# issue_comments

html_url: https://github.com/simonw/datasette/issues/215#issuecomment-507929913
issue_url: https://api.github.com/repos/simonw/datasette/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDUwNzkyOTkxMw==
created_at: 2019-07-03T04:08:28Z
updated_at: 2019-07-03T04:08:28Z
author_association: OWNER
body:
  I just closed #520 which means this is now technically possible.

  But... doing it using the new `asgi_wrapper` hook https://datasette.readthedocs.io/en/latest/plugins.html#asgi-wrapper-datasette isn't particularly obvious.

  I'm going to leave this ticket open for the moment. I think I need at least one example plugin to show that this approach is good enough - and it's still quite possible that I'll add an extra, easier hook for this. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/520#issuecomment-507935352
issue_url: https://api.github.com/repos/simonw/datasette/issues/520
node_id: MDEyOklzc3VlQ29tbWVudDUwNzkzNTM1Mg==
created_at: 2019-07-03T04:40:34Z
updated_at: 2019-07-03T04:40:34Z
author_association: OWNER
body:
  Documentation for the new hook: https://datasette.readthedocs.io/en/latest/plugins.html#asgi-wrapper-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/538#issuecomment-508224962
issue_url: https://api.github.com/repos/simonw/datasette/issues/538
node_id: MDEyOklzc3VlQ29tbWVudDUwODIyNDk2Mg==
created_at: 2019-07-03T19:24:32Z
updated_at: 2019-07-03T19:24:32Z
author_association: OWNER
body:
  Initial syntax suggestion:
  ```json
  {
      "title": "datasette-auth-github demo",
      "plugins": {
          "datasette-auth-github": {
              "client_id": "986f5d837b45e32ee6dd",
              "client_secret": {"$env": "GITHUB_CLIENT_SECRET"}
          }
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/538#issuecomment-508225524
issue_url: https://api.github.com/repos/simonw/datasette/issues/538
node_id: MDEyOklzc3VlQ29tbWVudDUwODIyNTUyNA==
created_at: 2019-07-03T19:26:32Z
updated_at: 2019-07-03T19:26:32Z
author_association: OWNER
body:
  Another useful option is the ability to load secrets from a file. This allows the file to have permissions set on it to only be read by the Datasette user. It also interacts well with the Kubernetes secrets mechanism, which is file-based.
  ```json
  {
      "plugins": {
          "datasette-auth-github": {
              "client_id": "986f5d837b45e32ee6dd",
              "client_secret": {"$file": "/secrets/github-client-secret"}
          }
      }
  }

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/539#issuecomment-508343359
issue_url: https://api.github.com/repos/simonw/datasette/issues/539
node_id: MDEyOklzc3VlQ29tbWVudDUwODM0MzM1OQ==
created_at: 2019-07-04T05:27:47Z
updated_at: 2019-07-04T05:27:47Z
author_association: OWNER
body:
  For some reason the way I have set these tests up keeps throwing [variations on this error](https://travis-ci.org/simonw/datasette/builds/554055865):
  ```
  self = <click.testing.CliRunner object at 0x7fccc0d940f0>
      @contextlib.contextmanager
      def isolated_filesystem(self):
          """A context manager that creates a temporary folder and changes
          the current working directory to it for isolated filesystem tests.
          """
  >       cwd = os.getcwd()
  E       FileNotFoundError: [Errno 2] No such file or directory
  /home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/click/testing.py:364: FileNotFoundError
  ```
  After much trial and error the cause appears to be the way I was trying to create custom metadata like so:
  ```python
   def test_plugin_config_env():
      os.environ["FOO_ENV"] = "FROM_ENVIRONMENT"
      for client in make_app_client(
          metadata={"plugins": {"env-plugin": {"foo": {"$env": "FOO_ENV"}}}}
      ):
          assert {"foo": "FROM_ENVIRONMENT"} == client.ds.plugin_config("env-plugin")
      del os.environ["FOO_ENV"]
  ```
  It wasnt't the `os.environ` manipulation OR the `tempfile` stuff - it appeared to be the usage of `make_app_client()`. When I stopped trying to do that (and moved my test configuration into `METADATA` in the `fixtures.py` module) the tests started passing again.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/539#issuecomment-508343497
issue_url: https://api.github.com/repos/simonw/datasette/issues/539
node_id: MDEyOklzc3VlQ29tbWVudDUwODM0MzQ5Nw==
created_at: 2019-07-04T05:28:30Z
updated_at: 2019-07-04T05:28:30Z
author_association: OWNER
body:
  This seems to have fixed the weird errors: https://github.com/simonw/datasette/pull/539/commits/fd6164b03ebe450a9a00df2e5be2dc7bbfbd9a3f

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/538#issuecomment-508346394
issue_url: https://api.github.com/repos/simonw/datasette/issues/538
node_id: MDEyOklzc3VlQ29tbWVudDUwODM0NjM5NA==
created_at: 2019-07-04T05:43:24Z
updated_at: 2019-07-04T05:43:24Z
author_association: OWNER
body:
  Re-opening this because I messed it up: the secret options are still visible in `/-/metadata` because I mutate the dictionary in place!

  https://github.com/simonw/datasette/blob/a2d45931935f6bb73605a94afedf9e78308c95d6/datasette/app.py#L273-L279

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/498#issuecomment-508590397
issue_url: https://api.github.com/repos/simonw/datasette/issues/498
node_id: MDEyOklzc3VlQ29tbWVudDUwODU5MDM5Nw==
created_at: 2019-07-04T23:34:41Z
updated_at: 2019-07-04T23:34:41Z
author_association: NONE
body:
  I'll take your suggestion and do this all in Javascript. Would I need to make a `static/` folder in my datasette project's root directory and make a custom `index.html` template that pulls from `static/js/search-all-fts.js`? Or would you suggest another way?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/540#issuecomment-508771246
issue_url: https://api.github.com/repos/simonw/datasette/issues/540
node_id: MDEyOklzc3VlQ29tbWVudDUwODc3MTI0Ng==
created_at: 2019-07-05T14:10:35Z
updated_at: 2019-07-05T14:10:35Z
author_association: OWNER
body:
  https://css-tricks.com/off-canvas-menu-with-css-target/ could be fun. See also https://medium.com/@heyoka/responsive-pure-css-off-canvas-hamburger-menu-aebc8d11d793

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/540#issuecomment-508806571
issue_url: https://api.github.com/repos/simonw/datasette/issues/540
node_id: MDEyOklzc3VlQ29tbWVudDUwODgwNjU3MQ==
created_at: 2019-07-05T16:14:55Z
updated_at: 2019-07-05T16:14:55Z
author_association: OWNER
body:
  I think I'll start with something ridiculously simple along these lines:

  <img width="482" alt="fixtures__roadside_attraction_characteristics" src="https://user-images.githubusercontent.com/9599/60734589-4dc63900-9f05-11e9-8603-34ed50ad0005.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/540#issuecomment-508819126
issue_url: https://api.github.com/repos/simonw/datasette/issues/540
node_id: MDEyOklzc3VlQ29tbWVudDUwODgxOTEyNg==
created_at: 2019-07-05T17:14:52Z
updated_at: 2019-07-05T17:14:52Z
author_association: OWNER
body:
  Having this be customized by a single plugin seems easy enough - define a named block region in the template, the plugin can then provide a template which extends the base template, over-rides the named block, adds some stuff, then calls [super](http://jinja.pocoo.org/docs/2.10/templates/#super-blocks) to include the default content.

  But... how would this work if there are multiple plugins that want to add to the menu area?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/540#issuecomment-508827948
issue_url: https://api.github.com/repos/simonw/datasette/issues/540
node_id: MDEyOklzc3VlQ29tbWVudDUwODgyNzk0OA==
created_at: 2019-07-05T17:56:57Z
updated_at: 2019-07-05T17:56:57Z
author_association: OWNER
body:
  For the moment I'm going to ignore the multiple plugins case. I shall revisit that at some point in the future, once I actually have multiple plugins that want to be able to customize the navigation at the same time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/540#issuecomment-508858510
issue_url: https://api.github.com/repos/simonw/datasette/issues/540
node_id: MDEyOklzc3VlQ29tbWVudDUwODg1ODUxMA==
created_at: 2019-07-05T20:43:20Z
updated_at: 2019-07-05T20:43:20Z
author_association: OWNER
body:
  https://latest.datasette.io/fixtures/attraction_characteristic/1 now looks like this:

  <img width="790" alt="fixtures__attraction_characteristic" src="https://user-images.githubusercontent.com/9599/60744452-d60b0500-9f2a-11e9-8ac6-b35aaaf8120c.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/540#issuecomment-508858729
issue_url: https://api.github.com/repos/simonw/datasette/issues/540
node_id: MDEyOklzc3VlQ29tbWVudDUwODg1ODcyOQ==
created_at: 2019-07-05T20:44:50Z
updated_at: 2019-07-05T20:44:50Z
author_association: OWNER
body:
  Here's an example custom `base.html` template that appears to have the desired effect:
  ```html+django
  {% extends "default:base.html" %}

  {% block extra_head %}
  <style type="text/css">
  .hd .logout {
      float: right;
      text-align: right;
      padding-left: 1em;
  }
  </style>
  {% endblock %}

  {% block nav %}
      {{ super() }}
      <p class="logout">
          <a href="/-/logout">Log out</a>
      </p>
  {% endblock %}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/541#issuecomment-508871981
issue_url: https://api.github.com/repos/simonw/datasette/issues/541
node_id: MDEyOklzc3VlQ29tbWVudDUwODg3MTk4MQ==
created_at: 2019-07-05T22:15:35Z
updated_at: 2019-07-05T22:15:35Z
author_association: OWNER
body:
  Example usage:
  ```python
  @hookimpl
  def extra_template_vars(request):
      return {
          "auth": request.scope.get("auth"),
      }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/541#issuecomment-508872137
issue_url: https://api.github.com/repos/simonw/datasette/issues/541
node_id: MDEyOklzc3VlQ29tbWVudDUwODg3MjEzNw==
created_at: 2019-07-05T22:16:57Z
updated_at: 2019-07-05T22:16:57Z
author_association: OWNER
body:
  It should allow for awaitables too, to allow for execution of SQL. For example:

  ```python
  @hookimpl
  def extra_template_vars(datasette):
      async def inner():
          first_db = list(datasette.databases.keys())[0]
          return {
              "sqlite_version": (
                  await datasette.execute(first_db, "select sqlite_version()")
              ).rows[0][0]
          }
      return inner
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/542#issuecomment-508873079
issue_url: https://api.github.com/repos/simonw/datasette/issues/542
node_id: MDEyOklzc3VlQ29tbWVudDUwODg3MzA3OQ==
created_at: 2019-07-05T22:25:33Z
updated_at: 2019-07-05T22:25:33Z
author_association: OWNER
body:
  I'm getting this weird failure again:
  ```
  self = <click.testing.CliRunner object at 0x7f4295b7bd30>
      @contextlib.contextmanager
      def isolated_filesystem(self):
          """A context manager that creates a temporary folder and changes
          the current working directory to it for isolated filesystem tests.
          """
  >       cwd = os.getcwd()
  E       FileNotFoundError: [Errno 2] No such file or directory
  ```
  I saw this the other day too, but managed to avoid it by skipping any code that messed around with temporary files.

  Baffled as to why this is showing up now, as none of the changes I'm making here have any relevance to the way temporary files are created or the current working directory is set.

  I understand the root cause here to be that the working directory for the tests was deleted while the tests were still running, which would seem to indicate that something to do with a `tmpdir` fixture is at fault.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/542#issuecomment-508873216
issue_url: https://api.github.com/repos/simonw/datasette/issues/542
node_id: MDEyOklzc3VlQ29tbWVudDUwODg3MzIxNg==
created_at: 2019-07-05T22:27:00Z
updated_at: 2019-07-05T22:27:00Z
author_association: OWNER
body:
  I bet that error is something to do with this:
  https://github.com/simonw/datasette/blob/e06e083240522a281b280b6461e935bf4b47bdbc/tests/test_plugins.py#L195-L197

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/542#issuecomment-508875041
issue_url: https://api.github.com/repos/simonw/datasette/issues/542
node_id: MDEyOklzc3VlQ29tbWVudDUwODg3NTA0MQ==
created_at: 2019-07-05T22:43:53Z
updated_at: 2019-07-05T22:43:53Z
author_association: OWNER
body:
  I've confirmed that it's the `template_dir=` line quoted above. It isn't the fault of `pathlib` - if you hard-code in the path string like this I still get the same failures:
  ```
      template_dir = "/Users/simonw/Dropbox/Development/datasette/tests/test_templates"
      for client in make_app_client(
          template_dir=template_dir
      ):
  ```
  So for some reason setting `template_dir` to a concrete directory breaks those other tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/542#issuecomment-508876259
issue_url: https://api.github.com/repos/simonw/datasette/issues/542
node_id: MDEyOklzc3VlQ29tbWVudDUwODg3NjI1OQ==
created_at: 2019-07-05T22:55:27Z
updated_at: 2019-07-05T22:55:46Z
author_association: OWNER
body:
  Here's what `isolated_filesystem()` does:

  > A context manager that creates a temporary folder and changes the current working directory to it for isolated filesystem tests.

  https://github.com/pallets/click/issues/395 is a four year old issue suggesting allowing it to have a parent tmp directory passed to it, but since that issue was never closed I don't think it's going to help me.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/542#issuecomment-508876468
issue_url: https://api.github.com/repos/simonw/datasette/issues/542
node_id: MDEyOklzc3VlQ29tbWVudDUwODg3NjQ2OA==
created_at: 2019-07-05T22:57:32Z
updated_at: 2019-07-05T22:57:32Z
author_association: OWNER
body:
  This looks relevant: https://github.com/pytest-dev/pytest/issues/1235 - "pytest explodes if getcwd fails"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/543#issuecomment-508938015
issue_url: https://api.github.com/repos/simonw/datasette/issues/543
node_id: MDEyOklzc3VlQ29tbWVudDUwODkzODAxNQ==
created_at: 2019-07-06T16:23:56Z
updated_at: 2019-07-06T16:24:16Z
author_association: OWNER
body:
  Potential design:
  ```
  $ datasette publish --plugin-secret datasette-auth-github client_id xxx
  ```
  This would populate the following in `metadata.json`:
  ```json
  {
      "plugins": {
          "datasette-auth-github": {
              "client_id": {"$env": "DATASETTE_AUTH_GITHUB_CLIENT_ID"}
          }
      }
  }
  ```
  And would ensure that the environment variable `DATASETTE_AUTH_GITHUB_CLIENT_ID` is set to the correct value.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/543#issuecomment-508942349
issue_url: https://api.github.com/repos/simonw/datasette/issues/543
node_id: MDEyOklzc3VlQ29tbWVudDUwODk0MjM0OQ==
created_at: 2019-07-06T17:34:05Z
updated_at: 2019-07-06T17:34:05Z
author_association: OWNER
body:
  As soon as I have this working I can use it to ship a live demo of https://github.com/simonw/datasette-auth-github

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/543#issuecomment-508955449
issue_url: https://api.github.com/repos/simonw/datasette/issues/543
node_id: MDEyOklzc3VlQ29tbWVudDUwODk1NTQ0OQ==
created_at: 2019-07-06T21:41:28Z
updated_at: 2019-07-06T21:41:50Z
author_association: OWNER
body:
      $ datasette publish now fixtures.db \
          --branch=master \
          --alias datasette-auth-demo \
          --install=datasette-auth-github \
          --plugin-secret datasette-auth-github client_id 86e397f7fd7a54d26a3a \
          --plugin-secret datasette-auth-github client_secret ...

  https://datasette-auth-demo.now.sh/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/507#issuecomment-509013413
issue_url: https://api.github.com/repos/simonw/datasette/issues/507
node_id: MDEyOklzc3VlQ29tbWVudDUwOTAxMzQxMw==
created_at: 2019-07-07T16:31:57Z
updated_at: 2019-07-07T16:31:57Z
author_association: CONTRIBUTOR
body:
  Chrome and Firefox [both support headless screengrabs]( https://www.bleepingcomputer.com/news/software/chrome-and-firefox-can-take-screenshots-of-sites-from-the-command-line/) from command line, but I don't know how parameterised they can be?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/547#issuecomment-509031132
issue_url: https://api.github.com/repos/simonw/datasette/issues/547
node_id: MDEyOklzc3VlQ29tbWVudDUwOTAzMTEzMg==
created_at: 2019-07-07T21:11:57Z
updated_at: 2019-07-07T21:11:57Z
author_association: OWNER
body:
  These release notes should specifically call out https://github.com/simonw/datasette-auth-github and https://github.com/simonw/datasette-cors as example plugins.

  Should also make sure there are accompanying Glitch.com demos of these.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/551#issuecomment-509038681
issue_url: https://api.github.com/repos/simonw/datasette/issues/551
node_id: MDEyOklzc3VlQ29tbWVudDUwOTAzODY4MQ==
created_at: 2019-07-07T23:21:54Z
updated_at: 2019-07-07T23:21:54Z
author_association: OWNER
body:
  Here's where I removed the faceting code: https://github.com/simonw/datasette/commit/c5542abba564a0b320a1201a8cc85b48c743005d - step one will be reverting that commit back again.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/551#issuecomment-509038953
issue_url: https://api.github.com/repos/simonw/datasette/issues/551
node_id: MDEyOklzc3VlQ29tbWVudDUwOTAzODk1Mw==
created_at: 2019-07-07T23:26:45Z
updated_at: 2019-07-07T23:26:45Z
author_association: OWNER
body:
  The code for this was getting a little bit gnarly. I need to come up with a rock-solid design for the feature first (solving #495), then carefully build it with comprehensive tests to make sure it all makes sense.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/498#issuecomment-509042334
issue_url: https://api.github.com/repos/simonw/datasette/issues/498
node_id: MDEyOklzc3VlQ29tbWVudDUwOTA0MjMzNA==
created_at: 2019-07-08T00:18:29Z
updated_at: 2019-07-08T00:18:29Z
author_association: NONE
body:
  @simonw I made this primitive search that I've put in my Datasette project's custom templates directory: https://gist.github.com/chrismp/e064b41f08208a6f9a93150a23cf7e03

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/544#issuecomment-509050775
issue_url: https://api.github.com/repos/simonw/datasette/issues/544
node_id: MDEyOklzc3VlQ29tbWVudDUwOTA1MDc3NQ==
created_at: 2019-07-08T01:43:15Z
updated_at: 2019-07-08T01:43:15Z
author_association: OWNER
body:
  Deployed the Cloud Run demo like this:
  ```
  datasette publish cloudrun fixtures.db \
    --branch=master \
    --install=datasette-auth-github \
    -n datasette-auth-demo \
    --service datasette-auth-demo \
    --show-files \
    --plugin-secret datasette-auth-github client_id 7524167294949f7886fc \
    --plugin-secret datasette-auth-github client_secret ...
  ```
  https://datasette-auth-demo-j7hipcg4aq-uc.a.run.app/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/552#issuecomment-509051355
issue_url: https://api.github.com/repos/simonw/datasette/issues/552
node_id: MDEyOklzc3VlQ29tbWVudDUwOTA1MTM1NQ==
created_at: 2019-07-08T01:47:30Z
updated_at: 2019-07-08T01:47:30Z
author_association: OWNER
body:
  This is a good opportunity to clean up some duplicate code too. This code is duplicated in a few places:

  https://github.com/simonw/datasette/blob/741760d05c6836396756ff6be44a4d375a83a06f/datasette/publish/heroku.py#L65-L87

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/478#issuecomment-509066513
issue_url: https://api.github.com/repos/simonw/datasette/issues/478
node_id: MDEyOklzc3VlQ29tbWVudDUwOTA2NjUxMw==
created_at: 2019-07-08T03:30:41Z
updated_at: 2019-07-08T03:30:41Z
author_association: OWNER
body:
  This worked as intended - thanks @glasnt!

  https://travis-ci.org/simonw/datasette/builds/555580006

  <img width="904" alt="Build__1058_-_simonw_datasette_-_Travis_CI" src="https://user-images.githubusercontent.com/9599/60781155-0b178300-a0f6-11e9-91c8-219334a1c213.png">

  The release has been deployed to PyPI even while the Docker image is still being built.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/547#issuecomment-509066738
issue_url: https://api.github.com/repos/simonw/datasette/issues/547
node_id: MDEyOklzc3VlQ29tbWVudDUwOTA2NjczOA==
created_at: 2019-07-08T03:31:54Z
updated_at: 2019-07-08T03:31:54Z
author_association: OWNER
body:
  Done! https://github.com/simonw/datasette/commit/fb7ee8e0ad59a15083234a48e935525f6e7257dd

  https://datasette.readthedocs.io/en/latest/changelog.html#v0-29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-509154312
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwOTE1NDMxMg==
created_at: 2019-07-08T09:36:25Z
updated_at: 2019-07-08T09:40:33Z
author_association: NONE
body:
  @chrismp: Ports 1024 and under are privileged and can usually only be bound by a root or supervisor user, so it makes sense if you're running as the user `chris` that port 8000 works but 80 doesn't.

  See [this generic question-and-answer](https://superuser.com/questions/710253/allow-non-root-process-to-bind-to-port-80-and-443) and [this systemd question-and-answer](https://stackoverflow.com/questions/40865775/linux-systemd-service-on-port-80) for more information about ways to skin this cat. Without knowing your specific circumstances, either extending those privileges to that service/executable/user, proxying them through something like nginx or indeed looking at what the nginx systemd job has to do to listen at port 80 all sound like good ways to start.

  At this point, this is more generic systemd/Linux support than a Datasette issue, which is why a complete rando like me is able to contribute anything. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/553#issuecomment-509277577
issue_url: https://api.github.com/repos/simonw/datasette/issues/553
node_id: MDEyOklzc3VlQ29tbWVudDUwOTI3NzU3Nw==
created_at: 2019-07-08T15:38:26Z
updated_at: 2019-07-08T15:38:26Z
author_association: OWNER
body:
  One thing that's clear is I need to add a demo to the fixtures that illustrates the feature, ideally with a thousand generated rows in it to give a better example of how it works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/553#issuecomment-509277853
issue_url: https://api.github.com/repos/simonw/datasette/issues/553
node_id: MDEyOklzc3VlQ29tbWVudDUwOTI3Nzg1Mw==
created_at: 2019-07-08T15:39:08Z
updated_at: 2019-07-08T15:39:15Z
author_association: OWNER
body:
  My reply on Twitter:

  https://twitter.com/simonw/status/1148254258900262913
  > Huh I hadn't thought about order-by-date - that's really useful. Not sure what I'll do for data that's spread over e.g. 1,000 days (other facets only return up to 30 data points right now) - maybe 30 most recent plus a link to view more than that?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/553#issuecomment-509278616
issue_url: https://api.github.com/repos/simonw/datasette/issues/553
node_id: MDEyOklzc3VlQ29tbWVudDUwOTI3ODYxNg==
created_at: 2019-07-08T15:41:05Z
updated_at: 2019-07-08T15:41:55Z
author_association: OWNER
body:
  Having a "more link" is very interesting. I could do that by generating a custom SQL query link, but that wouldn't play well in instances that disable custom SQL.

  This may be a case where a new URL is needed under the proposed `/database/table/-/namespace` namespace - #296

  Compatibility with `datasette-vega` is useful, since then anyone can get a chart by starting with a facet of some sort.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/531#issuecomment-509279790
issue_url: https://api.github.com/repos/simonw/datasette/issues/531
node_id: MDEyOklzc3VlQ29tbWVudDUwOTI3OTc5MA==
created_at: 2019-07-08T15:43:57Z
updated_at: 2019-07-08T15:43:57Z
author_association: OWNER
body:
  Including foreign keys here will let me build a plugin for visualizing table relationships based on https://observablehq.com/@d3/force-directed-graph

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-509431603
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwOTQzMTYwMw==
created_at: 2019-07-08T23:39:52Z
updated_at: 2019-07-08T23:39:52Z
author_association: NONE
body:
  In `datasette.service`, I edited

  ```
  User=chris
  ```

  To...

  ```
  User=root
  ```

  It worked. I can access `http://my-server.com`. I hope this is safe. Thanks for all the help, everyone.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-509480062
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwOTQ4MDA2Mg==
created_at: 2019-07-09T04:15:11Z
updated_at: 2019-07-09T04:15:11Z
author_association: OWNER
body:
  Running as root isn't ideal because it means that if there are any security vulnerabilities in Datasette an attacker could use them to execute any command they like as root on your machine.

  I'm moderately confident there aren't any vulnerabilities like that, but I'm definite not 100% certain!

  My recommendation is to run Datasette on 127.0.0.1 port 8001 and then have nginx proxy port 80 to it. See https://github.com/simonw/datasette/issues/514#issuecomment-504663201 for suggested nginx configuration.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/554#issuecomment-509629331
issue_url: https://api.github.com/repos/simonw/datasette/issues/554
node_id: MDEyOklzc3VlQ29tbWVudDUwOTYyOTMzMQ==
created_at: 2019-07-09T12:51:35Z
updated_at: 2019-07-09T12:51:35Z
author_association: CONTRIBUTOR
body:
  I wanted to add a test for it too, but I've realized it's impossible to test a server process as we cannot get its exit code.

  ```python
  # tests/test_cli.py
  def test_static_mounts_on_windows():
      if sys.platform != "win32":
          return
      runner = CliRunner()
      result = runner.invoke(
          cli, ["serve", "--static", r"s:C:\\"]
      )
      assert result.exit_code == 0
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/37#issuecomment-509681590
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/37
node_id: MDEyOklzc3VlQ29tbWVudDUwOTY4MTU5MA==
created_at: 2019-07-09T15:07:12Z
updated_at: 2021-06-22T18:17:53Z
author_association: OWNER
body:
  Here's a magic incantation for generating types detected through running the tests with https://github.com/Instagram/MonkeyType

  ```
  pip install pytest-monkeytype
  pytest --monkeytype-output=./monkeytype.sqlite3
  monkeytype list-modules
  monkeytype apply sqlite_utils.utils
  monkeytype apply sqlite_utils.cli
  monkeytype apply sqlite_utils.db
  ```

  Here's the result: https://github.com/simonw/sqlite-utils/commit/d18c694fc25b7dd3d76e250c77ddf56d10ddf935

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/37#issuecomment-509686432
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/37
node_id: MDEyOklzc3VlQ29tbWVudDUwOTY4NjQzMg==
created_at: 2019-07-09T15:18:49Z
updated_at: 2019-07-09T15:18:49Z
author_association: OWNER
body:
  Interestingly if I run that against my `monkeytype` branch I get the following errors (subset):
  ```
  _______________________________________________________________________________________ mypy db.py ________________________________________________________________________________________
  sqlite_utils/db.py:15: error: No library stub file for module 'numpy'
  sqlite_utils/db.py:15: note: (Stub files are from https://github.com/python/typeshed)
  sqlite_utils/db.py:197: error: Incompatible types in assignment (expression has type "Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], Dict[<nothing>, <nothing>]]", variable has type "Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]")
  sqlite_utils/db.py:204: error: Item "None" of "Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]" has no attribute "__iter__" (not iterable)
  sqlite_utils/db.py:206: error: Argument 1 to "set" has incompatible type "Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]"; expected "Iterable[str]"
  sqlite_utils/db.py:211: error: Item "None" of "Optional[Tuple[str, str, str]]" has no attribute "index"
  sqlite_utils/db.py:211: error: Unsupported right operand type for in ("Optional[Tuple[str, str, str]]")
  sqlite_utils/db.py:232: error: Unsupported right operand type for in ("Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]")
  sqlite_utils/db.py:234: error: Value of type "Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]" is not indexable
  sqlite_utils/db.py:307: error: Need type annotation for 'table_sql' (hint: "table_sql: Dict[<type>, <type>] = ...")
  sqlite_utils/db.py:427: error: Item "int" of "Union[int, None, str]" has no attribute "startswith"
  sqlite_utils/db.py:427: error: Item "None" of "Union[int, None, str]" has no attribute "startswith"
  sqlite_utils/db.py:434: error: Incompatible types in assignment (expression has type "List[Any]", target has type "int")
  sqlite_utils/db.py:438: error: Incompatible types in assignment (expression has type "object", target has type "int")
  sqlite_utils/db.py:670: error: Need type annotation for 'all_column_types' (hint: "all_column_types: Dict[<type>, <type>] = ...")
  sqlite_utils/db.py:757: error: Incompatible types in assignment (expression has type "List[Any]", variable has type "chain[Any]")
  sqlite_utils/db.py:761: error: Argument 1 to "create" of "Table" has incompatible type "Dict[str, Union[Type[str], Type[int], Type[bool], Type[time], Type[float]]]"; expected "Dict[str, Union[Type[str], Type[int], Type[bool], Type[time]]]"
  sqlite_utils/db.py:770: error: Need type annotation for 'all_columns' (hint: "all_columns: Set[<type>] = ...")
  sqlite_utils/db.py:773: error: Incompatible types in assignment (expression has type "List[Any]", variable has type "Set[Any]")
  sqlite_utils/db.py:775: error: "Set[Any]" has no attribute "insert"
  sqlite_utils/db.py:797: error: Need type annotation for 'values' (hint: "values: List[<type>] = ...")
  sqlite_utils/db.py:811: error: Argument 1 to "add_missing_columns" of "Table" has incompatible type "chain[Any]"; expected "Union[List[Union[Dict[str, Union[int, str]], Dict[str, Union[int, str, List[str]]]]], List[Dict[str, Union[str, int, float]]], List[Dict[str, Union[int, str]]]]"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/554#issuecomment-510542280
issue_url: https://api.github.com/repos/simonw/datasette/issues/554
node_id: MDEyOklzc3VlQ29tbWVudDUxMDU0MjI4MA==
created_at: 2019-07-11T15:47:31Z
updated_at: 2019-07-11T15:47:31Z
author_association: OWNER
body:
  Thanks for this!

  The tests are failing for Python 3.5 right now which is strange. https://travis-ci.org/simonw/datasette/jobs/556272017

  One failure looks like this:
  ```
  app_client = <tests.fixtures.TestClient object at 0x7fb7803285f8>
      def test_static(app_client):
          response = app_client.get("/-/static/app2.css")
  >       assert response.status == 404
  E       assert 500 == 404
  E        +  where 500 = <tests.fixtures.TestResponse object at 0x7fb780328a58>.status
  /home/travis/build/simonw/datasette/tests/test_html.py:56: AssertionError
  ----------------------------- Captured stderr call -----------------------------
  Traceback (most recent call last):
    File "/home/travis/build/simonw/datasette/datasette/utils/asgi.py", line 100, in __call__
      return await view(new_scope, receive, send)
    File "/home/travis/build/simonw/datasette/datasette/utils/asgi.py", line 303, in inner_static
      full_path = (Path(root_path) / path).resolve().absolute()
    File "/opt/python/3.5.6/lib/python3.5/pathlib.py", line 1109, in resolve
      s = self._flavour.resolve(self)
    File "/opt/python/3.5.6/lib/python3.5/pathlib.py", line 330, in resolve
      return _resolve(base, str(path)) or sep
    File "/opt/python/3.5.6/lib/python3.5/pathlib.py", line 315, in _resolve
      target = accessor.readlink(newpath)
    File "/opt/python/3.5.6/lib/python3.5/pathlib.py", line 422, in readlink
      return os.readlink(path)
  FileNotFoundError: [Errno 2] No such file or directory: '/home/travis/build/simonw/datasette/datasette/static/app2.css'
  ```
  Maybe an exception was renamed between 3.5 and 3.6?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/511#issuecomment-510559337
issue_url: https://api.github.com/repos/simonw/datasette/issues/511
node_id: MDEyOklzc3VlQ29tbWVudDUxMDU1OTMzNw==
created_at: 2019-07-11T16:31:06Z
updated_at: 2019-07-11T16:31:06Z
author_association: OWNER
body:
  @abdusco you mentioned Windows in #554 - can you confirm that Datasette is Windows compatible as of v0.29?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/511#issuecomment-510559836
issue_url: https://api.github.com/repos/simonw/datasette/issues/511
node_id: MDEyOklzc3VlQ29tbWVudDUxMDU1OTgzNg==
created_at: 2019-07-11T16:32:27Z
updated_at: 2019-07-11T16:32:27Z
author_association: OWNER
body:
  Travis CI supports Windows - Uvicorn is using it here: https://github.com/encode/uvicorn/blob/15c4e9cfe2d996bd73ed6b542e44160f620b9519/.travis.yml

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/557#issuecomment-510562059
issue_url: https://api.github.com/repos/simonw/datasette/issues/557
node_id: MDEyOklzc3VlQ29tbWVudDUxMDU2MjA1OQ==
created_at: 2019-07-11T16:38:27Z
updated_at: 2019-07-11T16:38:27Z
author_association: OWNER
body:
  Weird, Travis didn't even attempt to start a build for this branch/PR. Maybe something wrong with my `.travis.yml`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/511#issuecomment-510730200
issue_url: https://api.github.com/repos/simonw/datasette/issues/511
node_id: MDEyOklzc3VlQ29tbWVudDUxMDczMDIwMA==
created_at: 2019-07-12T03:23:22Z
updated_at: 2019-07-12T03:23:22Z
author_association: CONTRIBUTOR
body:
  @simonw yes it works fine on Windows, but test suite doesn't run properly, for that I had to use WSL

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/558#issuecomment-511052285
issue_url: https://api.github.com/repos/simonw/datasette/issues/558
node_id: MDEyOklzc3VlQ29tbWVudDUxMTA1MjI4NQ==
created_at: 2019-07-12T22:26:35Z
updated_at: 2019-07-14T01:44:32Z
author_association: OWNER
body:
  Thanks, this is really useful feedback. I'll take a look at this and see if I can figure out how the new ASGI/Uvicorn stuff differs from Sanic here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/558#issuecomment-511052658
issue_url: https://api.github.com/repos/simonw/datasette/issues/558
node_id: MDEyOklzc3VlQ29tbWVudDUxMTA1MjY1OA==
created_at: 2019-07-12T22:28:53Z
updated_at: 2019-07-12T22:28:53Z
author_association: OWNER
body:
  Are you definitely running the most recent release of uvicorn? The `raw_path` ASGI variable is only available in 0.8.1 or later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/558#issuecomment-511168964
issue_url: https://api.github.com/repos/simonw/datasette/issues/558
node_id: MDEyOklzc3VlQ29tbWVudDUxMTE2ODk2NA==
created_at: 2019-07-14T02:57:53Z
updated_at: 2019-07-14T02:57:53Z
author_association: OWNER
body:
  I think I've fixed it. Here's a demo: https://latest.datasette.io/fixtures/%F0%9D%90%9C%F0%9D%90%A2%F0%9D%90%AD%F0%9D%90%A2%F0%9D%90%9E%F0%9D%90%AC

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/560#issuecomment-511170166
issue_url: https://api.github.com/repos/simonw/datasette/issues/560
node_id: MDEyOklzc3VlQ29tbWVudDUxMTE3MDE2Ng==
created_at: 2019-07-14T03:31:42Z
updated_at: 2019-07-14T03:31:42Z
author_association: OWNER
body:
  ```
  TypeError: textarea is null[Learn More] codemirror-5.31.0.js:9508:3
      fromTextArea http://127.0.0.1:8001/-/static/codemirror-5.31.0.js:9508
      <anonymous> http://127.0.0.1:8001/fixtures:286
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/560#issuecomment-511170264
issue_url: https://api.github.com/repos/simonw/datasette/issues/560
node_id: MDEyOklzc3VlQ29tbWVudDUxMTE3MDI2NA==
created_at: 2019-07-14T03:34:24Z
updated_at: 2019-07-14T03:34:24Z
author_association: OWNER
body:
  Here's why:

  https://github.com/simonw/datasette/blob/6abe6faff6b035e9334dd05f8c741ae9b7a47440/datasette/templates/_codemirror_foot.html#L2

  This works on the `query.html` template:

  https://github.com/simonw/datasette/blob/6abe6faff6b035e9334dd05f8c741ae9b7a47440/datasette/templates/query.html#L38

  But fails on the `database.html` template:

  https://github.com/simonw/datasette/blob/6abe6faff6b035e9334dd05f8c741ae9b7a47440/datasette/templates/database.html#L28

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/35#issuecomment-511216484
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/35
node_id: MDEyOklzc3VlQ29tbWVudDUxMTIxNjQ4NA==
created_at: 2019-07-14T16:26:49Z
updated_at: 2019-07-14T16:26:49Z
author_association: OWNER
body:
  The last touched primary key should go in `last_pk`.

  https://github.com/simonw/sqlite-utils/blob/ba6bd47f15c154e348c3c85c05858053a4ab7bab/docs/python-api.rst#L126

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/35#issuecomment-511216664
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/35
node_id: MDEyOklzc3VlQ29tbWVudDUxMTIxNjY2NA==
created_at: 2019-07-14T16:29:22Z
updated_at: 2019-07-14T16:29:22Z
author_association: OWNER
body:
  Should this support both dictionaries AND keyword arguments?
  ```python
  db["events_venues"].update((3, 2), custom_label="Label")
  ```
  Pros: this is quite nice to look at.
  Cons: would have to continue supporting dictionaries as well, since otherwise you won't be able to update columns with names that are not valid Python identifiers. So now there are two ways.

  I'm going to stick with dictionaries only for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/28#issuecomment-511259493
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/28
node_id: MDEyOklzc3VlQ29tbWVudDUxMTI1OTQ5Mw==
created_at: 2019-07-15T02:37:20Z
updated_at: 2019-07-15T02:37:20Z
author_association: OWNER
body:
  I've changed my mind on this one. I don't think the docs will read particularly well if they try to cover both of these things at once. I could set up links between the two, but actually trying to combine them feels like it will result in documentation that is less useful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/557#issuecomment-511624403
issue_url: https://api.github.com/repos/simonw/datasette/issues/557
node_id: MDEyOklzc3VlQ29tbWVudDUxMTYyNDQwMw==
created_at: 2019-07-16T01:07:27Z
updated_at: 2019-07-16T01:10:24Z
author_association: OWNER
body:
  Woohoo! That got the tests running on Windows. They're pretty slow there unfortunately... and I'm seeing some failures.

  https://travis-ci.org/simonw/datasette/jobs/559205729 is the first run against Windows.
  ```

  collected 353 items

  tests\test_cli.py ...                                                    [  0%]
  tests\test_black.py .                                                    [  1%]
  tests\test_api.py ..........F........................................... [ 16%]
  .....s..F..................FF...............F......                      [ 30%]
  tests\test_cli.py F                                                      [ 31%]
  tests\test_csv.py .......                                                [ 33%]
  tests\test_database.py ....                                              [ 34%]
  tests\test_docs.py ..................................................... [ 49%]
  ....                                                                     [ 50%]
  tests\test_facets.py .....ss.                                            [ 52%]
  tests\test_filters.py ...........                                        [ 55%]
  tests\test_html.py ...FF.....FF..............................F.FF.F..... [ 70%]
  .....F                                                                   [ 72%]
  tests\test_plugins.py ...............F                                   [ 77%]
  tests\test_publish_cloudrun.py ..F.                                      [ 78%]
  tests\test_publish_heroku.py ...FF                                       [ 79%]
  tests\test_publish_now.py ...FF..                                        [ 81%]
  tests\test_utils.py .............................F...........FFF........ [ 96%]
  .............E                                                           [100%]

  =================================== ERRORS ====================================
  ________ ERROR at teardown of test_format_bytes[1099511627776-1.0 TB] _________

      @pytest.fixture(scope="session")
      def app_client():
  >       yield from make_app_client()

  ..\..\..\..\..\build\simonw\datasette\tests\fixtures.py:161: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmp_tkydmmd'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D0230CAF8>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmp_tkydmmd\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  ================================== FAILURES ===================================
  _____________________________ test_allow_sql_off ______________________________

      def test_allow_sql_off():
  >       for client in make_app_client(config={"allow_sql": False}):

  ..\..\..\..\build\simonw\datasette\tests\test_api.py:600: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmp6pkccymo'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D0261FD38>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmp6pkccymo\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  __________ test_table_filter_extra_where_disabled_if_no_sql_allowed ___________

      def test_table_filter_extra_where_disabled_if_no_sql_allowed():
  >       for client in make_app_client(config={"allow_sql": False}):

  ..\..\..\..\build\simonw\datasette\tests\test_api.py:1050: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpfa3zhjsq'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D025E0B88>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpfa3zhjsq\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  ____________________________ test_allow_facet_off _____________________________

      def test_allow_facet_off():
  >       for client in make_app_client(config={"allow_facet": False}):

  ..\..\..\..\build\simonw\datasette\tests\test_api.py:1454: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpvap5inq5'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D025DAAF8>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpvap5inq5\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  ___________________________ test_suggest_facets_off ___________________________

      def test_suggest_facets_off():
  >       for client in make_app_client(config={"suggest_facets": False}):

  ..\..\..\..\build\simonw\datasette\tests\test_api.py:1461: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmp7hn1_v44'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D0260C5E8>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmp7hn1_v44\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  ________________________ test_config_force_https_urls _________________________

      def test_config_force_https_urls():
  >       for client in make_app_client(config={"force_https_urls": True}):

  ..\..\..\..\build\simonw\datasette\tests\test_api.py:1589: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpu3lq69xr'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D02825EE8>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpu3lq69xr\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  ________ test_serve_with_inspect_file_prepopulates_table_counts_cache _________

      def test_serve_with_inspect_file_prepopulates_table_counts_cache():
          inspect_data = {"fixtures": {"tables": {"hithere": {"count": 44}}}}
  >       for client in make_app_client(inspect_data=inspect_data, is_immutable=True):

  ..\..\..\..\build\simonw\datasette\tests\test_cli.py:36: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpo65sp2ig'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D0240DC18>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpo65sp2ig\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  _____________________________ test_static_mounts ______________________________

      def test_static_mounts():
          for client in make_app_client(
  >           static_mounts=[("custom-static", str(pathlib.Path(__file__).parent))]
          ):

  ..\..\..\..\build\simonw\datasette\tests\test_html.py:64: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpqv82mm64'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D0294FD38>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpqv82mm64\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  ---------------------------- Captured stderr call -----------------------------
  Traceback (most recent call last):
    File "C:\Users\travis\build\simonw\datasette\datasette\utils\asgi.py", line 100, in __call__
      return await view(new_scope, receive, send)
    File "C:\Users\travis\build\simonw\datasette\datasette\utils\asgi.py", line 315, in inner_static
      await asgi_send_file(send, full_path, chunk_size=chunk_size)
    File "C:\Users\travis\build\simonw\datasette\datasette\utils\asgi.py", line 296, in asgi_send_file
      {"type": "http.response.body", "body": chunk, "more_body": more_body}
    File "c:\python37\lib\site-packages\aiofiles\base.py", line 92, in __aexit__
      yield from self._obj.close()
    File "c:\python37\lib\site-packages\aiofiles\threadpool\utils.py", line 36, in method
      return (yield from self._loop.run_in_executor(self._executor, cb))
  concurrent.futures._base.CancelledError
  __________________________ test_memory_database_page __________________________

      def test_memory_database_page():
  >       for client in make_app_client(memory=True):

  ..\..\..\..\build\simonw\datasette\tests\test_html.py:75: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmp6cagtj6v'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D040A2AF8>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmp6cagtj6v\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  _________________________ test_table_cell_truncation __________________________

      def test_table_cell_truncation():
  >       for client in make_app_client(config={"truncate_cells_html": 5}):

  ..\..\..\..\build\simonw\datasette\tests\test_html.py:123: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmp1zz78pre'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D03DFD5E8>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmp1zz78pre\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  _______________________ test_row_page_does_not_truncate _______________________

      def test_row_page_does_not_truncate():
  >       for client in make_app_client(config={"truncate_cells_html": 5}):

  ..\..\..\..\build\simonw\datasette\tests\test_html.py:148: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpsukb573e'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D023B9AF8>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpsukb573e\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  ________________ test_database_download_allowed_for_immutable _________________

      def test_database_download_allowed_for_immutable():
  >       for client in make_app_client(is_immutable=True):

  ..\..\..\..\build\simonw\datasette\tests\test_html.py:755: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpcivpcoop'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D02666A68>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpcivpcoop\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  ---------------------------- Captured stderr call -----------------------------
  Traceback (most recent call last):
    File "C:\Users\travis\build\simonw\datasette\datasette\utils\asgi.py", line 100, in __call__
      return await view(new_scope, receive, send)
    File "C:\Users\travis\build\simonw\datasette\datasette\utils\asgi.py", line 174, in view
      await response.asgi_send(send)
    File "C:\Users\travis\build\simonw\datasette\datasette\utils\asgi.py", line 383, in asgi_send
      return await asgi_send_file(send, self.filepath, content_type=self.content_type)
    File "C:\Users\travis\build\simonw\datasette\datasette\utils\asgi.py", line 296, in asgi_send_file
      {"type": "http.response.body", "body": chunk, "more_body": more_body}
    File "c:\python37\lib\site-packages\aiofiles\base.py", line 92, in __aexit__
      yield from self._obj.close()
    File "c:\python37\lib\site-packages\aiofiles\threadpool\utils.py", line 36, in method
      return (yield from self._loop.run_in_executor(self._executor, cb))
  concurrent.futures._base.CancelledError
  ________________ test_database_download_disallowed_for_memory _________________

      def test_database_download_disallowed_for_memory():
  >       for client in make_app_client(memory=True):

  ..\..\..\..\build\simonw\datasette\tests\test_html.py:773: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpvwpwrijf'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D02637D38>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpvwpwrijf\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  ___________________________ test_allow_download_off ___________________________

      def test_allow_download_off():
  >       for client in make_app_client(is_immutable=True, config={"allow_download": False}):

  ..\..\..\..\build\simonw\datasette\tests\test_html.py:782: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpmz0s6rqz'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D03EFA1F8>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpmz0s6rqz\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  _____________________________ test_allow_sql_off ______________________________

      def test_allow_sql_off():
  >       for client in make_app_client(config={"allow_sql": False}):

  ..\..\..\..\build\simonw\datasette\tests\test_html.py:800: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpmrr0edfk'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D03EFFD38>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpmrr0edfk\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  __________________________ test_custom_table_include __________________________

      def test_custom_table_include():
          for client in make_app_client(
  >           template_dir=str(pathlib.Path(__file__).parent / "test_templates")
          ):

  ..\..\..\..\build\simonw\datasette\tests\test_html.py:973: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpxcd2zp24'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D03FFBD38>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpxcd2zp24\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  ______________________ test_plugins_extra_template_vars _______________________

  restore_working_directory = None

      def test_plugins_extra_template_vars(restore_working_directory):
          for client in make_app_client(
  >           template_dir=str(pathlib.Path(__file__).parent / "test_templates")
          ):

  ..\..\..\..\build\simonw\datasette\tests\test_plugins.py:196: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\..\..\..\build\simonw\datasette\tests\fixtures.py:156: in make_app_client
      yield client
  c:\python37\lib\tempfile.py:805: in __exit__
      self.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:400: in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpkjbhg_ho'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D03DB6DC8>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
  >                   os.unlink(fullname)
  E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpkjbhg_ho\\fixtures.db'

  c:\python37\lib\shutil.py:398: PermissionError
  ____________________________ test_publish_cloudrun ____________________________

  mock_call = <MagicMock name='check_call' id='1636414304584'>
  mock_output = <MagicMock name='check_output' id='1636412936264'>
  mock_which = <MagicMock name='which' id='1636424648584'>

      @mock.patch("shutil.which")
      @mock.patch("datasette.publish.cloudrun.check_output")
      @mock.patch("datasette.publish.cloudrun.check_call")
      def test_publish_cloudrun(mock_call, mock_output, mock_which):
          mock_output.return_value = "myproject"
          mock_which.return_value = True
          runner = CliRunner()
          with runner.isolated_filesystem():
              open("test.db", "w").write("data")
              result = runner.invoke(cli.cli, ["publish", "cloudrun", "test.db"])
  >           assert 0 == result.exit_code
  E           AssertionError: assert 0 == 1
  E            +  where 1 = <Result PermissionError(13, 'The process cannot access the file because it is being used by another process')>.exit_code

  ..\..\..\..\build\simonw\datasette\tests\test_publish_cloudrun.py:37: AssertionError
  _____________________________ test_publish_heroku _____________________________

  mock_call = <MagicMock name='call' id='1636419273928'>
  mock_check_output = <MagicMock name='check_output' id='1636425565704'>
  mock_which = <MagicMock name='which' id='1636423528072'>

      @mock.patch("shutil.which")
      @mock.patch("datasette.publish.heroku.check_output")
      @mock.patch("datasette.publish.heroku.call")
      def test_publish_heroku(mock_call, mock_check_output, mock_which):
          mock_which.return_value = True
          mock_check_output.side_effect = lambda s: {
              "['heroku', 'plugins']": b"heroku-builds",
              "['heroku', 'apps:list', '--json']": b"[]",
              "['heroku', 'apps:create', 'datasette', '--json']": b'{"name": "f"}',
          }[repr(s)]
          runner = CliRunner()
          with runner.isolated_filesystem():
              open("test.db", "w").write("data")
              result = runner.invoke(cli.cli, ["publish", "heroku", "test.db"])
  >           assert 0 == result.exit_code, result.output
  E           AssertionError: 
  E           assert 0 == 1
  E            +  where 1 = <Result PermissionError(13, 'The process cannot access the file because it is being used by another process')>.exit_code

  ..\..\..\..\build\simonw\datasette\tests\test_publish_heroku.py:59: AssertionError
  _____________________ test_publish_heroku_plugin_secrets ______________________

  mock_call = <MagicMock name='call' id='1636418803912'>
  mock_check_output = <MagicMock name='check_output' id='1636422960392'>
  mock_which = <MagicMock name='which' id='1636421692616'>

      @mock.patch("shutil.which")
      @mock.patch("datasette.publish.heroku.check_output")
      @mock.patch("datasette.publish.heroku.call")
      def test_publish_heroku_plugin_secrets(mock_call, mock_check_output, mock_which):
          mock_which.return_value = True
          mock_check_output.side_effect = lambda s: {
              "['heroku', 'plugins']": b"heroku-builds",
              "['heroku', 'apps:list', '--json']": b"[]",
              "['heroku', 'apps:create', 'datasette', '--json']": b'{"name": "f"}',
          }[repr(s)]
          runner = CliRunner()
          with runner.isolated_filesystem():
              open("test.db", "w").write("data")
              result = runner.invoke(
                  cli.cli,
                  [
                      "publish",
                      "heroku",
                      "test.db",
                      "--plugin-secret",
                      "datasette-auth-github",
                      "client_id",
                      "x-client-id",
                  ],
              )
  >           assert 0 == result.exit_code, result.output
  E           AssertionError: 
  E           assert 0 == 1
  E            +  where 1 = <Result PermissionError(13, 'The process cannot access the file because it is being used by another process')>.exit_code

  ..\..\..\..\build\simonw\datasette\tests\test_publish_heroku.py:90: AssertionError
  ______________________________ test_publish_now _______________________________

  mock_run = <MagicMock name='run' id='1636418894600'>
  mock_which = <MagicMock name='which' id='1636418240136'>

      @mock.patch("shutil.which")
      @mock.patch("datasette.publish.now.run")
      def test_publish_now(mock_run, mock_which):
          mock_which.return_value = True
          runner = CliRunner()
          with runner.isolated_filesystem():
              open("test.db", "w").write("data")
              result = runner.invoke(cli.cli, ["publish", "nowv1", "test.db"])
  >           assert 0 == result.exit_code
  E           AssertionError: assert 0 == 1
  E            +  where 1 = <Result PermissionError(13, 'The process cannot access the file because it is being used by another process')>.exit_code

  ..\..\..\..\build\simonw\datasette\tests\test_publish_now.py:43: AssertionError
  ________________________ test_publish_now_force_token _________________________

  mock_run = <MagicMock name='run' id='1636447182536'>
  mock_which = <MagicMock name='which' id='1636420257416'>

      @mock.patch("shutil.which")
      @mock.patch("datasette.publish.now.run")
      def test_publish_now_force_token(mock_run, mock_which):
          mock_which.return_value = True
          runner = CliRunner()
          with runner.isolated_filesystem():
              open("test.db", "w").write("data")
              result = runner.invoke(
                  cli.cli, ["publish", "nowv1", "test.db", "--force", "--token=X"]
              )
  >           assert 0 == result.exit_code
  E           AssertionError: assert 0 == 1
  E            +  where 1 = <Result PermissionError(13, 'The process cannot access the file because it is being used by another process')>.exit_code

  ..\..\..\..\build\simonw\datasette\tests\test_publish_now.py:57: AssertionError
  _______________________________ test_detect_fts _______________________________

      def test_detect_fts():
          sql = """
          CREATE TABLE "Dumb_Table" (
            "TreeID" INTEGER,
            "qSpecies" TEXT
          );
          CREATE TABLE "Street_Tree_List" (
            "TreeID" INTEGER,
            "qSpecies" TEXT,
            "qAddress" TEXT,
            "SiteOrder" INTEGER,
            "qSiteInfo" TEXT,
            "PlantType" TEXT,
            "qCaretaker" TEXT
          );
          CREATE VIEW Test_View AS SELECT * FROM Dumb_Table;
          CREATE VIRTUAL TABLE "Street_Tree_List_fts" USING FTS4 ("qAddress", "qCaretaker", "qSpecies", content="Street_Tree_List");
          CREATE VIRTUAL TABLE r USING rtree(a, b, c);
          """
          conn = utils.sqlite3.connect(":memory:")
  >       conn.executescript(sql)
  E       sqlite3.OperationalError: no such module: rtree

  ..\..\..\..\build\simonw\datasette\tests\test_utils.py:182: OperationalError
  _______________ test_temporary_docker_directory_uses_hard_link ________________

      def test_temporary_docker_directory_uses_hard_link():
          with tempfile.TemporaryDirectory() as td:
              os.chdir(td)
              open("hello", "w").write("world")
              # Default usage of this should use symlink
              with utils.temporary_docker_directory(
                  files=["hello"],
                  name="t",
                  metadata=None,
                  extra_options=None,
                  branch=None,
                  template_dir=None,
                  plugins_dir=None,
                  static=[],
                  install=[],
                  spatialite=False,
                  version_note=None,
              ) as temp_docker:
                  hello = os.path.join(temp_docker, "hello")
                  assert "world" == open(hello).read()
                  # It should be a hard link
  >               assert 2 == os.stat(hello).st_nlink

  ..\..\..\..\..\build\simonw\datasette\tests\test_utils.py:239: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  c:\python37\lib\contextlib.py:119: in __exit__
      next(self.gen)
  ..\..\..\..\..\build\simonw\datasette\datasette\utils\__init__.py:397: in temporary_docker_directory
      tmp.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:395: in _rmtree_unsafe
      _rmtree_unsafe(fullname, onerror)
  c:\python37\lib\shutil.py:404: in _rmtree_unsafe
      onerror(os.rmdir, path, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmp0g07cby1\\t'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D0230C5E8>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
                      os.unlink(fullname)
                  except OSError:
                      onerror(os.unlink, fullname, sys.exc_info())
          try:
  >           os.rmdir(path)
  E           PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmp0g07cby1\\t'

  c:\python37\lib\shutil.py:402: PermissionError
  ________ test_temporary_docker_directory_uses_copy_if_hard_link_fails _________

  mock_link = <MagicMock name='link' id='1636414222984'>

      @patch("os.link")
      def test_temporary_docker_directory_uses_copy_if_hard_link_fails(mock_link):
          # Copy instead if os.link raises OSError (normally due to different device)
          mock_link.side_effect = OSError
          with tempfile.TemporaryDirectory() as td:
              os.chdir(td)
              open("hello", "w").write("world")
              # Default usage of this should use symlink
              with utils.temporary_docker_directory(
                  files=["hello"],
                  name="t",
                  metadata=None,
                  extra_options=None,
                  branch=None,
                  template_dir=None,
                  plugins_dir=None,
                  static=[],
                  install=[],
                  spatialite=False,
                  version_note=None,
              ) as temp_docker:
                  hello = os.path.join(temp_docker, "hello")
                  assert "world" == open(hello).read()
                  # It should be a copy, not a hard link
  >               assert 1 == os.stat(hello).st_nlink

  ..\..\..\..\..\build\simonw\datasette\tests\test_utils.py:266: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  c:\python37\lib\contextlib.py:119: in __exit__
      next(self.gen)
  ..\..\..\..\..\build\simonw\datasette\datasette\utils\__init__.py:397: in temporary_docker_directory
      tmp.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:395: in _rmtree_unsafe
      _rmtree_unsafe(fullname, onerror)
  c:\python37\lib\shutil.py:404: in _rmtree_unsafe
      onerror(os.rmdir, path, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpywj5p10g\\t'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D0230C558>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
                      os.unlink(fullname)
                  except OSError:
                      onerror(os.unlink, fullname, sys.exc_info())
          try:
  >           os.rmdir(path)
  E           PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmpywj5p10g\\t'

  c:\python37\lib\shutil.py:402: PermissionError
  _________________ test_temporary_docker_directory_quotes_args _________________

      def test_temporary_docker_directory_quotes_args():
          with tempfile.TemporaryDirectory() as td:
              os.chdir(td)
              open("hello", "w").write("world")
              with utils.temporary_docker_directory(
                  files=["hello"],
                  name="t",
                  metadata=None,
                  extra_options="--$HOME",
                  branch=None,
                  template_dir=None,
                  plugins_dir=None,
                  static=[],
                  install=[],
                  spatialite=False,
                  version_note="$PWD",
              ) as temp_docker:
                  df = os.path.join(temp_docker, "Dockerfile")
                  df_contents = open(df).read()
                  assert "'$PWD'" in df_contents
  >               assert "'--$HOME'" in df_contents

  ..\..\..\..\..\build\simonw\datasette\tests\test_utils.py:289: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  c:\python37\lib\contextlib.py:119: in __exit__
      next(self.gen)
  ..\..\..\..\..\build\simonw\datasette\datasette\utils\__init__.py:397: in temporary_docker_directory
      tmp.cleanup()
  c:\python37\lib\tempfile.py:809: in cleanup
      _shutil.rmtree(self.name)
  c:\python37\lib\shutil.py:516: in rmtree
      return _rmtree_unsafe(path, onerror)
  c:\python37\lib\shutil.py:395: in _rmtree_unsafe
      _rmtree_unsafe(fullname, onerror)
  c:\python37\lib\shutil.py:404: in _rmtree_unsafe
      onerror(os.rmdir, path, sys.exc_info())
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  path = 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmps8mll2fo\\t'
  onerror = <function rmtree.<locals>.onerror at 0x0000017D0230C948>

      def _rmtree_unsafe(path, onerror):
          try:
              with os.scandir(path) as scandir_it:
                  entries = list(scandir_it)
          except OSError:
              onerror(os.scandir, path, sys.exc_info())
              entries = []
          for entry in entries:
              fullname = entry.path
              try:
                  is_dir = entry.is_dir(follow_symlinks=False)
              except OSError:
                  is_dir = False
              if is_dir:
                  try:
                      if entry.is_symlink():
                          # This can only happen if someone replaces
                          # a directory with a symlink after the call to
                          # os.scandir or entry.is_dir above.
                          raise OSError("Cannot call rmtree on a symbolic link")
                  except OSError:
                      onerror(os.path.islink, fullname, sys.exc_info())
                      continue
                  _rmtree_unsafe(fullname, onerror)
              else:
                  try:
                      os.unlink(fullname)
                  except OSError:
                      onerror(os.unlink, fullname, sys.exc_info())
          try:
  >           os.rmdir(path)
  E           PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\travis\\AppData\\Local\\Temp\\tmps8mll2fo\\t'

  c:\python37\lib\shutil.py:402: PermissionError
  ========= 25 failed, 325 passed, 3 skipped, 1 error in 476.71 seconds =========
  travis_time:end:07fbb22c:start=1563238899952083900,finish=1563239378532569000,duration=478580485100
  [0K[31;1mThe command "pytest" exited with 1.[0m
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/557#issuecomment-511625212
issue_url: https://api.github.com/repos/simonw/datasette/issues/557
node_id: MDEyOklzc3VlQ29tbWVudDUxMTYyNTIxMg==
created_at: 2019-07-16T01:12:14Z
updated_at: 2019-07-16T01:12:14Z
author_association: OWNER
body:
  This looks useful for dealing with the `The process cannot access the file because it is being used by another process` error: https://stackoverflow.com/a/28032829

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/537#issuecomment-512126748
issue_url: https://api.github.com/repos/simonw/datasette/issues/537
node_id: MDEyOklzc3VlQ29tbWVudDUxMjEyNjc0OA==
created_at: 2019-07-17T06:48:35Z
updated_at: 2019-07-17T06:48:35Z
author_association: NONE
body:
  It looks as if the `datasette.utils.AsgiRouter.__call__` is the place to add this https://github.com/simonw/datasette/blob/90d4f497f9b3f6a5882937c91fddb496ac3e7368/datasette/utils/asgi.py#L101 .

  The sentry_asgi middleware uses the `__qualname__` or  `__name__` attributes of the `endpoint` https://github.com/encode/sentry-asgi/blob/c6a42d44d31f85885b79e4ee898683ecf8104971/sentry_asgi/middleware.py#L84

  Looking at the Starlette implementation `endpoint` is a `Callable` https://github.com/encode/starlette/commit/34d0097feb6f057bd050d5057df5a2f96b97384e#diff-34fba745b50527bfb4245d02afd59246R100 which as far as I can tell is analogous to the `view` function which is matched here https://github.com/simonw/datasette/blob/90d4f497f9b3f6a5882937c91fddb496ac3e7368/datasette/utils/asgi.py#L96 .

  A slight issue is that  `__qualname__` is matched *first* in the sentry_asgi middleware, and `__name__` is used if that doesn't exist. I think (please correct me if I am wrong) that for datasette, the `__name__` is what should be used. For example, when using the development fixtures and hitting `http://127.0.0.1:8001/fixtures/compound_three_primary_keys` the `view` function that is matched gives:

  ```python
  >>> view.__qualname__
  'AsgiView.as_asgi.<locals>.view'
  >>> view.__name__
  'TableView'
  ```

  Would `TableView` be the desired value here? Or am I looking in entirely the wrong place? :smile: 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/537#issuecomment-512664216
issue_url: https://api.github.com/repos/simonw/datasette/issues/537
node_id: MDEyOklzc3VlQ29tbWVudDUxMjY2NDIxNg==
created_at: 2019-07-18T04:53:18Z
updated_at: 2019-07-18T04:53:18Z
author_association: OWNER
body:
  Yes, `TableView` is desirable here I think. Maybe `datasette.views.table.TableView` if we can get that somehow. Thanks for taking a look!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/537#issuecomment-512930353
issue_url: https://api.github.com/repos/simonw/datasette/issues/537
node_id: MDEyOklzc3VlQ29tbWVudDUxMjkzMDM1Mw==
created_at: 2019-07-18T18:20:53Z
updated_at: 2019-07-18T18:34:03Z
author_association: NONE
body:
  Ok great, getting the `__qualname__` to be `TableView` and adding `endpoint` to the `scope` in `AsgiRouter` is simple enough (already done). However, (unless I'm missing a plugin hook or something) the suggestion of utilising it within a `datasette-sentry` plugin may not work. The only hook that would have access to the `scope` is the `asgi_wrapper` hook. But as this _wraps_ the existing `asgi` app, the `endpoint` won't yet have been added to the `scope` received by the hook https://github.com/SteadBytes/datasette/blob/107d47567dedd472eebec7f35bc34f5b58285ba8/datasette/app.py#L672 . However, I'm not sure where else the `endpoint` could be added to the asgi scope :thinking: 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513244121
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDUxMzI0NDEyMQ==
created_at: 2019-07-19T14:13:33Z
updated_at: 2019-07-19T14:13:33Z
author_association: OWNER
body:
  So what could the interface to this look like? Especially for the CLI?

  One option:

      sqlite-utils extract dea_sales company_name companies name

  Tricky thing here is that it's quite a large number of positional arguments:

      sqlite-utils extract dea_sales company_name companies name
                           Table     column       New table New column (maybe optional?)

  It would be great if this could supported multiple columns - for if a spreadsheet has e.g. a “Company Name”, “Company Address” pair of fields that always match each other and areduplicated many times.

  This could be handled by creating the new table with two columns that are indexed as a unique compound key. Then you can easily get-or-create on the pairs (or triples or whatever) from the original table.

  Challenge here is what does the CLI syntax look like. Something like this?

      $ sqlite-utils extract dea_sales -c company_name -c company_address \
        --to companies --to-col name --to-col address

  Perhaps the columns in the new table are FORCED to be the same as the old ones, hence avoiding some options? Bit restrictive… maybe they default to the same but you can customize?

      $ sqlite-utils extract dea_sales -c company_name -c company_address -t companies

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513246124
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDUxMzI0NjEyNA==
created_at: 2019-07-19T14:18:35Z
updated_at: 2019-07-19T14:19:40Z
author_association: OWNER
body:
  How about the Python version? That should be easier to design.

  ```python
  db["dea_sales"].extract(
      columns=["company_name", "company_address"],
      to_table="companies"
  )
  ```
  If we want to transform the extracted data (e.g. rename those columns) maybe support a `transform=` argument?

  ```python
  db["dea_sales"].extract(
      columns=["company_name", "company_address"],
      to_table="companies",
      transform = lambda extracted: {
          "name": extracted["company_name"],
          "address":  extracted["company_address"],
      }
  )
  ```
  This would create a new "companies" table with three columns: id, name and address.

  Would also be nice if there was a syntax for saying "... and use the value from this column as the primary key column in the newly created table".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513246831
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDUxMzI0NjgzMQ==
created_at: 2019-07-19T14:20:15Z
updated_at: 2019-07-19T14:20:49Z
author_association: OWNER
body:
  Since these operations could take a long time against large tables, it would be neat if there was a progress bar option for the CLI command.

  The operations are full table scans so calculating progress shouldn't be too difficult.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513262013
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDUxMzI2MjAxMw==
created_at: 2019-07-19T14:58:23Z
updated_at: 2020-09-22T18:12:11Z
author_association: OWNER
body:
  CLI design idea:

      $ sqlite-utils extract my.db \
          dea_sales company_name

  Here we just specify the original table and column - the new extracted table will automatically be called "company_name" and will have "id" and "value" columns, by default.

  To set a custom extract table:

      $ sqlite-utils extract my.db \
          dea_sales company_name \
          --table companies

  And for extracting multiple columns and renaming them on the created table, maybe something like this:

      $ sqlite-utils extract my.db \
          dea_sales company_name company_address \
          --table companies \
          --column company_name name \
          --column company_address address

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/537#issuecomment-513272392
issue_url: https://api.github.com/repos/simonw/datasette/issues/537
node_id: MDEyOklzc3VlQ29tbWVudDUxMzI3MjM5Mg==
created_at: 2019-07-19T15:27:03Z
updated_at: 2019-07-19T15:27:03Z
author_association: OWNER
body:
  Yeah that's a good call: the Datasette plugin mechanism where middleware is wrapped around the outside doesn't appear to be compatible with the Sentry mechanism of expecting that `scope` has been populated before it gets to their error handler.

  @tomchristie is this something you've thought about?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/537#issuecomment-513273003
issue_url: https://api.github.com/repos/simonw/datasette/issues/537
node_id: MDEyOklzc3VlQ29tbWVudDUxMzI3MzAwMw==
created_at: 2019-07-19T15:28:42Z
updated_at: 2019-07-19T15:28:42Z
author_association: OWNER
body: Asked about this on Twitter: https://twitter.com/simonw/status/1152238730259791877
reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 1, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/537#issuecomment-513279397
issue_url: https://api.github.com/repos/simonw/datasette/issues/537
node_id: MDEyOklzc3VlQ29tbWVudDUxMzI3OTM5Nw==
created_at: 2019-07-19T15:47:57Z
updated_at: 2019-07-19T15:48:09Z
author_association: NONE
body:
  The middleware implementation there works okay with a router nested inside if the scope is *mutated*. (Ie. "endpoint" doesn't need to exist at the point that the middleware starts running, but if it *has* been made available by the time an exception is thrown, then it can be used.)

  Starlette's usage of "endpoint" there is unilateral, rather than something I've discussed against the ASGI spec - certainly it's important for any monitoring ASGI middleware to be able to have some kind of visibility onto some limited subset of routing information, and `"endpoint"` in the scope referencing some routed-to callable seemed general enough to be useful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/537#issuecomment-513307487
issue_url: https://api.github.com/repos/simonw/datasette/issues/537
node_id: MDEyOklzc3VlQ29tbWVudDUxMzMwNzQ4Nw==
created_at: 2019-07-19T17:17:43Z
updated_at: 2019-07-19T17:17:43Z
author_association: OWNER
body:
  Huh, interesting. I'd got it into my head that scope should not be mutated under any circumstances - if that's not true and it's mutable there's all kinds of useful things we could do with it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/537#issuecomment-513317952
issue_url: https://api.github.com/repos/simonw/datasette/issues/537
node_id: MDEyOklzc3VlQ29tbWVudDUxMzMxNzk1Mg==
created_at: 2019-07-19T17:49:06Z
updated_at: 2019-07-19T17:49:06Z
author_association: OWNER
body:
  It strikes me that if scope is indeed meant to stay immutable the alternative way of solving this would be to add an outbound custom request header with the endpoint - `X-Endpoint: datasette.views.table.TableView` for example - and teach the Sentry plugin to optionally read that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/1#issuecomment-513437463
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDUxMzQzNzQ2Mw==
created_at: 2019-07-20T05:19:59Z
updated_at: 2019-07-20T05:19:59Z
author_association: MEMBER
body:
  I ran xml_analyser against the XML HealthKit `export.xml` file and got the following results:

  ```python
  {
    'ActivitySummary': {'attr_counts': {'activeEnergyBurned': 980,
                                       'activeEnergyBurnedGoal': 980,
                                       'activeEnergyBurnedUnit': 980,
                                       'appleExerciseTime': 980,
                                       'appleExerciseTimeGoal': 980,
                                       'appleStandHours': 980,
                                       'appleStandHoursGoal': 980,
                                       'dateComponents': 980},
                       'child_counts': {},
                       'count': 980,
                       'parent_counts': {'HealthData': 980}},
   'Correlation': {'attr_counts': {'creationDate': 1,
                                   'endDate': 1,
                                   'sourceName': 1,
                                   'sourceVersion': 1,
                                   'startDate': 1,
                                   'type': 1},
                   'child_counts': {'MetadataEntry': 1, 'Record': 2},
                   'count': 1,
                   'parent_counts': {'HealthData': 1}},
   'ExportDate': {'attr_counts': {'value': 1},
                  'child_counts': {},
                  'count': 1,
                  'parent_counts': {'HealthData': 1}},
   'HealthData': {'attr_counts': {'locale': 1},
                  'child_counts': {'ActivitySummary': 980,
                                   'Correlation': 1,
                                   'ExportDate': 1,
                                   'Me': 1,
                                   'Record': 2672231,
                                   'Workout': 663},
                  'count': 1,
                  'parent_counts': {}},
   'HeartRateVariabilityMetadataList': {'attr_counts': {},
                                        'child_counts': {'InstantaneousBeatsPerMinute': 93653},
                                        'count': 2318,
                                        'parent_counts': {'Record': 2318}},
   'InstantaneousBeatsPerMinute': {'attr_counts': {'bpm': 93653, 'time': 93653},
                                   'child_counts': {},
                                   'count': 93653,
                                   'parent_counts': {'HeartRateVariabilityMetadataList': 93653}},
   'Location': {'attr_counts': {'altitude': 398683,
                                'course': 398683,
                                'date': 398683,
                                'horizontalAccuracy': 398683,
                                'latitude': 398683,
                                'longitude': 398683,
                                'speed': 398683,
                                'verticalAccuracy': 398683},
                'child_counts': {},
                'count': 398683,
                'parent_counts': {'WorkoutRoute': 398683}},
   'Me': {'attr_counts': {'HKCharacteristicTypeIdentifierBiologicalSex': 1,
                          'HKCharacteristicTypeIdentifierBloodType': 1,
                          'HKCharacteristicTypeIdentifierDateOfBirth': 1,
                          'HKCharacteristicTypeIdentifierFitzpatrickSkinType': 1},
          'child_counts': {},
          'count': 1,
          'parent_counts': {'HealthData': 1}},
   'MetadataEntry': {'attr_counts': {'key': 290449, 'value': 290449},
                     'child_counts': {},
                     'count': 290449,
                     'parent_counts': {'Correlation': 1,
                                       'Record': 287974,
                                       'Workout': 1928,
                                       'WorkoutRoute': 546}},
   'Record': {'attr_counts': {'creationDate': 2672233,
                              'device': 2665111,
                              'endDate': 2672233,
                              'sourceName': 2672233,
                              'sourceVersion': 2671779,
                              'startDate': 2672233,
                              'type': 2672233,
                              'unit': 2650012,
                              'value': 2672232},
              'child_counts': {'HeartRateVariabilityMetadataList': 2318,
                               'MetadataEntry': 287974},
              'count': 2672233,
              'parent_counts': {'Correlation': 2, 'HealthData': 2672231}},
   'Workout': {'attr_counts': {'creationDate': 663,
                               'device': 230,
                               'duration': 663,
                               'durationUnit': 663,
                               'endDate': 663,
                               'sourceName': 663,
                               'sourceVersion': 663,
                               'startDate': 663,
                               'totalDistance': 663,
                               'totalDistanceUnit': 663,
                               'totalEnergyBurned': 663,
                               'totalEnergyBurnedUnit': 663,
                               'workoutActivityType': 663},
               'child_counts': {'MetadataEntry': 1928,
                                'WorkoutEvent': 2094,
                                'WorkoutRoute': 340},
               'count': 663,
               'parent_counts': {'HealthData': 663}},
   'WorkoutEvent': {'attr_counts': {'date': 2094,
                                    'duration': 837,
                                    'durationUnit': 837,
                                    'type': 2094},
                    'child_counts': {},
                    'count': 2094,
                    'parent_counts': {'Workout': 2094}},
   'WorkoutRoute': {'attr_counts': {'creationDate': 340,
                                    'endDate': 340,
                                    'sourceName': 340,
                                    'sourceVersion': 340,
                                    'startDate': 340},
                    'child_counts': {'Location': 398683, 'MetadataEntry': 546},
                    'count': 340,
                    'parent_counts': {'Workout': 340}}}
  ```

  The most interesting bit is this:

  ```python
   'HealthData': {'attr_counts': {'locale': 1},
                  'child_counts': {'ActivitySummary': 980,
                                   'Correlation': 1,
                                   'ExportDate': 1,
                                   'Me': 1,
                                   'Record': 2672231,
                                   'Workout': 663},
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/2#issuecomment-513439411
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/2
node_id: MDEyOklzc3VlQ29tbWVudDUxMzQzOTQxMQ==
created_at: 2019-07-20T05:58:57Z
updated_at: 2019-07-20T05:58:57Z
author_association: MEMBER
body:
  ```python
   'Workout': {'attr_counts': {'creationDate': 663,
                               'device': 230,
                               'duration': 663,
                               'durationUnit': 663,
                               'endDate': 663,
                               'sourceName': 663,
                               'sourceVersion': 663,
                               'startDate': 663,
                               'totalDistance': 663,
                               'totalDistanceUnit': 663,
                               'totalEnergyBurned': 663,
                               'totalEnergyBurnedUnit': 663,
                               'workoutActivityType': 663},
               'child_counts': {'MetadataEntry': 1928,
                                'WorkoutEvent': 2094,
                                'WorkoutRoute': 340},
               'count': 663,
               'parent_counts': {'HealthData': 663}},
   'WorkoutEvent': {'attr_counts': {'date': 2094,
                                    'duration': 837,
                                    'durationUnit': 837,
                                    'type': 2094},
                    'child_counts': {},
                    'count': 2094,
                    'parent_counts': {'Workout': 2094}},
   'WorkoutRoute': {'attr_counts': {'creationDate': 340,
                                    'endDate': 340,
                                    'sourceName': 340,
                                    'sourceVersion': 340,
                                    'startDate': 340},
                    'child_counts': {'Location': 398683, 'MetadataEntry': 546},
                    'count': 340,
                    'parent_counts': {'Workout': 340}}}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/537#issuecomment-513439736
issue_url: https://api.github.com/repos/simonw/datasette/issues/537
node_id: MDEyOklzc3VlQ29tbWVudDUxMzQzOTczNg==
created_at: 2019-07-20T06:05:01Z
updated_at: 2019-07-20T06:05:01Z
author_association: NONE
body:
  The asgi spec doesn't explicitly specify (at least as far as I can tell) whether the scope is immutable/mutable https://asgi.readthedocs.io/en/latest/specs/lifespan.html#scope . @simonw using a header for this would be a nice approach. It would also potentially increase the portability of any middleware/plugins/clients across different applications/frameworks as it's not tied directly to an asgi implementation

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/4#issuecomment-513440090
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDUxMzQ0MDA5MA==
created_at: 2019-07-20T06:11:50Z
updated_at: 2019-07-20T06:11:50Z
author_association: MEMBER
body:
  Some examples:

  https://github.com/dogsheep/healthkit-to-sqlite/blob/d016e70c31cf84ba0f5ec3102546db54a51aaffb/tests/export.xml#L4-L13

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/537#issuecomment-513442743
issue_url: https://api.github.com/repos/simonw/datasette/issues/537
node_id: MDEyOklzc3VlQ29tbWVudDUxMzQ0Mjc0Mw==
created_at: 2019-07-20T06:50:47Z
updated_at: 2019-07-20T06:50:47Z
author_association: NONE
body:
  Right now the spec does say “copy the scope, rather than mutate it” https://asgi.readthedocs.io/en/latest/specs/main.html#middleware 

  I wouldn’t be surprised if that there’s room for discussion on evolving the exact language there.

  There’s obvs a nice element to the strictness there, tho practically I’m not sure it’s something that implementations will follow, and its not something that Starlette chooses to abide by.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/537#issuecomment-513446227
issue_url: https://api.github.com/repos/simonw/datasette/issues/537
node_id: MDEyOklzc3VlQ29tbWVudDUxMzQ0NjIyNw==
created_at: 2019-07-20T07:50:44Z
updated_at: 2019-07-20T07:50:44Z
author_association: NONE
body:
  Oh yes well spotted thank you 😁 

  I agree that the strictness would be nice as it could help to avoid different middleware altering the scope in incompatible ways. However I do also agree that it's likely for not all implementations to follow 🤔

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/43#issuecomment-513481493
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/43
node_id: MDEyOklzc3VlQ29tbWVudDUxMzQ4MTQ5Mw==
created_at: 2019-07-20T16:34:54Z
updated_at: 2019-07-20T16:34:54Z
author_association: OWNER
body:
  Here's the implementation in `create_table()`:

  https://github.com/simonw/sqlite-utils/blob/c88f0a4d4617ff832aa728aac1da3500548c3137/sqlite_utils/db.py#L235-L249

  And here's where `add_column()` does its thing:

  https://github.com/simonw/sqlite-utils/blob/c88f0a4d4617ff832aa728aac1da3500548c3137/sqlite_utils/db.py#L547-L552

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/43#issuecomment-513481607
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/43
node_id: MDEyOklzc3VlQ29tbWVudDUxMzQ4MTYwNw==
created_at: 2019-07-20T16:36:34Z
updated_at: 2019-07-20T16:36:34Z
author_association: OWNER
body:
  Hmm... is it even possible to fix this within sqlite-utils? Maybe this is a SQLite implementation detail.

  It's possible to update the schema itself - that's how we add missing foreign key constraints - but it seems a little bit of overkill to use the writable schema when adding a new column JUST to get prettier indentation:

  https://github.com/simonw/sqlite-utils/blob/c88f0a4d4617ff832aa728aac1da3500548c3137/sqlite_utils/db.py#L316-L326

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/5#issuecomment-513625406
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDUxMzYyNTQwNg==
created_at: 2019-07-22T03:20:16Z
updated_at: 2019-07-22T03:20:16Z
author_association: MEMBER
body:
  It now renders like this:
  ```
  Importing from HealthKit  [#-----------------------------------]    5%  00:01:33
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/6#issuecomment-513626742
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/6
node_id: MDEyOklzc3VlQ29tbWVudDUxMzYyNjc0Mg==
created_at: 2019-07-22T03:28:55Z
updated_at: 2019-07-22T03:28:55Z
author_association: MEMBER
body:
  Here's what it looks like now as separate tables:

  <img width="1050" alt="hello9_and_Populate__endpoint__key_in_ASGI_scope_·_Issue__537_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/61604322-eba05000-abf5-11e9-8f8b-d0204e9a705a.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/537#issuecomment-513652597
issue_url: https://api.github.com/repos/simonw/datasette/issues/537
node_id: MDEyOklzc3VlQ29tbWVudDUxMzY1MjU5Nw==
created_at: 2019-07-22T06:03:18Z
updated_at: 2019-07-22T06:03:18Z
author_association: NONE
body:
  @simonw do you think it is still worth populating the `endpoint` key in the scope as originally intended by this issue, or should we hold off until a decision about possibly using an `X-Endpoint` header instead? :smile: 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/3#issuecomment-513952182
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/3
node_id: MDEyOklzc3VlQ29tbWVudDUxMzk1MjE4Mg==
created_at: 2019-07-22T20:58:55Z
updated_at: 2019-07-22T20:58:55Z
author_association: OWNER
body:
  Closing this as WONTFIX - it's not really an actionable ticket. I should still research these in the future though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/44#issuecomment-514162499
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/44
node_id: MDEyOklzc3VlQ29tbWVudDUxNDE2MjQ5OQ==
created_at: 2019-07-23T11:01:45Z
updated_at: 2019-07-23T11:01:45Z
author_association: OWNER
body:
  `id = table.lookup({"name":"Cleo"})`

  - If table does not exist, create it with id, name where name is unique
  - If table does exist, add unique name column
  - If table and column exist, add unique constraint - throw error if impossible
  - now either insert the new row or return the existing ID

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/44#issuecomment-514162930
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/44
node_id: MDEyOklzc3VlQ29tbWVudDUxNDE2MjkzMA==
created_at: 2019-07-23T11:03:03Z
updated_at: 2019-07-23T11:03:16Z
author_association: OWNER
body:
  I considered keyword arguments for this, but I am going with a dictionary instead - for two reasons:

  * leaves the option to add extra keyword arguments for further options later
  * supports column names that are not valid keyword arguments 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/46#issuecomment-514273140
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDUxNDI3MzE0MA==
created_at: 2019-07-23T15:57:53Z
updated_at: 2019-07-23T15:57:53Z
author_association: OWNER
body:
  This will play very well with the new ability to specify these arguments in the class constructor, like so:

  ```python
      table = db.table("trees", extracts={"species_id": "Species"})
      table.insert_all([{
          "id": 1,
          "species_id": "Oak",
      }, {
          "id": 2,
          "species_id": "Oak",
      }, {
          "id": 3,
          "species_id": "Palm",
      }])

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/38#issuecomment-514339130
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/38
node_id: MDEyOklzc3VlQ29tbWVudDUxNDMzOTEzMA==
created_at: 2019-07-23T18:57:33Z
updated_at: 2019-07-23T18:57:33Z
author_association: OWNER
body:
  This should switch to using `.get()`, introduced in https://github.com/simonw/sqlite-utils/commit/c65b67ca46f70e2da46a5b945f4ed358173262e9

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514496725
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7
node_id: MDEyOklzc3VlQ29tbWVudDUxNDQ5NjcyNQ==
created_at: 2019-07-24T06:20:59Z
updated_at: 2019-07-24T06:20:59Z
author_association: MEMBER
body:
  I'm using https://pypi.org/project/memory-profiler/ to explore this in more detail:

  ```
  $ pip install memory-profiler matplotlib
  ```

  Then:

  ```
  $ mprof run healthkit-to-sqlite ~/Downloads/healthkit-export.zip healthkit.db
  $ mprof plot
  ```

  <img width="1128" alt="Screen Shot 2019-07-24 at 8 17 06 AM" src="https://user-images.githubusercontent.com/9599/61769849-f3294a00-adeb-11e9-9026-02c24209530f.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514498221
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7
node_id: MDEyOklzc3VlQ29tbWVudDUxNDQ5ODIyMQ==
created_at: 2019-07-24T06:26:49Z
updated_at: 2019-07-24T06:26:49Z
author_association: MEMBER
body:
  Adding `el.clear()` got me a huge improvement:

  <img width="1128" alt="Screen Shot 2019-07-24 at 8 23 26 AM" src="https://user-images.githubusercontent.com/9599/61770130-c3c70d00-adec-11e9-886a-93a2cea5bf4b.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/7#issuecomment-514500253
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7
node_id: MDEyOklzc3VlQ29tbWVudDUxNDUwMDI1Mw==
created_at: 2019-07-24T06:34:28Z
updated_at: 2019-07-24T06:34:28Z
author_association: MEMBER
body:
  Clearing the root element each time saved even more:

  <img width="1128" alt="Screen Shot 2019-07-24 at 8 30 38 AM" src="https://user-images.githubusercontent.com/9599/61770555-d3932100-aded-11e9-8ffe-bebd682f94ed.png">

reactions:
  {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 2, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-514509307
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/49
node_id: MDEyOklzc3VlQ29tbWVudDUxNDUwOTMwNw==
created_at: 2019-07-24T07:09:43Z
updated_at: 2019-07-24T07:10:21Z
author_association: OWNER
body:
  This syntax should be shared with #42 as much as possible.

  Maybe something based on a namedtuple would work, since those are already used in the library.

  ```python
  workouts = db.table("workouts", extracts=[Extract(
      columns=["source", "source_version"],
      table="Sources"
  )])
  ```
  Since namedtuples cannot have default values this should probably be a class instead. Actually it looks like there is a trick for defaults here: https://stackoverflow.com/a/18348004

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-514745798
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDUxNDc0NTc5OA==
created_at: 2019-07-24T18:25:36Z
updated_at: 2019-07-24T18:25:36Z
author_association: NONE
body:
  This is on macOS 10.14.6, with Python 3.7.4, packages in the virtual environment:

  ```
  Package             Version
  ------------------- -------
  aiofiles            0.4.0
  Click               7.0
  click-default-group 1.2.1
  datasette           0.29.2
  h11                 0.8.1
  healthkit-to-sqlite 0.3.1
  httptools           0.0.13
  hupper              1.8.1
  importlib-metadata  0.18
  Jinja2              2.10.1
  MarkupSafe          1.1.1
  Pint                0.8.1
  pip                 19.2.1
  pluggy              0.12.0
  setuptools          41.0.1
  sqlite-utils        1.7
  tabulate            0.8.3
  uvicorn             0.8.4
  uvloop              0.12.2
  websockets          7.0
  zipp                0.5.2
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515226724
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDUxNTIyNjcyNA==
created_at: 2019-07-25T21:46:01Z
updated_at: 2019-07-25T21:46:01Z
author_association: MEMBER
body:
  I can work around this here (prior to the fix in sqlite-utils) by setting the batch size to something a bit lower here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515322294
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDUxNTMyMjI5NA==
created_at: 2019-07-26T06:07:12Z
updated_at: 2019-07-26T06:07:12Z
author_association: MEMBER
body:
  @tholo this should be fixed in just-released version 0.3.2 - could you run a `pip install -U healthkit-to-sqlite` and let me know if it works for you now?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/9#issuecomment-515370687
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDUxNTM3MDY4Nw==
created_at: 2019-07-26T09:01:19Z
updated_at: 2019-07-26T09:01:19Z
author_association: NONE
body:
  Yes, that did fix the issue I was seeing — it will now import my complete HealthKit data.

  Thorsten

  > On Jul 25, 2019, at 23:07, Simon Willison <notifications@github.com> wrote:
  > 
  > @tholo <https://github.com/tholo> this should be fixed in just-released version 0.3.2 - could you run a pip install -U healthkit-to-sqlite and let me know if it works for you now?
  > 
  > —
  > You are receiving this because you were mentioned.
  > Reply to this email directly, view it on GitHub <https://github.com/dogsheep/healthkit-to-sqlite/issues/9?email_source=notifications&email_token=AABIUPYTWYOYSLEAFS7TLMLQBKIBBA5CNFSM4IGSXNNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD23TDNQ#issuecomment-515322294>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABIUP7DCBQ37SESQL7D4WTQBKIBBANCNFSM4IGSXNNA>.
  > 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515751719
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/50
node_id: MDEyOklzc3VlQ29tbWVudDUxNTc1MTcxOQ==
created_at: 2019-07-28T10:40:11Z
updated_at: 2019-07-28T10:40:11Z
author_association: OWNER
body:
  I think the fix here is for me to switch to using `executemany()` - example from the Python docs: https://docs.python.org/3/library/sqlite3.html
  ```python
  purchases = [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
               ('2006-04-05', 'BUY', 'MSFT', 1000, 72.00),
               ('2006-04-06', 'SELL', 'IBM', 500, 53.00),
              ]
  c.executemany('INSERT INTO stocks VALUES (?,?,?,?,?)', purchases)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515752129
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/50
node_id: MDEyOklzc3VlQ29tbWVudDUxNTc1MjEyOQ==
created_at: 2019-07-28T10:46:49Z
updated_at: 2019-07-28T10:46:49Z
author_association: OWNER
body:
  The problem with `.executemany()` is it breaks `lastrowid`:

  > This read-only attribute provides the rowid of the last modified row. It is only set if you issued an INSERT or a REPLACE statement using the execute() method. For operations other than INSERT or REPLACE or when executemany() is called, lastrowid is set to None.

  So I think I need to continue to use my existing way of executing bulk inserts (with a giant repeated `INSERT INTO ... VALUES` block) but ensure that I calculate the chunk size such that I don't ever try to pass more than 999 values at once.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515752204
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/50
node_id: MDEyOklzc3VlQ29tbWVudDUxNTc1MjIwNA==
created_at: 2019-07-28T10:48:14Z
updated_at: 2019-07-28T10:48:14Z
author_association: OWNER
body:
  Here's the diff where I tried to use `.executemany()` and ran into the `lastrowid` problem:
  ```diff
  diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
  index ef55976..7f85759 100644
  --- a/sqlite_utils/db.py
  +++ b/sqlite_utils/db.py
  @@ -881,13 +881,10 @@ class Table:
                   or_what=or_what,
                   table=self.name,
                   columns=", ".join("[{}]".format(c) for c in all_columns),
  -                rows=", ".join(
  -                    """
  +                rows="""
                       ({placeholders})
                   """.format(
  -                        placeholders=", ".join(["?"] * len(all_columns))
  -                    )
  -                    for record in chunk
  +                    placeholders=", ".join(["?"] * len(all_columns))
                   ),
               )
               values = []
  @@ -902,15 +899,15 @@ class Table:
                           extract_table = extracts[key]
                           value = self.db[extract_table].lookup({"value": value})
                       record_values.append(value)
  -                values.extend(record_values)
  +                values.append(record_values)
               with self.db.conn:
                   try:
  -                    result = self.db.conn.execute(sql, values)
  +                    result = self.db.conn.executemany(sql, values)
                   except sqlite3.OperationalError as e:
                       if alter and (" has no column " in e.args[0]):
                           # Attempt to add any missing columns, then try again
                           self.add_missing_columns(chunk)
  -                        result = self.db.conn.execute(sql, values)
  +                        result = self.db.conn.executemany(sql, values)
                       else:
                           raise
                   self.last_rowid = result.lastrowid
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/51#issuecomment-515756563
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/51
node_id: MDEyOklzc3VlQ29tbWVudDUxNTc1NjU2Mw==
created_at: 2019-07-28T11:56:10Z
updated_at: 2019-07-28T11:56:10Z
author_association: OWNER
body:
  Interesting. The tests failed presumably because the version of SQLite I am running on Travis CI doesn't have that same 999 limit. I'm going to enforce the 999 limit within the library itself, to discourage people from writing code which will fail if it runs on a host that DOES have that limit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/23#issuecomment-517927854
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDUxNzkyNzg1NA==
created_at: 2019-08-03T14:17:12Z
updated_at: 2019-08-03T14:17:12Z
author_association: OWNER
body:
  It would be neat if this could interact with the lookup table mechanism introduced in #44 

  Maybe like this:

  ```python
  table.insert({"name": "Barry"}).m2m("tags", lookup={"tag": "Coworker"})
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/566#issuecomment-517936126
issue_url: https://api.github.com/repos/simonw/datasette/issues/566
node_id: MDEyOklzc3VlQ29tbWVudDUxNzkzNjEyNg==
created_at: 2019-08-03T16:13:36Z
updated_at: 2019-08-03T16:13:36Z
author_association: NONE
body:
  This example now works on my other machine when running from the anaconda prompt. So I'm not sure if this is actually a datasette issue or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/23#issuecomment-517971623
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDUxNzk3MTYyMw==
created_at: 2019-08-04T03:54:58Z
updated_at: 2019-08-04T03:54:58Z
author_association: OWNER
body:
  Released in version 1.9. Documentation is here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#working-with-many-to-many-relationships

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/411#issuecomment-519065799
issue_url: https://api.github.com/repos/simonw/datasette/issues/411
node_id: MDEyOklzc3VlQ29tbWVudDUxOTA2NTc5OQ==
created_at: 2019-08-07T12:00:36Z
updated_at: 2019-08-07T12:00:36Z
author_association: NONE
body:
  Hi,
  Apologies for the long delay.

  I tried your suggesting escaping approach:
  `SELECT a.pos AS rank, b.id, b.name, b.country, b.latitude AS latitude, b.longitude AS longitude, a.distance / 1000.0 AS dist_km FROM KNN AS a LEFT JOIN airports AS b ON (b.rowid = a.fid)WHERE f_table_name = 'airports' AND ref_geometry = MakePoint(:Long || ", " || :Lat) AND max_items = 6;
  `
  and it returns this error:
  `wrong number of arguments to function MakePoint()`

  Anything else you suggest I try?
  Thanks

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/54#issuecomment-522236053
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/54
node_id: MDEyOklzc3VlQ29tbWVudDUyMjIzNjA1Mw==
created_at: 2019-08-17T13:22:26Z
updated_at: 2019-08-17T13:22:26Z
author_association: OWNER
body:
  Good catch: `create_view()` is an earlier utility method from before I started adding methods like `db["table"].rows` and `.rows_where()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/569#issuecomment-522236218
issue_url: https://api.github.com/repos/simonw/datasette/issues/569
node_id: MDEyOklzc3VlQ29tbWVudDUyMjIzNjIxOA==
created_at: 2019-08-17T13:24:50Z
updated_at: 2019-08-17T13:24:50Z
author_association: OWNER
body:
  I think what I want it a mechanism where a thread can say "give me a connection for database X" and it either gets back connection to X instantly OR a new connection to X is created and returned OR it blocks (because a certain number of connections to that database exist already) until another thread returns their connection OR it times out and returns an error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/55#issuecomment-522238063
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/55
node_id: MDEyOklzc3VlQ29tbWVudDUyMjIzODA2Mw==
created_at: 2019-08-17T13:42:40Z
updated_at: 2019-08-17T13:42:40Z
author_association: OWNER
body:
  I still need to implement the part of this where certain actions against views (like updating/inserting) are not allowed.

  I think the better way to do that is to introduce a new class - `View` - rather than reusing the existing `Table` class - since there's a lot of stuff on Table (like primary key / foreign key introspection) which simply doesn't make sense for views.

  This will get a bit more complicated - maybe I need both Table and View to extend a common subclass here?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/569#issuecomment-522238222
issue_url: https://api.github.com/repos/simonw/datasette/issues/569
node_id: MDEyOklzc3VlQ29tbWVudDUyMjIzODIyMg==
created_at: 2019-08-17T13:44:24Z
updated_at: 2019-08-17T13:45:10Z
author_association: OWNER
body:
  Potential API design:
  ```python
  with pool.connection("fixtures") as conn:
      conn.set_time_limit(1000)
      conn.allow_all()
      conn.execute(...)
  ```
  Within this block the current thread has exclusive action to the connection - which has essentially been "checked out" from the pool. When the block ends that connection is made available to be checked out by other threads.

  This could accept multiple database names for the case where I want to join across databases:
  ```python
  with pool.connection("fixtures", "dogs") as conn:
      conn.execute(...)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/27#issuecomment-526794367
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDUyNjc5NDM2Nw==
created_at: 2019-08-31T03:05:18Z
updated_at: 2019-08-31T03:06:10Z
author_association: OWNER
body:
  Potential syntax:

      $ sqlite-utils create-table foo.db dogs \
          id integer \
          name string \
          age integer \
          --pk=id

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/swarm-to-sqlite/issues/4#issuecomment-526853542
issue_url: https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDUyNjg1MzU0Mg==
created_at: 2019-08-31T18:06:32Z
updated_at: 2019-08-31T18:06:32Z
author_association: MEMBER
body:
  https://your-foursquare-oauth-token.glitch.me/

  Source code: https://glitch.com/~your-foursquare-oauth-token

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527091278
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/57
node_id: MDEyOklzc3VlQ29tbWVudDUyNzA5MTI3OA==
created_at: 2019-09-02T10:13:32Z
updated_at: 2019-09-02T10:13:32Z
author_association: OWNER
body:
  This is brilliant!

  One tiny suggestion: I like `--create-triggers` and `create_triggers=True` better for this as they are shorter but still capture what it dies - especially since the underlying SQL uses `CREATE TRIGGER` statements.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-527200332
issue_url: https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3
node_id: MDEyOklzc3VlQ29tbWVudDUyNzIwMDMzMg==
created_at: 2019-09-02T16:32:20Z
updated_at: 2019-09-02T16:32:39Z
author_association: MEMBER
body:
  Also needed: an option for "fetch all checkins created within the last X days".

  This should help provide support for that Swarm feature where you can retroactively checkin to places in the past.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/56#issuecomment-527209840
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/56
node_id: MDEyOklzc3VlQ29tbWVudDUyNzIwOTg0MA==
created_at: 2019-09-02T17:23:21Z
updated_at: 2019-09-02T17:23:21Z
author_association: CONTRIBUTOR
body:
  I have updated the other PR with the changes from this one and added tests. I have also changed the escaping from double quotes to brackets. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527258212
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/57
node_id: MDEyOklzc3VlQ29tbWVudDUyNzI1ODIxMg==
created_at: 2019-09-02T23:40:56Z
updated_at: 2019-09-02T23:40:56Z
author_association: OWNER
body:
  This is fantastic, thanks so much.

  I spotted a Datasette bug triggered by this: Datasette has its own version of the `detect_fts` function - at https://github.com/simonw/datasette/blob/d224ee2c98ac39c2c6e21a0ac0c62e5c3e1ccd11/datasette/utils/__init__.py#L466-L479 - which fails to pick up FTS tables created using the new escaping pattern.

  It's a bug in Datasette, not sqlite-utils - so I'll open an issue there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/560#issuecomment-527266687
issue_url: https://api.github.com/repos/simonw/datasette/issues/560
node_id: MDEyOklzc3VlQ29tbWVudDUyNzI2NjY4Nw==
created_at: 2019-09-03T01:03:02Z
updated_at: 2019-09-03T01:03:02Z
author_association: OWNER
body:
  This is released in 0.29.3

  Compare https://v0-29.datasette.io/fixtures (broken) with https://v0-29-3.datasette.io/fixtures (fixed)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/4#issuecomment-527682713
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDUyNzY4MjcxMw==
created_at: 2019-09-03T23:48:57Z
updated_at: 2019-09-03T23:48:57Z
author_association: MEMBER
body:
  One interesting challenge here is that the JSON format for tweets in the archive is subtly different from the JSON format currently returned by the API.

  If we want to keep the tweets in the same database table (which feels like the right thing to me) we'll need to handle this.

  One thing we can do is have a column for `from_archive` which is set to 1 for tweets that were recovered from the archive.

  We can also ensure that tweets from the API always over-write the version that came from the archive (using `.upsert()`) while tweets from the archive use `.insert(..., ignore=True)` to avoid over-writing a better version that came from the API.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/5#issuecomment-527684202
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDUyNzY4NDIwMg==
created_at: 2019-09-03T23:56:28Z
updated_at: 2019-09-03T23:56:28Z
author_association: MEMBER
body:
  I previously used betamax here: https://github.com/simonw/github-contents/blob/master/test_github_contents.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527954898
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2
node_id: MDEyOklzc3VlQ29tbWVudDUyNzk1NDg5OA==
created_at: 2019-09-04T15:31:46Z
updated_at: 2019-09-04T15:31:46Z
author_association: MEMBER
body:
  I'm going to call this `twitter-to-sqlite user-timeline` to reflect the language used to describe the API endpoint: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527990908
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2
node_id: MDEyOklzc3VlQ29tbWVudDUyNzk5MDkwOA==
created_at: 2019-09-04T16:57:24Z
updated_at: 2019-09-04T16:57:24Z
author_association: MEMBER
body:
  I just tried this using `max_id=` pagination as described in [Working with timelines](https://developer.twitter.com/en/docs/tweets/timelines/guides/working-with-timelines) and I got back all 17,759 of my tweets.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/572#issuecomment-528112729
issue_url: https://api.github.com/repos/simonw/datasette/issues/572
node_id: MDEyOklzc3VlQ29tbWVudDUyODExMjcyOQ==
created_at: 2019-09-04T22:20:22Z
updated_at: 2019-09-04T22:20:38Z
author_association: OWNER
body:
  The bug is here:
  https://github.com/simonw/datasette/blob/9743e1d91b5f0a2b3c1c0bd6ffce8739341f43c4/datasette/templates/_description_source_license.html#L24

  `or metadata.source or metadat.source_url`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-529239307
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8
node_id: MDEyOklzc3VlQ29tbWVudDUyOTIzOTMwNw==
created_at: 2019-09-08T20:36:49Z
updated_at: 2019-09-08T20:36:49Z
author_association: MEMBER
body:
  `--attach` can optionally take a name for the database connection alias like this:

      $ twitter-to-sqlite users-lookup users.db --attach foo:attending.db ...

  If you omit the `alias:` bit the stem of the database (without the file extension) will be used.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/9#issuecomment-530028567
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDUzMDAyODU2Nw==
created_at: 2019-09-10T16:59:25Z
updated_at: 2019-09-10T16:59:25Z
author_association: MEMBER
body:
  By default in SQLite foreign key constraints are not enforced (you need to run `PRAGMA foreign_keys = ON;` to enforce them).

  We will take advantage of this - even though the `following` table has foreign keys against user we will allow IDs to populate that table without a corresponding user record.

  In the future we may add a command that can backfill missing user records.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/406#issuecomment-530985388
issue_url: https://api.github.com/repos/simonw/datasette/issues/406
node_id: MDEyOklzc3VlQ29tbWVudDUzMDk4NTM4OA==
created_at: 2019-09-12T20:08:22Z
updated_at: 2019-09-12T20:08:22Z
author_association: OWNER
body:
  This is particularly noticeable with databases produced by `twitter-to-sqlite` - e.g. https://twitter-to-sqlite-demo.now.sh/cleo/tweets

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/8#issuecomment-531404891
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8
node_id: MDEyOklzc3VlQ29tbWVudDUzMTQwNDg5MQ==
created_at: 2019-09-13T22:01:57Z
updated_at: 2019-09-13T22:01:57Z
author_association: MEMBER
body:
  I also wrote about this in https://simonwillison.net/2019/Sep/13/weeknotestwitter-sqlite-datasette-rure/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/3#issuecomment-531516956
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3
node_id: MDEyOklzc3VlQ29tbWVudDUzMTUxNjk1Ng==
created_at: 2019-09-14T21:56:31Z
updated_at: 2019-09-14T21:56:31Z
author_association: MEMBER
body:
  https://api.github.com/users/simonw/repos

  It would be useful to be able to fetch stargazers, forks etc as well. Not sure if that should be a separate command or a `--stargazers` option to this command.

  Probably a separate command since `issues` is a separate command already.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-531517138
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDUzMTUxNzEzOA==
created_at: 2019-09-14T21:59:59Z
updated_at: 2019-09-14T21:59:59Z
author_association: MEMBER
body:
  Paginate through https://api.github.com/repos/simonw/datasette/stargazers

  Send `Accept: application/vnd.github.v3.star+json` to get the `starred_at` dates.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/61#issuecomment-533818697
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/61
node_id: MDEyOklzc3VlQ29tbWVudDUzMzgxODY5Nw==
created_at: 2019-09-21T18:09:01Z
updated_at: 2019-09-21T18:09:28Z
author_association: CONTRIBUTOR
body:
  @witeshadow The library version doesn't have helpers around CSV (at least not from what I can see in the code). 

  But here's a snippet that makes it easy to insert from CSV using the library. 

  ```
  import csv
  from sqlite_utils import Database

  # CSV Reader

  csv_file = open("filename.csv")   # open the csv file.
  reader = csv.reader(csv_file)  # Create a CSV reader
  headers = next(reader)   # First line is the header
  docs = (dict(zip(headers, row)) for row in reader)

  # Now you can use the `sqlite_utils` library. 

  db = Database("my_database.db")
  db["table_name"].insert_all(docs)
  ```

  This snippet is adapted from reading the CLI source code on how it implements the csv option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/569#issuecomment-537711455
issue_url: https://api.github.com/repos/simonw/datasette/issues/569
node_id: MDEyOklzc3VlQ29tbWVudDUzNzcxMTQ1NQ==
created_at: 2019-10-02T22:41:12Z
updated_at: 2019-10-02T22:41:12Z
author_association: OWNER
body:
  I'm going to refactor the `execute()` and `execute_against_connection_in_thread()` methods.

  They currently live on the `Datasette` class, but in this new world it would make more sense for them to live on the `Database`, `ConnectionGroup` or `Connection` classes.

  I think I'll put them on the `Database` class.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/569#issuecomment-537712384
issue_url: https://api.github.com/repos/simonw/datasette/issues/569
node_id: MDEyOklzc3VlQ29tbWVudDUzNzcxMjM4NA==
created_at: 2019-10-02T22:44:36Z
updated_at: 2019-10-02T22:44:36Z
author_association: OWNER
body:
  I'm going to simplify things a bunch by continuing to ignore the cross-database joining issue #283 - I'll post some notes there on my latest thinking.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-537716955
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDUzNzcxNjk1NQ==
created_at: 2019-10-02T23:02:15Z
updated_at: 2019-10-02T23:02:15Z
author_association: OWNER
body:
  I've been thinking pretty hard about this as part of #569. My big concerns are:

  * If I'm caching and reusing connections I need to worry about the different combinations - if I have four databases do I cache separate connections for the ("one", "two") AND ("two", "three") AND ("one", "three") and so on pairs?
  * How does the API and interface deal with instances where you have a database connected as the primary and you want to ATTACH another database and talk to that as well?

  I think the best way to do this is to say that cross-database joins will only be available against the `:memory:` database. Maybe with an optional mode you can run like `datasette --crossdb` which causes every database to be `ATTACHd` to that connection with an alias so you can start running queries.

  If this proves to be a problem when hundreds of files are attached to a Datasette Library instance (#417) then maybe cross database joins are handled (in that case) by the authenticated user selecting which ones to ?_attach= and detaching them at the end of the request. Also perhaps limit to joining across a maximum of 3 databases at once in this case.

  I can probably avoid the scariest negative consequences of cross-database joins by having them turned off by default for signed-out users. The datasette-on-my-laptop or authenticated Datasette Library cases can be opt-in and can be a little less locked down.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/577#issuecomment-538173983
issue_url: https://api.github.com/repos/simonw/datasette/issues/577
node_id: MDEyOklzc3VlQ29tbWVudDUzODE3Mzk4Mw==
created_at: 2019-10-04T00:01:18Z
updated_at: 2019-10-04T00:01:18Z
author_association: OWNER
body:
  Here's the horrific (stack inspecting) workaround I came up with for a prototype of `datasette-atom`: https://github.com/simonw/datasette-atom/commit/c0e3bd9556d7b31f253a8bf666d42205cd24f4fc#diff-acba9942430bc5e616410567296f92ffR87

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/11#issuecomment-538711918
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/11
node_id: MDEyOklzc3VlQ29tbWVudDUzODcxMTkxOA==
created_at: 2019-10-06T04:54:17Z
updated_at: 2019-10-06T04:54:17Z
author_association: MEMBER
body:
  Shipped in 0.6. Here's the documentation: https://github.com/dogsheep/twitter-to-sqlite#capturing-tweets-in-real-time-with-track-and-follow

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/583#issuecomment-538766388
issue_url: https://api.github.com/repos/simonw/datasette/issues/583
node_id: MDEyOklzc3VlQ29tbWVudDUzODc2NjM4OA==
created_at: 2019-10-06T17:00:43Z
updated_at: 2019-10-06T17:00:43Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/2dc5c8dc259a0606162673d394ba8cc1c6f54428/datasette/utils/__init__.py#L165-L171

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/584#issuecomment-538809457
issue_url: https://api.github.com/repos/simonw/datasette/issues/584
node_id: MDEyOklzc3VlQ29tbWVudDUzODgwOTQ1Nw==
created_at: 2019-10-07T01:18:02Z
updated_at: 2019-10-07T01:18:02Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/f04deebec4f3842f7bd610cd5859de529f77d50e/datasette/views/base.py#L310

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/586#issuecomment-538829738
issue_url: https://api.github.com/repos/simonw/datasette/issues/586
node_id: MDEyOklzc3VlQ29tbWVudDUzODgyOTczOA==
created_at: 2019-10-07T03:49:30Z
updated_at: 2019-10-07T03:49:30Z
author_association: OWNER
body:
  Yes this is the fault of datasette-auth-github:

  https://github.com/simonw/datasette-auth-github/blob/b3aaa4cc2f9b05a480357727bdce39c57d56fd4c/datasette_auth_github/github_auth.py#L138-L145

  Bug filed over there: https://github.com/simonw/datasette-auth-github/issues/47

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/pocket-to-sqlite/issues/1#issuecomment-538847446
issue_url: https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDUzODg0NzQ0Ng==
created_at: 2019-10-07T05:41:17Z
updated_at: 2019-10-07T05:41:17Z
author_association: MEMBER
body:
  Prototype code:
  ```python
  offset = 0
  fetched = []
  size = 500
  while True:
      page = requests.get("https://getpocket.com/v3/get", {
          "consumer_key": consumer_key,
          "access_token": access_token,
          "sort": "oldest",
          "detailType": "complete",
          "count": size,
          "offset": offset,
      }).json()
      print(offset)
      fetched.append(page)
      offset += size
      if not len(page["list"]):
          break
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-539721880
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUzOTcyMTg4MA==
created_at: 2019-10-08T22:00:03Z
updated_at: 2019-10-08T22:00:03Z
author_association: NONE
body:
  If you are just using Nginx to open a reserved port, systemd can do that on its own. https://www.freedesktop.org/software/systemd/man/systemd.socket.html. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/567#issuecomment-540545863
issue_url: https://api.github.com/repos/simonw/datasette/issues/567
node_id: MDEyOklzc3VlQ29tbWVudDU0MDU0NTg2Mw==
created_at: 2019-10-10T12:20:29Z
updated_at: 2019-10-10T12:20:29Z
author_association: NONE
body:
  Hi Simon. Is there somewhere to read about its ability to run against read-only databases that are able to be modified by other processes? While we're waiting for "Datasette Edit" :)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/215#issuecomment-540548765
issue_url: https://api.github.com/repos/simonw/datasette/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDU0MDU0ODc2NQ==
created_at: 2019-10-10T12:27:56Z
updated_at: 2019-10-10T12:27:56Z
author_association: NONE
body:
  Hi Simon. Any news on the ability to add routes (with static content) to datasette? As a public institution I'm required to have at least privacy, cookie and availability policies in place, and it really would be nice to have these under the same url. Thank you for some great work!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/585#issuecomment-541052329
issue_url: https://api.github.com/repos/simonw/datasette/issues/585
node_id: MDEyOklzc3VlQ29tbWVudDU0MTA1MjMyOQ==
created_at: 2019-10-11T12:53:51Z
updated_at: 2019-10-11T12:53:51Z
author_association: CONTRIBUTOR
body:
  I think this would be good, yeah – currently, databases are explicitly sorted by name in the IndexView, we could just remove that part (and use an `OrderedDict` for consistency, I suppose)?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112588
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17
node_id: MDEyOklzc3VlQ29tbWVudDU0MTExMjU4OA==
created_at: 2019-10-11T15:31:30Z
updated_at: 2019-10-11T15:31:30Z
author_association: MEMBER
body:
  No need for an option:

  > This command will delete and recreate all of your `archive-*` tables every time you run it. If this is not what you want, run the command against a fresh SQLite database rather than running it again one that already exists.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/507#issuecomment-541118904
issue_url: https://api.github.com/repos/simonw/datasette/issues/507
node_id: MDEyOklzc3VlQ29tbWVudDU0MTExODkwNA==
created_at: 2019-10-11T15:48:49Z
updated_at: 2019-10-11T15:48:49Z
author_association: CONTRIBUTOR
body:
  Headless Chrome and Firefox via Selenium are a solid choice in my experience. You may be interested in how pretix and pretalx solve this problem: They use pytest to create those screenshots on release to make sure they are up to date. See [this writeup](https://behind.pretix.eu/2018/11/15/automated-screenshots/) and [this repo](https://github.com/pretix/pretix-screenshots).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/512#issuecomment-541119038
issue_url: https://api.github.com/repos/simonw/datasette/issues/512
node_id: MDEyOklzc3VlQ29tbWVudDU0MTExOTAzOA==
created_at: 2019-10-11T15:49:13Z
updated_at: 2019-10-11T15:49:13Z
author_association: CONTRIBUTOR
body:
  How open are you to changing the config variable names (with appropriate deprecation, of course)? `"about_url_text", "license_url_text"` etc might be better suited to convey that these are just meant as basically URL titles.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541119834
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18
node_id: MDEyOklzc3VlQ29tbWVudDU0MTExOTgzNA==
created_at: 2019-10-11T15:51:22Z
updated_at: 2019-10-11T16:51:33Z
author_association: MEMBER
body:
  In order to support multiple user timelines being saved in the same database, I'm going to import the tweets into the `tweets` table AND add a new `timeline_tweets` table recording that a specific tweet showed up in a specific user's timeline.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541141169
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18
node_id: MDEyOklzc3VlQ29tbWVudDU0MTE0MTE2OQ==
created_at: 2019-10-11T16:51:29Z
updated_at: 2019-10-11T16:51:29Z
author_association: MEMBER
body:
  Documented here: https://github.com/dogsheep/twitter-to-sqlite/blob/master/README.md#retrieving-tweets-from-your-home-timeline

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-541248629
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19
node_id: MDEyOklzc3VlQ29tbWVudDU0MTI0ODYyOQ==
created_at: 2019-10-11T22:48:56Z
updated_at: 2019-10-11T22:48:56Z
author_association: MEMBER
body:
  `since_id` documented here: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-home_timeline

  > Returns results with an ID greater than (that is, more recent than) the specified ID. There are limits to the number of Tweets which can be accessed through the API. If the limit of Tweets has occured since the since_id, the since_id will be forced to the oldest ID available.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/593#issuecomment-541323265
issue_url: https://api.github.com/repos/simonw/datasette/issues/593
node_id: MDEyOklzc3VlQ29tbWVudDU0MTMyMzI2NQ==
created_at: 2019-10-12T13:04:54Z
updated_at: 2019-10-12T13:07:01Z
author_association: NONE
body:
  hum, well, I fail also with hypercorn on "add_signal_handler()" not implemented directly  in Windows, python-3.8 included

   https://stackoverflow.com/questions/45987985/asyncio-loops-add-signal-handler-in-windows

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/6#issuecomment-541387822
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6
node_id: MDEyOklzc3VlQ29tbWVudDU0MTM4NzgyMg==
created_at: 2019-10-13T05:27:39Z
updated_at: 2019-10-13T05:27:39Z
author_association: MEMBER
body:
  This should be fixed by https://github.com/dogsheep/github-to-sqlite/commit/552543a74970f8a3a3f87f887be23a0c6eb1cb5b

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/595#issuecomment-541494080
issue_url: https://api.github.com/repos/simonw/datasette/issues/595
node_id: MDEyOklzc3VlQ29tbWVudDU0MTQ5NDA4MA==
created_at: 2019-10-14T03:43:11Z
updated_at: 2019-10-14T03:43:11Z
author_association: OWNER
body:
  Ooh this is tricky... https://travis-ci.org/simonw/datasette/jobs/597216285 failed because it looks like `websockets-8.0.2` isn't compatible with Python 3.5

  This is a problem. I need to keep Datasette working with 3.5 for compatibility with Glitch. But obviously I also want it to work with 3.8. 

  I wonder if there's a smart solution here. @tomchristie any thoughts?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/592#issuecomment-541494347
issue_url: https://api.github.com/repos/simonw/datasette/issues/592
node_id: MDEyOklzc3VlQ29tbWVudDU0MTQ5NDM0Nw==
created_at: 2019-10-14T03:45:29Z
updated_at: 2019-10-14T03:45:29Z
author_association: OWNER
body:
  This is a really nice implementation. I particularly like how you have the button as hidden until JavaScript runs and makes it visible.

  https://github.com/simonw/datasette/pull/592/files#diff-0932659dded7aa3ce2d4c7badb187a9cR6-R8

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/590#issuecomment-541495034
issue_url: https://api.github.com/repos/simonw/datasette/issues/590
node_id: MDEyOklzc3VlQ29tbWVudDU0MTQ5NTAzNA==
created_at: 2019-10-14T03:51:32Z
updated_at: 2019-10-14T03:51:32Z
author_association: OWNER
body:
  We need some unit tests for this.

  I'd be fine with us renaming `extra_database.db` to `extra database.py` in these existing fixtures:

  https://github.com/simonw/datasette/blob/af2e6a5cf186a7200d76cb67ac30fa59cc24d84e/tests/fixtures.py#L174-L185

  Then we could add an extra `test_database_name_with_spaces(...)` test in `test_api.py` which uses parametrize to exercise a bunch of URLs and check they all return a 200 - something a bit like this but covering a range of URL paths (one for database, table, row and table with filters):

  https://github.com/simonw/datasette/blob/90d4f497f9b3f6a5882937c91fddb496ac3e7368/tests/test_api.py#L746-L765

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/578#issuecomment-541495682
issue_url: https://api.github.com/repos/simonw/datasette/issues/578
node_id: MDEyOklzc3VlQ29tbWVudDU0MTQ5NTY4Mg==
created_at: 2019-10-14T03:57:17Z
updated_at: 2019-10-14T03:57:17Z
author_association: OWNER
body:
  So this is a new Docker feature from April 2019? https://engineering.docker.com/2019/04/multi-arch-images/

  > So why buildx?  Let’s first start with the name.  The x stands for experimental.  In the future, when these new build features are stable and made generally available, we will drop the x and integrate these features directly into the existing docker build command.  Note that as buildx is experimental, features and flags are subject to change.

  Is there a strong use-case for supporting this while it's still experimental? Users who want Datasette to run on ARM can build their own image pretty easily, so I'm not sure if it's worth depending on an experimental feature for the core Datasette project.

  Is there any indication from the Docker team as to when this experimental feature might stop being marked as experimental?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/590#issuecomment-541562581
issue_url: https://api.github.com/repos/simonw/datasette/issues/590
node_id: MDEyOklzc3VlQ29tbWVudDU0MTU2MjU4MQ==
created_at: 2019-10-14T08:57:46Z
updated_at: 2019-10-14T08:57:46Z
author_association: CONTRIBUTOR
body:
  Ah, thank you – I saw the need for unit tests but wasn't sure what the best way to add one would be.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/595#issuecomment-541664602
issue_url: https://api.github.com/repos/simonw/datasette/issues/595
node_id: MDEyOklzc3VlQ29tbWVudDU0MTY2NDYwMg==
created_at: 2019-10-14T13:03:10Z
updated_at: 2019-10-14T13:03:10Z
author_association: NONE
body:
  🤷‍♂️ @stonebig's suggestion would be the best I got too, *if* you want to support 3.5->3.8.

  It's either that, or hold off on 3.8 support until you're ready to go to 3.6->3.8.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/7#issuecomment-541721437
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/7
node_id: MDEyOklzc3VlQ29tbWVudDU0MTcyMTQzNw==
created_at: 2019-10-14T14:44:12Z
updated_at: 2019-10-14T14:44:12Z
author_association: MEMBER
body:
  Docs: https://github.com/dogsheep/github-to-sqlite/blob/0.5/README.md#retrieving-issue-comments-for-a-repository

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-541748580
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10
node_id: MDEyOklzc3VlQ29tbWVudDU0MTc0ODU4MA==
created_at: 2019-10-14T15:30:44Z
updated_at: 2019-10-14T15:30:44Z
author_association: MEMBER
body:
  Had several recommendations for https://github.com/tqdm/tqdm which is what goodreads-to-sqlite uses.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/578#issuecomment-541837823
issue_url: https://api.github.com/repos/simonw/datasette/issues/578
node_id: MDEyOklzc3VlQ29tbWVudDU0MTgzNzgyMw==
created_at: 2019-10-14T18:19:42Z
updated_at: 2019-10-14T18:19:42Z
author_association: NONE
body:
  My use case was: I wanted to use datasette on a Raspberry Pi. `docker pull datasetteproject/datasette` pulled the official image, which then failed to execute because it is not ARM ready. Building my own quite took some time (~60 minutes via Qemu on Intel i5). 

  You are right, the build method is quite new and I would not be surprised if the syntax / command will change in future. The outcome however, a Docker multi-architecture manifest, is aligned with Docker's strategy on how to tackle multiple architectures: transparently, on the registry-side.

  I just thought it would be nice to have the official image ready for multiple architectures. But I fully understand if the current methods feel too experimental to be mergable...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/595#issuecomment-541922637
issue_url: https://api.github.com/repos/simonw/datasette/issues/595
node_id: MDEyOklzc3VlQ29tbWVudDU0MTkyMjYzNw==
created_at: 2019-10-14T21:08:19Z
updated_at: 2019-10-14T21:08:19Z
author_association: OWNER
body:
  There more discussion around this on the Glitch forums here: https://support.glitch.com/t/can-you-upgrade-python-to-latest-version/7980

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/596#issuecomment-542462126
issue_url: https://api.github.com/repos/simonw/datasette/issues/596
node_id: MDEyOklzc3VlQ29tbWVudDU0MjQ2MjEyNg==
created_at: 2019-10-16T00:45:45Z
updated_at: 2019-10-16T00:45:45Z
author_association: OWNER
body:
  This means moving away from `select *`. I've been thinking this would be worthwhile anyway, since that way when you click "Edit SQL" you'll get a more useful SQL statement to start hacking away at.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542832952
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19
node_id: MDEyOklzc3VlQ29tbWVudDU0MjgzMjk1Mg==
created_at: 2019-10-16T18:30:11Z
updated_at: 2019-10-16T18:30:11Z
author_association: MEMBER
body:
  The `--since` option will derive the `since_id` from the max ID in the `timeline_tweets` table:

      $ twitter-to-sqlite home-timeline --since

  The `--since_id=xxx` option lets you specify that ID directly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542849963
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19
node_id: MDEyOklzc3VlQ29tbWVudDU0Mjg0OTk2Mw==
created_at: 2019-10-16T19:13:06Z
updated_at: 2019-10-16T19:13:06Z
author_association: MEMBER
body:
  Updated documentation: https://github.com/dogsheep/twitter-to-sqlite/blob/fced2a9b67d2cbdf9817f1eb75f7c28e413c963b/README.md#retrieving-tweets-from-your-home-timeline

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-542854749
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDU0Mjg1NDc0OQ==
created_at: 2019-10-16T19:26:01Z
updated_at: 2019-10-16T19:26:01Z
author_association: MEMBER
body:
  I'm not going to do this for "accounts that have followed me" and "new accounts that I have followed" - instead I will recommend running the `friend_ids` and `followers_ids` commands on a daily basis since that data doesn't really change much by the hour.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855081
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12
node_id: MDEyOklzc3VlQ29tbWVudDU0Mjg1NTA4MQ==
created_at: 2019-10-16T19:26:56Z
updated_at: 2019-10-16T19:26:56Z
author_association: MEMBER
body:
  This may be the first case where I want to be able to repair existing databases rather than discarding their contents.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855427
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12
node_id: MDEyOklzc3VlQ29tbWVudDU0Mjg1NTQyNw==
created_at: 2019-10-16T19:27:55Z
updated_at: 2019-10-16T19:27:55Z
author_association: MEMBER
body:
  I can do that by keeping `source` as a `TEXT` column but turning it into a non-enforced foreign key against a new `sources` table. Then I can run code that scans that column for any values beginning with a `<` and converts them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542858025
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12
node_id: MDEyOklzc3VlQ29tbWVudDU0Mjg1ODAyNQ==
created_at: 2019-10-16T19:35:31Z
updated_at: 2019-10-16T19:36:09Z
author_association: MEMBER
body:
  Maybe this means I need an `upgrade` command to apply these kinds of migrations? Total feature creep!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/597#issuecomment-542872267
issue_url: https://api.github.com/repos/simonw/datasette/issues/597
node_id: MDEyOklzc3VlQ29tbWVudDU0Mjg3MjI2Nw==
created_at: 2019-10-16T20:13:21Z
updated_at: 2019-10-16T20:13:21Z
author_association: OWNER
body:
  ```
  $ echo '{"hello": "world"}' | sqlite-utils insert foo.db hello -
  $ echo '{"hello": "world"}' | sqlite-utils insert foo-bar.db hello -
  $ datasette publish now \
    --about_url=https://github.com/simonw/datasette/issues/597 \
    --title="Issue #597" \
    --alias=datasette-issue-597 \
    foo.db foo-bar.db
  ```
  This failed to replicate the issue. https://datasette-issue-597.now.sh/ is working as it should.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542875885
issue_url: https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3
node_id: MDEyOklzc3VlQ29tbWVudDU0Mjg3NTg4NQ==
created_at: 2019-10-16T20:23:08Z
updated_at: 2019-10-16T20:23:08Z
author_association: MEMBER
body:
  https://developer.foursquare.com/docs/api/users/checkins documents `afterTimestamp`:
  > Retrieve the first results to follow these seconds since epoch. This should be useful for paging forward in time, or when polling for changes. To avoid missing results when polling, we recommend subtracting several seconds from the last poll time and then de-duplicating.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543217890
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDU0MzIxNzg5MA==
created_at: 2019-10-17T15:03:10Z
updated_at: 2019-10-17T15:03:10Z
author_association: MEMBER
body:
  Thinking about this further: the concept of migrations may end up being in direct conflict with the `sqlite-utils` concept of creating tables on demand the first time they are used - and of creating table schemas automatically to fit the shape of the JSON that is being inserted into them.

  I'm going to forge ahead anyway and build this because I think it will be an interesting exploration, but it's very likely this turns out to be a bad idea in the long run!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543222239
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDU0MzIyMjIzOQ==
created_at: 2019-10-17T15:12:33Z
updated_at: 2019-10-17T15:12:33Z
author_association: MEMBER
body:
  Migrations will run only if you open a database that previously existed (as opposed to opening a brand new empty database).

  This means that the first time you run a command against a fresh database, migrations will not run and the `migrations` table will not be created. The _second_ time you run any command against that database the migrations will execute and populate the `migrations` table.

  This also means that each migration needs to be able to sanity check the database to see if it should run or not. If it should NOT run, it will do nothing but still be marked as having executed by adding to the `migrations` table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543265058
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25
node_id: MDEyOklzc3VlQ29tbWVudDU0MzI2NTA1OA==
created_at: 2019-10-17T16:51:12Z
updated_at: 2019-10-17T16:51:12Z
author_association: MEMBER
body:
  This migration function only runs if there is a table called `tweets` and the migration has not run before.

  I think this can happen if the database has just been freshly created (by a command that fetches the user's user timeline for example) and is then run a SECOND time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543269396
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10
node_id: MDEyOklzc3VlQ29tbWVudDU0MzI2OTM5Ng==
created_at: 2019-10-17T17:02:07Z
updated_at: 2019-10-17T17:02:07Z
author_association: MEMBER
body:
  A neat trick that Click does is detecting if an interactive terminal is attached and NOT showing a progress bar if there isn't one. Need to figure out how to do that with tqdm.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543270714
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10
node_id: MDEyOklzc3VlQ29tbWVudDU0MzI3MDcxNA==
created_at: 2019-10-17T17:05:16Z
updated_at: 2019-10-17T17:05:16Z
author_association: MEMBER
body:
  https://github.com/pallets/click/blob/716a5be90f56ce6cd506bb53d5739d09374b1636/click/_termui_impl.py#L93 is how Click does this:
  ```
          self.is_hidden = not isatty(self.file)
  ```
  Where `isatty` is a Click utility function: `from ._compat import isatty`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543271000
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10
node_id: MDEyOklzc3VlQ29tbWVudDU0MzI3MTAwMA==
created_at: 2019-10-17T17:05:59Z
updated_at: 2019-10-17T17:05:59Z
author_association: MEMBER
body:
  Looks like tqdm already does a TTY check here: https://github.com/tqdm/tqdm/blob/89b73bdc30c099c5b53725806e7edf3a121c9b3a/tqdm/std.py#L889-L890

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543273540
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3
node_id: MDEyOklzc3VlQ29tbWVudDU0MzI3MzU0MA==
created_at: 2019-10-17T17:12:51Z
updated_at: 2019-10-17T17:12:51Z
author_association: MEMBER
body:
  Just importing tweets here isn't enough - how are we supposed to know which tweets were imported by which search?

  So I think the right thing to do here is to also create a `search_runs` table, which records each individual run of this tool (with a timestamp and the search terms used). Then have a `search_runs_tweets` m2m table which shows which Tweets were found by that search.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543290744
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3
node_id: MDEyOklzc3VlQ29tbWVudDU0MzI5MDc0NA==
created_at: 2019-10-17T17:57:14Z
updated_at: 2019-10-17T17:57:14Z
author_association: MEMBER
body:
  I have a working command now. I'm going to ship it early because it could do with some other people trying it out.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/597#issuecomment-543389140
issue_url: https://api.github.com/repos/simonw/datasette/issues/597
node_id: MDEyOklzc3VlQ29tbWVudDU0MzM4OTE0MA==
created_at: 2019-10-17T22:34:57Z
updated_at: 2019-10-17T22:34:57Z
author_association: OWNER
body:
  I've managed to replicate the bug again in my private instance, using databases called `twitter.db` and `twitter-bug.db`. When I visit `/twitter-bug` I see (but I am not redirected to) the `/twitter` page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/597#issuecomment-543389950
issue_url: https://api.github.com/repos/simonw/datasette/issues/597
node_id: MDEyOklzc3VlQ29tbWVudDU0MzM4OTk1MA==
created_at: 2019-10-17T22:37:41Z
updated_at: 2019-10-17T22:37:41Z
author_association: OWNER
body:
  Aha! It turns out my attempt to replicate the bug DID work:

  https://datasette-issue-597.now.sh/foo-bar

  <img width="742" alt="foo" src="https://user-images.githubusercontent.com/9599/67052770-0babb680-f0f4-11e9-86c7-487b42783394.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/584#issuecomment-543427426
issue_url: https://api.github.com/repos/simonw/datasette/issues/584
node_id: MDEyOklzc3VlQ29tbWVudDU0MzQyNzQyNg==
created_at: 2019-10-18T00:39:15Z
updated_at: 2019-10-18T00:39:15Z
author_association: OWNER
body:
  It looks like this is the problem:
  https://github.com/simonw/datasette/blob/90d4f497f9b3f6a5882937c91fddb496ac3e7368/datasette/utils/asgi.py#L216-L223
  That should be `utf-8`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/597#issuecomment-543900283
issue_url: https://api.github.com/repos/simonw/datasette/issues/597
node_id: MDEyOklzc3VlQ29tbWVudDU0MzkwMDI4Mw==
created_at: 2019-10-18T19:25:57Z
updated_at: 2019-10-18T19:27:06Z
author_association: OWNER
body:
  I'm suspicious of this code here:
  https://github.com/simonw/datasette/blob/b6ad1fdc7068cb8248787843e7438d1f19fa2e3a/datasette/views/base.py#L193-L205

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/601#issuecomment-544008944
issue_url: https://api.github.com/repos/simonw/datasette/issues/601
node_id: MDEyOklzc3VlQ29tbWVudDU0NDAwODk0NA==
created_at: 2019-10-18T23:40:48Z
updated_at: 2019-10-18T23:40:48Z
author_association: CONTRIBUTOR
body:
  The only negative impact that comes to mind is that now you have no way to get the read-only query to be formatted nicely, I think, so maybe a second PR adding the formatting functionality even to the read-only page would be good?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/601#issuecomment-544014518
issue_url: https://api.github.com/repos/simonw/datasette/issues/601
node_id: MDEyOklzc3VlQ29tbWVudDU0NDAxNDUxOA==
created_at: 2019-10-18T23:57:44Z
updated_at: 2019-10-18T23:57:44Z
author_association: OWNER
body:
  That's a good idea. It could even be smart about it and only show the "Format SQL" button on that page of the output of the function would differ from what's there already.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/588#issuecomment-544318343
issue_url: https://api.github.com/repos/simonw/datasette/issues/588
node_id: MDEyOklzc3VlQ29tbWVudDU0NDMxODM0Mw==
created_at: 2019-10-21T01:47:11Z
updated_at: 2019-10-21T01:47:11Z
author_association: OWNER
body:
  I think I see what's gone wrong here. Here's your example re-indented:
  ```json
  {
      "databases": {
          "MYDB": {
              "tables": {
                  "MYFIRSTTABLE": {
                      "source": "Test",
                      "source_url": "https://www.google.com",
                      "queries": {
                          "Query 1": {
                              "sql": "select * from MYFIRSTTABLE",
                              "title": "Query 1",
                              "description": "This is the first query"
                          }
                      }
                  },
                  "MYSECONDTABLE": {
                      "source": "Test2",
                      "source_url": "https://www.google.com",
                      "queries": {
                          "Query 2": {
                              "sql": "select * from MYSECONDTABLE;",
                              "title": "Query 2",
                              "description": "This is the second query"
                          }
                      }
                  }
              }
          }
      }
  }
  ```

  The `"queries" section isn't supported at the table level - it only works at the database level. So that example should look like this:

  ```json
  {
      "databases": {
          "MYDB": {
              "tables": {
                  "MYFIRSTTABLE": {
                      "source": "Test",
                      "source_url": "https://www.google.com"
                  },
                  "MYSECONDTABLE": {
                      "source": "Test2",
                      "source_url": "https://www.google.com"
                  }
              },
              "queries": {
                  "Query 1": {
                      "sql": "select * from MYFIRSTTABLE",
                      "title": "Query 1",
                      "description": "This is the first query"
                  },
                  "Query 2": {
                      "sql": "select * from MYSECONDTABLE;",
                      "title": "Query 2",
                      "description": "This is the second query"
                  }
              }
          }
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/260#issuecomment-544318517
issue_url: https://api.github.com/repos/simonw/datasette/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDU0NDMxODUxNw==
created_at: 2019-10-21T01:48:24Z
updated_at: 2019-10-21T01:48:24Z
author_association: OWNER
body:
  This came up in #588 - it would be helpful if this would spot things like `"queries"` defined against the tables block when they should be defined against a database.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-544335363
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDU0NDMzNTM2Mw==
created_at: 2019-10-21T03:32:04Z
updated_at: 2019-10-21T03:32:04Z
author_association: MEMBER
body:
  In case anyone is interested, here's an extract from the crontab I'm running these under at the moment:
  ```
  1,11,21,31,41,51 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite user-timeline /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --since
  2,7,12,17,22,27,32,37,42,47,52,57 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite home-timeline /home/ubuntu/timeline.db -a /home/ubuntu/auth.json --since
  6,16,26,36,46,56 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite favorites /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --stop_after=50
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/588#issuecomment-544502617
issue_url: https://api.github.com/repos/simonw/datasette/issues/588
node_id: MDEyOklzc3VlQ29tbWVudDU0NDUwMjYxNw==
created_at: 2019-10-21T12:58:22Z
updated_at: 2019-10-21T12:58:22Z
author_association: NONE
body:
  Thanks for the reply.  I was hoping queries per table were supported, as I have an application that builds tables depending on the user input to the application.  It will either create one table, or two.. and if one or the other is missing, certain queries will return errors.  Of course I can work around this by labeling the query name and hope users don't click queries they have not created a table for, but ideally the user would see the queries available based on the tables that exist in their database.  

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544646516
issue_url: https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDU0NDY0NjUxNg==
created_at: 2019-10-21T18:30:14Z
updated_at: 2019-10-21T18:30:14Z
author_association: MEMBER
body:
  Thanks to help from Dr. Laura Cantino at Science Hack Day San Francisco I've been able to pull together this query:

  ```sql
  select rsid, genotype, case genotype
    when 'AA' then 'brown eye color, 80% of the time'
    when 'AG' then 'brown eye color'
    when 'GG' then 'blue eye color, 99% of the time'
  end as interpretation from genome where rsid = 'rs12913832'
  ```

  See also https://www.snpedia.com/index.php/Rs12913832 - in particular this table:

  <img width="321" alt="rs12913832_-_SNPedia" src="https://user-images.githubusercontent.com/9599/67232392-216ff300-f3f6-11e9-8e14-b5f50c0c0d16.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-544648863
issue_url: https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDU0NDY0ODg2Mw==
created_at: 2019-10-21T18:36:03Z
updated_at: 2019-10-21T18:36:03Z
author_association: MEMBER
body:
  <img width="1418" alt="natalie__select_rsid__genotype__case_genotype_when__AA__then__brown_eye_color__80__of_the_time__when__AG__then__brown_eye_color__when__GG__then__blue_eye_color__99__of_the_time__end_as_interpretation_from_genome_where_rsid____rs12913832__an" src="https://user-images.githubusercontent.com/9599/67232810-f4701000-f3f6-11e9-90e2-8fe2cca1d98d.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/606#issuecomment-546134055
issue_url: https://api.github.com/repos/simonw/datasette/issues/606
node_id: MDEyOklzc3VlQ29tbWVudDU0NjEzNDA1NQ==
created_at: 2019-10-24T22:53:30Z
updated_at: 2019-10-24T22:53:30Z
author_association: OWNER
body:
  Here's a clue as to where to get it from:

  ```
  In [20]: mod, dist = pm.list_plugin_distinfo()[0]                                                                                                           

  In [21]: print(str(dist.metadata))                                                                                                                          
  Metadata-Version: 2.1
  Name: datasette-haversine
  Version: 0.1
  Summary: Datasette plugin that adds a custom SQL function for haversine distances
  Home-page: https://github.com/simonw/datasette-haversine
  Author: Simon Willison
  License: Apache License, Version 2.0
  Platform: UNKNOWN
  Description-Content-Type: text/markdown
  Requires-Dist: datasette
  Requires-Dist: haversine
  Provides-Extra: test
  Requires-Dist: pytest ; extra == 'test'

  # datasette-haversine

  [![PyPI](https://img.shields.io/pypi/v/datasette-haversine.svg)](https://pypi.org/project/datasette-haversine/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-haversine.svg?style=svg)](https://circleci.com/gh/simonw/datasette-haversine)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-haversine/blob/master/LICENSE)

  Datasette plugin that adds a custom SQL function for haversine distances
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/606#issuecomment-546134302
issue_url: https://api.github.com/repos/simonw/datasette/issues/606
node_id: MDEyOklzc3VlQ29tbWVudDU0NjEzNDMwMg==
created_at: 2019-10-24T22:54:39Z
updated_at: 2019-10-24T22:54:39Z
author_association: OWNER
body:
  Actually this is better:
  ```
  In [26]: dist.project_name                                                                                                                                  
  Out[26]: 'datasette-haversine'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/604#issuecomment-546155557
issue_url: https://api.github.com/repos/simonw/datasette/issues/604
node_id: MDEyOklzc3VlQ29tbWVudDU0NjE1NTU1Nw==
created_at: 2019-10-25T00:45:51Z
updated_at: 2019-10-25T00:45:51Z
author_association: OWNER
body:
  Relevant code:

  https://github.com/simonw/datasette/blob/912ce848b9fa8b1642c800b446f504518bc39f2a/datasette/templates/table.html#L94-L97

  And

  https://github.com/simonw/datasette/blob/b9ede4c1898616512b5d204f9c941deff473cbe4/datasette/views/table.py#L719-L722

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/604#issuecomment-546155658
issue_url: https://api.github.com/repos/simonw/datasette/issues/604
node_id: MDEyOklzc3VlQ29tbWVudDU0NjE1NTY1OA==
created_at: 2019-10-25T00:46:30Z
updated_at: 2019-10-25T00:47:41Z
author_association: OWNER
body:
  Just adding `_where` to that tuple won't do the trick, because `_where=` can be passed multiple times, e.g. https://latest.datasette.io/fixtures/roadside_attractions?_where=name%20like%20%27%museum%%27&_where=address%20like%20%27%Drive%%27

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/607#issuecomment-546722281
issue_url: https://api.github.com/repos/simonw/datasette/issues/607
node_id: MDEyOklzc3VlQ29tbWVudDU0NjcyMjI4MQ==
created_at: 2019-10-27T18:46:29Z
updated_at: 2019-10-27T19:00:40Z
author_association: NONE
body:
  Update: I've created a table of only unique names. This reduces the search space from over 16 million, to just about 640,000. Interestingly, it takes less than 2 seconds to create this table using Python. Performing the same search that we did earlier for `elon musk` takes nearly a second - much faster than before but still not speedy enough for an autocomplete feature (which usually needs to return results within 100ms to feel "real time"). 

  Any ideas for slashing the search speed nearly 10 fold?

  > ![image](https://user-images.githubusercontent.com/8431341/67639587-b6c02b00-f8bf-11e9-9344-1d8667cad395.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/607#issuecomment-546723302
issue_url: https://api.github.com/repos/simonw/datasette/issues/607
node_id: MDEyOklzc3VlQ29tbWVudDU0NjcyMzMwMg==
created_at: 2019-10-27T18:59:55Z
updated_at: 2019-10-27T19:00:48Z
author_association: NONE
body:
  Ultimately, I'm needing to serve searches like this to multiple users (at times concurrently). Given the size of the database I'm working with, can anyone comment as to whether I should be storing this in something like MySQL or Postgres rather than SQLite. I know there's been much [defense of sqlite being performant](https://www.sqlite.org/whentouse.html) but I wonder if those arguments break down as the database size increases.

  For example, if I scroll to the bottom of that linked page, where it says **Checklist For Choosing The Right Database Engine**, here's how I answer those questions:

   - Is the data separated from the application by a network?  → choose client/server
    __Yes__
  - Many concurrent writers? → choose client/server
    __Not exactly. I may have many concurrent readers but almost no concurrent writers.__
  - Big data? → choose client/server
    __No, my database is less than 40 gb and wont approach a terabyte in the next decade.__

  So is sqlite still a good idea here?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/607#issuecomment-546752311
issue_url: https://api.github.com/repos/simonw/datasette/issues/607
node_id: MDEyOklzc3VlQ29tbWVudDU0Njc1MjMxMQ==
created_at: 2019-10-28T00:37:10Z
updated_at: 2019-10-28T00:37:10Z
author_association: NONE
body:
  UPDATE:
  According to tips suggested in [Squeezing Performance from SQLite: Indexes? Indexes!](https://medium.com/@JasonWyatt/squeezing-performance-from-sqlite-indexes-indexes-c4e175f3c346) I have added an index to my large table and benchmarked query speeds in the case where I want to return `all rows`, `rows exactly equal to 'Musk Elon'` and, `rows like 'musk'`. Indexing reduced query time for each of those measures and **dramatically** reduced the time to return `rows exactly equal to 'Musk Elon'` as shown below:

  > table: edgar_idx
  > rows: 16,428,090 rows
  > **indexed: False**
  > Return all rows where company name exactly equal to Musk Elon
  > query: select rowid, * from edgar_idx where "company" = :p0 order by rowid limit 101
  > query time:   Query took 21821.031ms
  > 
  > Return all rows where company name contains Musk
  > query: select rowid, * from edgar_idx where "company" like :p0 order by rowid limit 101
  > query time: Query took 20505.029ms
  > 
  > Return everything
  > query: select rowid, * from edgar_idx order by rowid limit 101
  > query time:  Query took 7985.011ms
  > 
  > **indexed: True**
  > Return all rows where company name exactly equal to Musk Elon
  > query: select rowid, * from edgar_idx where "company" = :p0 order by rowid limit 101
  > query time:  Query took 30.0ms
  > 
  > Return all rows where company name contains Musk
  > query: select rowid, * from edgar_idx where "company" like :p0 order by rowid limit 101
  > query time:   Query took 13340.019ms
  > 
  > Return everything
  > query: select rowid, * from edgar_idx order by rowid limit 101
  > query time:   Query took 2190.003ms

  So indexing reduced query time for an exact match to "Musk Elon" from almost `22 seconds` to `30.0ms`. **That's amazing and truly promising!** However, an autocomplete feature relies on fuzzy / incomplete matching, which is more similar to the `contains 'musk'` query... Unfortunately, that takes 13 seconds even after indexing. So the hunt for a fast fuzzy /  autocomplete search capability persists.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/594#issuecomment-547373739
issue_url: https://api.github.com/repos/simonw/datasette/issues/594
node_id: MDEyOklzc3VlQ29tbWVudDU0NzM3MzczOQ==
created_at: 2019-10-29T11:21:52Z
updated_at: 2019-10-29T11:21:52Z
author_association: NONE
body:
  Just an FYI for folks wishing to run datasette with Python 3.8, I was able to successfully use datasette with the following in a virtual environment:

  ```
  pip install uvloop==0.14.0rc1
  pip install uvicorn==0.9.1
  ```

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0}


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/26#issuecomment-547713287
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDU0NzcxMzI4Nw==
created_at: 2019-10-30T02:36:13Z
updated_at: 2019-10-30T02:36:13Z
author_association: MEMBER
body:
  Shipped this in 0.13: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.13

  See also this Twitter thread: https://twitter.com/simonw/status/1189369677509623809

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/607#issuecomment-548055544
issue_url: https://api.github.com/repos/simonw/datasette/issues/607
node_id: MDEyOklzc3VlQ29tbWVudDU0ODA1NTU0NA==
created_at: 2019-10-30T18:37:44Z
updated_at: 2019-10-30T18:37:52Z
author_association: OWNER
body:
  .Hi @zeluspudding 

  You're running your search queries using the "contains" filter, which uses a `like` query under the hood.

  SQL `like` queries are generally slow because they force a full table scan. You can add an index on the column but it will only speed up prefix queries, like `... where name like 'apple%'` - they won't help if you are searching for text further along the string.

  Instead, you should take a look at SQLite's FTS - full text indexing feature. You can build a FTS index against a column and dramatically speed up searches for words within that column.

  This documentation should help get you started: https://datasette.readthedocs.io/en/stable/full_text_search.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/605#issuecomment-548056066
issue_url: https://api.github.com/repos/simonw/datasette/issues/605
node_id: MDEyOklzc3VlQ29tbWVudDU0ODA1NjA2Ng==
created_at: 2019-10-30T18:38:54Z
updated_at: 2019-10-30T18:38:54Z
author_association: OWNER
body:
  Could you flesh this out a little and help me understand what this might look like? If you define a query against a specific table in `metadata.json` where would you expect that query to be displayed in the Datasette UI?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/605#issuecomment-548058715
issue_url: https://api.github.com/repos/simonw/datasette/issues/605
node_id: MDEyOklzc3VlQ29tbWVudDU0ODA1ODcxNQ==
created_at: 2019-10-30T18:44:41Z
updated_at: 2019-10-30T18:55:37Z
author_association: NONE
body:
  Sure.  I imagine it being pretty straight forward.  Today when you click on the database, the UI displays:

  -Table 1-
    -fields-
    -row count-
  -Table 2-
    -fields-
    -row count-
  Queries:
    -query1-
    -query2-
    ..
    ...

  My proposal would be to display as follows:

  -Table 1-
    -fields-
    -row count-
    Queries:
    -query1-
    -query2-
    ..
    ...
  -Table 2-
    -fields-
    -row count-
    Queries:
    -query1-
    -query2-
    ..
    ...

  This way, if a given table is not present in the database, the associated queries are also not present.  Today, I have a list of queries, some work, some result in errors depending on whether the dependent tables exist in the database.

  Let me know if that makes sense.  Thanks again!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/607#issuecomment-548060038
issue_url: https://api.github.com/repos/simonw/datasette/issues/607
node_id: MDEyOklzc3VlQ29tbWVudDU0ODA2MDAzOA==
created_at: 2019-10-30T18:47:57Z
updated_at: 2019-10-30T18:47:57Z
author_association: NONE
body:
  Hi Simon, thanks for the pointer! Feeling good that I came to your conclusion a few days ago. I did hit a snag with figuring out how to compile a special version of sqlite for my windows machine (which I only realized I needed to do after running your command `sqlite-utils enable-fts mydatabase.db items name description`). 

  I'll try to solve that problem next week and report back here with my findings (if you know of a good tutorial for compiling on windows, I'm all ears). Either way, I'll try to close this issue out in the next two weeks. Thanks again!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/176#issuecomment-548508237
issue_url: https://api.github.com/repos/simonw/datasette/issues/176
node_id: MDEyOklzc3VlQ29tbWVudDU0ODUwODIzNw==
created_at: 2019-10-31T18:25:44Z
updated_at: 2019-10-31T18:25:44Z
author_association: NONE
body:
  👋 I'd be interested in building this out in Q1 or Q2 of 2020 if nobody has tackled it by then. I would love to integrate Datasette into @thechicagoreporter's practice, but we're also fully committed to GraphQL moving forward.

reactions:
  {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 2, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/606#issuecomment-548673320
issue_url: https://api.github.com/repos/simonw/datasette/issues/606
node_id: MDEyOklzc3VlQ29tbWVudDU0ODY3MzMyMA==
created_at: 2019-11-01T05:41:03Z
updated_at: 2019-11-01T05:41:03Z
author_association: OWNER
body:
  Couldn't easily write a test for this as plugin tests don't currently work against packaged plugins.

  Here's the new `datasette plugins` output:
  ```
  $ datasette plugins
  [
      {
          "name": "datasette-cluster-map",
          "static": true,
          "templates": false,
          "version": "0.6"
      },
      {
          "name": "datasette-json-html",
          "static": false,
          "templates": false,
          "version": "0.5"
      },
      {
          "name": "datasette-vega",
          "static": true,
          "templates": false,
          "version": "0.6.2"
      }
  ]
  ```
  Same output from `/-/plugins`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/574#issuecomment-548964895
issue_url: https://api.github.com/repos/simonw/datasette/issues/574
node_id: MDEyOklzc3VlQ29tbWVudDU0ODk2NDg5NQ==
created_at: 2019-11-01T21:58:17Z
updated_at: 2019-11-01T21:58:17Z
author_association: OWNER
body:
  New description:
  ```
    -h, --host TEXT           Host for server. Defaults to 127.0.0.1 which means
                              only connections from the local machine will be
                              allowed. Use 0.0.0.0 to listen to all IPs and
                              allow access from other machines.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/611#issuecomment-548967432
issue_url: https://api.github.com/repos/simonw/datasette/issues/611
node_id: MDEyOklzc3VlQ29tbWVudDU0ODk2NzQzMg==
created_at: 2019-11-01T22:08:57Z
updated_at: 2019-11-01T22:09:16Z
author_association: OWNER
body:
  Argh this is actually a bug in the already-shipped plugins themselves:

  ```python
  @hookimpl
  def extra_js_urls():
      return [
          {
              "url": "https://unpkg.com/leaflet@1.5.1/dist/leaflet.js",
              "sri": "sha512-GffPMF3RvMeYyc1LWMHtK8EbPv0iNZ8/oTtHPx9/cc2ILxQ+u905qIwdpULaqDkyBKgOaB57QTMg7ztg8Jm2Og==",
          },
          "/-/static-plugins/datasette_leaflet_geojson/datasette-leaflet-geojson.js",
      ]
  ```
  https://github.com/simonw/datasette-leaflet-geojson/blob/f8b5b1bd797b8c1b9a1771c259037dca258cce40/datasette_leaflet_geojson/__init__.py#L14-L22

  Maybe I should revert #606?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/611#issuecomment-548967774
issue_url: https://api.github.com/repos/simonw/datasette/issues/611
node_id: MDEyOklzc3VlQ29tbWVudDU0ODk2Nzc3NA==
created_at: 2019-11-01T22:10:31Z
updated_at: 2019-11-01T22:10:31Z
author_association: OWNER
body:
  The alternative would be to ship brand new fixed versions of all of these plugins and have those fixes require a minimum release of Datasette.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/611#issuecomment-548968034
issue_url: https://api.github.com/repos/simonw/datasette/issues/611
node_id: MDEyOklzc3VlQ29tbWVudDU0ODk2ODAzNA==
created_at: 2019-11-01T22:11:33Z
updated_at: 2019-11-01T22:11:33Z
author_association: OWNER
body:
  Or I could support both `with-hyphens` and `with_underscores` variants of the plugin names? That would provide a painless upgrade path - maybe this is something I could clean up in the backwards compatibility breaking 1.0 version?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/612#issuecomment-549088458
issue_url: https://api.github.com/repos/simonw/datasette/issues/612
node_id: MDEyOklzc3VlQ29tbWVudDU0OTA4ODQ1OA==
created_at: 2019-11-02T22:55:35Z
updated_at: 2019-11-02T22:55:35Z
author_association: OWNER
body:
  The bug is in this code here:

  https://github.com/simonw/datasette/blob/c3181d9a840dff7be8c990b21f5749db393a4ea0/datasette/views/base.py#L331-L338

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549094195
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8
node_id: MDEyOklzc3VlQ29tbWVudDU0OTA5NDE5NQ==
created_at: 2019-11-03T00:43:16Z
updated_at: 2019-11-03T00:43:28Z
author_association: MEMBER
body:
  Also need to take #5 into account - if this command creates incomplete user records, how do we repair them?

  And make sure that if we run this command first any future commands that populate users don't break (probably just a case of using `alter=True` in a few places).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/5#issuecomment-549094229
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDU0OTA5NDIyOQ==
created_at: 2019-11-03T00:44:03Z
updated_at: 2019-11-03T00:44:03Z
author_association: MEMBER
body:
  Might not need an incomplete boolean - may be possible to handle this with `alter=True` and then by filtering for users with null values in certain columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095217
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDU0OTA5NTIxNw==
created_at: 2019-11-03T01:06:25Z
updated_at: 2019-11-03T01:06:25Z
author_association: MEMBER
body:
  Wow, that `retweets_of_me` endpoint is almost completely useless:
  ```
  $ twitter-to-sqlite fetch https://api.twitter.com/1.1/statuses/retweets_of_me.json
  ```
  It returns my own tweets that have been retweeted, but with no indication at all of who retweeted them.

  It looks like this needs to be combined with this API - https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-retweets-id - to fetch the details of up to 100 recent users who actually DID retweet an individual status. But that has a one-every-12-seconds rate limit on it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095317
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDU0OTA5NTMxNw==
created_at: 2019-11-03T01:08:10Z
updated_at: 2019-11-03T01:08:10Z
author_association: MEMBER
body:
  Hmm... one thing that could be useful is that `retweets_of_me` can support a `--since` parameter - so if run frequently it should hopefully let us know which tweets we would need to run `statuses/retweets/:id.json` against.

  I'm not sure if the `--since` parameter would show me a tweet that was previously retweeted but has now been retweeted again. I'll have a bit of a test and see.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095463
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDU0OTA5NTQ2Mw==
created_at: 2019-11-03T01:10:52Z
updated_at: 2019-11-03T01:10:52Z
author_association: MEMBER
body:
  I imagine it won't, since the data I would be recording and then passing to `since_id` would be the highest ID of my own tweets that have been retweeted at least once. So it won't be able to spot if I should check for fresh retweets of a given tweet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095641
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDU0OTA5NTY0MQ==
created_at: 2019-11-03T01:12:58Z
updated_at: 2019-11-03T01:12:58Z
author_association: MEMBER
body:
  It looks like Twitter really want you to subscribe to a premium API for this kind of thing and consume retweets via webhooks: https://developer.twitter.com/en/docs/accounts-and-users/subscribe-account-activity/api-reference

  I'm going to give up on this for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549096321
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3
node_id: MDEyOklzc3VlQ29tbWVudDU0OTA5NjMyMQ==
created_at: 2019-11-03T01:27:55Z
updated_at: 2019-11-03T01:28:17Z
author_association: MEMBER
body:
  It would be neat if this could support `--since`, with that argument automatically finding the maximum tweet ID from a previous search that used the same exact arguments (using the `search_runs` table).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/577#issuecomment-549160499
issue_url: https://api.github.com/repos/simonw/datasette/issues/577
node_id: MDEyOklzc3VlQ29tbWVudDU0OTE2MDQ5OQ==
created_at: 2019-11-03T17:36:59Z
updated_at: 2019-11-03T17:36:59Z
author_association: OWNER
body:
  I think a `.render_template()` method on the `Datasette` class would be acceptable, since the purpose of that class will mainly be to provide a documented API for plugins to use: #576 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/613#issuecomment-549168834
issue_url: https://api.github.com/repos/simonw/datasette/issues/613
node_id: MDEyOklzc3VlQ29tbWVudDU0OTE2ODgzNA==
created_at: 2019-11-03T19:14:01Z
updated_at: 2019-11-03T19:14:01Z
author_association: OWNER
body:
  Consider https://latest.datasette.io/fixtures/facetable

  With `?_join=city_id` the underlying query would become this:

  ```
  select
    facetable.pk as 'facetable.pk',
    facetable.created as 'facetable.created',
    facetable.planet_int as 'facetable.planet_int',
    facetable.on_earth as 'facetable.on_earth',
    facetable.state as 'facetable.state',
    facetable.city_id as 'facetable.city_id',
    facetable.neighborhood as 'facetable.neighborhood',
    facetable.tags as 'facetable.tags',
    facetable.complex_array as 'facetable.complex_array',
    facet_cities.id as 'facet_cities.id',
    facet_cities.name as 'facet_cities.name'
  from
    facetable
    join facet_cities on facetable.city_id = facet_cities.id
  ```
  https://latest.datasette.io/fixtures?sql=select%0D%0A++facetable.pk+as+%27facetable.pk%27%2C%0D%0A++facetable.created+as+%27facetable.created%27%2C%0D%0A++facetable.planet_int+as+%27facetable.planet_int%27%2C%0D%0A++facetable.on_earth+as+%27facetable.on_earth%27%2C%0D%0A++facetable.state+as+%27facetable.state%27%2C%0D%0A++facetable.city_id+as+%27facetable.city_id%27%2C%0D%0A++facetable.neighborhood+as+%27facetable.neighborhood%27%2C%0D%0A++facetable.tags+as+%27facetable.tags%27%2C%0D%0A++facetable.complex_array+as+%27facetable.complex_array%27%2C%0D%0A++facet_cities.id+as+%27facet_cities.id%27%2C%0D%0A++facet_cities.name+as+%27facet_cities.name%27%0D%0Afrom%0D%0A++facetable%0D%0A++join+facet_cities+on+facetable.city_id+%3D+facet_cities.id

  I'm using `select foo.bar as 'foo.bar'` here to ensure that every column has an unambiguous name, to avoid any weirdness from joining against a table where both tables have a 'name' column.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/292#issuecomment-549169101
issue_url: https://api.github.com/repos/simonw/datasette/issues/292
node_id: MDEyOklzc3VlQ29tbWVudDU0OTE2OTEwMQ==
created_at: 2019-11-03T19:17:08Z
updated_at: 2019-11-03T19:17:16Z
author_association: OWNER
body:
  A good basic starting point for this would be to ignore the ability to add custom SQL fragments and instead focus on being able to show and hide specific columns. This will play particularly well with #613.

  Proposed syntax for that:

  `/db/table?_col=id&_col=name` - just show the `id` and `name` columns
  `/db/table?_nocol=extras&_nocol=age` - show all columns except for `extras` and `age`

  I don't think it makes sense to allow both `?_col=` and `?_nocol=` arguments in the same request, so if you provide both I think we throw a 400 error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/602#issuecomment-549210237
issue_url: https://api.github.com/repos/simonw/datasette/issues/602
node_id: MDEyOklzc3VlQ29tbWVudDU0OTIxMDIzNw==
created_at: 2019-11-04T02:49:18Z
updated_at: 2019-11-04T02:49:18Z
author_association: OWNER
body:
  Live demo can now be seen here: https://latest.datasette.io/fixtures/%F0%9D%90%9C%F0%9D%90%A2%F0%9D%90%AD%F0%9D%90%A2%F0%9D%90%9E%F0%9D%90%AC

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/602#issuecomment-549210352
issue_url: https://api.github.com/repos/simonw/datasette/issues/602
node_id: MDEyOklzc3VlQ29tbWVudDU0OTIxMDM1Mg==
created_at: 2019-11-04T02:50:15Z
updated_at: 2019-11-04T02:50:15Z
author_association: OWNER
body:
  I wonder if the `Format SQL` button should disappear in the read-only case once it has been clicked? It won't do anything if it's clicked again and there's no textarea for the user to mess up the now-formatted SQL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549226399
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3
node_id: MDEyOklzc3VlQ29tbWVudDU0OTIyNjM5OQ==
created_at: 2019-11-04T05:11:57Z
updated_at: 2019-11-04T05:11:57Z
author_association: MEMBER
body:
  I'm going to add a `hash` column to `search_runs` to support that. It's going to be the sha1 hash of the key-ordered JSON of the search arguments used by that run. Then `--since` can look for an identical hash and use it to identify the highest last fetched tweet to use in `since_id`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549228535
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3
node_id: MDEyOklzc3VlQ29tbWVudDU0OTIyODUzNQ==
created_at: 2019-11-04T05:31:55Z
updated_at: 2019-11-04T05:31:55Z
author_association: MEMBER
body:
  Documented here: https://github.com/dogsheep/twitter-to-sqlite/blob/801c0c2daf17d8abce9dcb5d8d610410e7e25dbe/README.md#running-searches

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-549230337
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10
node_id: MDEyOklzc3VlQ29tbWVudDU0OTIzMDMzNw==
created_at: 2019-11-04T05:47:18Z
updated_at: 2019-11-04T05:47:18Z
author_association: MEMBER
body:
  This definition isn't quite right - it's not pulling the identity of the user who starred the repo (`users.login` ends up being the owner login instead).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549233778
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8
node_id: MDEyOklzc3VlQ29tbWVudDU0OTIzMzc3OA==
created_at: 2019-11-04T06:14:40Z
updated_at: 2019-11-04T06:14:40Z
author_association: MEMBER
body:
  Spotted a tricky problem: running `github-to-sqlite starred stargazers.db` results in an incomplete `simonw` record. It creates a proper record for me thanks to this bit:

  https://github.com/dogsheep/github-to-sqlite/blob/ea07274667a08c67907e8bfbbccb6f0fb95ce817/github_to_sqlite/cli.py#L120-L126

  But then... when it gets to the `datasette` repository which I have starred it over-writes my full user record with one that's missing most of the details, thanks to this bit:

  https://github.com/dogsheep/github-to-sqlite/blob/ea07274667a08c67907e8bfbbccb6f0fb95ce817/github_to_sqlite/utils.py#L117-L124

  I need to find a way of NOT over-writing a good record with a thinner one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549425012
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/62
node_id: MDEyOklzc3VlQ29tbWVudDU0OTQyNTAxMg==
created_at: 2019-11-04T16:09:08Z
updated_at: 2019-11-04T16:09:08Z
author_association: OWNER
body:
  Documentation for `table.delete()`: https://github.com/simonw/sqlite-utils/blob/19073d6d972fad9d68dd74c28544cd29083f1c12/docs/python-api.rst#deleting-a-specific-record

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549425364
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/62
node_id: MDEyOklzc3VlQ29tbWVudDU0OTQyNTM2NA==
created_at: 2019-11-04T16:09:56Z
updated_at: 2019-11-04T16:09:56Z
author_association: OWNER
body:
  Since we have `table.rows_where(where, where_args)` it makes sense to me to also support `table.delete_where(where, where_args)`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549429512
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/62
node_id: MDEyOklzc3VlQ29tbWVudDU0OTQyOTUxMg==
created_at: 2019-11-04T16:18:48Z
updated_at: 2019-11-04T16:18:48Z
author_association: OWNER
body:
  Documentation for `.delete_where()`: https://github.com/simonw/sqlite-utils/blob/169ea455fc1f1d5e5b6e44cb339ba7ffa9d49c31/docs/python-api.rst#deleting-multiple-records

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/63#issuecomment-549430429
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/63
node_id: MDEyOklzc3VlQ29tbWVudDU0OTQzMDQyOQ==
created_at: 2019-11-04T16:20:35Z
updated_at: 2019-11-04T16:20:35Z
author_association: OWNER
body:
  I don't think we need this. We already have a `dogs.create_index(["name"], if_not_exists=True)` option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/61#issuecomment-549432592
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/61
node_id: MDEyOklzc3VlQ29tbWVudDU0OTQzMjU5Mg==
created_at: 2019-11-04T16:25:01Z
updated_at: 2019-11-04T16:25:01Z
author_association: OWNER
body:
  Yeah I've thought about this a bit and I'm OK leaving it out.

  The core idea of `sqlite-utils` is that if you can create a list, iterator or generator of Python dictionaries you can efficiently insert those into a SQLite table.

  The `--csv` function is actually implemented as just a few lines of code which turn that incoming CSV into a generator of dictionaries:

  https://github.com/simonw/sqlite-utils/blob/169ea455fc1f1d5e5b6e44cb339ba7ffa9d49c31/sqlite_utils/cli.py#L364-L368

  I could turn this into a reusable helper function but since it wouldn't have anything to do with database inserts (it would just be a helper that turns a CSV into a generator of dictionaries) it doesn't feel like it fits well in the Python library section of this package.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-549585571
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDU0OTU4NTU3MQ==
created_at: 2019-11-04T22:57:16Z
updated_at: 2019-11-04T22:57:16Z
author_association: OWNER
body:
  First step: stop using `select *` and switch to selecting columns instead. This will also make the `View and edit SQL` button more useful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-549591028
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDU0OTU5MTAyOA==
created_at: 2019-11-04T23:15:28Z
updated_at: 2019-11-04T23:15:28Z
author_association: OWNER
body:
  Demo: visit https://52fa79c.datasette.io/fixtures/roadside_attractions and click "View and edit SQL" - you'll be sent to https://52fa79c.datasette.io/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+limit+101

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/567#issuecomment-549665423
issue_url: https://api.github.com/repos/simonw/datasette/issues/567
node_id: MDEyOklzc3VlQ29tbWVudDU0OTY2NTQyMw==
created_at: 2019-11-05T05:11:14Z
updated_at: 2019-11-05T05:11:14Z
author_association: OWNER
body:
  @clausjuhl I wrote a bit about that here: https://simonwillison.net/2019/May/19/datasette-0-28/

  Short version: just point Datasette at a SQLite file and update it from another process - it should work fine! I do it all the time now - I'll have a script running that writes to a database and I'll use Datasette to monitor progress. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-550146151
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDU1MDE0NjE1MQ==
created_at: 2019-11-06T05:06:27Z
updated_at: 2019-11-06T05:06:27Z
author_association: OWNER
body:
  I'm going to always include the primary key (or rowid) - otherwise the table view will sometimes not include links to the row pages, which seems bad.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-550388354
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDU1MDM4ODM1NA==
created_at: 2019-11-06T16:26:55Z
updated_at: 2019-11-06T16:26:55Z
author_association: MEMBER
body:
  Here's a query I figured out using a window function that shows cumulative stargazers over time:
  ```sql
  select
    yyyymmdd,
    sum(n) over (
      order by
        yyyymmdd rows unbounded preceding
    ) as cumulative_count
  from
    (
      select
        substr(starred_at, 0, 11) as yyyymmdd,
        count(*) as n
      from
        stars
      group by
        yyyymmdd
    )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/607#issuecomment-550649607
issue_url: https://api.github.com/repos/simonw/datasette/issues/607
node_id: MDEyOklzc3VlQ29tbWVudDU1MDY0OTYwNw==
created_at: 2019-11-07T03:38:10Z
updated_at: 2019-11-07T03:38:10Z
author_association: NONE
body:
  I just got FTS5 working and it is incredible! The lookup time for returning all rows where company name contains "Musk" from my table of 16,428,090 rows has dropped from `13,340.019` ms to `15.6`ms. Well below the 100ms latency for the "real time autocomplete" feel (which doesn't currently include the http call).

  So cool! Thanks again for the pointers and awesome datasette!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550783316
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10
node_id: MDEyOklzc3VlQ29tbWVudDU1MDc4MzMxNg==
created_at: 2019-11-07T05:16:56Z
updated_at: 2019-11-07T05:34:29Z
author_association: MEMBER
body:
  It looks like Apple changed the location of these in iOS 13 - they are now in separate `.gpx` files:

  ![2FF70E95-CDEE-4241-A5C5-EE95A862E519](https://user-images.githubusercontent.com/9599/68362042-be12e000-00da-11ea-8925-7397410332d8.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550806302
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10
node_id: MDEyOklzc3VlQ29tbWVudDU1MDgwNjMwMg==
created_at: 2019-11-07T05:33:31Z
updated_at: 2019-11-07T05:33:31Z
author_association: MEMBER
body:
  The XML now includes references to these new files:

  ![CBBA54FC-51FB-4BB3-927C-C2CA99237B04](https://user-images.githubusercontent.com/9599/68362716-121ec400-00dd-11ea-9846-387c7cd64c8b.jpeg)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550824838
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10
node_id: MDEyOklzc3VlQ29tbWVudDU1MDgyNDgzOA==
created_at: 2019-11-07T05:47:07Z
updated_at: 2019-11-07T05:47:07Z
author_association: MEMBER
body:
  Relevant code:

  https://github.com/dogsheep/healthkit-to-sqlite/blob/d16f45f06fbae6ec8a78cc9ca7b5b7db0413f139/healthkit_to_sqlite/utils.py#L58-L64

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550828084
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10
node_id: MDEyOklzc3VlQ29tbWVudDU1MDgyODA4NA==
created_at: 2019-11-07T05:49:24Z
updated_at: 2019-11-07T05:49:24Z
author_association: MEMBER
body:
  So the fix there is going to be to detect the new `FileReference` element and load the corresponding points data from it.

  This will be a little tricky because that function will need access to the zip file.

  It probably won't work at all for the mode where the `export.xml` file is passed directly using the `--xml` option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/617#issuecomment-551349022
issue_url: https://api.github.com/repos/simonw/datasette/issues/617
node_id: MDEyOklzc3VlQ29tbWVudDU1MTM0OTAyMg==
created_at: 2019-11-08T01:56:28Z
updated_at: 2019-11-08T01:56:28Z
author_association: OWNER
body:
  Here's the current monstrosity:
  https://github.com/simonw/datasette/blob/f9c146b893856a48afa810ebcce1714f30d0d3a2/datasette/views/table.py#L210-L780

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/617#issuecomment-551349770
issue_url: https://api.github.com/repos/simonw/datasette/issues/617
node_id: MDEyOklzc3VlQ29tbWVudDU1MTM0OTc3MA==
created_at: 2019-11-08T01:59:20Z
updated_at: 2019-11-11T00:53:42Z
author_association: OWNER
body:
  This is what the code does:

  - Check if the table name is actually a canned query - if so execute that
  - Check for other parameters that need to be redirected - e.g. `?_filter_column=x` or `_sort_by_desc=y`
  - Turn the table + querystring parameters (`?foo=bar`, `?bar__contains=baz`, `?_where=`, `_sort=` etc) into a SQL query
  - Figure out pagination (apply limit, and handle [keyset pagination](https://simonwillison.net/2018/Oct/4/datasette-ideas/#Keyset_pagination))
  - Calculate suggested facets against the non-paginated version of the query
  - Calculate facet counts against the non-paginated version of the query
  - Execute the query
  - Expand any foreign key references from the results

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/617#issuecomment-551349885
issue_url: https://api.github.com/repos/simonw/datasette/issues/617
node_id: MDEyOklzc3VlQ29tbWVudDU1MTM0OTg4NQ==
created_at: 2019-11-08T01:59:43Z
updated_at: 2019-11-08T02:00:52Z
author_association: OWNER
body:
  A clean starting point would be to refactor out the "take a table and a bunch of querystring parameters and turn them into a SQL query" portion.

  Added bonus to this: I've long wanted to be able to apply the various trimmings of the TableView (like faceting and foreign key expansion) to other arbitrary SQL queries. Splitting out the code that builds the SELECT query will go a long way to making that a reality.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/616#issuecomment-551351578
issue_url: https://api.github.com/repos/simonw/datasette/issues/616
node_id: MDEyOklzc3VlQ29tbWVudDU1MTM1MTU3OA==
created_at: 2019-11-08T02:06:56Z
updated_at: 2019-11-08T02:06:56Z
author_association: OWNER
body:
  I think I know what's going on here.

  http://bases.vortex.media/-/versions says it's running `"version": "0.29.2"` of Datasette.

  http://teste-templates.herokuapp.com/-/versions (the one that works) is running `"version": "0.30.2"`

  I fixed a bug with FTS detection in 0.29.3 - https://datasette.readthedocs.io/en/stable/changelog.html#v0-29-3 fixed - #570

  So upgrading to the latest release should fix this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/608#issuecomment-551352256
issue_url: https://api.github.com/repos/simonw/datasette/issues/608
node_id: MDEyOklzc3VlQ29tbWVudDU1MTM1MjI1Ng==
created_at: 2019-11-08T02:09:43Z
updated_at: 2019-11-08T02:09:43Z
author_association: OWNER
body:
  I'm going to improve the UI to make it less likely people will accidentally over-write an existing service instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/608#issuecomment-551353133
issue_url: https://api.github.com/repos/simonw/datasette/issues/608
node_id: MDEyOklzc3VlQ29tbWVudDU1MTM1MzEzMw==
created_at: 2019-11-08T02:13:10Z
updated_at: 2019-11-08T02:13:10Z
author_association: OWNER
body:
  Right now you're not asked to select a service until AFTER the image has been compiled and pushed:
  ```
  7f41a4f30654: Pushed
  60c95c8aac78: Pushed
  5bad104547b2: Pushed
  latest: digest: sha256:7737f14d8490bbea49f29a750401fe2bf8a69f3da3bc2fd1ec5d1a031abf1b06 size: 2849
  DONE
  ------------------------------------------------------------------------------------------------------------------------------------------------------------------

  ID                                    CREATE_TIME                DURATION  SOURCE                                                                                      IMAGES                                       STATUS
  55b495e1-6862-464b-a99c-ee6eb1d8ac31  2019-11-08T02:08:14+00:00  35S       gs://datasette-222320_cloudbuild/source/1573178893.56-8a2500be28d540e1baac7c6726c5c975.tgz  gcr.io/datasette-222320/datasette (+1 more)  SUCCESS
  ...

  Service name (datasette): 
  ```
  This defaults to `datasette` which means you'll over-ride your previous deployment if you don't fully understand what is going on.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/608#issuecomment-551354256
issue_url: https://api.github.com/repos/simonw/datasette/issues/608
node_id: MDEyOklzc3VlQ29tbWVudDU1MTM1NDI1Ng==
created_at: 2019-11-08T02:17:50Z
updated_at: 2019-11-08T02:17:50Z
author_association: OWNER
body:
  If the user specifies `--service` I will use that without further questions.

  If they don't I'm going to ask them to enter a service manually (like happens at the moment) BUT I'm not going to give them a default and I'm going to show them the list of currently deployed services in the UI.

  I can get that by shelling out to `gcloud beta run services list --platform=managed --format json`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/608#issuecomment-551361489
issue_url: https://api.github.com/repos/simonw/datasette/issues/608
node_id: MDEyOklzc3VlQ29tbWVudDU1MTM2MTQ4OQ==
created_at: 2019-11-08T02:50:03Z
updated_at: 2019-11-08T02:50:03Z
author_association: OWNER
body:
  Here's the new interface:
  ```
  $ datasette publish cloudrun fixtures.db
  Please provide a service name for this deployment

  Using an existing service name will over-write it

  Your existing services:

    csvconf - created 2019-05-09T18:52:17.829Z - https://csvconf-j7hipcg4aq-uc.a.run.app
    fixtures - created 2019-11-08T02:22:14.048160Z - https://fixtures-j7hipcg4aq-uc.a.run.app

  Service name: 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/608#issuecomment-551361792
issue_url: https://api.github.com/repos/simonw/datasette/issues/608
node_id: MDEyOklzc3VlQ29tbWVudDU1MTM2MTc5Mg==
created_at: 2019-11-08T02:51:36Z
updated_at: 2019-11-08T02:51:36Z
author_association: OWNER
body:
  A nice undocumented bonus feature of this change is that you can see a full list of your Cloud Run services by running `datasette publish cloudrun fixtures.db` and then hitting `Ctrl+C` rather than continuing with the deploy.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/616#issuecomment-551872999
issue_url: https://api.github.com/repos/simonw/datasette/issues/616
node_id: MDEyOklzc3VlQ29tbWVudDU1MTg3Mjk5OQ==
created_at: 2019-11-08T15:31:33Z
updated_at: 2019-11-08T15:31:33Z
author_association: NONE
body:
  Thank you so much, Simon!

  Now, I'm contacting Heroku's support team to find a way to update the Datasette version on bases.vortex.media.

  Do you know how to do it?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129686
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29
node_id: MDEyOklzc3VlQ29tbWVudDU1MjEyOTY4Ng==
created_at: 2019-11-09T19:27:39Z
updated_at: 2019-11-09T19:27:39Z
author_association: MEMBER
body:
  I think this is fixed by the latest version of `sqlite-utils` - https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-12-1 - I'll bump the dependency.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129921
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29
node_id: MDEyOklzc3VlQ29tbWVudDU1MjEyOTkyMQ==
created_at: 2019-11-09T19:30:42Z
updated_at: 2019-11-09T19:30:42Z
author_association: MEMBER
body:
  Confirmed, that seems to fix it:
  ```
  (twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js 
  Traceback (most recent call last):
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite", line 11, in <module>
      load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')()
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py", line 633, in import_
      archive.import_from_file(db, path.name, open(path, "rb").read())
    File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/archive.py", line 224, in import_from_file
      db[table_name].upsert_all(rows, hash_id="pk")
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py", line 1094, in upsert_all
      extracts=extracts,
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py", line 961, in insert_all
      first_record = next(records)
  StopIteration
  (twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ pip install -U sqlite-utils
  Collecting sqlite-utils
    Using cached https://files.pythonhosted.org/packages/ee/a2/1b135010c7ac8e2d7545f659e9e6c6ede0f406f20b52e08d5817e1e31a9a/sqlite_utils-1.12.1-py3-none-any.whl
  Requirement already satisfied, skipping upgrade: click in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (7.0)
  Requirement already satisfied, skipping upgrade: tabulate in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (0.8.5)
  Requirement already satisfied, skipping upgrade: click-default-group in /Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages (from sqlite-utils) (1.2.2)
  Installing collected packages: sqlite-utils
    Found existing installation: sqlite-utils 1.11
      Uninstalling sqlite-utils-1.11:
        Successfully uninstalled sqlite-utils-1.11
  Successfully installed sqlite-utils-1.12.1
  (twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ twitter-to-sqlite import blah.db ~/Dropbox/dogsheep/twitter-2019-06-25-b31f246100821b551f2f9a23f21ac6fb565dab49dd23a35630cabbf2b94a1f03/account-suspension.js 
  (twitter-to-sqlite) ~/Dropbox/Development/twitter-to-sqlite $ 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/618#issuecomment-552139232
issue_url: https://api.github.com/repos/simonw/datasette/issues/618
node_id: MDEyOklzc3VlQ29tbWVudDU1MjEzOTIzMg==
created_at: 2019-11-09T21:26:11Z
updated_at: 2019-11-09T21:26:11Z
author_association: OWNER
body:
  Easiest solution here would be to append these on to the `create table` statement that is displayed on the table page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-552140870
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDU1MjE0MDg3MA==
created_at: 2019-11-09T21:49:51Z
updated_at: 2019-11-09T21:49:51Z
author_association: OWNER
body:
  Better idea: if you run Datasette in cross-database joining mode, all connections start out as memory connections and then have new databases attached to them on-demand.

  All table view queries will be automatically rewritten to start `SELECT db.table.one, db.table.two FROM db.table ...`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-552140975
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDU1MjE0MDk3NQ==
created_at: 2019-11-09T21:51:41Z
updated_at: 2019-11-09T21:51:41Z
author_association: OWNER
body:
  It may turn out that we have to recommend NOT exposing a Datasette instance to the public with dozens of database files that has multi-db queries enabled - will need to load test to understand if this recommendation is needed or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/617#issuecomment-552141417
issue_url: https://api.github.com/repos/simonw/datasette/issues/617
node_id: MDEyOklzc3VlQ29tbWVudDU1MjE0MTQxNw==
created_at: 2019-11-09T21:58:40Z
updated_at: 2019-11-09T21:58:40Z
author_association: OWNER
body:
  The function that builds the query could go in a new `datasette.utils.sql` module.

  I can design it to only take simple arguments (the table name, list of columns, list of primary keys and a list of key/value tuples from the query string). This will make it really easy to test (and means it won't need to be async since it won't have to use queries to retrieve those values).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/618#issuecomment-552154007
issue_url: https://api.github.com/repos/simonw/datasette/issues/618
node_id: MDEyOklzc3VlQ29tbWVudDU1MjE1NDAwNw==
created_at: 2019-11-10T01:39:35Z
updated_at: 2019-11-10T01:40:05Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/compound_three_primary_keys#export

  <img width="942" alt="fixtures__compound_three_primary_keys__1_001_rows" src="https://user-images.githubusercontent.com/9599/68537411-e2a7cb80-0317-11ea-9f89-fd072f6a76bf.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/620#issuecomment-552241725
issue_url: https://api.github.com/repos/simonw/datasette/issues/620
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI0MTcyNQ==
created_at: 2019-11-10T22:27:32Z
updated_at: 2019-11-10T22:27:32Z
author_association: OWNER
body:
  Added bonus: this would mean faceting results could be enhanced with a "more" link that points to a custom paginated SQL query - but that query could maintain the ability to expand the labels on foreign keys.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/621#issuecomment-552251645
issue_url: https://api.github.com/repos/simonw/datasette/issues/621
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI1MTY0NQ==
created_at: 2019-11-11T00:23:56Z
updated_at: 2019-11-11T00:33:31Z
author_association: OWNER
body:
  Current syntax: https://datasette.readthedocs.io/en/stable/json_api.html#special-table-arguments

  `/?_through={"table":"roadside_attraction_characteristics","column":"characteristic_id","value":"1"}`

  Demo: https://latest.datasette.io/fixtures/roadside_attractions?_through={%22table%22:%22roadside_attraction_characteristics%22,%22column%22:%22characteristic_id%22,%22value%22:%221%22}

  The alternative syntax for that could be: 

  `https://latest.datasette.io/fixtures/roadside_attractions?_through.roadside_attraction_characteristics.characteristic_id=1`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/621#issuecomment-552251666
issue_url: https://api.github.com/repos/simonw/datasette/issues/621
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI1MTY2Ng==
created_at: 2019-11-11T00:24:09Z
updated_at: 2019-11-11T00:32:49Z
author_association: OWNER
body:
  Here's the current code:

  https://github.com/simonw/datasette/blob/28c4a6db5b5e512db630d7ba6127196185de67c7/datasette/views/table.py#L316-L348

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/620#issuecomment-552251831
issue_url: https://api.github.com/repos/simonw/datasette/issues/620
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI1MTgzMQ==
created_at: 2019-11-11T00:25:58Z
updated_at: 2019-11-11T00:25:58Z
author_association: OWNER
body:
  There are three pieces of information that need to be described here: the column, the other table and the other table column.

  We already have a piece of API design that is similar to this: the `_through=` parameter, which looks like this:

  `?_through={"table":"m2m_characteristics","column":"characteristic_id","value":"1"}`

  I'm rethinking this syntax in #621  though to support a non-JSON variant that looks more like this:

  `?_through.roadside_attraction_characteristics.characteristic_id=1`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/620#issuecomment-552252074
issue_url: https://api.github.com/repos/simonw/datasette/issues/620
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI1MjA3NA==
created_at: 2019-11-11T00:28:28Z
updated_at: 2019-11-11T00:30:53Z
author_association: OWNER
body:
  So for foreign key definitions it could look like this:

  `/db/table?_fk.article_id=articles.id`

  Or for columns and table names that themselves contain dots it could be:

  `/db/table?_fk.article_id={"table":"articles","column":"id"}`

  The value (before the =) is unambiguous -it's `?fk.XXX` where XXX could be a column name that includes periods without breaking anything.

  Added bonus: if you're referencing another table's single primary key you can omit the `.id` entirely (since it can be automatically detected) - so you could do `?_fk.article_id=articles`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/620#issuecomment-552252199
issue_url: https://api.github.com/repos/simonw/datasette/issues/620
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI1MjE5OQ==
created_at: 2019-11-11T00:29:36Z
updated_at: 2019-11-11T00:29:36Z
author_association: OWNER
body:
  This new `?_fk.column_name=` syntax makes me wonder if the various filters should be `?colname.contains=x` rather than `?colname__contains=x` - but that's a conversation for another time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/621#issuecomment-552253208
issue_url: https://api.github.com/repos/simonw/datasette/issues/621
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI1MzIwOA==
created_at: 2019-11-11T00:39:57Z
updated_at: 2019-11-11T00:39:57Z
author_association: OWNER
body:
  This is a nice opportunity to help with the refactoring of the table view in #617 - the above code snippet takes the request, looks at the `_through=` parameters and appends things to the `where_clauses`, `params` and `extra_human_descriptions` lists. I can factor that out into a separate private method on `TableView` and use that as a pattern for further refactoring.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/617#issuecomment-552253893
issue_url: https://api.github.com/repos/simonw/datasette/issues/617
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI1Mzg5Mw==
created_at: 2019-11-11T00:46:42Z
updated_at: 2021-12-18T01:41:47Z
author_association: OWNER
body:
  As noted in https://github.com/simonw/datasette/issues/621#issuecomment-552253208 a common pattern in this method is blocks of code that append new items to the `where_clauses`, `params` and `extra_human_descriptions` arrays. This is a useful refactoring opportunity.

  Code that fits this pattern:

  * The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)`
  * Code that handles `?_where=`: `where_clauses.extend(request.args["_where"])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses
  * The `_through=` code, see #621 for details
  * The code that deals with `?_search=` FTS

  The keyset pagination code modifies `where_clauses` and `params` too, but I don't think it's quite going to work with the same abstraction that would cover the above examples.

  [UPDATE December 2021 - this comment became the basis for a new `filters_from_request` plugin hook, see also #473]

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/617#issuecomment-552254238
issue_url: https://api.github.com/repos/simonw/datasette/issues/617
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI1NDIzOA==
created_at: 2019-11-11T00:49:57Z
updated_at: 2019-11-11T00:51:35Z
author_association: OWNER
body:
  I experimented with a `table_filter()` plugin hook a while ago which looks very much like the abstraction I'm now talking about here: https://github.com/simonw/datasette/commit/5116c4ec8aed5091e1f75415424b80f613518dc6 - more details here: https://simonwillison.net/2018/Aug/6/russian-facebook-ads/#Weird_implementation_details_106

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/617#issuecomment-552254753
issue_url: https://api.github.com/repos/simonw/datasette/issues/617
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI1NDc1Mw==
created_at: 2019-11-11T00:54:51Z
updated_at: 2019-11-11T00:54:51Z
author_association: OWNER
body:
  That `table_filter()` plugin hook should probably be renamed though, since it could now apply to the regular custom SQL view as well as the table view.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/595#issuecomment-552275668
issue_url: https://api.github.com/repos/simonw/datasette/issues/595
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI3NTY2OA==
created_at: 2019-11-11T03:09:43Z
updated_at: 2019-11-11T03:09:43Z
author_association: OWNER
body:
  Glitch has been upgraded to Python 3.7. I think I'm happy to drop 3.5 support now - users who want Python 3.5 can get it by installing `datasette==0.30.2`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/623#issuecomment-552279522
issue_url: https://api.github.com/repos/simonw/datasette/issues/623
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI3OTUyMg==
created_at: 2019-11-11T03:31:17Z
updated_at: 2019-11-11T03:31:17Z
author_association: OWNER
body:
  `================= 379 passed, 4332 warnings in 195.71 seconds ==================`

  I think all of those warnings are the same:
  ```
    /home/travis/build/simonw/datasette/datasette/tracer.py:17: DeprecationWarning: Task.current_task() is deprecated, use asyncio.current_task() instead
      return id(asyncio.Task.current_task(loop=loop))
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/623#issuecomment-552281217
issue_url: https://api.github.com/repos/simonw/datasette/issues/623
node_id: MDEyOklzc3VlQ29tbWVudDU1MjI4MTIxNw==
created_at: 2019-11-11T03:40:59Z
updated_at: 2019-11-11T03:40:59Z
author_association: OWNER
body:
  That did the trick - still some warnings now but they are all in dependencies:
  ```
  =============================== warnings summary ===============================

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/pint/compat/chainmap.py:16

    /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/pint/compat/chainmap.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working

      from collections import MutableMapping

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/pint/util.py:21

    /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/pint/util.py:21: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working

      from collections import Mapping

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/base.py:17

    /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/base.py:17: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead

      def __anext__(self):

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/base.py:63

    /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/base.py:63: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead

      def __iter__(self):

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/base.py:72

    /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/base.py:72: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead

      def __anext__(self):

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/base.py:77

    /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/base.py:77: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead

      def __aenter__(self):

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/base.py:82

    /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/base.py:82: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead

      def __aexit__(self, exc_type, exc, tb):

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/base.py:91

    /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/base.py:91: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead

      def __aexit__(self, exc_type, exc_val, exc_tb):

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

    /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead

      def method(self, *args, **kwargs):

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/__init__.py:27

    /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/__init__.py:27: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead

      def _open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None,

  -- Docs: https://docs.pytest.org/en/latest/warnings.html

  ================== 379 passed, 54 warnings in 201.62 seconds ===================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/626#issuecomment-552729658
issue_url: https://api.github.com/repos/simonw/datasette/issues/626
node_id: MDEyOklzc3VlQ29tbWVudDU1MjcyOTY1OA==
created_at: 2019-11-12T04:31:13Z
updated_at: 2019-11-12T04:31:13Z
author_association: OWNER
body:
  I'm giving up on this. I spent a bunch of time trying to get it to work and couldn't.

  My final attempt was inspired by https://github.com/encode/uvicorn/blob/e821fd1ff5653f989d500d41fa6ba070858f6843/tests/test_main.py#L10-L32

  Here's the code that didn't work:
  ```
  def test_asgi_app(app_client):
      from uvicorn.config import Config
      from uvicorn.main import Server

      class CustomServer(Server):
          def install_signal_handlers(self):
              pass

      def run_server():
          asyncio.set_event_loop(asyncio.new_event_loop())
          print("ooh new loop")
          config = Config(app=app_client.asgi_app, limit_max_requests=1, port=8642)
          config.load()
          server = CustomServer(config=config)
          server.run()

      with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
          future = executor.submit(run_server)
          future.result()
          while not server.started:
              print(server.__dict__)
              time.sleep(0.01)
          assert 200 == urllib.request.urlopen("http://localhost:8642/").status
          # This line will raise if there's an exception:
          future.result()
  ```
  I got `RuntimeError: There is no current event loop in thread 'ThreadPoolExecutor-1_0'`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/622#issuecomment-552729978
issue_url: https://api.github.com/repos/simonw/datasette/issues/622
node_id: MDEyOklzc3VlQ29tbWVudDU1MjcyOTk3OA==
created_at: 2019-11-12T04:33:00Z
updated_at: 2019-11-12T04:33:00Z
author_association: OWNER
body:
  Here's the error I'm getting with Python 3.8 at the moment:
  ```
  ~/Dropbox/Development/datasette $ venv-py3.8.0/bin/datasette --memory -p 8055
  Serve! files=() (immutables=()) on port 8055
  Traceback (most recent call last):
    File "venv-py3.8.0/bin/datasette", line 11, in <module>
      load_entry_point('datasette', 'console_scripts', 'datasette')()
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/Users/simonw/Dropbox/Development/datasette/datasette/cli.py", line 365, in serve
      uvicorn.run(ds.app(), host=host, port=port, log_level="info")
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/uvicorn/main.py", line 279, in run
      server.run()
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/uvicorn/main.py", line 305, in run
      self.config.setup_event_loop()
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/uvicorn/config.py", line 218, in setup_event_loop
      loop_setup()
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/uvicorn/loops/auto.py", line 3, in auto_loop_setup
      import uvloop
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/uvloop/__init__.py", line 7, in <module>
      from .loop import Loop as __BaseLoop  # NOQA
    File "uvloop/includes/stdlib.pxi", line 114, in init uvloop.loop
  AttributeError: module 'sys' has no attribute 'set_coroutine_wrapper'
  ```
  I'm pretty confident upgrading uvicorn (and uvloop with it) will fix this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/629#issuecomment-552739324
issue_url: https://api.github.com/repos/simonw/datasette/issues/629
node_id: MDEyOklzc3VlQ29tbWVudDU1MjczOTMyNA==
created_at: 2019-11-12T05:23:22Z
updated_at: 2019-11-12T05:23:22Z
author_association: OWNER
body:
  This change needs to wait until `datasette 0.31` has been published to PyPI, because the Docker containers produced by `datasette publish` run `pip install datasette` and they need to be able to retrieve a version from PyPI that works with 3.8.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553170650
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDU1MzE3MDY1MA==
created_at: 2019-11-12T23:49:29Z
updated_at: 2019-11-12T23:49:29Z
author_association: OWNER
body:
  This relates to this bug: https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549233778

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553171011
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDU1MzE3MTAxMQ==
created_at: 2019-11-12T23:50:52Z
updated_at: 2019-11-12T23:50:52Z
author_association: OWNER
body:
  Fixing this is going to be a real pain. There's lots of code out there that uses `sqlite-utils` with the expectation that `upsert()` behaves as it currently does.

  Maybe I need to introduce new terms for both of these different patterns and deprecate the existing `.upsert()` and `.upsert_all()` since their behaviour can't be changed?

  Or maybe I fix this and ship `sqlite-utils 2.0` with a breaking change?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553171414
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDU1MzE3MTQxNA==
created_at: 2019-11-12T23:52:35Z
updated_at: 2019-11-12T23:52:35Z
author_association: OWNER
body:
  If I do implement the correct definition of `.upsert()` I think I'll use this pattern, since it works in versions of SQLite prior to 3.24:
  ```sql
  INSERT OR IGNORE INTO book(id) VALUES(1001);
  UPDATE book SET name = 'Programming' WHERE id = 1001;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/632#issuecomment-553264586
issue_url: https://api.github.com/repos/simonw/datasette/issues/632
node_id: MDEyOklzc3VlQ29tbWVudDU1MzI2NDU4Ng==
created_at: 2019-11-13T06:47:53Z
updated_at: 2019-11-13T06:47:53Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/973f8f139df6ad425354711052cfc2256de2e522/datasette/publish/heroku.py#L167

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/633#issuecomment-553483419
issue_url: https://api.github.com/repos/simonw/datasette/issues/633
node_id: MDEyOklzc3VlQ29tbWVudDU1MzQ4MzQxOQ==
created_at: 2019-11-13T16:35:34Z
updated_at: 2019-11-13T16:35:34Z
author_association: OWNER
body:
  The problem is this code in uvicorn: https://github.com/encode/uvicorn/blob/020dd085a0a5b609f14ddd2b36fc0a97ba4df019/uvicorn/main.py#L297-L302
  ```python
      if (config.reload or config.workers > 1) and not isinstance(app, str):
          logger = logging.getLogger("uvicorn.error")
          logger.warn(
              "You must pass the application as an import string to enable 'reload' or 'workers'."
          )
          sys.exit(1)
  ```
  From this commit 4 days ago: https://github.com/encode/uvicorn/commit/b034f7a136c5e57a33ffd10ce2404cc1faee9033 - see Uvicorn https://github.com/encode/uvicorn/issues/478 and https://github.com/encode/uvicorn/pull/481

  This is breaking because `--workers` "Defaults to the $WEB_CONCURRENCY environment variable  if available" - and it turns out on Heroku WEB_CONCURRENCY is set to 2 by default!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/633#issuecomment-553483633
issue_url: https://api.github.com/repos/simonw/datasette/issues/633
node_id: MDEyOklzc3VlQ29tbWVudDU1MzQ4MzYzMw==
created_at: 2019-11-13T16:36:02Z
updated_at: 2019-11-13T16:36:35Z
author_association: OWNER
body:
  Short term fix: set `--workers=1` when publishing with Heroku. Or even easier - set WEB_CONCURRENCY to 1.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/632#issuecomment-553487179
issue_url: https://api.github.com/repos/simonw/datasette/issues/632
node_id: MDEyOklzc3VlQ29tbWVudDU1MzQ4NzE3OQ==
created_at: 2019-11-13T16:44:07Z
updated_at: 2019-11-13T16:44:07Z
author_association: OWNER
body:
  Here's an instance deployed using the new code which shows Python 3.8.0: https://datasette-fixtures.herokuapp.com/-/versions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553526685
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDU1MzUyNjY4NQ==
created_at: 2019-11-13T17:58:59Z
updated_at: 2019-11-13T17:58:59Z
author_association: OWNER
body:
  This warrants making a backwards compatible change, which means I'll need to bump the major version number and release 2.0.

  I'm going to rename the existing `upsert()` and `upsert_all()` methods to `replace()` and `replace_all()` - then write new `upsert()` and `upsert_all()` methods that implement the correct behavior.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553527384
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDU1MzUyNzM4NA==
created_at: 2019-11-13T18:00:41Z
updated_at: 2019-11-13T18:00:41Z
author_association: OWNER
body:
  Is `replace()` a good name here? It doesn't really convey the idea that a brand new record will be created if there isn't an existing one to replace.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553528386
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDU1MzUyODM4Ng==
created_at: 2019-11-13T18:03:10Z
updated_at: 2019-11-13T18:03:54Z
author_association: OWNER
body:
  Maybe `inplace()` (combining "insert" and "replace")?

  It could be an alias for `.insert(..., replace=True)`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553540146
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDU1MzU0MDE0Ng==
created_at: 2019-11-13T18:33:30Z
updated_at: 2019-11-13T18:33:30Z
author_association: OWNER
body:
  Maybe instead of inventing a new term I should tell people to use `.insert(..., replace=True)` directly. That matches `ignore=True`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553574011
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDU1MzU3NDAxMQ==
created_at: 2019-11-13T19:53:45Z
updated_at: 2019-11-13T19:53:45Z
author_association: OWNER
body:
  First step: add a `replace=True` argument to `insert()` and `insert_all()` that does the same thing as the current `upsert=True`

  https://github.com/simonw/sqlite-utils/blob/8dab9fd1ccf571e188eec9ccf606a0c50fccf200/sqlite_utils/db.py#L938-L946

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/634#issuecomment-553662218
issue_url: https://api.github.com/repos/simonw/datasette/issues/634
node_id: MDEyOklzc3VlQ29tbWVudDU1MzY2MjIxOA==
created_at: 2019-11-14T00:05:22Z
updated_at: 2019-11-14T00:05:22Z
author_association: OWNER
body:
  Here's why: the `pytest` invocation is part of the `script` block shared by all of the stages:

  https://github.com/simonw/datasette/blob/b51f258d00bb3c3b401f15d46a1fbd50394dbe1c/.travis.yml#L11-L14

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/628#issuecomment-553665699
issue_url: https://api.github.com/repos/simonw/datasette/issues/628
node_id: MDEyOklzc3VlQ29tbWVudDU1MzY2NTY5OQ==
created_at: 2019-11-14T00:19:32Z
updated_at: 2019-11-14T00:19:32Z
author_association: OWNER
body:
  This is going to be a lot easier than the first attempt because I already modified the `.render()` method to be async four months ago: https://github.com/simonw/datasette/commit/fcfcae21e67cc15090942b1d2a47b5f016279337#diff-4296b31822a931646fa92cd2d5581521R105

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/580#issuecomment-554119604
issue_url: https://api.github.com/repos/simonw/datasette/issues/580
node_id: MDEyOklzc3VlQ29tbWVudDU1NDExOTYwNA==
created_at: 2019-11-14T22:51:00Z
updated_at: 2019-11-14T22:51:00Z
author_association: OWNER
body:
  One thing that would be useful is a way of constructing a `Datasette()` instance (and accompanying `TestClient`) which is attached to an in-memory database that the test itself can manipulate. This would make it trivial to run tests against hypothetical database tables - those tables could be created using `sqlite-utils` directly against the in-memory database that's being manipulated by the system-under-test.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/628#issuecomment-554132794
issue_url: https://api.github.com/repos/simonw/datasette/issues/628
node_id: MDEyOklzc3VlQ29tbWVudDU1NDEzMjc5NA==
created_at: 2019-11-14T23:27:51Z
updated_at: 2019-11-14T23:28:09Z
author_association: OWNER
body:
  I'm releasing this as Datasette 0.32 - I have a new plugin which takes advantage of it here: https://github.com/simonw/datasette-template-sql

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/580#issuecomment-554446317
issue_url: https://api.github.com/repos/simonw/datasette/issues/580
node_id: MDEyOklzc3VlQ29tbWVudDU1NDQ0NjMxNw==
created_at: 2019-11-15T17:11:02Z
updated_at: 2019-11-15T17:11:02Z
author_association: OWNER
body:
  Here's where I copied the testing utilities into `datasette-template-sql` - very slightly modified from the utilities in `datasette-atom`: https://github.com/simonw/datasette-template-sql/blob/0ef0c7304bb5e658723e229d3ee463b9a76058f9/tests/utils.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/579#issuecomment-554559848
issue_url: https://api.github.com/repos/simonw/datasette/issues/579
node_id: MDEyOklzc3VlQ29tbWVudDU1NDU1OTg0OA==
created_at: 2019-11-15T22:50:58Z
updated_at: 2019-11-15T22:50:58Z
author_association: OWNER
body:
  I'm going to extract out the refactoring of `.execute()` from `Datasette` to `Database` into a separate commit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/66#issuecomment-554565198
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDU1NDU2NTE5OA==
created_at: 2019-11-15T23:12:28Z
updated_at: 2019-11-15T23:12:28Z
author_association: OWNER
body:
  Urgh this is going to be quite a bit of work, especially in the CLI module which shares an implementation for `upsert` and `insert` in a way that looks like it will have to be unwrapped.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/620#issuecomment-554869243
issue_url: https://api.github.com/repos/simonw/datasette/issues/620
node_id: MDEyOklzc3VlQ29tbWVudDU1NDg2OTI0Mw==
created_at: 2019-11-18T06:16:17Z
updated_at: 2019-11-18T06:16:51Z
author_association: OWNER
body:
  I think I should move the `ds.expand_foreign_keys()` method somewhere else:

  https://github.com/simonw/datasette/blob/c2779e5af0d056ef1637f9f0e191dca421259a5e/datasette/app.py#L355-L390

  Moving it to `Database` makes sense to me. The question we then ask it is:

  "Given this column containing these values, give me back a dictionary mapping each column and value to a label - `(column, value) -> label`"

  Passing in the foreign keys we have calculated as an argument makes sense.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/617#issuecomment-555111571
issue_url: https://api.github.com/repos/simonw/datasette/issues/617
node_id: MDEyOklzc3VlQ29tbWVudDU1NTExMTU3MQ==
created_at: 2019-11-18T17:04:56Z
updated_at: 2019-11-18T17:04:56Z
author_association: OWNER
body:
  I made some comments on an annotated version of this method a few days ago: https://gist.github.com/simonw/66aca058195d77bae5f614ef73352eb5

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/546#issuecomment-555263597
issue_url: https://api.github.com/repos/simonw/datasette/issues/546
node_id: MDEyOklzc3VlQ29tbWVudDU1NTI2MzU5Nw==
created_at: 2019-11-18T23:46:01Z
updated_at: 2019-11-18T23:46:01Z
author_association: OWNER
body:
  Also maybe this should be added as a suggested facet if a column clearly contains commas that, when split upon, would result in a array recommendation?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/66#issuecomment-555690319
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDU1NTY5MDMxOQ==
created_at: 2019-11-19T20:10:17Z
updated_at: 2019-11-19T20:10:17Z
author_association: OWNER
body:
  Thinking about this further: I believe every time I've personally used `upsert` in the past (either with the Python library or the CLI tool) I've actually wanted the new behaviour, where "upsert" means "update existing record with these changes, or insert a new record if one does not exist".

  So I'm happy with `upsert` doing that, and `insert --replace` being added as an option that does what `upsert` does ta the moment.

  I'll still ship it as version 2.0 since it's technically a breaking change.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/636#issuecomment-555754912
issue_url: https://api.github.com/repos/simonw/datasette/issues/636
node_id: MDEyOklzc3VlQ29tbWVudDU1NTc1NDkxMg==
created_at: 2019-11-19T22:57:20Z
updated_at: 2019-11-19T22:57:20Z
author_association: OWNER
body:
  Weird, it looks like this was a deliberate decision:

  https://github.com/simonw/datasette/blob/a9909c29ccac771c23c2ef22b89d10697b5256b9/datasette/views/table.py#L590-L592

  From this commit - https://github.com/simonw/datasette/commit/52a5e95d21c9b2b7b50f5bc423e3746ae0538734 - which closed #162

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/636#issuecomment-555755906
issue_url: https://api.github.com/repos/simonw/datasette/issues/636
node_id: MDEyOklzc3VlQ29tbWVudDU1NTc1NTkwNg==
created_at: 2019-11-19T23:00:41Z
updated_at: 2019-11-19T23:00:41Z
author_association: OWNER
body:
  So I need to make sure the `Link` column isn't displayed in the filters, as seen in this screenshot from #162:

  <img width="729" alt="food-trucks__applicant__131_rows" src="https://user-images.githubusercontent.com/9599/33583381-ece18de6-d90e-11e7-8287-d5e4813dcb78.png">

  I'll add a test for that against this table: https://v0-32.datasette.io/fixtures/compound_primary_key

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/637#issuecomment-556295200
issue_url: https://api.github.com/repos/simonw/datasette/issues/637
node_id: MDEyOklzc3VlQ29tbWVudDU1NjI5NTIwMA==
created_at: 2019-11-20T19:38:51Z
updated_at: 2019-11-20T19:38:51Z
author_association: OWNER
body:
  Likewise tables with 0 rows - not clear that something hasn't broken on this page: https://latest.datasette.io/fixtures/123_starts_with_digits

  <img width="903" alt="fixtures__123_starts_with_digits__0_rows" src="https://user-images.githubusercontent.com/9599/69271756-50f75400-0b8a-11ea-8af1-f05bfc97c89d.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-556749086
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDU1Njc0OTA4Ng==
created_at: 2019-11-21T01:15:34Z
updated_at: 2019-11-21T01:21:45Z
author_association: CONTRIBUTOR
body:
  Hey @simonw is the url_prefix config option available in another branch, it looks like you've written some tests for it above?  In 0.32 I get "url_prefix is not a valid option". I think this would be *really helpful*!

  This would be really handy for proxying datasette in another domain's *subdirectory* I believe this will allow folks to run upstream authentication, but the links break if the url_prefix doesn't match. 

  I'd prefer not to host a proxied version of datasette on a subdomain (e.g. datasette.myurl.com b/c then I gotta worry about sharing authorization cookies with the subdomain, which I just assume not do, but...)

  Edit: I see the wip-url-prefix branch, I may try with that https://github.com/simonw/datasette/commit/8da2db4b71096b19e7a9ef1929369b8483d448bf

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/638#issuecomment-557331366
issue_url: https://api.github.com/repos/simonw/datasette/issues/638
node_id: MDEyOklzc3VlQ29tbWVudDU1NzMzMTM2Ng==
created_at: 2019-11-22T00:19:40Z
updated_at: 2019-11-22T00:19:40Z
author_association: OWNER
body:
  We currently use `select distinct wikipedia_url ...` to suggest facets. This query would only return rows which are represented twice or more:
  ```sql
  select
    wikipedia_url, count(*) as n
  from
    museums
  where
    wikipedia_url is not null
  group by wikipedia_url
  having n > 1
  ```
  https://www.niche-museums.com/museums?sql=select%0D%0A++wikipedia_url%2C+count%28*%29+as+n%0D%0Afrom%0D%0A++museums%0D%0Awhere%0D%0A++wikipedia_url+is+not+null%0D%0Agroup+by+wikipedia_url%0D%0Ahaving+n+%3E+1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/638#issuecomment-557333467
issue_url: https://api.github.com/repos/simonw/datasette/issues/638
node_id: MDEyOklzc3VlQ29tbWVudDU1NzMzMzQ2Nw==
created_at: 2019-11-22T00:28:07Z
updated_at: 2019-11-22T00:28:07Z
author_association: OWNER
body:
  It's not as simple as that - `planet_int` should be a suggested facet on https://latest.datasette.io/fixtures/facetable?_facet=planet_int because it returns two filters, even though one of those two is a value of 1. Switching to the new proposed SQL statement misses this.

  <img width="830" alt="fixtures__facetable__15_rows_and_museums__select___from__select_id__name__url__address__description__wikipedia_url__photo_url__photo_alt__latitude__longitude__press__permanently_closed__hours_from_museums_order_by_id___limit_0" src="https://user-images.githubusercontent.com/9599/69387809-dc044700-0c7b-11ea-8d78-5004af2a4434.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/638#issuecomment-557343750
issue_url: https://api.github.com/repos/simonw/datasette/issues/638
node_id: MDEyOklzc3VlQ29tbWVudDU1NzM0Mzc1MA==
created_at: 2019-11-22T01:14:59Z
updated_at: 2019-11-22T01:14:59Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/facetable doesn't suggest `distinct_some_null` as a facet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/637#issuecomment-557770129
issue_url: https://api.github.com/repos/simonw/datasette/issues/637
node_id: MDEyOklzc3VlQ29tbWVudDU1Nzc3MDEyOQ==
created_at: 2019-11-23T05:55:42Z
updated_at: 2019-11-23T05:56:16Z
author_association: OWNER
body:
  Design idea: 

  <img width="751" alt="fixtures__123_starts_with_digits__0_rows" src="https://user-images.githubusercontent.com/9599/69474195-ce7db880-0d72-11ea-9fdb-1b7bfe0be1b1.png">

  HTML:

  ```html
  <p style="border: 2px solid #ccc; padding: 0.5em; background-color: #eee; font-style: italic">
    0 records
  </p>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/637#issuecomment-557771312
issue_url: https://api.github.com/repos/simonw/datasette/issues/637
node_id: MDEyOklzc3VlQ29tbWVudDU1Nzc3MTMxMg==
created_at: 2019-11-23T06:17:23Z
updated_at: 2019-11-23T06:17:23Z
author_association: OWNER
body:
  Demo:
  * https://latest.datasette.io/fixtures/123_starts_with_digits
  * https://latest.datasette.io/fixtures/neighborhood_search?text=foop

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/639#issuecomment-558432868
issue_url: https://api.github.com/repos/simonw/datasette/issues/639
node_id: MDEyOklzc3VlQ29tbWVudDU1ODQzMjg2OA==
created_at: 2019-11-26T02:40:06Z
updated_at: 2019-11-26T02:40:06Z
author_association: OWNER
body:
  Unfortunately I don't think it's possible to do this with Heroku. Heroku treats all deployments as total replacements - that's part of how they achieve zero-downtime deployments, since they run the new deployment at the same time as the old deployment and then switch traffic over at the load balancer.

  I did have one idea that's relevant here: #238 - which would provide a mechanism for `metadata.json` to be hosted on a separate URL (e.g. a gist) and have Datasette periodically fetch a new copy. I closed that in favour of #357 - a plugin hook for loading metadata. That's still something I'm interested in exploring.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/639#issuecomment-558437707
issue_url: https://api.github.com/repos/simonw/datasette/issues/639
node_id: MDEyOklzc3VlQ29tbWVudDU1ODQzNzcwNw==
created_at: 2019-11-26T03:02:53Z
updated_at: 2019-11-26T03:03:29Z
author_association: NONE
body:
  @simonw - Thanks for the reply!

  My reading of the heroku documents is that if one sets things up using git, then one can use "git push" (from a {local, GitHub, GitLab} git repository to Heroku) to "update" a Heroku deployment, but I'm not sure exactly how this works. However, assuming there is some way to use "git push" to update the Heroku deployment, the question becomes how can one do this in conjunction with datasette.

  Again based on my reading the heroku documents, it would seem that the following should work (but it doesn't quite):

  1) Use datasette to create a deployment (named MYAPP)
  2) Put it in maintenance mode
  3) heroku git:clone -a MYAPP
    -- This results in an empty repository (as expected)
  4) In another directory, heroku slugs:download -a MYAPP
  5) Copy the downloaded slug into the repository
  6) Make some change to metadata.json
  6) Commit and push it back
  7) Take the deployment out of maintenance mode
  8) Refresh the deployment

  Using the heroku console, I've verified that the edits appear on heroku, but somehow they are not reflected in the running app.

  I'm hopeful that with some small tweak or perhaps the addition of a bit of voodoo, this strategy will work.  

  I think it will be important to get this working for another reason: getting Heroku, Cloudcube, and datasette to work together, to overcome the slug size limitation so that large SQLite databases can be deployed to Heroku using Datasette.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/357#issuecomment-558446045
issue_url: https://api.github.com/repos/simonw/datasette/issues/357
node_id: MDEyOklzc3VlQ29tbWVudDU1ODQ0NjA0NQ==
created_at: 2019-11-26T03:43:17Z
updated_at: 2019-11-26T03:43:17Z
author_association: OWNER
body:
  I think only one plugin gets to work at a time. The plugin can return a dictionary which is used for live lookups of metadata every time it's accessed - which means the plugin can itself mutate that dictionary.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/357#issuecomment-558459823
issue_url: https://api.github.com/repos/simonw/datasette/issues/357
node_id: MDEyOklzc3VlQ29tbWVudDU1ODQ1OTgyMw==
created_at: 2019-11-26T04:55:44Z
updated_at: 2019-11-26T04:56:24Z
author_association: OWNER
body:
  This needs to play nicely with `asyncio` - which means that the plugin hook needs to be able to interact with the event loop somehow.

  That said... I don't particularly want to change everywhere that accesses metadata into a `await` call. So this is tricky.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/357#issuecomment-558461851
issue_url: https://api.github.com/repos/simonw/datasette/issues/357
node_id: MDEyOklzc3VlQ29tbWVudDU1ODQ2MTg1MQ==
created_at: 2019-11-26T05:05:21Z
updated_at: 2019-11-26T05:05:21Z
author_association: OWNER
body:
  Here's an example plugin I set up using the experimental hook in d11fd2cbaa6b31933b1319f81b5d1520726cb0b6

  ```python
  import json
  from datasette import hookimpl
  import threading
  import requests
  import time


  def change_over_time(m, metadata_value):
      while True:
          print(metadata_value)
          fetched = requests.get(metadata_value).json()
          counter = m["counter"]
          m.clear()
          m["counter"] = counter + 1
          m.update(fetched)
          m["counter"] += 1
          m["title"] = "{} {}".format(m.get("title", ""), m["counter"])
          time.sleep(10)


  @hookimpl(trylast=True)
  def load_metadata(metadata_value):
      m = {
          "counter": 0,
      }
      x = threading.Thread(target=change_over_time, args=(m, metadata_value), daemon=True)
      x.start()
      x.setName("datasette-metadata-counter")
      return m
  ```
  It runs a separate thread that fetches the provided URL every 10 seconds:

  ```
  datasette -m metadata.json --memory -p 8069 -m https://gist.githubusercontent.com/simonw/e8e4fcd7c0a9c951f7dd976921992157/raw/b702d18a6a078a0fb94ef1cee62e11a3396e0336/demo-metadata.json
  ```
  I learned a bunch of things from this prototype.

  First, this is the wrong place to run the code:

  https://github.com/simonw/datasette/blob/d11fd2cbaa6b31933b1319f81b5d1520726cb0b6/datasette/cli.py#L337-L343

  I wanted the plugin hook to be able to receive a `datasette` instance, so implementations could potentially run their own database queries. Calling the hook in the CLI function here happens BEFORE the `Datasette()` instance is created, so that doesn't work.

  I wanted to build a demo of a plugin that would load metadata periodically from an external URL (see #238) - but this threaded implementation is pretty naive. It results in a hit every 10 seconds even if no-one is using Datasette!

  A smarter implementation would be to fetch and cache the results - then only re-fetch them if more than 10 seconds have passed since the last time the metadata was accessed.

  But... doing this neatly requires asyncio - and the plugin isn't running inside an event loop (since `uvicorn.run(ds.app()...)` has not run yet so the event loop hasn't even started).

  I could try and refactor everything so that all calls to read from `metadata` happen via `await`, but this feels like a pretty invasive change. It would be necessary if metadata might be read via a SQL query though.

  Or maybe I could set it up so the plugin can start itself running in the event loop and call back to the `datasette` object to update metadata whenever it feels like it?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/639#issuecomment-558687342
issue_url: https://api.github.com/repos/simonw/datasette/issues/639
node_id: MDEyOklzc3VlQ29tbWVudDU1ODY4NzM0Mg==
created_at: 2019-11-26T15:40:00Z
updated_at: 2019-11-26T15:40:00Z
author_association: CONTRIBUTOR
body:
  A bit of background: the reason `heroku git:clone` brings down an empty directory is because `datasette publish heroku` uses the [builds API](https://devcenter.heroku.com/articles/build-and-release-using-the-api), rather than a `git push`, to release the app. I originally did this because it seemed like a lower bar than having a working `git`, but the downside is, as you found out, that tweaking the created app is hard. 

  So there's one option -- change `datasette publish heroku` to use `git push` instead of `heroku builds:create`.

  @pkoppstein - what you suggested seems like it ought to work (you don't need maintenance mode, though). I'm not sure why it doesn't.

  You could also look into using the [slugs API](https://devcenter.heroku.com/articles/platform-api-deploying-slugs) to download the slug, change `metadata.json`, re-pack and re-upload the slug.

  Ultimately though I think I think @simonw's idea of reading `metadata.json` from an external source might be better (#357). Reading from an alternate URL would be fine, or you could also just stuff the whole `metadata.json` into a Heroku config var, and write a plugin to read it from there. 

  Hope this helps a bit!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/639#issuecomment-558852316
issue_url: https://api.github.com/repos/simonw/datasette/issues/639
node_id: MDEyOklzc3VlQ29tbWVudDU1ODg1MjMxNg==
created_at: 2019-11-26T22:54:23Z
updated_at: 2019-11-26T22:54:23Z
author_association: NONE
body:
  @jacobian - Thanks for your help.  Having to upload an entire slug each time a small change is needed in `metadata.json` seems no better than the current situation so I probably won't go down that rabbit hole just yet.  In any case, the really important goal is moving the SQLite file out of Heroku in a way that the Heroku app can still read it efficiently.  Is this possible?  Is Cloudcube the right place to start?  Is there any alternative?  

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/642#issuecomment-559142893
issue_url: https://api.github.com/repos/simonw/datasette/issues/642
node_id: MDEyOklzc3VlQ29tbWVudDU1OTE0Mjg5Mw==
created_at: 2019-11-27T15:47:36Z
updated_at: 2019-11-27T15:47:42Z
author_association: OWNER
body:
  It can include options for quickly bootstrapping custom template function or SQL function plugins, which are really simple.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/642#issuecomment-559146316
issue_url: https://api.github.com/repos/simonw/datasette/issues/642
node_id: MDEyOklzc3VlQ29tbWVudDU1OTE0NjMxNg==
created_at: 2019-11-27T15:55:51Z
updated_at: 2019-11-27T15:55:51Z
author_association: OWNER
body:
  One thing that put me off cookiecutter in the past is that I didn't think it could conditionally create files.

  I was wrong! You can use post- hooks to delete the files that you didn't need: https://github.com/audreyr/cookiecutter-pypackage/blob/master/hooks/post_gen_project.py

  I could use this mechanism to rename directories too if I needed to.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/642#issuecomment-559207224
issue_url: https://api.github.com/repos/simonw/datasette/issues/642
node_id: MDEyOklzc3VlQ29tbWVudDU1OTIwNzIyNA==
created_at: 2019-11-27T18:40:57Z
updated_at: 2019-11-27T18:41:07Z
author_association: CONTRIBUTOR
body:
  Would cookie cutter approaches also work for creating various flavours of customised templates?

  I need to try to create a couple of sites for myself to get a feel for what sorts of thing are easily doable, and what cribbable cookie cutter items might be. I'm guessing https://simonwillison.net/2019/Nov/25/niche-museums/ is a good place to start from?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/577#issuecomment-559610951
issue_url: https://api.github.com/repos/simonw/datasette/issues/577
node_id: MDEyOklzc3VlQ29tbWVudDU1OTYxMDk1MQ==
created_at: 2019-11-28T22:10:36Z
updated_at: 2019-11-28T22:10:49Z
author_association: OWNER
body:
  Better idea: take advantage of pluggy dependency injection. If a plugin takes a `render` argument we can send it a function that can be used to render a template.

  The need to `await render(...)` might be difficult here though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/573#issuecomment-559632608
issue_url: https://api.github.com/repos/simonw/datasette/issues/573
node_id: MDEyOklzc3VlQ29tbWVudDU1OTYzMjYwOA==
created_at: 2019-11-29T01:43:38Z
updated_at: 2019-11-29T01:43:38Z
author_association: CONTRIBUTOR
body:
  In passing, it looks like a start was made on a datasette Jupyter server extension in https://github.com/lucasdurand/jupyter-datasette although the build fails in MyBinder.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559883311
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDU1OTg4MzMxMQ==
created_at: 2019-11-29T21:30:37Z
updated_at: 2019-11-29T21:30:37Z
author_association: MEMBER
body:
  I should build the command to persist ETags and obey their polling guidelines:

  > Events are optimized for polling with the "ETag" header. If no new events have been triggered, you will see a "304 Not Modified" response, and your current rate limit will be untouched. There is also an "X-Poll-Interval" header that specifies how often (in seconds) you are allowed to poll. In times of high server load, the time may increase. Please obey the header.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/14#issuecomment-559902818
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDU1OTkwMjgxOA==
created_at: 2019-11-30T01:32:38Z
updated_at: 2019-11-30T01:32:38Z
author_association: MEMBER
body:
  Prototype:
  ```
  pip install sqlite-utils paginate-json
  paginate-json "https://api.github.com/users/simonw/events" | sqlite-utils insert /tmp/events.db events - --pk=id
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/639#issuecomment-559916057
issue_url: https://api.github.com/repos/simonw/datasette/issues/639
node_id: MDEyOklzc3VlQ29tbWVudDU1OTkxNjA1Nw==
created_at: 2019-11-30T06:08:50Z
updated_at: 2019-11-30T06:08:50Z
author_association: NONE
body:
  @simonw, @jacobian -  I was able to resolve the metadata.json issue by adding `-m metadata.json` to the Procfile.  Now `git push heroku master` picks up the changes, though I have the impression that heroku is doing more work than necessary (e.g. one of the information messages is: `Installing requirements with pip`).

  I also had to set the environment variable WEB_CONCURRENCY -- I used WEB_CONCURRENCY=1.

  I am still anxious to know whether it's possible for Datasette on Heroku to access the SQLite file at another location.  Cloudcube seems the most promising, and I'm hoping it can be done by tweaking the Procfile suitably, but maybe that's too optimistic?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/645#issuecomment-560036585
issue_url: https://api.github.com/repos/simonw/datasette/issues/645
node_id: MDEyOklzc3VlQ29tbWVudDU2MDAzNjU4NQ==
created_at: 2019-12-01T01:28:35Z
updated_at: 2019-12-01T01:28:35Z
author_association: OWNER
body:
  Plugins are currently expected to return this:
  ```python
  @hookimpl
  def register_output_renderer(datasette):
      return {
          "extension": "test",
          "callback": render_test
      }
  ```
  We can add an optional third argument, `"should_suggest"`, which takes the same arguments as the callback but simply returns `True` or `False` depending on if the plugin can work for the current set of data.

  If that dictionary key is omitted, Datasette will treat this test as returning `True`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/645#issuecomment-560036740
issue_url: https://api.github.com/repos/simonw/datasette/issues/645
node_id: MDEyOklzc3VlQ29tbWVudDU2MDAzNjc0MA==
created_at: 2019-12-01T01:29:58Z
updated_at: 2019-12-01T01:29:58Z
author_association: OWNER
body:
  It should be optionally awaitable - as should the existing `"callback"`. Can use the same pattern as this one: 

  https://github.com/simonw/datasette/blob/8c642f04e0608bf537fdd1f76d64c2367fb04d57/datasette/views/base.py#L124-L135

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/645#issuecomment-560913459
issue_url: https://api.github.com/repos/simonw/datasette/issues/645
node_id: MDEyOklzc3VlQ29tbWVudDU2MDkxMzQ1OQ==
created_at: 2019-12-02T23:38:22Z
updated_at: 2019-12-02T23:38:22Z
author_association: OWNER
body:
  I'm going to add unit tests for the hook, and as part of that I'll fix the weird thing at the moment where the plugins for the unit tests are defined inside a quoted string as opposed to their own separate file.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/646#issuecomment-561133534
issue_url: https://api.github.com/repos/simonw/datasette/issues/646
node_id: MDEyOklzc3VlQ29tbWVudDU2MTEzMzUzNA==
created_at: 2019-12-03T11:50:44Z
updated_at: 2019-12-03T11:50:44Z
author_association: NONE
body:
  Thanks for the reply.

  Will try to implement that on my end, if I have any success I will post here/ make a pull request.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/646#issuecomment-561247711
issue_url: https://api.github.com/repos/simonw/datasette/issues/646
node_id: MDEyOklzc3VlQ29tbWVudDU2MTI0NzcxMQ==
created_at: 2019-12-03T16:31:39Z
updated_at: 2019-12-03T17:31:33Z
author_association: NONE
body:
  > I don't think this is possible at the moment but you're right, it totally should be.

  Just give me a heads-up if you think you can do that quickly. I am trying to implement it with very little knowledge of how datasette works, so it will take loads of time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/580#issuecomment-561481377
issue_url: https://api.github.com/repos/simonw/datasette/issues/580
node_id: MDEyOklzc3VlQ29tbWVudDU2MTQ4MTM3Nw==
created_at: 2019-12-04T05:18:25Z
updated_at: 2019-12-04T05:18:25Z
author_association: OWNER
body:
  `datasette-atom` shipped with a copy of these classes too: https://github.com/simonw/datasette-atom/blob/c2e84207fccff0582d7152f3966dd2952fb0b74f/tests/utils.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-561494291
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDU2MTQ5NDI5MQ==
created_at: 2019-12-04T06:14:16Z
updated_at: 2019-12-04T06:14:16Z
author_association: OWNER
body:
  I've been successfully running the systemd recipe above by Russs on a couple of projects. I shared some notes about what's been working for me here: https://gist.github.com/simonw/63797bb10bb74e615695edd8f850844f

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/640#issuecomment-562462630
issue_url: https://api.github.com/repos/simonw/datasette/issues/640
node_id: MDEyOklzc3VlQ29tbWVudDU2MjQ2MjYzMA==
created_at: 2019-12-06T07:19:34Z
updated_at: 2019-12-06T07:19:34Z
author_association: OWNER
body:
  `heroku apps --json` returns a JSON list of apps.

  `heroku apps --json | sqlite-utils insert /tmp/heroku.db apps -` creates a SQLite database from them, useful for exploring them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/648#issuecomment-562913040
issue_url: https://api.github.com/repos/simonw/datasette/issues/648
node_id: MDEyOklzc3VlQ29tbWVudDU2MjkxMzA0MA==
created_at: 2019-12-08T04:56:43Z
updated_at: 2019-12-08T04:56:43Z
author_association: OWNER
body:
  Idea: do this with a simple template naming convention.

  If you hit `/about` and there is no matching database, check for a template file called `about-page.html`. If it exists, render it. Otherwise return a 404 database not found.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/649#issuecomment-562940252
issue_url: https://api.github.com/repos/simonw/datasette/issues/649
node_id: MDEyOklzc3VlQ29tbWVudDU2Mjk0MDI1Mg==
created_at: 2019-12-08T11:59:52Z
updated_at: 2019-12-08T12:00:12Z
author_association: OWNER
body:
  The easiest solution would be to only show counts on the index pages for immutable (`-i`) databases.

  I don't like this, because the most common uses of Datasette don't in my opinion justify it. Most of the time Datasette will be running against a single, small, mutable database. I'd like to show counts in that case.

  Some options:

  - disable counts on the index page for mutable databases of more than one is attached
  - disable counts on the index page for databases where the file in disk is larger than a specified threshold (maybe 10MB? I'm making up this number)
  - implement an overall timer which cuts off table counting once the sum of time spent on it has gone beyond a second

  Worth prototyping a bit to see what works best.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/648#issuecomment-563000942
issue_url: https://api.github.com/repos/simonw/datasette/issues/648
node_id: MDEyOklzc3VlQ29tbWVudDU2MzAwMDk0Mg==
created_at: 2019-12-08T22:08:14Z
updated_at: 2019-12-08T22:08:14Z
author_association: OWNER
body:
  Alternative idea: a new concept of "pages" which live inside `templates/pages/` and where the file name minus the `.html` extension defines the URL.

  `templates/about/me.html` would be served at `/about/me` - but only if no matching database and table were found.

  This only takes effect on 404 errors from core Datasette.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/648#issuecomment-563001064
issue_url: https://api.github.com/repos/simonw/datasette/issues/648
node_id: MDEyOklzc3VlQ29tbWVudDU2MzAwMTA2NA==
created_at: 2019-12-08T22:09:20Z
updated_at: 2019-12-08T22:09:20Z
author_association: OWNER
body:
  Stretch goal: it would be neat if these pages could return custom HTTP headers (eg content-type) and maybe even status codes (eg for redirects) somehow.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/648#issuecomment-563015290
issue_url: https://api.github.com/repos/simonw/datasette/issues/648
node_id: MDEyOklzc3VlQ29tbWVudDU2MzAxNTI5MA==
created_at: 2019-12-09T00:18:17Z
updated_at: 2019-12-09T00:18:17Z
author_association: OWNER
body:
  The implementation in https://github.com/simonw/datasette/commit/c5e8cd84d3ef55ed86771ac0bde0ca91d6b0e07a acts as a proof of concept. It has a big flaw though: it doesn't reuse the regular render() mechanism, which means it doesn't register custom template tags from plugins.

  This is bad because it means that pages rendered in this way cannot take advantage of things like [datasette-template-sql](https://github.com/simonw/datasette-template-sql).

  This means this issue is likely dependent on #577 - a documented mechanism to allow plugins to render templates.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/650#issuecomment-563016016
issue_url: https://api.github.com/repos/simonw/datasette/issues/650
node_id: MDEyOklzc3VlQ29tbWVudDU2MzAxNjAxNg==
created_at: 2019-12-09T00:24:20Z
updated_at: 2019-12-09T00:24:20Z
author_association: OWNER
body:
  In the future I may write a script which extracts the terms from this document into a separate database table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/577#issuecomment-563449488
issue_url: https://api.github.com/repos/simonw/datasette/issues/577
node_id: MDEyOklzc3VlQ29tbWVudDU2MzQ0OTQ4OA==
created_at: 2019-12-09T21:32:47Z
updated_at: 2019-12-09T21:32:47Z
author_association: OWNER
body:
  I'm going to go with `Datasette(...).render_template(...)` - I need that for #648, and it makes sense to me that the `Datasette` class is the documented interface that plugins interact with.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/577#issuecomment-564214355
issue_url: https://api.github.com/repos/simonw/datasette/issues/577
node_id: MDEyOklzc3VlQ29tbWVudDU2NDIxNDM1NQ==
created_at: 2019-12-10T19:47:17Z
updated_at: 2019-12-10T19:47:17Z
author_association: OWNER
body:
  The reason I've been dragging my heels on adding template rendering to the Datasette class is that it feels messy - should that class be responsible for both data access AND template rendering?

  I think I can come to terms with this thanks to plugins. The Datasette class can represent the family of features that plugins affect - which means that having it expose the template rendering API is reasonable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/644#issuecomment-565755208
issue_url: https://api.github.com/repos/simonw/datasette/issues/644
node_id: MDEyOklzc3VlQ29tbWVudDU2NTc1NTIwOA==
created_at: 2019-12-14T21:33:31Z
updated_at: 2019-12-14T21:33:31Z
author_association: CONTRIBUTOR
body:
  Hi @simonw

  Have you had a chance to look at this at all?

  I'm going to have a chunk of time free next week so if there is additional work needed on this, that would be a particularly convenient time for me to revisit this.

  Cheers

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/494#issuecomment-565832380
issue_url: https://api.github.com/repos/simonw/datasette/issues/494
node_id: MDEyOklzc3VlQ29tbWVudDU2NTgzMjM4MA==
created_at: 2019-12-15T18:08:54Z
updated_at: 2019-12-15T18:08:54Z
author_association: OWNER
body:
  Just saw this bug again. It's annoying.

  Here's the code that causes it:

  https://github.com/simonw/datasette/blob/d6b6c9171f3fd945c4e5e4144923ac831c43c208/datasette/cli.py#L327-L333

  `files` here is the list of mutable databases. `immutable` is the list of immutable ones.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/493#issuecomment-566265079
issue_url: https://api.github.com/repos/simonw/datasette/issues/493
node_id: MDEyOklzc3VlQ29tbWVudDU2NjI2NTA3OQ==
created_at: 2019-12-16T22:04:05Z
updated_at: 2019-12-16T22:04:05Z
author_association: OWNER
body:
  This is particularly relevant as plugins increasingly use `metadata.json` for their plugin configuration: https://datasette.readthedocs.io/en/0.32/plugins.html#plugin-configuration

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-567127981
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDU2NzEyNzk4MQ==
created_at: 2019-12-18T17:18:06Z
updated_at: 2019-12-18T17:18:06Z
author_association: NONE
body:
  Agreed, this would be nice to have. I'm currently working around it in `nginx` with additional location blocks:

  ```

      location /datasette/ {
          proxy_pass         http://127.0.0.1:8001/;
          proxy_redirect     off;
          include proxy_params;
      }

      location /dna-protein-genome/ {
          proxy_pass         http://127.0.0.1:8001/dna-protein-genome/;
          proxy_redirect     off;
          include proxy_params;
      }

      location /rna-protein-genome/ {
          proxy_pass         http://127.0.0.1:8001/rna-protein-genome/;
          proxy_redirect     off;
          include proxy_params;
      }
  ```

  The 2nd and 3rd above are my databases. This works, but I have a small problem with URLs like `/rna-protein-genome?params....` that I could fix with some more nginx munging. I seem to do this sort of thing once every 5 years and then have to look it all up again.

  Thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-567133734
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDU2NzEzMzczNA==
created_at: 2019-12-18T17:33:23Z
updated_at: 2019-12-18T17:33:23Z
author_association: CONTRIBUTOR
body:
  FWIW I did a dumb merge of the branch here: https://github.com/jsfenfen/datasette and it seemed to work in that I could run stuff at a subdirectory, but ended up abandoning it in favor of just posting a subdomain because getting the nginx configs right was making me crazy. I still would prefer posting at a subdirectory but the subdomain seems simpler at the moment. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-567219479
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDU2NzIxOTQ3OQ==
created_at: 2019-12-18T21:24:23Z
updated_at: 2019-12-18T21:24:23Z
author_association: NONE
body:
  @simonw What about allowing a base url. The `<base>....</base>` tag has been around forever. Then just use all relative URLs, which I guess is likely what you already do. See https://www.w3schools.com/TAGs/tag_base.asp

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/596#issuecomment-567225156
issue_url: https://api.github.com/repos/simonw/datasette/issues/596
node_id: MDEyOklzc3VlQ29tbWVudDU2NzIyNTE1Ng==
created_at: 2019-12-18T21:40:35Z
updated_at: 2019-12-18T21:40:35Z
author_association: NONE
body:
  I initially went looking for a way to hide a column completely. Today I found the setting to truncate cells, but it applies to all cells. In my case I have text columns that can have many thousands of characters. I was wondering whether the metadata JSON would be an appropriate place to indicate how columns are displayed (on a col-by-col basis). E.g., I'd like to be able to specify that only 20 chars of a given column be shown, and the font be monospace. But maybe I can do that in some other way - I barely know anything about datasette yet, sorry!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/596#issuecomment-567226048
issue_url: https://api.github.com/repos/simonw/datasette/issues/596
node_id: MDEyOklzc3VlQ29tbWVudDU2NzIyNjA0OA==
created_at: 2019-12-18T21:43:13Z
updated_at: 2019-12-18T21:43:13Z
author_association: NONE
body:
  Meant to add that of course it would be better not to reinvent CSS (one time was already enough). But one option would be to provide a mechanism to specify a CSS class for a column (a cell, a row...) and let the user give a URL path to a CSS file on the command line.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/651#issuecomment-568268746
issue_url: https://api.github.com/repos/simonw/datasette/issues/651
node_id: MDEyOklzc3VlQ29tbWVudDU2ODI2ODc0Ng==
created_at: 2019-12-22T14:37:37Z
updated_at: 2019-12-22T14:37:37Z
author_association: OWNER
body:
  I've not yet been able to figure out what the escaping rule are for FTS5 queries.

  If we figure out how those work maybe we can bundle them as a custom function?

      select ... where docs_fts match fts_escape(:search)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/651#issuecomment-568269476
issue_url: https://api.github.com/repos/simonw/datasette/issues/651
node_id: MDEyOklzc3VlQ29tbWVudDU2ODI2OTQ3Ng==
created_at: 2019-12-22T14:46:37Z
updated_at: 2019-12-22T14:47:03Z
author_association: OWNER
body:
  https://stackoverflow.com/a/43756146 says that an escaping mechanism that works is this one:

      select * from blah where term match '"bacon" "and" "eggs"'

  So split on whitespace and then encapsulate each search term in double quotes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/654#issuecomment-568274520
issue_url: https://api.github.com/repos/simonw/datasette/issues/654
node_id: MDEyOklzc3VlQ29tbWVudDU2ODI3NDUyMA==
created_at: 2019-12-22T15:51:58Z
updated_at: 2019-12-22T15:51:58Z
author_association: OWNER
body:
  Proof of concept:
  ```diff
  diff --git a/datasette/views/base.py b/datasette/views/base.py
  index 5182479..39d1f77 100644
  --- a/datasette/views/base.py
  +++ b/datasette/views/base.py
  @@ -1,6 +1,7 @@
   import asyncio
   import csv
   import itertools
  +import json
   import re
   import time
   import urllib
  @@ -138,28 +139,31 @@ class BaseView(AsgiView):
               )
               extra_template_vars.update(extra_vars)
   
  +        template_context = {
  +            **context,
  +            **{
  +                "app_css_hash": self.ds.app_css_hash(),
  +                "select_templates": select_templates,
  +                "zip": zip,
  +                "body_scripts": body_scripts,
  +                "extra_css_urls": self._asset_urls(
  +                    "extra_css_urls", template, context
  +                ),
  +                "extra_js_urls": self._asset_urls(
  +                    "extra_js_urls", template, context
  +                ),
  +                "format_bytes": format_bytes,
  +                "database_url": self.database_url,
  +                "database_color": self.database_color,
  +            },
  +            **extra_template_vars,
  +        }
  +        if request.args.get("_context"):
  +            return Response.html("<pre>{}</pre>".format(
  +                    escape(json.dumps(template_context, default=repr, indent=4))
  +            ))
           return Response.html(
  -            await template.render_async(
  -                {
  -                    **context,
  -                    **{
  -                        "app_css_hash": self.ds.app_css_hash(),
  -                        "select_templates": select_templates,
  -                        "zip": zip,
  -                        "body_scripts": body_scripts,
  -                        "extra_css_urls": self._asset_urls(
  -                            "extra_css_urls", template, context
  -                        ),
  -                        "extra_js_urls": self._asset_urls(
  -                            "extra_js_urls", template, context
  -                        ),
  -                        "format_bytes": format_bytes,
  -                        "database_url": self.database_url,
  -                        "database_color": self.database_color,
  -                    },
  -                    **extra_template_vars,
  -                }
  -            )
  +            await template.render_async(template_context)
           )
   ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/654#issuecomment-568274570
issue_url: https://api.github.com/repos/simonw/datasette/issues/654
node_id: MDEyOklzc3VlQ29tbWVudDU2ODI3NDU3MA==
created_at: 2019-12-22T15:52:43Z
updated_at: 2019-12-22T15:52:43Z
author_association: OWNER
body:
  One problem with this: what if secrets end up being dumped out in this debug view?

  This won't happen with default Datasette but could potentially happen with a plugin.

  This feature should be opt-in - maybe a `template_debug:1` config setting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/577#issuecomment-568276310
issue_url: https://api.github.com/repos/simonw/datasette/issues/577
node_id: MDEyOklzc3VlQ29tbWVudDU2ODI3NjMxMA==
created_at: 2019-12-22T16:10:31Z
updated_at: 2019-12-22T16:10:31Z
author_association: OWNER
body:
  The code in question currently lives in `BaseView.render()`:

  https://github.com/simonw/datasette/blob/d54318fc7f2565e6121920ce1ea9cb8b700e629a/datasette/views/base.py#L106-L163

  Should `datasette.render_template()` do exactly this, or should it be slightly different?

  Plugins need the option to not pass a `request` object - so maybe that parameter becomes optional.

  Perhaps plugins should be able to render templates without other plugins getting to inject their own variables?

  Does it always make sense to dump in all of those extra template context variables?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/654#issuecomment-568276548
issue_url: https://api.github.com/repos/simonw/datasette/issues/654
node_id: MDEyOklzc3VlQ29tbWVudDU2ODI3NjU0OA==
created_at: 2019-12-22T16:13:11Z
updated_at: 2019-12-22T16:13:11Z
author_association: OWNER
body:
  Documentation: https://datasette.readthedocs.io/en/latest/config.html#template-debug

  Demos:
  * https://latest.datasette.io/?_context=1
  * https://latest.datasette.io/fixtures?_context=1
  * https://latest.datasette.io/fixtures/roadside_attractions?_context=1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-569005894
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDU2OTAwNTg5NA==
created_at: 2019-12-26T08:03:59Z
updated_at: 2019-12-26T08:03:59Z
author_association: OWNER
body:
  To solve https://github.com/simonw/datasette-atom/issues/6 the name of the current canned query should be made available somehow.

  That way the plugin configuration could specify that the title for browse/feed should be X.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/655#issuecomment-569056952
issue_url: https://api.github.com/repos/simonw/datasette/issues/655
node_id: MDEyOklzc3VlQ29tbWVudDU2OTA1Njk1Mg==
created_at: 2019-12-26T13:16:14Z
updated_at: 2019-12-26T13:16:53Z
author_association: OWNER
body:
  I just tried copying and pasting in [Ace](https://ace.c9.io/) - an alternative rich code highlighting JavaScript editor - and it didn't work reliably there either.

  Maybe the fix here is to detect mobile safari and avoid loading codemirror entirely for that browser, falling back on a textarea?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/70#issuecomment-569130037
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/70
node_id: MDEyOklzc3VlQ29tbWVudDU2OTEzMDAzNw==
created_at: 2019-12-26T20:39:04Z
updated_at: 2019-12-26T20:39:04Z
author_association: OWNER
body:
  I hadn't thought about those at all.

  Are you suggesting a utility mechanism in the library for setting it up so that, for a specific foreign key, rows are deleted from other tables if the row they are pointing at is deleted?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/69#issuecomment-569130196
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/69
node_id: MDEyOklzc3VlQ29tbWVudDU2OTEzMDE5Ng==
created_at: 2019-12-26T20:40:21Z
updated_at: 2019-12-26T20:40:21Z
author_association: OWNER
body:
  This is a good idea. Datasette has this in the form of the `--load-extension` CLI argument, e.g. for SpatiaLite here: https://datasette.readthedocs.io/en/stable/spatialite.html#installation

  Having that available for `sqlite-utils` definitely makes sense.

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1}


html_url: https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569226620
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDU2OTIyNjYyMA==
created_at: 2019-12-27T09:05:29Z
updated_at: 2019-12-27T09:05:36Z
author_association: OWNER
body:
  I'm going to start by ignoring the existing `upsert` entirely and implementing `.insert(..., replace=True)` and `$ sqlite-utils insert --replace`. Including updating the tests.

  Then I'll figure out how to implement the new `.upsert()` / `$ sqlite-utils upsert`.

  Then I'll update the documentation, and ship `sqlite-utils` 2.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569233996
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/71
node_id: MDEyOklzc3VlQ29tbWVudDU2OTIzMzk5Ng==
created_at: 2019-12-27T09:45:17Z
updated_at: 2019-12-27T09:45:17Z
author_association: OWNER
body:
  It looks like those backports no longer include sqlite3 - Google Searches still find it but when you click through to launchpad you get 404s: https://launchpad.net/~jonathonf/+archive/ubuntu/codelite/+build/10511920

  Maybe Travis have a newer Ubuntu I can use that ships with FTS5 in its SQLite?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569844426
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDU2OTg0NDQyNg==
created_at: 2019-12-31T01:30:20Z
updated_at: 2019-12-31T01:30:20Z
author_association: OWNER
body:
  I shipped 2.0 - release notes here: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v2

  I also wrote about it on my blog: https://simonwillison.net/2019/Dec/30/sqlite-utils-2/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/73#issuecomment-570930239
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/73
node_id: MDEyOklzc3VlQ29tbWVudDU3MDkzMDIzOQ==
created_at: 2020-01-05T17:15:18Z
updated_at: 2020-01-05T17:15:18Z
author_association: OWNER
body:
  I think this is because you forgot to include a `pk=` argument. I'll change the code to throw a more useful error in this case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/73#issuecomment-571138093
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/73
node_id: MDEyOklzc3VlQ29tbWVudDU3MTEzODA5Mw==
created_at: 2020-01-06T13:28:31Z
updated_at: 2020-01-06T13:28:31Z
author_association: NONE
body:
  I think I actually had several issues in play...

  The missing key was one, but I think there is also an issue as per below.

  For example, in the following:

  ```python
  def init_testdb(dbname='test.db'):
      
      if os.path.exists(dbname):
          os.remove(dbname)

      conn = sqlite3.connect(dbname)
      db = Database(conn)
      
      return conn, db

  conn, db = init_testdb()

  c = conn.cursor()
  c.executescript('CREATE TABLE "test1" ("Col1" TEXT, "Col2" TEXT, PRIMARY KEY ("Col1"));')
  c.executescript('CREATE TABLE "test2" ("Col1" TEXT, "Col2" TEXT, PRIMARY KEY ("Col1"));')

  print('Test 1...')
  for i in range(3):
      db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
      db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))

  print('Test 2...')
  for i in range(3):
      db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
      db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'},
                              {'Col1':'c','Col2':'x'}], pk=('Col1'))
  print('Done...')

  ---------------------------------------------------------------------------
  Test 1...
  Test 2...
   IndexError: list index out of range 
  ---------------------------------------------------------------------------
  IndexError                                Traceback (most recent call last)
  <ipython-input-763-444132ca189f> in <module>
       22 print('Test 2...')
       23 for i in range(3):
  ---> 24     db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
       25     db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'},
       26                             {'Col1':'c','Col2':'x'}], pk=('Col1'))

  /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts)
     1157             alter=alter,
     1158             extracts=extracts,
  -> 1159             upsert=True,
     1160         )
     1161 

  /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert)
     1097                 # self.last_rowid will be 0 if a "INSERT OR IGNORE" happened
     1098                 if (hash_id or pk) and self.last_rowid:
  -> 1099                     row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
     1100                     if hash_id:
     1101                         self.last_pk = row[hash_id]

  IndexError: list index out of range
  ```

  the first test works but the second fails. Is the length of the list of items being upserted leaking somewhere?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/73#issuecomment-572870032
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/73
node_id: MDEyOklzc3VlQ29tbWVudDU3Mjg3MDAzMg==
created_at: 2020-01-10T04:38:41Z
updated_at: 2020-01-10T04:38:41Z
author_association: OWNER
body:
  Odd.. I'm not able to replicate that error. Here's what I got:
  <img width="869" alt="Tony_Bug" src="https://user-images.githubusercontent.com/9599/72126164-025f6e00-3320-11ea-9737-c0679950acbd.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/74#issuecomment-572871797
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/74
node_id: MDEyOklzc3VlQ29tbWVudDU3Mjg3MTc5Nw==
created_at: 2020-01-10T04:47:55Z
updated_at: 2020-01-10T04:47:55Z
author_association: OWNER
body:
  This is odd. I'd love to see more about that result object.

  Could you try running `pytest --pdb` and then `result.exit_code, result.exception` in the PDB prompt, something like this?

  ```
  $ pytest --pdb
  ========================================================= test session starts =========================================================
  platform darwin -- Python 3.7.4, pytest-5.2.2, py-1.8.0, pluggy-0.13.0
  rootdir: /Users/simonw/Dropbox/Development/sqlite-utils
  plugins: cov-2.8.1
  collected 216 items                                                                                                                   

  tests/test_black.py s                                                                                                           [  0%]
  tests/test_cli.py F
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

  db_path = '/private/var/folders/bl/5x847xbj2yb7xmp7f2tz7l280000gn/T/pytest-of-simonw/pytest-3/test_tables0/test.db'

      def test_tables(db_path):
          result = CliRunner().invoke(cli.cli, ["tables1", db_path])
  >       assert '[{"table": "Gosh"},\n {"table": "Gosh2"}]' == result.output.strip()
  E       assert '[{"table": "...e": "Gosh2"}]' == ''
  E         - [{"table": "Gosh"},
  E         -  {"table": "Gosh2"}]

  tests/test_cli.py:28: AssertionError
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  > /Users/simonw/Dropbox/Development/sqlite-utils/tests/test_cli.py(28)test_tables()
  -> assert '[{"table": "Gosh"},\n {"table": "Gosh2"}]' == result.output.strip()
  (Pdb) result.exit_code, result.exception
  (1, OperationalError('near "/": syntax error'))
  ```

  That should show the exception that caused the script to fail to run.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/73#issuecomment-573047321
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/73
node_id: MDEyOklzc3VlQ29tbWVudDU3MzA0NzMyMQ==
created_at: 2020-01-10T14:02:56Z
updated_at: 2020-01-10T14:09:23Z
author_association: NONE
body:
  Hmmm... just tried with installs from pip and the repo  (v2.0.0 and v2.0.1) and I get the error each time (start of second run through the second loop).

  Could it be sqlite3? I'm on 3.30.1.

  UPDATE: just tried it on jupyter.org/try and I get the error there, too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573388052
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/74
node_id: MDEyOklzc3VlQ29tbWVudDU3MzM4ODA1Mg==
created_at: 2020-01-12T06:51:30Z
updated_at: 2020-01-12T06:51:30Z
author_association: CONTRIBUTOR
body:
  Thanks.  That showed me that there was a click cli runner error, and setting `export LANG=en_US.UTF-8` fixed it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573389669
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/74
node_id: MDEyOklzc3VlQ29tbWVudDU3MzM4OTY2OQ==
created_at: 2020-01-12T07:21:17Z
updated_at: 2020-01-12T07:21:17Z
author_association: CONTRIBUTOR
body:
  I guess there is some extra flag for ` CliRunner.invoke` to check exitcode and raise the exception, or that should be an extra assert added.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/657#issuecomment-575321322
issue_url: https://api.github.com/repos/simonw/datasette/issues/657
node_id: MDEyOklzc3VlQ29tbWVudDU3NTMyMTMyMg==
created_at: 2020-01-16T20:01:43Z
updated_at: 2020-01-16T20:01:43Z
author_association: NONE
body:
  I have successfully tested datasette using a parquet VIRTUAL TABLE.  In the first terminal:

  ```datasette airports.db --load-extension=libparquet```

  In another terminal I load the same sqlite db file using the sqlite3 cli client.

  ```$ sqlite3 airports.db```

  and then load the parquet extension and create the virtual table.

  ```
  sqlite> .load /home/darreng/metars/libparquet
  sqlite> CREATE VIRTUAL TABLE mytable USING parquet('/home/xx/data.parquet');
  ```

  Now the parquet virtual table is usable by the datasette web UI.

  Its not an ideal solution but is a proof that datasette works the parquet extension.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/70#issuecomment-575799104
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/70
node_id: MDEyOklzc3VlQ29tbWVudDU3NTc5OTEwNA==
created_at: 2020-01-17T21:20:17Z
updated_at: 2020-01-17T21:20:17Z
author_association: NONE
body:
  Omg sorry I took so long to reply!

  On SQL we can say how the foreign key behaves when it is deleted or updated on the parent table (see https://www.sqlitetutorial.net/sqlite-foreign-key/ for more details).

  I did not see clearly how to create tables with this feature on sqlite-utils library.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/655#issuecomment-575950911
issue_url: https://api.github.com/repos/simonw/datasette/issues/655
node_id: MDEyOklzc3VlQ29tbWVudDU3NTk1MDkxMQ==
created_at: 2020-01-19T00:13:56Z
updated_at: 2020-01-19T00:13:56Z
author_association: OWNER
body:
  https://www.shellcheck.net/ has a workaround for this issue:
  ![9BA173CA-7F1E-447D-A672-1E02E46CA9B7](https://user-images.githubusercontent.com/9599/72672359-81822f80-3a0d-11ea-81a7-a9a0f1b0c34b.jpeg)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/659#issuecomment-575951056
issue_url: https://api.github.com/repos/simonw/datasette/issues/659
node_id: MDEyOklzc3VlQ29tbWVudDU3NTk1MTA1Ng==
created_at: 2020-01-19T00:15:39Z
updated_at: 2020-01-19T00:15:39Z
author_association: OWNER
body:
  Great feedback, thanks. This is mainly caused by the lack of an official Datasette website! I've been working on that - I'll move the news section out of the readme as soon as it's live.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/656#issuecomment-576293773
issue_url: https://api.github.com/repos/simonw/datasette/issues/656
node_id: MDEyOklzc3VlQ29tbWVudDU3NjI5Mzc3Mw==
created_at: 2020-01-20T14:17:11Z
updated_at: 2020-01-20T14:17:11Z
author_association: CONTRIBUTOR
body:
  Seems that headers and definitions has simply to be filled as an HTML table in the description field of matadata.json.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/657#issuecomment-576759416
issue_url: https://api.github.com/repos/simonw/datasette/issues/657
node_id: MDEyOklzc3VlQ29tbWVudDU3Njc1OTQxNg==
created_at: 2020-01-21T16:20:19Z
updated_at: 2020-01-21T16:20:19Z
author_association: NONE
body:
  Hi,

  I've completed some changes to my fork of datasette that allows it to automatically create the parquet virtual table when you supply it with a filename that has the ".parquet" extension.

  I had to figure out how to make the "CREATE VIRTUAL TABLE" statement only be applied to the fake in memory parquet database and not to any others that were also being loaded.  Thus it supports mixed mode databases e.g

  ```
  datasette my_test.parquet normal_sqlite_file.db  --load-extension=libparquet.so --load-extensio
  n=mod_spatialite.so
  ```

  Please see my changes here:  
  https://github.com/dazzag24/datasette/commit/8e18394353114f17291fd1857073b1e0485a1faf

  Thanks

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/651#issuecomment-579675357
issue_url: https://api.github.com/repos/simonw/datasette/issues/651
node_id: MDEyOklzc3VlQ29tbWVudDU3OTY3NTM1Nw==
created_at: 2020-01-29T09:45:00Z
updated_at: 2021-07-14T19:26:06Z
author_association: NONE
body:
  Hi Simon

  Thank you for adding the escape_function, but it does not work on my datasette-installation (0.33). I've added the following file to my datasette-dir: `/plugins/sql_functions.py`:

  ```python
  from datasette import hookimpl

  def escape_fts_query(query):
      bits = query.split()
      return ' '.join('"{}"'.format(bit.replace('"', '')) for bit in bits)

  @hookimpl
  def prepare_connection(conn):
      conn.create_function("escape_fts_query", 1, escape_fts_query)`
  ```
  It has no effect on the standard queries to the tables though, as they still produce errors when including any characters like '-', '/', '+' or '?'

  Does the function only work when using costum queries, where I can include the escape_fts-function explicitly in the sql-query?

  PS. I'm calling datasette with --plugins=plugins, and my other plugins work just fine.
  PPS. The fts5 virtual table is created with 'sqlite3' like so:

  `CREATE VIRTUAL TABLE "cases_fts" USING FTS5(
    title,
    subtitle,
    resume,
    suggestion,
    presentation,
    detail = full,
    content_rowid = 'id',
    content = 'cases',
    tokenize='unicode61', 'remove_diacritics 2', 'tokenchars "-_"'
  );`

  Thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/662#issuecomment-579787057
issue_url: https://api.github.com/repos/simonw/datasette/issues/662
node_id: MDEyOklzc3VlQ29tbWVudDU3OTc4NzA1Nw==
created_at: 2020-01-29T14:43:46Z
updated_at: 2020-01-29T14:43:46Z
author_association: OWNER
body:
  Can you share the exact queries you're having trouble with? The SQL itself or even just the full URL to the page (it doesn't matter if it's to a Datasette instance that isn't available online - I just need to see the URL parameters).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/662#issuecomment-579798917
issue_url: https://api.github.com/repos/simonw/datasette/issues/662
node_id: MDEyOklzc3VlQ29tbWVudDU3OTc5ODkxNw==
created_at: 2020-01-29T15:08:57Z
updated_at: 2020-01-29T15:08:57Z
author_association: NONE
body:
  Hi Simon

  Thankt you for a quick reply. Here are a few examples of urls, where I search the 'cases_fts'-virtual table for tokens in the title-column. It returns the same results, wether the other query-params are present or not.

  Searching for sky
  http://localhost:8001/db-7596a4e/cases?_search_title=sky&year__gte=1997&year__lte=2017&_sort_desc=last_deliberation_date
  Returns searchresults

  Searching for sky*
  http://localhost:8001/db-7596a4e/cases?_search_title=sky*&year__gte=1997&year__lte=2017&_sort_desc=last_deliberation_date
  Returns searchresults

  Searching for sky-tog
  http://localhost:8001/db-7596a4e/cases?_search_title=sky-tog&year__gte=1997&year__lte=2017&_sort_desc=last_deliberation_date
  Throws: No such column: tog

  searching for sky+
  http://localhost:8001/db-7596a4e/cases?_search_title=sky%2B&year__gte=1997&year__lte=2017&_sort_desc=last_deliberation_date
  Throws: Invalid SQL: fts5: syntax error near ""

  Searching for "madpakke" (including double quotes)
  http://localhost:8001/db-7596a4e/cases?_search_title=%22madpakke%22&year__gte=1997&year__lte=2017&_sort_desc=last_deliberation_date
  Returns searchresults even though 'madpakke' only appears in the fulltextindex without quotes

  As I said, my other plugins work just fine, and I just copied your sql_functions.py from the datasette-repo.

  Thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/661#issuecomment-579830682
issue_url: https://api.github.com/repos/simonw/datasette/issues/661
node_id: MDEyOklzc3VlQ29tbWVudDU3OTgzMDY4Mg==
created_at: 2020-01-29T16:07:41Z
updated_at: 2020-01-29T16:07:41Z
author_association: OWNER
body: Having `datasette package` take an optional port argument seems like a good feature to me.
reactions:
  {"total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/662#issuecomment-579832857
issue_url: https://api.github.com/repos/simonw/datasette/issues/662
node_id: MDEyOklzc3VlQ29tbWVudDU3OTgzMjg1Nw==
created_at: 2020-01-29T16:12:08Z
updated_at: 2020-01-29T16:12:08Z
author_association: OWNER
body:
  I think I see what's happening here.

  Adding the new plugin isn't quite enough: the change I made to master also alters the table view code to call the new function:

  https://github.com/simonw/datasette/commit/3c861f363df02a59a67c59036278338e4760d2ed#diff-5e0ffd62fced7d46339b9b2cd167c2f9

  If you add the escape function as a plugin in Datasette 0.33 you will have to use a custom SQL query to run it, like this:

  https://latest.datasette.io/fixtures?sql=select+pk%2C+text1%2C+text2%2C+%5Bname+with+.+and+spaces%5D+from+searchable+where+rowid+in+%28select+rowid+from+searchable_fts+where+searchable_fts+match+escape_fts%28%3Asearch%29%29+order+by+pk+limit+101&search=Dog

  Or you can hold out for Datasette 0.34 which will have this fix and will hopefully ship within the next 24 hours.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/662#issuecomment-579864036
issue_url: https://api.github.com/repos/simonw/datasette/issues/662
node_id: MDEyOklzc3VlQ29tbWVudDU3OTg2NDAzNg==
created_at: 2020-01-29T17:17:01Z
updated_at: 2020-01-29T17:17:01Z
author_association: NONE
body:
  This is excellent news. I'll wait until version 0.34. It would be tiresome to rewrite all standard-queries into custom queries. Thank you!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/661#issuecomment-580028593
issue_url: https://api.github.com/repos/simonw/datasette/issues/661
node_id: MDEyOklzc3VlQ29tbWVudDU4MDAyODU5Mw==
created_at: 2020-01-30T00:30:04Z
updated_at: 2020-01-30T00:30:04Z
author_association: OWNER
body:
  This has now shipped as part of Datasette 0.34: https://datasette.readthedocs.io/en/stable/changelog.html#v0-34

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/658#issuecomment-580029288
issue_url: https://api.github.com/repos/simonw/datasette/issues/658
node_id: MDEyOklzc3VlQ29tbWVudDU4MDAyOTI4OA==
created_at: 2020-01-30T00:32:43Z
updated_at: 2020-01-30T00:32:43Z
author_association: OWNER
body:
  Can you share how your file layout is working?

  You should have something like this:

  `static/app.css` - a CSS file

  Then run Datasette like this:

  `datasette my.db --static-dir=static:static/`

  Then `http://127.0.0.1:8001/static/app.css` should serve your CSS.

  Could you share the command you're using to deploy to Heroku?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/77#issuecomment-580515506
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/77
node_id: MDEyOklzc3VlQ29tbWVudDU4MDUxNTUwNg==
created_at: 2020-01-30T23:48:41Z
updated_at: 2020-01-30T23:48:41Z
author_association: OWNER
body:
  Potential design: a `conversions={}` option. Used like this:
  ```python
  db[table].insert(record, conversions={"geom": "GeomFromText(?, 4326)"})
  ```
  The `conversions=` key would be supported on `.insert()`, `.insert_all()`, `.upsert()` etc. It could also be passed to the `db.table()` constructor function:
  ```python
  table = db.table(
      "features",
      pk="id",
      conversions={
          "geom": "GeomFromText(?, 4326)"
      }
  )
  # Then used like this:
  table.insert(record)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567505
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/80
node_id: MDEyOklzc3VlQ29tbWVudDU4MDU2NzUwNQ==
created_at: 2020-01-31T03:39:19Z
updated_at: 2020-01-31T03:39:19Z
author_association: OWNER
body:
  Still needs documentation and tests. Also I'm not certain that this should be an argument you can pass to the `.table()` constructor, need to think that over.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580569059
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/80
node_id: MDEyOklzc3VlQ29tbWVudDU4MDU2OTA1OQ==
created_at: 2020-01-31T03:48:41Z
updated_at: 2020-01-31T03:48:41Z
author_association: OWNER
body:
  This may not be the right feature after all, see https://github.com/simonw/geojson-to-sqlite/issues/6#issuecomment-580569002

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/73#issuecomment-580745213
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/73
node_id: MDEyOklzc3VlQ29tbWVudDU4MDc0NTIxMw==
created_at: 2020-01-31T14:02:38Z
updated_at: 2020-01-31T14:21:09Z
author_association: NONE
body:
  So the conundrum continues.. The simple test case above now runs, but if I upsert a large number of new records (successfully) and then try to upsert a fewer number of new records to a different table, I get the same error.

  If I run the same upserts again (which in the first case means there are no new records to add, because they were already added), the second upsert works correctly.

  It feels as if the number of items added via an upsert >> the number of items I try to add in an upsert immediately after, I get the error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071010
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/81
node_id: MDEyOklzc3VlQ29tbWVudDU4MTA3MTAxMA==
created_at: 2020-02-01T21:27:00Z
updated_at: 2020-02-01T21:27:00Z
author_association: OWNER
body:
  Here's the current method:

  https://github.com/simonw/sqlite-utils/blob/f7289174e66ae4d91d57de94bbd9d09fabf7aff4/sqlite_utils/db.py#L823-L845

  If I make it a utility function instead of a class method I could ensure it is directly importable like so:

  ```python
  from sqlite_utils import detect_column_types
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071116
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/81
node_id: MDEyOklzc3VlQ29tbWVudDU4MTA3MTExNg==
created_at: 2020-02-01T21:28:35Z
updated_at: 2020-02-01T21:28:53Z
author_association: OWNER
body:
  Should I keep `table.detect_column_types()` working so as not to break existing code?

  If it was part of the documented API then I wouldn't break that without bumping to 3.x. Since it's undocumented I'm going to make it as a breaking change instead (and bump the `geojson-to-sqlite` dependency version).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071434
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/81
node_id: MDEyOklzc3VlQ29tbWVudDU4MTA3MTQzNA==
created_at: 2020-02-01T21:32:34Z
updated_at: 2020-02-01T21:32:34Z
author_association: OWNER
body:
  While I'm at it I think I'll rename it to `suggest_column_types` - it's not really detecting them since the input is just a list of dictionaries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/82#issuecomment-581652388
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/82
node_id: MDEyOklzc3VlQ29tbWVudDU4MTY1MjM4OA==
created_at: 2020-02-03T22:35:44Z
updated_at: 2020-02-03T22:35:44Z
author_association: OWNER
body:
  I can't replicate this problem:
  ```
  /tmp $ sqlite-utils --version
  sqlite-utils, version 2.2
  /tmp $ curl "https://data.nasa.gov/resource/y77d-th95.json" |     sqlite-utils insert meteorites.db meteorites - --pk=id
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100  240k    0  240k    0     0   185k      0 --:--:--  0:00:01 --:--:--  185k
  ```
  Could you run `sqlite-utils --version` and tell me what you get?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/577#issuecomment-581758728
issue_url: https://api.github.com/repos/simonw/datasette/issues/577
node_id: MDEyOklzc3VlQ29tbWVudDU4MTc1ODcyOA==
created_at: 2020-02-04T06:11:53Z
updated_at: 2020-02-04T06:11:53Z
author_association: OWNER
body:
  For the moment I'm going to move it to `async def render_template()` on `datasette` but otherwise keep the implementation the same.

  The new signature will be:

      async def render_template(self, template, context=None, request=None, view_name=None):

  `template` can be a list of strings or a single string. If a list of strings a template will be selected from them.

  I'll reconsider the large list of default context variables later on in a separate ticket.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/664#issuecomment-581770979
issue_url: https://api.github.com/repos/simonw/datasette/issues/664
node_id: MDEyOklzc3VlQ29tbWVudDU4MTc3MDk3OQ==
created_at: 2020-02-04T07:02:37Z
updated_at: 2020-02-04T07:02:37Z
author_association: OWNER
body:
  More importantly, I need to prototype up a plugin that actually uses this method - probably `datasette-upload-csvs` - to make sure it serves its purpose.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/664#issuecomment-581988901
issue_url: https://api.github.com/repos/simonw/datasette/issues/664
node_id: MDEyOklzc3VlQ29tbWVudDU4MTk4ODkwMQ==
created_at: 2020-02-04T16:17:55Z
updated_at: 2020-02-04T16:17:55Z
author_association: OWNER
body:
  It doesn't make sense for extra body scripts to be available for all template renders (including from plugins) but extra CSS and JS to only be handled by the BaseView render method - especially since extra body scripts are likely to depend on extra JS files.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/664#issuecomment-582089909
issue_url: https://api.github.com/repos/simonw/datasette/issues/664
node_id: MDEyOklzc3VlQ29tbWVudDU4MjA4OTkwOQ==
created_at: 2020-02-04T20:03:07Z
updated_at: 2020-02-04T20:03:07Z
author_association: OWNER
body:
  I'm going to ship this without documentation, get a couple of plugins out that use it and then add documentation once I'm confident it's right.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/653#issuecomment-582100643
issue_url: https://api.github.com/repos/simonw/datasette/issues/653
node_id: MDEyOklzc3VlQ29tbWVudDU4MjEwMDY0Mw==
created_at: 2020-02-04T20:29:56Z
updated_at: 2020-02-04T20:29:56Z
author_association: OWNER
body:
  I think there's one nasty edge-case here that we need to worry about: SQLite allows newlines inside of single quoted strings and I've actually started using that in quite a few places - it's great for [embedding markdown in a string](https://github.com/simonw/datasette-render-markdown) for example.

  The way you're stripping comments right now splits on newlines and removes any lines that start with `--`. I believe that will mangle the following example:

  ```sql
  select '# Hello there

  * This is a list
  * of items
  --
  [And a link](https://github.com/simonw/datasette-render-markdown).'
  as demo_markdown
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/653#issuecomment-582101530
issue_url: https://api.github.com/repos/simonw/datasette/issues/653
node_id: MDEyOklzc3VlQ29tbWVudDU4MjEwMTUzMA==
created_at: 2020-02-04T20:32:11Z
updated_at: 2020-02-04T20:32:11Z
author_association: OWNER
body:
  This is the kind of problem that has made me think that Datasette would really benefit from including a smart SQLite-syntax SQL parser.

  Writing one is a bit of a challenge though! There's an example (derived from SQLite SELECT statements) included in pyparsing here but I've not spent much time evaluating it: https://github.com/pyparsing/pyparsing/blob/master/examples/select_parser.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/665#issuecomment-582103280
issue_url: https://api.github.com/repos/simonw/datasette/issues/665
node_id: MDEyOklzc3VlQ29tbWVudDU4MjEwMzI4MA==
created_at: 2020-02-04T20:36:48Z
updated_at: 2020-02-04T20:36:48Z
author_association: OWNER
body:
  pyparsing has an example based on SQLite SELECT statements: https://github.com/pyparsing/pyparsing/blob/8d9ab59a2b2767ad56c9b852c325075113718c0a/examples/select_parser.py

  https://github.com/lark-parser/lark is a relatively new (less than two years old) parsing library that looks promising too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/653#issuecomment-582103856
issue_url: https://api.github.com/repos/simonw/datasette/issues/653
node_id: MDEyOklzc3VlQ29tbWVudDU4MjEwMzg1Ng==
created_at: 2020-02-04T20:38:18Z
updated_at: 2020-02-04T20:38:18Z
author_association: OWNER
body:
  We can probably solve this without a SQL parser though. Really all we care about here is that if the FIRST lines of the statement begin with `--` we ignore them and only validate the statement starting from the first non-commented line. I think we can do that without single quoted strings causing us confusion.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/653#issuecomment-582105810
issue_url: https://api.github.com/repos/simonw/datasette/issues/653
node_id: MDEyOklzc3VlQ29tbWVudDU4MjEwNTgxMA==
created_at: 2020-02-04T20:43:01Z
updated_at: 2020-02-04T20:43:01Z
author_association: CONTRIBUTOR
body:
  I *think* the existing code will be OK even if I strip the lines in the middle of a new line delimited string.

  It's only used for the validation, SQLite handles the `--` just fine and the whole SQL textarea still gets sent once it passes validation.

  I can add your test case to my branch later this evening though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/653#issuecomment-582141626
issue_url: https://api.github.com/repos/simonw/datasette/issues/653
node_id: MDEyOklzc3VlQ29tbWVudDU4MjE0MTYyNg==
created_at: 2020-02-04T22:13:31Z
updated_at: 2020-02-04T22:13:31Z
author_association: OWNER
body:
  Looks like SQLite supports `/* ... */` style comments as well. I don't think supporting those should be a requirement to land this though. https://www.sqlite.org/lang_comment.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/653#issuecomment-582211745
issue_url: https://api.github.com/repos/simonw/datasette/issues/653
node_id: MDEyOklzc3VlQ29tbWVudDU4MjIxMTc0NQ==
created_at: 2020-02-05T02:28:05Z
updated_at: 2020-02-05T02:28:05Z
author_association: OWNER
body:
  This is shipped in Datasette 0.35. Here's a demo of it working:

  https://latest.datasette.io/fixtures?sql=--+this+is+a+comment%0D%0Aselect+*+from+%5B123_starts_with_digits%5D

  Compare with https://v0-34.datasette.io/fixtures?sql=--+this+is+a+comment%0D%0Aselect+*+from+%5B123_starts_with_digits%5D which returned an error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/658#issuecomment-583177728
issue_url: https://api.github.com/repos/simonw/datasette/issues/658
node_id: MDEyOklzc3VlQ29tbWVudDU4MzE3NzcyOA==
created_at: 2020-02-07T00:28:55Z
updated_at: 2020-02-07T00:29:50Z
author_association: NONE
body:
  Simon,

  Yes, there is an "app.css" on static folder, however, anyone modification I do on this .css, doesn't apply on the datasette.

  I'm using this command: datasette publish heroku _"databases folder"_ -n _"herokuapp name"_ --extra-options="--config sql_time_limit_ms:60000 --config max_returned_rows:10000 --config force_https_urls:1" --template-dir _"templates folder"_ -m _"metadata.json folder"_

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/352#issuecomment-584203999
issue_url: https://api.github.com/repos/simonw/datasette/issues/352
node_id: MDEyOklzc3VlQ29tbWVudDU4NDIwMzk5OQ==
created_at: 2020-02-10T16:18:58Z
updated_at: 2020-02-10T16:18:58Z
author_association: NONE
body:
  I don't want to re-open this issue, but I'm wondering whether it would be possible to include the full row for which a specific cell is to be rendered in the hook signature. My use case are rows where custom rendering would need access to multiple values (specifically, rows containing the constituents of interlinear glossed text (IGT) in separate columns, see https://github.com/cldf/cldf/tree/master/components/examples).

  I could probably cobble this together with custom SQL and the sql-to-html plugin. But having a full row within a `render_cell` implementation seems a lot simpler.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/85#issuecomment-584426938
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/85
node_id: MDEyOklzc3VlQ29tbWVudDU4NDQyNjkzOA==
created_at: 2020-02-11T00:35:09Z
updated_at: 2020-02-11T00:35:09Z
author_association: OWNER
body:
  Here's why:
  https://github.com/simonw/sqlite-utils/blob/0c2451e0690c5f4e6463a2f339b0a280e30ed806/sqlite_utils/db.py#L627-L636

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/327#issuecomment-584657949
issue_url: https://api.github.com/repos/simonw/datasette/issues/327
node_id: MDEyOklzc3VlQ29tbWVudDU4NDY1Nzk0OQ==
created_at: 2020-02-11T14:21:15Z
updated_at: 2020-02-11T14:21:15Z
author_association: NONE
body:
  See https://github.com/simonw/datasette/issues/657 and my changes that allow datasette to load parquet files 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/667#issuecomment-585109972
issue_url: https://api.github.com/repos/simonw/datasette/issues/667
node_id: MDEyOklzc3VlQ29tbWVudDU4NTEwOTk3Mg==
created_at: 2020-02-12T09:21:22Z
updated_at: 2020-02-12T09:21:22Z
author_association: NONE
body:
  I think I found a better way to implement my use case: I wrap the `datasette serve` call into my own cli, which
  - creates the SQLite from CSV data
  - writes `metadata.json` for datasette
  - determines suitable config like `max_page_size`
  - then calls `datasette serve`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/667#issuecomment-585285753
issue_url: https://api.github.com/repos/simonw/datasette/issues/667
node_id: MDEyOklzc3VlQ29tbWVudDU4NTI4NTc1Mw==
created_at: 2020-02-12T16:18:22Z
updated_at: 2020-02-12T16:18:22Z
author_association: NONE
body:
  @simonw fwiw, here's the plugin I implemented to support CLDF datasets: https://github.com/cldf/datasette-cldf/blob/master/README.md
  It's a bit of a hybrid in that it does both, building the SQLite database **and** extending datasette by exploting what we know about the data format - so it may not be worth listing it with the other plugins.

  Having tools like datasette available definitely helps selling people on package formats like CLDF (or CSVW), many thanks for this!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-585870836
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDU4NTg3MDgzNg==
created_at: 2020-02-13T17:19:23Z
updated_at: 2020-02-13T17:19:37Z
author_association: OWNER
body:
  I'm excited about https://github.com/MagicStack/asyncpg for this - it's a true async PostgreSQL library (my SQLite queries run in a threadpool right now) with extremely impressive performance benchmarks, from the team behind `uvloop`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-585872538
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDU4NTg3MjUzOA==
created_at: 2020-02-13T17:22:54Z
updated_at: 2020-02-13T17:22:54Z
author_association: OWNER
body:
  A couple of things I'd like to support:
  - The same datasette instance can have both PostgreSQL and SQLite databases attached to it, and both types will be listed on the homepage.
  - The full test suite runs against both SQLite and PostgreSQL, with as few changes as possible (maybe a few `skipIf()` decorators for features not available on both). Probably easiest do do this with some kind of flag - e.g. `pytest --database=sqlite` v.s. `pytest --database=postgresql`

  I can implement that with this in `conftest.py`:
  ```python
  def pytest_addoption(parser):
      parser.addoption("--database", action="store", default="sqlite")
  ```
  See https://stackoverflow.com/questions/40880259/how-to-pass-arguments-in-pytest-by-command-line

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-585873401
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDU4NTg3MzQwMQ==
created_at: 2020-02-13T17:24:38Z
updated_at: 2020-02-13T17:24:38Z
author_association: OWNER
body:
  The biggest difference between the two will be around introspection. I searched the codebase for potential introspection queries, defined as `select ... from sqlite_master` and `pragma ...` statements.

  - 7 results for sqlite_master in database.py
  - 3 results in utils/__init__.py
  - (Ignoring the 4 in inspect.py)
  - 1 result for “pragma table_info” in app.py
  - 2 in utils/__init__.py
  - 2 results for “pragma foreign_key_list” in utils/__init__.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-585874699
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDU4NTg3NDY5OQ==
created_at: 2020-02-13T17:26:58Z
updated_at: 2020-02-13T17:27:13Z
author_association: OWNER
body:
  First page to get working: the database view page, which shows a list of tables.

  In the code this is entirely implemented with calls to the `Database` class, which is a good starting point - I can drop in a new version of that class that talks to PostgreSQL instead:

  https://github.com/simonw/datasette/blob/0091dfe3e5a3db94af8881038d3f1b8312bb857d/datasette/views/database.py#L25-L66

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-585883109
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDU4NTg4MzEwOQ==
created_at: 2020-02-13T17:44:08Z
updated_at: 2020-02-13T17:44:08Z
author_association: OWNER
body:
  Introspecting columns in PostgreSQL:
  ```
  In [3]: sql = """SELECT * 
     ...:   FROM information_schema.columns 
     ...:  WHERE table_schema = 'public' 
     ...:    AND table_name   = 'blog_blogmark' 
     ...:      ;"""                                                                                                                                                     
  ```
  Each column looks like this:
  ```pythton
  {'table_catalog': 'simonwillisonblog',
   'table_schema': 'public',
   'table_name': 'blog_blogmark',
   'column_name': 'id',
   'ordinal_position': 1,
   'column_default': "nextval('blog_blogmark_id_seq'::regclass)",
   'is_nullable': 'NO',
   'data_type': 'integer',
   'character_maximum_length': None,
   'character_octet_length': None,
   'numeric_precision': 32,
   'numeric_precision_radix': 2,
   'numeric_scale': 0,
   'datetime_precision': None,
   'interval_type': None,
   'interval_precision': None,
   'character_set_catalog': None,
   'character_set_schema': None,
   'character_set_name': None,
   'collation_catalog': None,
   'collation_schema': None,
   'collation_name': None,
   'domain_catalog': None,
   'domain_schema': None,
   'domain_name': None,
   'udt_catalog': 'simonwillisonblog',
   'udt_schema': 'pg_catalog',
   'udt_name': 'int4',
   'scope_catalog': None,
   'scope_schema': None,
   'scope_name': None,
   'maximum_cardinality': None,
   'dtd_identifier': '1',
   'is_self_referencing': 'NO',
   'is_identity': 'NO',
   'identity_generation': None,
   'identity_start': None,
   'identity_increment': None,
   'identity_maximum': None,
   'identity_minimum': None,
   'identity_cycle': 'NO',
   'is_generated': 'NEVER',
   'generation_expression': None,
   'is_updatable': 'YES'}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-585885242
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDU4NTg4NTI0Mg==
created_at: 2020-02-13T17:48:27Z
updated_at: 2020-02-13T17:48:27Z
author_association: OWNER
body:
  Finding out the primary keys for a table: https://wiki.postgresql.org/wiki/Retrieve_primary_key_columns

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-585885812
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDU4NTg4NTgxMg==
created_at: 2020-02-13T17:49:33Z
updated_at: 2020-02-13T17:49:33Z
author_association: OWNER
body:
  ```
  In [12]: await conn.fetch("""SELECT a.attname, format_type(a.atttypid, a.atttypmod) AS data_type 
      ...: FROM   pg_index i 
      ...: JOIN   pg_attribute a ON a.attrelid = i.indrelid 
      ...:                      AND a.attnum = ANY(i.indkey) 
      ...: WHERE  i.indrelid = 'blog_blogmark'::regclass 
      ...: AND    i.indisprimary;""")                                                                                                                                   
  Out[12]: [<Record attname='id' data_type='integer'>]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-585903240
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDU4NTkwMzI0MA==
created_at: 2020-02-13T18:28:42Z
updated_at: 2020-02-13T18:29:14Z
author_association: OWNER
body:
  Challenge: what's the equivalent for PostgreSQL of opening a database in read only mode? Will I have to talk users through creating read only credentials? Can I do this at runtime somehow? Can I detect if the connection has write permission and disable the arbitrary query feature?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-585962849
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDU4NTk2Mjg0OQ==
created_at: 2020-02-13T20:44:18Z
updated_at: 2020-02-13T20:46:53Z
author_association: OWNER
body:
  Got the database page working! It lists tables, their columns and their row count.

  Got the table page partially working! It can list rows.

  It can't apply filters yet (because PostgreSQL `$1` parameters differ from SQLite `:blah` parameters) and faceting doesn't work because PostgreSQL requires that subqueries have an alias:
  ```
  subquery in FROM must have an alias HINT: For example, FROM (SELECT ...) [AS] foo.
  ```
  Still a pretty promising start though!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/417#issuecomment-586047525
issue_url: https://api.github.com/repos/simonw/datasette/issues/417
node_id: MDEyOklzc3VlQ29tbWVudDU4NjA0NzUyNQ==
created_at: 2020-02-14T01:03:43Z
updated_at: 2020-02-14T01:59:02Z
author_association: OWNER
body:
  OK, I have a plan. I'm going to try and implement this is a core Datasette feature (no plugins) with the following design:

  - You can tell Datasette "load any databases you find in this directory" by passing the `--dir=path/to/dir` option to `datasette` that are valid SQLite files and will attach them to Datasette
  - Every 10 seconds Datasette will re-scan those directories to see if any new files have been added
  - That 10s will be the default for a new `--config directory_scan_s:10` config option. You can set this to `0` to disable scanning entirely, at which point Datasette will only run the scan once on startup.

  To check if a file is valid SQLite, Datasette will first check if the first few bytes of the file are `b"SQLite format 3\x00"`. If they are, it will open a connection to the file and attempt to run `select * from sqlite_master` against it. If that runs without any errors it will assume the file is usable and connect it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/417#issuecomment-586047995
issue_url: https://api.github.com/repos/simonw/datasette/issues/417
node_id: MDEyOklzc3VlQ29tbWVudDU4NjA0Nzk5NQ==
created_at: 2020-02-14T01:05:20Z
updated_at: 2020-02-14T01:05:20Z
author_association: OWNER
body:
  I'm going to add two methods to the Datasette class to help support this work (and to enable exciting new plugin opportunities in the future):

  - `datasette.add_database(name, db)` - adds a new named database to the list of connected databases. `db` will be a `Database()` object, which may prove useful in the future for things like #670 and could also allow some plugins to provide in-memory SQLite databases.
  - `datasette.remove_database(name)`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/576#issuecomment-586053947
issue_url: https://api.github.com/repos/simonw/datasette/issues/576
node_id: MDEyOklzc3VlQ29tbWVudDU4NjA1Mzk0Nw==
created_at: 2020-02-14T01:29:48Z
updated_at: 2020-05-30T13:22:09Z
author_association: OWNER
body:
  OK, I've made a start on this now in 3ffb8f3b98252531d11897fd431711e9b8045ace - still plenty more methods to document. More importantly that class has a LOT of junk methods on that no-one should ever call from a plugin, so I need to decide what to do about those.

  https://datasette.readthedocs.io/en/latest/internals.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-586067794
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDU4NjA2Nzc5NA==
created_at: 2020-02-14T02:29:16Z
updated_at: 2020-02-14T02:29:16Z
author_association: OWNER
body:
  One design issue: how to pick neat unique names for database files in a file hierarchy?

  Here's what I have so far:

  https://github.com/simonw/datasette/blob/fe6f9e6a7397cab2e4bc57745a8da9d824dad218/datasette/app.py#L231-L237

  For these files:
  ```
  ../travel-old.db
  ../sf-tree-history/trees.db
  ../library-of-congress/records-from-df.db
  ```
  It made these names:
  ```
  travel-old
  sf-tree-history_trees
  library-of-congress_records-from-df
  ```
  Maybe this is good enough? Needs some tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-586068095
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDU4NjA2ODA5NQ==
created_at: 2020-02-14T02:30:37Z
updated_at: 2020-02-14T02:30:46Z
author_association: OWNER
body:
  This can take a LONG time to run, and at the moment it's blocking and prevents Datasette from starting up.

  It would be much better if this ran in a thread, or an asyncio task. Probably have to be a thread because there's no easy `async` version of `pathlib.Path.glob()` that I've seen.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-586069529
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDU4NjA2OTUyOQ==
created_at: 2020-02-14T02:37:17Z
updated_at: 2020-02-14T02:37:17Z
author_association: OWNER
body:
  Another problem: if any of the found databases use SpatiaLite then Datasette will fail to start at all.

  It should skip them instead.

  The `select * from sqlite_master` check apparently isn't quite enough to catch this case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-586107989
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDU4NjEwNzk4OQ==
created_at: 2020-02-14T05:45:12Z
updated_at: 2020-02-14T05:45:12Z
author_association: OWNER
body:
  I tried running the `scan_dirs()` method in a thread and got an interesting error while trying to load the homepage: `RuntimeError: OrderedDict mutated during iteration`

  Makes sense - I had a thread that added an item to that dictionary right while the homepage was attempting to run this code:

  https://github.com/simonw/datasette/blob/efa54b439fd0394440c302602b919255047b59c5/datasette/views/index.py#L24-L27

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-586109032
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDU4NjEwOTAzMg==
created_at: 2020-02-14T05:50:15Z
updated_at: 2020-02-14T05:50:15Z
author_association: OWNER
body:
  So I need to ensure the `ds.databases` data structure is manipulated in a thread-safe manner.

  Mainly I need to ensure that it is locked during iterations over it, then unlocked at the end.

  Trickiest part is probably ensuring there is a test that proves this is working - I feel like I got lucky encountering that `RuntimeError` as early as I did.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-586109238
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDU4NjEwOTIzOA==
created_at: 2020-02-14T05:51:12Z
updated_at: 2020-02-14T05:51:12Z
author_association: OWNER
body:
  ... or maybe I can cheat and wrap the access to `self.ds.databases.items()` in `list()`, so I'm iterating over an atomically-created list of those things instead? I'll try that first.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-586109784
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDU4NjEwOTc4NA==
created_at: 2020-02-14T05:53:50Z
updated_at: 2020-02-14T05:54:21Z
author_association: OWNER
body:
  ... cheating like this seems to work:
  ```
  for name, db in list(self.ds.databases.items()):
  ```
  Python built-in operations are supposedly threadsafe, so in this case I can grab a copy of the list atomically (I think) and then safely iterate over it.

  Seems to work in my testing. Wish I could prove it with a unit test though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/672#issuecomment-586111102
issue_url: https://api.github.com/repos/simonw/datasette/issues/672
node_id: MDEyOklzc3VlQ29tbWVudDU4NjExMTEwMg==
created_at: 2020-02-14T05:59:24Z
updated_at: 2020-02-14T06:00:36Z
author_association: OWNER
body:
  Interesting new problem: hitting Ctrl+C no longer terminates the problem provided that `scan_dirs()` thread is still running.

  https://stackoverflow.com/questions/49992329/the-workers-in-threadpoolexecutor-is-not-really-daemon has clues. The workers are only meant to exit when their worker queues are empty.

  But... I want to run the worker every 10 seconds. How do I do that without having it loop forever and hence never quit?

reactions:
[cell repeats an earlier one in this file: 112 characters]


