# issue_comments

html_url: https://github.com/simonw/datasette/issues/272#issuecomment-394431323
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDM5NDQzMTMyMw==
created_at: 2018-06-04T17:17:37Z
updated_at: 2018-06-04T17:17:37Z
author_association: OWNER
body:
  I built this ASGI debugging tool to help with this migration: https://asgi-scope.now.sh/fivethirtyeight-34d6604/most-common-name%2Fsurnames.json?foo=bar&bazoeuto=onetuh&a=.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-394503399
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDM5NDUwMzM5OQ==
created_at: 2018-06-04T21:20:14Z
updated_at: 2018-06-04T21:20:14Z
author_association: OWNER
body:
  Results of an extremely simple micro-benchmark comparing the two shows that uvicorn is at least as fast as Sanic (benchmarks a little faster with a very simple payload): https://gist.github.com/simonw/418950af178c01c416363cc057420851

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-394764713
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDM5NDc2NDcxMw==
created_at: 2018-06-05T15:58:54Z
updated_at: 2018-06-05T16:00:40Z
author_association: OWNER
body:
  https://github.com/encode/uvicorn/blob/572b5fe6c811b63298d5350a06b664839624c860/uvicorn/run.py#L63 is how you start a Uvicorn server from code as opposed to the `uvicorn` CLI

      from uvicorn.run import UvicornServer
      UvicornServer().run(app, host=host, port=port)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/306#issuecomment-394894500
issue_url: https://api.github.com/repos/simonw/datasette/issues/306
node_id: MDEyOklzc3VlQ29tbWVudDM5NDg5NDUwMA==
created_at: 2018-06-05T23:40:40Z
updated_at: 2018-06-05T23:40:40Z
author_association: OWNER
body:
  Input:
  - function that says if a name is a valid database
  - Function that says if a table exists
  - URL

  Output:
  - view class
  - Arguments
  - Redirect (if it should redirect)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/306#issuecomment-394894910
issue_url: https://api.github.com/repos/simonw/datasette/issues/306
node_id: MDEyOklzc3VlQ29tbWVudDM5NDg5NDkxMA==
created_at: 2018-06-05T23:43:18Z
updated_at: 2018-06-05T23:49:41Z
author_association: OWNER
body:
  I'm going to use a named tuple for the output. That way I can support either tuple destructing or explicit property access on the returned value.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/306#issuecomment-394895267
issue_url: https://api.github.com/repos/simonw/datasette/issues/306
node_id: MDEyOklzc3VlQ29tbWVudDM5NDg5NTI2Nw==
created_at: 2018-06-05T23:45:26Z
updated_at: 2018-06-05T23:45:26Z
author_association: OWNER
body:
  To support a future where Datasette is an ASGI app that can be attached to a URL within a larger application the routing function should have the option to accept a path prefix which will then be automatically attached to any resulting redirects.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/306#issuecomment-394895750
issue_url: https://api.github.com/repos/simonw/datasette/issues/306
node_id: MDEyOklzc3VlQ29tbWVudDM5NDg5NTc1MA==
created_at: 2018-06-05T23:48:06Z
updated_at: 2018-06-06T23:50:31Z
author_association: OWNER
body:
  A neat trick could be that if the router returns a redirect it could then resolve that redirect to see if it will 404 (or redirect itself) before returning that response.

  This would need its own counter to guard against infinite redirects. 

  I'm not going to do this though: any view that results in a chain of redirects like this is a bug that should be fixed at the source.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/306#issuecomment-395463497
issue_url: https://api.github.com/repos/simonw/datasette/issues/306
node_id: MDEyOklzc3VlQ29tbWVudDM5NTQ2MzQ5Nw==
created_at: 2018-06-07T15:29:28Z
updated_at: 2018-06-07T15:29:28Z
author_association: OWNER
body:
  I started sketching this out in a branch, see pull request #307 - but I've decided I don't like it. I'm going to close this ticket and stick with regular expression URL routing for the moment. If I change my mind in the future the code in #307 lives in separate files (`datasette/routes.py` and `tests/test_routes.py`) so bringing it back into the project will be trivial.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/307#issuecomment-395463598
issue_url: https://api.github.com/repos/simonw/datasette/issues/307
node_id: MDEyOklzc3VlQ29tbWVudDM5NTQ2MzU5OA==
created_at: 2018-06-07T15:29:41Z
updated_at: 2018-06-07T15:29:41Z
author_association: OWNER
body:
  Closing this pull request for reasons outlined here: https://github.com/simonw/datasette/issues/306#issuecomment-395463497

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/305#issuecomment-396048471
issue_url: https://api.github.com/repos/simonw/datasette/issues/305
node_id: MDEyOklzc3VlQ29tbWVudDM5NjA0ODQ3MQ==
created_at: 2018-06-10T13:16:13Z
updated_at: 2018-06-10T13:16:13Z
author_association: OWNER
body:
  https://github.com/kubernetes/community/blob/master/contributors/devel/help-wanted.md Is worth stealing from too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397534196
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5NzUzNDE5Ng==
created_at: 2018-06-15T07:12:16Z
updated_at: 2018-06-15T07:12:16Z
author_association: OWNER
body:
  The first version of this is now shipped to master. I ended up rewriting most of the experimental branch to deal with the nasty issue described in #303

  Demo is available on https://fivethirtyeight.datasettes.com/fivethirtyeight-ab24e01/most-common-name%2Fsurnames

  ![2018-06-15 at 12 11 am](https://user-images.githubusercontent.com/9599/41455090-bd5ece30-7030-11e8-8da4-11fbb1f2ef8b.png)

  Here's the CSV version of that page: https://fivethirtyeight.datasettes.com/fivethirtyeight-ab24e01/most-common-name%2Fsurnames.csv

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397534404
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5NzUzNDQwNA==
created_at: 2018-06-15T07:13:20Z
updated_at: 2018-06-15T07:13:20Z
author_association: OWNER
body:
  Still to add: the streaming version that iterates through all of the pages, as seen in experimental commit https://github.com/simonw/datasette/commit/ced379ea325787b8c3bf0a614daba1fa4856a3bd

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/233#issuecomment-397637302
issue_url: https://api.github.com/repos/simonw/datasette/issues/233
node_id: MDEyOklzc3VlQ29tbWVudDM5NzYzNzMwMg==
created_at: 2018-06-15T14:24:08Z
updated_at: 2018-06-15T14:55:19Z
author_association: OWNER
body:
  I'm going with the terminology "labels" here. You'll be able to add ``?_labels=1`` and the JSON will look something like this:
  ```
  {
      "rowid": 233,
      "TreeID": 121240,
      "qLegalStatus": {
          "value" 2,
          "label": "Private"
      }
      "qSpecies": {
          "value": 16,
          "label": "Sycamore"
      }
      "qAddress": "91 Commonwealth Ave",
      ...
  }
  ```

  I need this to help build foreign key expansions for CSV files, see #266 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/233#issuecomment-397648080
issue_url: https://api.github.com/repos/simonw/datasette/issues/233
node_id: MDEyOklzc3VlQ29tbWVudDM5NzY0ODA4MA==
created_at: 2018-06-15T14:56:21Z
updated_at: 2018-06-15T14:56:21Z
author_association: OWNER
body:
  I considered including a `"table"` key like this:

  ```
      "qLegalStatus": {
          "value" 2,
          "label": "Private",
          "table": "qLegalStatus"
      }
  ```

  This would help generate the HTML links using just the JSON data. But... I realized that in a list of 50 rows that value would be duplicated 50 times which is a bit nasty.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/233#issuecomment-397663968
issue_url: https://api.github.com/repos/simonw/datasette/issues/233
node_id: MDEyOklzc3VlQ29tbWVudDM5NzY2Mzk2OA==
created_at: 2018-06-15T15:51:17Z
updated_at: 2018-06-15T15:51:17Z
author_association: OWNER
body:
  Nearly done, but I need the HTML view to ignore the `?_labels=1` param (it throws an error at the moment).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/233#issuecomment-397668427
issue_url: https://api.github.com/repos/simonw/datasette/issues/233
node_id: MDEyOklzc3VlQ29tbWVudDM5NzY2ODQyNw==
created_at: 2018-06-15T16:07:43Z
updated_at: 2018-06-15T16:07:43Z
author_association: OWNER
body:
  Demo: https://datasette-json-labels-demo.now.sh/fixtures-fda0fea/facetable.json?_labels=1&_shape=array

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/233#issuecomment-397729319
issue_url: https://api.github.com/repos/simonw/datasette/issues/233
node_id: MDEyOklzc3VlQ29tbWVudDM5NzcyOTMxOQ==
created_at: 2018-06-15T20:10:24Z
updated_at: 2018-06-15T20:10:24Z
author_association: OWNER
body:
  I'm also going to add the ability to specify individual columns that you want to expand using `?_label=city_id&_label=state_id`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/233#issuecomment-397729500
issue_url: https://api.github.com/repos/simonw/datasette/issues/233
node_id: MDEyOklzc3VlQ29tbWVudDM5NzcyOTUwMA==
created_at: 2018-06-15T20:11:14Z
updated_at: 2018-06-15T20:11:14Z
author_association: OWNER
body:
  The `.json` and `.csv` links displayed on the table page should default to using `?_labels=1` if Datasette detects that there are foreign key expansions available for the page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397729945
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5NzcyOTk0NQ==
created_at: 2018-06-15T20:13:05Z
updated_at: 2018-06-15T20:13:05Z
author_association: OWNER
body:
  The "This data as ..." area of the page is getting a bit untidy, especially if I'm going to add other download options in the future. I think I'll move the HTML to the page footer (less concerns about taking up lots of space there) and then have a bit of JavaScript that turns it into a show/hide menu of some sort in its current location.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/311#issuecomment-397823913
issue_url: https://api.github.com/repos/simonw/datasette/issues/311
node_id: MDEyOklzc3VlQ29tbWVudDM5NzgyMzkxMw==
created_at: 2018-06-16T16:32:07Z
updated_at: 2018-06-16T16:48:48Z
author_association: OWNER
body:
  Still todo:

  - [ ] HTML view to obey the ?_labels=1 param (it throws an error at the moment)
  - [ ] `?_label=one&_label=2` support for only expanding specific labels
  - [ ] Better docs

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/233#issuecomment-397824991
issue_url: https://api.github.com/repos/simonw/datasette/issues/233
node_id: MDEyOklzc3VlQ29tbWVudDM5NzgyNDk5MQ==
created_at: 2018-06-16T16:50:31Z
updated_at: 2018-06-16T16:50:42Z
author_association: OWNER
body:
  I'm going to support `?_labels=` on HTML views, but I'll allow it to be used to turn them off (they are on by default) using `?_labels=off`.

  Related: 7e0caa1e62607c6579101cc0e62bec8899013715 where I added a new `value_as_boolean` helper extracted from how `--config` works in `cli.py`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/233#issuecomment-397839482
issue_url: https://api.github.com/repos/simonw/datasette/issues/233
node_id: MDEyOklzc3VlQ29tbWVudDM5NzgzOTQ4Mg==
created_at: 2018-06-16T21:21:03Z
updated_at: 2018-06-16T21:21:03Z
author_association: OWNER
body:
  Should facets always have their labels expanded or should they also obey the `_labels` and `_label` querystring arguments?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/233#issuecomment-397839583
issue_url: https://api.github.com/repos/simonw/datasette/issues/233
node_id: MDEyOklzc3VlQ29tbWVudDM5NzgzOTU4Mw==
created_at: 2018-06-16T21:23:14Z
updated_at: 2018-06-16T21:23:44Z
author_association: OWNER
body:
  I'm a bit torn on naming - choices are:

  * `?_labels=on` and `?_label=col1&_label=col2`
  * `?_expands=on` (or `?_expand_all=on`) and `?_expand=col1&_expand=col2`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/233#issuecomment-397842194
issue_url: https://api.github.com/repos/simonw/datasette/issues/233
node_id: MDEyOklzc3VlQ29tbWVudDM5Nzg0MjE5NA==
created_at: 2018-06-16T22:26:21Z
updated_at: 2018-06-16T22:26:21Z
author_association: OWNER
body:
  Some demos:

  * https://datasette-labels-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List - regular HTML view
  * https://datasette-labels-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List?_labels=off - no labels
  * https://datasette-labels-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List.json?_labels=on - JSON with all labels
  * https://datasette-labels-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List.json?_label=qSpecies&_shape=array - JSON with specific labels in array shape
  * https://datasette-labels-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List.csv?_labels=on - CSV with all labels
  * https://datasette-labels-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List.csv?_label=qSpecies - CSV with specific labels

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397842246
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5Nzg0MjI0Ng==
created_at: 2018-06-16T22:27:59Z
updated_at: 2018-06-16T22:27:59Z
author_association: OWNER
body:
  Two demos of the new functionality in #233 as it applies to CSV:

  * https://datasette-labels-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List.csv?_labels=on - CSV with all foreign key columns expanded
  * https://datasette-labels-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List.csv?_label=qSpecies - CSV with specific columns expanded

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397842667
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5Nzg0MjY2Nw==
created_at: 2018-06-16T22:38:15Z
updated_at: 2018-06-18T05:55:11Z
author_association: OWNER
body:
  Still todo:

  - [x] Streaming version
  - [ ] Tidy up the "This data as ..." UI
  - [x] Default .csv (and .json) links to use `?_labels=on` (only if at least one foreign key detected)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/313#issuecomment-397900434
issue_url: https://api.github.com/repos/simonw/datasette/issues/313
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkwMDQzNA==
created_at: 2018-06-17T19:23:23Z
updated_at: 2018-06-17T19:23:23Z
author_association: OWNER
body:
  This will require some relatively sophisticated Travis build steps. Useful docs: https://docs.travis-ci.com/user/build-stages/ - useful example: https://docs.travis-ci.com/user/build-stages/deploy-heroku/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/313#issuecomment-397907987
issue_url: https://api.github.com/repos/simonw/datasette/issues/313
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkwNzk4Nw==
created_at: 2018-06-17T21:32:52Z
updated_at: 2018-06-17T21:32:52Z
author_association: OWNER
body:
  This very nearly works...

  * https://latest.datasette.io/
  * https://f0c1722.datasette.io/

  But... https://f0c1722.datasette.io/-/versions isn't showing the correct note:
  ```
  {
      "datasette": {
          "version": "0.22.1"
      }
      ...
  ```
  There should be a `"note"` field there with the full commit hash.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/313#issuecomment-397908093
issue_url: https://api.github.com/repos/simonw/datasette/issues/313
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkwODA5Mw==
created_at: 2018-06-17T21:34:52Z
updated_at: 2018-06-17T21:34:52Z
author_association: OWNER
body:
  It looks like all of my test deploys ended up going to the same Zeit deployment ID: https://zeit.co/simonw/datasette-latest/rbmtcedvlj

  This is strange... the Dockerfile should be different for each one (due to the differing version-note).

  https://github.com/simonw/datasette/commit/db1e6bc182d11f333e6addaa1a6be87625a4e12b#diff-34418c57343344c73271e13b01b7fcd9R255

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/313#issuecomment-397908185
issue_url: https://api.github.com/repos/simonw/datasette/issues/313
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkwODE4NQ==
created_at: 2018-06-17T21:36:50Z
updated_at: 2018-06-17T21:36:50Z
author_association: OWNER
body:
  ```
  The command "datasette publish now fixtures.db -m fixtures.json --token=$NOW_TOKEN --branch=$TRAVIS_COMMIT --version-note=$TRAVIS_COMMIT" exited with 0.
  ```

  Partial log of the ``datasette publish now`` output:

  ```
  > Step 5/7 : RUN datasette inspect fixtures.db --inspect-file inspect-data.json
  >  ---> Running in d373f330e53e
  >  ---> 09bab386aaa3
  > Removing intermediate container d373f330e53e
  > Step 6/7 : EXPOSE 8001
  >  ---> Running in e0fe37b3061c
  >  ---> 47798440e214
  > Removing intermediate container e0fe37b3061c
  > Step 7/7 : CMD datasette serve --host 0.0.0.0 fixtures.db --cors --port 8001 --inspect-file inspect-data.json --metadata metadata.json --version-note f0c17229b7a7914d3da02e087dfd0e25d8321448
  ```

  So it looks like `--version-note` is being correctly set there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/313#issuecomment-397908614
issue_url: https://api.github.com/repos/simonw/datasette/issues/313
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkwODYxNA==
created_at: 2018-06-17T21:44:51Z
updated_at: 2018-06-17T21:45:03Z
author_association: OWNER
body:
  Aha!
  ```1.03s$ now alias --token=$NOW_TOKEN
  > Error! Couldn't find a deployment to alias. Please provide one as an argument.
  The command "now alias --token=$NOW_TOKEN" exited with 1.
  ```
  That explains it. I need to set the same alias in my call to `datasette publish`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/313#issuecomment-397908947
issue_url: https://api.github.com/repos/simonw/datasette/issues/313
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkwODk0Nw==
created_at: 2018-06-17T21:51:34Z
updated_at: 2018-06-17T21:51:34Z
author_association: OWNER
body:
  That fixed it! https://958b75c.datasette.io/-/versions

  ```
  {
    "python": {
      "version": "3.6.5",
      "full": "3.6.5 (default, Jun  6 2018, 19:19:24) \n[GCC 6.3.0 20170516]"
    },
    "datasette": {
      "version": "0+unknown",
      "note": "958b75c69841ef5913da86e0eb2df634a9b95fda"
    },
    "sqlite": {
      "version": "3.16.2",
      "fts_versions": [
        "FTS5",
        "FTS4",
        "FTS3"
      ],
      "extensions": {
        "json1": null
      }
    }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397912840
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkxMjg0MA==
created_at: 2018-06-17T23:13:35Z
updated_at: 2018-06-17T23:16:42Z
author_association: OWNER
body:
  This worked! https://github.com/simonw/datasette/commit/5a0a82faf9cf9dd109d76181ed00eea19472087c - it spat out a 76MB CSV when I ran it against the sf-trees demo database.

  It was just a quick hack though - it currently ignores `_labels=` and `_dl=` which need to be supported.

  I'm going to add a config option for turning full CSV export off just in case any Datasette users are uncomfortable with URLs that churn out that much data in one go.

  ```
      ConfigOption("allow_csv_stream", True, """
          Allow .csv?_stream=1 to download all rows (ignoring max_returned_rows)
      """.strip()),
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397915258
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkxNTI1OA==
created_at: 2018-06-18T00:01:05Z
updated_at: 2018-06-18T00:01:05Z
author_association: OWNER
body:
  Someone malicious could use a UNION to generate an unpleasantly large CSV response. I'll add another config setting which limits the response size to 100MB but can be turned off by setting it to 0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397915403
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkxNTQwMw==
created_at: 2018-06-18T00:03:17Z
updated_at: 2018-06-18T00:14:37Z
author_association: OWNER
body:
  Since CSV streaming export doesn't work for custom SQL queries (since they don't support `_next=` pagination) there's no need to provide a option that disables streams just for custom SQL.

  Related: the UI should not show the option to download everything on custom SQL pages.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397916091
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkxNjA5MQ==
created_at: 2018-06-18T00:13:43Z
updated_at: 2018-06-18T00:15:50Z
author_association: OWNER
body:
  I was also worried about the performance of pagination over custom `_sort` orders or views which use offset pagination - but Datasette's SQL time limits should prevent those from getting out of hand. This does mean that a streaming CSV file may be truncated with an error - if this happens we should ensure the error is written out as the last line of the CSV so anyone who tried to import it gets a relevant error message informing them that the export did not complete.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397916321
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkxNjMyMQ==
created_at: 2018-06-18T00:17:44Z
updated_at: 2018-06-18T00:18:05Z
author_association: OWNER
body:
  The export UI could be a GET form controlling various parameters. This would discourage crawlers from hitting the export links and would also allow us to express the full range of export options.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397918264
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkxODI2NA==
created_at: 2018-06-18T00:49:35Z
updated_at: 2018-06-18T00:49:35Z
author_association: OWNER
body:
  Simpler design: the top of the page will link to basic .json and .csv and "advanced" - which will fragment link to an advanced export format the bottom of the page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397923253
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5NzkyMzI1Mw==
created_at: 2018-06-18T01:49:52Z
updated_at: 2018-06-18T03:02:28Z
author_association: OWNER
body:
  Ideally the downloadable filenames of exported CSVs would differ across different querystring parameters. Maybe S`treet_Trees-56cbd54.csv` where `56cbd54` is a hash of the querystring?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-397949002
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5Nzk0OTAwMg==
created_at: 2018-06-18T05:53:17Z
updated_at: 2018-06-18T05:53:17Z
author_association: OWNER
body:
  Advanced export pane:

  ![2018-06-17 at 10 52 pm](https://user-images.githubusercontent.com/9599/41520166-3809a45a-7281-11e8-9dfa-2b10f4cb9672.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/316#issuecomment-398030903
issue_url: https://api.github.com/repos/simonw/datasette/issues/316
node_id: MDEyOklzc3VlQ29tbWVudDM5ODAzMDkwMw==
created_at: 2018-06-18T12:00:43Z
updated_at: 2018-06-18T12:00:43Z
author_association: NONE
body:
  I should add that I'm using datasette version 0.22, Python 2.7.10 on Mac OS X.  Happy to send more info if helpful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-398098582
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5ODA5ODU4Mg==
created_at: 2018-06-18T15:40:32Z
updated_at: 2018-06-18T15:40:32Z
author_association: OWNER
body:
  This is now released in Datasette 0.23! http://datasette.readthedocs.io/en/latest/changelog.html#v0-23

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/316#issuecomment-398101670
issue_url: https://api.github.com/repos/simonw/datasette/issues/316
node_id: MDEyOklzc3VlQ29tbWVudDM5ODEwMTY3MA==
created_at: 2018-06-18T15:49:35Z
updated_at: 2018-06-18T15:50:38Z
author_association: OWNER
body:
  Wow, I've gone as high as 7GB but I've never tried it against 600GB.

  `datasette inspect` is indeed expected to take a long time for large databases. That's why it's available as a separate command: by running `datasette inspect` to generate `inspect-data.json` you can execute it just once against a large database and then have `datasette serve` take advantage of that cached metadata (hence avoiding `datasette serve` hanging on startup).

  As you spotted, most of the time is spent in those counts. I imagine you don't need those row counts in order for the rest of Datasette to function correctly (they are mainly used for display purposes - on the https://latest.datasette.io/fixtures index page for example).

  If your database changes infrequently, for the moment I recommend running `datasette inspect` once to generate the `inspect-data.json` file (let me know how long it takes) and then passing that file to `datasette serve mydb.db --inspect-file=inspect-data.json`

  If your database DOES change frequently then this workaround won't help you much. Let me know and I'll see how much work it would take to have those row counts be optional rather than required.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/265#issuecomment-398102537
issue_url: https://api.github.com/repos/simonw/datasette/issues/265
node_id: MDEyOklzc3VlQ29tbWVudDM5ODEwMjUzNw==
created_at: 2018-06-18T15:52:15Z
updated_at: 2018-06-18T15:52:15Z
author_association: OWNER
body:
  https://latest.datasette.io/ now always hosts the latest version of the code. I've started linking to it from our documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/316#issuecomment-398109204
issue_url: https://api.github.com/repos/simonw/datasette/issues/316
node_id: MDEyOklzc3VlQ29tbWVudDM5ODEwOTIwNA==
created_at: 2018-06-18T16:12:45Z
updated_at: 2018-06-18T16:12:45Z
author_association: NONE
body:
  Hi Simon,
  Thanks for the response.  Ok I'll try running `datasette inspect` up front.
  In principle the db won't change.  However, the site's in development and it's likely I'll need to add views and some auxiliary (smaller) tables as I go along.  I will need to be careful with this if it involves an inspect step in each iteration, though.
  g.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/316#issuecomment-398133159
issue_url: https://api.github.com/repos/simonw/datasette/issues/316
node_id: MDEyOklzc3VlQ29tbWVudDM5ODEzMzE1OQ==
created_at: 2018-06-18T17:29:59Z
updated_at: 2018-07-10T15:14:53Z
author_association: OWNER
body:
  For #271 I've been contemplating having Datasette work against an on-disk database that gets modified without needing to restart the server.

  For that to work, I'll have to dramatically change the inspect() mechanism. It may be that inspect becomes an optional optimization in the future.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/271#issuecomment-398133924
issue_url: https://api.github.com/repos/simonw/datasette/issues/271
node_id: MDEyOklzc3VlQ29tbWVudDM5ODEzMzkyNA==
created_at: 2018-06-18T17:32:22Z
updated_at: 2018-06-18T17:32:22Z
author_association: OWNER
body:
  As seen in #316 inspect is already taking a VERY long time to run against large (600GB) databases.

  To get this working I may have to make inspect an optional optimization and run introspection for columns and primary keys in demand.

  The one catch here is the `count(*)` queries - Datasette may need to learn not to return full table counts in circumstances where the count has not been pre-calculates and takes more than Xms to generate.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/321#issuecomment-398973176
issue_url: https://api.github.com/repos/simonw/datasette/issues/321
node_id: MDEyOklzc3VlQ29tbWVudDM5ODk3MzE3Ng==
created_at: 2018-06-21T04:34:11Z
updated_at: 2018-06-21T16:53:57Z
author_association: OWNER
body:
  This is a little bit fiddly, but it's possible to do it using SQLite string concatenation. Here's an example:

  ```
  select * from facetable
  where neighborhood like "%" || :text || "%";
  ```

  Try it here: https://latest.datasette.io/fixtures-35b6eb6?sql=select+*+from+facetable+where+neighborhood+like+%22%25%22+%7C%7C+%3Atext+%7C%7C+%22%25%22%3B&text=town

  ![2018-06-20 at 9 33 pm](https://user-images.githubusercontent.com/9599/41698185-a52143f2-74d1-11e8-8d16-32bfc4542104.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/318#issuecomment-398973309
issue_url: https://api.github.com/repos/simonw/datasette/issues/318
node_id: MDEyOklzc3VlQ29tbWVudDM5ODk3MzMwOQ==
created_at: 2018-06-21T04:35:12Z
updated_at: 2018-06-21T04:37:37Z
author_association: OWNER
body:
  Demo of fix: the `on_earth` facet on https://latest.datasette.io/fixtures-cafd088/facetable?_facet=planet_int&_facet=on_earth&_facet=city_id
  ![2018-06-20 at 9 35 pm](https://user-images.githubusercontent.com/9599/41698208-ebb6b72a-74d1-11e8-9d85-de7600177f69.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/321#issuecomment-398976488
issue_url: https://api.github.com/repos/simonw/datasette/issues/321
node_id: MDEyOklzc3VlQ29tbWVudDM5ODk3NjQ4OA==
created_at: 2018-06-21T04:59:33Z
updated_at: 2018-06-21T06:11:02Z
author_association: OWNER
body:
  I've added this to the unit tests and the documentation.

  Docs: http://datasette.readthedocs.io/en/latest/sql_queries.html#canned-queries
  Canned query demo: https://latest.datasette.io/fixtures/neighborhood_search?text=town

  New unit test:

  https://github.com/simonw/datasette/blob/3683a6b626b2e79f4dc9600d45853ca4ae8de11a/tests/test_api.py#L333-L344

  https://github.com/simonw/datasette/blob/3683a6b626b2e79f4dc9600d45853ca4ae8de11a/tests/fixtures.py#L145-L153

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/321#issuecomment-399106871
issue_url: https://api.github.com/repos/simonw/datasette/issues/321
node_id: MDEyOklzc3VlQ29tbWVudDM5OTEwNjg3MQ==
created_at: 2018-06-21T13:39:37Z
updated_at: 2018-06-21T13:39:37Z
author_association: NONE
body:
  One thing I've noticed with this approach is that the query is executed with no parameters which I do not believe was the case previously.  In the case the table contains a lot of data, this adds some time executing the query before the user can enter their input and run it with the parameters they want.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/321#issuecomment-399126228
issue_url: https://api.github.com/repos/simonw/datasette/issues/321
node_id: MDEyOklzc3VlQ29tbWVudDM5OTEyNjIyOA==
created_at: 2018-06-21T14:36:40Z
updated_at: 2018-06-21T14:36:53Z
author_association: OWNER
body:
  This seems to fix that:
  ```
  select neighborhood, facet_cities.name, state
  from facetable
      join facet_cities on facetable.city_id = facet_cities.id
  where :text != '' and neighborhood like '%' || :text || '%'
  order by neighborhood;
  ```
  Compare this (with empty string): https://latest.datasette.io/fixtures-cafd088?sql=select+neighborhood%2C+facet_cities.name%2C+state%0D%0Afrom+facetable%0D%0A++++join+facet_cities+on+facetable.city_id+%3D+facet_cities.id%0D%0Awhere+%3Atext+%21%3D+%22%22+and+neighborhood+like+%27%25%27+%7C%7C+%3Atext+%7C%7C+%27%25%27%0D%0Aorder+by+neighborhood%3B

  To this: https://latest.datasette.io/fixtures-cafd088?sql=select+neighborhood%2C+facet_cities.name%2C+state%0D%0Afrom+facetable%0D%0A++++join+facet_cities+on+facetable.city_id+%3D+facet_cities.id%0D%0Awhere+%3Atext+%21%3D+%22%22+and+neighborhood+like+%27%25%27+%7C%7C+%3Atext+%7C%7C+%27%25%27%0D%0Aorder+by+neighborhood%3B&text=town

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/321#issuecomment-399129220
issue_url: https://api.github.com/repos/simonw/datasette/issues/321
node_id: MDEyOklzc3VlQ29tbWVudDM5OTEyOTIyMA==
created_at: 2018-06-21T14:45:02Z
updated_at: 2018-06-21T14:45:02Z
author_association: NONE
body:
  Those queries look identical.  How can this be prevented if the queries are in a metadata.json file?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/309#issuecomment-399134680
issue_url: https://api.github.com/repos/simonw/datasette/issues/309
node_id: MDEyOklzc3VlQ29tbWVudDM5OTEzNDY4MA==
created_at: 2018-06-21T14:59:57Z
updated_at: 2018-06-21T14:59:57Z
author_association: OWNER
body:
  I can use Sanic middleware for this: http://sanic.readthedocs.io/en/latest/sanic/middleware.html#responding-early

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/319#issuecomment-399139462
issue_url: https://api.github.com/repos/simonw/datasette/issues/319
node_id: MDEyOklzc3VlQ29tbWVudDM5OTEzOTQ2Mg==
created_at: 2018-06-21T15:13:58Z
updated_at: 2018-06-21T15:13:58Z
author_association: OWNER
body:
  Demo of fix: https://latest.datasette.io/fixtures-e14e080/searchable_tags

  ![2018-06-21 at 8 13 am](https://user-images.githubusercontent.com/9599/41728203-0b571e9a-752b-11e8-9702-9887e3ede5bc.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/317#issuecomment-399144688
issue_url: https://api.github.com/repos/simonw/datasette/issues/317
node_id: MDEyOklzc3VlQ29tbWVudDM5OTE0NDY4OA==
created_at: 2018-06-21T15:29:06Z
updated_at: 2018-06-21T15:29:16Z
author_association: OWNER
body:
  From https://docs.travis-ci.com/user/deployment/pypi/

  > Note that if your PyPI password contains special characters you need to escape them before encrypting your password. Some people have [reported difficulties](https://github.com/travis-ci/dpl/issues/377) connecting to PyPI with passwords containing anything except alphanumeric characters.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/317#issuecomment-399150285
issue_url: https://api.github.com/repos/simonw/datasette/issues/317
node_id: MDEyOklzc3VlQ29tbWVudDM5OTE1MDI4NQ==
created_at: 2018-06-21T15:45:47Z
updated_at: 2018-06-21T15:45:47Z
author_association: OWNER
body:
  That fixed it! https://travis-ci.org/simonw/datasette/jobs/395078407 ran successfully and https://pypi.org/project/datasette/ now hosts Datasette 0.23.1 deployed via Travis.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/314#issuecomment-399156960
issue_url: https://api.github.com/repos/simonw/datasette/issues/314
node_id: MDEyOklzc3VlQ29tbWVudDM5OTE1Njk2MA==
created_at: 2018-06-21T16:04:59Z
updated_at: 2018-06-21T16:04:59Z
author_association: OWNER
body:
  Demo of fix: https://latest.datasette.io/fixtures-e14e080/simple_view

  ![2018-06-21 at 9 04 am](https://user-images.githubusercontent.com/9599/41731021-2be526aa-7532-11e8-9c3b-f787f918328e.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/259#issuecomment-399157944
issue_url: https://api.github.com/repos/simonw/datasette/issues/259
node_id: MDEyOklzc3VlQ29tbWVudDM5OTE1Nzk0NA==
created_at: 2018-06-21T16:07:49Z
updated_at: 2018-06-21T16:07:49Z
author_association: OWNER
body:
  Thanks to #319 the test suite now includes a m2m table: https://latest.datasette.io/fixtures-e14e080/searchable_tags

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/321#issuecomment-399171239
issue_url: https://api.github.com/repos/simonw/datasette/issues/321
node_id: MDEyOklzc3VlQ29tbWVudDM5OTE3MTIzOQ==
created_at: 2018-06-21T16:51:08Z
updated_at: 2018-06-21T16:51:29Z
author_association: OWNER
body:
  I may have misunderstood your problem here.

  I understood that the problem is that when using the `"%" || :text || "%"` construct the first hit to that page (with an empty string for `:text`) results in a `where neighborhood like "%%"` query which is slow because it matches every row in the database.

  My fix was to add this to the where clause:

      where :text != '' and ...

  Which means that when you first load the page the where fails to match any rows and you get no results (and hopefully instant loading times assuming SQLite is smart enough to optimize this away). That's why you don't see any rows returned on this page: https://latest.datasette.io/fixtures-cafd088?sql=select+neighborhood%2C+facet_cities.name%2C+state%0D%0Afrom+facetable%0D%0A++++join+facet_cities+on+facetable.city_id+%3D+facet_cities.id%0D%0Awhere+%3Atext+%21%3D+%22%22+and+neighborhood+like+%27%25%27+%7C%7C+%3Atext+%7C%7C+%27%25%27%0D%0Aorder+by+neighborhood%3B

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/321#issuecomment-399173916
issue_url: https://api.github.com/repos/simonw/datasette/issues/321
node_id: MDEyOklzc3VlQ29tbWVudDM5OTE3MzkxNg==
created_at: 2018-06-21T17:00:10Z
updated_at: 2018-06-21T17:00:10Z
author_association: NONE
body:
  Oh I see..  My issue is that the query executes with an empty string prior to the user submitting the parameters.  I'll try adding your workaround to some of my queries.  Thanks again,

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/326#issuecomment-399721346
issue_url: https://api.github.com/repos/simonw/datasette/issues/326
node_id: MDEyOklzc3VlQ29tbWVudDM5OTcyMTM0Ng==
created_at: 2018-06-24T01:10:26Z
updated_at: 2018-06-24T01:10:26Z
author_association: OWNER
body:
  Demo: go to https://vega.github.io/editor/ and paste in the following:

  ```
  {
    "data": {
      "url": "https://fivethirtyeight.datasettes.com/fivethirtyeight/twitter-ratio%2Fsenators.csv?_size=max&_sort_desc=replies",
      "format": {
        "type": "csv"
      }
    },
    "mark": "bar",
    "encoding": {
      "x": {
        "field": "created_at",
        "type": "temporal"
      },
      "y": {
        "field": "replies",
        "type": "quantitative"
      },
      "color": {
        "field": "user",
        "type": "nominal"
      }
    }
  }
  ```

  ![2018-06-23 at 6 10 pm](https://user-images.githubusercontent.com/9599/41814923-b1613370-7710-11e8-94ac-5b87b0b629ed.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-400571521
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDQwMDU3MTUyMQ==
created_at: 2018-06-27T07:30:07Z
updated_at: 2018-06-27T07:30:07Z
author_association: NONE
body:
  I’m up for helping with this.

  Looks like you’d need static files support, which I’m planning on adding a component for. Anything else obviously missing?

  For a quick overview it looks very doable - the test client ought to me your test cases stay roughly the same.

  Are you using any middleware or other components for the Sanic ecosystem? Do you use cookies or sessions at all?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/57#issuecomment-400903871
issue_url: https://api.github.com/repos/simonw/datasette/issues/57
node_id: MDEyOklzc3VlQ29tbWVudDQwMDkwMzg3MQ==
created_at: 2018-06-28T04:01:38Z
updated_at: 2018-06-28T04:01:38Z
author_association: OWNER
body:
  Shipped to Docker Hub: https://hub.docker.com/r/datasetteproject/datasette/

  I did this manually the first time. I'll set Travis up to do this automatically in #329

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/pull/280#issuecomment-401003061
issue_url: https://api.github.com/repos/simonw/datasette/issues/280
node_id: MDEyOklzc3VlQ29tbWVudDQwMTAwMzA2MQ==
created_at: 2018-06-28T11:26:23Z
updated_at: 2018-06-28T11:26:23Z
author_association: OWNER
body:
  I pushed this to Docker Hub https://hub.docker.com/r/datasetteproject/datasette/ and added notes on how to use it to the documentation: http://datasette.readthedocs.io/en/latest/installation.html#using-docker

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-401310732
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDQwMTMxMDczMg==
created_at: 2018-06-29T10:05:04Z
updated_at: 2018-06-29T10:07:25Z
author_association: CONTRIBUTOR
body:
  @russs Different map projections can presumably be handled on the client side using a leaflet plugin to transform the geometry (eg [kartena/Proj4Leaflet](https://kartena.github.io/Proj4Leaflet/)) although the leaflet side would need to detect or be informed of the original projection?

  Another possibility would be to provide an easy way/guidance for users to create an FK'd table containing the WGS84 projection of a non-WGS84 geometry in the original/principle table? This could then as a proxy for serving GeoJSON to the leaflet map?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-401312981
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDQwMTMxMjk4MQ==
created_at: 2018-06-29T10:14:54Z
updated_at: 2018-06-29T10:14:54Z
author_association: CONTRIBUTOR
body:
  > @RusSs Different map projections can presumably be handled on the client side using a leaflet plugin to transform the geometry (eg kartena/Proj4Leaflet) although the leaflet side would need to detect or be informed of the original projection?

  Well, as @simonw mentioned, GeoJSON only supports WGS84, and GeoJSON (and/or TopoJSON) is the standard we probably want to aim for. On-the-fly reprojection in spatialite is not an issue anyway, and in general I think you want to be serving stuff to web maps in WGS84 or Web Mercator.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/332#issuecomment-401477622
issue_url: https://api.github.com/repos/simonw/datasette/issues/332
node_id: MDEyOklzc3VlQ29tbWVudDQwMTQ3NzYyMg==
created_at: 2018-06-29T21:23:17Z
updated_at: 2018-06-29T21:23:55Z
author_association: OWNER
body:
  https://docs.python.org/3/library/json.html#json.dump

  > **json.dump**(obj, fp, *, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, default=None, sort_keys=False, **kw)¶
  > If `allow_nan` is false (default: True), then it will be a ValueError to serialize out of range float values (nan, inf, -inf) in strict compliance of the JSON specification. If allow_nan is true, their JavaScript equivalents (NaN, Infinity, -Infinity) will be used.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/332#issuecomment-401478223
issue_url: https://api.github.com/repos/simonw/datasette/issues/332
node_id: MDEyOklzc3VlQ29tbWVudDQwMTQ3ODIyMw==
created_at: 2018-06-29T21:26:12Z
updated_at: 2018-06-29T21:26:19Z
author_association: OWNER
body:
  I'm not sure what the correct thing to do here is. I don't want to throw a `ValueError` when trying to render that data as JSON, but I also want to produce JSON that doesn't break when fetched by JavaScript.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/332#issuecomment-402243153
issue_url: https://api.github.com/repos/simonw/datasette/issues/332
node_id: MDEyOklzc3VlQ29tbWVudDQwMjI0MzE1Mw==
created_at: 2018-07-03T17:58:50Z
updated_at: 2018-07-12T16:10:39Z
author_association: OWNER
body:
  I think I'm going to return `null` in the JSON for infinity/nan values by default, but if you send `_nan=1` I will instead return invalid JSON with `Infinity` or `NaN` in it (since you have opted in to getting those and hence should be able to handle them).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/325#issuecomment-403263890
issue_url: https://api.github.com/repos/simonw/datasette/issues/325
node_id: MDEyOklzc3VlQ29tbWVudDQwMzI2Mzg5MA==
created_at: 2018-07-08T05:35:20Z
updated_at: 2018-07-09T17:28:27Z
author_association: OWNER
body:
  Fixed: https://v0-23-2.datasette.io/fixtures-e14e080/table%2Fwith%2Fslashes.csv / https://v0-23-2.datasette.io/fixtures-e14e080/table%2Fwith%2Fslashes.csv/3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/334#issuecomment-403672561
issue_url: https://api.github.com/repos/simonw/datasette/issues/334
node_id: MDEyOklzc3VlQ29tbWVudDQwMzY3MjU2MQ==
created_at: 2018-07-10T01:45:28Z
updated_at: 2018-07-10T01:45:28Z
author_association: OWNER
body:
  Tested with `datasette publish heroku fixtures.db --extra-options="--config sql_time_limit_ms:4000"`

  https://blooming-anchorage-31561.herokuapp.com/-/config

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/330#issuecomment-403855963
issue_url: https://api.github.com/repos/simonw/datasette/issues/330
node_id: MDEyOklzc3VlQ29tbWVudDQwMzg1NTk2Mw==
created_at: 2018-07-10T15:04:31Z
updated_at: 2018-07-10T15:04:31Z
author_association: OWNER
body:
  This relates to #276 - I'm definitely convinced now that displaying a giant `b'...'` blob on the page is not a useful default.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/331#issuecomment-403858949
issue_url: https://api.github.com/repos/simonw/datasette/issues/331
node_id: MDEyOklzc3VlQ29tbWVudDQwMzg1ODk0OQ==
created_at: 2018-07-10T15:12:53Z
updated_at: 2018-07-10T15:13:04Z
author_association: OWNER
body:
  ```
  $ datasette airports.sqlite 
  Serve! files=('airports.sqlite',) on port 8001
  Usage: datasette airports.sqlite [OPTIONS] [FILES]...

  Error: It looks like you're trying to load a SpatiaLite database without first loading the SpatiaLite module.

  Read more: https://datasette.readthedocs.io/en/latest/spatialite.html
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/335#issuecomment-403863927
issue_url: https://api.github.com/repos/simonw/datasette/issues/335
node_id: MDEyOklzc3VlQ29tbWVudDQwMzg2MzkyNw==
created_at: 2018-07-10T15:26:27Z
updated_at: 2018-07-10T15:29:54Z
author_association: OWNER
body:
  Here are some useful examples of other Python apps that have been packaged using the recipe described above: https://github.com/Homebrew/homebrew-core/search?utf8=%E2%9C%93&q=virtualenv_install_with_resources&type=

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/335#issuecomment-403865063
issue_url: https://api.github.com/repos/simonw/datasette/issues/335
node_id: MDEyOklzc3VlQ29tbWVudDQwMzg2NTA2Mw==
created_at: 2018-07-10T15:29:32Z
updated_at: 2018-07-10T15:29:32Z
author_association: OWNER
body:
  Huh... from https://docs.brew.sh/Acceptable-Formulae

  > We frown on authors submitting their own work unless it is very popular.

  Marking this one as "help wanted" :)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/335#issuecomment-403866099
issue_url: https://api.github.com/repos/simonw/datasette/issues/335
node_id: MDEyOklzc3VlQ29tbWVudDQwMzg2NjA5OQ==
created_at: 2018-07-10T15:32:14Z
updated_at: 2018-07-10T15:32:14Z
author_association: OWNER
body:
  I can host a custom tap without needing to get anything accepted into homebrew-core: https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap

  Since my principle goal here is ensuring an easy installation path for people who are familiar with `brew` but don't know how to use pip and Python 3 that could be a good option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/330#issuecomment-403868584
issue_url: https://api.github.com/repos/simonw/datasette/issues/330
node_id: MDEyOklzc3VlQ29tbWVudDQwMzg2ODU4NA==
created_at: 2018-07-10T15:39:12Z
updated_at: 2018-07-10T16:21:08Z
author_association: OWNER
body:
  I think this makes sense for the HTML view (not for JSON or CSV). It could be controlled be a new [config option](http://datasette.readthedocs.io/en/latest/config.html), `truncate_cells_html` - which is on by default but can be turned off.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/330#issuecomment-403906747
issue_url: https://api.github.com/repos/simonw/datasette/issues/330
node_id: MDEyOklzc3VlQ29tbWVudDQwMzkwNjc0Nw==
created_at: 2018-07-10T17:39:46Z
updated_at: 2018-07-10T17:39:46Z
author_association: OWNER
body:
  ```
  datasette publish now timezones.db --spatialite \
    --extra-options="--config truncate_cells_html:200" \
    --name=datasette-issue-330-demo \
    --branch=master
  ```
  https://datasette-issue-330-demo-sbelwxttfn.now.sh/timezones-3cb9f64/timezones

  ![2018-07-10 at 10 39 am](https://user-images.githubusercontent.com/9599/42527428-7eabc6c8-842d-11e8-91ac-5666dbc5872c.png)

  But https://datasette-issue-330-demo-sbelwxttfn.now.sh/timezones-3cb9f64/timezones/1 displays the full blob.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/87#issuecomment-403909671
issue_url: https://api.github.com/repos/simonw/datasette/issues/87
node_id: MDEyOklzc3VlQ29tbWVudDQwMzkwOTY3MQ==
created_at: 2018-07-10T17:49:12Z
updated_at: 2018-07-10T17:49:12Z
author_association: OWNER
body:
  This was fixed by https://github.com/simonw/datasette/commit/6a32684ebba89dfe882e1147b23aa8778479f5d8#diff-354f30a63fb0907d4ad57269548329e3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/140#issuecomment-403939399
issue_url: https://api.github.com/repos/simonw/datasette/issues/140
node_id: MDEyOklzc3VlQ29tbWVudDQwMzkzOTM5OQ==
created_at: 2018-07-10T19:30:17Z
updated_at: 2018-07-10T19:30:41Z
author_association: OWNER
body:
  Building this using Svelte would also produce a neat example of a plugin that uses Svelte: https://svelte.technology/guide - and if I like it I might part datasette-vega to it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-403959704
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDQwMzk1OTcwNA==
created_at: 2018-07-10T20:44:47Z
updated_at: 2018-07-10T20:44:47Z
author_association: OWNER
body:
  No cookies or sessions - no POST requests in fact, Datasette just cares about GET (path and querystring) and being able to return custom HTTP headers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/336#issuecomment-403996143
issue_url: https://api.github.com/repos/simonw/datasette/issues/336
node_id: MDEyOklzc3VlQ29tbWVudDQwMzk5NjE0Mw==
created_at: 2018-07-10T23:21:27Z
updated_at: 2018-07-10T23:21:27Z
author_association: OWNER
body:
  Easiest way to do this I think would be to make those help blocks separate files in the docs/ directory (publish-help.txt perhaps) and then include them with a sphinx directive: https://reinout.vanrees.org/weblog/2010/12/08/include-external-in-sphinx.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/336#issuecomment-404021890
issue_url: https://api.github.com/repos/simonw/datasette/issues/336
node_id: MDEyOklzc3VlQ29tbWVudDQwNDAyMTg5MA==
created_at: 2018-07-11T02:09:25Z
updated_at: 2018-07-11T02:09:25Z
author_association: OWNER
body:
  I decided against the unit tests, instead I have a new script called `./update-docs-help.sh` which I can run any time I want to refresh the included documentation: https://github.com/simonw/datasette/commit/aec3ae53237e43b0c268dbf9b58fa265ef38cfe1#diff-cb15a1e5a244bb82ad4afce67f252543

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/338#issuecomment-404209205
issue_url: https://api.github.com/repos/simonw/datasette/issues/338
node_id: MDEyOklzc3VlQ29tbWVudDQwNDIwOTIwNQ==
created_at: 2018-07-11T15:21:47Z
updated_at: 2018-07-11T15:21:47Z
author_association: OWNER
body:
  Oops, opened this in the wrong repo - moved it here: https://github.com/simonw/datasette-vega/issues/13

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/339#issuecomment-404338345
issue_url: https://api.github.com/repos/simonw/datasette/issues/339
node_id: MDEyOklzc3VlQ29tbWVudDQwNDMzODM0NQ==
created_at: 2018-07-11T23:09:24Z
updated_at: 2018-07-11T23:09:24Z
author_association: OWNER
body:
  It sounds like you're running into the Sanic default response timeout value of 60 seconds: https://github.com/channelcat/sanic/blob/master/docs/sanic/config.md#builtin-configuration-values

  For the moment you can over-ride that using an environment variable like this:

      SANIC_RESPONSE_TIMEOUT=6000 datasette fivethirtyeight.db -p 8008 --config sql_time_limit_ms:600000

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-404514973
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDQwNDUxNDk3Mw==
created_at: 2018-07-12T13:38:24Z
updated_at: 2018-07-12T13:38:24Z
author_association: NONE
body:
  Okay. I reckon the latest version should have all the kinds of components you'd need:

  Recently added ASGI components for Routing and Static Files support, as well as making few tweaks to make sure requests and responses are instantiated efficiently.

  Don't have any redirect-to-slash / redirect-to-non-slash stuff out of the box yet, which it looks like you might miss.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/332#issuecomment-404567587
issue_url: https://api.github.com/repos/simonw/datasette/issues/332
node_id: MDEyOklzc3VlQ29tbWVudDQwNDU2NzU4Nw==
created_at: 2018-07-12T16:15:29Z
updated_at: 2018-07-12T16:17:54Z
author_association: OWNER
body:
  Here's how plotly handled this issue: https://github.com/plotly/plotly.py/pull/203 - see also https://github.com/plotly/plotly.py/blob/213602df6c89b45ce2b811ed2591171c961408e7/plotly/utils.py#L137

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/332#issuecomment-404569003
issue_url: https://api.github.com/repos/simonw/datasette/issues/332
node_id: MDEyOklzc3VlQ29tbWVudDQwNDU2OTAwMw==
created_at: 2018-07-12T16:20:06Z
updated_at: 2018-07-12T16:20:06Z
author_association: OWNER
body:
  And here's how django-rest-framework did it: https://github.com/encode/django-rest-framework/pull/4918/files

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/332#issuecomment-404574598
issue_url: https://api.github.com/repos/simonw/datasette/issues/332
node_id: MDEyOklzc3VlQ29tbWVudDQwNDU3NDU5OA==
created_at: 2018-07-12T16:39:51Z
updated_at: 2018-07-12T16:39:51Z
author_association: OWNER
body:
  Since my data is all flat lists of values I don't think I need to customize the JSON encoder itself (no need to deal with nested values). I'll fix the data on its way into the encoder instead. This will also help if I decide to move to uJSON for better performance #48

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/327#issuecomment-404923318
issue_url: https://api.github.com/repos/simonw/datasette/issues/327
node_id: MDEyOklzc3VlQ29tbWVudDQwNDkyMzMxOA==
created_at: 2018-07-13T18:58:11Z
updated_at: 2018-07-13T18:58:11Z
author_association: OWNER
body:
  Relevant: https://code.fb.com/data-infrastructure/xars-a-more-efficient-open-source-system-for-self-contained-executables/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/342#issuecomment-404953877
issue_url: https://api.github.com/repos/simonw/datasette/issues/342
node_id: MDEyOklzc3VlQ29tbWVudDQwNDk1Mzg3Nw==
created_at: 2018-07-13T21:05:12Z
updated_at: 2018-07-13T21:05:12Z
author_association: OWNER
body:
  That's a good idea. We already do this for tables - e.g. on https://fivethirtyeight.datasettes.com/fivethirtyeight-ac35616/most-common-name%2Fsurnames - so having it as an option for canned queries definitely makes sense.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/342#issuecomment-404954202
issue_url: https://api.github.com/repos/simonw/datasette/issues/342
node_id: MDEyOklzc3VlQ29tbWVudDQwNDk1NDIwMg==
created_at: 2018-07-13T21:06:53Z
updated_at: 2018-07-13T21:07:13Z
author_association: OWNER
body:
  https://timezones-api.now.sh/-/metadata currently shows this:
  ```
  {
      "databases": {
          "timezones": {
              "license": "ODbL",
              "license_url": "http://opendatacommons.org/licenses/odbl/",
              "queries": {
                  "by_point": "select tzid\nfrom\n    timezones\nwhere\n    within(GeomFromText(\u0027POINT(\u0027 || :longitude || \u0027 \u0027 || :latitude || \u0027)\u0027), timezones.Geometry)\n    and rowid in (\n        SELECT pkid FROM idx_timezones_Geometry\n        where xmin \u003c :longitude\n        and xmax \u003e :longitude\n        and ymin \u003c :latitude\n        and ymax \u003e :latitude\n    )"
              },
              "source": "timezone-boundary-builder",
              "source_url": "https://github.com/evansiroky/timezone-boundary-builder",
              "tables": {
                  "timezones": {
                      "license": "ODbL",
                      "license_url": "http://opendatacommons.org/licenses/odbl/",
                      "sortable_columns": [
                          "tzid"
                      ],
                      "source": "timezone-boundary-builder",
                      "source_url": "https://github.com/evansiroky/timezone-boundary-builder"
                  }
              }
          }
      },
      "license": "ODbL",
      "license_url": "http://opendatacommons.org/licenses/odbl/",
      "source": "timezone-boundary-builder",
      "source_url": "https://github.com/evansiroky/timezone-boundary-builder",
      "title": "OpenStreetMap Time Zone Boundaries"
  }
  ```

  We could support the value part of the `"queries"` array optionally being a dictionary with the same set of metadata fields supported for a table, plus a new `"sql"` key to hold the SQL for the query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/342#issuecomment-404954672
issue_url: https://api.github.com/repos/simonw/datasette/issues/342
node_id: MDEyOklzc3VlQ29tbWVudDQwNDk1NDY3Mg==
created_at: 2018-07-13T21:09:01Z
updated_at: 2018-07-13T21:09:01Z
author_association: OWNER
body:
  So it would look like this:
  ```
  {
      "databases": {
          "timezones": {
              "license": "ODbL",
              "license_url": "http://opendatacommons.org/licenses/odbl/",
              "queries": {
                  "by_point": {
                      "title": "Timezones by point",
                      "description": "Find the timezone for a latitude/longitude point",
                      "sql": "select tzid\nfrom\n    timezones\nwhere\n    within(GeomFromText('POINT(' || :longitude || ' ' || :latitude || ')'), timezones.Geometry)\n    and rowid in (\n        SELECT pkid FROM idx_timezones_Geometry\n        where xmin < :longitude\n        and xmax > :longitude\n        and ymin < :latitude\n        and ymax > :latitude\n    )"
                  }
              }
          }
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/294#issuecomment-405026800
issue_url: https://api.github.com/repos/simonw/datasette/issues/294
node_id: MDEyOklzc3VlQ29tbWVudDQwNTAyNjgwMA==
created_at: 2018-07-14T14:24:31Z
updated_at: 2018-07-14T14:24:31Z
author_association: CONTRIBUTOR
body:
  I had a quick look at this in relation to #343 and I feel like it might be worth modelling the inspected table metadata internally as an object rather than a dict. (We'd still have to serialise it back to JSON.)

  There are a few places where we rely on the structure of this metadata dict for various reasons, including in templates (and potentially also in user templates). It would be nice to have a reasonably well defined API for accessing metadata internally so that it's clearer what we're breaking.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/342#issuecomment-405138460
issue_url: https://api.github.com/repos/simonw/datasette/issues/342
node_id: MDEyOklzc3VlQ29tbWVudDQwNTEzODQ2MA==
created_at: 2018-07-16T02:42:32Z
updated_at: 2018-07-16T02:42:32Z
author_association: OWNER
body:
  Demos:

  * https://latest.datasette.io/fixtures/neighborhood_search
  * https://timezones-api.now.sh/timezones/by_point

  Documentation: http://datasette.readthedocs.io/en/latest/sql_queries.html#canned-queries

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/332#issuecomment-405968983
issue_url: https://api.github.com/repos/simonw/datasette/issues/332
node_id: MDEyOklzc3VlQ29tbWVudDQwNTk2ODk4Mw==
created_at: 2018-07-18T15:18:57Z
updated_at: 2018-07-18T15:18:57Z
author_association: OWNER
body:
  Maybe argument should be `?_json_nan=1` since that makes it more explicitly obvious what is going on here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/308#issuecomment-405971920
issue_url: https://api.github.com/repos/simonw/datasette/issues/308
node_id: MDEyOklzc3VlQ29tbWVudDQwNTk3MTkyMA==
created_at: 2018-07-18T15:27:12Z
updated_at: 2018-07-18T15:27:12Z
author_association: OWNER
body:
  It looks like there are a few extra options we should support:

  https://devcenter.heroku.com/articles/heroku-cli-commands

  ```
    -t, --team=team            team to use
    --region=region            specify region for the app to run in
    --space=space              the private space to create the app in
  ```

  Since these differ from the options for Zeit Now I think this means splitting up `datasette publish now` and `datasette publish Heroku` into separate subcommands.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/333#issuecomment-405975025
issue_url: https://api.github.com/repos/simonw/datasette/issues/333
node_id: MDEyOklzc3VlQ29tbWVudDQwNTk3NTAyNQ==
created_at: 2018-07-18T15:36:11Z
updated_at: 2018-07-18T15:40:04Z
author_association: OWNER
body:
  A `force_https_api_urls` config option would work here - if set, Datasette will ignore the incoming protocol and always use https. The `datasette deploy now` command could then add that as an option passed to `datasette serve`.

  This is the pattern which is producing incorrect URLs on Zeit Now, because the Sanic `request.url` property is not being correctly set.

  https://github.com/simonw/datasette/blob/6e37f091edec35e2706197489f54fff5d890c63c/datasette/views/table.py#L653-L655

  Suggested help text:

  > Always use https:// for URLs output as part of Datasette API responses

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/333#issuecomment-405988035
issue_url: https://api.github.com/repos/simonw/datasette/issues/333
node_id: MDEyOklzc3VlQ29tbWVudDQwNTk4ODAzNQ==
created_at: 2018-07-18T16:12:35Z
updated_at: 2018-07-18T16:12:35Z
author_association: OWNER
body:
  I'll add a `absolute_url(request, path)` method on the base view class which knows to check the new config option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/333#issuecomment-407109113
issue_url: https://api.github.com/repos/simonw/datasette/issues/333
node_id: MDEyOklzc3VlQ29tbWVudDQwNzEwOTExMw==
created_at: 2018-07-23T15:59:02Z
updated_at: 2018-07-23T15:59:02Z
author_association: OWNER
body:
  I still need to modify `datasette publish now` to set this config option on the instances that it deploys.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/332#issuecomment-407262311
issue_url: https://api.github.com/repos/simonw/datasette/issues/332
node_id: MDEyOklzc3VlQ29tbWVudDQwNzI2MjMxMQ==
created_at: 2018-07-24T02:43:03Z
updated_at: 2018-07-24T02:43:03Z
author_association: OWNER
body:
  Actually SQLite doesn't handle NaN at all (it treats it as null) so I'm going to change this ticket to just deal with Infinity and -Infinity.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/332#issuecomment-407262561
issue_url: https://api.github.com/repos/simonw/datasette/issues/332
node_id: MDEyOklzc3VlQ29tbWVudDQwNzI2MjU2MQ==
created_at: 2018-07-24T02:44:39Z
updated_at: 2018-07-24T02:44:39Z
author_association: OWNER
body:
  According to https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg110573.html you can insert Infinity/-Infinity in raw SQL (as used by our fixtures) using 1e999 and -1e999.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/332#issuecomment-407267707
issue_url: https://api.github.com/repos/simonw/datasette/issues/332
node_id: MDEyOklzc3VlQ29tbWVudDQwNzI2NzcwNw==
created_at: 2018-07-24T03:20:08Z
updated_at: 2018-07-24T03:20:08Z
author_association: OWNER
body:
  Demo:
  * https://700d83d.datasette.io/fixtures-dcc1dbf/infinity.json - Infinity converted to Null
  * https://700d83d.datasette.io/fixtures-dcc1dbf/infinity.json?_json_infinity=on - invalid JSON containing `Infinity` and `-Infinity`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/320#issuecomment-407269243
issue_url: https://api.github.com/repos/simonw/datasette/issues/320
node_id: MDEyOklzc3VlQ29tbWVudDQwNzI2OTI0Mw==
created_at: 2018-07-24T03:30:32Z
updated_at: 2018-07-24T03:30:32Z
author_association: OWNER
body:
  * No primary key => no "object" option: https://latest.datasette.io/fixtures-dcc1dbf/no_primary_key
  * Has a primary key => show "object" option: https://latest.datasette.io/fixtures-dcc1dbf/complex_foreign_keys
  * Has a next page => has "stream all rows" option: https://latest.datasette.io/fixtures-dcc1dbf/no_primary_key
  * Has foreign key references = show default-checked "expand labels" option: https://latest.datasette.io/fixtures-dcc1dbf/complex_foreign_keys
  * Does not have a next page => do not show "stream all rows" option: https://latest.datasette.io/fixtures-dcc1dbf/complex_foreign_keys

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/298#issuecomment-407274059
issue_url: https://api.github.com/repos/simonw/datasette/issues/298
node_id: MDEyOklzc3VlQ29tbWVudDQwNzI3NDA1OQ==
created_at: 2018-07-24T04:03:05Z
updated_at: 2018-07-24T04:03:05Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures-dcc1dbf?sql=select+%28%27https%3A%2F%2Ftwitter.com%2F%27+%7C%7C+%27simonw%27%29+as+user_url%3B

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/329#issuecomment-407275996
issue_url: https://api.github.com/repos/simonw/datasette/issues/329
node_id: MDEyOklzc3VlQ29tbWVudDQwNzI3NTk5Ng==
created_at: 2018-07-24T04:18:28Z
updated_at: 2018-07-24T04:18:28Z
author_association: OWNER
body:
  Hopefully this will do the trick: https://github.com/simonw/datasette/commit/2bdab66772dca51b0c729b4e1063610cb2edd890

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/329#issuecomment-407280689
issue_url: https://api.github.com/repos/simonw/datasette/issues/329
node_id: MDEyOklzc3VlQ29tbWVudDQwNzI4MDY4OQ==
created_at: 2018-07-24T04:52:58Z
updated_at: 2018-07-24T04:52:58Z
author_association: OWNER
body:
  It almost worked... but I had to fix the `docker login` command: https://github.com/simonw/datasette/commit/3a46d5e3c4278e74c3694f36995ea134bff800bc

  Hopefully the next release will be published correctly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/336#issuecomment-407450815
issue_url: https://api.github.com/repos/simonw/datasette/issues/336
node_id: MDEyOklzc3VlQ29tbWVudDQwNzQ1MDgxNQ==
created_at: 2018-07-24T15:35:03Z
updated_at: 2018-07-24T15:35:03Z
author_association: OWNER
body:
  Actually I do like the idea of a unit test that reminds me if I've forgotten to update the included files.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/349#issuecomment-407980716
issue_url: https://api.github.com/repos/simonw/datasette/issues/349
node_id: MDEyOklzc3VlQ29tbWVudDQwNzk4MDcxNg==
created_at: 2018-07-26T05:28:54Z
updated_at: 2018-07-26T05:28:54Z
author_association: OWNER
body:
  Documentation here: http://datasette.readthedocs.io/en/latest/plugins.html#publish-subcommand-publish

  The best way to write a new publish plugin is to check out how the Heroku and Now default plugins are implemented: https://github.com/simonw/datasette/tree/master/datasette/publish

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-408093480
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDQwODA5MzQ4MA==
created_at: 2018-07-26T13:15:55Z
updated_at: 2018-07-26T13:46:40Z
author_association: OWNER
body:
  I'm now hacking around with an initial version of this in the [starlette branch](https://github.com/simonw/datasette/tree/starlette).

  Here's my work in progress, deployed using `datasette publish now fixtures.db -n datasette-starlette-demo --branch=starlette --extra-options="--asgi"`

  https://datasette-starlette-demo.now.sh/

  Lots more work to do - the CSS isn't being served correctly for example, it's showing this error when I hit `/-/static/app.css`:
  ```
  INFO: 127.0.0.1 - "GET /-/static/app.css HTTP/1.1" 200
  ERROR: Exception in ASGI application
  Traceback (most recent call last):
    File "/Users/simonw/Dropbox/Development/datasette/venv/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py", line 363, in run_asgi
      result = await asgi(self.receive, self.send)
    File "/Users/simonw/Dropbox/Development/datasette/venv/lib/python3.6/site-packages/starlette/staticfiles.py", line 91, in __call__
      await response(receive, send)
    File "/Users/simonw/Dropbox/Development/datasette/venv/lib/python3.6/site-packages/starlette/response.py", line 180, in __call__
      {"type": "http.response.body", "body": chunk, "more_body": False}
    File "/Users/simonw/Dropbox/Development/datasette/venv/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py", line 483, in send
      raise RuntimeError("Response content shorter than Content-Length")
  RuntimeError: Response content shorter than Content-Length
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-408105251
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDQwODEwNTI1MQ==
created_at: 2018-07-26T13:54:06Z
updated_at: 2018-07-26T13:54:06Z
author_association: OWNER
body:
  Tom shipped my fix for that bug already, so https://datasette-starlette-demo.now.sh/ is now serving CSS!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/355#issuecomment-409087871
issue_url: https://api.github.com/repos/simonw/datasette/issues/355
node_id: MDEyOklzc3VlQ29tbWVudDQwOTA4Nzg3MQ==
created_at: 2018-07-31T04:06:22Z
updated_at: 2018-07-31T04:06:22Z
author_association: OWNER
body:
  I started playing with this in the `m2m` branch - work so far: https://github.com/simonw/datasette/compare/295d005ca48747faf046ed30c3c61e7563c61ed2...af4ce463e7518f9d7828b846efd5b528a1905eca

  Here's a demo: https://datasette-m2m-work-in-progress.now.sh/russian-ads-e8e09e2/ads?_m2m_ad_targets__target_id=ec3ac&_m2m_ad_targets__target_id=e128e

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/356#issuecomment-409088967
issue_url: https://api.github.com/repos/simonw/datasette/issues/356
node_id: MDEyOklzc3VlQ29tbWVudDQwOTA4ODk2Nw==
created_at: 2018-07-31T04:14:44Z
updated_at: 2018-07-31T04:14:44Z
author_association: OWNER
body:
  Here's the query I'm playing with for facet counts:

  https://datasette-m2m-work-in-progress.now.sh/russian-ads-e8e09e2?sql=select+target_id%2C+count%28*%29+as+n+from+ad_targets%0D%0Awhere%0D%0A++target_id+not+in+%28%22ec3ac%22%2C+%22e128e%22%29%0D%0A++and+ad_id+in+%28select+ad_id+from+ad_targets+where+target_id+%3D+%22ec3ac%22%29%0D%0A++and+ad_id+in+%28select+ad_id+from+ad_targets+where+target_id+%3D+%22e128e%22%29%0D%0Agroup+by+target_id+order+by+n+desc%3B

  ```
  select target_id, count(*) as n from ad_targets
  where
    target_id not in ("ec3ac", "e128e")
    and ad_id in (select ad_id from ad_targets where target_id = "ec3ac")
    and ad_id in (select ad_id from ad_targets where target_id = "e128e")
  group by target_id order by n desc;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/352#issuecomment-409715112
issue_url: https://api.github.com/repos/simonw/datasette/issues/352
node_id: MDEyOklzc3VlQ29tbWVudDQwOTcxNTExMg==
created_at: 2018-08-01T20:41:04Z
updated_at: 2018-08-01T20:41:04Z
author_association: OWNER
body:
  The hook is currently only used on the custom SQL results page - it needs to run on table/view pages as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/352#issuecomment-410485995
issue_url: https://api.github.com/repos/simonw/datasette/issues/352
node_id: MDEyOklzc3VlQ29tbWVudDQxMDQ4NTk5NQ==
created_at: 2018-08-05T00:16:21Z
updated_at: 2018-08-05T00:16:21Z
author_association: OWNER
body:
  First plugin using this hook: https://github.com/simonw/datasette-json-html

  Hook documentation: http://datasette.readthedocs.io/en/latest/plugins.html#render-cell-value

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/352#issuecomment-410580202
issue_url: https://api.github.com/repos/simonw/datasette/issues/352
node_id: MDEyOklzc3VlQ29tbWVudDQxMDU4MDIwMg==
created_at: 2018-08-06T03:39:40Z
updated_at: 2018-08-06T03:39:40Z
author_association: OWNER
body:
  I used `datasette-json-html` to build this: https://russian-ira-facebook-ads-datasette-whmbonekoj.now.sh/russian-ads-919cbfd/display_ads

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/174#issuecomment-412290986
issue_url: https://api.github.com/repos/simonw/datasette/issues/174
node_id: MDEyOklzc3VlQ29tbWVudDQxMjI5MDk4Ng==
created_at: 2018-08-11T17:46:51Z
updated_at: 2018-08-11T17:46:51Z
author_association: OWNER
body:
  This was fixed in https://github.com/simonw/datasette/commit/89d9fbb91bfc0dd9091b34dbf3cf540ab849cc44

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/188#issuecomment-412291327
issue_url: https://api.github.com/repos/simonw/datasette/issues/188
node_id: MDEyOklzc3VlQ29tbWVudDQxMjI5MTMyNw==
created_at: 2018-08-11T17:53:17Z
updated_at: 2018-08-11T17:53:17Z
author_association: OWNER
body:
  Potential problem: the existing `metadata.json` format looks like this:

  ```
  {
      "title": "Custom title for your index page",
      "description": "Some description text can go here",
      "license": "ODbL",
      "license_url": "https://opendatacommons.org/licenses/odbl/",
      "databases": {
          "database1": {
              "source": "Alternative source",
              "source_url": "http://example.com/",
              "tables": {
                  "example_table": {
                      "description_html": "Custom <em>table</em> description",
                      "license": "CC BY 3.0 US",
                      "license_url": "https://creativecommons.org/licenses/by/3.0/us/"
                  }
              }
          }
      }
  }
  ```

  This doesn't make sense for metadata that is bundled with a specific database - there's no point in having the `databases` key, instead the content of `database1` in the above example should be at the top level. This also means that if you rename the `*.db` file you won't have to edit its metadata at the same time.

  Calling such an embedded file `metadata.json` when the shape is different could be confusing. Maybe call it `database-metadata.json` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/231#issuecomment-412291395
issue_url: https://api.github.com/repos/simonw/datasette/issues/231
node_id: MDEyOklzc3VlQ29tbWVudDQxMjI5MTM5NQ==
created_at: 2018-08-11T17:54:41Z
updated_at: 2018-08-11T17:54:41Z
author_association: OWNER
body:
  I'm going to separate the issue of enabling and disabling plugins from the existence of the `plugins` key. The format will simply be:

  ```
  {
      "plugins": {
          "name-of-plugin": {
              ... any structures you like go here, defined by the plugin ...
          }
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/238#issuecomment-412291437
issue_url: https://api.github.com/repos/simonw/datasette/issues/238
node_id: MDEyOklzc3VlQ29tbWVudDQxMjI5MTQzNw==
created_at: 2018-08-11T17:55:26Z
updated_at: 2018-08-11T18:02:48Z
author_association: OWNER
body:
  On further thought, I'd much rather implement this using some kind of metadata plugin hook - see #357

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/185#issuecomment-412299013
issue_url: https://api.github.com/repos/simonw/datasette/issues/185
node_id: MDEyOklzc3VlQ29tbWVudDQxMjI5OTAxMw==
created_at: 2018-08-11T20:14:54Z
updated_at: 2018-08-11T20:14:54Z
author_association: OWNER
body:
  I've been worrying about how this one relates to #260 - I'd like to validate metadata (to help protect against people e.g. misspelling `license_url` and then being confused when their license isn't displayed properly), but this issue requests the ability to add arbitrary additional keys to the metadata structure.

  I think the solution is to introduce a metadata key called `extra_metadata_keys` which allows you to specifically list the extra keys that you want to enable. Something like this:

  ```
  {
      "title": "My title",
      "source": "Source",
      "source_url": "https://www.example.com/",
      "release_date": "2018-04-01",
      "extra_metadata_keys": ["release_date"]
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/359#issuecomment-412356537
issue_url: https://api.github.com/repos/simonw/datasette/issues/359
node_id: MDEyOklzc3VlQ29tbWVudDQxMjM1NjUzNw==
created_at: 2018-08-12T17:01:21Z
updated_at: 2018-08-12T17:01:39Z
author_association: OWNER
body:
  Example table: https://latest-code.datasette.io/code/definitions

  Here's a query that does facet counting against that column:

  https://latest-code.datasette.io/code-a26fa3c?sql=select+count%28*%29+as+n%2C+j.value+from+definitions+join+json_each%28params%29+j+group+by+j.value+order+by+n+desc%3B

  ```
  select count(*) as n, j.value
  from definitions join json_each(params) j
  group by j.value order by n desc;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/359#issuecomment-412356746
issue_url: https://api.github.com/repos/simonw/datasette/issues/359
node_id: MDEyOklzc3VlQ29tbWVudDQxMjM1Njc0Ng==
created_at: 2018-08-12T17:05:00Z
updated_at: 2018-08-12T17:05:00Z
author_association: OWNER
body:
  And here's the query for pulling back every record tagged with a specific tag:

  https://latest-code.datasette.io/code-a26fa3c?sql=select+*+from+definitions+where+rowid+in+%28%0D%0A++select+definitions.rowid%0D%0A++from+definitions+join+json_each%28params%29+j%0D%0A++where+j.value+%3D+%3Atag%0D%0A%29&tag=filename

  ```
  select * from definitions where rowid in (
    select definitions.rowid
    from definitions join json_each(params) j
    where j.value = :tag
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/359#issuecomment-412357691
issue_url: https://api.github.com/repos/simonw/datasette/issues/359
node_id: MDEyOklzc3VlQ29tbWVudDQxMjM1NzY5MQ==
created_at: 2018-08-12T17:17:29Z
updated_at: 2018-08-12T17:17:29Z
author_association: OWNER
body:
  Note that there doesn't seem to be a way to use indexes (even [indexes on expressions](https://www.sqlite.org/expridx.html)) to speed these up, so this will only ever be effective on smaller data sets, probably in the 10,000-100,000 range. Datasette is often used with smaller data sets so this is still worth pursuing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/360#issuecomment-413387424
issue_url: https://api.github.com/repos/simonw/datasette/issues/360
node_id: MDEyOklzc3VlQ29tbWVudDQxMzM4NzQyNA==
created_at: 2018-08-16T00:57:25Z
updated_at: 2018-08-16T00:57:25Z
author_association: OWNER
body:
  I deployed a working demo of this here: https://pysqlite3-datasette.now.sh

  I used this command to deploy it:

      datasette publish now \
          fixtures.db fivethirtyeight.db \
          --branch=pysqlite3 \
          --install=https://github.com/karlb/pysqlite3/archive/master.zip \
          -n pysqlite3-datasette

  https://pysqlite3-datasette.now.sh/-/versions confirms version of SQLite is `3.25.0`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/360#issuecomment-413396812
issue_url: https://api.github.com/repos/simonw/datasette/issues/360
node_id: MDEyOklzc3VlQ29tbWVudDQxMzM5NjgxMg==
created_at: 2018-08-16T01:50:42Z
updated_at: 2018-08-16T01:50:42Z
author_association: OWNER
body:
  Now that this has merged into master the command for deploying it can use `--branch=master` instead:

      datasette publish now \
          fixtures.db fivethirtyeight.db \
          --branch=master \
          --install=https://github.com/karlb/pysqlite3/archive/master.zip \
          -n pysqlite3-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/267#issuecomment-414860009
issue_url: https://api.github.com/repos/simonw/datasette/issues/267
node_id: MDEyOklzc3VlQ29tbWVudDQxNDg2MDAwOQ==
created_at: 2018-08-21T23:57:51Z
updated_at: 2018-08-21T23:57:51Z
author_association: NONE
body:
  Looks to me like hashing, redirects and caching were documented as part of https://github.com/simonw/datasette/commit/788a542d3c739da5207db7d1fb91789603cdd336#diff-3021b0e065dce289c34c3b49b3952a07 - so perhaps this can be closed? :tada:

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/362#issuecomment-416727898
issue_url: https://api.github.com/repos/simonw/datasette/issues/362
node_id: MDEyOklzc3VlQ29tbWVudDQxNjcyNzg5OA==
created_at: 2018-08-28T20:24:00Z
updated_at: 2018-08-28T20:24:00Z
author_association: OWNER
body:
  Are you talking about these filters here?

  ![2018-08-28 at 9 22 pm](https://user-images.githubusercontent.com/9599/44748784-8688cb00-ab08-11e8-8baf-ace2e04e181f.png)

  I haven't thought much about how those could be made more usable - right now they basically expose all available options, but customizing them for particular use-cases is certainly an interesting potential space.

  Could you sketch out a bit more about how your ideal interface here would work?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/308#issuecomment-418106781
issue_url: https://api.github.com/repos/simonw/datasette/issues/308
node_id: MDEyOklzc3VlQ29tbWVudDQxODEwNjc4MQ==
created_at: 2018-09-03T12:53:21Z
updated_at: 2018-09-03T12:53:21Z
author_association: OWNER
body:
  Now that I've split the heroku command out into a separate default plugin this is a much easier thing to add: https://github.com/simonw/datasette/blob/master/datasette/publish/heroku.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-418695115
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDQxODY5NTExNQ==
created_at: 2018-09-05T11:21:25Z
updated_at: 2018-09-05T11:21:25Z
author_association: NONE
body:
  Some notes:

  * Starlette just got a bump to 0.3.0 - there's some renamings in there. It's got enough functionality now that you can treat it either as a framework or as a toolkit. Either way the component design is all just *here's an ASGI app* all the way through.
  * Uvicorn got a bump to 0.3.3 - Removed some cyclical references that were causing garbage collection to impact performance. Ought to be a decent speed bump.
  * Wrt. passing config - Either use a single envvar that points to a config, or use multiple envvars for the config. Uvicorn could get a flag to read a `.env` file, but I don't see ASGI itself having a specific interface there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/329#issuecomment-422821483
issue_url: https://api.github.com/repos/simonw/datasette/issues/329
node_id: MDEyOklzc3VlQ29tbWVudDQyMjgyMTQ4Mw==
created_at: 2018-09-19T14:17:42Z
updated_at: 2018-09-19T14:17:42Z
author_association: CONTRIBUTOR
body:
  I'm using the docker image (0.23.2) and notice some differences/bugs between the docs and the published version with canned queries.  (submitted a tiny doc fix also)

  I was able to build the docker container locally using `master` and I'm using that for now.
  Would it be possible to manually push 0.24 to DockerHub until the TravisCI stuff is fixed?

  I would like to run this in our Kubernetes cluster but don't want to publish a version in our internal registry if I don't have to.
  Thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/329#issuecomment-422903031
issue_url: https://api.github.com/repos/simonw/datasette/issues/329
node_id: MDEyOklzc3VlQ29tbWVudDQyMjkwMzAzMQ==
created_at: 2018-09-19T18:07:09Z
updated_at: 2018-09-19T18:07:09Z
author_association: OWNER
body:
  The new 0.25 release has been successfully pushed to Docker Hub! https://hub.docker.com/r/datasetteproject/datasette/tags/

  One catch: it looks like it didn't update the "latest" tag to point at it. Looking into that now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/329#issuecomment-422908130
issue_url: https://api.github.com/repos/simonw/datasette/issues/329
node_id: MDEyOklzc3VlQ29tbWVudDQyMjkwODEzMA==
created_at: 2018-09-19T18:23:02Z
updated_at: 2018-09-19T18:23:02Z
author_association: OWNER
body:
  I fixed that by running the following on my laptop:

      $ docker pull datasetteproject/datasette:0.25
      $ docker tag datasetteproject/datasette:0.25 datasetteproject/datasette:latest
      $ docker push datasetteproject/datasette

  The `latest` tag now points to the most recent release.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/329#issuecomment-422915450
issue_url: https://api.github.com/repos/simonw/datasette/issues/329
node_id: MDEyOklzc3VlQ29tbWVudDQyMjkxNTQ1MA==
created_at: 2018-09-19T18:45:02Z
updated_at: 2018-09-20T10:50:50Z
author_association: CONTRIBUTOR
body:
  That works for me.  Was able to pull the public image and no errors on my canned query.  (~although a small rendering bug.  I'll create an issue and if I have time today, a PR to fix~  this turned out to be my error.)
  Thanks for the quick response!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/292#issuecomment-423543060
issue_url: https://api.github.com/repos/simonw/datasette/issues/292
node_id: MDEyOklzc3VlQ29tbWVudDQyMzU0MzA2MA==
created_at: 2018-09-21T14:06:31Z
updated_at: 2018-09-21T14:09:06Z
author_association: OWNER
body:
  I keep on finding new reasons that I want this.

  The latest is that I'm playing with the more advanced features of FTS5 - in particular the highlight() function and the ability to sort by rank.

  The problem is... in order to do this, I need to properly join against the `_fts` table. Here's an example query:

      select
        highlight(events_fts, 0, '<b>', '</b>'),
        events_fts.rank,
        events.*
      from events
        join events_fts on events.rowid = events_fts.rowid
      where events_fts match :search 
      order by rank

  Note that this is a different query from the usual FTS one (which does `where rowid in (select rowid from events_fts...)`) because I need the rank column somewhere I can sort against.

  I'd like to be able to use this on the table view page so I can get faceting etc for free, but this is a completely different query from the default. Maybe I need a way to customize the entire query? That feels weird though - why am I not using a view in that case?

  Answer: because views can't accept `:search` style parameters. I could use a canned query, but canned queries don't get faceting etc.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/328#issuecomment-427261369
issue_url: https://api.github.com/repos/simonw/datasette/issues/328
node_id: MDEyOklzc3VlQ29tbWVudDQyNzI2MTM2OQ==
created_at: 2018-10-05T06:37:06Z
updated_at: 2018-10-05T06:37:06Z
author_association: NONE
body:
  ```
  ~ $ docker pull datasetteproject/datasette
  ~ $ docker run -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
  Usage: datasette -p [OPTIONS] [FILES]...

  Error: Invalid value for "files": Path "/mnt/fixtures.db" does not exist.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/187#issuecomment-427943710
issue_url: https://api.github.com/repos/simonw/datasette/issues/187
node_id: MDEyOklzc3VlQ29tbWVudDQyNzk0MzcxMA==
created_at: 2018-10-08T18:58:05Z
updated_at: 2018-10-08T18:58:05Z
author_association: NONE
body:
  I have same error:
  ```
  Collecting uvloop
    Using cached https://files.pythonhosted.org/packages/5c/37/6daa39aac42b2deda6ee77f408bec0419b600e27b89b374b0d440af32b10/uvloop-0.11.2.tar.gz
      Complete output from command python setup.py egg_info:
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "C:\Users\sageev\AppData\Local\Temp\pip-install-bq64l8jy\uvloop\setup.py", line 15, in <module>
          raise RuntimeError('uvloop does not support Windows at the moment')
      RuntimeError: uvloop does not support Windows at the moment
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/366#issuecomment-429737929
issue_url: https://api.github.com/repos/simonw/datasette/issues/366
node_id: MDEyOklzc3VlQ29tbWVudDQyOTczNzkyOQ==
created_at: 2018-10-15T07:32:57Z
updated_at: 2018-10-15T07:32:57Z
author_association: CONTRIBUTOR
body:
  Very hacky solution is to write now.json file forcing the usage of v1 of Zeit cloud, see https://github.com/slygent/datasette/commit/3ab824793ec6534b6dd87078aa46b11c4fa78ea3

  This does work, at least.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/176#issuecomment-431867885
issue_url: https://api.github.com/repos/simonw/datasette/issues/176
node_id: MDEyOklzc3VlQ29tbWVudDQzMTg2Nzg4NQ==
created_at: 2018-10-22T15:24:57Z
updated_at: 2018-10-22T15:24:57Z
author_association: NONE
body:
  I'd like this as well. It would let me access Datasette-driven projects from GatsbyJS the same way I can access Postgres DBs via Hasura. While I don't see SQLite replacing Postgres for the 50m row datasets I sometimes have to work with, there's a whole class of smaller datasets that are great with Datasette but currently would find another option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/366#issuecomment-433680598
issue_url: https://api.github.com/repos/simonw/datasette/issues/366
node_id: MDEyOklzc3VlQ29tbWVudDQzMzY4MDU5OA==
created_at: 2018-10-28T06:38:43Z
updated_at: 2018-10-28T06:38:43Z
author_association: OWNER
body:
  I've just started running into this as well. Looks like I'll have to anchor to v1 for the moment - I'm hoping the discussion on https://github.com/zeit/now-cli/issues/1523 encourages an increase in this limit policy :/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/371#issuecomment-435767775
issue_url: https://api.github.com/repos/simonw/datasette/issues/371
node_id: MDEyOklzc3VlQ29tbWVudDQzNTc2Nzc3NQ==
created_at: 2018-11-05T06:27:33Z
updated_at: 2018-11-05T06:27:33Z
author_association: OWNER
body:
  This would be fantastic - that tutorial looks like many of the details needed for this.

  Do you know if Digital Ocean have the ability to provision URLs for a droplet without you needing to buy your own domain name? Heroku have https://example.herokuapp.com/ and Zeit have https://blah.now.sh/ - does Digital Ocean have an equivalent?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/369#issuecomment-435767827
issue_url: https://api.github.com/repos/simonw/datasette/issues/369
node_id: MDEyOklzc3VlQ29tbWVudDQzNTc2NzgyNw==
created_at: 2018-11-05T06:27:55Z
updated_at: 2018-11-05T06:28:48Z
author_association: OWNER
body:
  This is a good idea. Basically a version of this bug but on the custom SQL query page:

  ![2018-11-04 at 10 28 pm](https://user-images.githubusercontent.com/9599/47981499-fd9a8c80-e080-11e8-9c59-00e626d3aa4c.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/369#issuecomment-435768450
issue_url: https://api.github.com/repos/simonw/datasette/issues/369
node_id: MDEyOklzc3VlQ29tbWVudDQzNTc2ODQ1MA==
created_at: 2018-11-05T06:31:59Z
updated_at: 2018-11-05T06:31:59Z
author_association: CONTRIBUTOR
body:
  That would be ideal, but you know better than me whether the CSV streaming trick works for custom SQL queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/329#issuecomment-435772031
issue_url: https://api.github.com/repos/simonw/datasette/issues/329
node_id: MDEyOklzc3VlQ29tbWVudDQzNTc3MjAzMQ==
created_at: 2018-11-05T06:53:28Z
updated_at: 2018-11-05T06:54:10Z
author_association: OWNER
body:
  This works now! The `0.25.1` release was the first release which successfully pushed to Docker Hub: https://hub.docker.com/r/datasetteproject/datasette/tags/

  ![2018-11-04 at 10 53 pm](https://user-images.githubusercontent.com/9599/47982395-70593700-e084-11e8-8870-9100677c2bde.png)

  Here's the log from the successful Travis release job: https://travis-ci.org/simonw/datasette/jobs/450714602

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/371#issuecomment-435862009
issue_url: https://api.github.com/repos/simonw/datasette/issues/371
node_id: MDEyOklzc3VlQ29tbWVudDQzNTg2MjAwOQ==
created_at: 2018-11-05T12:48:35Z
updated_at: 2018-11-05T12:48:35Z
author_association: CONTRIBUTOR
body:
  I think you need to register a domain name you own separately in order to get a non-IP address address? https://www.digitalocean.com/docs/networking/dns/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/370#issuecomment-435974786
issue_url: https://api.github.com/repos/simonw/datasette/issues/370
node_id: MDEyOklzc3VlQ29tbWVudDQzNTk3NDc4Ng==
created_at: 2018-11-05T18:06:56Z
updated_at: 2018-11-05T18:06:56Z
author_association: OWNER
body:
  I've been thinking a bit about ways of using Jupyter Notebook more effectively with Datasette (thinks like a `publish_dataframes(df1, df2, df3)` function which publishes some Pandas dataframes and returns you a URL to a new hosted Datasette instance) but you're right, Jupyter Lab is potentially a much more interesting fit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/374#issuecomment-435976262
issue_url: https://api.github.com/repos/simonw/datasette/issues/374
node_id: MDEyOklzc3VlQ29tbWVudDQzNTk3NjI2Mg==
created_at: 2018-11-05T18:11:10Z
updated_at: 2018-11-05T18:11:10Z
author_association: OWNER
body:
  I think there is a useful way forward here though: the image size may be limited to 100MB, but once the instance launches it gets access to a filesystem with a lot more space than that (possibly as much as 15GB given my initial poking around).

  So... one potential solution here is to teach Datasette to launch from a smaller image and then download a larger SQLite file from a known URL as part of its initial startup.

  Combined with the ability to get Now to always run at least one copy of an instance this could allow Datasette to host much larger SQLite databases on that platform while playing nicely with the Zeit v2 platform.

  See also https://github.com/zeit/now-cli/issues/1523

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/370#issuecomment-436037692
issue_url: https://api.github.com/repos/simonw/datasette/issues/370
node_id: MDEyOklzc3VlQ29tbWVudDQzNjAzNzY5Mg==
created_at: 2018-11-05T21:15:47Z
updated_at: 2018-11-05T21:18:37Z
author_association: CONTRIBUTOR
body:
  In terms of integration with `pandas`, I was pondering two different ways `datasette`/`csvs_to_sqlite` integration may work:

  - like [`pandasql`](https://github.com/yhat/pandasql), to provide a SQL query layer either by a direct connection to the sqlite db or via `datasette` API;
  - as an improvement of `pandas.to_sql()`, which is a bit ropey (e.g. `pandas.to_sql_from_csvs()`, routing the dataframe to sqlite via `csvs_tosqlite` rather than the dodgy mapping that `pandas` supports).

  The `pandas.publish_*` idea could be quite interesting though... Would it be useful/fruitful to think about `publish_` as a complement to [`pandas.to_`](https://pandas.pydata.org/pandas-docs/stable/api.html#id12)?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/370#issuecomment-436042445
issue_url: https://api.github.com/repos/simonw/datasette/issues/370
node_id: MDEyOklzc3VlQ29tbWVudDQzNjA0MjQ0NQ==
created_at: 2018-11-05T21:30:42Z
updated_at: 2018-11-05T21:31:48Z
author_association: CONTRIBUTOR
body:
  Another route would be something like creating a `datasette` IPython magic for notebooks to take a dataframe and easily render it as a `datasette`. You'd need to run the app in the background rather than block execution in the notebook. Related to that, or to publishing a dataframe in notebook cell for use in other cells in a non-blocking way, there may be cribs in something like https://github.com/micahscopes/nbmultitask .

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/227#issuecomment-439194286
issue_url: https://api.github.com/repos/simonw/datasette/issues/227
node_id: MDEyOklzc3VlQ29tbWVudDQzOTE5NDI4Ng==
created_at: 2018-11-15T21:20:37Z
updated_at: 2018-11-15T21:20:37Z
author_association: NONE
body:
  I'm diving back into https://salaries.news.baltimoresun.com and what I really want is the ability to inject the request into my context.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/374#issuecomment-439762759
issue_url: https://api.github.com/repos/simonw/datasette/issues/374
node_id: MDEyOklzc3VlQ29tbWVudDQzOTc2Mjc1OQ==
created_at: 2018-11-19T03:41:36Z
updated_at: 2018-11-19T03:41:36Z
author_association: OWNER
body:
  It turned out Zeit didn't end up shipping the new 100MB-limit Docker-based Zeit 2.0 after all - they ended up going in a completely different direction, towards lambdas instead (which don't really fit the Datasette model): https://zeit.co/blog/now-2

  But... as far as I can tell they have introduced the 100MB image size for all free Zeit accounts ever against their 1.0 platform. So we still need to solve this, or free Zeit users won't be able to use `datasette publish now` even while 1.0 is still available.

  I made some notes on this here: https://simonwillison.net/2018/Nov/19/smaller-python-docker-images/

  I've got it working for the Datasette Publish webapp, but I still need to fix `datasette publish now` to create much smaller patterns.

  I know how to do this for regular datasette, but I haven't yet figured out an Alpine Linux pattern for spatialite extras:

  https://github.com/simonw/datasette/blob/5e3a432a0caa23837fa58134f69e2f82e4f632a6/datasette/utils.py#L287-L300

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/374#issuecomment-439763196
issue_url: https://api.github.com/repos/simonw/datasette/issues/374
node_id: MDEyOklzc3VlQ29tbWVudDQzOTc2MzE5Ng==
created_at: 2018-11-19T03:45:13Z
updated_at: 2018-11-19T03:45:13Z
author_association: OWNER
body:
  This looks like it might be a recipe for spatialite Python on Alpine Linux: https://github.com/bentrm/geopython/blob/8e52062d9545f4b7c1f04a3516354a5a9155e31f/Dockerfile

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/374#issuecomment-439763268
issue_url: https://api.github.com/repos/simonw/datasette/issues/374
node_id: MDEyOklzc3VlQ29tbWVudDQzOTc2MzI2OA==
created_at: 2018-11-19T03:45:44Z
updated_at: 2018-11-19T03:45:44Z
author_association: OWNER
body:
  Another example that might be useful: https://github.com/poc-flask/alpine/blob/8e9f48a2351e106347dab36d08cf21dee865993e/Dockerfile

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/389#issuecomment-440128762
issue_url: https://api.github.com/repos/simonw/datasette/issues/389
node_id: MDEyOklzc3VlQ29tbWVudDQ0MDEyODc2Mg==
created_at: 2018-11-20T03:52:11Z
updated_at: 2018-11-20T03:52:11Z
author_association: OWNER
body:
  The problem is Sanic. Here's the error I'm getting:

  ```
  (venv) datasette $ pytest -x 
  ============================================================= test session starts ==============================================================
  platform darwin -- Python 3.7.1, pytest-4.0.0, py-1.7.0, pluggy-0.8.0
  rootdir: /Users/simonw/Dropbox/Development/datasette, inifile:
  collected 258 items                                                                                                                            

  tests/test_api.py ...................F

  =================================================================== FAILURES ===================================================================
  _______________________________________________________ test_table_with_slashes_in_name ________________________________________________________

  app_client = <tests.fixtures.TestClient object at 0x10ada7d68>

      def test_table_with_slashes_in_name(app_client):
          response = app_client.get('/fixtures/table%2Fwith%2Fslashes.csv?_shape=objects&_format=json')
  >       assert response.status == 200
  E       AssertionError: assert 404 == 200
  ```

  That's because something about how Sanic handles escape characters in URLs changed between 0.7.0 and 0.8.3.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/374#issuecomment-448437245
issue_url: https://api.github.com/repos/simonw/datasette/issues/374
node_id: MDEyOklzc3VlQ29tbWVudDQ0ODQzNzI0NQ==
created_at: 2018-12-19T01:35:59Z
updated_at: 2018-12-19T01:35:59Z
author_association: OWNER
body:
  Closing this as Zeit went on a different direction with Now v2, so the 100MB limit is no longer a concern.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/393#issuecomment-450943632
issue_url: https://api.github.com/repos/simonw/datasette/issues/393
node_id: MDEyOklzc3VlQ29tbWVudDQ1MDk0MzYzMg==
created_at: 2019-01-02T18:30:20Z
updated_at: 2019-01-02T18:30:20Z
author_association: OWNER
body:
  This is the code which is meant to add those options as hidden form fields:

  https://github.com/simonw/datasette/blob/fe5b6ea95a973534fe8a44907c0ea2449aae7602/datasette/templates/table.html#L150-L155

  It's clearly not working. Need to fix this and add a corresponding unit test.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/393#issuecomment-450944166
issue_url: https://api.github.com/repos/simonw/datasette/issues/393
node_id: MDEyOklzc3VlQ29tbWVudDQ1MDk0NDE2Ng==
created_at: 2019-01-02T18:32:12Z
updated_at: 2019-01-02T18:32:12Z
author_association: OWNER
body:
  Here's the test that needs updating:

  https://github.com/simonw/datasette/blob/8b8ae55e7c8b9e1dceef53f55a330b596ca44d41/tests/test_html.py#L427-L435

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/391#issuecomment-450964512
issue_url: https://api.github.com/repos/simonw/datasette/issues/391
node_id: MDEyOklzc3VlQ29tbWVudDQ1MDk2NDUxMg==
created_at: 2019-01-02T19:45:12Z
updated_at: 2019-01-02T19:45:12Z
author_association: OWNER
body:
  Thanks, I've fixed this. I had to re-alias it against now:
  ```
  ~ $ now alias google-trends-pnwhfwvgqf.now.sh https://google-trends.datasettes.com/
  > Assigning alias google-trends.datasettes.com to deployment google-trends-pnwhfwvgqf.now.sh
  > Certificate for google-trends.datasettes.com (cert_uXaADIuNooHS3tZ) created [18s]
  > Success! google-trends.datasettes.com now points to google-trends-pnwhfwvgqf.now.sh [20s]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/393#issuecomment-451046123
issue_url: https://api.github.com/repos/simonw/datasette/issues/393
node_id: MDEyOklzc3VlQ29tbWVudDQ1MTA0NjEyMw==
created_at: 2019-01-03T03:05:07Z
updated_at: 2019-01-03T03:05:07Z
author_association: OWNER
body:
  The fix was released as part of Datasette 0.26 - you can see the fix working here: https://v0-26.datasette.io/fixtures-dd88475/facetable?_facet=planet_int&planet_int=1#export

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/393#issuecomment-451047426
issue_url: https://api.github.com/repos/simonw/datasette/issues/393
node_id: MDEyOklzc3VlQ29tbWVudDQ1MTA0NzQyNg==
created_at: 2019-01-03T03:19:04Z
updated_at: 2019-01-03T03:19:04Z
author_association: OWNER
body:
  https://fivethirtyeight.datasettes.com/-/versions is now running 0.26 - so your initial bug demo is now fixed: https://fivethirtyeight.datasettes.com/fivethirtyeight-c300360/classic-rock%2Fclassic-rock-song-list?Release+Year__exact=1989#export

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-451704724
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDQ1MTcwNDcyNA==
created_at: 2019-01-06T00:32:23Z
updated_at: 2019-01-06T00:33:44Z
author_association: OWNER
body:
  I found a really nice pattern for writing the unit tests for this (though it would look even nicer with a solution to #395)
  ```python
  @pytest.mark.parametrize("prefix", ["/prefix/", "https://example.com/"])
  @pytest.mark.parametrize("path", [
      "/",
      "/fixtures",
      "/fixtures/compound_three_primary_keys",
      "/fixtures/compound_three_primary_keys/a,a,a",
      "/fixtures/paginated_view",
  ])
  def test_url_prefix_config(prefix, path):
      for client in make_app_client(config={
          "url_prefix": prefix,
      }):
          response = client.get(path)
          soup = Soup(response.body, "html.parser")
          for a in soup.findAll("a"):
              href = a["href"]
              if href not in {
                  "https://github.com/simonw/datasette",
                  "https://github.com/simonw/datasette/blob/master/LICENSE",
                  "https://github.com/simonw/datasette/blob/master/tests/fixtures.py",
              }:
                  assert href.startswith(prefix), (href, a.parent)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/397#issuecomment-453251589
issue_url: https://api.github.com/repos/simonw/datasette/issues/397
node_id: MDEyOklzc3VlQ29tbWVudDQ1MzI1MTU4OQ==
created_at: 2019-01-10T20:59:42Z
updated_at: 2019-01-10T20:59:42Z
author_association: OWNER
body:
  What version of SQLite are you seeing in Datasette? You can tell by hitting http://localhost:8001/-/versions - e.g. here: https://latest.datasette.io/-/versions

  My best guess is that your Python SQLite module is running an older version that doesn't support window functions.

  One way you can fix that is with the `pysqlite3` module - try running this in your virtual environment:

      pip install git+git://github.com/karlb/pysqlite3

  That's using a fork of the official module that embeds a full recent SQLite. See this issue thread for more details: https://github.com/coleifer/pysqlite3/issues/2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/397#issuecomment-453252024
issue_url: https://api.github.com/repos/simonw/datasette/issues/397
node_id: MDEyOklzc3VlQ29tbWVudDQ1MzI1MjAyNA==
created_at: 2019-01-10T21:00:57Z
updated_at: 2019-01-10T21:00:57Z
author_association: OWNER
body:
  Oh I just saw you're using the official Datasette docker package - yeah, that's not bundled with a recent SQLite at the moment. We should update that:

  https://github.com/simonw/datasette/blob/5b026115126bedbb66457767e169139146d1c9fd/Dockerfile#L9-L11

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/271#issuecomment-453262703
issue_url: https://api.github.com/repos/simonw/datasette/issues/271
node_id: MDEyOklzc3VlQ29tbWVudDQ1MzI2MjcwMw==
created_at: 2019-01-10T21:35:18Z
updated_at: 2019-01-10T21:35:18Z
author_association: OWNER
body:
  It turns out this was much easier to support than I expected: https://github.com/simonw/datasette/commit/eac08f0dfc61a99e8887442fc247656d419c76f8

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/397#issuecomment-453330680
issue_url: https://api.github.com/repos/simonw/datasette/issues/397
node_id: MDEyOklzc3VlQ29tbWVudDQ1MzMzMDY4MA==
created_at: 2019-01-11T01:17:11Z
updated_at: 2019-01-11T01:25:33Z
author_association: OWNER
body:
  If you pull [the latest image](https://hub.docker.com/r/datasetteproject/datasette) you should get the right SQLite version now:

      docker pull datasetteproject/datasette
      docker run -p 8001:8001 \
          datasetteproject/datasette \
          datasette -p 8001 -h 0.0.0.0

  http://0.0.0.0:8001/-/versions now gives me:

  ```
          "version": "3.26.0"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/399#issuecomment-453874429
issue_url: https://api.github.com/repos/simonw/datasette/issues/399
node_id: MDEyOklzc3VlQ29tbWVudDQ1Mzg3NDQyOQ==
created_at: 2019-01-13T23:09:09Z
updated_at: 2019-01-13T23:09:09Z
author_association: OWNER
body:
  It looks like there are two reasons for this:

  - The `.git` directory was listed in `.dockerignore` so it wasn't being copied into the build process
  - The docker build stage wasn't installing the `git` executable, so it couldn't read the current version

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/399#issuecomment-453876023
issue_url: https://api.github.com/repos/simonw/datasette/issues/399
node_id: MDEyOklzc3VlQ29tbWVudDQ1Mzg3NjAyMw==
created_at: 2019-01-13T23:31:59Z
updated_at: 2019-01-13T23:31:59Z
author_association: OWNER
body:
  ```
  docker pull datasetteproject/datasette
  docker run -p 8001:8001 datasetteproject/datasette datasette -p 8001 -h 0.0.0.0
  ```
  http://0.0.0.0:8001/-/versions now returns:
  ```
  {
      "datasette": {
          "version": "0.26.2+0.ga418c8b.dirty"
      },
  ```
  I'm not sure why it's showing `.dirty` there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/402#issuecomment-455223551
issue_url: https://api.github.com/repos/simonw/datasette/issues/402
node_id: MDEyOklzc3VlQ29tbWVudDQ1NTIyMzU1MQ==
created_at: 2019-01-17T15:55:06Z
updated_at: 2019-01-17T15:55:06Z
author_association: OWNER
body:
  It's new in SQLite 3.26.0 so I will need to figure out how to only apply it in that version or higher. 

  https://sqlite.org/releaselog/3_26_0.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/402#issuecomment-455224327
issue_url: https://api.github.com/repos/simonw/datasette/issues/402
node_id: MDEyOklzc3VlQ29tbWVudDQ1NTIyNDMyNw==
created_at: 2019-01-17T15:56:57Z
updated_at: 2019-01-17T15:56:57Z
author_association: OWNER
body:
  https://sqlite.org/security.html has other recommmendations for apps that accept SQLite files from untrusted sources that we should apply.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/401#issuecomment-455230501
issue_url: https://api.github.com/repos/simonw/datasette/issues/401
node_id: MDEyOklzc3VlQ29tbWVudDQ1NTIzMDUwMQ==
created_at: 2019-01-17T16:12:59Z
updated_at: 2019-01-17T16:12:59Z
author_association: OWNER
body:
  Datasette-cluster-map doesn't use the new plugin configuration mechanism yet - it really should!

  The best example of how to use this mechanism right now is embedded in the Datasette unit tests: 

  https://github.com/simonw/datasette/blob/b7257a21bf3dfa7353980f343c83a616da44daa7/tests/fixtures.py#L266-L270

  https://github.com/simonw/datasette/blob/b7257a21bf3dfa7353980f343c83a616da44daa7/tests/test_plugins.py#L139-L145

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/402#issuecomment-455231411
issue_url: https://api.github.com/repos/simonw/datasette/issues/402
node_id: MDEyOklzc3VlQ29tbWVudDQ1NTIzMTQxMQ==
created_at: 2019-01-17T16:15:21Z
updated_at: 2019-01-17T16:15:21Z
author_association: OWNER
body:
  Unfortunately it looks like there isn't currently a mechanism in the Python sqlite3 library for setting configuration flags like SQLITE_DBCONFIG_DEFENSIVE

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/401#issuecomment-455445069
issue_url: https://api.github.com/repos/simonw/datasette/issues/401
node_id: MDEyOklzc3VlQ29tbWVudDQ1NTQ0NTA2OQ==
created_at: 2019-01-18T06:49:07Z
updated_at: 2019-01-18T06:49:07Z
author_association: OWNER
body:
  I've released a new version of the datasette-cluster-map plugin to illustrate how plugin configuration can work: https://github.com/simonw/datasette-cluster-map/commit/fcc86c450e3df3e6b81c41f31df458923181527a

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/403#issuecomment-455445392
issue_url: https://api.github.com/repos/simonw/datasette/issues/403
node_id: MDEyOklzc3VlQ29tbWVudDQ1NTQ0NTM5Mg==
created_at: 2019-01-18T06:51:14Z
updated_at: 2019-01-18T06:51:14Z
author_association: OWNER
body:
  I talk about that a bit here: https://simonwillison.net/2018/Oct/4/datasette-ideas/#Bundling_the_data_with_the_code

  One of the key ideas behind Datasette is that if your data is read-only you can package it up with the rest of your code - so the normal limitations that apply with hosting services like now.sh no longer prevent you from including a database. The SQLite database is just another static binary file that gets packaged up as part of your deployment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/401#issuecomment-455520561
issue_url: https://api.github.com/repos/simonw/datasette/issues/401
node_id: MDEyOklzc3VlQ29tbWVudDQ1NTUyMDU2MQ==
created_at: 2019-01-18T11:48:13Z
updated_at: 2019-01-18T11:48:13Z
author_association: NONE
body:
  Thanks.  I'll take a look at your changes.
  I must admit I was struggling to see how to pass info from the python code in __init__.py into the javascript document.addEventListener function.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/405#issuecomment-457975857
issue_url: https://api.github.com/repos/simonw/datasette/issues/405
node_id: MDEyOklzc3VlQ29tbWVudDQ1Nzk3NTg1Nw==
created_at: 2019-01-28T01:48:37Z
updated_at: 2019-01-28T01:49:00Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures-dd88475/facetable.json?_shape=array&_nl=on

  Also https://b5dd839.datasette.io/fixtures-dd88475/facetable.json?_shape=array&_nl=on

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/404#issuecomment-457976864
issue_url: https://api.github.com/repos/simonw/datasette/issues/404
node_id: MDEyOklzc3VlQ29tbWVudDQ1Nzk3Njg2NA==
created_at: 2019-01-28T01:56:55Z
updated_at: 2019-01-28T01:56:55Z
author_association: OWNER
body:
  This failed in Python 3.5:

  ```
    File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/jinja2/environment.py", line 1020, in render_async
      raise NotImplementedError('This feature is not available for this '
  NotImplementedError: This feature is not available for this version of Python
  ```

  It looks like this is caused by this feature detection code:

  https://github.com/pallets/jinja/blob/a7ba0b637805c53d442e975e3864d3ea38d8743f/jinja2/utils.py#L633-L638

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/7#issuecomment-457980966
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/7
node_id: MDEyOklzc3VlQ29tbWVudDQ1Nzk4MDk2Ng==
created_at: 2019-01-28T02:29:32Z
updated_at: 2019-01-28T02:29:32Z
author_association: OWNER
body:
  Remember to remove this TODO (and turn the `[]` into `()` on this line) as part of this task:

  https://github.com/simonw/sqlite-utils/blob/5309c5c7755818323a0f5353bad0de98ecc866be/sqlite_utils/cli.py#L78-L80

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011906
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/7
node_id: MDEyOklzc3VlQ29tbWVudDQ1ODAxMTkwNg==
created_at: 2019-01-28T06:25:55Z
updated_at: 2019-01-28T06:25:55Z
author_association: OWNER
body:
  I tested this with a script called `churn_em_out.py`
  ```
  i = 0
  while True:
      i += 1
      print(
          '{"id": I, "another": "row", "number": J}'.replace("I", str(i)).replace(
              "J", str(i + 1)
          )
      )
  ```

  Then I ran this:
  ```
  python churn_em_out.py | \
      sqlite-utils insert /tmp/getbig.db stats - \
          --nl --batch-size=10000
  ```
  And used `watch 'ls -lah /tmp/getbig.db'` to watch the file growing as it had 10,000 lines of junk committed in batches. The memory used by the process never grew about around 50MB.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/160#issuecomment-459915995
issue_url: https://api.github.com/repos/simonw/datasette/issues/160
node_id: MDEyOklzc3VlQ29tbWVudDQ1OTkxNTk5NQ==
created_at: 2019-02-02T00:43:16Z
updated_at: 2019-02-02T00:58:20Z
author_association: CONTRIBUTOR
body:
  Do you have any simple working examples of how to use `--static`? Inspection of default served files suggests locations such as `http://example.com/-/static/app.css?0e06ee`.

  If `datasette` is being proxied to `http://example.com/foo/datasette`, what form should arguments to  `--static` take so that static files are correctly referenced?

  Use case is here: https://github.com/psychemedia/jupyterserverproxy-datasette-demo Trying to do a really simple `datasette` demo in MyBinder using jupyter-server-proxy.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/398#issuecomment-460901857
issue_url: https://api.github.com/repos/simonw/datasette/issues/398
node_id: MDEyOklzc3VlQ29tbWVudDQ2MDkwMTg1Nw==
created_at: 2019-02-06T05:01:19Z
updated_at: 2019-02-06T05:01:19Z
author_association: OWNER
body:
  I'd really like to use the content-length header here, but Sanic hasn't yet fixed the bug I filed about it: https://github.com/huge-success/sanic/issues/1194

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/172#issuecomment-460902824
issue_url: https://api.github.com/repos/simonw/datasette/issues/172
node_id: MDEyOklzc3VlQ29tbWVudDQ2MDkwMjgyNA==
created_at: 2019-02-06T05:09:05Z
updated_at: 2019-02-06T05:09:05Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures-dd88475

  <img width="373" alt="screen shot 2019-02-05 at 9 08 49 pm" src="https://user-images.githubusercontent.com/9599/52321564-41f03e00-298a-11e9-9fad-a8bb4e89ad24.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/187#issuecomment-463917744
issue_url: https://api.github.com/repos/simonw/datasette/issues/187
node_id: MDEyOklzc3VlQ29tbWVudDQ2MzkxNzc0NA==
created_at: 2019-02-15T05:58:44Z
updated_at: 2019-02-15T05:58:44Z
author_association: NONE
body:
  is this supported or not? you can comment if it is not supported so that people like me can stop trying.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/187#issuecomment-466325528
issue_url: https://api.github.com/repos/simonw/datasette/issues/187
node_id: MDEyOklzc3VlQ29tbWVudDQ2NjMyNTUyOA==
created_at: 2019-02-22T09:03:50Z
updated_at: 2019-02-22T09:03:50Z
author_association: NONE
body:
  I ran into the same issue when trying to install datasette on windows after successfully using it on linux. Unfortunately, there has not been any progress in implementing uvloop for windows - so I recommend not to use it on win. You can read about this issue here:
  [https://github.com/MagicStack/uvloop/issues/14](url)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/8#issuecomment-466695500
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/8
node_id: MDEyOklzc3VlQ29tbWVudDQ2NjY5NTUwMA==
created_at: 2019-02-23T21:09:03Z
updated_at: 2019-02-23T21:09:03Z
author_association: OWNER
body:
  Fixed in https://github.com/simonw/sqlite-utils/commit/228d595f7d10994f34e948888093c2cd290267c4

  <img width="745" alt="screen shot 2019-02-23 at 1 07 14 pm" src="https://user-images.githubusercontent.com/9599/53291690-2d72ba80-376c-11e9-99b8-074045d6290d.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/11#issuecomment-466695672
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/11
node_id: MDEyOklzc3VlQ29tbWVudDQ2NjY5NTY3Mg==
created_at: 2019-02-23T21:10:23Z
updated_at: 2019-02-23T21:10:23Z
author_association: OWNER
body:
  Rough sketch:

  ```
  +try:
  +    import numpy
  +except ImportError:
  +    numpy = None
  +
   Column = namedtuple(
       "Column", ("cid", "name", "type", "notnull", "default_value", "is_pk")
   )

  @@ -70,6 +79,22 @@ class Database:
               datetime.time: "TEXT",
               None.__class__: "TEXT",
           }
  +        # If numpy is available, add more types
  +        if numpy:
  +            col_type_mapping.update({
  +                numpy.int8: "INTEGER",
  +                numpy.int16: "INTEGER",
  +                numpy.int32: "INTEGER",
  +                numpy.int64: "INTEGER",
  +                numpy.uint8: "INTEGER",
  +                numpy.uint16: "INTEGER",
  +                numpy.uint32: "INTEGER",
  +                numpy.uint64: "INTEGER",
  +                numpy.float16: "FLOAT",
  +                numpy.float32: "FLOAT",
  +                numpy.float64: "FLOAT",
  +                numpy.float128: "FLOAT",
  +            })
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/13#issuecomment-466732039
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/13
node_id: MDEyOklzc3VlQ29tbWVudDQ2NjczMjAzOQ==
created_at: 2019-02-24T04:07:57Z
updated_at: 2019-02-24T04:07:57Z
author_association: OWNER
body:
  Example: http://api.nobelprize.org/v1/laureate.json

  This includes affiliations which look like this:

            "affiliations": [
              {
                "name": "Sorbonne University",
                "city": "Paris",
                "country": "France"
              }
            ]

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/10#issuecomment-466794069
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/10
node_id: MDEyOklzc3VlQ29tbWVudDQ2Njc5NDA2OQ==
created_at: 2019-02-24T16:55:37Z
updated_at: 2019-02-24T16:55:37Z
author_association: OWNER
body:
  This was fixed by https://github.com/simonw/sqlite-utils/commit/228d595f7d10994f34e948888093c2cd290267c4 - see also #8 
  ```
  >>> db = sqlite_utils.Database(":memory:")
  >>> dfX=pd.DataFrame({'order':range(3),'col2':range(3)})
  >>> db["test"].upsert_all(dfX.to_dict(orient='records'))
  <Table test>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/14#issuecomment-466794369
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDQ2Njc5NDM2OQ==
created_at: 2019-02-24T16:59:11Z
updated_at: 2019-02-24T16:59:43Z
author_association: OWNER
body:
  https://www.sqlite.org/lang_createindex.html

  ![image](https://user-images.githubusercontent.com/9599/53302378-72512c80-3812-11e9-8828-46a03d893879.png)

  May as well support ``--if-not-exists`` as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/14#issuecomment-466800090
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDQ2NjgwMDA5MA==
created_at: 2019-02-24T18:01:10Z
updated_at: 2019-02-24T18:01:10Z
author_association: OWNER
body:
  The `WHERE` clause can be used to create partial indexes: https://www.sqlite.org/partialindex.html

  I'm going to ignore it for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/14#issuecomment-466800210
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDQ2NjgwMDIxMA==
created_at: 2019-02-24T18:02:23Z
updated_at: 2019-02-24T18:02:23Z
author_association: OWNER
body:
  Likewise I'm going to ignore indexes on expressions (as opposed to just columns): https://www.sqlite.org/expridx.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/2#issuecomment-466807308
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/2
node_id: MDEyOklzc3VlQ29tbWVudDQ2NjgwNzMwOA==
created_at: 2019-02-24T19:18:19Z
updated_at: 2019-02-24T19:18:25Z
author_association: OWNER
body:
  Python API:

      db["articles"].add_foreign_key("author_id", "authors", "id")

  CLI:

      $ sqlite-utils add-foreign-key articles author_id authors id

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/17#issuecomment-466820167
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/17
node_id: MDEyOklzc3VlQ29tbWVudDQ2NjgyMDE2Nw==
created_at: 2019-02-24T21:42:33Z
updated_at: 2019-02-24T21:42:33Z
author_association: OWNER
body:
  It looks like the type information isn't actually used for anything at all, so this:

  https://github.com/simonw/sqlite-utils/blob/f8d3b7cfe5c1950b0749d40eb2640df50b52f651/tests/test_create.py#L97-L103

  Could actually be written like this:

  ```
      fresh_db["m2m"].insert(
          {"one_id": 1, "two_id": 1},
          foreign_keys=(
              ("one_id", "one", "id"),
              ("two_id", "two", "id"),
          ),
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/17#issuecomment-466821200
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/17
node_id: MDEyOklzc3VlQ29tbWVudDQ2NjgyMTIwMA==
created_at: 2019-02-24T21:55:08Z
updated_at: 2019-02-24T21:55:54Z
author_association: OWNER
body:
  This involves a breaking API change. I need to call that out in the README and also fix my two other projects which use the old four-tuple version of `foreign_keys=`:

  https://github.com/simonw/db-to-sqlite/blob/c2f8e93bc6bbdfd135de3656ea0f497859ae49ff/db_to_sqlite/cli.py#L30-L42

  And

  https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/e7106710abdd7bdcae035bedd8bdaba75ae56a12/fetch_and_build_russian_ads.py#L71-L74

  I'll also need to set a minimum version for `sqlite-utils` in the `db-to-sqlite` setup.py:

  https://github.com/simonw/db-to-sqlite/blob/c2f8e93bc6bbdfd135de3656ea0f497859ae49ff/setup.py#L25

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/187#issuecomment-467264937
issue_url: https://api.github.com/repos/simonw/datasette/issues/187
node_id: MDEyOklzc3VlQ29tbWVudDQ2NzI2NDkzNw==
created_at: 2019-02-26T02:14:28Z
updated_at: 2019-02-26T02:14:28Z
author_association: OWNER
body:
  I'm working on a port of Datasette to Starlette which I think would fix this issue: https://github.com/encode/starlette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/409#issuecomment-472844001
issue_url: https://api.github.com/repos/simonw/datasette/issues/409
node_id: MDEyOklzc3VlQ29tbWVudDQ3Mjg0NDAwMQ==
created_at: 2019-03-14T13:04:20Z
updated_at: 2019-03-14T13:04:42Z
author_association: NONE
body:
  It seems this affects the Datasette Publish -site as well: https://github.com/simonw/datasette-publish-support/issues/3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/409#issuecomment-472875713
issue_url: https://api.github.com/repos/simonw/datasette/issues/409
node_id: MDEyOklzc3VlQ29tbWVudDQ3Mjg3NTcxMw==
created_at: 2019-03-14T14:14:39Z
updated_at: 2019-03-14T14:14:39Z
author_association: NONE
body:
  also linking this zeit issue in case it is helpful: https://github.com/zeit/now-examples/issues/163#issuecomment-440125769

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/416#issuecomment-473154643
issue_url: https://api.github.com/repos/simonw/datasette/issues/416
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzE1NDY0Mw==
created_at: 2019-03-15T04:27:47Z
updated_at: 2019-03-15T04:28:00Z
author_association: OWNER
body:
  Deployed a demo: https://datasette-optional-hash-demo.now.sh/

      datasette publish now \
          ../demo-databses/russian-ads.db \
          ../demo-databses/polar-bears.db \
          --branch=optional-hash \
          -n datasette-optional-hash \
          --alias datasette-optional-hash-demo \
          --install=datasette-cluster-map \
          --install=datasette-json-html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/416#issuecomment-473156513
issue_url: https://api.github.com/repos/simonw/datasette/issues/416
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzE1NjUxMw==
created_at: 2019-03-15T04:40:29Z
updated_at: 2019-03-15T04:40:29Z
author_association: OWNER
body:
  Still TODO: need to figure out what to do about cache TTL. Defaulting to 365 days no longer makes sense without the hash_urls setting.

  Maybe drop that setting default to 0?

  Here's the setting:

  https://github.com/simonw/datasette/blob/9743e1d91b5f0a2b3c1c0bd6ffce8739341f43c4/datasette/app.py#L84-L86

  And here's where it takes affect:

  https://github.com/simonw/datasette/blob/4462a5ab2817ac0d9ffe20dafbbf27c5c5b81466/datasette/views/base.py#L491-L501

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/414#issuecomment-473156774
issue_url: https://api.github.com/repos/simonw/datasette/issues/414
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzE1Njc3NA==
created_at: 2019-03-15T04:42:06Z
updated_at: 2019-03-15T04:42:06Z
author_association: OWNER
body:
  This has been bothering me as well, especially when I try to install `datasette` and `sqlite-utils` at the same time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/415#issuecomment-473157770
issue_url: https://api.github.com/repos/simonw/datasette/issues/415
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzE1Nzc3MA==
created_at: 2019-03-15T04:49:03Z
updated_at: 2019-03-15T04:49:03Z
author_association: OWNER
body:
  Interesting idea. I can see how this would make sense if you are dealing with really long SQL queries.

  My own example of a long query that might benefit from this: https://russian-ads-demo.herokuapp.com/russian-ads-a42c4e8?sql=select%0D%0A++++target_id%2C%0D%0A++++targets.name%2C%0D%0A++++count(*)+as+n%2C%0D%0A++++json_object(%0D%0A++++++++%22href%22%2C+%22%2Frussian-ads%2Ffaceted-targets%3Ftargets%3D%22+||+%0D%0A++++++++++++json_insert(%3Atargets%2C+%27%24[%27+||+json_array_length(%3Atargets)+||+%27]%27%2C+target_id)%0D%0A++++++++%2C%0D%0A++++++++%22label%22%2C+json_insert(%3Atargets%2C+%27%24[%27+||+json_array_length(%3Atargets)+||+%27]%27%2C+target_id)%0D%0A++++)+as+apply_this_facet%2C%0D%0A++++json_object(%0D%0A++++++++%22href%22%2C+%22%2Frussian-ads%2Fdisplay_ads%3F_targets_json%3D%22+||+%0D%0A++++++++++++json_insert(%3Atargets%2C+%27%24[%27+||+json_array_length(%3Atargets)+||+%27]%27%2C+target_id)%0D%0A++++++++%2C%0D%0A++++++++%22label%22%2C+%22See+%22+||+count(*)+||+%22+ads+matching+%22+||+json_insert(%3Atargets%2C+%27%24[%27+||+json_array_length(%3Atargets)+||+%27]%27%2C+target_id)%0D%0A++++)+as+browse_these_ads%0D%0Afrom+ad_targets%0D%0Ajoin+targets+on+ad_targets.target_id+%3D+targets.id%0D%0Awhere%0D%0A++++json_array_length(%3Atargets)+%3D%3D+0+or%0D%0A++++ad_id+in+(%0D%0A++++++++select+ad_id%0D%0A++++++++from+%22ad_targets%22%0D%0A++++++++where+%22ad_targets%22.target_id+in+(select+value+from+json_each(%3Atargets))%0D%0A++++++++group+by+%22ad_targets%22.ad_id%0D%0A++++++++having+count(distinct+%22ad_targets%22.target_id)+%3D+json_array_length(%3Atargets)%0D%0A++++)%0D%0A++++and+target_id+not+in+(select+value+from+json_each(%3Atargets))%0D%0Agroup+by%0D%0A++++target_id+order+by+n+desc%0D%0A&targets=[%22e6200%22]

  Having a `show/hide` link would be an easy way to support this in the UI, and those could add/remove a `_hide_sql=1` parameter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/412#issuecomment-473158506
issue_url: https://api.github.com/repos/simonw/datasette/issues/412
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzE1ODUwNg==
created_at: 2019-03-15T04:53:53Z
updated_at: 2019-03-15T04:53:53Z
author_association: OWNER
body:
  I've been thinking about how Datasette instances could query each other for a while - it's a really interesting direction.

  There are some tricky problems to solve to get this to work. There's a SQLite mechanism called "virtual table functions" which can implement things like this, but it's not supported by Python's `sqlite3` module out of the box.

  https://github.com/coleifer/sqlite-vtfunc is a library that enables this feature. I experimented with using that to implement a function that scrapes HTML content (with an eye to accessing data from other APIs and Datasette instances) a while ago: https://github.com/coleifer/sqlite-vtfunc/issues/6

  The bigger challenge is how to get this kind of thing to behave well within a Python 3 async environment. I have some ideas here but they're going to require some very crafty engineering.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/416#issuecomment-473159679
issue_url: https://api.github.com/repos/simonw/datasette/issues/416
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzE1OTY3OQ==
created_at: 2019-03-15T05:01:27Z
updated_at: 2019-03-15T05:01:27Z
author_association: OWNER
body:
  Also: if the option is False and the user visits a URL with a hash in it, should we redirect them?

  I'm inclined to say no: furthermore, I'd be OK continuing to serve a far-future cache header for that case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/416#issuecomment-473160702
issue_url: https://api.github.com/repos/simonw/datasette/issues/416
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzE2MDcwMg==
created_at: 2019-03-15T05:08:13Z
updated_at: 2019-03-15T05:08:13Z
author_association: OWNER
body:
  This also needs extensive tests to ensure that with the option turned on all of the redirects behave as they should.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/415#issuecomment-473164038
issue_url: https://api.github.com/repos/simonw/datasette/issues/415
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzE2NDAzOA==
created_at: 2019-03-15T05:31:21Z
updated_at: 2019-03-15T05:31:21Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures-dd88475?sql=select+%2A+from+sortable+order+by+pk1%2C+pk2+limit+101

  <img width="761" alt="Screen Shot 2019-03-14 at 10 30 47 PM" src="https://user-images.githubusercontent.com/9599/54410469-d5631000-46a8-11e9-8f77-d8fb5dd8fd23.png">

  v.s. https://latest.datasette.io/fixtures-dd88475?sql=select+%2A+from+sortable+order+by+pk1%2C+pk2+limit+101&_hide_sql=1

  <img width="761" alt="Screen Shot 2019-03-14 at 10 31 05 PM" src="https://user-images.githubusercontent.com/9599/54410489-e27fff00-46a8-11e9-9207-8245b62f1cf4.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/417#issuecomment-473308631
issue_url: https://api.github.com/repos/simonw/datasette/issues/417
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzMwODYzMQ==
created_at: 2019-03-15T14:32:13Z
updated_at: 2019-03-15T14:32:13Z
author_association: OWNER
body:
  This would allow Datasette to be easily used as a "data library" (like a data warehouse but less expectation of big data querying technology such as Presto).

  One of the things I learned at the NICAR CAR 2019 conference in Newport Beach is that there is a very real need for some kind of easily accessible data library at most newsrooms.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/417#issuecomment-473312514
issue_url: https://api.github.com/repos/simonw/datasette/issues/417
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzMxMjUxNA==
created_at: 2019-03-15T14:42:07Z
updated_at: 2019-03-17T22:12:30Z
author_association: OWNER
body:
  A neat ability of Datasette Library would be if it can work against other files that have been dropped into the folder. In particular: if a user drops a CSV file into the folder, how about automatically converting that CSV file to SQLite using [sqlite-utils](https://github.com/simonw/sqlite-utils)?

reactions:
  {"total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/123#issuecomment-473313975
issue_url: https://api.github.com/repos/simonw/datasette/issues/123
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzMxMzk3NQ==
created_at: 2019-03-15T14:45:46Z
updated_at: 2019-03-15T14:45:46Z
author_association: OWNER
body:
  I'm reopening this one as part of #417.

  Further experience with Python's CSV standard library module has convinced me that pandas is not a required dependency for this. My [sqlite-utils](https://github.com/simonw/sqlite-utils) package can do most of the work here with very few dependencies.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/123#issuecomment-473323329
issue_url: https://api.github.com/repos/simonw/datasette/issues/123
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzMyMzMyOQ==
created_at: 2019-03-15T15:09:15Z
updated_at: 2019-05-14T15:53:05Z
author_association: OWNER
body:
  How would Datasette accepting URLs work?

  I want to support not just SQLite files and CSVs but other extensible formats (geojson, Atom, shapefiles etc) as well.

  So `datasette serve` needs to be able to take filepaths or URLs to a variety of different content types.

  If it's a URL, we can use the first 200 downloaded bytes to decide which type of file it is. This is likely more reliable than hoping the web server provided the correct content-type.

  Also: let's have a threshold for downloading to disk. We will start downloading to a temp file (location controlled by an environment variable) if either the content length header is above that threshold OR we hit that much data cached in memory already and don't know how much more is still to come.

  There needs to be a command line option for saying "grab from this URL but force treat it as CSV" - same thing for files on disk.

      datasette mydb.db --type=db http://blah/blah --type=csv

  If you provide less `--type` options thatn you did URLs then the default behavior is used for all of the subsequent URLs.

  Auto detection could be tricky. Probably do this with a plugin hook.

  https://github.com/h2non/filetype.py is interesting but deals with images video etc so not right for this purpose. 

  I think we need our own simple content sniffing code via a plugin hook.

  What if two plugin type hooks can both potentially handle a sniffed file? The CLI can quit and return an error saying content is ambiguous and you need to specify a `--type`, picking from the following list.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/419#issuecomment-473708724
issue_url: https://api.github.com/repos/simonw/datasette/issues/419
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzcwODcyNA==
created_at: 2019-03-17T19:55:21Z
updated_at: 2019-05-16T03:35:59Z
author_association: OWNER
body:
  Thinking about this further: I think I may have made a mistake establishing "immutable" as the default mode for databases opened by Datasette.

  What would it look like if files were NOT opened in immutable mode by default?

  Maybe the command to start Datasette looks like this:

      datasette mutable1.db mutable2.db --immutable=this_is_immutable.db --immutable=this_is_immutable2.db

  So regular file arguments are treated as mutable (and opened in `?mode=ro`) while file arguments passed using the new `--immutable` option are opened in immutable mode.

  The `-i` shortcut has not yet been taken, so this could be abbreviated to:

      datasette mutable1.db mutable2.db -i this_is_immutable.db -i this_is_immutable2.db

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/419#issuecomment-473708941
issue_url: https://api.github.com/repos/simonw/datasette/issues/419
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzcwODk0MQ==
created_at: 2019-03-17T19:58:11Z
updated_at: 2019-03-17T19:58:11Z
author_association: OWNER
body:
  Some problems to solve:

  * Right now Datasette assumes it can always show the count of rows in a table, because this has been pre-calculated. If a database is mutable the pre-calculation trick no longer works, and for giant tables a `select count(*) from X` query can be expensive to run. Maybe we set a time limit on these? If time limit expires show "many rows"?
  * Maintaining a content hash of the table no longer makes sense if it is changing (though interestingly there's a `.sha3sum` built-in SQLite CLI command which takes a hash of the content and stays the same even through vacuum runs). Without that we need a different mechanism for calculating table colours. It also means that we can't do the special dbname-hash URL trick (see #418) at all if the database is opened as mutable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/418#issuecomment-473709815
issue_url: https://api.github.com/repos/simonw/datasette/issues/418
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzcwOTgxNQ==
created_at: 2019-03-17T20:08:31Z
updated_at: 2019-03-17T20:08:31Z
author_association: OWNER
body:
  In #419 I'm now proposing that Datasette default to opening files in "mutable" mode, in which case it would not make sense to support hash URLs for those files at all. So actually this feature will only be available for files that are explicitly opened in immutable mode.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/419#issuecomment-473709883
issue_url: https://api.github.com/repos/simonw/datasette/issues/419
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzcwOTg4Mw==
created_at: 2019-03-17T20:09:47Z
updated_at: 2019-03-17T20:37:45Z
author_association: OWNER
body:
  Could I persist the last calculated count for a table and somehow detect if that table has been changed in any way by another process, hence invalidating the cached count (and potentially scheduling a new count)?

  https://www.sqlite.org/c3ref/update_hook.html says that `sqlite3_update_hook()` can be used to register a handler invoked on almost all update/insert/delete operations to a specific table... except that it misses out on deletes triggered by `ON CONFLICT REPLACE` and only works for `ROWID` tables.

  Also this hook is not exposed in the Python `sqlite3` library - though it may be available using some terrifying `ctypes` hacks: https://stackoverflow.com/a/16920926

  So on further research, I think the answer is *no*: I should assume that it won't be possible to cache counts and magically invalidate the cache when the underlying file is changed by another process.

  Instead I need to assume that counts will be an expensive operation.

  As such, I can introduce a time limit on counts and use that anywhere a count is displayed. If the time limit is exceeded by the `count(*)` query I can show "many" instead.

  That said... running `count(*)` against a table with 200,000 rows in only takes about 3ms, so even a timeout of 20ms is likely to work fine for tables of around a million rows.

  It would be really neat if I could generate a lower bound count in a limited amount of time. If I counted up to 4m rows before the timeout I could show "more than 4m rows". No idea if that would be possible though.

  Relevant: https://stackoverflow.com/questions/8988915/sqlite-count-slow-on-big-tables - reports of very slow counts on 6GB database file. Consensus seems to be "yeah, that's just how SQLite is built" - though there was a suggestion that you can use `select max(ROWID) from table` provided you are certain there have been no deletions.

  Also relevant: http://sqlite.1065341.n5.nabble.com/sqlite3-performance-on-select-count-very-slow-for-16-GB-file-td80176.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/419#issuecomment-473712820
issue_url: https://api.github.com/repos/simonw/datasette/issues/419
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzcxMjgyMA==
created_at: 2019-03-17T20:43:23Z
updated_at: 2019-03-17T20:43:51Z
author_association: OWNER
body:
  So the differences here are:

  * For immutable databases we calculate content hash and table counts; mutable databases we do not
  * Immutable databasse open with `file:{}?immutable=1`, mutable databases open with `file:{}?mode=ro`
  * Anywhere that shows a table count now needs to call a new method which knows to run `count(*)` with a timeout for mutable databases, read from the precalculated counts for immutable databases
  * The url-hash option should no longer be available at all for mutable databases
  * New command-line tool syntax: `datasette mutable.db` v.s. `datasette -i immutable.db`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/419#issuecomment-473713363
issue_url: https://api.github.com/repos/simonw/datasette/issues/419
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzcxMzM2Mw==
created_at: 2019-03-17T20:49:39Z
updated_at: 2019-03-17T20:52:46Z
author_association: OWNER
body:
  And a really important difference: the whole model of caching inspect data no longer works for mutable files, because another process might make a change to the database schema (adding a new table for example).

  https://fivethirtyeight.datasettes.com/-/inspect

  So everywhere that uses `self.ds.inspect()` right now will have to change to calling a routine which knows the difference between mutable and immutable databases and queries for live schema data for mutables while using a cache for immutables.

  I'll track this as a separate ticket.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/420#issuecomment-473713946
issue_url: https://api.github.com/repos/simonw/datasette/issues/420
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzcxMzk0Ng==
created_at: 2019-03-17T20:56:38Z
updated_at: 2019-03-17T20:58:17Z
author_association: OWNER
body:
  Some examples:

  https://github.com/simonw/datasette/blob/1f54e092306b208125f39d06712b02895eb75168/datasette/views/table.py#L34-L40

  https://github.com/simonw/datasette/blob/1f54e092306b208125f39d06712b02895eb75168/datasette/views/table.py#L45-L48

  https://github.com/simonw/datasette/blob/1f54e092306b208125f39d06712b02895eb75168/datasette/views/table.py#L62-L65

  https://github.com/simonw/datasette/blob/1f54e092306b208125f39d06712b02895eb75168/datasette/views/table.py#L112-L123

  https://github.com/simonw/datasette/blob/1f54e092306b208125f39d06712b02895eb75168/datasette/views/index.py#L11-L19

  https://github.com/simonw/datasette/blob/afe9aa3ae03c485c5d6652741438d09445a486c1/datasette/views/base.py#L143-L147

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/416#issuecomment-473714545
issue_url: https://api.github.com/repos/simonw/datasette/issues/416
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzcxNDU0NQ==
created_at: 2019-03-17T21:03:08Z
updated_at: 2019-03-17T21:04:17Z
author_association: OWNER
body:
  I'm going to introduce a new config setting: `default_cache_ttl_hashed` - and set the default value for `default_cache_ttl` to 10s (to protect against dog-piling).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/416#issuecomment-473715254
issue_url: https://api.github.com/repos/simonw/datasette/issues/416
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzcxNTI1NA==
created_at: 2019-03-17T21:11:37Z
updated_at: 2019-03-17T21:11:37Z
author_association: OWNER
body:
  The code for this has got a bit tricky. I need to make a decision at some point as to if the current request is a hashed_url request (if it includes a DB hash in the URL which is the current correct hash). I then need to be able to use that fact to decide which default TTL value to apply when returning the response.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/416#issuecomment-473717052
issue_url: https://api.github.com/repos/simonw/datasette/issues/416
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzcxNzA1Mg==
created_at: 2019-03-17T21:32:24Z
updated_at: 2019-03-17T21:33:16Z
author_association: OWNER
body:
  Since this feature is now controlled by a config setting, I'm inclined to make it also available via a URL parameter.

  If you hit this URL:

      /fixtures/table.json?_hash=1

  We can redirect to:

      /fixtures-c2342/table.json

  In this way developers can opt-in to a hashed (and hence far-future cached) response on a per-query basis.

  This option won't be available against mutable databases though, which are coming in #419 

  This means that the `hash_urls:1` config basically has the effect of assuming `?_hash=1` on all URLs to mutable databases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/418#issuecomment-473724868
issue_url: https://api.github.com/repos/simonw/datasette/issues/418
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzcyNDg2OA==
created_at: 2019-03-17T23:07:31Z
updated_at: 2019-03-17T23:07:31Z
author_association: OWNER
body:
  The design of this feature is discussed extensively in the comments on pull request #416 

  Some demos:

  * https://latest.datasette.io/fixtures/facetable now no longer redirects to the hash
  * https://latest.datasette.io/fixtures/facetable?_hash=1 redirects to https://latest.datasette.io/fixtures-dd88475/facetable

  ```
  ~ $ curl -i 'https://latest.datasette.io/fixtures-dd88475/facetable' 
  HTTP/2 200 
  date: Sun, 17 Mar 2019 23:05:21 GMT
  content-type: text/html; charset=utf-8
  content-length: 17555
  cache-control: max-age=31536000

  </html>~ $ curl -i 'https://latest.datasette.io/fixtures/facetable' 
  HTTP/2 200 
  date: Sun, 17 Mar 2019 23:05:40 GMT
  content-type: text/html; charset=utf-8
  content-length: 17410
  cache-control: max-age=5
  ```

  There are now three config settings relevant to the above:

  `default_cache_ttl` - defaults to 5s. The default cache TTL for non-hashed resources.
  `default_cache_ttl_hashed` - defaults to 31536000s. The default cache TTL for hashed resources.
  `hash_urls` - defaults to False. If True, all URLs will attempt to redirect to their hashed version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/419#issuecomment-473726527
issue_url: https://api.github.com/repos/simonw/datasette/issues/419
node_id: MDEyOklzc3VlQ29tbWVudDQ3MzcyNjUyNw==
created_at: 2019-03-17T23:28:41Z
updated_at: 2019-05-16T14:54:50Z
author_association: OWNER
body:
  I've added the `-i` option, so this now works:

      datasette -i fixtures.db

  This feature is incomplete though. Some extra changes I need to make:

  * The `?_hash=1` and `--config hash_urls:1` options (introduced in #418) should only work for immutable databases #471
  * Would be useful if there was a debug screen that could show which databases were mounted as mutable v.s. immutable - maybe a `/-/databases` page? - #470 
  * Need to rework how `.inspect()` works, see #420 
  * Documentation is needed #421

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/420#issuecomment-473744172
issue_url: https://api.github.com/repos/simonw/datasette/issues/420
node_id: MDEyOklzc3VlQ29tbWVudDQ3Mzc0NDE3Mg==
created_at: 2019-03-18T02:08:12Z
updated_at: 2019-03-18T02:08:12Z
author_association: OWNER
body:
  Maybe this is a good opportunity to improve the introspection capabilities in [sqlite-utils](https://github.com/simonw/sqlite-utils) and add it as a dependency.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/417#issuecomment-474280581
issue_url: https://api.github.com/repos/simonw/datasette/issues/417
node_id: MDEyOklzc3VlQ29tbWVudDQ3NDI4MDU4MQ==
created_at: 2019-03-19T10:06:42Z
updated_at: 2019-03-19T10:06:42Z
author_association: CONTRIBUTOR
body:
  This would be really interesting but several possibilities in use arise, I think?

  For example:

  - I put a new CSV file into the import dir and a new table is created therefrom
  - I put a CSV file into the import dir that replaces a previous file / table of the same name as a pre-existing table (eg files that contain monthly data in year to date). The data may also patch previous months, so a full replace / DROP on the original table may well be in order.
  - I put a CSV file into the import dir that updates a table of the same name as a pre-existing table (eg files that contain last month's data)

  CSV files may also have messy names compared to the table you want. Or for an update CSV, may have the form `MYTABLENAME-February2019.csv` etc

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/412#issuecomment-474282321
issue_url: https://api.github.com/repos/simonw/datasette/issues/412
node_id: MDEyOklzc3VlQ29tbWVudDQ3NDI4MjMyMQ==
created_at: 2019-03-19T10:09:46Z
updated_at: 2019-03-19T10:09:46Z
author_association: CONTRIBUTOR
body:
  Does this also relate to https://github.com/simonw/datasette/issues/283 and the ability to `ATTACH DATABASE`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/420#issuecomment-474398127
issue_url: https://api.github.com/repos/simonw/datasette/issues/420
node_id: MDEyOklzc3VlQ29tbWVudDQ3NDM5ODEyNw==
created_at: 2019-03-19T14:34:55Z
updated_at: 2019-03-19T14:34:55Z
author_association: OWNER
body:
  I systematically reviewed the codebase for things that `.inspect()` is used for:

  In `app.py`:

  * `table_exists()` uses `table in self.inspect().get(database, {}).get("tables")`
  * `.execute()` looks up the database name to get the `info["file"]` (the correct filename with the `.db` extension)

  In `cli.py`:

  * The `datasette inspect` command dumps it to JSON
  * `datasette skeleton` iterates over it
  * `datasette serve` calls it on startup (to populate static cache of inspect data)

  In `base.py`:

  * `.database_url(database)` calls it to lookup the hash (if `hash_urls` config turned on)
  * `.resolve_db_name()` uses it to lookup the hash

  In `database.py`:

  * `DatabaseView` uses it to find up the list of tables and views to display, plus the size of the DB file in bytes
  * `DatabaseDownload` uses it to get the filepath for download

  In `index.py`:

  * `IndexView` uses it _extensively_ - to loop through every database and every table. This would make a good starting point for the refactor.

  In `table.py`:

  * `sortable_columns_for_table()` uses it to find the columns in a table
  * `expandable_columns()` uses it to find foreign keys
  * `expand_foreign_keys()` uses it to find foreign keys
  * `display_columns_and_rows()` uses it to find primary keys and foreign keys... but also has access to a `cursor.description` which it uses to list the columns
  * `TableView.data` uses it to lookup columns and primary keys and the `table_rows_count` (used if the thing isn't a view) and probably a few more things, this method is huge!
  * `RowView.data` uses it for primary keys
  * `foreign_key_tables()` uses it for foreign keys

  In the tests it's used by `test_api.test_inspect_json()` and by a couple of tests in `test_inspect`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/420#issuecomment-474399630
issue_url: https://api.github.com/repos/simonw/datasette/issues/420
node_id: MDEyOklzc3VlQ29tbWVudDQ3NDM5OTYzMA==
created_at: 2019-03-19T14:38:14Z
updated_at: 2019-03-19T14:38:14Z
author_association: OWNER
body:
  Most of these can be replaced with relatively straight-forward direct introspection of the SQLite table.

  The one exception is the incoming foreign keys: these can only be found by inspecting ALL of the other tables. 

  This requires running `PRAGMA foreign_key_list([table_name])` against every other table in the database. How expensive is doing this on a database with hundreds of tables?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/420#issuecomment-474407617
issue_url: https://api.github.com/repos/simonw/datasette/issues/420
node_id: MDEyOklzc3VlQ29tbWVudDQ3NDQwNzYxNw==
created_at: 2019-03-19T14:55:51Z
updated_at: 2019-03-19T14:55:51Z
author_association: OWNER
body:
  A microbenchmark against `fivethirtyeight.db` (415 tables):

      In [1]: import sqlite3                                                                                              
      In [2]: c = sqlite3.connect("fivethirtyeight.db")                                                                   
      In [3]: %timeit c.execute("select name from sqlite_master where type = 'table'").fetchall()                         
      283 µs ± 12.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
      In [4]: tables = [r[0] for r in c.execute("select name from sqlite_master where type = 'table'").fetchall()]        
      In [5]: len(tables)                                                                                                 
      Out[5]: 415
      In [6]: %timeit [c.execute("pragma foreign_keys([{}])".format(t)).fetchall() for t in tables]                       
      1.81 ms ± 161 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

  So running `pragma foreign_keys()` against 415 tables only takes 1.81ms. This is going to be fine.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/422#issuecomment-474888132
issue_url: https://api.github.com/repos/simonw/datasette/issues/422
node_id: MDEyOklzc3VlQ29tbWVudDQ3NDg4ODEzMg==
created_at: 2019-03-20T15:34:37Z
updated_at: 2019-03-20T15:34:37Z
author_association: OWNER
body:
  Here's a trick for lower bound counts which looks like it might actually work. Consider the following queries:

  ```
  select count(*) from (
    select rowid from [most-common-name/surnames] limit 1000
  )
  ```
  https://fivethirtyeight.datasettes.com/fivethirtyeight-b76415d?sql=select+count%28*%29+from+%28%0D%0A++select+rowid+from+%5Bmost-common-name%2Fsurnames%5D+limit+1000%0D%0A%29

  Takes 0.827ms (it took longer with `select * from` in the subquery).

  Same query but with limit 10,000:

  https://fivethirtyeight.datasettes.com/fivethirtyeight-b76415d?sql=select+count%28*%29+from+%28%0D%0A++select+rowid++from+%5Bmost-common-name%2Fsurnames%5D+limit+10000%0D%0A%29

  Took 2.335ms

  With 100,000 limit:

  https://fivethirtyeight.datasettes.com/fivethirtyeight-b76415d?sql=select+count%28*%29+from+%28%0D%0A++select+rowid++from+%5Bmost-common-name%2Fsurnames%5D+limit+100000%0D%0A%29

  Took 27.558ms

  So one solution here would be to pick an upper bound (maybe 100,001) and use this query, which should give an accurate count below that upper bound but allow us to show "100,000+" as a count if the table exceeds that boundary.

  Maybe the boundary is a config setting? Also, if a tighter timeout (maybe 20ms) is exceeded for that boundary we could halve it and try again.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/422#issuecomment-474890920
issue_url: https://api.github.com/repos/simonw/datasette/issues/422
node_id: MDEyOklzc3VlQ29tbWVudDQ3NDg5MDkyMA==
created_at: 2019-03-20T15:39:58Z
updated_at: 2019-03-20T15:39:58Z
author_association: OWNER
body:
  The page with the most table counts on it is the index page: https://fivethirtyeight.datasettes.com/fivethirtyeight-b76415d

  <img width="1032" alt="Screen Shot 2019-03-20 at 8 36 00 AM" src="https://user-images.githubusercontent.com/9599/54697753-5710ce80-4aeb-11e9-89aa-13f843a78d2c.png">

  If I paginate this (which needs to happen anyway for Datasette Library #417) the impact here won't be as bad. I could even load in the table row counts asynchronously via JavaScript?

  Bigger problem is this total summed count representation on the homepage:

  <img width="1022" alt="Screen Shot 2019-03-20 at 8 38 53 AM" src="https://user-images.githubusercontent.com/9599/54697913-a3f4a500-4aeb-11e9-97a4-be501d737642.png">

  I think that feature just won't be feasibly against large databases in a mutable world. Maybe we consider to show that total but only for immutable databases? May be easier just to drop it entirely (we will still show the table count).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/420#issuecomment-477633354
issue_url: https://api.github.com/repos/simonw/datasette/issues/420
node_id: MDEyOklzc3VlQ29tbWVudDQ3NzYzMzM1NA==
created_at: 2019-03-28T15:01:37Z
updated_at: 2019-03-28T15:01:37Z
author_association: OWNER
body:
  I started looking at how I would implement `table_exists()` with a direct call that uses `sqlite-utils` to see if a table exists.

  https://github.com/simonw/datasette/blob/82fec6048148b58748040a7e2caa163387e982a3/datasette/app.py#L303-L304

  `sqlite-utils` needs access to the database connection - but the database connection itself is currently only available in code that runs in a thread inside the `.execute()` method:

  https://github.com/simonw/datasette/blob/82fec6048148b58748040a7e2caa163387e982a3/datasette/app.py#L413-L426

  So I'm going to need to refactor this a bit. I think I need a way to say "here is a function which needs access to the connection object for database named X - run that function in a thread, give it access to that connection and then give me back the result".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/420#issuecomment-477636768
issue_url: https://api.github.com/repos/simonw/datasette/issues/420
node_id: MDEyOklzc3VlQ29tbWVudDQ3NzYzNjc2OA==
created_at: 2019-03-28T15:09:27Z
updated_at: 2019-03-28T15:09:27Z
author_association: OWNER
body:
  Even more tricky: `table_exists()` is currently a synchronous function. If it's going to be executing a SQL query it needs to become an async function.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/420#issuecomment-478391708
issue_url: https://api.github.com/repos/simonw/datasette/issues/420
node_id: MDEyOklzc3VlQ29tbWVudDQ3ODM5MTcwOA==
created_at: 2019-03-31T22:33:32Z
updated_at: 2019-03-31T22:34:02Z
author_association: OWNER
body:
  Next I need to fix this:

  https://github.com/simonw/datasette/blob/0209a0a344503157351e625f0629b686961763c9/datasette/app.py#L420-L435

  Given the name of the database (from the URL e.g. https://latest.datasette.io/fixtures) I need to figure out what name I used to cache the collection.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/420#issuecomment-478393116
issue_url: https://api.github.com/repos/simonw/datasette/issues/420
node_id: MDEyOklzc3VlQ29tbWVudDQ3ODM5MzExNg==
created_at: 2019-03-31T22:52:48Z
updated_at: 2019-03-31T22:52:48Z
author_association: OWNER
body:
  This means the `Datasette` class needs a new property, keeping track of all of the connected databases.

  ```
  ds.databases = {
      "name_used_in_urls": {
          "type": "file", # or "memory"
          "path": filepath # or None if memory
          "mutable": True # or False,
          "hash": "..." # or None if mutable
      }
  }
  ```

  Maybe these should be objects, not dictionaries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/420#issuecomment-480552387
issue_url: https://api.github.com/repos/simonw/datasette/issues/420
node_id: MDEyOklzc3VlQ29tbWVudDQ4MDU1MjM4Nw==
created_at: 2019-04-07T02:06:20Z
updated_at: 2019-04-07T02:06:20Z
author_association: OWNER
body:
  `expand_foreign_keys()` relies on the `.inspect()` command having automatically derived the `label_column` for a table, which it does using this code:

  https://github.com/simonw/datasette/blob/97331f3435ba1583a0f9dbcaffc25de8894cf1f8/datasette/inspect.py#L34-L42

  This needs access to the column names for the table. I think we can drop this entirely in favour of a new utility function - and that function can incorporate the metadata check as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/420#issuecomment-480556166
issue_url: https://api.github.com/repos/simonw/datasette/issues/420
node_id: MDEyOklzc3VlQ29tbWVudDQ4MDU1NjE2Ng==
created_at: 2019-04-07T03:35:59Z
updated_at: 2019-04-07T03:48:14Z
author_association: OWNER
body:
  Still need to solve: `TableView.data()` - but this is the one with a row count in hence the need to solve #422 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/425#issuecomment-480558501
issue_url: https://api.github.com/repos/simonw/datasette/issues/425
node_id: MDEyOklzc3VlQ29tbWVudDQ4MDU1ODUwMQ==
created_at: 2019-04-07T04:32:28Z
updated_at: 2019-04-07T04:32:28Z
author_association: OWNER
body:
  Here's the problem:

  https://github.com/simonw/datasette/blob/6f6d0ff2b41f1cacaf42287b1b230b646bcba9ee/datasette/templates/query.html#L30-L36

  Need an else block here that adds the SQL as a hidden form field.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/18#issuecomment-480621924
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/18
node_id: MDEyOklzc3VlQ29tbWVudDQ4MDYyMTkyNA==
created_at: 2019-04-07T19:31:42Z
updated_at: 2019-04-07T19:31:42Z
author_association: NONE
body:
  I've just noticed that SQLite lets you IGNORE inserts that collide with a pre-existing key. This can be quite handy if you have a dataset that keeps changing in part, and you don't want to upsert and replace pre-existing PK rows but you do want to ignore collisions to existing PK rows.

  Do `sqlite_utils` support such (cavalier!) behaviour?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/420#issuecomment-481310295
issue_url: https://api.github.com/repos/simonw/datasette/issues/420
node_id: MDEyOklzc3VlQ29tbWVudDQ4MTMxMDI5NQ==
created_at: 2019-04-09T15:50:52Z
updated_at: 2019-04-09T15:50:52Z
author_association: OWNER
body:
  Efficient row counts are even more important for the `DatabaseView` and `IndexView` pages.

  The row counts on those pages don't have to be precise, so one option is for me to calculate them and cache them occasionally. I could even have a dedicated thread which just does the counting?

  In #422 I've figured out a mechanism for getting accurate or lower-bound counts within a time limit (accurate if possible, lower-bound otherwise).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/359#issuecomment-481724452
issue_url: https://api.github.com/repos/simonw/datasette/issues/359
node_id: MDEyOklzc3VlQ29tbWVudDQ4MTcyNDQ1Mg==
created_at: 2019-04-10T14:52:59Z
updated_at: 2019-04-10T14:52:59Z
author_association: OWNER
body:
  I'm going to go with `?_facet_array=definitions` as the querystring argument for this.

  And `?definitions_arraycontains=foo` as the filter argument.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/359#issuecomment-481939013
issue_url: https://api.github.com/repos/simonw/datasette/issues/359
node_id: MDEyOklzc3VlQ29tbWVudDQ4MTkzOTAxMw==
created_at: 2019-04-11T02:17:55Z
updated_at: 2019-04-11T02:17:55Z
author_association: OWNER
body:
  Challenge: facets can also be defined in `metadata.json` like this:

  ```
  {
    "databases": {
      "sf-trees": {
        "tables": {
          "Street_Tree_List": {
            "facets": ["qLegalStatus"]
          }
        }
      }
    }
  }
  ```

  But... `?_facet_array=definitions` doesn't fit in that data structure. Need to have an alternative mechanism for defining this kind of facet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/427#issuecomment-481940539
issue_url: https://api.github.com/repos/simonw/datasette/issues/427
node_id: MDEyOklzc3VlQ29tbWVudDQ4MTk0MDUzOQ==
created_at: 2019-04-11T02:26:43Z
updated_at: 2019-04-11T02:26:43Z
author_association: OWNER
body:
  I quite like the Solr idea. It could look like this for Datasette:

  `?_facet=name` - default behaviour, same as today. But that's actually an alias for `?_facet.name=name` - which defines a name for the facet.

  `?_facet.tags.array=tags` - would define a facet called `tags` that uses an array facet against the `tags` column.

  I don't like the need to say `tags` twice in that though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/427#issuecomment-481957014
issue_url: https://api.github.com/repos/simonw/datasette/issues/427
node_id: MDEyOklzc3VlQ29tbWVudDQ4MTk1NzAxNA==
created_at: 2019-04-11T04:05:07Z
updated_at: 2019-04-11T04:05:07Z
author_association: OWNER
body:
  OK, I have a plan:

  `?_facet=foo`
  `?_facet_facettype=options`

  Options here can be one of the following:

  - A single value which is the name of a table
  - A comma separated list of options
  - A JSON object starting with `{` or `[`

  If the table name itself contains a `,`, `{` or `]` then you have to escape it by putting it in a JSON object, `?_facet_percentile={"column":"{this_is,a_weird[column_name"}` for example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/427#issuecomment-481957313
issue_url: https://api.github.com/repos/simonw/datasette/issues/427
node_id: MDEyOklzc3VlQ29tbWVudDQ4MTk1NzMxMw==
created_at: 2019-04-11T04:07:00Z
updated_at: 2019-04-11T04:07:40Z
author_association: OWNER
body:
  This means the `metadata.json` format can look like this:
  ```
  {
    "databases": {
      "sf-trees": {
        "tables": {
          "Street_Tree_List": {
            "facets": ["qLegalStatus", {"array": "tags"}, {"percentile": {"blah": "options"}}]
          }
        }
      }
    }
  }
  ```

  So any advanced facets are represented here as a dictionary with a single key - the type - that maps to the options.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/428#issuecomment-482424314
issue_url: https://api.github.com/repos/simonw/datasette/issues/428
node_id: MDEyOklzc3VlQ29tbWVudDQ4MjQyNDMxNA==
created_at: 2019-04-12T03:33:35Z
updated_at: 2019-04-12T03:33:35Z
author_association: OWNER
body:
  It looks like I accidentally broke the `fts_table` metadata mechanism here: https://github.com/simonw/datasette/commit/3a208a41d4dce35b97eca8b25f37055c3fda5aed#diff-5e0ffd62fced7d46339b9b2cd167c2f9L297

  I'll fix that as part of this work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/428#issuecomment-482433121
issue_url: https://api.github.com/repos/simonw/datasette/issues/428
node_id: MDEyOklzc3VlQ29tbWVudDQ4MjQzMzEyMQ==
created_at: 2019-04-12T04:30:29Z
updated_at: 2019-04-12T04:30:29Z
author_association: OWNER
body:
  Configured in metadata: https://latest.datasette.io/fixtures/searchable_view_configured_by_metadata?_search=weasel

  Configured with querystring: https://latest.datasette.io/fixtures/searchable_view?_search=weasel&_fts_table=searchable_fts&_fts_pk=pk

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/410#issuecomment-482434925
issue_url: https://api.github.com/repos/simonw/datasette/issues/410
node_id: MDEyOklzc3VlQ29tbWVudDQ4MjQzNDkyNQ==
created_at: 2019-04-12T04:42:27Z
updated_at: 2019-04-12T04:42:27Z
author_association: OWNER
body:
  You can pass multiple databases as command-line arguments, and each one will be loaded. For example:

      datasette mydb.db otherdb.db

  Then the URLs should be:

  http://127.0.0.1:8001/mydb for the first one
  http://127.0.0.1:8001/otherdb for the second one

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/425#issuecomment-482439686
issue_url: https://api.github.com/repos/simonw/datasette/issues/425
node_id: MDEyOklzc3VlQ29tbWVudDQ4MjQzOTY4Ng==
created_at: 2019-04-12T05:12:12Z
updated_at: 2019-04-12T05:12:12Z
author_association: OWNER
body:
  Fixed: https://latest.datasette.io/fixtures?sql=select+*+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101&_hide_sql=1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/356#issuecomment-482620313
issue_url: https://api.github.com/repos/simonw/datasette/issues/356
node_id: MDEyOklzc3VlQ29tbWVudDQ4MjYyMDMxMw==
created_at: 2019-04-12T15:35:44Z
updated_at: 2019-04-12T15:35:44Z
author_association: OWNER
body:
  One question here is how these facets should be defined in the table page query string.

  #427 started exploring this.

  For any m2m facet we need to know:

  - what is the join table?
  - how is the join table related to our current table?
  - what is the table on the other side of the relationship?
  - how does that table relate to the join table?
  - how should that table be displayed (what's the label column)?

  The simplest form of m2m relationship can be automatically derived from just knowing the table. We can support that like so:

  ?_facet_m2m=tagged

  This could work automatically if the following constraints turn out to apply:

  - the tagged table has a foreign key back to our table, against our primary key 
  - the tagged table has a single other foreign key to one other table
  - that other table has a single text column we can use as the label (or has a label column defined in metadata)

  If any of the above rules don't hold, I think the solution is to have explicit configuration. Per #427 this will likely be done using JSON in the query string. Something like this (would be one line but indented for readability):

  ```
  ?_facet_m2m={
    "through":"tagged",
    "through_fk_us":"tree_id",
    "other":"tags",
    "through_fk_other":"tag_id",
    "other_label": "tag"
  }
  ```

  Probably also need a way of specifying the outbound column used on both us and other - if the m2m table isn't linking to the foreign keys.

  I don't yet like the names of the above keys.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/427#issuecomment-482626534
issue_url: https://api.github.com/repos/simonw/datasette/issues/427
node_id: MDEyOklzc3VlQ29tbWVudDQ4MjYyNjUzNA==
created_at: 2019-04-12T15:52:53Z
updated_at: 2019-04-12T15:52:53Z
author_association: OWNER
body:
  I just realized: a key part of faceting is being able to correctly apply the facet (and know that it has been applied).

  Existing facets are exact match only, so they can be applied and detected with ?foo=bar

  More advanced facets like _facet_array and _facet_m2m will need different ways of applying themselves. This needs to be bundled up in the new Facet abstraction somehow.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/427#issuecomment-482627099
issue_url: https://api.github.com/repos/simonw/datasette/issues/427
node_id: MDEyOklzc3VlQ29tbWVudDQ4MjYyNzA5OQ==
created_at: 2019-04-12T15:54:41Z
updated_at: 2019-04-12T15:54:41Z
author_association: OWNER
body:
  Bonus idea: since we are having a Facet abstraction we should allow additional facet type apps to be registered using a plugin.

  Fun idea for a (very inefficient) demo plugin: facet-by-emoji! Would work by counting all emoji in text fields using a horrible slow full-scan regular expression, then would apply selected emoji facets using a LIKE query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/429#issuecomment-482628978
issue_url: https://api.github.com/repos/simonw/datasette/issues/429
node_id: MDEyOklzc3VlQ29tbWVudDQ4MjYyODk3OA==
created_at: 2019-04-12T16:00:04Z
updated_at: 2019-04-12T16:00:04Z
author_association: OWNER
body:
  I originally thought of this as a plugin but then realized that it's 100% compatible with Datasette's existing arbitrary SQL clauses and would make some of my other projects (especially involving custom queries that still need faceting) a whole lot easier.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/429#issuecomment-482638695
issue_url: https://api.github.com/repos/simonw/datasette/issues/429
node_id: MDEyOklzc3VlQ29tbWVudDQ4MjYzODY5NQ==
created_at: 2019-04-12T16:29:25Z
updated_at: 2019-04-13T01:14:17Z
author_association: OWNER
body:
  Getting a prototype working was hardly any code at all:

  http://127.0.0.1:8001/fixtures/facetable?_where=city_id+in+(select+id+from+facet_cities+where+name+like+%22%25an%25%22)

  ```
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index b7c9a4b..7ca9572 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -295,6 +295,10 @@ class TableView(RowTableShared):
           filters = Filters(sorted(other_args.items()), units, ureg)
           where_clauses, params = filters.build_where_clauses(table)
   
  +        # Add _where= from querystring
  +        if self.ds.config("allow_sql") and "_where" in request.args:
  +            where_clauses.extend(request.args["_where"])
  +
           # _search support:
           fts_table = special_args.get("_fts_table")
           fts_table = fts_table or table_metadata.get("fts_table")
  ```

  Still needed:

  - [x] Unit tests
  - [x] Probably some kind of visual display on the table page so you know that extra clauses have been added (and maybe a UI for dropping them again)

  I'm going to leave the `:named` parameter support out of the first version of this feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/429#issuecomment-482640079
issue_url: https://api.github.com/repos/simonw/datasette/issues/429
node_id: MDEyOklzc3VlQ29tbWVudDQ4MjY0MDA3OQ==
created_at: 2019-04-12T16:34:01Z
updated_at: 2019-04-12T16:34:01Z
author_association: OWNER
body:
  UI concept:

  <img width="681" alt="fixtures__facetable__10_rows" src="https://user-images.githubusercontent.com/9599/56052339-ff3e3f80-5d05-11e9-89b2-8c37056c254f.png">

  ```
  <h3>1 extra where clause:</h3>
  <p><code>city_id in (select id from facet_cities where name like "%an%")</code> [<a href="#">remove</a>]</p>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/429#issuecomment-482766801
issue_url: https://api.github.com/repos/simonw/datasette/issues/429
node_id: MDEyOklzc3VlQ29tbWVudDQ4Mjc2NjgwMQ==
created_at: 2019-04-13T01:56:19Z
updated_at: 2019-04-13T01:56:19Z
author_association: OWNER
body:
  Documentation is here: https://datasette.readthedocs.io/en/latest/json_api.html#special-table-arguments

  Demo:

  * https://latest.datasette.io/fixtures/facetable?_where=state=%22MI%22&_where=city_id=3
  * https://latest.datasette.io/fixtures/facetable?_where=city_id%20in%20(select%20id%20from%20facet_cities%20where%20name%20!=%20%22Detroit%22)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/427#issuecomment-482864457
issue_url: https://api.github.com/repos/simonw/datasette/issues/427
node_id: MDEyOklzc3VlQ29tbWVudDQ4Mjg2NDQ1Nw==
created_at: 2019-04-13T18:51:44Z
updated_at: 2019-04-13T18:57:51Z
author_association: OWNER
body:
  A facet needs to:
  - given a sql query and a list of configs, return a list of buckets
    - Know how to generate URLs for selecting and deselecting a filter (along with underlying filter application sql logic)
    - Tell if a specific filter is currently selected or not
    - Set a time limit and report if it times out
    - Generate human readable labels
    - In some cases: expand foreign keys - which means they need access to foreign key information - just the name of the table and the name of the column is enough to call `expand_foreign_keys()` (I [moved that](https://github.com/simonw/datasette/commit/274ef43bb7b129ddc2e68805b4f4ff3776fb9503) to the Datasette class to make it easier to access)
  - Make suggestions for facets. Let's give it access to the whole table here so it could either run against each column in return and rely with a list of suggestions or it could spot eg a latitude and a longitude column

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/427#issuecomment-482864837
issue_url: https://api.github.com/repos/simonw/datasette/issues/427
node_id: MDEyOklzc3VlQ29tbWVudDQ4Mjg2NDgzNw==
created_at: 2019-04-13T18:53:43Z
updated_at: 2019-04-13T18:53:43Z
author_association: OWNER
body:
  `TableView.data` is currently the longest, hairiest method in the codebase. It's 775 - 177 = 598 lines of code! Extracting faceting logic should help reduce that quite a bit.

  https://github.com/simonw/datasette/blob/274ef43bb7b129ddc2e68805b4f4ff3776fb9503/datasette/views/table.py#L177-L775

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/427#issuecomment-482865424
issue_url: https://api.github.com/repos/simonw/datasette/issues/427
node_id: MDEyOklzc3VlQ29tbWVudDQ4Mjg2NTQyNA==
created_at: 2019-04-13T18:56:25Z
updated_at: 2019-04-13T19:42:08Z
author_association: OWNER
body:
  I think there's a `Facet` base class.

  `class ColumnFacet(Facet):` is the default behaviour we have today

  `class ArrayFacet(Facet):` facet by JSON array

  `class ManyToManyFacet(Facet):` facet by M2M table

  `class DateFacet(Facet):` facet by date

  `class DateTimeFacet(Facet):` facet by datetime

  `class EmojiFacet(Facet):` super-fun demo plugin I have planned

  Could even have a facet against a numerical column which loads the entire set of column values into numpy or pandas and calculates complex statistics facets in memory .

  There’s actually a lot of potential for Datasette plugins that load several MBs of  data and analyze using other Python libraries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/431#issuecomment-482872210
issue_url: https://api.github.com/repos/simonw/datasette/issues/431
node_id: MDEyOklzc3VlQ29tbWVudDQ4Mjg3MjIxMA==
created_at: 2019-04-13T19:37:57Z
updated_at: 2019-04-13T19:37:57Z
author_association: OWNER
body:
  You should be able to see the reload happening in the console logs:

  <img width="1045" alt="tmp_—_sqlite3_fixtures_db_—_80×25_and_tmp_—_Python_◂_datasette_--reload_fixtures_db_-p_8052_—_154×57_and_datasette_—_python3_7__usr_local_Cellar_pipenv_2018_11_26_libexec_bin_pipenv_shell_▸_bash_—_147×43" src="https://user-images.githubusercontent.com/9599/56084524-d347ba80-5de8-11e9-9fc8-6e902244a92a.png">

  I'm doing some work at the moment to handle mutating files MUCH better - #419 - my goal is to have Datasette work against SQLite files that are being updated out-of-the box, and change the current immutable behaviour to be an option rather than the default.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/432#issuecomment-482876432
issue_url: https://api.github.com/repos/simonw/datasette/issues/432
node_id: MDEyOklzc3VlQ29tbWVudDQ4Mjg3NjQzMg==
created_at: 2019-04-13T20:06:32Z
updated_at: 2019-04-13T20:06:32Z
author_association: OWNER
body:
  This has a bug which isn't being caught by the unit tests (yet) - facet suggestion suggests facets that have already been enabled:

  <img width="845" alt="fixtures__facetable__15_rows" src="https://user-images.githubusercontent.com/9599/56084775-eceb0100-5dec-11e9-8d98-24f1ea1862eb.png">

  There is also a test failure due to missing plugin hook documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/8#issuecomment-482994231
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/8
node_id: MDEyOklzc3VlQ29tbWVudDQ4Mjk5NDIzMQ==
created_at: 2019-04-14T15:04:07Z
updated_at: 2019-04-14T15:29:33Z
author_association: NONE
body:
  PLEASE IGNORE THE BELOW... I did a package update and rebuilt the kernel I was working in... may just have been an old version of sqlite_utils, seems to be working now. (Too many containers / too many environments!)


  Has an issue been reintroduced here with FTS? eg I'm getting an error thrown by spaces in column names here:

  ```
  /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)

  def enable_fts(self, columns, fts_version="FTS5"):
  --> 329         "Enables FTS on the specified columns"
      330         sql = """
      331             CREATE VIRTUAL TABLE "{table}_fts" USING {fts_version} (
  ```

  when trying an `insert_all`.

  Also, if a col has a `.` in it, I seem to get:

  ```
  /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)
      327                     jsonify_if_needed(record.get(key, None)) for key in all_columns
      328                 )
  --> 329             result = self.db.conn.execute(sql, values)
      330             self.db.conn.commit()
      331             self.last_id = result.lastrowid

  OperationalError: near ".": syntax error
  ```

  (Can't post a worked minimal example right now; racing trying to build something against a live timing screen that will stop until next weekend in an hour or two...)

  PS Hmmm I did a test and they seem to work; I must be messing up s/where else...

  ```
  import sqlite3
  from sqlite_utils import Database

  dbname='testingDB_sqlite_utils.db'

  #!rm $dbname
  conn = sqlite3.connect(dbname, timeout=10)


  #Setup database tables
  c = conn.cursor()

  setup='''
  CREATE TABLE IF NOT EXISTS  "test1" (
      "NO" INTEGER,
      "NAME" TEXT
  );

  CREATE TABLE IF NOT EXISTS  "test2" (
      "NO" INTEGER,
      `TIME OF DAY` TEXT
  );

  CREATE TABLE IF NOT EXISTS  "test3" (
      "NO" INTEGER,
      `AVG. SPEED (MPH)` FLOAT
  );
  '''

  c.executescript(setup)


  DB = Database(conn)

  import pandas as pd

  df1 = pd.DataFrame({'NO':[1,2],'NAME':['a','b']})
  DB['test1'].insert_all(df1.to_dict(orient='records'))

  df2 = pd.DataFrame({'NO':[1,2],'TIME OF DAY':['early on','late']})
  DB['test2'].insert_all(df2.to_dict(orient='records'))

  df3 = pd.DataFrame({'NO':[1,2],'AVG. SPEED (MPH)':['123.3','123.4']})
  DB['test3'].insert_all(df3.to_dict(orient='records'))
  ```

  all seem to work ok. I'm still getting errors in my set up though, which is not too different to the text cases?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/288#issuecomment-483106262
issue_url: https://api.github.com/repos/simonw/datasette/issues/288
node_id: MDEyOklzc3VlQ29tbWVudDQ4MzEwNjI2Mg==
created_at: 2019-04-15T04:48:59Z
updated_at: 2019-04-15T04:48:59Z
author_association: OWNER
body:
  This has got more urgent now that I've added the `?column__arraycontains=foo` filter as part of the effort to implement facet-by-array for #359

  I added that filter in https://github.com/simonw/datasette/commit/78e45ead4d771007c57b307edf8fc920101f8733 but it can only be applied once - for proper faceting this needs to work:

  https://latest.datasette.io/fixtures/facetable?tags__arraycontains=tag1&tags__arraycontains=tag2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/429#issuecomment-483202658
issue_url: https://api.github.com/repos/simonw/datasette/issues/429
node_id: MDEyOklzc3VlQ29tbWVudDQ4MzIwMjY1OA==
created_at: 2019-04-15T10:48:01Z
updated_at: 2019-04-15T10:48:01Z
author_association: CONTRIBUTOR
body:
  Minor UI observation:

  ![image](https://user-images.githubusercontent.com/82988/56127017-2bf78e80-5f74-11e9-9120-9393eb5d4988.png)

  `_where=` renders a `[remove]` link whereas `_facet=` gets a cross to remove it. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/288#issuecomment-483458569
issue_url: https://api.github.com/repos/simonw/datasette/issues/288
node_id: MDEyOklzc3VlQ29tbWVudDQ4MzQ1ODU2OQ==
created_at: 2019-04-15T23:45:04Z
updated_at: 2019-04-15T23:45:04Z
author_association: OWNER
body:
  https://latest.datasette.io/fixtures/facetable?tags__arraycontains=tag1&tags__arraycontains=tag2 now works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/432#issuecomment-484584234
issue_url: https://api.github.com/repos/simonw/datasette/issues/432
node_id: MDEyOklzc3VlQ29tbWVudDQ4NDU4NDIzNA==
created_at: 2019-04-18T16:33:52Z
updated_at: 2019-04-18T16:33:52Z
author_association: OWNER
body:
  It would be nice to decouple the `request` object from the `Facet` class. The request is needed for two things at the moment:

  * To decide if a specific facet bucket has been selected or not
  * To construct the `toggle_url` for turning the selection on or off

  Can I pull those needs out of the Facet class somehow?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/434#issuecomment-484694648
issue_url: https://api.github.com/repos/simonw/datasette/issues/434
node_id: MDEyOklzc3VlQ29tbWVudDQ4NDY5NDY0OA==
created_at: 2019-04-18T21:23:56Z
updated_at: 2019-04-18T21:23:56Z
author_association: OWNER
body:
  Thanks for looking into this!

  To clarify: currently, the Dockerfile that we generate looks something like this:

  ```
  CMD ["datasette", "serve", "--host", "0.0.0.0", "fixtures.db", "--cors", "--port", "8001"]
  ```

  Your code here changes that CMD line to look like this instead, in order to set the port based on an environment variable:

  ```
  CMD ["sh", "-c", "datasette serve --port $PORT ..."]
  ```

  I wonder if this is the only way to do this?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/434#issuecomment-484699119
issue_url: https://api.github.com/repos/simonw/datasette/issues/434
node_id: MDEyOklzc3VlQ29tbWVudDQ4NDY5OTExOQ==
created_at: 2019-04-18T21:40:45Z
updated_at: 2019-04-18T21:40:45Z
author_association: OWNER
body:
  I asked @andrewgodwin about this and he confirmed that if we want to read an environment variable we can't use the `CMD [...]` syntax in the way that we were using it.

  He did suggest that if we're doing `CMD ["sh", "-c", "datasette serve --port $PORT ..."]` we may as well do this instead:

  `CMD "datasette serve --port $PORT ..."`

  We should apply some command-line escaping here - if the user passes `--version-note=hello$there` to `datasette publish` we need that $ not to be accidentally evaluated as an environment variable.

  It looks like [shlex.quote](https://docs.python.org/dev/library/shlex.html#shlex.quote) is the right way to do that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/435#issuecomment-485269362
issue_url: https://api.github.com/repos/simonw/datasette/issues/435
node_id: MDEyOklzc3VlQ29tbWVudDQ4NTI2OTM2Mg==
created_at: 2019-04-21T17:38:05Z
updated_at: 2019-04-21T17:38:05Z
author_association: OWNER
body:
  I built a first version of this in https://github.com/simonw/datasette/commit/7d01ca34a10b5f8a993859cfd05790eb2870b94e which dumped SQL queries out to the terminal logs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/435#issuecomment-485270230
issue_url: https://api.github.com/repos/simonw/datasette/issues/435
node_id: MDEyOklzc3VlQ29tbWVudDQ4NTI3MDIzMA==
created_at: 2019-04-21T17:52:03Z
updated_at: 2019-04-21T17:52:03Z
author_association: OWNER
body:
  Demos:

  * https://latest.datasette.io/fixtures/facetable?_trace=1 - SQL dumped at bottom of HTML
  * https://latest.datasette.io/fixtures/facetable.json?_trace=1 - SQL in a "_traces" key

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/439#issuecomment-487542486
issue_url: https://api.github.com/repos/simonw/datasette/issues/439
node_id: MDEyOklzc3VlQ29tbWVudDQ4NzU0MjQ4Ng==
created_at: 2019-04-29T11:20:30Z
updated_at: 2019-04-29T11:20:30Z
author_association: CONTRIBUTOR
body:
  Actually I think this is not the whole story because of the rowid issue. I'm going to think about this one a bit more.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/441#issuecomment-487686655
issue_url: https://api.github.com/repos/simonw/datasette/issues/441
node_id: MDEyOklzc3VlQ29tbWVudDQ4NzY4NjY1NQ==
created_at: 2019-04-29T18:14:25Z
updated_at: 2019-04-29T18:14:25Z
author_association: CONTRIBUTOR
body:
  Subsidiary note which I forgot in the commit message:

  I've decided to give each view a short string name to aid in differentiating which view a hook is being called from. Since hooks are functions and not subclasses, and can get called from different places in the URL hierarchy, it's sometimes difficult to distinguish what data you're actually operating on. I think this will come in handy for other hooks as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/424#issuecomment-487689477
issue_url: https://api.github.com/repos/simonw/datasette/issues/424
node_id: MDEyOklzc3VlQ29tbWVudDQ4NzY4OTQ3Nw==
created_at: 2019-04-29T18:22:40Z
updated_at: 2019-04-29T18:22:40Z
author_association: CONTRIBUTOR
body:
  This is pretty conflicty because I forgot how to use git fetch. If you're interested in merging this I'll rewrite it against an actual modern checkout...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/424#issuecomment-487692377
issue_url: https://api.github.com/repos/simonw/datasette/issues/424
node_id: MDEyOklzc3VlQ29tbWVudDQ4NzY5MjM3Nw==
created_at: 2019-04-29T18:30:46Z
updated_at: 2019-04-29T18:30:46Z
author_association: CONTRIBUTOR
body:
  Actually no, I ended up not using the inspected column types in my plugin, and the binary column issue can be solved a lot more simply, so I'll close this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/441#issuecomment-487723476
issue_url: https://api.github.com/repos/simonw/datasette/issues/441
node_id: MDEyOklzc3VlQ29tbWVudDQ4NzcyMzQ3Ng==
created_at: 2019-04-29T20:05:23Z
updated_at: 2019-04-29T20:05:23Z
author_association: CONTRIBUTOR
body:
  This is the minimal example (I also included it in the docs):

  ```python
  from datasette import hookimpl

  def render_test(args, data, view_name):
      return {
         'body': 'Hello World',
         'content_type': 'text/plain'
      }

  @hookimpl
  def register_output_renderer():
      return {
          'extension': 'test',
          'callback': render_test
      }
  ```

  I'm working on the GeoJSON one now and it should be ready soon. (I forgot I was going to run into the same problem as before - that Spatialite's stupid binary format isn't WKB and I have no way of altering the query to change that - but I've just managed to write some code to rearrange the bytes from Spatialite blob-geometry into WKB...)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/441#issuecomment-487724539
issue_url: https://api.github.com/repos/simonw/datasette/issues/441
node_id: MDEyOklzc3VlQ29tbWVudDQ4NzcyNDUzOQ==
created_at: 2019-04-29T20:08:32Z
updated_at: 2019-04-29T20:08:32Z
author_association: CONTRIBUTOR
body:
  I also just realised that I should be passing the datasette object into the hook function...as I just found I need it. So hold off merging until I've fixed that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/441#issuecomment-487735247
issue_url: https://api.github.com/repos/simonw/datasette/issues/441
node_id: MDEyOklzc3VlQ29tbWVudDQ4NzczNTI0Nw==
created_at: 2019-04-29T20:39:43Z
updated_at: 2019-04-29T20:39:43Z
author_association: CONTRIBUTOR
body:
  I updated the hook to pass the datasette object through now.

  You can see the working [GeoJSON render function here](https://github.com/russss/datasette-geo/blob/master/datasette_plugin_geo/geojson.py) - the [hook function is here](https://github.com/russss/datasette-geo/blob/master/datasette_plugin_geo/__init__.py#L65-L70).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/441#issuecomment-487748271
issue_url: https://api.github.com/repos/simonw/datasette/issues/441
node_id: MDEyOklzc3VlQ29tbWVudDQ4Nzc0ODI3MQ==
created_at: 2019-04-29T21:20:17Z
updated_at: 2019-04-29T21:20:17Z
author_association: CONTRIBUTOR
body:
  Also I just pushed a change to add registered output renderers to the templates:
  ![image](https://user-images.githubusercontent.com/45057/56927799-f18e0580-6acc-11e9-8ea9-a0ee961323ec.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/439#issuecomment-487859345
issue_url: https://api.github.com/repos/simonw/datasette/issues/439
node_id: MDEyOklzc3VlQ29tbWVudDQ4Nzg1OTM0NQ==
created_at: 2019-04-30T08:21:19Z
updated_at: 2019-04-30T08:21:19Z
author_association: CONTRIBUTOR
body:
  I think the best approach to this is to pass through the `view_name` parameter I added in #441. It's then simple enough for me to add `.geojson` to the URL in JS - I don't need the pkey.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/438#issuecomment-488370672
issue_url: https://api.github.com/repos/simonw/datasette/issues/438
node_id: MDEyOklzc3VlQ29tbWVudDQ4ODM3MDY3Mg==
created_at: 2019-05-01T18:33:57Z
updated_at: 2019-05-01T18:33:57Z
author_association: OWNER
body:
  Yeah this is a good call. Having pytest set some kind of flag (maybe an environment variable?) that disables auto-plugin discovery is a very reasonable fix here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/441#issuecomment-488373631
issue_url: https://api.github.com/repos/simonw/datasette/issues/441
node_id: MDEyOklzc3VlQ29tbWVudDQ4ODM3MzYzMQ==
created_at: 2019-05-01T18:43:17Z
updated_at: 2019-05-01T18:43:17Z
author_association: OWNER
body:
  I plan to merge this in about 7 hours time (after work).

  I may tweak the plugin format a little bit before the next Datasette release though (best to merge first, make changes later I think) - for consistency with some other upcoming hooks (the facet hook in particular). I'll discuss that with you while I'm working on it.

  Thanks so much for this, it's a really neat addition!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/419#issuecomment-488528111
issue_url: https://api.github.com/repos/simonw/datasette/issues/419
node_id: MDEyOklzc3VlQ29tbWVudDQ4ODUyODExMQ==
created_at: 2019-05-02T01:14:58Z
updated_at: 2019-05-02T01:14:58Z
author_association: OWNER
body:
  I just closed #420 - all of the places in the codebase that were using `.inspect()` should have been eliminated.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/431#issuecomment-488555399
issue_url: https://api.github.com/repos/simonw/datasette/issues/431
node_id: MDEyOklzc3VlQ29tbWVudDQ4ODU1NTM5OQ==
created_at: 2019-05-02T05:13:54Z
updated_at: 2019-05-02T05:13:54Z
author_association: OWNER
body:
  Datasette master now treats databases as readonly but NOT immutable. This means you can make changes to those databases from another process and those changes will be instantly reflected in the Datasette interface.

  As such, reloading on database change is no longer necessary. Closing this ticket.

reactions:
  {"total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/427#issuecomment-488564761
issue_url: https://api.github.com/repos/simonw/datasette/issues/427
node_id: MDEyOklzc3VlQ29tbWVudDQ4ODU2NDc2MQ==
created_at: 2019-05-02T06:24:49Z
updated_at: 2019-05-03T00:07:16Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/compare/facet-refactor-2 is almost ready to merge now. The remaining things to do are listed as TODOs there:

  - [x] Ensure facet is not suggested if it is already active
  - [x] Don't allow facets to be hidden if they were configured in metadata.json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/427#issuecomment-488564891
issue_url: https://api.github.com/repos/simonw/datasette/issues/427
node_id: MDEyOklzc3VlQ29tbWVudDQ4ODU2NDg5MQ==
created_at: 2019-05-02T06:25:41Z
updated_at: 2019-05-02T06:25:41Z
author_association: OWNER
body:
  It would be neat to ship at least one additional face with this work - probably either `ArrayFacet` or `DateFacet`. I think `ArrayFacet` because it demonstrates the only-if-json1-enabled functionality.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/432#issuecomment-488595724
issue_url: https://api.github.com/repos/simonw/datasette/issues/432
node_id: MDEyOklzc3VlQ29tbWVudDQ4ODU5NTcyNA==
created_at: 2019-05-02T08:50:53Z
updated_at: 2019-05-02T08:50:53Z
author_association: CONTRIBUTOR
body:
  > Can I pull those needs out of the Facet class somehow?

  I was thinking that it might be handy for datasette to have a request object which wraps the Sanic Request. This could include the datasette-specific querystring decoding and the `special_args` parsing from TableView.data.

  This would mean that we could expose the request object to plugin hooks without coupling them to Sanic.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/432#issuecomment-488874364
issue_url: https://api.github.com/repos/simonw/datasette/issues/432
node_id: MDEyOklzc3VlQ29tbWVudDQ4ODg3NDM2NA==
created_at: 2019-05-03T00:04:23Z
updated_at: 2019-05-03T00:04:23Z
author_association: OWNER
body:
  Abandoning this in favour of #445 - which contains the code from this branch but updated to incorporate recent changes in master.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/419#issuecomment-489060765
issue_url: https://api.github.com/repos/simonw/datasette/issues/419
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTA2MDc2NQ==
created_at: 2019-05-03T11:07:42Z
updated_at: 2019-05-03T11:07:42Z
author_association: CONTRIBUTOR
body:
  Are you planning on removing inspect entirely? 

  I didn't spot this work before I started on datasette-geo, but ironically I think it has a use case which really needs the inspect functionality (or some replacement). 

  Datasette-geo uses it to store the bounding box of all the geographic features in the table. This is needed when rendering the map because it avoids having to send loads of tile requests for areas which are empty. 

  Even with relatively small datasets, calculating the bounding box seems to take around 5 seconds, so I don't think it's really feasible to do this on page load.

  One possible fix would be to do this on startup, and then in a thread which watches the database for changes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/434#issuecomment-489138554
issue_url: https://api.github.com/repos/simonw/datasette/issues/434
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTEzODU1NA==
created_at: 2019-05-03T15:36:48Z
updated_at: 2019-05-03T15:36:48Z
author_association: OWNER
body:
  Here's my first working deployment: https://datasette-j7hipcg4aq-uc.a.run.app/fixtures-c35b6a5/facetable?_facet_array=tags

  I deployed it using this:

      datasette publish cloudrun fixtures.db --branch=master

  The second time I ran the command I got an error:

      ERROR: (gcloud.beta.run.deploy) Deployment endpoint was not found. Perhaps the
      provided region was invalid. Set the `run/region` property to a valid region and
      retry. Ex: `gcloud config set run/region us-central1`

  So I ran the command it suggested and then everything worked:

      gcloud config set run/region us-central1
      datasette publish cloudrun fixtures.db --branch=master

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/434#issuecomment-489154360
issue_url: https://api.github.com/repos/simonw/datasette/issues/434
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTE1NDM2MA==
created_at: 2019-05-03T16:18:18Z
updated_at: 2019-05-03T16:18:18Z
author_association: OWNER
body:
  Documentation is now available here: https://datasette.readthedocs.io/en/latest/publish.html#publishing-to-google-cloud-run

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/434#issuecomment-489163939
issue_url: https://api.github.com/repos/simonw/datasette/issues/434
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTE2MzkzOQ==
created_at: 2019-05-03T16:49:45Z
updated_at: 2019-05-03T16:50:03Z
author_association: CONTRIBUTOR
body:
  > The second time I ran the command I got an error:

  > 
  > ERROR: (gcloud.beta.run.deploy) Deployment endpoint was not found. Perhaps the
  > provided region was invalid. Set the `run/region` property to a valid region and
  > retry. Ex: `gcloud config set run/region us-central1`
  > 

  Yes, I was able to reproduce this; I used to get prompted for a run region interactively by the `gcloud` tool before, but maybe this is changing? (the [documentation](https://cloud.google.com/run/docs/deploying) now assumes `run/region` is set).

  Not sure which course of action is best: making `datasette` ensure that `run/region` is set beforehand or wait a bit until the gcloud CLI stabilizes?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/446#issuecomment-489167692
issue_url: https://api.github.com/repos/simonw/datasette/issues/446
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTE2NzY5Mg==
created_at: 2019-05-03T17:02:24Z
updated_at: 2019-05-03T17:02:24Z
author_association: OWNER
body:
  I looked at using namedtuples for this but hey have one major constraint: there isn't a clean way to convert them to dictionary-style JSON: https://bugs.python.org/issue30343

  So something that uses a class which knows how to be rendered as JSON would be a better fit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/447#issuecomment-489190440
issue_url: https://api.github.com/repos/simonw/datasette/issues/447
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTE5MDQ0MA==
created_at: 2019-05-03T18:13:56Z
updated_at: 2019-05-03T18:13:56Z
author_association: OWNER
body:
  This appears to fix a very weird error we were getting just on Python 3.7-dev: https://travis-ci.org/simonw/datasette/jobs/527858613

  That weird error boiled down to `count` being `None`: 

  ```
  {
      "columns": ["pk", "distance", "frequency"],
      "name": "units",
      "count": 3,
      "hidden": False,
      "foreign_keys": {"incoming": [], "outgoing": []},
      "fts_table": None,
      "primary_keys": ["pk"],
  }
  # compared to:
  {
      "name": "units",
      "columns": ["pk", "distance", "frequency"],
      "primary_keys": ["pk"],
      "count": None,
      "hidden": False,
      "fts_table": None,
      "foreign_keys": {"incoming": [], "outgoing": []},
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/442#issuecomment-489194450
issue_url: https://api.github.com/repos/simonw/datasette/issues/442
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTE5NDQ1MA==
created_at: 2019-05-03T18:26:48Z
updated_at: 2019-05-03T18:26:48Z
author_association: OWNER
body:
  Demo here: https://latest.datasette.io/fixtures/binary_data

  I slightly tweaked the copy:

  <img width="694" alt="image" src="https://user-images.githubusercontent.com/9599/57157708-79b32a00-6daf-11e9-9544-d8660c06cf65.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/446#issuecomment-489204605
issue_url: https://api.github.com/repos/simonw/datasette/issues/446
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTIwNDYwNQ==
created_at: 2019-05-03T18:59:51Z
updated_at: 2019-05-03T18:59:59Z
author_association: OWNER
body:
  Potential design:

  ```python
  from collections import OrderedDict


  class DataSpec:
      __slots__ = []

      def __init__(self, **kwargs):
          if list(kwargs.keys()) != self.__slots__:
              raise TypeError(
                  "{}() has required arguments {} (got {})".format(
                      self.__class__.__name__, self.__slots__, list(kwargs.keys())
                  )
              )
          for key in self.__slots__:
              setattr(self, key, kwargs[key])

      def __repr__(self):
          return "<{} {}>".format(self.__class__.__name__, dict(self.as_dict()))

      def as_dict(self):
          return OrderedDict([(key, getattr(self, key)) for key in self.__slots__])


  class Output(DataSpec):
      __slots__ = ["body", "content_type", "status_code"]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/446#issuecomment-489209255
issue_url: https://api.github.com/repos/simonw/datasette/issues/446
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTIwOTI1NQ==
created_at: 2019-05-03T19:15:23Z
updated_at: 2019-05-03T19:16:34Z
author_association: OWNER
body:
  The `register_output_renderer()` hook currently returns a dictionary with `body`, `content_type` and `status_code` keys but each of these keys are optionaly. I'm tempted to make all three required to better fit this model - @russss any objections?

  Alternative would be to support default values for properties of the `DataSpec` subclass - maybe:

  `__defaults__ = {"body": "", "content_type": "text/plain", "status_code": 200}`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/446#issuecomment-489221481
issue_url: https://api.github.com/repos/simonw/datasette/issues/446
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTIyMTQ4MQ==
created_at: 2019-05-03T19:58:31Z
updated_at: 2019-05-03T19:58:31Z
author_association: CONTRIBUTOR
body:
  In this particular case I don't think there's an issue making all those required. However, I suspect we might have to allow optional values at some point - my preferred solution to russss/datasette-geo#2 would need one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/446#issuecomment-489222223
issue_url: https://api.github.com/repos/simonw/datasette/issues/446
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTIyMjIyMw==
created_at: 2019-05-03T20:01:19Z
updated_at: 2019-05-03T20:01:29Z
author_association: CONTRIBUTOR
body:
  Also I have a slight preference against (ab)using `__slots__` to enforce fields, although I have done it myself in the past. It would be possible to do this with `__setattr__` instead, although that's an implementation detail and I'm not too fussed about it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/448#issuecomment-489240609
issue_url: https://api.github.com/repos/simonw/datasette/issues/448
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTI0MDYwOQ==
created_at: 2019-05-03T21:09:13Z
updated_at: 2019-05-03T21:09:13Z
author_association: OWNER
body:
  It may be that some facet implementations (`ArrayFacet` in this case) need a way to detect if they are supported by the thing they are running against (must be a rowid table in this case) and avoid suggesting themselves if they are not compatible. This may require a change to the information we make available to the `suggest()` method (information passed to the Facet class constructor).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/367#issuecomment-489241377
issue_url: https://api.github.com/repos/simonw/datasette/issues/367
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTI0MTM3Nw==
created_at: 2019-05-03T21:12:09Z
updated_at: 2019-05-03T21:12:09Z
author_association: OWNER
body:
  Before applying this fix, GitHub showed the following statistics:

      Python 50.1% JavaScript 46.0% HTML 3.0% Other 0.9% 

  Afterwards, it shows:



      Python 92.8% HTML 5.5% CSS 1.3% Dockerfile 0.4% 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/434#issuecomment-489250828
issue_url: https://api.github.com/repos/simonw/datasette/issues/434
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTI1MDgyOA==
created_at: 2019-05-03T21:50:44Z
updated_at: 2019-05-03T21:50:44Z
author_association: OWNER
body:
  Since there's a useful error message I'm OK with revisiting this in a few weeks to see if they change the CLI tool.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/450#issuecomment-489342728
issue_url: https://api.github.com/repos/simonw/datasette/issues/450
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTM0MjcyOA==
created_at: 2019-05-04T16:37:35Z
updated_at: 2019-05-04T16:37:35Z
author_association: CONTRIBUTOR
body:
  For a bit more context: this fixes a crash with `unsupported operand type(s) for +: 'int' and 'NoneType'` on the index page for me.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/187#issuecomment-489353316
issue_url: https://api.github.com/repos/simonw/datasette/issues/187
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTM1MzMxNg==
created_at: 2019-05-04T18:36:36Z
updated_at: 2019-05-04T18:36:36Z
author_association: NONE
body:
  Hi @simonw - I just hit this issue when trying out Datasette after your PyCon talk today. Datasette is pinned to Sanic 0.7.0, but it looks like 0.8.0 added the option to remove the uvloop dependency for Windows by having an environment variable `SANIC_NO_UVLOOP` at install time. Maybe that'll be sufficient before a port to Starlette?

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/454#issuecomment-489420385
issue_url: https://api.github.com/repos/simonw/datasette/issues/454
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTQyMDM4NQ==
created_at: 2019-05-05T12:07:56Z
updated_at: 2019-05-05T12:10:13Z
author_association: OWNER
body:
  Since I want the option to store more than one host, I don't think this should be a command-line option or a `--config` setting. Instead, I'm inclined to add this to `metadata.json`.

  Maybe this should be a plugin? That way the `metadata.json` setting could look like this:
  ```
  {
      "title": "Title of this instance",
      "plugins": {
          "datasette-cors": {
              "allowed_origins": ["https://example.com"]
          }
      }
  }
  ```

  This could be implemented easily on top of ASGI #272.

  (It should probably raise an exception on startup if any of the `allowed_origins` ends with a slash e.g. `"https://example.com/"` since that's not actually a valid origin, and it's an easy mistake to make.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/454#issuecomment-489420661
issue_url: https://api.github.com/repos/simonw/datasette/issues/454
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTQyMDY2MQ==
created_at: 2019-05-05T12:11:01Z
updated_at: 2019-05-05T12:11:01Z
author_association: OWNER
body:
  Also worth considering: `Access-Control-Max-Age: 86400` support - maybe as a `"max_age"` setting for the plugin. This can reduce the number of preflight checks the browser needs to make.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/453#issuecomment-489421634
issue_url: https://api.github.com/repos/simonw/datasette/issues/453
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTQyMTYzNA==
created_at: 2019-05-05T12:24:24Z
updated_at: 2019-05-05T12:24:24Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures.json?sql=select+blah

  <img width="793" alt="Mozilla_Firefox" src="https://user-images.githubusercontent.com/9599/57193981-2c4dce80-6f0f-11e9-859e-032fb21e75ed.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/454#issuecomment-489433651
issue_url: https://api.github.com/repos/simonw/datasette/issues/454
node_id: MDEyOklzc3VlQ29tbWVudDQ4OTQzMzY1MQ==
created_at: 2019-05-05T14:52:46Z
updated_at: 2019-05-05T14:52:46Z
author_association: OWNER
body:
  I really like the idea of this as a plugin, because it will provide a great example of an ASGI plugin including how to build unit tests against Datasette plugins which actually start up a Datasette server and run some requests through it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/457#issuecomment-490979355
issue_url: https://api.github.com/repos/simonw/datasette/issues/457
node_id: MDEyOklzc3VlQ29tbWVudDQ5MDk3OTM1NQ==
created_at: 2019-05-09T16:43:29Z
updated_at: 2019-05-09T16:43:29Z
author_association: OWNER
body:
  From https://cloud.google.com/sdk/gcloud/reference/beta/run/deploy it looks like the service can be provided as an optional positional argument. How about `datasette publish cloudrun ... --service=sf-trees`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/454#issuecomment-491003082
issue_url: https://api.github.com/repos/simonw/datasette/issues/454
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTAwMzA4Mg==
created_at: 2019-05-09T17:53:45Z
updated_at: 2019-05-09T17:53:57Z
author_association: OWNER
body:
  I built a new ASGI middleware component for CORS headers which I can use to implement this: https://pypi.org/project/asgi-cors/ and https://github.com/simonw/asgi-cors 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/457#issuecomment-491038808
issue_url: https://api.github.com/repos/simonw/datasette/issues/457
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTAzODgwOA==
created_at: 2019-05-09T19:41:31Z
updated_at: 2019-05-09T19:41:31Z
author_association: OWNER
body:
  Updated docs now live at https://datasette.readthedocs.io/en/latest/publish.html#publishing-to-google-cloud-run 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/459#issuecomment-491531399
issue_url: https://api.github.com/repos/simonw/datasette/issues/459
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTUzMTM5OQ==
created_at: 2019-05-11T17:51:11Z
updated_at: 2019-05-11T17:51:11Z
author_association: OWNER
body:
  This is the publish code that needs updating:

  https://github.com/simonw/datasette/blob/e7b31ae8c1a28cab9db8e165b3f21407c2e581e6/datasette/publish/now.py#L79-L96

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/461#issuecomment-491532380
issue_url: https://api.github.com/repos/simonw/datasette/issues/461
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTUzMjM4MA==
created_at: 2019-05-11T18:06:01Z
updated_at: 2019-05-11T18:06:01Z
author_association: OWNER
body:
  I plan to add a search filter box too, but only if there are more than X (probably 10) connected databases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/450#issuecomment-491532671
issue_url: https://api.github.com/repos/simonw/datasette/issues/450
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTUzMjY3MQ==
created_at: 2019-05-11T18:10:09Z
updated_at: 2019-05-11T18:10:09Z
author_association: OWNER
body:
  I fixed this in https://github.com/simonw/datasette/commit/de005b9b7d3db72375e6b8b048d1616a98e6347a instead (hadn't seen this PR)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/459#issuecomment-491533931
issue_url: https://api.github.com/repos/simonw/datasette/issues/459
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTUzMzkzMQ==
created_at: 2019-05-11T18:29:37Z
updated_at: 2019-05-11T19:26:11Z
author_association: OWNER
body:
  This needs doing before the next release because it is interfering with my Travis code that actually runs the release: https://travis-ci.org/simonw/datasette/jobs/530597261

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/435#issuecomment-491534162
issue_url: https://api.github.com/repos/simonw/datasette/issues/435
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTUzNDE2Mg==
created_at: 2019-05-11T18:33:02Z
updated_at: 2019-05-11T18:36:33Z
author_association: OWNER
body:
  I don't like the shape of the JSON:

  ```
    "_traces": {
      "num_traces": 20,
      "traces": {
        "duration": 0.015190839767456055,
        "queries": [
          [
            "sql",
            [
              "fixtures",
              "select 1 from sqlite_master where type='table' and name=?",
              [
                "facetable.json"
              ]
            ],
            1557599406.7231224,
            1557599406.723611,
            0.4887580871582031
          ],
          [
  ```

  I want this instead:

  ```
    "_traces": {
      "num_traces": 20,
      "sum_duration_ms": 0.015190839767456055,
      "traces": [
        {
          "type": "sql",
          "database": "fixtures",
          "sql": "select 1 from sqlite_master where type='table' and name=?",
          "args": ["facetable.json"],
          "start": 1557599406.7231224,
          "end": 1557599406.723611,
          "duration_ms": 0.4887580871582031
        }

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/435#issuecomment-491541721
issue_url: https://api.github.com/repos/simonw/datasette/issues/435
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTU0MTcyMQ==
created_at: 2019-05-11T20:32:21Z
updated_at: 2019-05-11T20:32:21Z
author_association: OWNER
body:
  Demo of the finished feature: https://latest.datasette.io/fixtures/facetable?_trace=1

  <img width="1294" alt="image" src="https://user-images.githubusercontent.com/9599/57574617-2ebd9600-73f1-11e9-9a1a-458f1b7d0d66.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/462#issuecomment-491543635
issue_url: https://api.github.com/repos/simonw/datasette/issues/462
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTU0MzYzNQ==
created_at: 2019-05-11T21:03:10Z
updated_at: 2019-05-11T21:03:23Z
author_association: OWNER
body:
  `test_inspect.py` currently just contains two tests that exercise a small portion of what `.inspect()` does - I'm going to repurpose that module and have it only test the `datasette inspect` CLI command instead.

  Here's the current contents of that file: https://github.com/simonw/datasette/blob/ce09e5d2d392634eced44c3c8d603d7c628e2822/tests/test_inspect.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/462#issuecomment-491543785
issue_url: https://api.github.com/repos/simonw/datasette/issues/462
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTU0Mzc4NQ==
created_at: 2019-05-11T21:05:54Z
updated_at: 2019-05-11T21:05:54Z
author_association: OWNER
body:
  So I think `datasette inspect fixtures.db other.db` should output something like this:

  ```json
  {
    "fixtures": {
      "hash": "894870db97229e9e18b40921dc32b581da813465d672445e96e040ab2adbd229",
      "file": "fixtures.db",
      "size": 225280,
      "tables": {
        "facetable": {
          "count": 34,
        }
     }
  }
  ```

  It currently writes it out to a file called `inspect-data.json`. Should I keep that as the default behaviour or switch it to outputting to stdout instead?

  Here's the current `datasette inspect --help`:

  ```$ datasette inspect --help
  Usage: datasette inspect [OPTIONS] [FILES]...

  Options:
    --inspect-file TEXT
    --load-extension PATH  Path to a SQLite extension to load
    --help                 Show this message and exit.```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/462#issuecomment-491545952
issue_url: https://api.github.com/repos/simonw/datasette/issues/462
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTU0NTk1Mg==
created_at: 2019-05-11T21:41:49Z
updated_at: 2019-05-11T21:41:49Z
author_association: OWNER
body:
  I now need to update `datasette serve ... --inspect-data=X` to understand and correctly handle the new format.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/466#issuecomment-491548189
issue_url: https://api.github.com/repos/simonw/datasette/issues/466
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTU0ODE4OQ==
created_at: 2019-05-11T22:21:40Z
updated_at: 2019-05-11T22:21:47Z
author_association: OWNER
body:
  This is a little bit tricky. This SQL looks like it may detect Spatialite tables:
  ```sql
  select * from sqlite_master where type = "table"
  and sql like "%CREATE VIRTUAL TABLE%"
  and sql like "%USING VirtualSpatialIndex%"
  ```
  But where to put it? I think this should go in a new "checks" mechanism, where we run checks against every connected database on Datasette startup.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/466#issuecomment-491549929
issue_url: https://api.github.com/repos/simonw/datasette/issues/466
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTU0OTkyOQ==
created_at: 2019-05-11T22:55:23Z
updated_at: 2019-05-11T22:55:23Z
author_association: OWNER
body:
  To build a unit test for this I'm going to have to ship a small spatialite.db binary database as part of the git repo.

  This is because I need the tests to run even when the spatialite module is not available - but you cannot create a spatialite database without having access to that module.

  I'll include a build script in the repo for constructing that database.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/418#issuecomment-491551647
issue_url: https://api.github.com/repos/simonw/datasette/issues/418
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTU1MTY0Nw==
created_at: 2019-05-11T23:31:23Z
updated_at: 2019-05-11T23:31:23Z
author_association: OWNER
body:
  Actually right now https://latest.datasette.io/fixtures/facetable?_hash=1 redirects to https://latest.datasette.io/fixtures-000/facetable - because we are no longer calculating hashes on startup for non-immutable databases. So that's weird.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/418#issuecomment-491551702
issue_url: https://api.github.com/repos/simonw/datasette/issues/418
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTU1MTcwMg==
created_at: 2019-05-11T23:32:21Z
updated_at: 2019-05-11T23:32:21Z
author_association: OWNER
body:
  I'm going to re-open this, because some of this needs revisiting now that we aren't running `.inspect()` and hence are not calculating hashes for anything other than immutable databases (and databases are treated as mutable by default).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/231#issuecomment-491943956
issue_url: https://api.github.com/repos/simonw/datasette/issues/231
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTk0Mzk1Ng==
created_at: 2019-05-13T18:56:21Z
updated_at: 2019-05-13T18:56:21Z
author_association: OWNER
body:
  I implemented this a while ago but forgot to close the issue: https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-491944613
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDQ5MTk0NDYxMw==
created_at: 2019-05-13T18:58:19Z
updated_at: 2019-05-13T18:58:19Z
author_association: OWNER
body:
  We've grown a bunch of plugin hooks over the past two years: https://datasette.readthedocs.io/en/latest/plugins.html#plugin-hooks

  Since the plugin system will never be 100% "finished", I'm closing this in favor of the label: https://github.com/simonw/datasette/labels/plugins

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/460#issuecomment-492285114
issue_url: https://api.github.com/repos/simonw/datasette/issues/460
node_id: MDEyOklzc3VlQ29tbWVudDQ5MjI4NTExNA==
created_at: 2019-05-14T15:22:58Z
updated_at: 2019-05-14T15:22:58Z
author_association: OWNER
body:
  If a database has less than 10 tables AND I can get a full count of all 10 of them in under 10ms each then I think I'll still show the row counts.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/461#issuecomment-492296836
issue_url: https://api.github.com/repos/simonw/datasette/issues/461
node_id: MDEyOklzc3VlQ29tbWVudDQ5MjI5NjgzNg==
created_at: 2019-05-14T15:50:34Z
updated_at: 2019-05-14T15:51:11Z
author_association: OWNER
body:
  This is needed by Datasette Library #417 since that's going to demand listing a LOT of databases on the homepage.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/467#issuecomment-492883561
issue_url: https://api.github.com/repos/simonw/datasette/issues/467
node_id: MDEyOklzc3VlQ29tbWVudDQ5Mjg4MzU2MQ==
created_at: 2019-05-16T01:40:09Z
updated_at: 2019-05-16T01:40:09Z
author_association: OWNER
body:
  I'm setting X to 30 because the fixtures database currently has 26 tables (22 visible, 4 hidden) and I want to display counts for it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/467#issuecomment-492898241
issue_url: https://api.github.com/repos/simonw/datasette/issues/467
node_id: MDEyOklzc3VlQ29tbWVudDQ5Mjg5ODI0MQ==
created_at: 2019-05-16T03:02:27Z
updated_at: 2019-05-16T03:02:27Z
author_association: OWNER
body:
  I'm going to be lazy and skip the unit test for this, because I don't currently have a neat way of mocking a SQL interrupted exception to simulate a query taking too long (at least for these counts).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/460#issuecomment-492899100
issue_url: https://api.github.com/repos/simonw/datasette/issues/460
node_id: MDEyOklzc3VlQ29tbWVudDQ5Mjg5OTEwMA==
created_at: 2019-05-16T03:07:41Z
updated_at: 2019-05-16T03:07:41Z
author_association: OWNER
body:
  I'm going to sort by row counts first, but if row counts aren't available I'll fall back to number of inbound/outbound foreign keys.

  To make unit testing easier, I'll accept an undocumented ?_sort=relationships parameter

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/419#issuecomment-492903398
issue_url: https://api.github.com/repos/simonw/datasette/issues/419
node_id: MDEyOklzc3VlQ29tbWVudDQ5MjkwMzM5OA==
created_at: 2019-05-16T03:33:01Z
updated_at: 2019-05-16T03:33:01Z
author_association: OWNER
body:
  @russss sorry I only just spotted your comment here.

  I think I have an alternative suggestion for what you need to do here. It sounds to me like you need to calculate a specific piece of information against a specific database. Instead of doing this in inspect, how about having a separate tool which runs this once against the database file and writes the result into a database file there?

  I've been thinking about this pattern a bit as part of the sqlite-utils work I've been doing. It's already something that's needed for SQLite FTS support - it's no good just creating a FTS index, you have to populate it as well. In sqlite-utils world you do that like this: https://sqlite-utils.readthedocs.io/en/latest/cli.html#configuring-full-text-search

      $ sqlite-utils enable-fts mydb.db documents title summary

  But then later if you've inserted new records you have to call this:

      $ sqlite-utils populate-fts mydb.db documents title summary

  So one option here could be for `datasette-geo` to know to look for a special `datasette_geo_bounding_box` database table and, if it's missing, to calculate at runtime (probably once on startup and then cache it).

  Another option: Datasette now has an option to open a database file in "immutable" mode, using `datasette -i mydatabase.db`. When you do that we calculate counts on startup - and we'll also be able to load counts from the `inspect-data.json` file (that's pretty much all that will be in there). I'm open to making this available as a plugin hook - all kinds of optimizations could be run against these `-i` databases. It would essentially be what we have with inspect today but just for databases opened in that specific mode.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/460#issuecomment-492903581
issue_url: https://api.github.com/repos/simonw/datasette/issues/460
node_id: MDEyOklzc3VlQ29tbWVudDQ5MjkwMzU4MQ==
created_at: 2019-05-16T03:34:08Z
updated_at: 2019-05-16T03:34:08Z
author_association: OWNER
body:
  Demo of above: https://latest.datasette.io/?_sort=relationships compared to https://latest.datasette.io/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/465#issuecomment-492904704
issue_url: https://api.github.com/repos/simonw/datasette/issues/465
node_id: MDEyOklzc3VlQ29tbWVudDQ5MjkwNDcwNA==
created_at: 2019-05-16T03:41:27Z
updated_at: 2019-05-16T03:41:27Z
author_association: OWNER
body:
  The main use-case for this endpoint now is going to be [Datasette Registry](https://github.com/simonw/datasette-registry) (which really needs some more love).

  That tool needs to be able to query a Datasette and find out:

  * What tables are available
  * What their columns are
  * Ideally, their row counts

  A single `/-/inspect` call is no good here because with Datasette Library #417 I'm going to be encouraging MUCH larger Datasette instances, potentially with hundreds of attached databases and thousands of attached tables. So pagination will be essential.

  Maybe a smarter approach will be the older idea of having a separate inspect for each database (and maybe each table):

  * `/mydatabase/-/inspect`
  * `/mydatabase/mytable/-/inspect`

  Either way, I'm going to decouple this from milestone 0.28.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/472#issuecomment-493564683
issue_url: https://api.github.com/repos/simonw/datasette/issues/472
node_id: MDEyOklzc3VlQ29tbWVudDQ5MzU2NDY4Mw==
created_at: 2019-05-17T19:03:49Z
updated_at: 2019-05-17T19:03:49Z
author_association: OWNER
body:
  Should set up an alias for the existing `now` to avoid breaking existing automation scripts: http://click.palletsprojects.com/en/5.x/advanced/#command-aliases

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/473#issuecomment-493781417
issue_url: https://api.github.com/repos/simonw/datasette/issues/473
node_id: MDEyOklzc3VlQ29tbWVudDQ5Mzc4MTQxNw==
created_at: 2019-05-19T18:45:15Z
updated_at: 2019-05-19T18:45:15Z
author_association: OWNER
body:
  This expands on the refactoring work from https://github.com/simonw/datasette/commit/6da567dda953c7ac0e5500f17d8e220467a3499e

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/474#issuecomment-493785170
issue_url: https://api.github.com/repos/simonw/datasette/issues/474
node_id: MDEyOklzc3VlQ29tbWVudDQ5Mzc4NTE3MA==
created_at: 2019-05-19T19:36:25Z
updated_at: 2019-05-19T19:36:25Z
author_association: OWNER
body:
  This needs to happen in two places:

  https://github.com/simonw/datasette/blob/260085838887ee343f4d3b177c422e7aef5ade9d/datasette/templates/database.html#L59-L61

  https://github.com/simonw/datasette/blob/260085838887ee343f4d3b177c422e7aef5ade9d/datasette/views/database.py#L75-L80

  Plus new unit test.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/478#issuecomment-493794975
issue_url: https://api.github.com/repos/simonw/datasette/issues/478
node_id: MDEyOklzc3VlQ29tbWVudDQ5Mzc5NDk3NQ==
created_at: 2019-05-19T21:53:04Z
updated_at: 2019-05-19T21:53:04Z
author_association: OWNER
body:
  Example build that's taking ages to push the release: https://travis-ci.org/simonw/datasette/builds/534573406

  Here's the relevant Travis CI configuration:

  https://github.com/simonw/datasette/blob/e518f76c5f5dd0138032bfb26387f5bb91086a3f/.travis.yml#L31-L52

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/469#issuecomment-493799012
issue_url: https://api.github.com/repos/simonw/datasette/issues/469
node_id: MDEyOklzc3VlQ29tbWVudDQ5Mzc5OTAxMg==
created_at: 2019-05-19T22:53:18Z
updated_at: 2019-05-19T22:53:18Z
author_association: OWNER
body:
  I manually tested this against heroku, cloud run and nowv1 and all three worked as expected - in particular they all offered working "download this database" links.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/480#issuecomment-494064357
issue_url: https://api.github.com/repos/simonw/datasette/issues/480
node_id: MDEyOklzc3VlQ29tbWVudDQ5NDA2NDM1Nw==
created_at: 2019-05-20T16:43:14Z
updated_at: 2019-05-20T16:43:14Z
author_association: OWNER
body:
  Thanks for this - looks like exactly what I wanted. I'll test this next time I ship a minor release.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-494190922
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDQ5NDE5MDkyMg==
created_at: 2019-05-21T00:00:40Z
updated_at: 2019-05-21T00:01:09Z
author_association: OWNER
body:
  Wow, this issue has been open for a full year now!

  I've been thinking about this a lot. I've decided I want Datasette to use ASGI 3.0 internally with no dependencies on anything else - then I want the option to run Datasette under both daphne and uvicorn - because uvicorn doesn't support Python 3.5 but Datasette still needs to (primarily for Glitch), and daphne works with 3.5.

  So I'm going to try to go the following route:

  - Every Datasette view becomes an ASGI app
  - The Datasette application itself is an ASGI app that routes to those views
  - When you `pip install datasette` you get Daphne as a dependency (I'd like you to be able to opt-out of installing Daphne, I'm not yet sure how that would work)
  - A new `asgi_serve` plugin hook allows a plugin to serve Datasette using uvicorn (or hypercorn) instead

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-494191378
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDQ5NDE5MTM3OA==
created_at: 2019-05-21T00:02:48Z
updated_at: 2019-05-21T00:02:48Z
author_association: OWNER
body:
  I said earlier that I only need to support GET - I actually need to be able to support POST too, mainly to support plugins (e.g. a plugin that allows authenticated login before you can view Datasette, but potentially also plugins that let you write data directly to SQLite as well).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-494191738
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDQ5NDE5MTczOA==
created_at: 2019-05-21T00:05:02Z
updated_at: 2019-05-21T00:05:02Z
author_association: OWNER
body:
  While I'm not depending on Starlette any more I will need to instead depend on https://github.com/andrew-d/python-multipart for POST form parsing - as used by Starlette here https://github.com/encode/starlette/blob/ab86530eddfcf56e0f7e5ca56f6ab69c15594a7d/starlette/requests.py#L178-L193

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-494192163
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDQ5NDE5MjE2Mw==
created_at: 2019-05-21T00:07:25Z
updated_at: 2019-05-21T00:07:25Z
author_association: OWNER
body:
  Bah, I'd much rather depend on Starlette for things like form parsing - but it's 3.6+ only!

  https://github.com/encode/starlette/blob/ab86530eddfcf56e0f7e5ca56f6ab69c15594a7d/setup.py#L39

  Maybe I could require Python 3.6 or higher if you want to handle POST data? This would make my internals far too complicated though I think.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-494192779
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDQ5NDE5Mjc3OQ==
created_at: 2019-05-21T00:10:47Z
updated_at: 2019-05-21T00:10:47Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/commit/9fdb47ca952b93b7b60adddb965ea6642b1ff523 added `decode_path_component()` and `encode_path_component()` functions because ASGI decodes %2F encoded slashes in URLs automatically. The new encoding scheme looks like this:

      "table/and/slashes" => "tableU+002FandU+002Fslashes"
      "~table" => "U+007Etable"
      "+bobcats!" => "U+002Bbobcats!"
      "U+007Etable" => "UU+002B007Etable"

  For background see this comment: https://github.com/django/asgiref/issues/51#issuecomment-450603464

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/481#issuecomment-494255074
issue_url: https://api.github.com/repos/simonw/datasette/issues/481
node_id: MDEyOklzc3VlQ29tbWVudDQ5NDI1NTA3NA==
created_at: 2019-05-21T06:18:17Z
updated_at: 2019-05-21T06:18:17Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/facetable?_facet_date=created

  <img width="842" alt="fixtures__facetable__15_rows" src="https://user-images.githubusercontent.com/9599/58072755-88426500-7b55-11e9-948a-2ca8517956fc.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-494297022
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDQ5NDI5NzAyMg==
created_at: 2019-05-21T08:39:17Z
updated_at: 2019-05-21T08:39:17Z
author_association: NONE
body:
  Useful context stuff:

  > ASGI decodes %2F encoded slashes in URLs automatically

  `raw_path` for ASGI looks to be under consideration: https://github.com/django/asgiref/issues/87

  > uvicorn doesn't support Python 3.5

  That was an issue specifically against the <=3.5.2 minor point releases of Python, now resolved: https://github.com/encode/uvicorn/issues/330 👍

  > Starlette for things like form parsing - but it's 3.6+ only!

  Yeah - the bits that require 3.6 are anywhere with the "async for" syntax. If it wasn't for that I'd downport it, but that one's a pain. It's the one bit of syntax to watch out for if you're looking to bring any bits of implementation across to Datasette.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/184#issuecomment-494459264
issue_url: https://api.github.com/repos/simonw/datasette/issues/184
node_id: MDEyOklzc3VlQ29tbWVudDQ5NDQ1OTI2NA==
created_at: 2019-05-21T16:17:29Z
updated_at: 2019-05-21T16:17:29Z
author_association: NONE
body:
  Reopening this because it still raises 500 for incorrect table capitalization. 

  Example:

  - https://salaries.news.baltimoresun.com/salaries/2018+Maryland+state+salaries/1 200 OK
  - https://salaries.news.baltimoresun.com/salaries/bad-table/1 400
  - https://salaries.news.baltimoresun.com/salaries/2018+maryland+state+salaries/1 500 Internal Error (note lowercase 'm')

  I think because the table name exists but is not in its canonical form, it triggers a dict lookup error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/483#issuecomment-495032818
issue_url: https://api.github.com/repos/simonw/datasette/issues/483
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTAzMjgxOA==
created_at: 2019-05-23T01:28:06Z
updated_at: 2019-05-23T01:28:06Z
author_association: OWNER
body:
  Here's a UI concept:

  <img width="418" alt="fixtures__facetable__15_rows" src="https://user-images.githubusercontent.com/9599/58219350-551be500-7cbf-11e9-84ae-3b0386ae2063.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/483#issuecomment-495032933
issue_url: https://api.github.com/repos/simonw/datasette/issues/483
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTAzMjkzMw==
created_at: 2019-05-23T01:28:45Z
updated_at: 2019-05-23T01:28:45Z
author_association: OWNER
body:
  Would this be useful for other facets? How many facet types are likely to have a small number of options that could be linked to in this way?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/483#issuecomment-495034774
issue_url: https://api.github.com/repos/simonw/datasette/issues/483
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTAzNDc3NA==
created_at: 2019-05-23T01:38:32Z
updated_at: 2019-05-23T01:43:04Z
author_association: NONE
body:
  I think that location information is one of the other common pieces of hierarchical data. At least one that is general enough that extra dimensions could be auto-generated.

  Also, I think this is an awesome project. Thank you for creating this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/259#issuecomment-495058104
issue_url: https://api.github.com/repos/simonw/datasette/issues/259
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA1ODEwNA==
created_at: 2019-05-23T03:55:37Z
updated_at: 2019-05-23T03:55:37Z
author_association: OWNER
body:
  I got rid of inspect in #462 - I will still be doing many-to-many detection (initially as part of #356) but it doesn't need a separate ticket.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/355#issuecomment-495058622
issue_url: https://api.github.com/repos/simonw/datasette/issues/355
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA1ODYyMg==
created_at: 2019-05-23T03:58:58Z
updated_at: 2019-05-23T03:58:58Z
author_association: OWNER
body:
  So the design I have so far is:

  `?_m2m_linktablename__linktablecolumn=value`

  I'm concerned that this doesn't take tables or columns with `__` in their name into account. Does that matter? Could I support this without them?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/355#issuecomment-495058828
issue_url: https://api.github.com/repos/simonw/datasette/issues/355
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA1ODgyOA==
created_at: 2019-05-23T04:00:27Z
updated_at: 2019-05-23T04:00:27Z
author_association: OWNER
body:
  The alternative would be to use JSON:

  `?_m2m={"table":"ad_targets","column":"target_id","value":"ec3ac"}`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/355#issuecomment-495058964
issue_url: https://api.github.com/repos/simonw/datasette/issues/355
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA1ODk2NA==
created_at: 2019-05-23T04:01:17Z
updated_at: 2019-05-23T04:01:17Z
author_association: OWNER
body:
  I think I like this better. I don't think `?_m2m=` is the correct name for it though.

  `?_through={"table":"ad_targets","column":"target_id","value":"ec3ac"}` feels a little more accurate.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/355#issuecomment-495059236
issue_url: https://api.github.com/repos/simonw/datasette/issues/355
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA1OTIzNg==
created_at: 2019-05-23T04:03:04Z
updated_at: 2019-05-23T04:03:04Z
author_association: OWNER
body:
  This assumes that our current table has a single, unambiguous foreign key relationship with the table indicated by the `?through=` parameter.

  I think that's reasonable. The JSON format could be extended to allow that side of the relationship to optionally be defined there (if the foreign key relationship is missing).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/355#issuecomment-495061686
issue_url: https://api.github.com/repos/simonw/datasette/issues/355
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA2MTY4Ng==
created_at: 2019-05-23T04:21:00Z
updated_at: 2019-05-23T04:21:00Z
author_association: OWNER
body:
  Filtering through one table already works - you need to know that table's primary key, then you do `?column_id=pk` against the first table.

  Filtering through a m2m table will be handled by the new `?_through=` parameter.

  I'm going to leave out filtering through a second levels of joins for the moment. Potentially that could be added later as some extra complicated JSON.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/484#issuecomment-495068273
issue_url: https://api.github.com/repos/simonw/datasette/issues/484
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA2ODI3Mw==
created_at: 2019-05-23T05:03:48Z
updated_at: 2019-05-23T05:04:35Z
author_association: OWNER
body:
  Ideally we would display a limited number of m2m related records with a "..." if there are more than our limit. I could also show a count of the total number of records, but this would have to be agressively time-limited or it could cause extremely poor performance.

  This could be implemented as a SQL query for every displayed row, taking advantage of [Many Small Queries Are Efficient In SQLite](https://sqlite.org/np1queryprob.html). Provided that SQL runs against an index this should be fast to display even on a table with hundreds of rows.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/355#issuecomment-495077443
issue_url: https://api.github.com/repos/simonw/datasette/issues/355
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA3NzQ0Mw==
created_at: 2019-05-23T05:52:52Z
updated_at: 2019-05-23T05:52:52Z
author_association: OWNER
body:
  Documentation here: https://datasette.readthedocs.io/en/latest/json_api.html#special-table-arguments

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/355#issuecomment-495077528
issue_url: https://api.github.com/repos/simonw/datasette/issues/355
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA3NzUyOA==
created_at: 2019-05-23T05:53:20Z
updated_at: 2019-05-23T05:53:20Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/roadside_attractions?_through={%22table%22:%22roadside_attraction_characteristics%22,%22column%22:%22characteristic_id%22,%22value%22:%221%22}

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/355#issuecomment-495079393
issue_url: https://api.github.com/repos/simonw/datasette/issues/355
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA3OTM5Mw==
created_at: 2019-05-23T06:02:11Z
updated_at: 2019-05-23T06:02:11Z
author_association: OWNER
body:
  I'm re-opening because we need a UI mechanism for deselecting this:

  <img width="937" alt="fixtures__roadside_attractions__2_rows_where_where_roadside_attraction_characteristics_characteristic_id____1__and_fixtures__roadside_attractions__2_rows_where_w here_roadside_attraction_characteristics_characteristic_id____1_" src="https://user-images.githubusercontent.com/9599/58229099-9626f000-7ce5-11e9-84d6-a7077b498599.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/355#issuecomment-495079705
issue_url: https://api.github.com/repos/simonw/datasette/issues/355
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA3OTcwNQ==
created_at: 2019-05-23T06:03:40Z
updated_at: 2019-05-23T06:04:03Z
author_association: OWNER
body:
  I think an approach similar to how `?_where=` works would do the job here.

  Can address this feedback from @psychemedia while I'm at it: https://github.com/simonw/datasette/issues/429#issuecomment-483202658

  ![image](https://user-images.githubusercontent.com/82988/56127017-2bf78e80-5f74-11e9-9120-9393eb5d4988.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/483#issuecomment-495080390
issue_url: https://api.github.com/repos/simonw/datasette/issues/483
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA4MDM5MA==
created_at: 2019-05-23T06:06:53Z
updated_at: 2019-05-23T06:06:53Z
author_association: OWNER
body:
  Yes there's definitely something exciting to be done with location facets.

  The easiest one would be a radius-distance-from-a-point facet (5km, 10km etc). A more sophisticated thing might be possible on top of GeoJSON and SpatiaLite - that's probably something I would put in a plugin rather than shipping in Datasette itself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/483#issuecomment-495080591
issue_url: https://api.github.com/repos/simonw/datasette/issues/483
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA4MDU5MQ==
created_at: 2019-05-23T06:07:53Z
updated_at: 2019-05-23T06:09:05Z
author_association: OWNER
body:
  As far as URL design goes... I'm going to stick with `?_facet_date=` for this and use the not-yet-fully-baked JSON alternative syntax. Probably something like this:

      ?_facet_date={"column":"created","interval":"month"}

  Where interval can be day (the default), month or year. And maybe week? Not sure about that.

  Still not sure what/if I should do about exposing these options in the UI.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/485#issuecomment-495083670
issue_url: https://api.github.com/repos/simonw/datasette/issues/485
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA4MzY3MA==
created_at: 2019-05-23T06:21:52Z
updated_at: 2019-05-23T06:22:36Z
author_association: OWNER
body:
  If a table has more than two columns we could do a betterl job at guessing the label column. A few potential tricks:


  * look for a column called name or title
  * look for the first column of type text
  * check for the text column with the most diversity in values

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/485#issuecomment-495085021
issue_url: https://api.github.com/repos/simonw/datasette/issues/485
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTA4NTAyMQ==
created_at: 2019-05-23T06:27:57Z
updated_at: 2019-05-26T23:15:51Z
author_association: OWNER
body:
  I could attempt to calculate the statistics needed for this in a time limited SQL query something like this one: https://latest.datasette.io/fixtures?sql=select+%27name%27+as+column%2C+count+%28distinct+name%29+as+count_distinct%2C+avg%28length%28name%29%29+as+avg_length+from+roadside_attractions%0D%0A++union%0D%0Aselect+%27address%27+as+column%2C+count%28distinct+address%29+as+count_distinct%2C+avg%28length%28address%29%29+as+avg_length+from+roadside_attractions

  ```
  select 'name' as column, count (distinct name) as count_distinct, avg(length(name)) as avg_length from roadside_attractions
    union
  select 'address' as column, count(distinct address) as count_distinct, avg(length(address)) as avg_length from roadside_attractions
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/486#issuecomment-495659567
issue_url: https://api.github.com/repos/simonw/datasette/issues/486
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTY1OTU2Nw==
created_at: 2019-05-24T14:41:45Z
updated_at: 2019-05-24T14:41:45Z
author_association: OWNER
body:
  I'm really keen to offer this as a plugin hook once I have Datasette working on ASGI - #272 

  I'll hopefully have that working in the next few weeks, but in the meantime there are a couple of tricks you can use:

  - you can add static HTML files (no templates though) using the static route configuration options
  - you can link to external hosted pages using the `about_url` metadata option
  - you can add information to an existing page with a custom template. I do that here for example: https://russian-ira-facebook-ads.datasettes.com/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/18#issuecomment-495818358
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/18
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTgxODM1OA==
created_at: 2019-05-25T00:16:50Z
updated_at: 2019-05-25T00:16:50Z
author_association: OWNER
body:
  Oh that's really interesting - yeah I think having an option to insert if missing, skip otherwise would be a good feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/20#issuecomment-495818849
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTgxODg0OQ==
created_at: 2019-05-25T00:21:04Z
updated_at: 2019-05-25T00:21:04Z
author_association: OWNER
body:
  I'm going to implement this using a new `--json-cols` option which works by detecting any string value that starts with [ or { and is valid JSON.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/356#issuecomment-495931078
issue_url: https://api.github.com/repos/simonw/datasette/issues/356
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTkzMTA3OA==
created_at: 2019-05-25T16:30:09Z
updated_at: 2019-05-25T16:30:09Z
author_association: OWNER
body:
  Implemented in https://github.com/simonw/datasette/commit/d923d847545e829bf946bb9170bebfc7c3f9d993

  Documentation here: https://datasette.readthedocs.io/en/latest/facets.html#facet-by-many-to-many

  Demo here: https://latest.datasette.io/fixtures/roadside_attractions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/356#issuecomment-495931140
issue_url: https://api.github.com/repos/simonw/datasette/issues/356
node_id: MDEyOklzc3VlQ29tbWVudDQ5NTkzMTE0MA==
created_at: 2019-05-25T16:30:59Z
updated_at: 2019-05-25T16:30:59Z
author_association: OWNER
body:
  I went with a much more simple URL scheme: `?_facet_m2m=destination_table` (it then figures out which the middle table is by looking at the foreign keys).

  This can be extended to be more complicated in the future if needed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/485#issuecomment-496038601
issue_url: https://api.github.com/repos/simonw/datasette/issues/485
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjAzODYwMQ==
created_at: 2019-05-26T23:08:41Z
updated_at: 2019-05-26T23:08:41Z
author_association: OWNER
body:
  The code currently assumes the primary key is called "id" or "pk" - improving it to detect the primary key using database introspection should work much better.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/485#issuecomment-496039267
issue_url: https://api.github.com/repos/simonw/datasette/issues/485
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjAzOTI2Nw==
created_at: 2019-05-26T23:19:38Z
updated_at: 2019-05-26T23:20:10Z
author_association: OWNER
body:
  Thinking about that union query: I imagine doing this with union could encourage multiple full table scans. Maybe this query would only do one? https://latest.datasette.io/fixtures?sql=select%0D%0A++count+%28distinct+name%29+as+count_distinct_column_1%2C%0D%0A++avg%28length%28name%29%29+as+avg_length_column_1%2C%0D%0A++count%28distinct+address%29+as+count_distinct_column_2%2C%0D%0A++avg%28length%28address%29%29+as+avg_length_column_2%0D%0Afrom+roadside_attractions

  ```
  select
    count (distinct name) as count_distinct_column_1,
    avg(length(name)) as avg_length_column_1,
    count(distinct address) as count_distinct_column_2,
    avg(length(address)) as avg_length_column_2
  from roadside_attractions
  ```

  <img width="800" alt="fixtures__select_count__distinct_name__as_count_distinct_column_1__avg_length_name___as_avg_length_column_1__count_distinct_address__as_count_distinct_column_2__avg_length_address___as_avg_length_column_2_from_roadside_attractions" src="https://user-images.githubusercontent.com/9599/58388316-201ad580-7fd2-11e9-95c3-c98e2758fc1e.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/485#issuecomment-496039483
issue_url: https://api.github.com/repos/simonw/datasette/issues/485
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjAzOTQ4Mw==
created_at: 2019-05-26T23:22:53Z
updated_at: 2019-05-26T23:22:53Z
author_association: OWNER
body:
  Comparing these two SQL queries (the one with union and the one without) using explain:

  With union: https://latest.datasette.io/fixtures?sql=explain+select+%27name%27+as+column%2C+count+%28distinct+name%29+as+count_distinct%2C+avg%28length%28name%29%29+as+avg_length+from+roadside_attractions%0D%0A++union%0D%0Aselect+%27address%27+as+column%2C+count%28distinct+address%29+as+count_distinct%2C+avg%28length%28address%29%29+as+avg_length+from+roadside_attractions produces 52 rows

  Without union: https://latest.datasette.io/fixtures?sql=explain+select%0D%0A++count+(distinct+name)+as+count_distinct_column_1%2C%0D%0A++avg(length(name))+as+avg_length_column_1%2C%0D%0A++count(distinct+address)+as+count_distinct_column_2%2C%0D%0A++avg(length(address))+as+avg_length_column_2%0D%0Afrom+roadside_attractions produces 32 rows

  So I'm going to use the one without the union.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/487#issuecomment-496077220
issue_url: https://api.github.com/repos/simonw/datasette/issues/487
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjA3NzIyMA==
created_at: 2019-05-27T04:56:14Z
updated_at: 2019-05-27T04:56:14Z
author_association: OWNER
body:
  I'm going to rename `ConnectedDatabase` to just `Database` and move it out of the `app.py` module too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/485#issuecomment-496283728
issue_url: https://api.github.com/repos/simonw/datasette/issues/485
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjI4MzcyOA==
created_at: 2019-05-27T18:44:07Z
updated_at: 2019-05-27T18:44:07Z
author_association: OWNER
body:
  This code now lives in a method on the new `datasette.database.Database` class, which should make it easier to write unit tests for.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/473#issuecomment-496337533
issue_url: https://api.github.com/repos/simonw/datasette/issues/473
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjMzNzUzMw==
created_at: 2019-05-28T01:59:19Z
updated_at: 2019-05-28T01:59:19Z
author_association: OWNER
body:
  It would be nice if this plugin was passed the current database/table so it can decide to enable new filters only for specific tables. This will require a bit of refactoring because the filters list is static at the moment - it would instead have to be returned by a function that runs when the table view is rendered.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/473#issuecomment-496338533
issue_url: https://api.github.com/repos/simonw/datasette/issues/473
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjMzODUzMw==
created_at: 2019-05-28T02:05:39Z
updated_at: 2019-05-28T02:05:39Z
author_association: OWNER
body:
  I wonder if this is the right hook?

  The more likely case is that we need a hook that registers a new type of lookup entirely - `?_spatial_within={geojson}` for example. These aren't necessarily attached to a specific column and may have entirely custom syntax - but they still need to be able to influence the query (like the `_where=` clause does) and the human-readable description of the page.

  Is there a strong case for supporting both custom filter plugins AND custom table where plugins, or could those where plugins cover both bases?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/473#issuecomment-496338666
issue_url: https://api.github.com/repos/simonw/datasette/issues/473
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjMzODY2Ng==
created_at: 2019-05-28T02:06:23Z
updated_at: 2019-05-28T02:06:23Z
author_association: OWNER
body:
  I'm having trouble coming up with interesting column-based filters which don't make sense to ship as default behaviour.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/473#issuecomment-496338808
issue_url: https://api.github.com/repos/simonw/datasette/issues/473
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjMzODgwOA==
created_at: 2019-05-28T02:07:23Z
updated_at: 2019-05-28T02:07:23Z
author_association: OWNER
body:
  Assuming I do go ahead with this plugin hook, the existing `InFilter` makes for a nice simple example that illustrates the two key methods: `.where_clause()` and `.human_clause()`:

  ```python
  class InFilter(Filter):
      key = "in"
      display = "in"

      def split_value(self, value):
          if value.startswith("["):
              return json.loads(value)
          else:
              return [v.strip() for v in value.split(",")]

      def where_clause(self, table, column, value, param_counter):
          values = self.split_value(value)
          params = [":p{}".format(param_counter + i) for i in range(len(values))]
          sql = "{} in ({})".format(escape_sqlite(column), ", ".join(params))
          return sql, values

      def human_clause(self, column, value):
          return "{} in {}".format(column, json.dumps(self.split_value(value)))
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/473#issuecomment-496339574
issue_url: https://api.github.com/repos/simonw/datasette/issues/473
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjMzOTU3NA==
created_at: 2019-05-28T02:12:13Z
updated_at: 2019-05-28T02:13:07Z
author_association: OWNER
body:
  Here's an older version of what that custom table filtering plugin might look like: https://github.com/simonw/datasette/commit/5116c4ec8aed5091e1f75415424b80f613518dc6

  ```python
  from datasette.utils import TableFilter

  @hookimpl
  def table_filter():
      async def inner(view, name, table, request):
          extra_human_descriptions = []
          where_clauses = []
          params = {}
          # ... build those things here
          return TableFilter(
              human_description_extras=extra_human_descriptions,
              where_clauses=where_clauses,
              params=params,
          )
      return inner
  ```

  I built this for the https://github.com/simonw/russian-ira-facebook-ads-datasette project.

  It's pretty neat. Maybe I should go with that?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/489#issuecomment-496345660
issue_url: https://api.github.com/repos/simonw/datasette/issues/489
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjM0NTY2MA==
created_at: 2019-05-28T02:48:57Z
updated_at: 2019-05-28T02:48:57Z
author_association: OWNER
body:
  Fixed: https://latest.datasette.io/fixtures/roadside_attraction_characteristics?_sort=attraction_id&_size=2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/485#issuecomment-496367866
issue_url: https://api.github.com/repos/simonw/datasette/issues/485
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjM2Nzg2Ng==
created_at: 2019-05-28T05:14:06Z
updated_at: 2019-05-28T05:14:06Z
author_association: OWNER
body:
  I'm going to generate statistics for every TEXT column.

  Any column with more than 90% distinct rows (compared to the total count of rows) will be a candidate for the label.

  I will then pick the candidate column with the shortest average length.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/21#issuecomment-496665669
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDQ5NjY2NTY2OQ==
created_at: 2019-05-28T19:57:47Z
updated_at: 2019-05-28T19:57:47Z
author_association: OWNER
body:
  https://www.sqlite.org/lang_insert.html

  Looks like it's as simple as `INSERT OR IGNORE` - and we already have code related to that here: 

  https://github.com/simonw/sqlite-utils/blob/1e28eeee8ce55ea68eddb228294a1eff6785b497/sqlite_utils/db.py#L498-L501

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/16#issuecomment-496756348
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDQ5Njc1NjM0OA==
created_at: 2019-05-29T02:12:43Z
updated_at: 2019-05-29T02:12:43Z
author_association: OWNER
body:
  For the CLI I'm thinking:

      sqlite-utils add-column data.db mytable col text --fk othertable id

  So the foreign key is added using `--fk othertable othercolumn`

  If the user doesn't provide the column type we can derive it from the foreign key.

  Can we make the `othercolumn` optional? That would be neat - then we could automatically detect it using the sole primary key of the referenced table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/16#issuecomment-496757010
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDQ5Njc1NzAxMA==
created_at: 2019-05-29T02:16:17Z
updated_at: 2019-05-29T02:16:17Z
author_association: OWNER
body:
  In that case `--fk othertable` detects the primary key.

  If you want to point at a different column, use `--fk-col X`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/21#issuecomment-496786354
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDQ5Njc4NjM1NA==
created_at: 2019-05-29T05:09:01Z
updated_at: 2019-05-29T05:09:01Z
author_association: OWNER
body:
  Shipped this feature in sqlite-utils 1.1: https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/492#issuecomment-496970219
issue_url: https://api.github.com/repos/simonw/datasette/issues/492
node_id: MDEyOklzc3VlQ29tbWVudDQ5Njk3MDIxOQ==
created_at: 2019-05-29T14:50:10Z
updated_at: 2019-05-29T14:50:10Z
author_association: OWNER
body:
  Code in question:
  https://github.com/simonw/datasette/blob/2a4b892d6c0f6609ea48df0dc393397af2b7b1c1/datasette/templates/table.html#L84-L86

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/495#issuecomment-497194179
issue_url: https://api.github.com/repos/simonw/datasette/issues/495
node_id: MDEyOklzc3VlQ29tbWVudDQ5NzE5NDE3OQ==
created_at: 2019-05-30T04:04:01Z
updated_at: 2019-05-30T04:04:01Z
author_association: OWNER
body:
  I think there are a few steps to solving this:

  * Define a variant of `?_facet_m2m={..}` that takes a JSON object which specifies which middle and destination table should be used, and via which columns
  * Teach the Facet `ManyToManyFacet.suggest()` method to check for multiple conflicting m2m relationships and use the new JSON URL for them. Also pick non-conflicting display names for those facets.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/496#issuecomment-497885590
issue_url: https://api.github.com/repos/simonw/datasette/issues/496
node_id: MDEyOklzc3VlQ29tbWVudDQ5Nzg4NTU5MA==
created_at: 2019-05-31T23:05:05Z
updated_at: 2019-05-31T23:05:05Z
author_association: NONE
body:
  Upon doing a "fix" which allowed a longer build timeout the cloudrun container was too slow when it actually ran. So I would say if your sqlite database is over 1 GB heroku and cloudrun are not good options.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/446#issuecomment-498838841
issue_url: https://api.github.com/repos/simonw/datasette/issues/446
node_id: MDEyOklzc3VlQ29tbWVudDQ5ODgzODg0MQ==
created_at: 2019-06-04T20:51:37Z
updated_at: 2019-06-04T20:51:37Z
author_association: OWNER
body:
  So the Python 3.7 way to do this would be dataclasses, but I still want to support 3.5.

  This excellent essay has made me think attrs might be the way to go: https://www.revsys.com/tidbits/dataclasses-and-attrs-when-and-why/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/498#issuecomment-498839428
issue_url: https://api.github.com/repos/simonw/datasette/issues/498
node_id: MDEyOklzc3VlQ29tbWVudDQ5ODgzOTQyOA==
created_at: 2019-06-04T20:53:21Z
updated_at: 2019-06-04T20:53:21Z
author_association: OWNER
body:
  It does not, but that's a really great idea for a feature.

  One challenge here is that FTS ranking calculations take overall table statistics into account, which means it's usually not possible to combine rankings from different tables in a sensible way. But that doesn't mean it's not possible to return grouped results.

  I think this makes a lot of sense as a plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/499#issuecomment-498840129
issue_url: https://api.github.com/repos/simonw/datasette/issues/499
node_id: MDEyOklzc3VlQ29tbWVudDQ5ODg0MDEyOQ==
created_at: 2019-06-04T20:55:30Z
updated_at: 2019-06-04T21:01:22Z
author_association: OWNER
body:
  I really want this too!

  It's one of the goals of the Datasette Library #417 concept, which I'm hoping to turn into an actual feature in the coming months.

  It's also going to be a major focus of my ten month JSK fellowship at Stanford, which starts in September. https://twitter.com/simonw/status/1123624552867565569

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/499#issuecomment-498840597
issue_url: https://api.github.com/repos/simonw/datasette/issues/499
node_id: MDEyOklzc3VlQ29tbWVudDQ5ODg0MDU5Nw==
created_at: 2019-06-04T20:56:53Z
updated_at: 2019-06-04T20:56:53Z
author_association: OWNER
body:
  Have you played with Datasette on Glitch yet? https://simonwillison.net/2019/Apr/23/datasette-glitch/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/499#issuecomment-499260727
issue_url: https://api.github.com/repos/simonw/datasette/issues/499
node_id: MDEyOklzc3VlQ29tbWVudDQ5OTI2MDcyNw==
created_at: 2019-06-05T21:22:55Z
updated_at: 2019-06-05T21:22:55Z
author_association: NONE
body:
  I was thinking of having some kind of GUI in which regular reporters can upload a CSV and choose how to name the tables, columns and whatnot. Maybe it's possible to make such a GUI using Jinja template language? I ask because I'm unsure how to pursue this but I'd like to try. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/498#issuecomment-499262397
issue_url: https://api.github.com/repos/simonw/datasette/issues/498
node_id: MDEyOklzc3VlQ29tbWVudDQ5OTI2MjM5Nw==
created_at: 2019-06-05T21:28:32Z
updated_at: 2019-06-05T21:28:32Z
author_association: NONE
body:
  Thinking about this more, I'd probably have to make a template page to go along with this, right? I'm guessing there's no way to add an all-databases-all-tables search to datasette's "home page" except by copying the "home page" template and editing it?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/24#issuecomment-499603495
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDQ5OTYwMzQ5NQ==
created_at: 2019-06-06T18:02:20Z
updated_at: 2019-06-06T18:02:20Z
author_association: OWNER
body:
  This seems reasonable. It could look like this:

  ```python
  db["dogs"].add_column("instagram", str, not_null=True)
  db["dogs"].add_column("is_good_dog", bool, default=True)
  ```
  And this:
  ```bash
  $ sqlite-utils add-column mydb.db dogs instagram --not-null
  $ sqlite-utils add-column mydb.db dogs is_good_dog integer --default=1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-499923145
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDQ5OTkyMzE0NQ==
created_at: 2019-06-07T15:10:57Z
updated_at: 2019-06-07T15:11:07Z
author_association: CONTRIBUTOR
body:
  Putting this here in case anyone else encounters the same issue with nginx, I was able to resolve it by passing the header in the nginx proxy config (i.e., `proxy_set_header Host $host`).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/506#issuecomment-500173881
issue_url: https://api.github.com/repos/simonw/datasette/issues/506
node_id: MDEyOklzc3VlQ29tbWVudDUwMDE3Mzg4MQ==
created_at: 2019-06-08T23:55:52Z
updated_at: 2019-06-08T23:55:52Z
author_association: OWNER
body:
  Possible plugin direction:

  ```python
  import filetype # https://pypi.org/project/filetype/

  @hookimpl(trylast=True)
  def render_cell(value):
      if isinstance(value, bytes):
          info = repr(value)
          # May still want to truncate this on table view (but not on row page)
          guess = filetype.guess(value)
          if guess is not None:
              # Need jinja2 markup here for \n to display
              info = "Guess: mime={}, extension={}\n\n{}".format(
                  guess.mime, guess.extension, info
              )
          return info

      return None
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/506#issuecomment-500183002
issue_url: https://api.github.com/repos/simonw/datasette/issues/506
node_id: MDEyOklzc3VlQ29tbWVudDUwMDE4MzAwMg==
created_at: 2019-06-09T03:56:06Z
updated_at: 2019-06-09T03:56:06Z
author_association: OWNER
body:
  What are some other interesting tricks we can use to make binary data a bit more interesting to look at?

  https://martin.varela.fi/2017/09/09/simple-binary-data-visualization/ has some really clever visualization tricks - probably a bit much for this plugin though. See also https://codisec.com/binary-visualization-explained/ 

  https://github.com/tryexceptpass/perceptio is some much simpler code for rendering an image for a binary.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/506#issuecomment-500183106
issue_url: https://api.github.com/repos/simonw/datasette/issues/506
node_id: MDEyOklzc3VlQ29tbWVudDUwMDE4MzEwNg==
created_at: 2019-06-09T03:59:12Z
updated_at: 2019-06-09T03:59:12Z
author_association: OWNER
body:
  Another cheap trick is the equivalent of the Unix `strings` command - https://stackoverflow.com/questions/6804582/extract-strings-from-a-binary-file-in-python

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/506#issuecomment-500183948
issue_url: https://api.github.com/repos/simonw/datasette/issues/506
node_id: MDEyOklzc3VlQ29tbWVudDUwMDE4Mzk0OA==
created_at: 2019-06-09T04:22:58Z
updated_at: 2019-06-09T04:26:53Z
author_association: OWNER
body:
  This is quite nice:

  ```
  $ od -c /tmp/Thumb64Segment_11.data | head -n 10
  0000000   \0  \0   @  \0  \0  \0 005   5   X   T   S   F  \0  \0  \0 001
  0000020   \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
  *
  0010000  025 030   . 377 026 032   . 377 027 033   - 377 031 035   0 377
  0010020  032 036   1 377 036       5 377 037   !   8 377 036   "   : 377
  0010040  036       8 377       $   : 377   !   &   ; 377   $   *   ? 377
  0010060    '   -   ? 377   %   *   < 377   %   ,   > 377   -   3   E 377
  0010100    6   ;   M 377   :   @   O 377   =   C   R 377   @   G   V 377
  0010120    @   I   X 377   <   B   Q 377   8   @   N 377   8   @   P 377
  0010140    :   C   T 377   ;   C   U 377   :   C   V 377   9   C   W 377
  ```

  Here's a rough Python equivalent http://code.activestate.com/recipes/579120-data_dumppy-like-the-unix-od-octal-dump-command/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/506#issuecomment-500220646
issue_url: https://api.github.com/repos/simonw/datasette/issues/506
node_id: MDEyOklzc3VlQ29tbWVudDUwMDIyMDY0Ng==
created_at: 2019-06-09T15:22:12Z
updated_at: 2019-06-09T15:22:12Z
author_association: OWNER
body:
  ![3C9CCDBA-F346-47CB-BFEC-964B0426E728](https://user-images.githubusercontent.com/9599/59160835-789ca900-8a8f-11e9-9767-0f50890d17fe.jpeg)

  New idea: show essentially this but differentiate the escape sequences in some way. Maybe wrap them in `<code>` or put the non-escape sequences in bold?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/506#issuecomment-500224864
issue_url: https://api.github.com/repos/simonw/datasette/issues/506
node_id: MDEyOklzc3VlQ29tbWVudDUwMDIyNDg2NA==
created_at: 2019-06-09T16:07:39Z
updated_at: 2019-06-09T16:07:39Z
author_association: OWNER
body:
  Shipped 0.1 of the plugin! I'm pretty happy with this display format:

  <img width="543" alt="many-photos-tables__RKFaceCrop__58_rows" src="https://user-images.githubusercontent.com/9599/59161307-011e4800-8a96-11e9-9768-a168320d8979.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/506#issuecomment-500238035
issue_url: https://api.github.com/repos/simonw/datasette/issues/506
node_id: MDEyOklzc3VlQ29tbWVudDUwMDIzODAzNQ==
created_at: 2019-06-09T19:21:18Z
updated_at: 2019-06-09T19:21:18Z
author_association: NONE
body:
  If you don't mind calling out to Java, then Apache Tika is able to tell you what a load of "binary stuff" is, plus render it to XHTML where possible.

  There's a python wrapper around the Apache Tika server, but for a more typical datasette usecase you'd probably just want to grab the Tika CLI jar, and call it with `--detect` and/or `--xhtml` to process the unknown binary blob

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/506#issuecomment-500903287
issue_url: https://api.github.com/repos/simonw/datasette/issues/506
node_id: MDEyOklzc3VlQ29tbWVudDUwMDkwMzI4Nw==
created_at: 2019-06-11T15:48:27Z
updated_at: 2019-06-11T15:48:27Z
author_association: OWNER
body:
  Calling out to Tika does make me a little nervous, but that's why Datasette has plugins! A plugin that calls Tika (and caches the results) could be really interesting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/502#issuecomment-500975113
issue_url: https://api.github.com/repos/simonw/datasette/issues/502
node_id: MDEyOklzc3VlQ29tbWVudDUwMDk3NTExMw==
created_at: 2019-06-11T18:50:42Z
updated_at: 2019-06-11T18:50:42Z
author_association: OWNER
body:
  There should be a "Download SQLite DB" link at the bottom of each database page - e.g. on https://latest.datasette.io/fixtures 

  <img width="815" alt="fixtures" src="https://user-images.githubusercontent.com/9599/59298300-1bdff080-8c3f-11e9-8d75-b3eb6e1ae385.png">

  Does that solve this? Re-open the issue if it doesn't!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/26#issuecomment-501508302
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDUwMTUwODMwMg==
created_at: 2019-06-13T00:57:52Z
updated_at: 2019-06-13T00:57:52Z
author_association: OWNER
body:
  Two challenges here:

  1. We need a way to specify which tables should be used - e.g. "put records from the `"user"` key in a `users` table, put multiple records from the `"labels"` key in a table called `labels`" (we can pick an automatic name for the m2m table, though it might be nice to have an option to customize it)

  2. Should we deal with nested objects? Consider https://api.github.com/repos/simonw/datasette/pulls for example:

  <img width="620" alt="Mozilla_Firefox" src="https://user-images.githubusercontent.com/9599/59395956-73fc1d00-8d3b-11e9-8fbe-41a04ac8ff41.png">

  Here we have `head.user` as a user, `head.repo` as a repo, and `head.repo.owner` as another user.

  Ideally our mechanism for specifying which table things should be pulled out into would handle this, but it's getting a bit complicated.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/24#issuecomment-501509642
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDUwMTUwOTY0Mg==
created_at: 2019-06-13T01:06:09Z
updated_at: 2019-06-13T01:06:09Z
author_association: OWNER
body:
  Hmm... we need the ability to pass `--not-null` when we are creating a table as well.

  If you attempt to add `NOT NULL` to a column after a table has first been created you get this error:

  ```sqlite3.OperationalError: Cannot add a NOT NULL column with default value NULL```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/24#issuecomment-501514575
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDUwMTUxNDU3NQ==
created_at: 2019-06-13T01:34:55Z
updated_at: 2019-06-13T01:34:55Z
author_association: OWNER
body:
  Since you can't have one without the other, I'm going with `--not-null-default=` and `not_null_default=` for the add column versions of this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/24#issuecomment-501515609
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDUwMTUxNTYwOQ==
created_at: 2019-06-13T01:40:12Z
updated_at: 2019-06-13T01:40:47Z
author_association: OWNER
body:
  But what to do for creating a table?

  For the Python function I could do this:
  ```python
  db["cats"].create({
      "id": int,
      "name": str,
      "score": int, 
      "weight": float,
  }, pk="id", not_null={"weight"}, defaults={"score": 1})
  ```

  The CLI tool only every creates tables as a side-effect of a `sqlite-utils insert` or `sqlite-utils upsert`. I can have them accept optional arguments, `--not-null colname` and `--default colname value`:
  ```
  echo '{"name": "Cleo", "age": 4, "score": 2}' | \
    sqlite-utils insert dogs.db dogs - \
      --not-null age \
      --not-null name \
      --default score 1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/24#issuecomment-501516028
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDUwMTUxNjAyOA==
created_at: 2019-06-13T01:42:36Z
updated_at: 2019-06-13T01:42:36Z
author_association: OWNER
body:
  Maybe it's time to create a `sqlite-utils create-table` command here too, rather than forcing people to create tables only by inserting example data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/25#issuecomment-501516797
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/25
node_id: MDEyOklzc3VlQ29tbWVudDUwMTUxNjc5Nw==
created_at: 2019-06-13T01:46:36Z
updated_at: 2019-06-13T01:47:35Z
author_association: OWNER
body:
  Maybe `foreign_keys` could even optionally just be a list of columns - it could then attempt to detect the related tables based on some rules-of-thumb and raise an error if there's no obvious candidate.

  Rules:
  * If the column name ends in `_id`, remove that suffix and look for a matching table.
  * Try for a table which is the column name without the `_id` suffix with an `s` appended to it
  * Try for a table that's the exact match for the column name

  If none of these rules match, raise an error.

  So the above example could be further simplified to:
  ```python
  db["usages"].insert_all(
      usages_to_insert,
      foreign_keys=["line_id", "definition_id"]
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/25#issuecomment-501517535
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/25
node_id: MDEyOklzc3VlQ29tbWVudDUwMTUxNzUzNQ==
created_at: 2019-06-13T01:50:34Z
updated_at: 2019-06-13T01:50:34Z
author_association: OWNER
body:
  If I'm going to do this then I should make the `other_table` and `other_column` arguments optional here too:

  https://github.com/simonw/sqlite-utils/blob/2fed87da6ea990d295672e4db2c8ae97b787913e/sqlite_utils/cli.py#L201-L215

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/26#issuecomment-501536495
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDUwMTUzNjQ5NQ==
created_at: 2019-06-13T03:40:21Z
updated_at: 2019-06-13T03:40:21Z
author_association: OWNER
body:
  I think I can do something here with a very simple `head.repo.owner` path syntax. Normally this kind of syntax would have to take the difference between dictionaries and lists into account but I don't think that matters here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/26#issuecomment-501537812
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDUwMTUzNzgxMg==
created_at: 2019-06-13T03:49:37Z
updated_at: 2019-06-13T03:50:39Z
author_association: OWNER
body:
  There's an interesting difference here between nested objects with a primary-key style ID and nested objects without.

  If a nested object does not have a primary key, we could still shift it out to another table but it would need to be in a context where it has an automatic foreign key back to our current record.

  A good example of something where that would be useful is the `outageDevices` key in https://github.com/simonw/pge-outages/blob/d890d09ff6e2997948028528e06c82e1efe30365/pge-outages.json#L13-L25 

  ```json
    {
      "outageNumber": "407367",
      "outageStartTime": "1560355216",
      "crewCurrentStatus": "PG&E repair crew is on-site working to restore power.",
      "currentEtor": "1560376800",
      "cause": "Our preliminary determination is that your outage was caused by scheduled maintenance work.",
      "estCustAffected": "3",
      "lastUpdateTime": "1560355709",
      "hazardFlag": "0",
      "latitude": "37.35629",
      "longitude": "-119.70469",
      "outageDevices": [
        {
          "latitude": "37.35409",
          "longitude": "-119.70575"
        },
        {
          "latitude": "37.35463",
          "longitude": "-119.70525"
        },
        {
          "latitude": "37.35562",
          "longitude": "-119.70467"
        }
      ],
      "regionName": "Ahwahnee"
    }
  ```

  These could either be inserted into an `outageDevices` table that uses `rowid`... or we could have a mechanism where we automatically derive a primary key for them based on a hash of their data, hence avoiding creating duplicates even though we don't have a provided primary key.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/26#issuecomment-501538100
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDUwMTUzODEwMA==
created_at: 2019-06-13T03:51:27Z
updated_at: 2019-06-13T03:51:27Z
author_association: OWNER
body:
  I like the term "extract" for what we are doing here, partly because that's the terminology I used in `csvs-to-sqlite`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/26#issuecomment-501539452
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDUwMTUzOTQ1Mg==
created_at: 2019-06-13T03:59:32Z
updated_at: 2019-06-13T03:59:32Z
author_association: OWNER
body:
  Another complexity from the https://api.github.com/repos/simonw/datasette/pulls example:

  <img width="552" alt="Mozilla_Firefox" src="https://user-images.githubusercontent.com/9599/59402680-55eee680-8d54-11e9-9694-9443c8bbbf8a.png">

  We don't actually want `head` and `base` to be pulled out into a separate table. Our ideal table design would probably look something like this:

  - `url`: ...
  - `id`: `285698310`
  - ...
  - `user_id`: `9599` -> refs `users`
  - `head_label`: `simonw:travis-38dev`
  - `head_ref`: `travis-38dev`
  - `head_sha`: `f274f9004302c5ca75ce89d0abfd648457957e31`
  - `head_user_id`: `9599` -> refs `users`
  - `head_repo_id`: `107914493` -> refs `repos`
  - `base_label`: `simonw:master`
  - `base_ref`: `master`
  - `base_sha`: `5e8fbf7f6fbc0b63d0479da3806dd9ccd6aaa945`
  - `base_user_id`: `9599` -> refs `users`
  - `base_repo_id`: `107914493` -> refs `repos`

  So the nested `head` and `base` sections here, instead of being extracted into another table, were flattened into their own columns.

  So perhaps we need a flatten-nested-into-columns mechanism which can be used in conjunction with a extract-to-tables mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/26#issuecomment-501541902
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDUwMTU0MTkwMg==
created_at: 2019-06-13T04:15:22Z
updated_at: 2019-06-13T16:55:42Z
author_association: OWNER
body:
  So maybe something like this:
  ```
  curl https://api.github.com/repos/simonw/datasette/pulls?state=all | \
    sqlite-utils insert git.db pulls - \
      --flatten=base \
      --flatten=head \
      --extract=user:users:id \
      --extract=head_repo.license:licenses:key \
      --extract=head_repo.owner:users \
      --extract=head_repo
      --extract=base_repo.license:licenses:key \
      --extract=base_repo.owner:users \
      --extract=base_repo
  ```
  Is the order of those nested `--extract` lines significant I wonder? It would be nice if the order didn't matter and the code figured out the right execution plan on its own.

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/sqlite-utils/issues/26#issuecomment-501542025
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDUwMTU0MjAyNQ==
created_at: 2019-06-13T04:16:10Z
updated_at: 2019-06-13T04:16:42Z
author_association: OWNER
body:
  So for `--extract` the format is `path-to-property:table-to-extract-to:primary-key`

  If we find an array (as opposed to a direct nested object) at the end of the dotted path we do a m2m table.

  And if `primary-key` is omitted maybe we do the rowid thing with a foreign key back to ourselves.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/26#issuecomment-501543688
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDUwMTU0MzY4OA==
created_at: 2019-06-13T04:26:15Z
updated_at: 2019-06-13T04:26:15Z
author_association: OWNER
body:
  I may ignore `--flatten` for the moment - users can do their own flattening using `jq` if they need that.

  ```
  curl https://api.github.com/repos/simonw/datasette/pulls?state=all | jq "
      [.[] | . + {
          base_label: .base.label,
          base_ref: .base.ref,
          base_sha: .base.sha,
          base_user: .base.user,
          base_repo: .base.repo,
          head_label: .head.label,
          head_ref: .head.ref,
          head_sha: .head.sha,
          head_user: .head.user,
          head_repo: .head.repo
      } | del(.base, .head, ._links)]
  "
  ```
  Output: https://gist.github.com/simonw/2703ed43fcfe96eb8cfeee7b558b61e1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/25#issuecomment-501548097
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/25
node_id: MDEyOklzc3VlQ29tbWVudDUwMTU0ODA5Nw==
created_at: 2019-06-13T04:54:33Z
updated_at: 2019-06-13T04:54:33Z
author_association: OWNER
body:
  Still need to add this mechanism to `.create_table()` - this code here is all that needs to be modified - it needs to learn to deal with the alternative syntax for foreign keys and guess the missing data if necessary:

  https://github.com/simonw/sqlite-utils/blob/d645032cfa4edbccd0542eecdddca29edf9f7b07/sqlite_utils/db.py#L115-L119

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/25#issuecomment-501548676
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/25
node_id: MDEyOklzc3VlQ29tbWVudDUwMTU0ODY3Ng==
created_at: 2019-06-13T04:58:12Z
updated_at: 2019-06-13T04:58:12Z
author_association: OWNER
body:
  I'm going to reuse the `ForeignKey` named tuple here:

  https://github.com/simonw/sqlite-utils/blob/d645032cfa4edbccd0542eecdddca29edf9f7b07/sqlite_utils/db.py#L17-L19

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/24#issuecomment-501572149
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDUwMTU3MjE0OQ==
created_at: 2019-06-13T06:47:17Z
updated_at: 2019-06-13T06:47:17Z
author_association: OWNER
body:
  @IgnoredAmbience this is now shipped in sqlite-utils 1.2 - documentation here:

  * https://sqlite-utils.readthedocs.io/en/latest/python-api.html#python-api-defaults-not-null
  * https://sqlite-utils.readthedocs.io/en/latest/cli.html#cli-defaults-not-null

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/498#issuecomment-501903071
issue_url: https://api.github.com/repos/simonw/datasette/issues/498
node_id: MDEyOklzc3VlQ29tbWVudDUwMTkwMzA3MQ==
created_at: 2019-06-13T22:35:06Z
updated_at: 2019-06-13T22:35:06Z
author_association: NONE
body:
  I'd like to start working on this. I've made a custom template for `index.html` that contains a `form` that contains a search `input`. But I'm not sure where to go from here. When user enters a search term, I'd like for that term to go into a function I'll make that will search all tables with full text search enabled. 

  Can I make additional custom Python scripts for this or must I edit datasette's files directly?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/111#issuecomment-502135123
issue_url: https://api.github.com/repos/simonw/datasette/issues/111
node_id: MDEyOklzc3VlQ29tbWVudDUwMjEzNTEyMw==
created_at: 2019-06-14T14:39:59Z
updated_at: 2019-06-14T14:39:59Z
author_association: OWNER
body:
  This may be the feature that causes me to add dateutilas a dependency (so I can use dateutil.parser.parse)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-502393107
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwMjM5MzEwNw==
created_at: 2019-06-15T19:25:54Z
updated_at: 2019-06-19T01:20:14Z
author_association: OWNER
body:
  OK, time for a solid implementation plan.

  As soon as https://github.com/django/asgiref/pull/92 is merged (hopefully very soon) the ASGI spec will have support for an optional `raw_path` - which means we can continue to use `table%2Fnames` with embedded `/` without being unable to tell if a path has been decoded or not.

  Steps to implement:

  ## Refactor classes, then add .asgi() method to BaseView

  Add a `.asgi(self, scope, receive, send)` method to my base view class. This will expose an ASGI interface to the outside world: the method itself will construct a request object and call the existing `.get()` method.

  My only true shared base class is actually `RenderMixin` because the `IndexView` doesn't extend `BaseView`. I'm going to refactor the class hierarchy a bit here - `AsgiView` will be my top level class with the `.asgi()` method on it. `RenderMixin` will be renamed `BaseView(AsgiView)`, while existing `BaseView` will be renamed `DataView(BaseView)` since it mainly exists to introduce the handy `.data()` abstraction.

  So...

  * `AsgiView` - has `.asgi()` method, extends Sanic `HTTPMethodView` (for the moment)
  * `BaseView(AsgiView)` - defines utility methods currently on `RenderMixin`
  * `IndexView(BaseView)` - the current `IndexView`
  * `DataView(BaseView)` - defines the utilities currently on `BaseView`, including `data()`
  * Everything else subclasses `DataView`

  ## Extract routing logic out into a new `DatasetteView`

  I considered calling this `RouteView`, but one of the goals of this project is to allow other ASGI apps to import Datasette itself and reuse it as its own ASGI function.

  So `DatasetteView` will subclass `BaseView` and will do all of the routing logic. That logic currently lives here:

  https://github.com/simonw/datasette/blob/aa911122feab13f8e65875c98edb00fd3832b7b8/datasette/app.py#L594-L640

  ## For tests: Implement a version of app_client.get() that calls ASGI instead

  Almost all of the unit tests currently use `app_client.get("/path...")`. I want to be able to run tests against both ASGI and existing-Sanic, so for the moment I'm going to teach `app_client.get()` to use ASGI instead but only in the presence of a new environment variable. I can then have Travis run the tests twice - once with that environement variable and once without.

  ## Make datasette serve --asgi run ASGI and uvicorn

  Uvicorn supports Python 3.5 again as of https://github.com/encode/uvicorn/issues/330 - so it's going to be the new dependency for Datasette. 

  ## Do some comparative testing of ASGI and non-ASGI

  Just some sanity checking to make sure there aren't any weird issues.

  ## Final step: refactor out Sanic

  Hopefully this will just involve changes being made to the AsgiView base class, since that subclasses Sanic `HTTPMethodView`.

  Bonus: It looks like dropping Sanic as a dependency in favour of Uvicorn should give us Windows support! https://github.com/encode/uvicorn/issues/82

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-502393267
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwMjM5MzI2Nw==
created_at: 2019-06-15T19:28:27Z
updated_at: 2019-06-15T19:28:27Z
author_association: OWNER
body:
  I'll probably revert 9fdb47ca952b93b7b60adddb965ea6642b1ff523 from https://github.com/simonw/datasette/issues/272#issuecomment-494192779 since I won't need it now that ASGI is getting `raw_path` support.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/509#issuecomment-502393573
issue_url: https://api.github.com/repos/simonw/datasette/issues/509
node_id: MDEyOklzc3VlQ29tbWVudDUwMjM5MzU3Mw==
created_at: 2019-06-15T19:32:56Z
updated_at: 2019-06-15T19:32:56Z
author_association: OWNER
body:
  Experimental exploratory patch:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 2ef7da4..ca51866 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -164,8 +164,10 @@ class Datasette:
                   is_memory = True
               is_mutable = path not in self.immutables
               db = Database(self, path, is_mutable=is_mutable, is_memory=is_memory)
  +            i = 1
               if db.name in self.databases:
  -                raise Exception("Multiple files with same stem: {}".format(db.name))
  +                db.stem = db.name + "-" + str(i)
  +                i += 1
               self.databases[db.name] = db
           self.cache_headers = cache_headers
           self.cors = cors
  diff --git a/datasette/database.py b/datasette/database.py
  index e491577..75c8681 100644
  --- a/datasette/database.py
  +++ b/datasette/database.py
  @@ -14,6 +14,8 @@ from .inspect import inspect_hash
   
   
   class Database:
  +    stem = None
  +
       def __init__(self, ds, path=None, is_mutable=False, is_memory=False):
           self.ds = ds
           self.path = path
  @@ -73,6 +75,8 @@ class Database:
       def name(self):
           if self.is_memory:
               return ":memory:"
  +        elif self.stem:
  +            return self.stem
           else:
               return Path(self.path).stem
   ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-502394420
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwMjM5NDQyMA==
created_at: 2019-06-15T19:45:46Z
updated_at: 2019-06-15T19:45:46Z
author_association: OWNER
body:
  For reference, here's some WIP code I wrote last year against the old ASGI 2 spec: https://github.com/simonw/datasette/commit/4fd36ba2f3f91da7258859808616078e3464fb97

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-502395689
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwMjM5NTY4OQ==
created_at: 2019-06-15T20:05:26Z
updated_at: 2019-06-15T20:05:26Z
author_association: OWNER
body:
  For the routing component: I'm going to base my implementation on the one from Django Channels.

  https://github.com/django/channels/blob/507cb54fcb36df63282dd19653ea743036e7d63c/channels/routing.py#L123-L149

  Documented here: https://channels.readthedocs.io/en/latest/topics/routing.html#urlrouter

  Particularly relevant: my view classes need access to the components that were already parsed out of the URL by the router. I'm going to copy the Django Channels mechanism of stashing those in `scope["url_route"]["kwargs"]`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-502401078
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwMjQwMTA3OA==
created_at: 2019-06-15T21:35:26Z
updated_at: 2019-06-15T21:35:26Z
author_association: OWNER
body:
  Started sketching out the router in the `asgi` branch: https://github.com/simonw/datasette/commit/7cdc55c6836fe246b1ca8a13a965a39991c9ffec

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/187#issuecomment-502401636
issue_url: https://api.github.com/repos/simonw/datasette/issues/187
node_id: MDEyOklzc3VlQ29tbWVudDUwMjQwMTYzNg==
created_at: 2019-06-15T21:44:23Z
updated_at: 2019-06-15T21:44:23Z
author_association: OWNER
body:
  I'm closing his as a duplicate of the new #511 - I hope to have his working very shortly as a result of #272 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/507#issuecomment-502402586
issue_url: https://api.github.com/repos/simonw/datasette/issues/507
node_id: MDEyOklzc3VlQ29tbWVudDUwMjQwMjU4Ng==
created_at: 2019-06-15T22:00:14Z
updated_at: 2019-06-15T22:00:14Z
author_association: OWNER
body:
  I need to find a good tool for taking screenshots on OSX that are reliably the same dimension across multiple sessions. Maybe in Firefox?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/511#issuecomment-502420036
issue_url: https://api.github.com/repos/simonw/datasette/issues/511
node_id: MDEyOklzc3VlQ29tbWVudDUwMjQyMDAzNg==
created_at: 2019-06-16T04:51:39Z
updated_at: 2019-06-16T04:52:14Z
author_association: OWNER
body:
  It looks like AppVeyor is the right solution for Windows CI. I've set up a Datasette project there but I do t yet have the configuration figured out (plus Datasette won't work on windows yet anyway): https://ci.appveyor.com/project/simonw/datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-502466466
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwMjQ2NjQ2Ng==
created_at: 2019-06-16T16:28:10Z
updated_at: 2019-06-16T16:28:10Z
author_association: OWNER
body:
  I have an open pull request to Uvicorn with an implementation of `raw_path`: https://github.com/encode/uvicorn/pull/372

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-503195217
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwMzE5NTIxNw==
created_at: 2019-06-18T15:46:31Z
updated_at: 2019-06-18T15:54:18Z
author_association: OWNER
body:
  How should file serving work?

  Starlette and Sanic both use `aiofiles` - https://github.com/Tinche/aiofiles  - which is a small wrapper around file operations which runs them all in an executor thread. It doesn't have any C dependencies so it looks like a good option. [Quart uses it too](https://gitlab.com/pgjones/quart/blob/317562ea660edb7159efc20fa57b95223d408ea0/quart/wrappers/response.py#L122-169).

  `aiohttp` does things differently: it has [an implementation based on sendfile](https://github.com/aio-libs/aiohttp/blob/7a324fd46ff7dc9bb0bb1bc5afb326e04cf7cef0/aiohttp/web_fileresponse.py#L46-L122) with [an alternative fallback](https://github.com/aio-libs/aiohttp/blob/7a324fd46ff7dc9bb0bb1bc5afb326e04cf7cef0/aiohttp/web_fileresponse.py#L175-L200) which reads chunks from a file object and yields them one chunk at a time, 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/513#issuecomment-503199253
issue_url: https://api.github.com/repos/simonw/datasette/issues/513
node_id: MDEyOklzc3VlQ29tbWVudDUwMzE5OTI1Mw==
created_at: 2019-06-18T15:56:29Z
updated_at: 2019-06-18T15:56:29Z
author_association: OWNER
body:
  Unfortunately not - I really wish this was possible. I have not yet found a great serverless solution for publishing 1GB+ databases - they're too big for Heroku, Cloud Run OR Zeit Now. Once databases get that big the only option I've found is to run a VPS (or an EC2 instance) with a mounted hard drive volume and execute `datasette serve` on that instance, with an nginx running on port 80 that proxies traffic back to Datasette.

  I'd love to figure out a way to make hosting larger databases as easy as it currently is to host small ones.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/512#issuecomment-503200024
issue_url: https://api.github.com/repos/simonw/datasette/issues/512
node_id: MDEyOklzc3VlQ29tbWVudDUwMzIwMDAyNA==
created_at: 2019-06-18T15:58:18Z
updated_at: 2019-06-18T15:58:18Z
author_association: OWNER
body:
  The `about` and `license` and `source` keys are all intended to be used for links - so if you provide a `about_url` it will be displayed as a URL, and you can then use the `about` key to customize the link label.

  There are `description` and `title` fields which can be used to display text without linking to anything.

  I'm definitely open to reconsidering how these work - I don't think they quite serve people's needs as they are right now, so suggestions for improving them would be very welcome.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/502#issuecomment-503237884
issue_url: https://api.github.com/repos/simonw/datasette/issues/502
node_id: MDEyOklzc3VlQ29tbWVudDUwMzIzNzg4NA==
created_at: 2019-06-18T17:39:18Z
updated_at: 2019-06-18T17:46:08Z
author_association: NONE
body:
  It appears that I cannot reopen this issue but the proposed solution did not solve it. The link is not there. I have full text search enabled for a bunch of tables in my database and even clicking the link to reveal hidden tables did not show the download DB link.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/513#issuecomment-503249999
issue_url: https://api.github.com/repos/simonw/datasette/issues/513
node_id: MDEyOklzc3VlQ29tbWVudDUwMzI0OTk5OQ==
created_at: 2019-06-18T18:11:36Z
updated_at: 2019-06-18T18:11:36Z
author_association: NONE
body:
  Ah, so basically put the SQLite databases on Linode, for example, and run `datasette serve` on there? I'm comfortable with that. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-503369834
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwMzM2OTgzNA==
created_at: 2019-06-19T01:26:24Z
updated_at: 2019-06-19T01:26:24Z
author_association: OWNER
body:
  I need to be able to define the URL routes once and have them work for both Sanic and ASGI.

  I'm going to extract the web application bits out of the `Datasette` class into a `DatasetteServer` class. Then I can have a `add_route()` method on that class, then have `DatasetteSanic` and `DatasetteAsgi` subclasses which redefine that method.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504661909
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY2MTkwOQ==
created_at: 2019-06-22T12:29:46Z
updated_at: 2019-06-22T12:29:46Z
author_association: OWNER
body:
  I'm still trying to figure this out myself.

  I'm confident that running nginx on port 80 and using it to proxy traffic to Datasette is a sensible way to solve the port problem.

  As for running Datasette itself: the two options that seem best to me are some kind of Init.d service or running it under supervisord. I have to admit I haven't worked out the necessary incantation for either of those yet: the solitary instance I have that's not running as a Docker container is sitting in a "screen" instance for the moment!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504661990
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY2MTk5MA==
created_at: 2019-06-22T12:30:47Z
updated_at: 2019-06-22T12:30:47Z
author_association: OWNER
body:
  A section in the Datasette docs that acts as recommendations plus a tutorial for running Datasette on a VPS without using a Docker would be excellent.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504662904
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY2MjkwNA==
created_at: 2019-06-22T12:45:21Z
updated_at: 2019-06-22T12:45:39Z
author_association: CONTRIBUTOR
body:
  On most modern Linux distros, systemd is the easiest answer.

  Example systemd unit file (save to `/etc/systemd/system/datasette.service`):
  ```
  [Unit]
  Description=Datasette
  After=network.target

  [Service]
  Type=simple
  User=<username>
  WorkingDirectory=/path/to/data
  ExecStart=/path/to/datasette serve -h 0.0.0.0 ./my.db
  Restart=on-failure

  [Install]
  WantedBy=multi-user.target
  ```

  Activate it with:
  ```bash
  $ sudo systemctl daemon-reload
  $ sudo systemctl enable datasette
  $ sudo systemctl start datasette
  ```

  Logs are best viewed using `journalctl -u datasette -f`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504662931
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY2MjkzMQ==
created_at: 2019-06-22T12:45:47Z
updated_at: 2019-06-22T12:45:47Z
author_association: OWNER
body:
  I asked about this on Twitter and got a solid recommendation for systemd and this tutorial: https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units - via https://twitter.com/sil/status/1142412145990221825

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504663201
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY2MzIwMQ==
created_at: 2019-06-22T12:49:56Z
updated_at: 2019-06-22T12:49:56Z
author_association: OWNER
body:
  Here are some partial notes I have saved from an nginx configuration I've used in the past:
  ```
  cat /etc/nginx/sites-available/default
  server {
  	listen 80 default_server;
  	listen [::]:80 default_server;

          location / {
                  proxy_pass http://127.0.0.1:8001/;
  		proxy_set_header Host $host;
          }
  ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504663390
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY2MzM5MA==
created_at: 2019-06-22T12:52:27Z
updated_at: 2019-06-22T12:52:27Z
author_association: OWNER
body:
  This example is useful to - I like how it has a Makefile that knows how to set up systemd: https://github.com/pikesley/Queube

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504663766
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY2Mzc2Ng==
created_at: 2019-06-22T12:57:59Z
updated_at: 2019-06-22T12:57:59Z
author_association: CONTRIBUTOR
body:
  > This example is useful to - I like how it has a Makefile that knows how to set up systemd: https://github.com/pikesley/Queube

  I wasn't even aware it was possible to add a systemd service at an arbitrary path, but it seems a little messy to me.

  Maybe worth noting that systemd does support [per-user services](https://wiki.archlinux.org/index.php/Systemd/User) which don't require root access. Cool but probably overkill for most people (especially when you're going to need root to listen on port 80 anyway, directly or via a reverse proxy).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504684831
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY4NDgzMQ==
created_at: 2019-06-22T17:38:23Z
updated_at: 2019-06-22T17:38:23Z
author_association: CONTRIBUTOR
body:
  > > WorkingDirectory=/path/to/data
  > 
  > @russss, Which directory does this represent?

  It's the working directory (cwd) of the spawned process. In this case if you set it to the directory your data is in, you can use relative paths to the db (and metadata/templates/etc) in the `ExecStart` command.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504685187
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY4NTE4Nw==
created_at: 2019-06-22T17:43:24Z
updated_at: 2019-06-22T17:43:24Z
author_association: NONE
body:
  > > > WorkingDirectory=/path/to/data
  > > 
  > > 
  > > @russss, Which directory does this represent?
  > 
  > It's the working directory (cwd) of the spawned process. In this case if you set it to the directory your data is in, you can use relative paths to the db (and metadata/templates/etc) in the `ExecStart` command.

  In my case, on a remote server, I set up a virtual environment in `/home/chris/Env/datasette`, and when I activated that environment I ran `pip install datasette`. 

  My datasette project is in `/home/chris/datatsette-project`, so I guess I'd use that directory in the `WorkingDirectory` parameter?

  And the `ExecStart` parameter would be `/home/chris/Env/datasette/lib/python3.7/site-packages/datasette serve -h 0.0.0.0 my.db` I'm guessing?
   

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504686266
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY4NjI2Ng==
created_at: 2019-06-22T17:58:50Z
updated_at: 2019-06-23T21:21:57Z
author_association: NONE
body:
  @russss 

  Actually, here's what I've got in `/etc/systemd/system/datasette.service`

  ```
  [Unit]
  Description=Datasette
  After=network.target

  [Service]
  Type=simple
  User=chris
  WorkingDirectory=/home/chris/digital-library
  ExecStart=/home/chris/Env/datasette/lib/python3.7/site-packages/datasette serve -h 0.0.0.0 databases/*.db --cors --metadata metadata.json
  Restart=on-failure

  [Install]
  WantedBy=multi-user.target
  ```

  I ran: 
  ```
  $ sudo systemctl daemon-reload
  $ sudo systemctl enable datasette
  $ sudo systemctl start datasette
  ```
  Then I ran:
  `$ journalctl -u datasette -f`

  Got this message.

  ```
  Hint: You are currently not seeing messages from other users and the system.
        Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.
        Pass -q to turn off this notice.
  -- Logs begin at Thu 2019-06-20 00:05:23 CEST. --
  Jun 22 19:55:57 ns331247 systemd[16176]: datasette.service: Failed to execute command: Permission denied
  Jun 22 19:55:57 ns331247 systemd[16176]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied
  Jun 22 19:55:57 ns331247 systemd[16184]: datasette.service: Failed to execute command: Permission denied
  Jun 22 19:55:57 ns331247 systemd[16184]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied
  Jun 22 19:55:58 ns331247 systemd[16186]: datasette.service: Failed to execute command: Permission denied
  Jun 22 19:55:58 ns331247 systemd[16186]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied
  Jun 22 19:55:58 ns331247 systemd[16190]: datasette.service: Failed to execute command: Permission denied
  Jun 22 19:55:58 ns331247 systemd[16190]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied
  Jun 22 19:55:58 ns331247 systemd[16191]: datasette.service: Failed to execute command: Permission denied
  Jun 22 19:55:58 ns331247 systemd[16191]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied
  ```
  When I go to the address for my server, I am met with the standard "Welcome to nginx" message:

  ```
  Welcome to nginx!
  If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

  For online documentation and support please refer to nginx.org.
  Commercial support is available at nginx.com.

  Thank you for using nginx.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504690927
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY5MDkyNw==
created_at: 2019-06-22T19:06:07Z
updated_at: 2019-06-22T19:06:07Z
author_association: CONTRIBUTOR
body:
  I'd rather not turn this into a systemd support thread, but you're trying to execute the package directory there. Your datasette executable is probably at `/home/chris/Env/datasette/bin/datasette`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/516#issuecomment-504696610
issue_url: https://api.github.com/repos/simonw/datasette/issues/516
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY5NjYxMA==
created_at: 2019-06-22T20:36:57Z
updated_at: 2019-06-22T20:36:57Z
author_association: OWNER
body:
  According to [the black documentation](https://black.readthedocs.io/en/stable/the_black_code_style.html?highlight=isort.cfg) the following config file is necessary to avoid isort and black getting into an edit war with each other:
  ```
  [settings]
  multi_line_output=3
  include_trailing_comma=True
  force_grid_wrap=0
  use_parentheses=True
  line_length=88
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/516#issuecomment-504696638
issue_url: https://api.github.com/repos/simonw/datasette/issues/516
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY5NjYzOA==
created_at: 2019-06-22T20:37:25Z
updated_at: 2019-06-22T20:38:42Z
author_association: OWNER
body:
  This test added do `test_black.py` seems to do the trick:
  ```python
  @pytest.mark.parametrize("path", code_root.glob("**/*.py"))
  def test_isort(path):
      # Have to capture stdout because isort uses print() directly
      stdout = sys.stdout
      sys.stdout = io.StringIO()
      result = isort.SortImports(path, check=True)
      assert (
          not result.incorrectly_sorted
      ), "{} has incorrectly sorted imports, fix with 'isort -rc . && black .'"
      sys.stdout = stdout
  ```
  I'll rename `test_black.py` to `test_lint.py` to reflect that it now covers more forms of linting.

  Also needed: add `move_to_front(items, "test_isort")` to `conftest.py`. And add `isort` to `extras_require` in `setup.py`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-504697742
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwNDY5Nzc0Mg==
created_at: 2019-06-22T20:55:59Z
updated_at: 2019-06-22T20:56:22Z
author_association: OWNER
body:
  Getting this to work with both Sanic AND ASGI at the same time (via the classes described previously with an `--asgi` command-line option) is proving way trickier than I expected, mainly because of the complexity of [the current Datasette.app() method](https://github.com/simonw/datasette/blob/35429f90894321eda7f2db31b9ea7976f31f73ac/datasette/app.py#L545-L721).

  I'm going to drop the compatibility path for a bit and see if I can make progress on a pure-ASGI port.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-504710331
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwNDcxMDMzMQ==
created_at: 2019-06-23T01:08:45Z
updated_at: 2019-06-23T01:08:45Z
author_association: OWNER
body:
  Lots still to do:

  * Static files are not being served
  * Streaming CSV files don't work
  * Tests all fail
  * Some URLs (e.g. the 'next' link on tables) are incorrect

  I'm going to work on getting the unit test framework to be ASGI-compatible next.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-504711468
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwNDcxMTQ2OA==
created_at: 2019-06-23T01:36:33Z
updated_at: 2019-06-23T01:36:33Z
author_association: OWNER
body:
  Published an in-progress demo:

      datasette publish now fixtures.db -n datasette-asgi-early-demo --branch=asgi

  Here it is: https://datasette-asgi-early-demo-qahhxctqpw.now.sh/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-504716988
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwNDcxNjk4OA==
created_at: 2019-06-23T03:43:46Z
updated_at: 2019-06-23T15:15:26Z
author_association: OWNER
body:
  OK, it's beginning to shape up now. Next steps:

  - [x] Static file support (including for plugins) - plus tests
  - [x] Streaming support so the CSV tests will pass
  - [x] Ability to download the database file
  - [x] Implement missing-slash redirects

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/226#issuecomment-504720379
issue_url: https://api.github.com/repos/simonw/datasette/issues/226
node_id: MDEyOklzc3VlQ29tbWVudDUwNDcyMDM3OQ==
created_at: 2019-06-23T05:05:32Z
updated_at: 2019-06-23T05:05:32Z
author_association: OWNER
body:
  The mechanism I described here - having a `tests/example_plugin` folder - is probably the right solution for #517 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-504754433
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc1NDQzMw==
created_at: 2019-06-23T13:51:53Z
updated_at: 2019-06-23T13:51:53Z
author_association: OWNER
body:
  CSV tests all pass as of https://github.com/simonw/datasette/commit/ff9efa668ebc33f17ef9b30139960e29906a18fb

  This code could be a lot neater though. At the very least I'm going to refactor `datasette/utils.py` into a `datasette/utils` package and put all of my new ASGI utilities in `datasette/utils/asgi.py`

  The way I implemented streaming on top of a writer object (inspired by Sanic) is a bit of a weird hack. I think I'd rather use an abstraction where my view functions can yield chunks of body data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-504754552
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc1NDU1Mg==
created_at: 2019-06-23T13:53:39Z
updated_at: 2019-06-23T13:53:39Z
author_association: OWNER
body:
  Next test to fix (because by new test harness doesn't actually obey the `allow_redirects=` parameter):
  ```
  _____________ test_database_page_redirects_with_url_hash _____________

  app_client_with_hash = <tests.fixtures.TestClient object at 0x10981f240>

      def test_database_page_redirects_with_url_hash(app_client_with_hash):
          response = app_client_with_hash.get("/fixtures", allow_redirects=False)
          assert response.status == 302
          response = app_client_with_hash.get("/fixtures")
  >       assert "fixtures" in response.text
  E       AssertionError: assert 'fixtures' in ''
  E        +  where '' = <tests.fixtures.TestResponse object at 0x10981f550>.text
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-504759683
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc1OTY4Mw==
created_at: 2019-06-23T14:57:50Z
updated_at: 2019-06-23T14:57:50Z
author_association: OWNER
body:
  All of the tests are now passing!

  I still need a solution for this:

  https://github.com/simonw/datasette/blob/5bd510b01adae3f719e4426b9bfbc346a946ba5c/datasette/app.py#L706-L714

  I think the answer is ASGI lifespan, which is supported by Uvicorn. https://asgi.readthedocs.io/en/latest/specs/lifespan.html#startup

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-504759842
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc1OTg0Mg==
created_at: 2019-06-23T15:00:06Z
updated_at: 2019-06-23T15:00:06Z
author_association: OWNER
body:
  I also need to actually take advantage of `raw_path` such that pages like https://fivethirtyeight.datasettes.com/fivethirtyeight/twitter-ratio%2Fsenators can be correctly served.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-504760061
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc2MDA2MQ==
created_at: 2019-06-23T15:02:52Z
updated_at: 2019-06-23T15:02:52Z
author_association: OWNER
body:
  Tests are failing on Python 3.5: https://travis-ci.org/simonw/datasette/jobs/549380098 - error is `TypeError: the JSON object must be str, not 'bytes'`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-504761039
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc2MTAzOQ==
created_at: 2019-06-23T15:15:41Z
updated_at: 2019-06-23T15:18:36Z
author_association: OWNER
body:
  And now the tests are all passing!

  Still to do:

  * Use `raw_path` so table names containing `/` can work correctly
  * Get ?_trace=1 working again
  * Replacement for `@app.listener("before_server_start")`
  * Replace Sanic request object with my own request class, so I can remove Sanic dependency

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/518#issuecomment-504762769
issue_url: https://api.github.com/repos/simonw/datasette/issues/518
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc2Mjc2OQ==
created_at: 2019-06-23T15:37:26Z
updated_at: 2019-06-23T15:37:26Z
author_association: OWNER
body:
  This is strange: on my local machine `http://127.0.0.1:8001/fixtures/table%2Fwith%2Fslashes.csv` is returning a 404 BUT there's a test for that which is passing under pytest:

  https://github.com/simonw/datasette/blob/d60fbfcae2658e71cab6d7b3b9f53f8d895064ef/tests/test_api.py#L721-L727

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/518#issuecomment-504762887
issue_url: https://api.github.com/repos/simonw/datasette/issues/518
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc2Mjg4Nw==
created_at: 2019-06-23T15:38:58Z
updated_at: 2019-06-23T15:38:58Z
author_association: OWNER
body:
  Mystery solved: that's because I'm constructing my own scope object and testing via `ApplicationCommunicator` rather than exercising Uvicorn directly.

  https://github.com/simonw/datasette/blob/d60fbfcae2658e71cab6d7b3b9f53f8d895064ef/tests/fixtures.py#L42-L57

  I don't want to introduce the complexity of launching a real Uvicorn as part of the tests, so I guess I'll have to carefully update my `ApplicationCommunicator` test harness to more correctly emulate real life.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/498#issuecomment-504763919
issue_url: https://api.github.com/repos/simonw/datasette/issues/498
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc2MzkxOQ==
created_at: 2019-06-23T15:50:49Z
updated_at: 2019-06-23T15:50:49Z
author_association: OWNER
body:
  One interesting way to approach this could be to do it entirely in JavaScript. I've had a lot of success building small apps on top of Datasette's JavaScript API - I wrote up one example here: https://24ways.org/2018/fast-autocomplete-search-for-your-website/

  Once #272 is done I'll be adding a plugin hook that allows plugins to define entirely new pages within the Datasette application, which may also be a good way to work on this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/518#issuecomment-504765018
issue_url: https://api.github.com/repos/simonw/datasette/issues/518
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc2NTAxOA==
created_at: 2019-06-23T16:03:20Z
updated_at: 2019-06-23T16:03:20Z
author_association: OWNER
body:
  Weird new bug: http://127.0.0.1:8001/fixtures/table%2Fwith%2Fslashes.csv?_format=json is downloading CSV for me now.

  https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=json does the right thing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/518#issuecomment-504765145
issue_url: https://api.github.com/repos/simonw/datasette/issues/518
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc2NTE0NQ==
created_at: 2019-06-23T16:04:37Z
updated_at: 2019-06-23T16:04:37Z
author_association: OWNER
body:
  Another bug: JSON is being served without a content-type header:
  ```
  ~ $ curl -i 'http://127.0.0.1:8001/fivethirtyeight/ahca-polls%2Fahca_polls.json'
  HTTP/1.1 200 OK
  date: Sun, 23 Jun 2019 16:04:01 GMT
  server: uvicorn
  referrer-policy: no-referrer
  transfer-encoding: chunked

  {"database": "fivethirtyeight", "table": "ahca-polls/ahca_polls", ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/518#issuecomment-504765738
issue_url: https://api.github.com/repos/simonw/datasette/issues/518
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc2NTczOA==
created_at: 2019-06-23T16:11:49Z
updated_at: 2019-06-23T16:20:44Z
author_association: OWNER
body:
  OK, for **Get ?_trace=1 working again**. The old code lives in two places:

  https://github.com/simonw/datasette/blob/35429f90894321eda7f2db31b9ea7976f31f73ac/datasette/app.py#L546-L560

  And then:

  https://github.com/simonw/datasette/blob/35429f90894321eda7f2db31b9ea7976f31f73ac/datasette/app.py#L653-L672

  So it's stashing something on the request to tell the rest of the code it should be tracing, then using that collected data from the request to add information to the final body.

  One possible shape for the replacement is a new ASGI middleware that wraps everything else. We don't have a mutable request object here though, so we will need to untangle this entirely from the request object.

  Also tricky is that in ASGI land we handle streams - we don't usually wait around for the entire response body to be compiled for us. This means the code that modifies the response (adding to the JSON or appending inside the `</body>`) needs to be reconsidered.

  As usual, Starlette seems to have figured this out: https://github.com/encode/starlette/blob/8c8cc2ec0a5cb834a9a15b871ae8b480503abb67/starlette/middleware/gzip.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/516#issuecomment-504768147
issue_url: https://api.github.com/repos/simonw/datasette/issues/516
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc2ODE0Nw==
created_at: 2019-06-23T16:43:23Z
updated_at: 2019-06-23T16:43:23Z
author_association: OWNER
body:
  The Starlette lint and test scripts do this, and also apply autoflake to remove any unnecessary imports: https://github.com/encode/starlette/tree/8c8cc2ec0a5cb834a9a15b871ae8b480503abb67/scripts

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/520#issuecomment-504772599
issue_url: https://api.github.com/repos/simonw/datasette/issues/520
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc3MjU5OQ==
created_at: 2019-06-23T17:44:36Z
updated_at: 2019-06-23T17:44:36Z
author_association: OWNER
body:
  These plugins will need access to configuration and the ability to execute SQL - which means we need to make the `datasette` instance available to them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/518#issuecomment-504782618
issue_url: https://api.github.com/repos/simonw/datasette/issues/518
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc4MjYxOA==
created_at: 2019-06-23T20:05:44Z
updated_at: 2019-06-23T20:05:59Z
author_association: OWNER
body:
  **Replacement for @app.listener("before_server_start")** - this is what the [ASGI lifespan protocol](https://asgi.readthedocs.io/en/latest/specs/lifespan.html) is for.

  I know Uvicorn supports this because it keeps saying `ASGI 'lifespan' protocol appears unsupported` on the console.

  I think the solution here will be to introduce another ASGI wrapper class similar to `AsgiTracer`. I'll model this on the example in the ASGI lifespan spec.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/498#issuecomment-504785662
issue_url: https://api.github.com/repos/simonw/datasette/issues/498
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc4NTY2Mg==
created_at: 2019-06-23T20:47:37Z
updated_at: 2019-06-23T20:47:37Z
author_association: NONE
body:
  Very cool, thank you.

  Using http://search-24ways.herokuapp.com as an example, let's say I want to search all FTS columns in all tables in all databases for the word "web." 

  [Here's a link](http://search-24ways.herokuapp.com/24ways-f8f455f?sql=select+count%28*%29from+articles+where+rowid+in+%28select+rowid+from+articles_fts+where+articles_fts+match+%3Asearch%29&search=web) to the query I'd need to run to search "web" on FTS columns in `articles` table of the `24ways` database. 

  And [here's a link](http://search-24ways.herokuapp.com/24ways-f8f455f.json?sql=select+count%28*%29from+articles+where+rowid+in+%28select+rowid+from+articles_fts+where+articles_fts+match+%3Asearch%29&search=web) to the JSON version of the above result. I'd like to get the JSON result of that query for each FTS table of each database in my datasette project. 

  Is it possible in Javascript to automate the construction of query URLs like the one I linked, but for every FTS table in my datasette project?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504789231
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc4OTIzMQ==
created_at: 2019-06-23T21:35:33Z
updated_at: 2019-06-23T21:35:33Z
author_association: NONE
body:
  @russss 

  Thanks, just one more thing.

  I edited `datasette.service`:

  ```
  [Unit]
  Description=Datasette
  After=network.target

  [Service]
  Type=simple
  User=chris
  WorkingDirectory=/home/chris/digital-library
  ExecStart=/home/chris/Env/datasette/bin/datasette serve -h 0.0.0.0 databases/*.db --cors --metadata metadata.json
  Restart=on-failure

  [Install]
  WantedBy=multi-user.target
  ```

  Then ran:

  ```
  $ sudo systemctl daemon-reload
  $ sudo systemctl enable datasette
  $ sudo systemctl start datasette
  ```

  But the logs from `journalctl` show this datasette error:

  ```
  Jun 23 23:31:41 ns331247 datasette[1771]: Error: Invalid value for "[FILES]...": Path "databases/*.db" does not exist.
  Jun 23 23:31:44 ns331247 datasette[1778]: Usage: datasette serve [OPTIONS] [FILES]...
  Jun 23 23:31:44 ns331247 datasette[1778]: Try "datasette serve --help" for help.
  ```

  But the `databases` directory does exist in the directory specified by `WorkingDirectory`. Is this a datasette problem or did I write something incorrectly in the `.service` file?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/518#issuecomment-504790825
issue_url: https://api.github.com/repos/simonw/datasette/issues/518
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc5MDgyNQ==
created_at: 2019-06-23T21:57:58Z
updated_at: 2019-06-23T21:57:58Z
author_association: OWNER
body:
  The big one: **Replace Sanic request and response objects with my own classes, so I can remove Sanic dependency**

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/518#issuecomment-504791053
issue_url: https://api.github.com/repos/simonw/datasette/issues/518
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc5MTA1Mw==
created_at: 2019-06-23T22:00:56Z
updated_at: 2019-06-23T22:00:56Z
author_association: OWNER
body:
  The `InvalidUsage` exception is thrown by Sanic when it gets an unhandled request - usually a HEAD. It was added in efbb4e83374a2c795e436c72fa79f70da72309b8

  I'm going to replace it with specific handling for HEAD requests plus a unit test.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/518#issuecomment-504791796
issue_url: https://api.github.com/repos/simonw/datasette/issues/518
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc5MTc5Ng==
created_at: 2019-06-23T22:10:02Z
updated_at: 2019-06-24T13:42:50Z
author_association: OWNER
body:
  The Sanic stuff I import at the moment is:

  - [x] `from sanic.views import HTTPMethodView`
  - [x] `from sanic.request import Request` https://github.com/huge-success/sanic/blob/0.7.0/sanic/request.py#L44
  - [x] `from sanic.exceptions import NotFound`
  - [x] `from sanic import response` - used for `response.text("")`, `response.redirect(path)`, `response.HTTPResponse` and `response.html()` https://github.com/huge-success/sanic/blob/0.7.0/sanic/response.py
  - [x] `from sanic.request import RequestParameters` https://github.com/huge-success/sanic/blob/0.7.0/sanic/request.py#L30

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504793379
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc5MzM3OQ==
created_at: 2019-06-23T22:31:29Z
updated_at: 2019-06-23T22:31:48Z
author_association: OWNER
body:
  I suggest trying a full path in `ExecStart` like this:

  `ExecStart=/home/chris/Env/datasette/bin/datasette serve -h 0.0.0.0 /home/chris/digital-library/databases/*.db --cors --metadata /home/chris/digital-library/metadata.json`

  That should eliminate the chance of some kind of path confusion.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/498#issuecomment-504793988
issue_url: https://api.github.com/repos/simonw/datasette/issues/498
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc5Mzk4OA==
created_at: 2019-06-23T22:40:25Z
updated_at: 2019-06-23T22:41:25Z
author_association: OWNER
body:
  You can use a database query against `sqlite_master` to find all the FTS tables:

  http://sf-trees.datasettes.com/trees-b64f0cb?sql=select+name+from+sqlite_master+where+name+like+%22%25_fts%22+and+type+%3D+%22table%22

  `select name from sqlite_master where name like "%_fts" and type = "table"`

  You could then construct a crafty UNION query to get results back from all of those tables at once:

  http://sf-trees.datasettes.com/trees-b64f0cb?sql=select+%27PlantType_value_fts%27%2C+rowid%2C+value+from+PlantType_value_fts+where+PlantType_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qCareAssistant_value_fts%27%2C+rowid%2C+value+from+qCareAssistant_value_fts+where+qCareAssistant_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qCaretaker_value_fts%27%2C+rowid%2C+value+from+qCaretaker_value_fts+where+qCaretaker_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qLegalStatus_value_fts%27%2C+rowid%2C+value+from+qLegalStatus_value_fts+where+qLegalStatus_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qSiteInfo_value_fts%27%2C+rowid%2C+value+from+qSiteInfo_value_fts+where+qSiteInfo_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qSpecies_value_fts%27%2C+rowid%2C+value+from+qSpecies_value_fts+where+qSpecies_value_fts+match+%3Asearch&search=fi%2A

  (I'm searching for `fi*` here to demonstrate wildcards)

  The problem, as discussed earlier, is relevance: there's no way to compare the scores you're getting across different tables, so you won't be able to order by anything.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/518#issuecomment-504795648
issue_url: https://api.github.com/repos/simonw/datasette/issues/518
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc5NTY0OA==
created_at: 2019-06-23T23:07:06Z
updated_at: 2019-06-23T23:07:06Z
author_association: OWNER
body:
  For the request object.... what are the fields of it I actually use?

  * `request.url`
  * `request.query_string`
  * `request.path`
  * `request.method`
  * `request.args`
  * `request.raw_args`

  ALL of those are things that can be derived from the `scope` - so I think my new `Request` class (in `utils/asgi.py`) is just going to be a wrapper around a `scope`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/518#issuecomment-504798977
issue_url: https://api.github.com/repos/simonw/datasette/issues/518
node_id: MDEyOklzc3VlQ29tbWVudDUwNDc5ODk3Nw==
created_at: 2019-06-23T23:52:38Z
updated_at: 2019-06-23T23:52:38Z
author_association: OWNER
body:
  Last thing is to replace `sanic.response`:

  * `response.text("")`
  * `response.html()`
  * `response.redirect(path)`
  * `response.HTTPResponse`

  Implementations here: https://github.com/huge-success/sanic/blob/0.7.0/sanic/response.py#L175-L285

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/523#issuecomment-504807727
issue_url: https://api.github.com/repos/simonw/datasette/issues/523
node_id: MDEyOklzc3VlQ29tbWVudDUwNDgwNzcyNw==
created_at: 2019-06-24T01:24:07Z
updated_at: 2019-06-24T01:24:07Z
author_association: OWNER
body:
  For databases opened in immutable mode we pre-calculate the total row count for every table precisely so we can offer this kind of functionality without too much of a performance hit. The total row count is already available to the template (you can hit the .json endpoint to see it), so implementing this should be possible just by updating the template.

  For mutable databases we have a mechanism for attempting the count and giving up after a specified time limit - we can use that to get "3 of many".

  It looks like this is actually a dupe of #127 and #134.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/523#issuecomment-504809397
issue_url: https://api.github.com/repos/simonw/datasette/issues/523
node_id: MDEyOklzc3VlQ29tbWVudDUwNDgwOTM5Nw==
created_at: 2019-06-24T01:38:14Z
updated_at: 2019-06-24T01:38:14Z
author_association: CONTRIBUTOR
body:
  Ah, apologies – I had found and read those issues, but I was under the impression that they refered only to the filtered row count, not the unfiltered total row count.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-504844339
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDUwNDg0NDMzOQ==
created_at: 2019-06-24T03:33:06Z
updated_at: 2019-06-24T03:33:06Z
author_association: OWNER
body:
  It's alive! Here's the first deployed version: https://a559123.datasette.io/

  You can confirm it's running under ASGI by viewing https://a559123.datasette.io/-/versions and looking for the `"asgi"` key.

  Compare to the last version of master running on Sanic here: http://aa91112.datasette.io/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/398#issuecomment-504863901
issue_url: https://api.github.com/repos/simonw/datasette/issues/398
node_id: MDEyOklzc3VlQ29tbWVudDUwNDg2MzkwMQ==
created_at: 2019-06-24T05:33:26Z
updated_at: 2019-06-24T05:33:26Z
author_association: OWNER
body:
  I no longer depend on Sanic so I should be able to solve this entirely within the Datasette codebase. I need to figure out how `Transfer-Encoding: chunked` and the `Content-Length` header interact.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/268#issuecomment-504880796
issue_url: https://api.github.com/repos/simonw/datasette/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDUwNDg4MDc5Ng==
created_at: 2019-06-24T06:47:23Z
updated_at: 2019-06-24T06:47:23Z
author_association: OWNER
body:
  I did a bunch of research relevant to this a while ago: https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/48#issuecomment-504883688
issue_url: https://api.github.com/repos/simonw/datasette/issues/48
node_id: MDEyOklzc3VlQ29tbWVudDUwNDg4MzY4OA==
created_at: 2019-06-24T06:57:43Z
updated_at: 2019-06-24T06:57:43Z
author_association: OWNER
body:
  I've seen no evidence that JSON handling is even close to being a performance bottleneck, so wontfix.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-504998302
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNDk5ODMwMg==
created_at: 2019-06-24T12:57:19Z
updated_at: 2019-06-24T12:57:19Z
author_association: NONE
body:
  Same error when I used the full path.

  On Sun, Jun 23, 2019 at 18:31 Simon Willison <notifications@github.com>
  wrote:

  > I suggest trying a full path in ExecStart like this:
  >
  > ExecStart=/home/chris/Env/datasette/bin/datasette serve -h 0.0.0.0
  > /home/chris/digital-library/databases/*.db --cors --metadata metadata.json
  >
  > That should eliminate the chance of some kind of path confusion.
  >
  > —
  > You are receiving this because you authored the thread.
  > Reply to this email directly, view it on GitHub
  > <https://github.com/simonw/datasette/issues/514?email_source=notifications&email_token=AB4RUOZESVC5ZI2AZFJJD2TP372UJA5CNFSM4H2VJCSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYLISIY#issuecomment-504793379>,
  > or mute the thread
  > <https://github.com/notifications/unsubscribe-auth/AB4RUO243SPJ7YSWJU7YKF3P372UJANCNFSM4H2VJCSA>
  > .
  >
  -- 
  *Chris Persaud*
  ChrisPersaud.com

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/527#issuecomment-505052224
issue_url: https://api.github.com/repos/simonw/datasette/issues/527
node_id: MDEyOklzc3VlQ29tbWVudDUwNTA1MjIyNA==
created_at: 2019-06-24T15:08:52Z
updated_at: 2019-06-24T15:08:52Z
author_association: OWNER
body:
  The `select rank, *` feature is only available with FTS5 - it won't work with FTS4. So my best guess is that `csvs-to-sqlite` is setting up FTS with FTS4 when you want FTS5.

  ... I tested on my own machine and that is indeed what's happening! And in fact it looks like it's a known bug - I should fix that! https://github.com/simonw/csvs-to-sqlite/issues/41

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/527#issuecomment-505057520
issue_url: https://api.github.com/repos/simonw/datasette/issues/527
node_id: MDEyOklzc3VlQ29tbWVudDUwNTA1NzUyMA==
created_at: 2019-06-24T15:21:18Z
updated_at: 2019-06-24T15:21:18Z
author_association: OWNER
body:
  I just released csvs-to-sqlite 0.9.1 with this bug fix: https://github.com/simonw/csvs-to-sqlite/releases/tag/0.9.1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-505060332
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: MDEyOklzc3VlQ29tbWVudDUwNTA2MDMzMg==
created_at: 2019-06-24T15:28:16Z
updated_at: 2019-06-24T15:28:16Z
author_association: OWNER
body:
  This is currently a deliberate feature decision.

  The problem is that the streaming CSV feature relies on Datasette's automated efficient pagination under the hood. When you stream a CSV you're actually causing Datasette to paginate through the full set of "pages" under the hood, streaming each page out as a new chunk of CSV rows.

  This mechanism only works if the `next_url` has been generated for the page. Currently the `next_url` is available for table views (where it uses [the primary key or the sort column](https://datasette.readthedocs.io/en/stable/sql_queries.html#pagination)) and for views, but it's not set for canned queries because I can't be certain they can be efficiently paginated.

  Offset/limit pagination for canned queries would be a pretty nasty performance hit, because each subsequent page would require even more time for SQLite to scroll through to the specified offset.

  This does seem like it's worth fixing though: pulling every row for a canned queries would definitely be useful. The problem is that the pagination trick used elsewhere isn't right for canned queries - instead I would need to keep the database cursor open until ALL rows had been fetched. Figuring out how to do that efficiently within an asyncio managed thread pool may take some thought.

  Maybe this feature ends up as something which is turned off by default (due to the risk of it causing uptime problems for public sites) but that users working on their own private environments can turn on?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-505061703
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNTA2MTcwMw==
created_at: 2019-06-24T15:31:25Z
updated_at: 2019-06-24T15:31:25Z
author_association: OWNER
body:
  I'm suspicious of the wildcard. Does it work if you do the following?

      ExecStart=/home/chris/Env/datasette/bin/datasette serve -h 0.0.0.0 /home/chris/digital-library/databases/actual-database.db --cors --metadata /home/chris/digital-library/metadata.json

  If that does work then it means the ExecStart line doesn't support bash wildcard expansion. You'll need to create a separate script like this:

  ```
  #!/bin/bash
  /home/chris/Env/datasette/bin/datasette serve -h 0.0.0.0 /home/chris/digital-library/databases/*.db --cors --metadata /home/chris/digital-library/metadata.json
  ```

  Then save that as `/home/chris/digital-library/run-datasette.sh` and try this:

      ExecStart=/home/chris/digital-library/run-datasette.sh

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/525#issuecomment-505083671
issue_url: https://api.github.com/repos/simonw/datasette/issues/525
node_id: MDEyOklzc3VlQ29tbWVudDUwNTA4MzY3MQ==
created_at: 2019-06-24T16:29:30Z
updated_at: 2019-06-24T16:29:30Z
author_association: OWNER
body:
  It's mentioned here at the moment, but I'm going to expand that:

  https://github.com/simonw/datasette/blob/34e292d24dc2b8376236472bec3cce1c556ddfe5/docs/full_text_search.rst#L74

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/525#issuecomment-505086213
issue_url: https://api.github.com/repos/simonw/datasette/issues/525
node_id: MDEyOklzc3VlQ29tbWVudDUwNTA4NjIxMw==
created_at: 2019-06-24T16:36:35Z
updated_at: 2019-06-24T16:36:35Z
author_association: OWNER
body:
  https://datasette.readthedocs.io/en/latest/full_text_search.html#adding-full-text-search-to-a-sqlite-table

  <img width="683" alt="Full-text_search_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/60036214-8cd1d000-9663-11e9-9ee0-11d64711f5ad.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/437#issuecomment-505087020
issue_url: https://api.github.com/repos/simonw/datasette/issues/437
node_id: MDEyOklzc3VlQ29tbWVudDUwNTA4NzAyMA==
created_at: 2019-06-24T16:38:56Z
updated_at: 2019-06-24T16:38:56Z
author_association: OWNER
body:
  Closing this because it doesn't really fit the new model of inspect (though we should discuss in #465 how to further evolve this feature) and because as-of #272 we no longer use Sanic - though #520 will implement the equivalent of `prepare_sanic` against ASGI.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-505161008
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: MDEyOklzc3VlQ29tbWVudDUwNTE2MTAwOA==
created_at: 2019-06-24T20:11:15Z
updated_at: 2019-06-24T20:11:15Z
author_association: OWNER
body:
  Views already use offset/limit pagination so actually I may be over-thinking this.

  Maybe the right thing to do here is to have the feature enabled by default, since it will work for the VAST majority of queries - the only ones that might cause problems are complex queries across millions of rows. It can continue to use aggressive internal time limits so if someone DOES trigger something expensive they'll get an error.

  I can allow users to disable the feature with a config setting, or increase the time limit if they need to.

  Downgrading this from a medium to a small since it's much less effort to enable the existing pagination method for this type of query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-505162238
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: MDEyOklzc3VlQ29tbWVudDUwNTE2MjIzOA==
created_at: 2019-06-24T20:14:51Z
updated_at: 2019-06-24T20:14:51Z
author_association: OWNER
body:
  The other reason I didn't implement this in the first place is that adding offset/limit to a custom query (as opposed to a view) requires modifying the existing SQL - what if that SQL already has its own offset/limit clause?

  It looks like I can solve that using a nested query:
  ```sql
  select * from (
    select * from compound_three_primary_keys limit 1000
  ) limit 10 offset 100
  ```
  https://latest.datasette.io/fixtures?sql=select+*+from+%28%0D%0A++select+*+from+compound_three_primary_keys+limit+1000%0D%0A%29+limit+10+offset+100

  So I can wrap any user-provided SQL query in an outer offset/limit and implement pagination that way.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/498#issuecomment-505228873
issue_url: https://api.github.com/repos/simonw/datasette/issues/498
node_id: MDEyOklzc3VlQ29tbWVudDUwNTIyODg3Mw==
created_at: 2019-06-25T00:21:17Z
updated_at: 2019-06-25T00:21:17Z
author_association: NONE
body:
  Eh, I'm not concerned with a relevance score right now. I think I'd be fine with a search whose results show links to data tables with at least one result.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-505232675
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDUwNTIzMjY3NQ==
created_at: 2019-06-25T00:43:12Z
updated_at: 2019-06-25T00:43:12Z
author_association: NONE
body:
  Yep, that worked to get the site up and running at `my-server.com:8000` but when I edited `run-datasette.sh` to contain this...

  ```
  #!/bin/bash
  /home/chris/Env/datasette/bin/datasette serve -h 0.0.0.0 -p 80 /home/chris/digital-library/databases/*.db --cors --metadata /home/chris/digital-library/metadata.json
  ```

  I got this error.

  ```
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]: [2019-06-25 02:42:41 +0200] [752] [INFO] Goin' Fast @ http://0.0.0.0:80
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]: [2019-06-25 02:42:41 +0200] [752] [ERROR] Unable to start server
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]: Traceback (most recent call last):
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]:   File "uvloop/loop.pyx", line 1111, in uvloop.loop.Loop._create_server
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]:   File "uvloop/handles/tcp.pyx", line 89, in uvloop.loop.TCPServer.bind
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]:   File "uvloop/handles/streamserver.pyx", line 95, in uvloop.loop.UVStreamServer._fatal_error
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]:   File "uvloop/handles/tcp.pyx", line 87, in uvloop.loop.TCPServer.bind
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]:   File "uvloop/handles/tcp.pyx", line 26, in uvloop.loop.__tcp_bind
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]: PermissionError: [Errno 13] Permission denied
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]: During handling of the above exception, another exception occurred:
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]: Traceback (most recent call last):
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]:   File "/home/chris/Env/datasette/lib/python3.7/site-packages/sanic/server.py", line 591, in serve
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]:     http_server = loop.run_until_complete(server_coroutine)
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]:   File "uvloop/loop.pyx", line 1451, in uvloop.loop.Loop.run_until_complete
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]:   File "uvloop/loop.pyx", line 1684, in create_server
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]:   File "uvloop/loop.pyx", line 1116, in uvloop.loop.Loop._create_server
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]: PermissionError: [Errno 13] error while attempting to bind on address ('0.0.0.0', 80): permission denied
  Jun 25 02:42:41 ns331247 run-datasette.sh[747]: [2019-06-25 02:42:41 +0200] [752] [INFO] Server Stopped
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/521#issuecomment-505279560
issue_url: https://api.github.com/repos/simonw/datasette/issues/521
node_id: MDEyOklzc3VlQ29tbWVudDUwNTI3OTU2MA==
created_at: 2019-06-25T04:40:16Z
updated_at: 2019-06-25T04:40:38Z
author_association: OWNER
body:
  I think the trick is to redefine what a "cell_row" is. Each row is currently a list of cells:

  https://github.com/simonw/datasette/blob/6341f8cbc7833022012804dea120b838ec1f6558/datasette/views/table.py#L159-L163

  I can redefine the row (the `cells` variable in the above example) as a thing-that-iterates-cells (hence behaving like a list) but that also supports `__getitem__` access for looking up cell values if you know the name of the column.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/521#issuecomment-505280611
issue_url: https://api.github.com/repos/simonw/datasette/issues/521
node_id: MDEyOklzc3VlQ29tbWVudDUwNTI4MDYxMQ==
created_at: 2019-06-25T04:46:10Z
updated_at: 2019-06-25T04:46:10Z
author_association: OWNER
body:
  Of course I can't do it in this part of he code because it's constructing plain JSON objects here. 

  I'll need to do it in the section that prepares special objects for the HTML.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/521#issuecomment-505405030
issue_url: https://api.github.com/repos/simonw/datasette/issues/521
node_id: MDEyOklzc3VlQ29tbWVudDUwNTQwNTAzMA==
created_at: 2019-06-25T11:34:21Z
updated_at: 2019-06-25T11:34:21Z
author_association: OWNER
body:
  Actually I'm OK here - `display_columns_and_rows()` is already only used by the HTML template rendering path.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/521#issuecomment-505421892
issue_url: https://api.github.com/repos/simonw/datasette/issues/521
node_id: MDEyOklzc3VlQ29tbWVudDUwNTQyMTg5Mg==
created_at: 2019-06-25T12:26:54Z
updated_at: 2019-06-25T12:26:54Z
author_association: OWNER
body:
  This is starting to shape up. Example template:
  ```
  {% for row in display_rows %}
    <div> 
      <hr>
      <h2 class="scientist">
        {{ row["First_Name"] }} {{ row["Last_Name"] }}
      </h2>
      {% if row["Link_to_Photo"] %}
        <img style="max-width: 300px" src="{{ row["Link_to_Photo"] }}">
      {% endif %}
      {% for cell in row.cells[3:] %}
        {% if cell.raw and cell.column != "createdTime" %}
          <p><strong>{{ cell.column }}: </strong>{{ cell.value }}</p>
        {% endif %}
      {% endfor %}
    </div>
  {% endfor %}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/521#issuecomment-505422550
issue_url: https://api.github.com/repos/simonw/datasette/issues/521
node_id: MDEyOklzc3VlQ29tbWVudDUwNTQyMjU1MA==
created_at: 2019-06-25T12:28:48Z
updated_at: 2019-06-25T12:28:48Z
author_association: OWNER
body:
  Still needed before merge: documentation! I can expand this section: https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-templates

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/498#issuecomment-505635729
issue_url: https://api.github.com/repos/simonw/datasette/issues/498
node_id: MDEyOklzc3VlQ29tbWVudDUwNTYzNTcyOQ==
created_at: 2019-06-25T21:48:46Z
updated_at: 2019-06-25T21:48:46Z
author_association: OWNER
body:
  Hmmm... you know I think I actually need this for my https://github.com/simonw/fara-datasette project...

  https://fara.datasettes.com/fara?sql=select+1+as+sortme%2C+%27FARA_All_RegistrantDocs%27+as+%27table%27%2C+rowid+as+id%2C+Document_Type+%7C%7C+%27+%27+%7C%7C+Short_Form_Name+%7C%7C+%27+for+%27+%7C%7C+Registrant_Name+as+title%0D%0Afrom+FARA_All_RegistrantDocs%0D%0A++where+rowid+in+%28select+rowid+from+FARA_All_RegistrantDocs_fts+where+FARA_All_RegistrantDocs_fts+match+%3Asearch%0D%0A++++order+by+rank+limit+10%0D%0A%29%0D%0Aunion%0D%0Aselect+2+as+sortme%2C+%27FARA_All_ForeignPrincipals%27+as+%27table%27%2C+rowid+as+id%2C+Foreign_Principal+%7C%7C+%27+for+%27+%7C%7C+%22Country%2FLocation_Represented%22+as+title%0D%0Afrom+FARA_All_ForeignPrincipals%0D%0A++where+rowid+in+%28select+rowid+from+FARA_All_ForeignPrincipals_fts+where+FARA_All_ForeignPrincipals_fts+match+%3Asearch%0D%0A++++order+by+rank+limit+10%0D%0A%29%0D%0Aunion%0D%0Aselect+3+as+sortme%2C+%27FARA_All_Registrants%27+as+%27table%27%2C+rowid+as+id%2C+Name+%7C%7C+%27+on+%27+%7C%7C+Registration_Date+%7C%7C+%27+in+%27+%7C%7C+City+as+title%0D%0Afrom+FARA_All_Registrants%0D%0A++where+rowid+in+%28select+rowid+from+FARA_All_Registrants_fts+where+FARA_All_Registrants_fts+match+%3Asearch%0D%0A++++order+by+rank+limit+10%0D%0A%29%0D%0Aunion%0D%0Aselect+4+as+sortme%2C+%27FARA_All_ShortForms%27+as+%27table%27%2C+rowid+as+id%2C+Short_Form_First_Name+%7C%7C+%27+%27+%7C%7C+Short_Form_Last_Name+%7C%7C+%27+in+%27+%7C%7C+City+as+title%0D%0Afrom+FARA_All_ShortForms%0D%0A++where+rowid+in+%28select+rowid+from+FARA_All_ShortForms_fts+where+FARA_All_ShortForms_fts+match+%3Asearch%0D%0A++++order+by+rank+limit+10%0D%0A%29%0D%0A%0D%0A&search=cohen

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/522#issuecomment-506000023
issue_url: https://api.github.com/repos/simonw/datasette/issues/522
node_id: MDEyOklzc3VlQ29tbWVudDUwNjAwMDAyMw==
created_at: 2019-06-26T18:48:53Z
updated_at: 2019-06-26T18:48:53Z
author_association: NONE
body:
  Reference implementation from Requests: https://github.com/kennethreitz/requests/blob/3.0/requests/structures.py#L14

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/32#issuecomment-506784912
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/32
node_id: MDEyOklzc3VlQ29tbWVudDUwNjc4NDkxMg==
created_at: 2019-06-28T15:55:13Z
updated_at: 2019-06-28T15:55:13Z
author_association: OWNER
body:
  Documentation, to be added as a subsection of https://sqlite-utils.readthedocs.io/en/latest/python-api.html#adding-foreign-key-constraints

  ### Adding multiple foreign key constraints at once

  The final step in adding a new foreign key to a SQLite database is to run VACUUM, to ensure the new foreign key is available in future introspection queries.

  VACUUM against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using table.add_foreign_key(...) these can quickly add up.

  Instead, you can use db.add_foreign_keys(...) to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a table, column, other_table and other_column.

  Here's an example adding two foreign keys at once:

      db.add_foreign_keys([
          ("dogs", "breed_id", "breeds", "id"),
          ("dogs", "home_town_id", "towns", "id")
      ])

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/465#issuecomment-506797086
issue_url: https://api.github.com/repos/simonw/datasette/issues/465
node_id: MDEyOklzc3VlQ29tbWVudDUwNjc5NzA4Ng==
created_at: 2019-06-28T16:32:33Z
updated_at: 2019-06-28T16:32:33Z
author_association: OWNER
body:
  Next step:

  - [ ] build `/database/-/inspect` which shows tables, columns, column types and foreign keys

  It won't show table counts. Or maybe it will include them optionally but only for `-I` databases, in a special area of the JSON reserved for immutable-only inspect details.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/498#issuecomment-506985050
issue_url: https://api.github.com/repos/simonw/datasette/issues/498
node_id: MDEyOklzc3VlQ29tbWVudDUwNjk4NTA1MA==
created_at: 2019-06-29T20:28:21Z
updated_at: 2019-06-29T20:28:21Z
author_association: NONE
body:
  In my case, I have an ever-growing number of databases and tables within them. Most tables have FTS enabled. I cannot predict the names of future tables and databases, nor can I predict the names of the columns for which I wish to enable FTS.

  For my purposes, I was thinking of writing up something that sends these two GET requests to each of my databases' tables.

  ```
  http://my-server.com/database-name/table-name.json?_search=mySearchString
  http://my-server.com/database-name/table-name.json
  ```

  In the resulting JSON strings, I'd check the value of the key `filtered_table_rows_count`. If the value is `0` in the first URL's result, or if values from both requests are the same, that means FTS is either disabled for the table or it has no rows matching the search query.

  Is this feasible within the datasette library, or would it require some type of plugin? Or maybe you know of a better way of accomplishing this goal. Maybe I overlooked something.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507050894
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDUwNzA1MDg5NA==
created_at: 2019-06-30T16:51:39Z
updated_at: 2019-06-30T16:55:16Z
author_association: OWNER
body:
  One option:

      db.insert_m2m("dogs", 1, "breeds", 3)

  This would create the missing m2m table if it did not already exist, and call it `dogs_breeds`.

  You can pass an optional `m2m_table=` argument to specify the table name:

      db.insert_m2m("dogs", 1, "breeds", 3, m2m_table="dog_m2m_breeds")

  Maybe allow passing extra key/value pairs that will be inserted into the m2m table?

      db.insert_m2m("dogs", 1, "breeds", 3, m2m_extra={"classified_on_date": "2019-04-01"})

  Could even accept `alter=True` to cause the m2m table to be automatically altered to fit any `m2m_extra=` columns that do not already exist.

  It would be nice if there was a `table.*` method for this which allowed you to send three arguments rather than four. Maybe something like this:

      db["dogs"].insert_m2m(1, "breeds", 3)

  It's a bit confusing them both having the same method name though. Also calling it `insert_m2m` here is odd because you're not inserting into the `dogs` table.

  Maybe this instead?

      db["dogs"].m2m(1, "breeds", 3)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507051067
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDUwNzA1MTA2Nw==
created_at: 2019-06-30T16:54:08Z
updated_at: 2019-06-30T16:59:05Z
author_association: OWNER
body:
  There could be a version of this which inserts the related record into the other table if it is missing, but reuses it if it's already there. That might look like this:

      db["events"].m2m(3, "venues", {
          "id": 4,
          "venue_name": "Blah",
          "latitude": 37.77
          "longitude": 122.42
      }, pk="id")

  Since this is doing an `.upsert()` against `venues`, the `.m2m()` method could accept all of the other arguments to `.upsert()` - probably `pk=`, `column_order=` and `ignore=True` as well (actually borrowed from `.insert(...)`, it would mean don't update this row if you find it).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507051516
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDUwNzA1MTUxNg==
created_at: 2019-06-30T17:01:26Z
updated_at: 2019-06-30T17:03:16Z
author_association: OWNER
body:
  Question: do I set a compound primary key on `(event_id, venue_id)` or do I add an auto-incrementing surrogate key as well?

  I'm leaning towards compound primary key. Maybe that's the default and there's an option for including a regular incrementing primary key.
  ```
  db["events"].m2m(3, "venues", {
      "id": 4,
      "venue_name": "Blah",
      "latitude": 37.77
      "longitude": 122.42
  }, pk="id", m2m_surrogate_id=True)
  ```
  If you use `m2m_surrogate_id` I'll still make sure to set up a unique constraint on `(event_id, venue_id)`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507051825
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDUwNzA1MTgyNQ==
created_at: 2019-06-30T17:06:29Z
updated_at: 2019-06-30T17:06:29Z
author_association: OWNER
body:
  For this method:
  ```python
  db["events"].m2m(3, "venues", {
      "id": 4,
      "venue_name": "Blah",
      "latitude": 37.77
      "longitude": 122.42
  }, pk="id")
  ```
  Let's say that the first argument is allowed to be either a string/integer representing a primary key, OR it can be a full row record (a dictionary) which will have its primary key automatically extracted based on the table definition.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507052051
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDUwNzA1MjA1MQ==
created_at: 2019-06-30T17:09:51Z
updated_at: 2019-06-30T17:15:09Z
author_association: OWNER
body:
  Maybe there's a case where we want to be able to insert a row AND add its m2m records in a single operation? Could look something like this:

  ```python
  db["events"].insert({
    "id": 1,
    "title": "Some event"
  }, m2m={
    "venues": {
       "id": 4,
       "venue_name": "Blah",
       "latitude": 37.77
       "longitude": 122.42
    }
  })
  ```
  The `venues` key in that dictionary could be a single dictionary or a list of dictionary.

  I'm not sure about this though. If the `m2m=` argument causes the creation of the `venues` table, how would we pass in extra creation options like `pk=` for that table?

  Maybe some complex nested function mechanism like this?

  ```python
  db["events"].insert({
    "id": 1,
    "title": "Some event"
  }, m2m=[db["venues"].upsert({
       "id": 4,
       "venue_name": "Blah",
       "latitude": 37.77
       "longitude": 122.42
    }, pk="id")]
  })
  ```

  This would require having `.insert()` and friends return a more interesting object than they do at the moment. They currently return `self` to support chaining. Changing this would be a backwards-incompatible change and hence would require a major version bump... _unless_ they continued to return `self` but remembered the last inserted record in a way that could be handled by that `m2m=` argument.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507053558
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDUwNzA1MzU1OA==
created_at: 2019-06-30T17:34:29Z
updated_at: 2019-06-30T17:34:29Z
author_association: OWNER
body:
  Or... how about we use chaining like this:

  ```python
  db["events"].insert({...}, pk="id").m2m("venues", {...}, pk="id")
  ```
  This makes me think that maybe there should be some kind of mechanism for saying "I'd should always be treated as a primary key". Maybe as an argument passed to the `Database` constructor.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507053731
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDUwNzA1MzczMQ==
created_at: 2019-06-30T17:37:33Z
updated_at: 2019-06-30T17:37:33Z
author_association: OWNER
body:
  Implementing the `table.m2m()` method to have an optional _first_ argument (optional in that if it's in a chained operation it uses the last inserted thing) could be tricky. How to tell the difference between the two uses? A different method name would be better.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507055345
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDUwNzA1NTM0NQ==
created_at: 2019-06-30T18:02:25Z
updated_at: 2019-06-30T18:02:33Z
author_association: OWNER
body:
  I have a solution. If I introduce a `db[table_name].update(row_pk, {...})` method I can use that method without the dictionary argument to select a specific row as the "currently editing / last inserted" row in the table context. Then I can support the following:
  ```python
  db["events"].update(3).m2m("venues", {...}, pk="id")
  ```
  This means that the `table.m2m()` method will only work in the context of a chain, and will throw an error if you attempt to call it without first using `.insert()` or `.update()` to select a record that you will be manipulating first.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/533#issuecomment-507895104
issue_url: https://api.github.com/repos/simonw/datasette/issues/533
node_id: MDEyOklzc3VlQ29tbWVudDUwNzg5NTEwNA==
created_at: 2019-07-03T00:46:20Z
updated_at: 2019-07-03T00:46:33Z
author_association: OWNER
body:
  I'd like to rename `_rows_and_columns.html` to `_table.html`.

  There is at least one project out there using the current filename though: https://github.com/baltimore-sun-data/salaries-datasette/blob/master/templates/_rows_and_columns.html - I shall file an issue there to let them know about the backwards incompatible change.

reactions:
[cell repeats an earlier one in this file: 112 characters]


