# issue_comments

html_url: https://github.com/simonw/datasette/issues/859#issuecomment-647189666
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0NzE4OTY2Ng==
created_at: 2020-06-21T22:26:55Z
updated_at: 2020-06-21T22:26:55Z
author_association: OWNER
body:
  This makes a lot of sense. I implemented the mechanism for the index page because I have my own instance of Datasette that was running slow, but it had a dozen database files attached to it. I've not run into this with a single giant database file but it absolutely makes sense that the same optimization would be necessary for the database page there too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-647189948
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0NzE4OTk0OA==
created_at: 2020-06-21T22:30:12Z
updated_at: 2020-06-21T22:30:43Z
author_association: OWNER
body:
  I'll write a little script which generates a 300MB SQLite file with a bunch of tables with lots of randomly generated rows in to help test this.

  Having a tool like that which can generate larger databases with different gnarly performance characteristics will be useful for other performance work too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/687#issuecomment-647190144
issue_url: https://api.github.com/repos/simonw/datasette/issues/687
node_id: MDEyOklzc3VlQ29tbWVudDY0NzE5MDE0NA==
created_at: 2020-06-21T22:32:13Z
updated_at: 2020-06-21T22:32:13Z
author_association: OWNER
body:
  So the new plan is NOT to have a `plugins/` folder, but instead have several top-level pages:

  - Plugins (exists)
  - Writing plugins
  - Plugin hooks
  - Testing plugins
  - Internals for plugins (exists)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/687#issuecomment-647190177
issue_url: https://api.github.com/repos/simonw/datasette/issues/687
node_id: MDEyOklzc3VlQ29tbWVudDY0NzE5MDE3Nw==
created_at: 2020-06-21T22:32:36Z
updated_at: 2020-06-21T22:32:36Z
author_association: OWNER
body:
  I'm going to break out the plugin hooks first in a single commit to make for a cleaner commit history (since that way git can hopefully detect that the content moved).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-647194131
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0NzE5NDEzMQ==
created_at: 2020-06-21T23:15:54Z
updated_at: 2020-06-21T23:26:09Z
author_association: CONTRIBUTOR
body:
  I'm not sure if table counts are to blame. There shouldn't be a ~3 orders of magnitude difference.

  ```fish
  user@klein /a/w/scrapyard (master)> set sql "select count(*) from table_1; select count(*) from table_2; select count(*) from table_3;"
  user@klein /a/w/scrapyard (master)> time sqlite3 scrapyard.db "$sql"
  187489
  46492
  2229

  ________________________________________________________
  Executed in   25.57 millis    fish           external
     usr time    3.55 millis    0.00 micros    3.55 millis
     sys time   22.42 millis  1123.00 micros   21.30 millis
  ```

  but not letting datasette count the tables definitely helps.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/687#issuecomment-647203845
issue_url: https://api.github.com/repos/simonw/datasette/issues/687
node_id: MDEyOklzc3VlQ29tbWVudDY0NzIwMzg0NQ==
created_at: 2020-06-22T00:32:42Z
updated_at: 2020-06-22T00:32:42Z
author_association: OWNER
body:
  Maybe add this to the plugins.rst page near the top:
  ```

  .. toctree::
     :caption: See also
     :maxdepth: 1

     plugin_hooks
     internals
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/687#issuecomment-647237091
issue_url: https://api.github.com/repos/simonw/datasette/issues/687
node_id: MDEyOklzc3VlQ29tbWVudDY0NzIzNzA5MQ==
created_at: 2020-06-22T02:44:10Z
updated_at: 2020-06-22T02:44:10Z
author_association: OWNER
body:
  Now split into four pages:

  - https://datasette.readthedocs.io/en/latest/plugins.html
  - https://datasette.readthedocs.io/en/latest/writing_plugins.html
  - https://datasette.readthedocs.io/en/latest/plugin_hooks.html
  - https://datasette.readthedocs.io/en/latest/internals.html

  Still need to add the "Testing plugins" page, then I can close this issue.

  I should also do #855, documenting the new `datasette-plugin` cookiecutter template. That can go in `writing_plugins.rst`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/838#issuecomment-647803394
issue_url: https://api.github.com/repos/simonw/datasette/issues/838
node_id: MDEyOklzc3VlQ29tbWVudDY0NzgwMzM5NA==
created_at: 2020-06-22T22:36:34Z
updated_at: 2020-06-22T22:36:34Z
author_association: NONE
body:
  I also am seeing the same issue with an Apache setup (same even w/o `ProxyPassReverse`, though I typically use it as @tsibley stated).

  But also want to say thanks for a great tool (this issue not withstanding)!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/861#issuecomment-647822757
issue_url: https://api.github.com/repos/simonw/datasette/issues/861
node_id: MDEyOklzc3VlQ29tbWVudDY0NzgyMjc1Nw==
created_at: 2020-06-22T23:40:43Z
updated_at: 2020-06-22T23:40:43Z
author_association: OWNER
body:
  I started building that tool here: https://github.com/simonw/sqlite-generate

  (I built a new cookiecutter template for that too, https://github.com/simonw/click-app)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/861#issuecomment-647889674
issue_url: https://api.github.com/repos/simonw/datasette/issues/861
node_id: MDEyOklzc3VlQ29tbWVudDY0Nzg4OTY3NA==
created_at: 2020-06-23T03:44:17Z
updated_at: 2020-06-23T03:44:17Z
author_association: OWNER
body:
  https://github.com/simonw/sqlite-generate is now ready to be used - see also https://pypi.org/project/sqlite-generate/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-647890378
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0Nzg5MDM3OA==
created_at: 2020-06-23T03:47:19Z
updated_at: 2020-06-23T03:47:19Z
author_association: OWNER
body:
  I generated a 600MB database using [sqlite-generate](https://github.com/simonw/sqlite-generate) just now - with 100 tables at 100,00 rows and 3 tables at 1,000,000 rows - and performance of the database page was fine, 250ms.

  Those tables only had 4 columns each though.

  You said "200k+, 50+ rows in a couple of tables" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-647890619
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0Nzg5MDYxOQ==
created_at: 2020-06-23T03:48:21Z
updated_at: 2020-06-23T03:48:21Z
author_association: OWNER
body:
      sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50

  Looks like that will take 35 minutes to run (it's not a particularly fast tool).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/862#issuecomment-647892930
issue_url: https://api.github.com/repos/simonw/datasette/issues/862
node_id: MDEyOklzc3VlQ29tbWVudDY0Nzg5MjkzMA==
created_at: 2020-06-23T03:58:48Z
updated_at: 2020-06-23T03:58:48Z
author_association: OWNER
body:
  Should this be controlled be a separate configuration setting? I'm inclined to say no - I think instead I'll set the limit to be 10 * whatever `facet_suggest_time_limit_ms` is.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-647894903
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0Nzg5NDkwMw==
created_at: 2020-06-23T04:07:59Z
updated_at: 2020-06-23T04:07:59Z
author_association: OWNER
body:
  Just to check: are you seeing the problem on this page: https://latest.datasette.io/fixtures (the database page) - or this page (the table page): https://latest.datasette.io/fixtures/compound_three_primary_keys

  If it's the table page then the problem may well be #862.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-647922203
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0NzkyMjIwMw==
created_at: 2020-06-23T05:44:58Z
updated_at: 2021-01-05T08:22:43Z
author_association: CONTRIBUTOR
body:
  I'm seeing the problem on database page. Index page and table page runs quite fast.

  - Tables have <10 columns (`id`, `url`, `title`, `body_html`, `date`, `author`, `meta` (for keeping unstructured json)). I've added index on `date` columns (using `sqlite-utils`) in addition to the index present on `id` columns. 
  - All tables have FTS enabled on `text` and `varchar` columns (`title`, `body_html` etc) to speed up searching.
  - There are couple of tables related with foreign keys (think a thread in a forum and posts in that thread, related with `thread_id`)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-647923666
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0NzkyMzY2Ng==
created_at: 2020-06-23T05:49:31Z
updated_at: 2020-06-23T05:49:31Z
author_association: CONTRIBUTOR
body:
  I think I should mention that having FTS on all tables mean I have 5 visible, 25 hidden (FTS) tables displayed on database page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-647925594
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0NzkyNTU5NA==
created_at: 2020-06-23T05:55:21Z
updated_at: 2020-06-23T06:28:29Z
author_association: CONTRIBUTOR
body:
  Hmm, not seeing the problem now.  
  I've removed the commented out sections in `database.py` and restarted the process. Database page now loads in <250ms.

  I have couple of workers that check some pages regularly and scrape new content and save to the DB. Could it be that datasette tries to recount tables every time database size changes? Normally it keeps a count cache, but as DB gets updated so often (new content every 5 min or so) it's practically recounting every time I go to the database page?

  EDIT: 
  It turns out it doesn't hold cache with mutable databases.

  I'll update the issue with more findings and a better way to reproduce the problem if I encounter it again.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-647935300
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0NzkzNTMwMA==
created_at: 2020-06-23T06:23:01Z
updated_at: 2020-06-23T06:23:01Z
author_association: CONTRIBUTOR
body:
  > You said "200k+, 50+ rows in a couple of tables" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.

  Ah that was a typo, I meant 50k.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-647936117
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0NzkzNjExNw==
created_at: 2020-06-23T06:25:17Z
updated_at: 2020-06-23T06:25:17Z
author_association: CONTRIBUTOR
body:
  > 
  > 
  > ```
  > sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50
  > ```
  > 
  > Looks like that will take 35 minutes to run (it's not a particularly fast tool).

  Try chunking write operations into batches every 1000 records or so.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-648163272
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0ODE2MzI3Mg==
created_at: 2020-06-23T13:52:23Z
updated_at: 2020-06-23T13:52:23Z
author_association: OWNER
body:
  I'm chunking inserts at 100 at a time right now: https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L1030

  I think the performance is more down to using Faker to create the test data - generating millions of entirely fake, randomized records takes a fair bit of time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-648232645
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0ODIzMjY0NQ==
created_at: 2020-06-23T15:19:53Z
updated_at: 2020-06-23T15:19:53Z
author_association: CONTRIBUTOR
body:
  The issue seems to appear sporadically, like when I return to database page after a while, during which some records have been added to the database.

  I've just visited database, page first visit took ~10s, consecutive visits took 0.3s.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-648234787
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0ODIzNDc4Nw==
created_at: 2020-06-23T15:22:51Z
updated_at: 2020-06-23T15:22:51Z
author_association: OWNER
body:
  I wonder if this is a SQLite caching issue then?

  Datasette has a configuration option for this but I haven't spent much time experimenting with it so I don't know how much of an impact it can have: https://datasette.readthedocs.io/en/stable/config.html#cache-size-kb

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/694#issuecomment-648296323
issue_url: https://api.github.com/repos/simonw/datasette/issues/694
node_id: MDEyOklzc3VlQ29tbWVudDY0ODI5NjMyMw==
created_at: 2020-06-23T17:10:51Z
updated_at: 2020-06-23T17:10:51Z
author_association: NONE
body:
  @simonw 

  Did you find the reason? I had similar situation and I check this on millions ways. I am sure app doesn't consume such memory.

  I was trying the app with:
  `docker run --rm -it -p 80:80 -m 128M foo`

  I was watching app with `docker stats`. Even limited memory by `CMD ["java", "-Xms60M", "-Xmx60M", "-jar", "api.jar"]`.
  Checked memory usage by app in code and print bash commands. The app definitely doesn't use this memory. Also doesn't write files.

  Only one solution is to change memory to 512M.

  It is definitely something wrong with `cloud run`.

  I even did special app for testing this. It looks like when I cross very small amount of code / memory / app size in random when, then memory needs grow +hundreds. Nothing make sense here. Especially it works everywhere expect cloud run.

  Please let me know if you discovered something more.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/116#issuecomment-648403834
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/116
node_id: MDEyOklzc3VlQ29tbWVudDY0ODQwMzgzNA==
created_at: 2020-06-23T20:36:29Z
updated_at: 2020-06-23T20:36:29Z
author_association: OWNER
body:
  Should go in this section https://sqlite-utils.readthedocs.io/en/stable/python-api.html#introspection - under `.columns_dict` and before `.foreign_keys`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648440525
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/117
node_id: MDEyOklzc3VlQ29tbWVudDY0ODQ0MDUyNQ==
created_at: 2020-06-23T21:35:01Z
updated_at: 2020-06-23T21:35:01Z
author_association: OWNER
body:
  Here's what's missing:
  ```
  In [11]: db.conn.execute('PRAGMA foreign_key_list(song)').fetchall()                                                                       
  Out[11]: 
  [(0,
    0,
    'album',
    'songartist',
    'albumartist',
    'NO ACTION',
    'NO ACTION',
    'NONE'),
   (0, 1, 'album', 'songalbum', 'albumname', 'NO ACTION', 'NO ACTION', 'NONE')]
  ```
  Compare with this code here:
  https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/sqlite_utils/db.py#L563-L579

  The first two columns returned by `PRAGMA foreign_key_list(table)` are `id` and `seq` - these show when two foreign key records are part of the same compound foreign key. `sqlite-utils` entirely ignores those at the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648442511
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/117
node_id: MDEyOklzc3VlQ29tbWVudDY0ODQ0MjUxMQ==
created_at: 2020-06-23T21:39:41Z
updated_at: 2020-06-23T21:39:41Z
author_association: OWNER
body:
  So there are two sides to supporting this:

  - Being able to sensibly introspect composite foreign keys
  - Being able to define composite foreign keys when creating a table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/864#issuecomment-648580236
issue_url: https://api.github.com/repos/simonw/datasette/issues/864
node_id: MDEyOklzc3VlQ29tbWVudDY0ODU4MDIzNg==
created_at: 2020-06-24T04:39:39Z
updated_at: 2020-06-24T04:39:39Z
author_association: OWNER
body:
  Urgh, fixing this is going to be a bit of a pain.

  Here's where I added that custom `dispatch_request()` method - it was to implement flash messaging in #790: https://github.com/simonw/datasette/blame/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/views/base.py#L85

  If I want this to be made available to `register_routes()` views as well, I'm going to have to move the logic somewhere else. In particular I need to make sure that the `request` object is created once and used throughout the whole request cycle.

  Currently `register_routes()` view functions get their own separate request object which is created here:

  https://github.com/simonw/datasette/blob/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/app.py#L1057-L1068

  So I'm going to have to refactor this quite a bit to get that shared request object which can be passed both to `register_routes` views and to my various `BaseView` subclasses.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/864#issuecomment-648580556
issue_url: https://api.github.com/repos/simonw/datasette/issues/864
node_id: MDEyOklzc3VlQ29tbWVudDY0ODU4MDU1Ng==
created_at: 2020-06-24T04:40:49Z
updated_at: 2020-06-24T04:40:49Z
author_association: OWNER
body:
  The ideal fix here would be to rework my `BaseView` subclass mechanism to work with `register_routes()` so that those views don't have any special privileges above plugin-provided views.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-648669523
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0ODY2OTUyMw==
created_at: 2020-06-24T08:13:23Z
updated_at: 2020-06-24T10:30:36Z
author_association: CONTRIBUTOR
body:
  I tried setting `cache_size_kb=0` then `cache_size_kb=100000`, still getting this behavior. I even changed `Database::table_counts` and lowered time limit to 1

  ```py
  table_count = (
      await self.execute(
          "select count(*) from [{}]".format(table),
          custom_time_limit=1,
      )
  ).rows[0][0]
  counts[table] = table_count
  ```

  I feel like 10 seconds is a magic number, like a processing timeout and datasette gives up and returns the page. 
  Index page loads instantly, table page, query page, as well. But when I return to database page after some time, it loads in 10s.

  EDIT:

  It's always like 10 + 0.3s, like 10s wait and timeout then 300ms to render the page

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/866#issuecomment-648818707
issue_url: https://api.github.com/repos/simonw/datasette/issues/866
node_id: MDEyOklzc3VlQ29tbWVudDY0ODgxODcwNw==
created_at: 2020-06-24T13:26:14Z
updated_at: 2020-06-24T13:26:14Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=h1) Report
  > Merging [#866](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f&el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/866/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##           master     #866   +/-   ##
  =======================================
    Coverage   82.99%   82.99%           
  =======================================
    Files          26       26           
    Lines        3547     3547           
  =======================================
    Hits         2944     2944           
    Misses        603      603           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=footer). Last update [1a5b7d3...fb64dda](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/858#issuecomment-648997857
issue_url: https://api.github.com/repos/simonw/datasette/issues/858
node_id: MDEyOklzc3VlQ29tbWVudDY0ODk5Nzg1Nw==
created_at: 2020-06-24T18:42:10Z
updated_at: 2020-06-24T18:42:10Z
author_association: OWNER
body:
  I really need to get myself a Windows 10 development environment working so I can dig into this kind of bug properly. I have a gaming PC lying around that I could re-task for that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-649000075
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY0OTAwMDA3NQ==
created_at: 2020-06-24T18:46:36Z
updated_at: 2020-06-24T18:47:37Z
author_association: OWNER
body:
  Another magic parameter that would be useful would be `_random`. Consider https://github.com/simonw/datasette-auth-tokens/issues/1 for example - I'd like to be able to provide a writable canned query which can create new authentication tokens in the database, but ideally it would automatically populate a secure random secret for each one.

  Maybe `_random_chars_128` to create a 128 character long random string (using `os.urandom(64).hex()`).

  This would be the first example of a magic parameter where part of the parameter name is used to configure the resulting value. Maybe neater to separate that with a different character? Unfortunately `_random_chars:128` wouldn't work because these parameters are used in a SQLite query where `:` has special meaning: `insert into blah (secret) values (:_random_chars:128)` wouldn't make sense.

  Actually this is already supported by the proposed design - `_random_chars_128` would become `random("chars_128")` so the `random()` function could split off the 128 itself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-649014757
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY0OTAxNDc1Nw==
created_at: 2020-06-24T19:15:46Z
updated_at: 2020-06-24T19:31:52Z
author_association: OWNER
body:
  I'm building this documentation-first - here's the documentation so far: https://github.com/simonw/datasette/blob/6fc8bd9c473f4a25e0a076f24c7e5a9b2f353bb8/docs/sql_queries.rst#magic-parameters

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/849#issuecomment-649908756
issue_url: https://api.github.com/repos/simonw/datasette/issues/849
node_id: MDEyOklzc3VlQ29tbWVudDY0OTkwODc1Ng==
created_at: 2020-06-26T02:09:09Z
updated_at: 2022-10-27T13:57:08Z
author_association: OWNER
body:
  I mentioned this issue here: https://simonwillison.net/2020/Jun/26/weeknotes-plugins-sqlite-generate/

  Repositories created by following the README in https://github.com/simonw/datasette-template and https://github.com/simonw/click-app have a `main` branch instead of `master` so I have a few examples live now. https://github.com/simonw/datasette-saved-queries is one example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/867#issuecomment-649931714
issue_url: https://api.github.com/repos/simonw/datasette/issues/867
node_id: MDEyOklzc3VlQ29tbWVudDY0OTkzMTcxNA==
created_at: 2020-06-26T03:12:51Z
updated_at: 2020-06-26T03:12:51Z
author_association: OWNER
body:
  Here's the relevant code:
  https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/datasette/app.py#L1057-L1070
  And the relevant test code:
  https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/test_plugins.py#L567-L573

  https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/plugins/my_plugin.py#L162-L196

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/868#issuecomment-650340914
issue_url: https://api.github.com/repos/simonw/datasette/issues/868
node_id: MDEyOklzc3VlQ29tbWVudDY1MDM0MDkxNA==
created_at: 2020-06-26T18:53:02Z
updated_at: 2021-07-10T23:41:42Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#868](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b452fcb) into [master](https://codecov.io/gh/simonw/datasette/commit/000528192eaf891118932250141dabe7a1561ece?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0005281) will **increase** coverage by `0.49%`.
  > The diff coverage is `96.19%`.

  > :exclamation: Current head b452fcb differs from pull request most recent head c99caba. Consider uploading reports for the commit c99caba to get more accurate results
  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/868/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##           master     #868      +/-   ##
  ==========================================
  + Coverage   82.91%   83.40%   +0.49%     
  ==========================================
    Files          26       27       +1     
    Lines        3547     3634      +87     
  ==========================================
  + Hits         2941     3031      +90     
  + Misses        606      603       -3     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | |
  | [datasette/default\_magic\_parameters.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWFnaWNfcGFyYW1ldGVycy5weQ==) | `91.17% <91.17%> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.99% <97.91%> (+1.32%)` | :arrow_up: |
  | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.93% <100.00%> (+0.08%)` | :arrow_up: |
  | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `91.32% <100.00%> (+0.41%)` | :arrow_up: |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.39% <100.00%> (-0.01%)` | :arrow_down: |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.37% <100.00%> (-1.96%)` | :arrow_down: |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.67% <0.00%> (-0.03%)` | :arrow_down: |
  | ... and [6 more](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [180c7a5...c99caba](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-650455353
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY1MDQ1NTM1Mw==
created_at: 2020-06-26T23:55:40Z
updated_at: 2020-06-26T23:55:40Z
author_association: OWNER
body:
  `_request_ip` is actually quite hard to implement - should it take into account things like the `x-forwarded-for` header?

  It probably should - but that means it now needs a bunch of extra configuration to tell it which of those headers can be trusted in the current environment.

  As such I think I'll leave that for a plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-650455793
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY1MDQ1NTc5Mw==
created_at: 2020-06-26T23:57:30Z
updated_at: 2020-06-27T00:00:16Z
author_association: OWNER
body:
  Maybe I should ship a default `_scope_headers_...` parameter instead, which reads from a dictionary of `scope["headers"]` - https://asgi-scope.now.sh/ shows what those look like.

  ```
  {'client': ('148.64.98.14', 0),
   'headers': [[b'host', b'asgi-scope.now.sh'],
               [b'x-forwarded-for', b'148.64.98.14'],
               [b'x-vercel-id', b'sw72x-1593215573008-024e4e603806'],
               [b'x-forwarded-host', b'asgi-scope.now.sh'],
               [b'accept',
                b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
                b'webp,*/*;q=0.8'],
               [b'x-real-ip', b'148.64.98.14'],
               [b'x-vercel-deployment-url', b'asgi-scope-9eyeojbek.now.sh'],
               [b'upgrade-insecure-requests', b'1'],
               [b'x-vercel-trace', b'sfo1'],
               [b'x-forwarded-proto', b'https'],
               [b'accept-language', b'en-US,en;q=0.5'],
               [b'user-agent',
                b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko'
                b'/20100101 Firefox/77.0'],
               [b'x-vercel-forwarded-for', b'148.64.98.14'],
               [b'accept-encoding', b'gzip, deflate, br'],
               [b'dnt', b'1'],
               [b'te', b'trailers']],
   'http_version': '1.1',
   'method': 'GET',
   'path': '/',
   'query_string': b'',
   'raw_path': b'/',
   'root_path': '',
   'scheme': 'https',
   'server': ('asgi-scope.now.sh', 80),
   'type': 'http'}
  ```

  I'm going to have `_request_X` actually mean "find the first value for X in `scope["headers"`]" - with underscores converted to hyphens.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-650458857
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY1MDQ1ODg1Nw==
created_at: 2020-06-27T00:11:04Z
updated_at: 2020-06-27T00:11:04Z
author_association: OWNER
body:
  Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter `_cookie_ds_actor` to be resolved and returned as JSON data that the attacker can see.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-650593122
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY1MDU5MzEyMg==
created_at: 2020-06-27T18:03:02Z
updated_at: 2020-06-27T18:03:10Z
author_association: OWNER
body:
  > Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter `_cookie_ds_actor` to be resolved and returned as JSON data that the attacker can see.

  This is an open security hole in https://github.com/simonw/datasette/commit/94c1315f0030fd58ce46a9294052c5c9d9d181c7 - it's useful for testing, but I need to remove it before I land that branch.

  https://github.com/simonw/datasette/blob/94c1315f0030fd58ce46a9294052c5c9d9d181c7/datasette/views/database.py#L231-L237

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/835#issuecomment-650598710
issue_url: https://api.github.com/repos/simonw/datasette/issues/835
node_id: MDEyOklzc3VlQ29tbWVudDY1MDU5ODcxMA==
created_at: 2020-06-27T18:32:22Z
updated_at: 2020-06-27T18:32:22Z
author_association: OWNER
body:
  Skipping CSRF on `Authorization: Bearer xxx` headers also makes sense for JWT applications, which tend to send JWTs using that form of header.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/869#issuecomment-650600176
issue_url: https://api.github.com/repos/simonw/datasette/issues/869
node_id: MDEyOklzc3VlQ29tbWVudDY1MDYwMDE3Ng==
created_at: 2020-06-27T18:41:31Z
updated_at: 2020-06-28T02:54:21Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=h1) Report
  > Merging [#869](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c&el=desc) will **increase** coverage by `0.23%`.
  > The diff coverage is `90.62%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/869/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##           master     #869      +/-   ##
  ==========================================
  + Coverage   82.99%   83.23%   +0.23%     
  ==========================================
    Files          26       27       +1     
    Lines        3547     3609      +62     
  ==========================================
  + Hits         2944     3004      +60     
  - Misses        603      605       +2     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.45% <86.36%> (-1.88%)` | :arrow_down: |
  | [datasette/default\_magic\_parameters.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWFnaWNfcGFyYW1ldGVycy5weQ==) | `91.17% <91.17%> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.07% <100.00%> (+0.81%)` | :arrow_up: |
  | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.87% <100.00%> (+0.02%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=footer). Last update [1bb33da...9e693a7](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/868#issuecomment-650600606
issue_url: https://api.github.com/repos/simonw/datasette/issues/868
node_id: MDEyOklzc3VlQ29tbWVudDY1MDYwMDYwNg==
created_at: 2020-06-27T18:44:28Z
updated_at: 2020-06-27T18:44:28Z
author_association: OWNER
body:
  This is really exciting! Thanks so much for looking into this.

  I'm interested in moving CI for this repo over to GitHub Actions, so I'd be fine with you getting this to work as an Action rather than through Travis. If you can get it working in Travis though I'll happily land that and figure out how to convert that to GitHub Actions later on.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-650678951
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY1MDY3ODk1MQ==
created_at: 2020-06-28T02:59:52Z
updated_at: 2020-06-28T02:59:52Z
author_association: OWNER
body:
  Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#magic-parameters

  Plugin hook documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#plugin-hook-register-magic-parameters

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-650679100
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY1MDY3OTEwMA==
created_at: 2020-06-28T03:00:44Z
updated_at: 2020-06-28T03:00:44Z
author_association: OWNER
body:
  I'm going to add some canned queries to the `metadata.json` used by the live demo that illustrate this feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/805#issuecomment-650784162
issue_url: https://api.github.com/repos/simonw/datasette/issues/805
node_id: MDEyOklzc3VlQ29tbWVudDY1MDc4NDE2Mg==
created_at: 2020-06-28T15:48:32Z
updated_at: 2020-06-28T15:48:32Z
author_association: OWNER
body:
  https://github.com/simonw/datasette-glitch is my new plugin that outputs the root login link on Glitch when the server starts.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/834#issuecomment-650811919
issue_url: https://api.github.com/repos/simonw/datasette/issues/834
node_id: MDEyOklzc3VlQ29tbWVudDY1MDgxMTkxOQ==
created_at: 2020-06-28T19:38:50Z
updated_at: 2020-06-28T19:38:50Z
author_association: OWNER
body:
  I have two plugins in progress that use this hook now:

  - https://github.com/simonw/datasette-init creates tables and views on startup
  - https://github.com/simonw/datasette-glitch outputs the login-as-root secret link on Glitch

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/871#issuecomment-650812444
issue_url: https://api.github.com/repos/simonw/datasette/issues/871
node_id: MDEyOklzc3VlQ29tbWVudDY1MDgxMjQ0NA==
created_at: 2020-06-28T19:43:27Z
updated_at: 2020-06-28T19:43:27Z
author_association: OWNER
body:
  Currently:

  > `_timestamp_epoch`
  > 
  > The number of seconds since the Unix epoch.
  > 
  > `_timestamp_date_utc`
  > 
  > The date in UTC, e.g. `2020-06-01`
  > 
  > `_timestamp_datetime_utc`
  > 
  > The ISO 8601 datetime in UTC, e.g. `2020-06-24T18:01:07Z`

  I'm going to rename them to:

  - `_now_epoch`
  - `_now_date_utc`
  - `_now_datetime_utc`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/870#issuecomment-650815278
issue_url: https://api.github.com/repos/simonw/datasette/issues/870
node_id: MDEyOklzc3VlQ29tbWVudDY1MDgxNTI3OA==
created_at: 2020-06-28T20:09:07Z
updated_at: 2020-06-28T20:11:21Z
author_association: OWNER
body:
  There's a lot of complex logic in the `DataView` class, which handles conditionally returning content as `.json` or as HTML or as `.csv`.

  That view subclasses `AsgiView` which is itself request-aware, so maybe I don't need to reconsider how those classes work - just figure out how to hook them up with `register_routes`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/870#issuecomment-650818086
issue_url: https://api.github.com/repos/simonw/datasette/issues/870
node_id: MDEyOklzc3VlQ29tbWVudDY1MDgxODA4Ng==
created_at: 2020-06-28T20:34:33Z
updated_at: 2020-06-28T20:34:33Z
author_association: OWNER
body:
  The key to all of this may be the `DatasetteRouter` class. It deals with `scope` right now but if it internally dealt with `request` that could be enough to fix #864 by adding logic needed by the `.add_message()` mechanism.

  https://github.com/simonw/datasette/blob/0991ea75cc7b265389aa8362414a305ba532d31a/datasette/app.py#L904-L938

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/870#issuecomment-650818309
issue_url: https://api.github.com/repos/simonw/datasette/issues/870
node_id: MDEyOklzc3VlQ29tbWVudDY1MDgxODMwOQ==
created_at: 2020-06-28T20:36:28Z
updated_at: 2020-06-28T20:36:52Z
author_association: OWNER
body:
  Since `AsgiRouter` is only used as the super-class of the `DatasetteRouter` class maybe I should get rid of `AsgiRouter` entirely - no point in having a Datasette-specific subclass of it if the parent class isn't ever used by anything else.

  I could also rename it to just `Router` which is a nicer name than `DatasetteRouter`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/847#issuecomment-650819895
issue_url: https://api.github.com/repos/simonw/datasette/issues/847
node_id: MDEyOklzc3VlQ29tbWVudDY1MDgxOTg5NQ==
created_at: 2020-06-28T20:50:21Z
updated_at: 2020-06-28T20:50:21Z
author_association: OWNER
body:
  I'm happy enough with https://codecov.io/gh/simonw/datasette that I'm not going to spend any more time on this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/870#issuecomment-650820068
issue_url: https://api.github.com/repos/simonw/datasette/issues/870
node_id: MDEyOklzc3VlQ29tbWVudDY1MDgyMDA2OA==
created_at: 2020-06-28T20:52:09Z
updated_at: 2020-06-28T20:53:00Z
author_association: OWNER
body:
  Maybe I could add a `as_request_view` method as an alternative to `as_asgi`:

  https://github.com/simonw/datasette/blob/a8bcafc1775c8a8655b365ae22a3d64f6361c74a/datasette/utils/asgi.py#L150-L174

  Or I could teach the `Router` to spot the `dispatch_request` method and call it directly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/870#issuecomment-650834666
issue_url: https://api.github.com/repos/simonw/datasette/issues/870
node_id: MDEyOklzc3VlQ29tbWVudDY1MDgzNDY2Ng==
created_at: 2020-06-28T23:07:19Z
updated_at: 2020-06-28T23:07:19Z
author_association: OWNER
body:
  So now the problem is simpler: I need to get `BaseView` to a state where it can accept a shared `request` object and it can be used in conjunction with `register_routes()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/870#issuecomment-650838691
issue_url: https://api.github.com/repos/simonw/datasette/issues/870
node_id: MDEyOklzc3VlQ29tbWVudDY1MDgzODY5MQ==
created_at: 2020-06-28T23:44:12Z
updated_at: 2020-06-28T23:44:25Z
author_association: OWNER
body:
  This code is interesting:

  https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L948-L955

  I want to change the signature of that `return await view(new_scope, receive, send)` method to instead take `(request, send)` - so I can have a single shared request object that's created just once per HTTP request.

  The problem is the scope modification: I have code that modifies the scope, but how should that impact a shared `Request` instance? Should its `.scope` be replaced with alternative scopes as it travels through the codebase?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/870#issuecomment-650838972
issue_url: https://api.github.com/repos/simonw/datasette/issues/870
node_id: MDEyOklzc3VlQ29tbWVudDY1MDgzODk3Mg==
created_at: 2020-06-28T23:46:40Z
updated_at: 2020-06-28T23:46:40Z
author_association: OWNER
body:
  I'm going to create the single `Request()` instance in the `DatasetteRouter` class - at the beginning of the `route_path` method: https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L905-L925

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/870#issuecomment-650842381
issue_url: https://api.github.com/repos/simonw/datasette/issues/870
node_id: MDEyOklzc3VlQ29tbWVudDY1MDg0MjM4MQ==
created_at: 2020-06-29T00:12:07Z
updated_at: 2020-06-29T00:12:07Z
author_association: OWNER
body:
  I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with `register_routes()`) as part of Datasette 0.45 but it would be OK if it slipped to a later release.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/864#issuecomment-650842514
issue_url: https://api.github.com/repos/simonw/datasette/issues/864
node_id: MDEyOklzc3VlQ29tbWVudDY1MDg0MjUxNA==
created_at: 2020-06-29T00:12:59Z
updated_at: 2020-06-29T00:12:59Z
author_association: OWNER
body:
  > I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with `register_routes()`) as part of Datasette 0.45 but it would be OK if it slipped to a later release.
  https://github.com/simonw/datasette/issues/870#issuecomment-650842381

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/864#issuecomment-650846473
issue_url: https://api.github.com/repos/simonw/datasette/issues/864
node_id: MDEyOklzc3VlQ29tbWVudDY1MDg0NjQ3Mw==
created_at: 2020-06-29T00:39:04Z
updated_at: 2020-06-29T00:39:04Z
author_association: OWNER
body:
  Re-opening: plugins may get to set messages but they don't display them, even if they render a template that extends `base.html`. For example, this code in a plugin:
  ```python
          return Response.html(
              await datasette.render_template(
                  "write.html",
                  {"databases": databases, "sql": request.args.get("sql") or ""},
                  request=request,
              )
          )
  ```
  This won't display messages. The reason is that the messages are made available to the template context in the `BaseView.render()` method here:
  https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/views/base.py#L87-L95

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/864#issuecomment-650846625
issue_url: https://api.github.com/repos/simonw/datasette/issues/864
node_id: MDEyOklzc3VlQ29tbWVudDY1MDg0NjYyNQ==
created_at: 2020-06-29T00:39:47Z
updated_at: 2020-06-29T00:39:47Z
author_association: OWNER
body:
  I think the fix is to move the `"show_messages"` variable to here:

  https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/app.py#L735-L748

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/864#issuecomment-650847013
issue_url: https://api.github.com/repos/simonw/datasette/issues/864
node_id: MDEyOklzc3VlQ29tbWVudDY1MDg0NzAxMw==
created_at: 2020-06-29T00:41:55Z
updated_at: 2020-06-29T00:41:55Z
author_association: OWNER
body:
  To test this I'll need a plugin test that renders a custom template. Here's an example I can imitate: https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/tests/test_plugins.py#L588-L596

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/805#issuecomment-650891257
issue_url: https://api.github.com/repos/simonw/datasette/issues/805
node_id: MDEyOklzc3VlQ29tbWVudDY1MDg5MTI1Nw==
created_at: 2020-06-29T03:56:48Z
updated_at: 2020-06-29T03:56:48Z
author_association: OWNER
body:
  Using `datasette-glitch` and the new https://github.com/simonw/datasette-write - currently running on `datasette==0.45a4` - works on Glitch. The console shows a login link which gives you a cookie which allows you access to the `/-/write` interface.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/840#issuecomment-650891502
issue_url: https://api.github.com/repos/simonw/datasette/issues/840
node_id: MDEyOklzc3VlQ29tbWVudDY1MDg5MTUwMg==
created_at: 2020-06-29T03:58:08Z
updated_at: 2020-06-29T03:58:08Z
author_association: OWNER
body:
  Step one: a "logout" page at `/-/logout` - which shows you a single CSRF-protected "logout" button if you do a GET against it and logs you out if you do a POST against it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/875#issuecomment-650898808
issue_url: https://api.github.com/repos/simonw/datasette/issues/875
node_id: MDEyOklzc3VlQ29tbWVudDY1MDg5ODgwOA==
created_at: 2020-06-29T04:32:31Z
updated_at: 2020-06-29T04:33:30Z
author_association: OWNER
body:
  I could borrow the implementation for this from `datasette-auth-github`
  https://github.com/simonw/datasette-auth-github/blob/182298b034ecb647971b65057d1d3e7b7fbbb482/datasette_auth_github/templates/base.html
  ```html+jinja
  {% extends "default:base.html" %}

  {% block extra_head %}
  <style type="text/css">
  .hd .logout {
      float: right;
      text-align: right;
      padding-left: 1em;
  }
  </style>
  {% endblock %}

  {% block nav %}
      {{ super() }}
      {% if auth and auth.username %}
          <p class="logout">
              <strong>{{ auth.username }}</strong> &middot; <a href="/-/logout">Log out</a>
          </p>
      {% endif %}
  {% endblock %}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/875#issuecomment-650899265
issue_url: https://api.github.com/repos/simonw/datasette/issues/875
node_id: MDEyOklzc3VlQ29tbWVudDY1MDg5OTI2NQ==
created_at: 2020-06-29T04:34:32Z
updated_at: 2020-06-29T04:34:32Z
author_association: OWNER
body:
  From https://github.com/simonw/datasette/issues/840#issuecomment-643454625
  > Another problem: what to display in the "you are logged in as", since we don't dictate an actor design.
  > 
  > I'm going to use a includes template for this that can easily be over-ridden by administrators or by plugins.
  > 
  > The default will look for the first available of the following keys:
  > 
  >     * display
  >     * name
  >     * username
  >     * login
  >     * id

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-650905399
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY1MDkwNTM5OQ==
created_at: 2020-06-29T05:01:03Z
updated_at: 2020-06-29T05:01:03Z
author_association: OWNER
body:
  This is a bit tricky to fix. This change to uvicorn is relevant: https://github.com/encode/uvicorn/commit/a75fe1381f6b1f78901691c71894f3cf487b5d30

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-650906318
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY1MDkwNjMxOA==
created_at: 2020-06-29T05:04:04Z
updated_at: 2020-06-29T05:04:12Z
author_association: OWNER
body:
  Within uvicorn it does this:
  ```python
              if port == 0:
                  port = server.sockets[0].getsockname()[1]
  ```
  That `server` variable is later stashed here:
  ```
  self.servers = [server]
  ```
  Where `self` is the instance of `class Server` - which is the class that Uvicorn instantiates and calls `.run()` on when we do `uvicorn.run()` here: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/cli.py#L409

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-650906533
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY1MDkwNjUzMw==
created_at: 2020-06-29T05:04:44Z
updated_at: 2020-06-29T05:04:44Z
author_association: OWNER
body:
  The challenge is... can we run our own custom code after that line has executed that has access to `server` and can hence access `server.servers[0].sockets[0].getsockname()[1]` to find the port?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-650907323
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY1MDkwNzMyMw==
created_at: 2020-06-29T05:07:16Z
updated_at: 2020-06-29T05:07:16Z
author_association: OWNER
body:
  This line is interesting: is this a hook I can attach to somehow?
  ```python
          await self.lifespan.startup()
  ```
  From https://github.com/encode/uvicorn/blob/a75fe1381f6b1f78901691c71894f3cf487b5d30/uvicorn/main.py#L475

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-650908534
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY1MDkwODUzNA==
created_at: 2020-06-29T05:11:06Z
updated_at: 2020-06-29T05:11:06Z
author_association: OWNER
body:
  Uvicorn's lifespan stuff isn't easy to figure out, but this test suite holds some clues: https://github.com/encode/uvicorn/blob/master/tests/test_lifespan.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-650908854
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY1MDkwODg1NA==
created_at: 2020-06-29T05:12:04Z
updated_at: 2020-06-29T05:12:04Z
author_association: OWNER
body:
  Can I detect the port the server is running on from within the regular Datasette ASGI code? If so I could use that ability and maybe output the magic `--root` link a second after the server starts up somehow.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-650909136
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY1MDkwOTEzNg==
created_at: 2020-06-29T05:12:58Z
updated_at: 2020-06-29T05:12:58Z
author_association: OWNER
body:
  On startup Datasette currently outputs:
  ```
  INFO:     Waiting for application startup.
  INFO:     ASGI 'lifespan' protocol appears unsupported.
  INFO:     Application startup complete.
  ```
  So the ASGI lifespan protocol is almost certainly the right way to solve this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-650909476
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY1MDkwOTQ3Ng==
created_at: 2020-06-29T05:14:08Z
updated_at: 2020-06-29T05:14:08Z
author_association: OWNER
body:
  I already have a `AsgiLifespan` class:
  https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L896-L905

  It runs this function: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L890-L894

  Could that startup function also output the `--root` login URL, if needed?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-650910137
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY1MDkxMDEzNw==
created_at: 2020-06-29T05:16:32Z
updated_at: 2020-06-29T05:16:32Z
author_association: OWNER
body:
  I'm not convinced that function is ever actually being called - I added a `print()` statement to it and it's not executing. I don't think the tests cover it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-651193131
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY1MTE5MzEzMQ==
created_at: 2020-06-29T15:27:00Z
updated_at: 2020-06-29T15:27:00Z
author_association: OWNER
body:
  Aha! Yes it's not being called, and the reason is this: https://github.com/encode/starlette/issues/486

  Short version: by default an exception raised during that phase is silently swallowed! You can avoid the swallowing by adding `lifespan="on"` to the call to `uvicorn.run()`.

  When I did that here:

  `uvicorn.run(ds.app(), host=host, port=port, log_level="info", lifespan="on")`

  The server failed to start with this error:

  ```
  INFO:     Started server process [68849]
  INFO:     Waiting for application startup.
  ERROR:    Exception in 'lifespan' protocol
  Traceback (most recent call last):
    File ".../uvicorn/lifespan/on.py", line 48, in main
      await app(scope, self.receive, self.send)
    File ".../uvicorn/middleware/proxy_headers.py", line 45, in __call__
      return await self.app(scope, receive, send)
    File ".../datasette_debug_asgi.py", line 9, in wrapped_app
      if scope["path"] == "/-/asgi-scope":
  KeyError: 'path'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-651203178
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY1MTIwMzE3OA==
created_at: 2020-06-29T15:44:38Z
updated_at: 2020-06-29T15:44:54Z
author_association: OWNER
body:
  I'm having real trouble figuring out how to gain access to the port that was used to start the server. I'm treating this as a very low priority - it only affects the exact `-p 0 --root` combination which isn't going to affect many people at all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/875#issuecomment-651293559
issue_url: https://api.github.com/repos/simonw/datasette/issues/875
node_id: MDEyOklzc3VlQ29tbWVudDY1MTI5MzU1OQ==
created_at: 2020-06-29T18:43:50Z
updated_at: 2020-06-29T18:43:50Z
author_association: OWNER
body:
  <img width="470" alt="_memory_" src="https://user-images.githubusercontent.com/9599/86043599-ca4b0500-b9fd-11ea-9378-a053e96cfde7.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/805#issuecomment-651301202
issue_url: https://api.github.com/repos/simonw/datasette/issues/805
node_id: MDEyOklzc3VlQ29tbWVudDY1MTMwMTIwMg==
created_at: 2020-06-29T19:00:37Z
updated_at: 2020-06-29T19:00:37Z
author_association: OWNER
body:
  How about a blog? Pre-configured canned queries that are only available to `"root"`, plus datasette-template-sql and default templates for the index page and blog entry pages.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/805#issuecomment-651302221
issue_url: https://api.github.com/repos/simonw/datasette/issues/805
node_id: MDEyOklzc3VlQ29tbWVudDY1MTMwMjIyMQ==
created_at: 2020-06-29T19:02:45Z
updated_at: 2020-06-29T19:05:26Z
author_association: OWNER
body:
  No I prefer the idea that logged out users can still perform some writes, in a not-likely-to-attract-abuse way.

  So a root-user-can-configure-polls, logged-out-users-can-vote-in-them demo would be good.

  Or... crazy idea: a collaborative drawing program? A grid of cells of emoji, anyone can add an emoji to a cell. Would involve a bit of JavaScript. I could use https://github.com/joeattardi/emoji-button for this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/877#issuecomment-651984355
issue_url: https://api.github.com/repos/simonw/datasette/issues/877
node_id: MDEyOklzc3VlQ29tbWVudDY1MTk4NDM1NQ==
created_at: 2020-06-30T19:02:15Z
updated_at: 2020-06-30T19:02:15Z
author_association: OWNER
body:
  https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#login-csrf

  > Login CSRF can be mitigated by creating pre-sessions (sessions before a user is authenticated) and including tokens in login form.

  Sounds like regular CSRF protection to me.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/832#issuecomment-651992737
issue_url: https://api.github.com/repos/simonw/datasette/issues/832
node_id: MDEyOklzc3VlQ29tbWVudDY1MTk5MjczNw==
created_at: 2020-06-30T19:19:33Z
updated_at: 2020-06-30T19:20:02Z
author_association: OWNER
body:
  I already have this method on Datasette:
  ```python
  async def permission_allowed(self, actor, action, resource=None, default=False):
  ```
  What would be a good method name that complements that and indicates "check a list of permissions in order"? Should it even run against the request or should you have to hand it `request.actor`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/832#issuecomment-651993537
issue_url: https://api.github.com/repos/simonw/datasette/issues/832
node_id: MDEyOklzc3VlQ29tbWVudDY1MTk5MzUzNw==
created_at: 2020-06-30T19:21:15Z
updated_at: 2020-06-30T19:21:15Z
author_association: OWNER
body:
  I could rename `permission_allowed()` to `check_permission()` and have a complementary `check_permissions()` method.

  This is a breaking change but we're pre-1.0 so I think that's OK. I could even set up a temporary `permission_allowed()` alias which prints a deprecation warning to the console, then remove that at 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/832#issuecomment-651993977
issue_url: https://api.github.com/repos/simonw/datasette/issues/832
node_id: MDEyOklzc3VlQ29tbWVudDY1MTk5Mzk3Nw==
created_at: 2020-06-30T19:22:06Z
updated_at: 2020-06-30T19:22:06Z
author_association: OWNER
body:
  `permission_allowed` is already the name of the pugin hook. It's actually a bit confusing that it's also the name of a method on `datasette.`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/832#issuecomment-651994978
issue_url: https://api.github.com/repos/simonw/datasette/issues/832
node_id: MDEyOklzc3VlQ29tbWVudDY1MTk5NDk3OA==
created_at: 2020-06-30T19:24:12Z
updated_at: 2020-06-30T19:24:12Z
author_association: OWNER
body:
  Hah... but check_permission` is a method on `BaseView`. Here are the various permission methods at the moment:

  https://github.com/simonw/datasette/blob/6c2634583627bfab750c115cb13850252821d637/datasette/default_permissions.py#L5-L14

  And on BaseView:

  https://github.com/simonw/datasette/blob/a8a5f813722f72703a7aae41135ccc40635cc02f/datasette/views/base.py#L65-L70

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/832#issuecomment-651995453
issue_url: https://api.github.com/repos/simonw/datasette/issues/832
node_id: MDEyOklzc3VlQ29tbWVudDY1MTk5NTQ1Mw==
created_at: 2020-06-30T19:25:13Z
updated_at: 2020-06-30T19:25:26Z
author_association: OWNER
body:
  I'm going to put the new `check_permissions()` method on `BaseView` as well. If I want that method to be available to plugins I can do so by turning that `BaseView` class into a documented API that plugins are encouraged to use themselves.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/832#issuecomment-651999516
issue_url: https://api.github.com/repos/simonw/datasette/issues/832
node_id: MDEyOklzc3VlQ29tbWVudDY1MTk5OTUxNg==
created_at: 2020-06-30T19:33:49Z
updated_at: 2020-06-30T21:34:59Z
author_association: OWNER
body:
  Tests needed for this:

  - If a user has view table but NOT view database / view instance, can they view the table page?
  - If a user has view canned query but NOT view database / view instance, can they view the canned query page?
  - If a user has view database but NOT view instance, can they view the database page?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/832#issuecomment-652103895
issue_url: https://api.github.com/repos/simonw/datasette/issues/832
node_id: MDEyOklzc3VlQ29tbWVudDY1MjEwMzg5NQ==
created_at: 2020-06-30T23:41:22Z
updated_at: 2020-06-30T23:41:22Z
author_association: OWNER
body:
  I don't think this needs any additional documentation - the new behaviour matches how the permissions are documented here: https://datasette.readthedocs.io/en/0.44/authentication.html#built-in-permissions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/876#issuecomment-652106227
issue_url: https://api.github.com/repos/simonw/datasette/issues/876
node_id: MDEyOklzc3VlQ29tbWVudDY1MjEwNjIyNw==
created_at: 2020-06-30T23:49:55Z
updated_at: 2020-06-30T23:50:04Z
author_association: OWNER
body:
  Done: https://latest.datasette.io/-/patterns

  <img width="1004" alt="Datasette__Pattern_Portfolio" src="https://user-images.githubusercontent.com/9599/86187862-b680cb00-baf1-11ea-8905-8c257dd67665.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-652160909
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY1MjE2MDkwOQ==
created_at: 2020-07-01T03:09:32Z
updated_at: 2020-07-01T03:10:21Z
author_association: CONTRIBUTOR
body:
  I've just realized Datasette tries to count hidden tables too. There are 5 visible tables, 25 hidden tables, which I haven't realize earlier to consider their effect. I've turned off counting for hidden tables to see if it has any effect.

  What's the point of counting FTS tables?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/880#issuecomment-652162722
issue_url: https://api.github.com/repos/simonw/datasette/issues/880
node_id: MDEyOklzc3VlQ29tbWVudDY1MjE2MjcyMg==
created_at: 2020-07-01T03:16:07Z
updated_at: 2020-07-01T03:16:07Z
author_association: OWNER
body:
  The response from this will never be a 302 - it will always be a 200 if the response worked or a 400 for bad parameters or a 500 for errors. The body returned will always be in JSON format.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/812#issuecomment-652163450
issue_url: https://api.github.com/repos/simonw/datasette/issues/812
node_id: MDEyOklzc3VlQ29tbWVudDY1MjE2MzQ1MA==
created_at: 2020-07-01T03:18:51Z
updated_at: 2020-07-01T03:20:28Z
author_association: OWNER
body:
  This can be a plugin hook:

  ```python
  @hookspec
  def forbidden(datasette, request, message, send):
      "Custom response for a 403 forbidden error"
  ```
  If the hook returns a `Response` object, it will be returned to the user. Plugins are likely to want to return a redirect response.

  Maybe the hook can instead use the `send` argument to respond to the request and return `True` which means "I've responded to this"?

  I'm going to leave `send` off for the moment - I can add that in the future if it turns out it would have been a good idea.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/812#issuecomment-652165709
issue_url: https://api.github.com/repos/simonw/datasette/issues/812
node_id: MDEyOklzc3VlQ29tbWVudDY1MjE2NTcwOQ==
created_at: 2020-07-01T03:26:35Z
updated_at: 2020-07-01T03:26:35Z
author_association: OWNER
body:
  This case may not be covered without extra work:
  https://github.com/simonw/datasette/blob/3ec5b1abf6afa2d22a3378092809a1a8c0249d26/datasette/views/database.py#L122-L123

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/877#issuecomment-652166115
issue_url: https://api.github.com/repos/simonw/datasette/issues/877
node_id: MDEyOklzc3VlQ29tbWVudDY1MjE2NjExNQ==
created_at: 2020-07-01T03:28:07Z
updated_at: 2020-07-01T03:28:07Z
author_association: CONTRIBUTOR
body:
  Does this mean custom routes get to expose endpoints accepting POST requests? I've tried earlier to add some POST endpoints, but requests were being rejected by Datasette due to CSRF

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/877#issuecomment-652182990
issue_url: https://api.github.com/repos/simonw/datasette/issues/877
node_id: MDEyOklzc3VlQ29tbWVudDY1MjE4Mjk5MA==
created_at: 2020-07-01T04:29:38Z
updated_at: 2020-07-01T04:42:59Z
author_association: OWNER
body:
  Have you tried the method described here? https://datasette.readthedocs.io/en/latest/internals.html#csrf-protection - I'm happy to bulk out that section of the documentation if that doesn't help solve your problem.

  I just closed #835 which should make CSRF protection easier to work with - it won't interfere with requests without cookies or requests with `Authentication: Bearer token` tokens. See also https://github.com/simonw/asgi-csrf/issues/11

  You can try out `pip install datasette==0.45a5` to get those features. Hopefully releasing a full 0.45 tomorrow.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/877#issuecomment-652255960
issue_url: https://api.github.com/repos/simonw/datasette/issues/877
node_id: MDEyOklzc3VlQ29tbWVudDY1MjI1NTk2MA==
created_at: 2020-07-01T07:52:25Z
updated_at: 2020-07-01T08:10:00Z
author_association: CONTRIBUTOR
body:
  I am calling the API from another origin, so injecting CSRF token into templates wouldn't work.

  EDIT:

  I'll try the new version, it sounds promising

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/877#issuecomment-652261382
issue_url: https://api.github.com/repos/simonw/datasette/issues/877
node_id: MDEyOklzc3VlQ29tbWVudDY1MjI2MTM4Mg==
created_at: 2020-07-01T08:03:17Z
updated_at: 2020-07-01T08:03:23Z
author_association: CONTRIBUTOR
body:
  Bearer tokens sound interesting. Where do tokens come from? An auth provider of my choosing? How do they get verified?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/883#issuecomment-652297139
issue_url: https://api.github.com/repos/simonw/datasette/issues/883
node_id: MDEyOklzc3VlQ29tbWVudDY1MjI5NzEzOQ==
created_at: 2020-07-01T09:11:29Z
updated_at: 2020-07-01T09:11:29Z
author_association: CONTRIBUTOR
body:
  Turns out we should include hidden tables in the result dict, or we're breaking tests. I've committed a refactor https://github.com/simonw/datasette/pull/883/commits/4f06e1bf6fbe4b73be770b87f610bf7c0e6e3ea7

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/883#issuecomment-652311990
issue_url: https://api.github.com/repos/simonw/datasette/issues/883
node_id: MDEyOklzc3VlQ29tbWVudDY1MjMxMTk5MA==
created_at: 2020-07-01T09:40:40Z
updated_at: 2020-07-01T09:40:40Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=h1) Report
  > Merging [#883](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/676bb64c877d73f8ff496cef4632f5a8a5a9283c&el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/883/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##           master     #883   +/-   ##
  =======================================
    Coverage   83.42%   83.42%           
  =======================================
    Files          27       27           
    Lines        3632     3632           
  =======================================
    Hits         3030     3030           
    Misses        602      602           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=footer). Last update [676bb64...251884f](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/877#issuecomment-652520496
issue_url: https://api.github.com/repos/simonw/datasette/issues/877
node_id: MDEyOklzc3VlQ29tbWVudDY1MjUyMDQ5Ng==
created_at: 2020-07-01T16:26:52Z
updated_at: 2020-07-01T16:26:52Z
author_association: OWNER
body:
  Tokens get verified by plugins. So far there's only one: https://github.com/simonw/datasette-auth-tokens - which has you hard-coding plugins in a configuration file. I have a issue there to add support for database-backed tokens too: https://github.com/simonw/datasette-auth-tokens/issues/1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/877#issuecomment-652597975
issue_url: https://api.github.com/repos/simonw/datasette/issues/877
node_id: MDEyOklzc3VlQ29tbWVudDY1MjU5Nzk3NQ==
created_at: 2020-07-01T19:12:15Z
updated_at: 2020-07-01T19:12:15Z
author_association: OWNER
body:
  The latest release of https://github.com/simonw/datasette-auth-tokens (0.2) now supports SQL configuration of tokens.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/880#issuecomment-652646487
issue_url: https://api.github.com/repos/simonw/datasette/issues/880
node_id: MDEyOklzc3VlQ29tbWVudDY1MjY0NjQ4Nw==
created_at: 2020-07-01T21:05:48Z
updated_at: 2020-07-01T21:05:48Z
author_association: OWNER
body:
  I've been testing the WIP using this in the console:
  ```javascript
  fetch('/data/add_name.json', {
    method: 'POST',
    body: 'name=XXXfetch',
    credentials: 'omit',
    headers: {'Content-Type': 'application/x-www-form-urlencoded'}
  })
  .then(response => console.log(response))
  ```
  Against a canned query configured like this:
  ```yaml
  databases:
    data:
      queries:
        add_name:
          sql: insert into names (name) values (:name)
          write: true
  ```
  I haven't got it to work yet. Latest error is this one:
  ```
  INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
  Traceback (most recent call last):
    File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 975, in route_path
      await response.asgi_send(send)
  AttributeError: 'tuple' object has no attribute 'asgi_send'
  INFO:     127.0.0.1:49938 - "POST /data/add_name.json HTTP/1.1" 500 Internal Server Error
  ```
  It looks like I'm going to have to rethink how the `BaseView` code around tables, formats and hashes is structured in order to fix this. That's a big refactoring! I'm moving this to a new milestone for Datasette 0.46.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/887#issuecomment-652709199
issue_url: https://api.github.com/repos/simonw/datasette/issues/887
node_id: MDEyOklzc3VlQ29tbWVudDY1MjcwOTE5OQ==
created_at: 2020-07-02T00:21:54Z
updated_at: 2020-07-02T00:21:54Z
author_association: OWNER
body:
  Example in the live demo: https://latest.datasette.io/fixtures/magic_parameters

  <img width="858" alt="fixtures__select___header_user_agent_as_user_agent____now_datetime_utc_as_datetime" src="https://user-images.githubusercontent.com/9599/86303008-58bab480-bbbf-11ea-9c99-aec7ed1f4f56.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/887#issuecomment-652711562
issue_url: https://api.github.com/repos/simonw/datasette/issues/887
node_id: MDEyOklzc3VlQ29tbWVudDY1MjcxMTU2Mg==
created_at: 2020-07-02T00:30:43Z
updated_at: 2020-07-02T00:30:43Z
author_association: OWNER
body:
  Demo has updated: https://latest.datasette.io/fixtures/magic_parameters

  <img width="900" alt="fixtures__select___header_user_agent_as_user_agent____now_datetime_utc_as_datetime" src="https://user-images.githubusercontent.com/9599/86303410-95d37680-bbc0-11ea-892c-0ef4744c261a.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/887#issuecomment-652711822
issue_url: https://api.github.com/repos/simonw/datasette/issues/887
node_id: MDEyOklzc3VlQ29tbWVudDY1MjcxMTgyMg==
created_at: 2020-07-02T00:31:33Z
updated_at: 2020-07-02T00:31:33Z
author_association: OWNER
body:
  If a canned query has a title defined that will be used instead: https://latest.datasette.io/fixtures/neighborhood_search

  <img width="821" alt="fixtures__select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________order_by_neighborhood_" src="https://user-images.githubusercontent.com/9599/86303455-b26fae80-bbc0-11ea-9054-0cc6c086f143.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/886#issuecomment-652731459
issue_url: https://api.github.com/repos/simonw/datasette/issues/886
node_id: MDEyOklzc3VlQ29tbWVudDY1MjczMTQ1OQ==
created_at: 2020-07-02T01:48:08Z
updated_at: 2020-07-02T01:48:08Z
author_association: OWNER
body:
  A common error with this (and other) magic parameters is for the database query to result in the following:

      You did not supply a value for binding 3.

  This is a pretty crufty error. I'm inclined to say that ANY missing or invalid magic parameter should be treated as a `None` value instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/886#issuecomment-652732460
issue_url: https://api.github.com/repos/simonw/datasette/issues/886
node_id: MDEyOklzc3VlQ29tbWVudDY1MjczMjQ2MA==
created_at: 2020-07-02T01:52:02Z
updated_at: 2020-07-02T01:52:02Z
author_association: OWNER
body:
  In investigating this I'm not convinced 500 errors are being correctly raised by errors in canned writable queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/889#issuecomment-653002499
issue_url: https://api.github.com/repos/simonw/datasette/issues/889
node_id: MDEyOklzc3VlQ29tbWVudDY1MzAwMjQ5OQ==
created_at: 2020-07-02T13:22:13Z
updated_at: 2020-07-02T13:22:13Z
author_association: CONTRIBUTOR
body:
  I was able to narrow this down to the fact that lifespan protocol is turned on. 

  I see the workaround you've used here: https://github.com/simonw/datasette-debug-asgi/commit/72d568d32a3159c763ce908c0b269736935c6987

  If so, maybe it's time to update some of the asg_wrapper [plugins](https://datasette.readthedocs.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette). 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/890#issuecomment-653309545
issue_url: https://api.github.com/repos/simonw/datasette/issues/890
node_id: MDEyOklzc3VlQ29tbWVudDY1MzMwOTU0NQ==
created_at: 2020-07-03T02:52:25Z
updated_at: 2020-07-03T03:03:00Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=h1) Report
  > Merging [#890](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/57879dc8b346a435804a9e45ffaacbf2a0228bc6&el=desc) will **decrease** coverage by `0.01%`.
  > The diff coverage is `80.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/890/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##           master     #890      +/-   ##
  ==========================================
  - Coverage   83.42%   83.40%   -0.02%     
  ==========================================
    Files          27       27              
    Lines        3632     3634       +2     
  ==========================================
  + Hits         3030     3031       +1     
  - Misses        602      603       +1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/890/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.99% <80.00%> (-0.17%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=footer). Last update [57879dc...745af3b](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/890#issuecomment-653314465
issue_url: https://api.github.com/repos/simonw/datasette/issues/890
node_id: MDEyOklzc3VlQ29tbWVudDY1MzMxNDQ2NQ==
created_at: 2020-07-03T03:07:41Z
updated_at: 2020-07-03T03:07:41Z
author_association: OWNER
body:
  This is an excellent fix. Thanks!

  Not sure why codecov is complaining. I'm going to merge it as-is.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962418
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41
node_id: MDEyOklzc3VlQ29tbWVudDY1Mzk2MjQxOA==
created_at: 2020-07-06T00:41:38Z
updated_at: 2020-07-06T00:41:38Z
author_association: MEMBER
body:
  https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/logs?project=datasette-222320 has some clues.

  <img width="818" alt="github-to-sqlite_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/86545887-92086280-bee6-11ea-8787-b3b85a7bb4b9.png">
  <img width="1451" alt="github-to-sqlite_–_Service_details_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/86545917-c11ed400-bee6-11ea-9cf8-957911ca9f6a.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962530
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41
node_id: MDEyOklzc3VlQ29tbWVudDY1Mzk2MjUzMA==
created_at: 2020-07-06T00:42:13Z
updated_at: 2020-07-06T00:42:13Z
author_association: MEMBER
body:
  So it looks like it's the ASGI lifespan change I made in https://github.com/simonw/datasette/commit/16f592247a2a0e140ada487e9972645406dcae69 - It must be incompatible with one of the plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962669
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41
node_id: MDEyOklzc3VlQ29tbWVudDY1Mzk2MjY2OQ==
created_at: 2020-07-06T00:42:57Z
updated_at: 2020-07-06T00:42:57Z
author_association: MEMBER
body:
  https://github-to-sqlite.dogsheep.net/-/plugins

  ```json
  [
      {
          "name": "datasette-json-html",
          "static": false,
          "templates": false,
          "version": "0.6",
          "hooks": [
              "prepare_connection",
              "render_cell"
          ]
      },
      {
          "name": "datasette-render-markdown",
          "static": false,
          "templates": false,
          "version": "1.1.2",
          "hooks": [
              "extra_template_vars",
              "render_cell"
          ]
      },
      {
          "name": "datasette-pretty-json",
          "static": false,
          "templates": false,
          "version": "0.2",
          "hooks": [
              "render_cell"
          ]
      },
      {
          "name": "datasette-search-all",
          "static": false,
          "templates": true,
          "version": "0.2.1",
          "hooks": [
              "asgi_wrapper",
              "extra_template_vars"
          ]
      },
      {
          "name": "datasette-vega",
          "static": true,
          "templates": false,
          "version": "0.6.2",
          "hooks": [
              "extra_css_urls",
              "extra_js_urls"
          ]
      }
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653966670
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41
node_id: MDEyOklzc3VlQ29tbWVudDY1Mzk2NjY3MA==
created_at: 2020-07-06T01:07:02Z
updated_at: 2020-07-06T01:07:02Z
author_association: MEMBER
body:
  OK that fix worked.https://github.com/dogsheep/github-to-sqlite/runs/839764768?check_suite_focus=true

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/118
node_id: MDEyOklzc3VlQ29tbWVudDY1NTIzOTcyOA==
created_at: 2020-07-08T02:16:42Z
updated_at: 2020-07-08T02:16:42Z
author_association: CONTRIBUTOR
body:
  I fixed my original oops by moving the `DELETE FROM $table` out of the chunking loop and repushed. I think this change can be considered in isolation from issues around transactions, which I discuss next.

  I wanted to make the DELETE + INSERT happen all in the same transaction so it was robust, but that was more complicated than I expected. The transaction handling in the Database/Table classes isn't systematic, and this poses big hurdles to making `Table.insert_all` (or other operations) consistent and robust in the face of errors.

  For example, I wanted to do this (whitespace ignored in diff, so indentation change not highlighted):

  ```diff
  diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
  index d6b9ecf..4107ceb 100644
  --- a/sqlite_utils/db.py
  +++ b/sqlite_utils/db.py
  @@ -1028,6 +1028,11 @@ class Table(Queryable):
           batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))
           self.last_rowid = None
           self.last_pk = None
  +        with self.db.conn:
  +            # Explicit BEGIN is necessary because Python's sqlite3 doesn't
  +            # issue implicit BEGINs for DDL, only DML.  We mix DDL and DML
  +            # below and might execute DDL first, e.g. for table creation.
  +            self.db.conn.execute("BEGIN")
               if truncate and self.exists():
                   self.db.conn.execute("DELETE FROM [{}];".format(self.name))
               for chunk in chunks(itertools.chain([first_record], records), batch_size):
  @@ -1038,7 +1043,11 @@ class Table(Queryable):
                           # Use the first batch to derive the table names
                           column_types = suggest_column_types(chunk)
                           column_types.update(columns or {})
  -                    self.create(
  +                        # Not self.create() because that is wrapped in its own
  +                        # transaction and Python's sqlite3 doesn't support
  +                        # nested transactions.
  +                        self.db.create_table(
  +                            self.name,
                               column_types,
                               pk,
                               foreign_keys,
  @@ -1139,7 +1148,6 @@ class Table(Queryable):
                       flat_values = list(itertools.chain(*values))
                       queries_and_params = [(sql, flat_values)]
   
  -            with self.db.conn:
                   for query, params in queries_and_params:
                       try:
                           result = self.db.conn.execute(query, params)
  ```

  but that fails in tests because other methods call `insert/upsert/insert_all/upsert_all` in the middle of their transactions, so the BEGIN statement throws an error (no nested transactions allowed).

  Stepping back, it would be nice to make the transaction handling systematic and predictable. One way to do this is to make the `sqlite_utils/db.py` code generally not begin or commit any transactions, and require the caller to do that instead. This lets the caller mix and match the Python API calls into transactions as appropriate (which is impossible for the API methods themselves to fully determine). Then, make `sqlite_utils/cli.py` begin and commit a transaction in each `@cli.command` function, making each command robust and consistent in the face of errors. The big change here, and why I didn't just submit a patch, is that it dramatically changes the Python API to _require_ callers to begin a transaction rather than just immediately calling methods.

  There is also the caveat that for each transaction, an explicit `BEGIN` is also necessary so that DDL as well as DML (as well as `SELECT`s) are consistent and rolled back on error. There are several bugs.python.org discussions around this particular problem of DDL and some plans to make it better and consistent with DBAPI2, eventually. In the meantime, the sqlite-utils Database class could be a context manager which supports the incantations necessary to do proper transactions. This would still be a Python API change for callers but wouldn't expose them to the weirdness of the sqlite3's default transaction handling.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/118
node_id: MDEyOklzc3VlQ29tbWVudDY1NTI4MzM5Mw==
created_at: 2020-07-08T04:55:18Z
updated_at: 2020-07-08T04:55:18Z
author_association: OWNER
body:
  This is a really good idea - and thank you for the detailed discussion in the pull request.

  I'm keen to discuss how transactions can work better. I tend to use this pattern in my own code:

      with db.conn:
          db["table"].insert(...)

  But it's not documented and I've not though very hard about it!

  I like having inserts that handle 10,000+ rows commit on every chunk so I can watch their progress from another process, but the library should absolutely support people who want to commit all of the rows in a single transaction - or combine changes with DML.

  Lots to discuss here. I'll start a new issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655286864
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/118
node_id: MDEyOklzc3VlQ29tbWVudDY1NTI4Njg2NA==
created_at: 2020-07-08T05:05:27Z
updated_at: 2020-07-08T05:05:36Z
author_association: OWNER
body:
  The only thing missing from this PR is updates to the documentation. Those need to go in two places:

  - In the Python API docs. I suggest adding a note to this section about bulk inserts: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/python-api.rst#bulk-inserts
  - In the CLI docs, in this section: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/cli.rst#inserting-json-data

  Here's an example of a previous commit that includes updates to both CLI and API documentation: https://github.com/simonw/sqlite-utils/commit/f9473ace14878212c1fa968b7bd2f51e4f064dba#diff-e3e2a9bfd88566b05001b02a3f51d286

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655290625
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY1NTI5MDYyNQ==
created_at: 2020-07-08T05:15:45Z
updated_at: 2020-07-08T05:15:45Z
author_association: OWNER
body:
  Ideally this would all happen in a single transaction, such that other processes talking to the database would not see any inconsistent state while the table copy was taking place. Need to confirm that this is possible. Also refs transactions thoughts in #121.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655643078
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/118
node_id: MDEyOklzc3VlQ29tbWVudDY1NTY0MzA3OA==
created_at: 2020-07-08T17:05:59Z
updated_at: 2020-07-08T17:05:59Z
author_association: CONTRIBUTOR
body:
  > The only thing missing from this PR is updates to the documentation.

  Ah, yes, thanks for this reminder! I've repushed with doc bits added.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655652679
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/121
node_id: MDEyOklzc3VlQ29tbWVudDY1NTY1MjY3OQ==
created_at: 2020-07-08T17:24:46Z
updated_at: 2020-07-08T17:24:46Z
author_association: CONTRIBUTOR
body:
  Better transaction handling would be really great. Some of my thoughts on implementing better transaction discipline are in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728.

  My preferences:

  - Each CLI command should operate in a single transaction so that either the whole thing succeeds or the whole thing is rolled back. This avoids partially completed operations when an error occurs part way through processing. Partially completed operations are typically much harder to recovery from gracefully and may cause inconsistent data states.

  - The Python API should be transaction-agnostic and rely on the caller to coordinate transactions. Only the caller knows how individual insert, create, update, etc operations/methods should be bundled conceptually into transactions. When the caller is the CLI, for example, that bundling would be at the CLI command-level. Other callers might want to break up operations into multiple transactions. Transactions are usually most useful when controlled at the application-level (like logging configuration) instead of the library level. The library needs to provide an API that's conducive to transaction use, though.

  - The Python API should provide a context manager to provide consistent transactions handling with more useful defaults than Python's `sqlite3` module. The latter issues implicit `BEGIN` statements by default for most DML (`INSERT`, `UPDATE`, `DELETE`, … but not `SELECT`, I believe), but **not** DDL (`CREATE TABLE`, `DROP TABLE`, `CREATE VIEW`, …). Notably, the `sqlite3` module doesn't issue the implicit `BEGIN` until the first DML statement. It _does not_ issue it when entering the `with conn` block, like other DBAPI2-compatible modules do. The `with conn` block for `sqlite3` only arranges to commit or rollback an existing transaction when exiting. Including DDL and `SELECT`s in transactions is important for operation consistency, though. There are several existing bugs.python.org tickets about this and future changes are in the works, but sqlite-utils can provide its own API sooner. sqlite-utils's `Database` class could itself be a context manager (built on the `sqlite3` connection context manager) which additionally issues an explicit `BEGIN` when entering. This would then let Python API callers do something like:

  ```python
  db = sqlite_utils.Database(path)

  with db: # ← BEGIN issued here by Database.__enter__
      db.insert(…)
      db.create_view(…)
  # ← COMMIT/ROLLBACK issue here by sqlite3.connection.__exit__
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655673896
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/121
node_id: MDEyOklzc3VlQ29tbWVudDY1NTY3Mzg5Ng==
created_at: 2020-07-08T18:08:11Z
updated_at: 2020-07-08T18:08:11Z
author_association: OWNER
body:
  I'm with you on most of this. Completely agreed that the CLI should do everything in a transaction.

  The one thing I'm not keen on is forcing calling code to explicitly start a transaction, for a couple of reasons:

  1. It will break all of the existing code out there
  2. It doesn't match to how I most commonly use this library - as an interactive tool in a Jupyter notebook, where I'm generally working against a brand new scratch database and any errors don't actually matter

  So... how about this: IF you wrap your code in a `with db:` block then the `.insert()` and suchlike methods expect you to manage transactions yourself. But if you don't use the context manager they behave like they do at the moment (or maybe a bit more sensibly).

  That way existing code works as it does today, lazy people like me can call `.insert()` without thinking about transactions, but people writing actual production code (as opposed to Jupyter hacks) have a sensible way to take control of the transactions themselves.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/119#issuecomment-655674910
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/119
node_id: MDEyOklzc3VlQ29tbWVudDY1NTY3NDkxMA==
created_at: 2020-07-08T18:10:18Z
updated_at: 2020-07-08T18:10:18Z
author_association: OWNER
body:
  This will work similar to how `.add_foreign_keys()` works: turn on `writable_schema` and rewrite the `sql` for that table in the `sqlite_master` table.

  Here's that code today - it could be adapted to include removal of foreign keys that we no longer want:

  https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L391-L401

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677099
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY1NTY3NzA5OQ==
created_at: 2020-07-08T18:15:02Z
updated_at: 2020-07-08T18:15:02Z
author_association: OWNER
body:
  I'm not so keen on that chained API - it's pretty complicated.

  Here's an idea for a much simpler interface. Essentially it lets you say "take table X and migrate its contents to a new table with this structure - then atomically rename the tables to switch them":
  ```python
  db["mytable"].migrate_table({"id": int, "name": str"}, pk="id")
  ```
  The `migrate_table()` method would take the same exact signature as the `table.create()` method: https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L615-L625

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677396
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY1NTY3NzM5Ng==
created_at: 2020-07-08T18:15:39Z
updated_at: 2020-07-08T18:15:39Z
author_association: OWNER
body:
  Alternative possible names:
  - `.transform_table()`
  - `.migrate()`
  - `.transform()`

  I'm torn between `.migrate_table()` and `.transform_table()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677909
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY1NTY3NzkwOQ==
created_at: 2020-07-08T18:16:39Z
updated_at: 2020-07-08T18:16:39Z
author_association: OWNER
body:
  Since neither the term "transform" or "migrate" are used in the codebase at the moment, I think I'll go with `.transform_table()` - that leaves the term "migrate" available for any future database migrations system (similar to Django's).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655778058
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY1NTc3ODA1OA==
created_at: 2020-07-08T21:54:30Z
updated_at: 2020-07-08T21:54:30Z
author_association: OWNER
body:
  Don't forget this step:
  >  If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655782477
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY1NTc4MjQ3Nw==
created_at: 2020-07-08T22:06:23Z
updated_at: 2020-07-08T22:06:23Z
author_association: OWNER
body:
  Thinking about the method signature:
  ```python
      def transform_table(
          self,
          columns,
          pk=None,
          foreign_keys=None,
          column_order=None,
          not_null=None,
          defaults=None,
          hash_id=None,
          extracts=None,
      ):
  ```
  This requires the caller to provide the exact set of columns for the new table.

  It would be useful if this was optional - if you could omit the columns and have it automatically use the previous columns. This would let you change things like the primary key or the column order using the other arguments.

  Even better: allow column renaming using an optional `rename={...}` argument:

  ```python
  db["dogs"].transform_table(rename={"name": "dog_name"})
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655785396
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY1NTc4NTM5Ng==
created_at: 2020-07-08T22:14:10Z
updated_at: 2020-07-08T22:14:10Z
author_association: OWNER
body:
  Work in progress: not quite right yet, I need smarter logic for how renamed columns are reflected in the generated `INSERT INTO ... SELECT ...` query:
  ```python
      def transform_table(
          self,
          columns=None,
          rename=None,
          change_type=None,
          pk=None,
          foreign_keys=None,
          column_order=None,
          not_null=None,
          defaults=None,
          hash_id=None,
          extracts=None,
      ):
          assert self.exists(), "Cannot transform a table that doesn't exist yet"
          columns = columns or self.columns_dict
          if rename is not None or change_type is not None:
              columns = {rename.get(key, key): change_type.get(key, value) for key, value in columns.items()}
          new_table_name = "{}_new_{}".format(self.name, os.urandom(6).hex())
          previous_columns = set(self.columns_dict.keys())
          with self.db.conn:
              columns = {name: value for (name, value) in columns.items()}
              new_table = self.db.create_table(
                  new_table_name,
                  columns,
                  pk=pk,
                  foreign_keys=foreign_keys,
                  column_order=column_order,
                  not_null=not_null,
                  defaults=defaults,
                  hash_id=hash_id,
                  extracts=extracts,
              )
              # Copy across data - but only for columns that exist in both
              new_columns = set(columns.keys())
              columns_to_copy = new_columns.intersection(previous_columns)
              copy_sql = "INSERT INTO [{new_table}] ({new_cols}) SELECT {old_cols} FROM [{old_table}]".format(
                  new_table=new_table_name,
                  old_table=self.name,
                  old_cols=", ".join("[{}]".format(col) for col in columns_to_copy),
                  new_cols=", ".join("[{}]".format(rename.get(col, col)) for col in columns_to_copy),
              )
              self.db.conn.execute(copy_sql)
              # Drop the old table
              self.db.conn.execute("DROP TABLE [{}]".format(self.name))
              # Rename the new one
              self.db.conn.execute(
                  "ALTER TABLE [{}] RENAME TO [{}]".format(new_table_name, self.name)
              )
          return self
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655786374
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY1NTc4NjM3NA==
created_at: 2020-07-08T22:16:54Z
updated_at: 2020-07-08T22:16:54Z
author_association: OWNER
body:
  According to https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes the hardest bits to consider are how to deal with existing foreign key relationships, triggers and views.

  I'm OK leaving views as an exercise for the caller - many of these transformations may not need any view changes at all.

  Foreign key relationships are important: it should handle these automatically as effectively as possible.

  Likewise trigger changes: need to think about what this means.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655898722
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/121
node_id: MDEyOklzc3VlQ29tbWVudDY1NTg5ODcyMg==
created_at: 2020-07-09T04:53:08Z
updated_at: 2020-07-09T04:53:08Z
author_association: CONTRIBUTOR
body:
  Yep, I agree that makes more sense for backwards compat and more casual use cases. I think it should be possible for the Database/Queryable methods to DTRT based on seeing if it's within a context-manager-managed transaction.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-656363548
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY1NjM2MzU0OA==
created_at: 2020-07-09T21:37:28Z
updated_at: 2020-07-09T21:37:28Z
author_association: OWNER
body:
  I'm going to add a second method `.transform_table_sql(...)` - which returns the SQL that would have been executed but does NOT execute it.

  Advanced callers can use this to include their own additional steps in the same transaction - e.g. recreating views or triggers.

  More importantly it gives me a useful hook for writing some unit tests against the generated SQL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/892#issuecomment-657268051
issue_url: https://api.github.com/repos/simonw/datasette/issues/892
node_id: MDEyOklzc3VlQ29tbWVudDY1NzI2ODA1MQ==
created_at: 2020-07-12T19:58:24Z
updated_at: 2020-07-12T19:58:24Z
author_association: OWNER
body:
  ```css
  .wy-side-nav-search > div.version {
      margin-top: -.4045em;
      margin-bottom: .809em;
      font-weight: normal;
      color: rgba(255,255,255,0.3);
  }
  ```
  Fix can go here: https://github.com/simonw/datasette/blob/ee0ef016523a765b6ef6eaa43cad9ad568f78ae4/docs/_static/css/custom.css#L1-L3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/892#issuecomment-657268433
issue_url: https://api.github.com/repos/simonw/datasette/issues/892
node_id: MDEyOklzc3VlQ29tbWVudDY1NzI2ODQzMw==
created_at: 2020-07-12T20:02:17Z
updated_at: 2020-07-12T20:02:35Z
author_association: OWNER
body:
  Fixed https://datasette.readthedocs.io/en/latest/

  <img width="805" alt="Datasette_—_Datasette_documentation_and_How_to_set_a_custom_theme__·_Issue__638_·_readthedocs_readthedocs_org" src="https://user-images.githubusercontent.com/9599/87255480-f32cba80-c43f-11ea-8928-01528859929f.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/895#issuecomment-659085528
issue_url: https://api.github.com/repos/simonw/datasette/issues/895
node_id: MDEyOklzc3VlQ29tbWVudDY1OTA4NTUyOA==
created_at: 2020-07-16T00:32:47Z
updated_at: 2020-07-16T00:32:47Z
author_association: OWNER
body:
  This was added in https://github.com/simonw/datasette/commit/504196341c49840270bd75ea1a1871ef386ba7ea - here's the relevant code (which only applies on the table page, not the query page):

  https://github.com/simonw/datasette/blob/d6e03b04302a0852e7133dc030eab50177c37be7/datasette/views/table.py#L196-L204

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/896#issuecomment-659615034
issue_url: https://api.github.com/repos/simonw/datasette/issues/896
node_id: MDEyOklzc3VlQ29tbWVudDY1OTYxNTAzNA==
created_at: 2020-07-16T19:14:07Z
updated_at: 2020-07-16T19:14:07Z
author_association: OWNER
body:
  Demo: https://srccon-2020.datasette.io/srccon?sql=select+id%2C+day%2C+time%2C+event_name%2C+event_description%2C+facilitators+from+sessions+order+by+event_dtstart+limit+101

  I really like this:

  <img width="1396" alt="srccon__select_id__day__time__event_name__event_description__facilitators_from_sessions_order_by_event_dtstart_limit_101" src="https://user-images.githubusercontent.com/9599/87712630-d228da00-c75d-11ea-8f44-69886e599a07.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/896#issuecomment-659734703
issue_url: https://api.github.com/repos/simonw/datasette/issues/896
node_id: MDEyOklzc3VlQ29tbWVudDY1OTczNDcwMw==
created_at: 2020-07-16T23:34:57Z
updated_at: 2020-07-16T23:34:57Z
author_association: OWNER
body:
  I'm worried about how this will interact with some of the plugins:

  * https://github.com/simonw/datasette-json-html
  * https://github.com/simonw/datasette-pretty-json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/896#issuecomment-659773897
issue_url: https://api.github.com/repos/simonw/datasette/issues/896
node_id: MDEyOklzc3VlQ29tbWVudDY1OTc3Mzg5Nw==
created_at: 2020-07-17T01:26:08Z
updated_at: 2020-07-17T01:26:08Z
author_association: OWNER
body:
  I manually tested it with those plugins and it seems to interoperate just fine - since both of those use `<pre>` tags for the cases that I care about so they're already expecting white-space to be pre wrapped in some way.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/898#issuecomment-660419499
issue_url: https://api.github.com/repos/simonw/datasette/issues/898
node_id: MDEyOklzc3VlQ29tbWVudDY2MDQxOTQ5OQ==
created_at: 2020-07-18T03:55:13Z
updated_at: 2020-07-18T03:55:13Z
author_association: OWNER
body:
  Maybe I should make `httpx` a testing dependency of Datasette itself. It's usage is already encouraged in plugins by https://datasette.readthedocs.io/en/stable/testing_plugins.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/898#issuecomment-660419792
issue_url: https://api.github.com/repos/simonw/datasette/issues/898
node_id: MDEyOklzc3VlQ29tbWVudDY2MDQxOTc5Mg==
created_at: 2020-07-18T03:57:46Z
updated_at: 2020-07-18T03:57:46Z
author_association: OWNER
body:
  This requires some thought. There are various testing utilities that don't exist yet that plugins might benefit from - off the top of my head:

  - `assert_permissions_checked`
  - `assert_template_rendered`

  I should resist the temptation to provide a reusable version of `make_app_client` that provides a fully configured Datasette instance because I need to be able to change the design of the Datasette `fixtures.db` test database without accidentally breaking any plugins that depend on it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660536265
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43
node_id: MDEyOklzc3VlQ29tbWVudDY2MDUzNjI2NQ==
created_at: 2020-07-18T20:15:12Z
updated_at: 2020-07-18T20:15:12Z
author_association: MEMBER
body:
  I want to create a SQL query which shows me all of my repositories that have commits that are NOT in the most recent release.

  The releases table doesn't have enough information for this because it doesn't tell you the commit hash associated with each release, just the tag: https://github-to-sqlite.dogsheep.net/github/releases

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660547502
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43
node_id: MDEyOklzc3VlQ29tbWVudDY2MDU0NzUwMg==
created_at: 2020-07-18T21:50:37Z
updated_at: 2020-07-18T21:50:37Z
author_association: MEMBER
body:
  ```
  $ github-to-sqlite tags tags.db simonw/datasette dogsheep/github-to-sqlite
  $ sqlite-utils tables tags.db --counts
  [{"table": "users", "count": 2},
   {"table": "licenses", "count": 1},
   {"table": "repos", "count": 2},
   {"table": "tags", "count": 76},
   {"table": "licenses_fts", "count": 1},
   {"table": "licenses_fts_data", "count": 3},
   {"table": "licenses_fts_idx", "count": 1},
   {"table": "licenses_fts_docsize", "count": 1},
   {"table": "licenses_fts_config", "count": 1},
   {"table": "repos_fts", "count": 2},
   {"table": "repos_fts_data", "count": 3},
   {"table": "repos_fts_idx", "count": 1},
   {"table": "repos_fts_docsize", "count": 2},
   {"table": "repos_fts_config", "count": 1},
   {"table": "users_fts", "count": 2},
   {"table": "users_fts_data", "count": 3},
   {"table": "users_fts_idx", "count": 1},
   {"table": "users_fts_docsize", "count": 2},
   {"table": "users_fts_config", "count": 1}]
  $ sqlite-utils rows tags.db tags
  [{"repo_id": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"},
   {"repo_id": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"},
   {"repo_id": 107914493, "name": "0.45a4", "sha": "265483173bc8341dc02c8b782b9b59d2ce8bbedb"},
   {"repo_id": 107914493, "name": "0.45a3", "sha": "1f55a4a2b68fa65e56a28baeb7f44122fdeca7e7"},
   {"repo_id": 107914493, "name": "0.45a2", "sha": "1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f"},
   {"repo_id": 107914493, "name": "0.45a1", "sha": "b59b92b1b0517cf18fa748ff9d0a0bf86298dd43"},
   {"repo_id": 107914493, "name": "0.45a0", "sha": "dda932d818b34ccab11730a76554f0a3748d8348"},
   {"repo_id": 107914493, "name": "0.44", "sha": "b906030235efbdff536405d66078f4868ce0d3bd"},
   {"repo_id": 107914493, "name": "0.43", "sha": "d56f402822df102f9cf1a9a056449d01a15e3aae"},
   {"repo_id": 107914493, "name": "0.42", "sha": "af6c6c5d6f929f951c0e63bfd1c82e37a071b50f"},
   {"repo_id": 107914493, "name": "0.41", "sha": "182e5c8745c94576718315f7596ccc81e5e2417b"},
   {"repo_id": 107914493, "name": "0.40", "sha": "8da108193b08abf140716f8ac499f32309dfe9cf"},
   {"repo_id": 107914493, "name": "0.39", "sha": "dedd775512daee49925882654f252df61a9e3b6d"},
   {"repo_id": 107914493, "name": "0.38", "sha": "7e357abbc38dcc9d19a2f1df3252668a48e941e4"},
   {"repo_id": 107914493, "name": "0.37.1", "sha": "be20e6991eac2baa9b43e9b26ae209bae805ede5"},
   {"repo_id": 107914493, "name": "0.37", "sha": "c9e6841482b299fceadc5ad548c2dbf58a8f1227"},
   {"repo_id": 107914493, "name": "0.36", "sha": "b031fe97636b80b05fec409ee1dffb7d044fd4e9"},
   {"repo_id": 107914493, "name": "0.35", "sha": "30b6f71b306a43605c99bef79302ed5cb22d1924"},
   {"repo_id": 107914493, "name": "0.34", "sha": "e7f60d2a9b59752e20de8412f7b0a3e9a5359a31"},
   {"repo_id": 107914493, "name": "0.33", "sha": "59e7014c8a0f4102d7dc79f517540c55c49e1554"},
   {"repo_id": 107914493, "name": "0.32", "sha": "a95bedb9c423fa6d772c93ef47bc40f13a5bea50"},
   {"repo_id": 107914493, "name": "0.31.2", "sha": "b51f258d00bb3c3b401f15d46a1fbd50394dbe1c"},
   {"repo_id": 107914493, "name": "0.31.1", "sha": "a22c7761b61baa61b8e3da7d30887468d61d6b83"},
   {"repo_id": 107914493, "name": "0.31", "sha": "7f89928062b1a1fdb2625a946f7cd5161e597401"},
   {"repo_id": 107914493, "name": "0.30.2", "sha": "2bf7ce5f517d772a16d7855a35a8a75d4456aad7"},
   {"repo_id": 107914493, "name": "0.30.1", "sha": "3ca290e0db03bb4747e24203c445873f74512107"},
   {"repo_id": 107914493, "name": "0.30", "sha": "8050f9e1ece9afd0236ad38c6458c12a4ad917e6"},
   {"repo_id": 107914493, "name": "0.29.3", "sha": "0fc8afde0eb5ef677f4ac31601540d6168c8208d"},
   {"repo_id": 107914493, "name": "0.29.2", "sha": "6abe6faff6b035e9334dd05f8c741ae9b7a47440"},
   {"repo_id": 107914493, "name": "0.29.1", "sha": "2a94f3719fb2c4335fcda374fa92f87272b02d34"},
   {"repo_id": 107914493, "name": "0.29", "sha": "fb7ee8e0ad59a15083234a48e935525f6e7257dd"},
   {"repo_id": 107914493, "name": "0.28", "sha": "e518f76c5f5dd0138032bfb26387f5bb91086a3f"},
   {"repo_id": 107914493, "name": "0.27.1", "sha": "3f3f29ac9afe7c41ffc48a3bd2af473a53eecc8a"},
   {"repo_id": 107914493, "name": "0.27", "sha": "436b8bc1d17c2ab415800ab209204f94e7f7929e"},
   {"repo_id": 107914493, "name": "0.26.2", "sha": "a418c8b44f82d456be523c8690cf7236bb648c22"},
   {"repo_id": 107914493, "name": "0.26.1", "sha": "4722acc73ce761556b18f5dcbe36b7fef2ee2c69"},
   {"repo_id": 107914493, "name": "0.26", "sha": "424e146697309a54c05d5d1ba1f840849ddbafdc"},
   {"repo_id": 107914493, "name": "0.25.2", "sha": "b5128fc53fce6a1bf3b16bad9f318451bc1d1263"},
   {"repo_id": 107914493, "name": "0.25.1", "sha": "3dc0b3fa8c9b9bd81540ffe20c8b7e7a72465274"},
   {"repo_id": 107914493, "name": "0.25", "sha": "57a71377c992753327a16b417daf79df7f506dd1"},
   {"repo_id": 107914493, "name": "0.24", "sha": "28872a1fa789f314b0342f4e6182f1c78d6e2bca"},
   {"repo_id": 107914493, "name": "0.23.2", "sha": "6df6f712b36f0fe75694174906e31242427a8d1d"},
   {"repo_id": 107914493, "name": "0.23.1", "sha": "dea86b9fba78e032ad09673e884e764387daf209"},
   {"repo_id": 107914493, "name": "0.23", "sha": "e04f5b0d348ef7275a0a5ab9eb53527105132885"},
   {"repo_id": 107914493, "name": "0.22.1", "sha": "5d6252788230d168ba09f379d1d2af867e3302ab"},
   {"repo_id": 107914493, "name": "0.22", "sha": "558d9d7bfef3dd633eb16389281b67d42c9bdeef"},
   {"repo_id": 107914493, "name": "0.21", "sha": "403211de632cd15f0820cc9399305fc43c187b47"},
   {"repo_id": 107914493, "name": "0.20", "sha": "3a5d7951ce8f35118ffdd7f8d86e09b909e1218c"},
   {"repo_id": 107914493, "name": "0.19", "sha": "ba9bfa583179c25aaef94b1f44da7eba74620b9a"},
   {"repo_id": 107914493, "name": "0.18", "sha": "43ae15c0d14b3e968e8d5bfef72ac0c39783c3a2"},
   {"repo_id": 107914493, "name": "0.17", "sha": "fb988ace7c7e2bee5ac142a0eab22431d0675a77"},
   {"repo_id": 107914493, "name": "0.16", "sha": "b6539ff04502536bd1fa96e3b1430bdafc456826"},
   {"repo_id": 107914493, "name": "0.15", "sha": "7706fe0c67aba5cfe905c7906cae9e0c43cd75b2"},
   {"repo_id": 107914493, "name": "0.14", "sha": "2edc652df6d786e4f2c3f073e3567002d248be09"},
   {"repo_id": 107914493, "name": "0.13", "sha": "c160f15c3937f8fbe581276f811e8c58f9137bb1"},
   {"repo_id": 107914493, "name": "0.12", "sha": "51bdd67691bd69082ae7690af8b905f06050ee80"},
   {"repo_id": 107914493, "name": "0.11", "sha": "b0f3d4e375655f0764f3137dbcede324f9bbc0cb"},
   {"repo_id": 107914493, "name": "0.10", "sha": "5928c11ee798a232aa4096706cd47e639d1c9fc2"},
   {"repo_id": 107914493, "name": "0.9", "sha": "d75f423b6fcfc074b7c6f8f7679da8876f181edd"},
   {"repo_id": 107914493, "name": "0.8", "sha": "fe279ab7b4ae99dab295d5cf4d39ad06d782997e"},
   {"repo_id": 107914493, "name": "0.7", "sha": "6b3b05b6db0d2a7b7cec8b8dbb4ddc5e12a376b2"},
   {"repo_id": 207052882, "name": "2.3", "sha": "7090e43d804724ef3b31ae5ca9efd6ac05f76cbc"},
   {"repo_id": 207052882, "name": "2.2", "sha": "4fe69783b55465e7692a807d3a02a710f69c9c42"},
   {"repo_id": 207052882, "name": "2.1", "sha": "9d7aed336c8e62bf372caa800cb4aae3985cbae9"},
   {"repo_id": 207052882, "name": "2.0", "sha": "44611df1524a03ce305405e5902c9615e3c73a72"},
   {"repo_id": 207052882, "name": "1.1", "sha": "5cd34bd07d704487d48ac741ee5da5317afe88d2"},
   {"repo_id": 207052882, "name": "1.0.1", "sha": "3b7ab5685de89fcb6fc92d320c0e24b17be05570"},
   {"repo_id": 207052882, "name": "1.0", "sha": "1ea30c8fb1d080bd5e38c577e3ad20bb527a2fe6"},
   {"repo_id": 207052882, "name": "0.7", "sha": "e35eec4343aa560c58c1634cc228d0d46c442304"},
   {"repo_id": 207052882, "name": "0.6", "sha": "9eb737090fafd0e5a7e314be48402374d99e9828"},
   {"repo_id": 207052882, "name": "0.5", "sha": "ae9035f8fe5aff1c54bff4c6b4c2e808a44f0f2a"},
   {"repo_id": 207052882, "name": "0.4", "sha": "8c6251c31a05c58c2bfbef114247642d1b3dbb44"},
   {"repo_id": 207052882, "name": "0.3", "sha": "f697f247468516aa4ee13b1862b59e0dba18d00f"},
   {"repo_id": 207052882, "name": "0.2", "sha": "0fe96bc50fb3d7b1c7e4577db0ddf207eaeebbb9"},
   {"repo_id": 207052882, "name": "0.1.1", "sha": "321e0284c64dc48b2143311009886293c05edb07"},
   {"repo_id": 207052882, "name": "0.1", "sha": "7387c88a3f84704548e81d43b91615c02b61a957"}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660551397
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43
node_id: MDEyOklzc3VlQ29tbWVudDY2MDU1MTM5Nw==
created_at: 2020-07-18T22:27:32Z
updated_at: 2020-07-18T23:05:45Z
author_association: MEMBER
body:
  ```sql
  with most_recent_releases as (
    with ranked as (
      select
        repo,
        tag_name,
        published_at,
        row_number() OVER (
          partition BY repo
          ORDER BY
            published_at DESC
        ) rank
      FROM
        releases
    )
    select
      *
    from
      ranked
    where
      rank = 1
  )
  select
    repos.full_name as repo,
    most_recent_releases.tag_name as release,
    commits.committer_date as release_commit_date,
    (
      select
        count(*)
      from
        commits c2
      where
        c2.repo = repos.id
        and c2.committer_date > commits.committer_date
    ) as commits_since_release,
    'https://github.com/' || repos.full_name || '/compare/' || most_recent_releases.tag_name || '...' || repos.default_branch as view_commits
  from
    most_recent_releases
    join repos on most_recent_releases.repo = repos.id
    join tags on tags.repo = repos.id
    and tags.name = most_recent_releases.tag_name
    join commits on tags.sha = commits.sha
  order by
    commits_since_release desc
  ```


  repo | release | release_commit_date | commits_since_release | view_commits
  -- | -- | -- | -- | --
  simonw/datasette | 0.45 | 2020-07-01T21:43:07Z | 9 | https://github.com/simonw/datasette/compare/0.45...master
  dogsheep/twitter-to-sqlite | 0.21.1 | 2020-04-30T18:20:43Z | 2 | https://github.com/dogsheep/twitter-to-sqlite/compare/0.21.1...master
  dogsheep/github-to-sqlite | 2.3 | 2020-07-09T23:26:34Z | 2 | https://github.com/dogsheep/github-to-sqlite/compare/2.3...master
  dogsheep/dogsheep-photos | 0.4.1 | 2020-05-25T20:11:20Z | 2 | https://github.com/dogsheep/dogsheep-photos/compare/0.4.1...master
  dogsheep/swarm-to-sqlite | 0.3.1 | 2020-03-28T02:29:41Z | 1 | https://github.com/dogsheep/swarm-to-sqlite/compare/0.3.1...master
  dogsheep/hacker-news-to-sqlite | 0.3.1 | 2020-03-21T22:39:34Z | 1 | https://github.com/dogsheep/hacker-news-to-sqlite/compare/0.3.1...master
  simonw/sqlite-utils | 2.11 | 2020-07-08T17:36:07Z | 0 | https://github.com/simonw/sqlite-utils/compare/2.11...master
  dogsheep/healthkit-to-sqlite | 0.5 | 2020-03-28T01:50:51Z | 0 | https://github.com/dogsheep/healthkit-to-sqlite/compare/0.5...master
  dogsheep/inaturalist-to-sqlite | 0.2 | 2020-03-24T00:35:44Z | 0 | https://github.com/dogsheep/inaturalist-to-sqlite/compare/0.2...master
  dogsheep/genome-to-sqlite | 0.1 | 2019-09-19T15:38:10Z | 0 | https://github.com/dogsheep/genome-to-sqlite/compare/0.1...master
  dogsheep/pocket-to-sqlite | 0.2 | 2020-03-27T22:23:16Z | 0 | https://github.com/dogsheep/pocket-to-sqlite/compare/0.2...master

  https://github-to-sqlite.dogsheep.net/github?sql=with+most_recent_releases+as+%28%0D%0A++with+ranked+as+%28%0D%0A++++select%0D%0A++++++repo%2C%0D%0A++++++tag_name%2C%0D%0A++++++published_at%2C%0D%0A++++++row_number%28%29+OVER+%28%0D%0A++++++++partition+BY+repo%0D%0A++++++++ORDER+BY%0D%0A++++++++++published_at+DESC%0D%0A++++++%29+rank%0D%0A++++FROM%0D%0A++++++releases%0D%0A++%29%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++ranked%0D%0A++where%0D%0A++++rank+%3D+1%0D%0A%29%0D%0Aselect%0D%0A++repos.full_name+as+repo%2C%0D%0A++most_recent_releases.tag_name+as+release%2C%0D%0A++commits.committer_date+as+release_commit_date%2C%0D%0A++%28%0D%0A++++select%0D%0A++++++count%28*%29%0D%0A++++from%0D%0A++++++commits+c2%0D%0A++++where%0D%0A++++++c2.repo+%3D+repos.id%0D%0A++++++and+c2.committer_date+%3E+commits.committer_date%0D%0A++%29+as+commits_since_release%2C%0D%0A++%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos.full_name+%7C%7C+%27%2Fcompare%2F%27+%7C%7C+most_recent_releases.tag_name+%7C%7C+%27...%27+%7C%7C+repos.default_branch+as+view_commits%0D%0Afrom%0D%0A++most_recent_releases%0D%0A++join+repos+on+most_recent_releases.repo+%3D+repos.id%0D%0A++join+tags+on+tags.repo+%3D+repos.id%0D%0A++and+tags.name+%3D+most_recent_releases.tag_name%0D%0A++join+commits+on+tags.sha+%3D+commits.sha%0D%0Aorder+by%0D%0A++commits_since_release+desc

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553646
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45
node_id: MDEyOklzc3VlQ29tbWVudDY2MDU1MzY0Ng==
created_at: 2020-07-18T22:51:41Z
updated_at: 2020-07-18T22:51:41Z
author_association: MEMBER
body:
  I could fix this by putting `REFRESH_DB` in a commit message:

  https://github.com/dogsheep/github-to-sqlite/blob/4ae4aa6f172344b19ff3513707195ee6d2654bd4/.github/workflows/deploy-demo.yml#L41-L46

  But... doing so would lose the data I've collected in https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc concerning the first time each dependent repo was spotted.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553711
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45
node_id: MDEyOklzc3VlQ29tbWVudDY2MDU1MzcxMQ==
created_at: 2020-07-18T22:52:16Z
updated_at: 2020-07-18T22:52:16Z
author_association: MEMBER
body:
  I think the best fix is to download the `github.db` database, manually fix it and then manually deploy it to Cloud Run from my laptop.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554162
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45
node_id: MDEyOklzc3VlQ29tbWVudDY2MDU1NDE2Mg==
created_at: 2020-07-18T22:56:58Z
updated_at: 2020-07-18T22:56:58Z
author_association: MEMBER
body:
  Manually fixing the database:
  ```
  $ wget 'https://github-to-sqlite.dogsheep.net/github.db'
  --2020-07-18 15:52:33--  https://github-to-sqlite.dogsheep.net/github.db
  Resolving github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)... 172.217.5.115
  Connecting to github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)|172.217.5.115|:443... connected.
  HTTP request sent, awaiting response... 200 OK
  Length: 14626816 (14M) [application/octet-stream]
  Saving to: ‘github.db’

  github.db                              100%[============================================================================>]  13.95M  1.22MB/s    in 18s     

  2020-07-18 15:52:53 (773 KB/s) - ‘github.db’ saved [14626816/14626816]

  $ sqlite3 github.db 
  SQLite version 3.28.0 2019-04-15 14:49:49
  Enter ".help" for usage hints.
  sqlite> drop table tags;
  sqlite> ^D
  $ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils dogsheep/healthkit-to-sqlite dogsheep/swarm-to-sqlite dogsheep/twitter-to-sqlite dogsheep/inaturalist-to-sqlite dogsheep/google-takeout-to-sqlite dogsheep/github-to-sqlite dogsheep/genome-to-sqlite dogsheep/pocket-to-sqlite dogsheep/hacker-news-to-sqlite dogsheep/dogsheep-photos 
  $ sqlite-utils tables github.db --counts
  [{"table": "users", "count": 4048},
   {"table": "repos", "count": 210},
   ...
   {"table": "stars", "count": 4140},
   {"table": "tags", "count": 188}]
  $ sqlite-utils rows github.db tags    
  [{"repo": 107914493, "name": "0.45", "sha": "f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb"},
   {"repo": 107914493, "name": "0.45a5", "sha": "676bb64c877d73f8ff496cef4632f5a8a5a9283c"},
   ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554299
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45
node_id: MDEyOklzc3VlQ29tbWVudDY2MDU1NDI5OQ==
created_at: 2020-07-18T22:58:24Z
updated_at: 2020-07-18T23:02:52Z
author_association: MEMBER
body:
  Deploying the fixed version like this:
  ```
  $ gcloud config set run/region us-central1
  $ gcloud config set project datasette-222320
  $ datasette publish cloudrun /tmp/github.db \
              -m demo-metadata.json \
              --service github-to-sqlite \
              --install=py-gfm \
              --install='datasette-search-all>=0.3' \
              --install='datasette-render-markdown>=1.1.2' \
              --install=datasette-pretty-json \
              --install=datasette-json-html \
              --install=datasette-vega
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554811
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45
node_id: MDEyOklzc3VlQ29tbWVudDY2MDU1NDgxMQ==
created_at: 2020-07-18T23:03:13Z
updated_at: 2020-07-18T23:03:13Z
author_association: MEMBER
body:
  https://github-to-sqlite.dogsheep.net/github/tags now shows a `repo` column instead of a `repo_id` column.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/456#issuecomment-661524006
issue_url: https://api.github.com/repos/simonw/datasette/issues/456
node_id: MDEyOklzc3VlQ29tbWVudDY2MTUyNDAwNg==
created_at: 2020-07-21T01:15:07Z
updated_at: 2020-07-21T01:15:07Z
author_association: CONTRIBUTOR
body:
  Bumping this, as the previous fix is passing the wrong type, and not actually addressing the issue...

  The `exclude` argument needs an iterable of packages instead of a single string (but since `str` is iterable, it's currently excluding packages `t`, `e`, and `s`.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/902#issuecomment-661587375
issue_url: https://api.github.com/repos/simonw/datasette/issues/902
node_id: MDEyOklzc3VlQ29tbWVudDY2MTU4NzM3NQ==
created_at: 2020-07-21T02:44:49Z
updated_at: 2020-07-21T02:44:49Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=h1) Report
  > Merging [#902](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/d9a5ef1c32a4390e398653ebfd570f8e1a03d93e&el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/902/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##           master     #902   +/-   ##
  =======================================
    Coverage   83.41%   83.41%           
  =======================================
    Files          27       27           
    Lines        3636     3636           
  =======================================
    Hits         3033     3033           
    Misses        603      603           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=footer). Last update [d9a5ef1...9aa139d](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/905#issuecomment-662114881
issue_url: https://api.github.com/repos/simonw/datasette/issues/905
node_id: MDEyOklzc3VlQ29tbWVudDY2MjExNDg4MQ==
created_at: 2020-07-21T21:25:37Z
updated_at: 2020-07-21T21:25:37Z
author_association: OWNER
body:
  I can use `aiofiles.os.stat` for this: https://github.com/Tinche/aiofiles/blob/master/aiofiles/os.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/905#issuecomment-662241702
issue_url: https://api.github.com/repos/simonw/datasette/issues/905
node_id: MDEyOklzc3VlQ29tbWVudDY2MjI0MTcwMg==
created_at: 2020-07-22T04:59:46Z
updated_at: 2020-07-22T04:59:46Z
author_association: OWNER
body:
  Deployed and working:
  ```
  % curl -I 'https://fivethirtyeight.datasettes.com/fivethirtyeight.db'
  HTTP/1.1 200 OK
  Date: Wed, 22 Jul 2020 04:59:23 GMT
  Content-Type: application/octet-stream
  Content-Length: 281845760
  Connection: keep-alive
  Set-Cookie: __cfduid=d550b15c99aa59144e49557ced64fc48a1595393963; expires=Fri, 21-Aug-20 04:59:23 GMT; path=/; domain=.datasettes.com; HttpOnly; SameSite=Lax
  Via: 1.1 vegur
  Cache-Control: max-age=14400
  CF-Cache-Status: MISS
  Accept-Ranges: bytes
  cf-request-id: 04167d0c7100000540f98e8200000001
  Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
  Server: cloudflare
  CF-RAY: 5b6a978d89b30540-LAX
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-663143160
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48
node_id: MDEyOklzc3VlQ29tbWVudDY2MzE0MzE2MA==
created_at: 2020-07-23T17:46:07Z
updated_at: 2020-07-23T17:46:07Z
author_association: MEMBER
body:
  Frustratingly, these links don't work on PyPI: https://pypi.org/project/twitter-to-sqlite/

  There's an issue about that here: https://github.com/pypa/readme_renderer/issues/169

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/906#issuecomment-663720907
issue_url: https://api.github.com/repos/simonw/datasette/issues/906
node_id: MDEyOklzc3VlQ29tbWVudDY2MzcyMDkwNw==
created_at: 2020-07-24T20:29:24Z
updated_at: 2020-07-24T20:29:24Z
author_association: OWNER
body:
  Here are the existing test cases:
  https://github.com/simonw/datasette/blob/2115d7e3457b48b3cf9c81551b9fed2d0e9cd111/tests/test_utils.py#L468-L505

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/907#issuecomment-663726146
issue_url: https://api.github.com/repos/simonw/datasette/issues/907
node_id: MDEyOklzc3VlQ29tbWVudDY2MzcyNjE0Ng==
created_at: 2020-07-24T20:43:27Z
updated_at: 2020-07-24T20:43:27Z
author_association: OWNER
body:
  It might be good to have a little interactive tool which helps debug these things, since there are quite a few edge-cases and the damage caused if people use them incorrectly is substantial.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/908#issuecomment-663726318
issue_url: https://api.github.com/repos/simonw/datasette/issues/908
node_id: MDEyOklzc3VlQ29tbWVudDY2MzcyNjMxOA==
created_at: 2020-07-24T20:43:57Z
updated_at: 2020-07-24T20:45:38Z
author_association: OWNER
body:
  I can implement this as a plugin. Or it could ship as part of Datasette, somewhere under the `/-/` namespace like the `PermissionsDebugView` and `MessagesDebugView` tools.

  I'm going to ship it in Datasette core, to further reinforce the philosophy that debugging tools are important.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/901#issuecomment-663727716
issue_url: https://api.github.com/repos/simonw/datasette/issues/901
node_id: MDEyOklzc3VlQ29tbWVudDY2MzcyNzcxNg==
created_at: 2020-07-24T20:47:57Z
updated_at: 2020-07-24T20:47:57Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=h1) Report
  > Merging [#901](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/d9a5ef1c32a4390e398653ebfd570f8e1a03d93e&el=desc) will **decrease** coverage by `0.00%`.
  > The diff coverage is `83.33%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/901/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##           master     #901      +/-   ##
  ==========================================
  - Coverage   83.41%   83.41%   -0.01%     
  ==========================================
    Files          27       27              
    Lines        3636     3642       +6     
  ==========================================
  + Hits         3033     3038       +5     
  - Misses        603      604       +1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/901/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.76% <66.66%> (-0.18%)` | :arrow_down: |
  | [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/901/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `94.35% <100.00%> (+0.09%)` | :arrow_up: |
  | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/901/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `91.47% <0.00%> (+0.07%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=footer). Last update [d9a5ef1...1285f28](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/907#issuecomment-663764203
issue_url: https://api.github.com/repos/simonw/datasette/issues/907
node_id: MDEyOklzc3VlQ29tbWVudDY2Mzc2NDIwMw==
created_at: 2020-07-24T22:53:07Z
updated_at: 2020-07-24T22:53:07Z
author_association: OWNER
body:
  Actually that is already covered here: 
  https://github.com/simonw/datasette/blob/6be5654ffab282e8cf39cc138ba2d4496ebc7407/docs/authentication.rst#L158

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/906#issuecomment-663767678
issue_url: https://api.github.com/repos/simonw/datasette/issues/906
node_id: MDEyOklzc3VlQ29tbWVudDY2Mzc2NzY3OA==
created_at: 2020-07-24T23:07:22Z
updated_at: 2020-07-24T23:07:22Z
author_association: OWNER
body:
  Illustration of current system:

  https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22terry%22%0D%0A%7D&allow=null - `null` allows

  https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22terry%22%0D%0A%7D&allow={} - `{}` denies

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/908#issuecomment-663779179
issue_url: https://api.github.com/repos/simonw/datasette/issues/908
node_id: MDEyOklzc3VlQ29tbWVudDY2Mzc3OTE3OQ==
created_at: 2020-07-25T00:05:48Z
updated_at: 2020-07-25T00:06:15Z
author_association: OWNER
body:
  The documentation section here now has a bunch of different links to live demos illustrating different "allow" block syntax: https://github.com/simonw/datasette/blob/092874202c8748d6e0d4800eaf707c0145d95ffe/docs/authentication.rst#defining-permissions-with-allow-blocks

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/906#issuecomment-663779460
issue_url: https://api.github.com/repos/simonw/datasette/issues/906
node_id: MDEyOklzc3VlQ29tbWVudDY2Mzc3OTQ2MA==
created_at: 2020-07-25T00:07:10Z
updated_at: 2020-07-25T00:07:10Z
author_association: OWNER
body:
  Demo of the new functionality:

  * https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22root%22%0D%0A%7D&allow=false
  * https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22root%22%0D%0A%7D&allow=true

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/868#issuecomment-663924220
issue_url: https://api.github.com/repos/simonw/datasette/issues/868
node_id: MDEyOklzc3VlQ29tbWVudDY2MzkyNDIyMA==
created_at: 2020-07-26T01:29:00Z
updated_at: 2020-07-26T01:29:00Z
author_association: NONE
body:
  Ok, so it's been a while but I think I'm making progress. The good news: I have come up with a configuration change for running the tests on Windows on GitHub Actions. The bad news: there's a bunch of extraneous errors on the Windows case. I *think* this is due to Windows file IO and sqlite in a lot of cases, so I'm working through it. I will eventually clean up this PR.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931279
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/122
node_id: MDEyOklzc3VlQ29tbWVudDY2MzkzMTI3OQ==
created_at: 2020-07-26T03:33:23Z
updated_at: 2020-07-27T04:30:49Z
author_association: OWNER
body:
  One idea: `sqlite-utils insert-files`

  It could work something like this:

      sqlite-utils insert-files files.db /tmp/blah.jpg /tmp/foo.gif \
        --table files \
        -c key:filename -c hash:sha256 -c body:content \
        --pk key

  This would insert those two image files into the database in a table called `files` with a schema that looks something like this:

  ```sql
  CREATE TABLE files (
      key text primary key,
      hash text,
      body blob
  );
  ```

  The `-c key:filename` options here are the most interesting: they let you create the table with a specific layout. The bit before the `:` is the column name. The bit after the `:` can be a range of different things:

  - `filename` - just the filename
  - `filepath` - the full filepath (provided on the command-line)
  - `absolutepath` - the filepath expanded to start with `/home/...` or whatever
  - `sha256` - the SHA256 of the contents
  - `md5` - the MD5
  - `content` - the binary content itself
  - `mtime` - the mtime (floating point timestamp)
  - `ctime` - the ctime (floating point timestamp)
  - `mtime_iso` - the mtime as an ISO datetime
  - `ctime_iso` - the mtime as an ISO datetime
  - `size` - the size of the file in bytes

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931317
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/122
node_id: MDEyOklzc3VlQ29tbWVudDY2MzkzMTMxNw==
created_at: 2020-07-26T03:33:54Z
updated_at: 2020-07-26T03:33:54Z
author_association: OWNER
body:
  The command also accepts one or more directories, in which case it will recursively scan them for all files that they contain.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012148
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/125
node_id: MDEyOklzc3VlQ29tbWVudDY2NDAxMjE0OA==
created_at: 2020-07-26T16:47:51Z
updated_at: 2020-07-26T16:47:51Z
author_association: OWNER
body:
  Best solution I can think of is to return the data as base64. It's a bit nasty since it means you can't round trip it back again.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/125
node_id: MDEyOklzc3VlQ29tbWVudDY2NDAxMjI0Nw==
created_at: 2020-07-26T16:48:46Z
updated_at: 2020-07-26T16:48:46Z
author_association: OWNER
body:
  I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says "these columns are base64 encoded, store the decoded data in a BLOB".

  That would solve inserting binary data using JSON too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664013338
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/122
node_id: MDEyOklzc3VlQ29tbWVudDY2NDAxMzMzOA==
created_at: 2020-07-26T16:57:35Z
updated_at: 2020-07-26T16:57:35Z
author_association: OWNER
body:
  I should consider easy compatibility with https://www.sqlite.org/sqlar.html

  > An SQLite Archive is an ordinary SQLite database file that contains the following table as part of its schema:
  > ```
  > CREATE TABLE sqlar(
  >   name TEXT PRIMARY KEY,  -- name of the file
  >   mode INT,               -- access permissions
  >   mtime INT,              -- last modification time
  >   sz INT,                 -- original file size
  >   data BLOB               -- compressed content
  > );
  > ```
  > Each row of the SQLAR table holds the content of a single file. The filename (the full pathname relative to the root of the archive) is in the "name" field. The "mode" field is an integer which is the unix-style access permissions for the file. "mtime" is the modification time of the file in seconds since 1970. "sz" is the original uncompressed size of the file. The "data" field contains the file content. The content is usually compressed using [Deflate](http://zlib.net/), though not always. If the "sz" field is equal to the size of the "data" field, then the content is stored uncompressed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664048432
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/122
node_id: MDEyOklzc3VlQ29tbWVudDY2NDA0ODQzMg==
created_at: 2020-07-26T22:29:31Z
updated_at: 2020-07-26T22:29:31Z
author_association: OWNER
body:
  I'm trying to play with `sqlite3 -A` on my Mac.

  `sqlite3 -A` tells me that it's an unknown option - but I used `brew info sqlite` to find my homebrew installed version and it turns out this works:

  ```
   % /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 -A
  Wrong number of arguments.  Usage:
  .archive ...             Manage SQL archives
     Each command must have exactly one of the following options:
       -c, --create               Create a new archive
       -u, --update               Add or update files with changed mtime
       -i, --insert               Like -u but always add even if unchanged
       -t, --list                 List contents of archive
       -x, --extract              Extract files from archive
     Optional arguments:
       -v, --verbose              Print each filename as it is processed
       -f FILE, --file FILE       Use archive FILE (default is current db)
       -a FILE, --append FILE     Open FILE using the apndvfs VFS
       -C DIR, --directory DIR    Read/extract files from directory DIR
       -n, --dryrun               Show the SQL that would have occurred
     Examples:
       .ar -cf ARCHIVE foo bar  # Create ARCHIVE from files foo and bar
       .ar -tf ARCHIVE          # List members of ARCHIVE
       .ar -xvf ARCHIVE         # Verbosely extract files from ARCHIVE
     See also:
        http://sqlite.org/cli.html#sqlar_archive_support
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664048720
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/122
node_id: MDEyOklzc3VlQ29tbWVudDY2NDA0ODcyMA==
created_at: 2020-07-26T22:32:50Z
updated_at: 2020-07-26T22:33:20Z
author_association: OWNER
body:
  This seems to work in creating a SQLite archive containing all `.gif` files in the current directory:

      /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 archive.db -A -c *.gif

  Then listing files like this:
  ```
  $ /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 archive.db -A -t
  copyable.gif
  debug-allow.gif
  flash.gif
  table-md.gif
  ```
  Here's the schema:
  ```
  $ sqlite3 archive.db .schema                                  
  CREATE TABLE sqlar(
    name TEXT PRIMARY KEY,  -- name of the file
    mode INT,               -- access permissions
    mtime INT,              -- last modification time
    sz INT,                 -- original file size
    data BLOB               -- compressed content
  );
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664062546
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/125
node_id: MDEyOklzc3VlQ29tbWVudDY2NDA2MjU0Ng==
created_at: 2020-07-27T00:33:03Z
updated_at: 2020-07-27T00:33:03Z
author_association: OWNER
body:
  I'm going to imitate how Datasette solves this problem:
  ```json
  [
    {
      "name": "lorem.txt",
      "mode": 33188,
      "mtime": 1595805965,
      "sz": 16984,
      "data": {
        "$base64": true,
        "encoded": "eJzt0c1xAyEMBeC7q1ABHleR3HxNAQrIjmb4M0gelx+RTY7p4N2WBYT0vmufUknH8kq5lz5pqRFXsTOl3pYkE/NJnHXoStruJEVjc0mOCyTqq/ZMJnXEZW1Js2ZvRm5U+DPKk9hRWqjyvTFx0YfzhT6MpGmN2lR1fzxjyfVMD9dFrS+bnkleMpMam/ZGXgrX1I/K+5Au3S/9lNQRh0k4Gq/RUz8GiKfsQm+7JLsJ6fTo5JhVG00ZU76kZZkxePx49uIjnpNoJyYlWUsoaSl/CcVATje/Kxu13RANnrHweaH3V5Jh4jvGyKCnxJLiXPKhmW3fiCnG7Jql7RR3UvFo8jJ4z039dtOkTFmWzL1be9lt8A5II471m6vXy+l0BR/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984PuP7xubBoN9"
      }
    }
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664065341
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/125
node_id: MDEyOklzc3VlQ29tbWVudDY2NDA2NTM0MQ==
created_at: 2020-07-27T00:49:41Z
updated_at: 2020-07-27T00:49:41Z
author_association: OWNER
body:
  Documentation: https://github.com/simonw/sqlite-utils/commit/20e543e9a492f2e764caae73c38e87f18eaec444?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/126#issuecomment-664065597
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/126
node_id: MDEyOklzc3VlQ29tbWVudDY2NDA2NTU5Nw==
created_at: 2020-07-27T00:51:11Z
updated_at: 2020-07-27T00:51:11Z
author_association: OWNER
body:
  I'm going to implement this as the reverse of #125 - binary columns in JSON are now output like this:
  ```json
    {
      "name": "lorem.txt",
      "mode": 33188,
      "mtime": 1595805965,
      "sz": 16984,
      "data": {
        "$base64": true,
        "encoded": "eJzt0c1xAyEMBeC7q1ABHleR3HxNAQrIjmb4M0gelx+RTY7p4N2WBYT0vmufUknH8kq5lz5pqRFXsTOl3pYkE/NJnHXoStruJEVjc0mOCyTqq/ZMJnXEZW1Js2ZvRm5U+DPKk9hRWqjyvTFx0YfzhT6MpGmN2lR1fzxjyfVMD9dFrS+bnkleMpMam/ZGXgrX1I/K+5Au3S/9lNQRh0k4Gq/RUz8GiKfsQm+7JLsJ6fTo5JhVG00ZU76kZZkxePx49uIjnpNoJyYlWUsoaSl/CcVATje/Kxu13RANnrHweaH3V5Jh4jvGyKCnxJLiXPKhmW3fiCnG7Jql7RR3UvFo8jJ4z039dtOkTFmWzL1be9lt8A5II471m6vXy+l0BR/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984PuP7xubBoN9"
      }
    }
  ]
  ```
  So the `sqlite-utils insert` command should learn to spot `{"$base64": true...}` values and base64 decode them before inserting them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/124#issuecomment-664105302
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/124
node_id: MDEyOklzc3VlQ29tbWVudDY2NDEwNTMwMg==
created_at: 2020-07-27T03:54:24Z
updated_at: 2020-07-30T22:57:51Z
author_association: OWNER
body:
  Documentation: https://github.com/simonw/sqlite-utils/commit/814d4a7f90991be865d38aac45ff12e36df1c67d?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40

  > You can pass named parameters to the query using -p:
  >
  >     $ sqlite-utils query dogs.db "select :num * :num2" -p num 5 -p num2 6
  >     [{":num * :num2": 30}]

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/126#issuecomment-664106405
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/126
node_id: MDEyOklzc3VlQ29tbWVudDY2NDEwNjQwNQ==
created_at: 2020-07-27T04:00:08Z
updated_at: 2020-07-27T04:00:33Z
author_association: OWNER
body:
  ```
  $ echo '[
      {
          "name": "transparent.gif",
          "content": {
              "$base64": true,
              "encoded": "R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"
          }
      }
  ]' | sqlite-utils insert trans.db files - --pk=name
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664127741
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/122
node_id: MDEyOklzc3VlQ29tbWVudDY2NDEyNzc0MQ==
created_at: 2020-07-27T05:29:48Z
updated_at: 2020-07-27T05:29:48Z
author_association: OWNER
body:
  Test command:
  ```
  sqlite-utils insert-files gifs.db *.gif \
      -c filename:filename \
      -c filepath:filepath \
      -c absolutepath:absolutepath \
      -c sha256:sha256 \
      -c md5:md5 \
      -c content:content \
      -c mtime:mtime \
      -c ctime:ctime \
      -c mtime_iso:mtime_iso \
      -c ctime_iso:ctime_iso \
      -c size:size \
      --pk absolutepath
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/122
node_id: MDEyOklzc3VlQ29tbWVudDY2NDEyODA3MQ==
created_at: 2020-07-27T05:30:54Z
updated_at: 2020-07-27T05:30:54Z
author_association: OWNER
body:
  Inserting files by piping them in should work - but since a filename cannot be derived this will need a `--name blah.gif` option.

      cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/127#issuecomment-664163206
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/127
node_id: MDEyOklzc3VlQ29tbWVudDY2NDE2MzIwNg==
created_at: 2020-07-27T07:10:05Z
updated_at: 2020-07-27T07:10:05Z
author_association: OWNER
body:
  I tried to get this working but it was a bit tricky because `-` doesn't behave like a regular `pathlib.Path` - needs a bit more thought on how the implementation would work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664163524
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/122
node_id: MDEyOklzc3VlQ29tbWVudDY2NDE2MzUyNA==
created_at: 2020-07-27T07:10:41Z
updated_at: 2020-07-27T07:10:41Z
author_association: OWNER
body:
  Docs: https://github.com/simonw/sqlite-utils/blob/ebc802f7ff0e640b6ae11ea525290fea0115228c/docs/cli.rst#inserting-binary-data-from-files

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/128#issuecomment-664683608
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/128
node_id: MDEyOklzc3VlQ29tbWVudDY2NDY4MzYwOA==
created_at: 2020-07-27T23:09:22Z
updated_at: 2020-07-27T23:09:22Z
author_association: OWNER
body:
  This seems to work, but needs more tests:
  ```diff
  diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
  index d6b9ecf..ee26433 100644
  --- a/sqlite_utils/db.py
  +++ b/sqlite_utils/db.py
  @@ -7,6 +7,7 @@ import itertools
   import json
   import os
   import pathlib
  +import uuid
   
   SQLITE_MAX_VARS = 999
   
  @@ -40,11 +41,13 @@ COLUMN_TYPE_MAPPING = {
       str: "TEXT",
       bytes.__class__: "BLOB",
       bytes: "BLOB",
  +    memoryview: "BLOB",
       datetime.datetime: "TEXT",
       datetime.date: "TEXT",
       datetime.time: "TEXT",
       decimal.Decimal: "FLOAT",
       None.__class__: "TEXT",
  +    uuid.UUID: "TEXT",
       # SQLite explicit types
       "TEXT": "TEXT",
       "INTEGER": "INTEGER",
  @@ -1336,6 +1339,8 @@ def jsonify_if_needed(value):
           return json.dumps(value, default=repr)
       elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)):
           return value.isoformat()
  +    elif isinstance(value, uuid.UUID):
  +        return str(value)
       else:
           return value
   ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/910#issuecomment-665663131
issue_url: https://api.github.com/repos/simonw/datasette/issues/910
node_id: MDEyOklzc3VlQ29tbWVudDY2NTY2MzEzMQ==
created_at: 2020-07-29T13:26:14Z
updated_at: 2020-07-29T13:26:14Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=h1) Report
  > Merging [#910](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/3c33b421320c0be81a625ca7307b2e4416a9ed5b&el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/910/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##           master     #910   +/-   ##
  =======================================
    Coverage   83.55%   83.55%           
  =======================================
    Files          27       27           
    Lines        3666     3666           
  =======================================
    Hits         3063     3063           
    Misses        603      603           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=footer). Last update [3c33b42...3493915](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/909#issuecomment-665854704
issue_url: https://api.github.com/repos/simonw/datasette/issues/909
node_id: MDEyOklzc3VlQ29tbWVudDY2NTg1NDcwNA==
created_at: 2020-07-29T19:22:31Z
updated_at: 2020-07-29T19:22:31Z
author_association: OWNER
body:
  I think this results in a bug where the "download database" link doesn't include the correct filename: https://github.com/simonw/datasette/blob/549b1c2063db48c4622ee5c7b478a1e3cbc1ac07/datasette/views/database.py#L110-L131

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/909#issuecomment-666010395
issue_url: https://api.github.com/repos/simonw/datasette/issues/909
node_id: MDEyOklzc3VlQ29tbWVudDY2NjAxMDM5NQ==
created_at: 2020-07-30T00:56:17Z
updated_at: 2020-07-30T00:56:17Z
author_association: OWNER
body:
  ```
  $ curl -I https://latest.datasette.io/fixtures.db
  HTTP/1.1 200 OK
  content-disposition: attachment; filename="fixtures.db"
  content-type: application/octet-stream
  Date: Thu, 30 Jul 2020 00:56:05 GMT
  Server: Google Frontend
  Transfer-Encoding: chunked
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/129#issuecomment-666046819
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/129
node_id: MDEyOklzc3VlQ29tbWVudDY2NjA0NjgxOQ==
created_at: 2020-07-30T02:28:34Z
updated_at: 2020-07-30T02:28:34Z
author_association: OWNER
body:
  This code looks useful as inspiration: https://github.com/j4mie/sqlsite/blob/f2dadb8db5ed7880f8872b6591d8cb1487f777ea/sqlsite/sqlar.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/127#issuecomment-666047928
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/127
node_id: MDEyOklzc3VlQ29tbWVudDY2NjA0NzkyOA==
created_at: 2020-07-30T02:31:05Z
updated_at: 2020-07-30T02:31:05Z
author_association: OWNER
body:
  Maybe could do this using an improved version of this lambda? Could teach it to look for `-` and read from `sys.stdin` if it sees it.

  https://github.com/simonw/sqlite-utils/blob/710454d72aed5094573e642344fd075a0ef5372c/sqlite_utils/cli.py#L839

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/127#issuecomment-666063689
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/127
node_id: MDEyOklzc3VlQ29tbWVudDY2NjA2MzY4OQ==
created_at: 2020-07-30T03:08:51Z
updated_at: 2020-07-30T03:08:51Z
author_association: OWNER
body:
  Documentation at the bottom of this section: https://github.com/simonw/sqlite-utils/blob/8fe1e6d1be021aeeb8f08b0f77f03b75a83b6f75/docs/cli.rst#inserting-binary-data-from-files

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/129#issuecomment-666752039
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/129
node_id: MDEyOklzc3VlQ29tbWVudDY2Njc1MjAzOQ==
created_at: 2020-07-30T22:40:55Z
updated_at: 2020-07-30T22:40:55Z
author_association: OWNER
body:
  This should be a separate command from `insert-files`. SQLite Archives should use a table with this schema:
  ```sql
  CREATE TABLE sqlar(
    name TEXT PRIMARY KEY,  -- name of the file
    mode INT,               -- access permissions
    mtime INT,              -- last modification time
    sz INT,                 -- original file size
    data BLOB               -- compressed content
  );
  ```
  `insert-files` currently treats the table name as a required argument - but it's not necessary for this table. Also there shouldn't be any support for the `--column` option.

  So if I write this command it should be this instead:

      sqlite-utils sqlar files.db file.txt file2.txt

  But at that point, why bother? Users can use `sqlite3 files.db -Ac *.txt` instead.

  So I'm not going to bother implementing this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/849#issuecomment-667295759
issue_url: https://api.github.com/repos/simonw/datasette/issues/849
node_id: MDEyOklzc3VlQ29tbWVudDY2NzI5NTc1OQ==
created_at: 2020-07-31T18:45:35Z
updated_at: 2020-07-31T18:45:35Z
author_association: OWNER
body:
  Watch out for places in the documentation that might link to `master` - e.g. here:

  https://github.com/simonw/datasette/blob/2d7fa8b9058dfbf9c7c371cdeec115d32a177dc9/docs/custom_templates.rst#L247

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/849#issuecomment-667424020
issue_url: https://api.github.com/repos/simonw/datasette/issues/849
node_id: MDEyOklzc3VlQ29tbWVudDY2NzQyNDAyMA==
created_at: 2020-07-31T23:21:30Z
updated_at: 2020-07-31T23:21:30Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/tree/main branch now exists and will automatically mirror master (and vice-versa).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/849#issuecomment-667424128
issue_url: https://api.github.com/repos/simonw/datasette/issues/849
node_id: MDEyOklzc3VlQ29tbWVudDY2NzQyNDEyOA==
created_at: 2020-07-31T23:21:56Z
updated_at: 2020-07-31T23:23:24Z
author_association: OWNER
body:
  I'm going to change the default branch on the GitHub repository. If something breaks I can always change it back again.

  Done that! Default is now `main`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/913#issuecomment-667429616
issue_url: https://api.github.com/repos/simonw/datasette/issues/913
node_id: MDEyOklzc3VlQ29tbWVudDY2NzQyOTYxNg==
created_at: 2020-07-31T23:48:25Z
updated_at: 2020-07-31T23:49:59Z
author_association: OWNER
body:
  I could let plugins add additional options to `datasette serve` - but what if two plugins both try to register an option with the same name?

  A better solution could be to use the existing `--config` option - and allow plugins to register their own, namespaced config options. So you could do things like:

      datasette my.db --config datasette-insert:unsafe:1

  Maybe even drop the `datasette-` prefix?

      datasette my.db --config insert:unsafe:1

  I think I prefer keeping the prefix to be honest - it makes it more obvious that this is a setting which comes from a specific named plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/913#issuecomment-667429690
issue_url: https://api.github.com/repos/simonw/datasette/issues/913
node_id: MDEyOklzc3VlQ29tbWVudDY2NzQyOTY5MA==
created_at: 2020-07-31T23:48:48Z
updated_at: 2020-07-31T23:48:48Z
author_association: OWNER
body:
  Here's the code in Datasette that parses `--config` options at the moment:

  https://github.com/simonw/datasette/blob/7ca8c0521ac1ea48a3cd8d0fe9275d1316e54b43/datasette/cli.py#L25-L40

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/913#issuecomment-667430352
issue_url: https://api.github.com/repos/simonw/datasette/issues/913
node_id: MDEyOklzc3VlQ29tbWVudDY2NzQzMDM1Mg==
created_at: 2020-07-31T23:52:10Z
updated_at: 2020-07-31T23:52:10Z
author_association: OWNER
body:
  The bigger question here is when this mechanism should be used in place of `metadata.json` or `metadata.yml`. Especially since I'm already considering renaming or reworking that mechanism since plugin configuration has nothing to do with database metadata: #493

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/899#issuecomment-667430790
issue_url: https://api.github.com/repos/simonw/datasette/issues/899
node_id: MDEyOklzc3VlQ29tbWVudDY2NzQzMDc5MA==
created_at: 2020-07-31T23:54:40Z
updated_at: 2020-07-31T23:54:40Z
author_association: OWNER
body:
  There's no mechanism that can do this at the moment.

  You could absolutely support this with a plugin, probably using the `asgi_wrapper` plugin hook. There's an existing package at https://pypi.org/project/asgi-ratelimit/ which may be usable for this - it may even be possible to configure that using https://github.com/simonw/datasette-configure-asgi rather than using it to write a custom plugin.

  Using a separate revers proxy would also be a good way to solve this. It depends which option would work best in your environment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667584567
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/130
node_id: MDEyOklzc3VlQ29tbWVudDY2NzU4NDU2Nw==
created_at: 2020-08-01T20:41:09Z
updated_at: 2020-08-01T20:41:09Z
author_association: OWNER
body:
  API documentation here: https://github.com/simonw/sqlite-utils/commit/617e6f070c85be66ea04c80b78dafd08c875f8c8?short_path=e156262#diff-e1562629b8def6da772d9b0903faf703

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667585598
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/130
node_id: MDEyOklzc3VlQ29tbWVudDY2NzU4NTU5OA==
created_at: 2020-08-01T20:51:28Z
updated_at: 2020-08-01T20:51:28Z
author_association: OWNER
body:
  CLI documentation: https://github.com/simonw/sqlite-utils/commit/57e4eb8e5564af5d97f892b3be8342451ee177a2?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-669241341
issue_url: https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8
node_id: MDEyOklzc3VlQ29tbWVudDY2OTI0MTM0MQ==
created_at: 2020-08-05T14:55:14Z
updated_at: 2020-08-05T14:55:14Z
author_association: MEMBER
body:
  Looks like there's a column that's not consistently there, so the table got created without it.

  Easiest fix is to add `alter=True` to this line:

  https://github.com/dogsheep/swarm-to-sqlite/blob/f4a82633da927cde672c9d9af92930bfca2e3ddf/swarm_to_sqlite/utils.py#L94

  That will cause `sqlite-utils` to notice if there's a missing column and add it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/335#issuecomment-670999832
issue_url: https://api.github.com/repos/simonw/datasette/issues/335
node_id: MDEyOklzc3VlQ29tbWVudDY3MDk5OTgzMg==
created_at: 2020-08-09T03:12:14Z
updated_at: 2020-08-09T03:12:14Z
author_association: OWNER
body:
  Another useful example: https://github.com/Homebrew/homebrew-core/blob/master/Formula/trailscraper.rb

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/335#issuecomment-671001457
issue_url: https://api.github.com/repos/simonw/datasette/issues/335
node_id: MDEyOklzc3VlQ29tbWVudDY3MTAwMTQ1Nw==
created_at: 2020-08-09T03:37:39Z
updated_at: 2020-08-09T03:37:39Z
author_association: OWNER
body:
  Here's what happened when I installed `homebrew-vd`: https://gist.github.com/simonw/7bfd971a62743d7ca248e6b5e696c240

  It worked! And from digging around, it has a virtual environment at `/usr/local/Cellar/visidata/1.5.2/libexec/`

  Which means `/usr/local/Cellar/visidata/1.5.2/libexec/bin/pip` is a working `pip`

  And I tried running these commands and confirmed that I get a `datasette` with an additional plugin:

  ```
  /usr/local/Cellar/visidata/1.5.2/libexec/bin/pip install datasette
  /usr/local/Cellar/visidata/1.5.2/libexec/bin/pip install datasette-graphql
  /usr/local/Cellar/visidata/1.5.2/libexec/bin/datasette plugins
  [
      {
          "name": "datasette-graphql",
          "static": false,
          "templates": true,
          "version": "0.11",
          "hooks": [
              "register_routes",
              "startup"
          ]
      }
  ]
  ```
  So I can package Datasette as a homebrew package AND I can give people instructions for installing plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/335#issuecomment-671005731
issue_url: https://api.github.com/repos/simonw/datasette/issues/335
node_id: MDEyOklzc3VlQ29tbWVudDY3MTAwNTczMQ==
created_at: 2020-08-09T04:44:13Z
updated_at: 2020-08-09T17:04:21Z
author_association: OWNER
body:
  Telling people how to figure out that `pip` location is going to be pretty unpleasant.

  How about instead providing a `datasette plugins --install=datasette-graphql` command?

  Or `datasette install datasette-vega`

  It would run `pip install` in the same virtualenv as Datasette itself.

  http://jelly.codes/articles/python-pip-module/ shows how to do this:

  ```python

  import pip

  try:
      pip.main(["install", "plumbum"])
  except SystemExit as e:
      pass
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/918#issuecomment-671070486
issue_url: https://api.github.com/repos/simonw/datasette/issues/918
node_id: MDEyOklzc3VlQ29tbWVudDY3MTA3MDQ4Ng==
created_at: 2020-08-09T16:11:59Z
updated_at: 2020-08-09T16:11:59Z
author_association: OWNER
body:
  Fix has been released in Datasette 0.46: https://datasette.readthedocs.io/en/latest/changelog.html#v0-46

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/918#issuecomment-671070528
issue_url: https://api.github.com/repos/simonw/datasette/issues/918
node_id: MDEyOklzc3VlQ29tbWVudDY3MTA3MDUyOA==
created_at: 2020-08-09T16:12:16Z
updated_at: 2020-08-09T16:12:16Z
author_association: OWNER
body:
  It's worth noting that in order to exploit this issue the following would all need to be true:

  - A user is running a copy of Datasette protected by a cookie-based authentication plugin AND configured with at least one writable canned query
  - An attacker is in control of a URL that could concievably be returned on a page that is displayed as the result of submitting a read-only canned query
  - An authenticated user of that Datasette instance, who is running a browser that doesn't support the `SameSite=lax` cookie parameter (which is [widely supported](https://caniuse.com/#feat=same-site-cookie-attribute) by modern browsers), submits the read-only canned query form and then clicks a link to the attacker's off-site page, exposing their CSRFToken in the attacker's HTTP referer logs
  - The attacker then tricks that user into visiting their own malicious web page which includes a POST form that auto-submits against the writable canned query that the attacker wishes to exploit, including the CSRF token as a hidden field

  The attacker would need full knowledge of the URL and form layout of the Datasette instance that they are exploiting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/918#issuecomment-671071710
issue_url: https://api.github.com/repos/simonw/datasette/issues/918
node_id: MDEyOklzc3VlQ29tbWVudDY3MTA3MTcxMA==
created_at: 2020-08-09T16:21:41Z
updated_at: 2020-08-09T16:21:41Z
author_association: OWNER
body:
  Submitting the form on https://latest.datasette.io/fixtures/neighborhood_search demonstrates that this is fixed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/919#issuecomment-671072223
issue_url: https://api.github.com/repos/simonw/datasette/issues/919
node_id: MDEyOklzc3VlQ29tbWVudDY3MTA3MjIyMw==
created_at: 2020-08-09T16:26:17Z
updated_at: 2020-08-09T16:26:17Z
author_association: OWNER
body:
  Should be released in a couple of minutes: https://travis-ci.org/github/simonw/datasette/builds/716328883

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/915#issuecomment-671073223
issue_url: https://api.github.com/repos/simonw/datasette/issues/915
node_id: MDEyOklzc3VlQ29tbWVudDY3MTA3MzIyMw==
created_at: 2020-08-09T16:35:20Z
updated_at: 2020-08-09T16:36:10Z
author_association: OWNER
body:
  `datasette-graphql` uses the logic from `TableView` right now. It wasn't too unpleasant, but I do worry about the two of them being coupled together in this way.

  https://github.com/simonw/datasette-graphql/blob/cc65ec294b0bf8e26213fc68bb5487066de9caab/datasette_graphql/utils.py#L412-L417

  ```python
          request = Request.fake(path_with_query_string)

          view = TableView(DatasetteSpecialConfig(datasette))
          data, _, _ = await view.data(
              request, database=database_name, hash=None, table=table_name, _next=after
          )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/918#issuecomment-671075764
issue_url: https://api.github.com/repos/simonw/datasette/issues/918
node_id: MDEyOklzc3VlQ29tbWVudDY3MTA3NTc2NA==
created_at: 2020-08-09T16:56:48Z
updated_at: 2020-08-09T16:56:48Z
author_association: OWNER
body:
  GitHub security advisory: https://github.com/simonw/datasette/security/advisories/GHSA-q6j3-c4wc-63vw

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/335#issuecomment-671076975
issue_url: https://api.github.com/repos/simonw/datasette/issues/335
node_id: MDEyOklzc3VlQ29tbWVudDY3MTA3Njk3NQ==
created_at: 2020-08-09T17:08:34Z
updated_at: 2020-08-09T17:09:21Z
author_association: OWNER
body:
  Quick prototype of `datasette install`:
  ```diff
  diff --git a/datasette/cli.py b/datasette/cli.py
  index 287195a..95b6eb7 100644
  --- a/datasette/cli.py
  +++ b/datasette/cli.py
  @@ -231,6 +231,18 @@ def package(
           call(args)
   
   
  +@cli.command()
  +@click.argument("packages", nargs=-1, required=True)
  +def install(packages):
  +    "Install Python packages - e.g. Datasette plugins - into the same environment as Datasett"
  +    import pip
  +
  +    try:
  +        pip.main(["install"] + list(packages))
  +    except SystemExit as e:
  +        pass
  +
  +
   @cli.command()
   @click.argument("files", type=click.Path(exists=True), nargs=-1)
   @click.option(
  ```

  ```
  $ datasette install
  Usage: datasette install [OPTIONS] PACKAGES...
  Try 'datasette install --help' for help.

  Error: Missing argument 'PACKAGES...'.
  $ datasette install datasette-vega
  WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
  Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
  To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
  Collecting datasette-vega
    Using cached datasette_vega-0.6.2-py3-none-any.whl (1.8 MB)
  ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/335#issuecomment-671077168
issue_url: https://api.github.com/repos/simonw/datasette/issues/335
node_id: MDEyOklzc3VlQ29tbWVudDY3MTA3NzE2OA==
created_at: 2020-08-09T17:10:15Z
updated_at: 2020-08-09T18:13:39Z
author_association: OWNER
body:
  Here's the issue that explains that warning: https://github.com/pypa/pip/issues/5599

  This should fix it (risky):

      from pip._internal.cli.main import main

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/131#issuecomment-671088832
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/131
node_id: MDEyOklzc3VlQ29tbWVudDY3MTA4ODgzMg==
created_at: 2020-08-09T19:00:41Z
updated_at: 2020-08-09T19:00:41Z
author_association: OWNER
body:
  Should be consistent with the `create-table` command as much as possible:
  ```
  $ sqlite-utils create-table mydb.db mytable \
      id integer \
      name text \
      age integer \
      is_good integer \
      --not-null name \
      --not-null age \
      --default is_good 1 \
      --pk=id
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671146948
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/132
node_id: MDEyOklzc3VlQ29tbWVudDY3MTE0Njk0OA==
created_at: 2020-08-10T03:26:51Z
updated_at: 2020-08-10T03:26:51Z
author_association: OWNER
body:
  For the CLI:

      $ sqlite-utils enable-wal github.db
      $ sqlite-utils disable-wal github.db

  For the Python library:

  ```python
  import sqlite_utils

  db = sqlite_utils.Database("github.db")
  db.enable_wal()
  db.disable_wal()
  mode = db.journal_mode # "wal" or "delete" or others
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671147148
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/132
node_id: MDEyOklzc3VlQ29tbWVudDY3MTE0NzE0OA==
created_at: 2020-08-10T03:27:50Z
updated_at: 2020-08-10T03:27:50Z
author_association: OWNER
body:
  https://www.sqlite.org/pragma.html#pragma_journal_mode lists six modes: DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF

  I'm only going to implement utilities for DELETE (wal-off) and WAL (wal-on) - the other modes look like they're for specialist purposes that I don't need to support.

  If it turns out I do need them I can add those to `sqlite-utils` later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671147344
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/132
node_id: MDEyOklzc3VlQ29tbWVudDY3MTE0NzM0NA==
created_at: 2020-08-10T03:29:00Z
updated_at: 2020-08-10T03:29:00Z
author_association: OWNER
body:
  The CLI options should take multiple database files:

      $ sqlite-utils enable-wal *.db

  It's possible for this to fail if the DB is locked. How about a `--retry` option that causes it to retry a bunch of times if that happens?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671151170
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/132
node_id: MDEyOklzc3VlQ29tbWVudDY3MTE1MTE3MA==
created_at: 2020-08-10T03:52:02Z
updated_at: 2020-08-10T03:52:02Z
author_association: OWNER
body:
  I'm having trouble figuring out how to write a test that locks a SQLite database (so I can test that `--retry` actually works). I tried this recipe but it didn't seem to prevent another process from running `pragma journal_mode='wal';` against that database:

  ```python
  import time
  import sys
  import sqlite3

  filename = sys.argv[-1]

  db = sqlite3.connect(filename)
  with db:
      db.execute("create table if not exists counter(id integer primary key, counter text)")
      time.sleep(100)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/335#issuecomment-671733187
issue_url: https://api.github.com/repos/simonw/datasette/issues/335
node_id: MDEyOklzc3VlQ29tbWVudDY3MTczMzE4Nw==
created_at: 2020-08-11T05:25:23Z
updated_at: 2020-08-11T05:25:23Z
author_association: OWNER
body:
  I got this almost working in `simonw/homebrew-datasette` - see https://github.com/simonw/homebrew-datasette/issues/2 for the last missing detail.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/335#issuecomment-672088880
issue_url: https://api.github.com/repos/simonw/datasette/issues/335
node_id: MDEyOklzc3VlQ29tbWVudDY3MjA4ODg4MA==
created_at: 2020-08-11T16:54:06Z
updated_at: 2020-08-11T16:54:06Z
author_association: OWNER
body:
  It works!
  ```
  $ brew tap simonw/datasette
  $ brew install simonw/datasette/datasette
  $ datasette --version
  datasette, version 0.46
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/923#issuecomment-672287754
issue_url: https://api.github.com/repos/simonw/datasette/issues/923
node_id: MDEyOklzc3VlQ29tbWVudDY3MjI4Nzc1NA==
created_at: 2020-08-11T21:25:33Z
updated_at: 2020-08-11T21:25:33Z
author_association: OWNER
body:
  .. and confirm if `brew tap ...` is even needed if you run `brew install simonw/datasette/datasette`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/923#issuecomment-672288845
issue_url: https://api.github.com/repos/simonw/datasette/issues/923
node_id: MDEyOklzc3VlQ29tbWVudDY3MjI4ODg0NQ==
created_at: 2020-08-11T21:28:17Z
updated_at: 2020-08-11T21:28:17Z
author_association: OWNER
body:
  Here's a pattern for installing plugins:
  ```
  $ datasette plugins
  []
  $ /usr/local/opt/datasette/libexec/bin/pip install datasette-vega
  Collecting datasette-vega
    Using cached datasette_vega-0.6.2-py3-none-any.whl (1.8 MB)
  Requirement already satisfied: datasette in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette-vega) (0.46)
  Requirement already satisfied: click~=7.1.1 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (7.1.2)
  Requirement already satisfied: click-default-group~=1.2.2 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.2.2)
  Requirement already satisfied: Jinja2<2.12.0,>=2.10.3 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (2.11.2)
  Requirement already satisfied: hupper~=1.9 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.10.2)
  Requirement already satisfied: pint~=0.9 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.14)
  Requirement already satisfied: pluggy~=0.13.0 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.13.1)
  Requirement already satisfied: uvicorn~=0.11 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.11.8)
  Requirement already satisfied: aiofiles<0.6,>=0.4 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.5.0)
  Requirement already satisfied: janus<0.6,>=0.4 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.5.0)
  Requirement already satisfied: asgi-csrf>=0.6 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.6.1)
  Requirement already satisfied: PyYAML~=5.3 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (5.3.1)
  Requirement already satisfied: mergedeep<1.4.0,>=1.1.1 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.3.0)
  Requirement already satisfied: itsdangerous~=1.1 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.1.0)
  Requirement already satisfied: python-baseconv==1.2.2 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.2.2)
  Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from Jinja2<2.12.0,>=2.10.3->datasette->datasette-vega) (1.1.1)
  Requirement already satisfied: setuptools in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from pint~=0.9->datasette->datasette-vega) (49.3.1)
  Requirement already satisfied: packaging in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from pint~=0.9->datasette->datasette-vega) (20.4)
  Requirement already satisfied: h11<0.10,>=0.8 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (0.9.0)
  Requirement already satisfied: websockets==8.* in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (8.1)
  Requirement already satisfied: httptools==0.1.* in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (0.1.1)
  Requirement already satisfied: uvloop>=0.14.0 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (0.14.0)
  Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from packaging->pint~=0.9->datasette->datasette-vega) (2.4.7)
  Requirement already satisfied: six in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from packaging->pint~=0.9->datasette->datasette-vega) (1.15.0)
  Installing collected packages: datasette-vega
  Successfully installed datasette-vega-0.6.2
  $ datasette plugins                                              
  [
      {
          "name": "datasette-vega",
          "static": true,
          "templates": false,
          "version": "0.6.2",
          "hooks": [
              "extra_css_urls",
              "extra_js_urls"
          ]
      }
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/925#issuecomment-672328436
issue_url: https://api.github.com/repos/simonw/datasette/issues/925
node_id: MDEyOklzc3VlQ29tbWVudDY3MjMyODQzNg==
created_at: 2020-08-11T22:33:32Z
updated_at: 2020-08-11T22:33:42Z
author_association: OWNER
body:
  ```
  $ datasette install --help
  Usage: datasette install [OPTIONS] PACKAGES...

    Install Python packages - e.g. Datasette plugins - into the same
    environment as Datasette

  Options:
    --help  Show this message and exit.
  $ datasette uninstall --help
  Usage: datasette uninstall [OPTIONS] PACKAGES...

    Uninstall Python packages (e.g. plugins) from the Datasette environment

  Options:
    -y, --yes  Don't ask for confirmation
    --help     Show this message and exit.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/925#issuecomment-672328807
issue_url: https://api.github.com/repos/simonw/datasette/issues/925
node_id: MDEyOklzc3VlQ29tbWVudDY3MjMyODgwNw==
created_at: 2020-08-11T22:34:37Z
updated_at: 2020-08-11T22:34:37Z
author_association: OWNER
body:
  This will simplify the instructions for installing plugins with Datasette install via homebrew, refs #923

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/923#issuecomment-672329101
issue_url: https://api.github.com/repos/simonw/datasette/issues/923
node_id: MDEyOklzc3VlQ29tbWVudDY3MjMyOTEwMQ==
created_at: 2020-08-11T22:35:13Z
updated_at: 2020-08-11T22:35:13Z
author_association: OWNER
body:
  I added the `datasette install name-of-plugin` command in #925 mainly to simplify the process of installing plugins if Datasette itself was installed using homebrew.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/923#issuecomment-672336720
issue_url: https://api.github.com/repos/simonw/datasette/issues/923
node_id: MDEyOklzc3VlQ29tbWVudDY3MjMzNjcyMA==
created_at: 2020-08-11T22:53:07Z
updated_at: 2020-08-11T22:53:07Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/5126ecb1267ed3850bf3b0ab270accd031a02e79/docs/installation.rst#using-homebrew

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/926#issuecomment-672338113
issue_url: https://api.github.com/repos/simonw/datasette/issues/926
node_id: MDEyOklzc3VlQ29tbWVudDY3MjMzODExMw==
created_at: 2020-08-11T22:57:28Z
updated_at: 2020-08-11T22:57:28Z
author_association: OWNER
body:
  I partly want this so I can easily implement a better `test` method for the Homebrew package. The test I have right now looks like this:

  https://github.com/simonw/homebrew-datasette/blob/8aa30aa183158051a987a7e3f50e7e3ee05d8ee9/Formula/datasette.rb#L125-L127

  ```
    test do
      system bin/"datasette", "--help"
    end
  ```
  The Homebrew docs at https://docs.brew.sh/Formula-Cookbook#add-a-test-to-the-formula say:

  > We want tests that don't require any user input and test the basic functionality of the application. For example `foo build-foo input.foo` is a good test and (despite their widespread use) `foo --version` and `foo --help` are bad tests. However, a bad test is better than no test at all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/927#issuecomment-672357902
issue_url: https://api.github.com/repos/simonw/datasette/issues/927
node_id: MDEyOklzc3VlQ29tbWVudDY3MjM1NzkwMg==
created_at: 2020-08-11T23:32:39Z
updated_at: 2020-08-11T23:32:39Z
author_association: OWNER
body:
  It works:
  ```
  $ datasette --get '/:memory:.json?sql=select * from sqlite_master' | jq .                            
  {
    "database": ":memory:",
    "query_name": null,
    "rows": [],
    "truncated": false,
    "columns": [
      "type",
      "name",
      "tbl_name",
      "rootpage",
      "sql"
    ],
    "query": {
      "sql": "select * from sqlite_master",
      "params": {}
    },
    "private": false,
    "allow_execute_sql": true,
    "query_ms": 0.8032321929931641
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/928#issuecomment-672372465
issue_url: https://api.github.com/repos/simonw/datasette/issues/928
node_id: MDEyOklzc3VlQ29tbWVudDY3MjM3MjQ2NQ==
created_at: 2020-08-11T23:54:28Z
updated_at: 2020-08-11T23:54:28Z
author_association: OWNER
body:
  While debugging this I found a useful clue in https://github.com/pypa/pip/blob/e060970d51c5946beac8447eb95585d83019582d/src/pip/_internal/cli/main.py#L23-L47

  ```
  # Do not import and use main() directly! Using it directly is actively
  # discouraged by pip's maintainers. The name, location and behavior of
  # this function is subject to change, so calling it directly is not
  # portable across different pip versions.

  # In addition, running pip in-process is unsupported and unsafe. This is
  # elaborated in detail at
  # https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program.
  # That document also provides suggestions that should work for nearly
  # all users that are considering importing and using main() directly.

  # However, we know that certain users will still want to invoke pip
  # in-process. If you understand and accept the implications of using pip
  # in an unsupported manner, the best approach is to use runpy to avoid
  # depending on the exact location of this entry point.

  # The following example shows how to use runpy to invoke pip in that
  # case:
  #
  #     sys.argv = ["pip", your, args, here]
  #     runpy.run_module("pip", run_name="__main__")
  #
  # Note that this will exit the process after running, unlike a direct
  # call to main. As it is not safe to do any processing after calling
  # main, this should not be an issue in practice.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/928#issuecomment-672373061
issue_url: https://api.github.com/repos/simonw/datasette/issues/928
node_id: MDEyOklzc3VlQ29tbWVudDY3MjM3MzA2MQ==
created_at: 2020-08-11T23:56:19Z
updated_at: 2020-08-11T23:56:19Z
author_association: OWNER
body:
  New implementation of the `install` command:
  https://github.com/simonw/datasette/blob/afdeda8216d4d3027f87583ccdbef17ad85022ef/datasette/cli.py#L235-L240

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/927#issuecomment-672375214
issue_url: https://api.github.com/repos/simonw/datasette/issues/927
node_id: MDEyOklzc3VlQ29tbWVudDY3MjM3NTIxNA==
created_at: 2020-08-12T00:01:23Z
updated_at: 2020-08-12T00:23:45Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=h1) Report
  > Merging [#927](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/83eda049af3f38d4289118d3576f96b2535084b1&el=desc) will **increase** coverage by `0.39%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/927/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main     #927      +/-   ##
  ==========================================
  + Coverage   83.62%   84.02%   +0.39%     
  ==========================================
    Files          27       28       +1     
    Lines        3682     3774      +92     
  ==========================================
  + Hits         3079     3171      +92     
    Misses        603      603              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/927/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `75.13% <100.00%> (+1.00%)` | :arrow_up: |
  | [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/927/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `100.00% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=footer). Last update [83eda04...2111da0](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/927#issuecomment-672382108
issue_url: https://api.github.com/repos/simonw/datasette/issues/927
node_id: MDEyOklzc3VlQ29tbWVudDY3MjM4MjEwOA==
created_at: 2020-08-12T00:09:18Z
updated_at: 2020-08-12T00:09:18Z
author_association: OWNER
body:
  Documentation can go here: https://datasette.readthedocs.io/en/latest/getting_started.html#datasette-serve-options

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/927#issuecomment-672391299
issue_url: https://api.github.com/repos/simonw/datasette/issues/927
node_id: MDEyOklzc3VlQ29tbWVudDY3MjM5MTI5OQ==
created_at: 2020-08-12T00:19:20Z
updated_at: 2020-08-12T00:19:20Z
author_association: OWNER
body:
  Docs: https://github.com/simonw/datasette/blob/2111da01a03cfc62303b6a4b59ea9f96d22c0f78/docs/getting_started.rst#datasette---get

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/926#issuecomment-672393737
issue_url: https://api.github.com/repos/simonw/datasette/issues/926
node_id: MDEyOklzc3VlQ29tbWVudDY3MjM5MzczNw==
created_at: 2020-08-12T00:26:17Z
updated_at: 2020-08-12T00:26:17Z
author_association: OWNER
body:
  ```
  $ datasette --get '/:memory:.json?sql=select+sqlite_version()' | jq .
  {
    "database": ":memory:",
    "query_name": null,
    "rows": [
      [
        "3.32.3"
      ]
    ],
    "truncated": false,
    "columns": [
      "sqlite_version()"
    ],
    "query": {
      "sql": "select sqlite_version()",
      "params": {}
    },
    "private": false,
    "allow_execute_sql": true,
    "query_ms": 1.165628433227539
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/930#issuecomment-672472518
issue_url: https://api.github.com/repos/simonw/datasette/issues/930
node_id: MDEyOklzc3VlQ29tbWVudDY3MjQ3MjUxOA==
created_at: 2020-08-12T02:24:49Z
updated_at: 2020-08-12T02:24:49Z
author_association: OWNER
body:
  I checked and `datasette publish` deploys a 0.47 version that works fine too: https://datasette-0-47-j7hipcg4aq-uc.a.run.app/-/versions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/930#issuecomment-672480811
issue_url: https://api.github.com/repos/simonw/datasette/issues/930
node_id: MDEyOklzc3VlQ29tbWVudDY3MjQ4MDgxMQ==
created_at: 2020-08-12T02:31:38Z
updated_at: 2020-08-12T02:31:38Z
author_association: OWNER
body:
  The root cause appears to be that the `templates/` are missing from the `sdist` `.tar.gz`: https://github.com/simonw/homebrew-datasette/issues/6#issuecomment-672477352

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/930#issuecomment-672488293
issue_url: https://api.github.com/repos/simonw/datasette/issues/930
node_id: MDEyOklzc3VlQ29tbWVudDY3MjQ4ODI5Mw==
created_at: 2020-08-12T02:35:34Z
updated_at: 2020-08-12T02:35:34Z
author_association: OWNER
body:
  OK, this has fixed it - I used `sdist` to create the `.tar.gz` and then `pip installed` it into a new environment, and everything worked just fine. Going to ship this as Datasette 0.47.1.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/930#issuecomment-672519787
issue_url: https://api.github.com/repos/simonw/datasette/issues/930
node_id: MDEyOklzc3VlQ29tbWVudDY3MjUxOTc4Nw==
created_at: 2020-08-12T02:52:46Z
updated_at: 2020-08-12T02:52:46Z
author_association: OWNER
body:
  Homebrew install is now fixed, using a temporary URL while waiting for Travis to ship the new release to PyPI: https://github.com/simonw/homebrew-datasette/issues/6

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/931#issuecomment-673068327
issue_url: https://api.github.com/repos/simonw/datasette/issues/931
node_id: MDEyOklzc3VlQ29tbWVudDY3MzA2ODMyNw==
created_at: 2020-08-12T19:33:42Z
updated_at: 2020-08-12T19:33:42Z
author_association: OWNER
body:
  https://hub.docker.com/r/datasetteproject/datasette/tags shows this:

  <img width="1282" alt="datasetteproject_datasette_Tags_-_Docker_Hub" src="https://user-images.githubusercontent.com/9599/90059445-0b3e6680-dc98-11ea-9e21-c473b6e9b55d.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/931#issuecomment-673068919
issue_url: https://api.github.com/repos/simonw/datasette/issues/931
node_id: MDEyOklzc3VlQ29tbWVudDY3MzA2ODkxOQ==
created_at: 2020-08-12T19:34:57Z
updated_at: 2020-08-12T19:34:57Z
author_association: OWNER
body:
  Looks like SpatiaLite released new versions: https://www.gaia-gis.it/fossil/freexl/index says "Sources: current version is 1.0.6 (released on 2020-08-02)" and links to http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/931#issuecomment-673070308
issue_url: https://api.github.com/repos/simonw/datasette/issues/931
node_id: MDEyOklzc3VlQ29tbWVudDY3MzA3MDMwOA==
created_at: 2020-08-12T19:37:55Z
updated_at: 2020-08-12T19:37:55Z
author_association: OWNER
body:
  Project news here: https://groups.google.com/g/spatialite-users/c/8AG3hQzXDmo

  > FreeXL
  > ==============================
  > the current stable version is now 1.0.6
  > (supporting few minor bug fixes and refreshed
  > build scripts)
  > DONE: no further development is expected in
  > the near future.
  > 
  > http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz
  > http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.zip
  > 
  > 
  > ReadOSM
  > ===============================
  > the current stable version is now 1.1.0a
  > (supporting few minor bug fixes and refreshed
  > build scripts)
  > DONE: no further development is expected in
  > the near future.
  > 
  > http://www.gaia-gis.it/gaia-sins/readosm-1.1.0a.tar.gz
  > http://www.gaia-gis.it/gaia-sins/readosm-1.1.0a.zip
  > 
  > 
  > VirtualPG
  > ===============================
  > the current stanle version is now 2.0.0
  > (supporting few minor bug fixes and refreshed
  > build scripts)
  > DONE: no further development is expected in
  > the near future.
  > 
  > http://www.gaia-gis.it/gaia-sins/virtualpg-2.0.0.tar.gz
  > http://www.gaia-gis.it/gaia-sins/virtualpg-2.0.0.zip
  > 
  > 
  > libspatialite
  > ===============================
  > 5.0.0 Release Candidate 1 (RC1) is now ready
  > to become a stable release.
  > I'll simply wait for more or less a week so to
  > allow for a reasonable community testing; if
  > no critical issue will be reported in the meanwhile
  > I'll go ASAP to release 5.0.0 "stable"
  > 
  > http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.0-RC1.tar.gz
  > http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.0-RC1.zip
  > 
  > 
  > librasterlite2
  > ================================
  > still under active development.
  > a very relevant milestone has been achieved,
  > now the DB layout required for fully integrating
  > libspatialte and librasterlie2 is definetely 
  > consolidated, no further changes are expected.
  > 
  > http://www.gaia-gis.it/gaia-sins/librasterlite2-sources/librasterlite2-1.1.0-beta1.tar.gz
  > http://www.gaia-gis.it/gaia-sins/librasterlite2-sources/librasterlite2-1.1.0-beta1.zip
  > 
  > more development activity is still required for
  > fully implementing SQL-driven graphic rendering 
  > of both Vector and Raster Coverages based on
  > standard SLD/SE styles.
  > 
  > a reasonable estimate is two man-month (and
  > not necessarily one development month equals
  > one calendar month)
  > 
  > 
  > spatialite_gui
  > =================================
  > the GUI tool is expected to take full profit from
  > the advanced features of RasterLite2 so to
  > become a complete self-contained GIS viewer 
  > with map editing capabilities.
  >  
  > consequently it wiil surely be the last
  > member of the SpatiaLite family to be
  > released in a final stable form. 
  > several transient development versions
  > will be possibly released so to closely
  > follow the evolution of RasterLite2.
  > 
  > http://www.gaia-gis.it/gaia-sins/spatialite-gui-sources/spatialite_gui-2.1.0-beta1.tar.gz
  > http://www.gaia-gis.it/gaia-sins/spatialite-gui-sources/spatialite_gui-2.1.0-beta1.zip

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/931#issuecomment-673074297
issue_url: https://api.github.com/repos/simonw/datasette/issues/931
node_id: MDEyOklzc3VlQ29tbWVudDY3MzA3NDI5Nw==
created_at: 2020-08-12T19:46:29Z
updated_at: 2020-08-12T19:46:29Z
author_association: OWNER
body:
  Looks like the old files have moved to:

  - http://www.gaia-gis.it/gaia-sins/spatialite-tools-sources/
  - http://www.gaia-gis.it/gaia-sins/libspatialite-sources/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/931#issuecomment-673088110
issue_url: https://api.github.com/repos/simonw/datasette/issues/931
node_id: MDEyOklzc3VlQ29tbWVudDY3MzA4ODExMA==
created_at: 2020-08-12T20:15:28Z
updated_at: 2020-08-12T20:15:28Z
author_association: OWNER
body:
  I changed the Dockerfile and built it on my laptop using:

      docker build -f Dockerfile -t datasette-updated-spatialite .

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/931#issuecomment-673104851
issue_url: https://api.github.com/repos/simonw/datasette/issues/931
node_id: MDEyOklzc3VlQ29tbWVudDY3MzEwNDg1MQ==
created_at: 2020-08-12T20:52:08Z
updated_at: 2020-08-12T20:52:08Z
author_association: OWNER
body:
  ```
  docker run -p 8001:8001 -v `pwd`:/mnt \                     
    datasette-updated-spatialite \
    datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db --load-extension=/usr/local/lib/mod_spatialite.so
  ```
  This seems to work `/-/versions` reports the SpatiaLite versions as expected:
  ```
          "extensions": {
              "json1": null,
              "spatialite": "4.4.0-RC0"
          },
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/932#issuecomment-673735299
issue_url: https://api.github.com/repos/simonw/datasette/issues/932
node_id: MDEyOklzc3VlQ29tbWVudDY3MzczNTI5OQ==
created_at: 2020-08-13T22:10:40Z
updated_at: 2020-08-13T22:11:06Z
author_association: OWNER
body:
  Idea: plugins can provide their own user-facing documentation. Datasette can like to eg `datasette.io/help?plugins=datasette-vega,datasette-cluster-map` to get the user manual with extra sections for those plugins.

  Or... link to `?url=datasette-url` and the documentation site can hit `/-/plugins.json` to figure out what extra manual sections to display!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/932#issuecomment-674144798
issue_url: https://api.github.com/repos/simonw/datasette/issues/932
node_id: MDEyOklzc3VlQ29tbWVudDY3NDE0NDc5OA==
created_at: 2020-08-14T16:02:24Z
updated_at: 2020-08-14T16:02:24Z
author_association: OWNER
body:
  Things to go in here:

  - What is Datasette?
  - A *database* contains *tables* full of *records*. A table has *rows* and *columns*.
  - Understanding faceting
  - How to use the filter interface
  - How to export data
  - How to link to data
  - How to run SQL

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/935#issuecomment-674450607
issue_url: https://api.github.com/repos/simonw/datasette/issues/935
node_id: MDEyOklzc3VlQ29tbWVudDY3NDQ1MDYwNw==
created_at: 2020-08-15T21:50:41Z
updated_at: 2020-08-15T21:50:41Z
author_association: OWNER
body:
  The bug is here:

  https://github.com/simonw/datasette/blob/13b3b51087964d5e1a8c1cdd2495e07bdbe176b8/datasette/database.py#L89-L100

  If `conn = self.connect(write=True)` raises an exception the entire server hangs, like this:
  ```
  % datasette -i fixtures.db  --get /            
  Exception in thread Thread-1:
  Traceback (most recent call last):
    File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
      self.run()
    File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 870, in run
      self._target(*self._args, **self._kwargs)
    File "/Users/simon/.local/share/virtualenvs/latest-datasette-with-all-plugins-PJL_Xy9e/lib/python3.8/site-packages/datasette/database.py", line 92, in _execute_writes
      conn = self.connect(write=True)
    File "/Users/simon/.local/share/virtualenvs/latest-datasette-with-all-plugins-PJL_Xy9e/lib/python3.8/site-packages/datasette/database.py", line 55, in connect
      assert not (write and not self.is_mutable)
  AssertionError

  ... server hangs here ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/935#issuecomment-674450652
issue_url: https://api.github.com/repos/simonw/datasette/issues/935
node_id: MDEyOklzc3VlQ29tbWVudDY3NDQ1MDY1Mg==
created_at: 2020-08-15T21:51:22Z
updated_at: 2020-08-15T21:51:22Z
author_association: OWNER
body:
  The easiest way to recreate this is to attempt a write against an immutable database, which triggers this assertion error:

  https://github.com/simonw/datasette/blob/13b3b51087964d5e1a8c1cdd2495e07bdbe176b8/datasette/database.py#L47-L55

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/935#issuecomment-674451012
issue_url: https://api.github.com/repos/simonw/datasette/issues/935
node_id: MDEyOklzc3VlQ29tbWVudDY3NDQ1MTAxMg==
created_at: 2020-08-15T21:56:13Z
updated_at: 2020-08-15T21:56:13Z
author_association: OWNER
body:
  This implementation seems to fix it, need to work out how to test though.
  ```diff
  diff --git a/datasette/database.py b/datasette/database.py
  index ffa7a79..7ba1456 100644
  --- a/datasette/database.py
  +++ b/datasette/database.py
  @@ -89,14 +89,22 @@ class Database:
       def _execute_writes(self):
           # Infinite looping thread that protects the single write connection
           # to this database
  -        conn = self.connect(write=True)
  +        conn_exception = None
  +        conn = None
  +        try:
  +            conn = self.connect(write=True)
  +        except Exception as e:
  +            conn_exception = e
           while True:
               task = self._write_queue.get()
  -            try:
  -                result = task.fn(conn)
  -            except Exception as e:
  -                print(e)
  -                result = e
  +            if conn_exception is not None:
  +                result = conn_exception
  +            else:
  +                try:
  +                    result = task.fn(conn)
  +                except Exception as e:
  +                    print(e)
  +                    result = e
               task.reply_queue.sync_q.put(result)
   
       async def execute_fn(self, fn):
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/936#issuecomment-674453318
issue_url: https://api.github.com/repos/simonw/datasette/issues/936
node_id: MDEyOklzc3VlQ29tbWVudDY3NDQ1MzMxOA==
created_at: 2020-08-15T22:29:15Z
updated_at: 2020-08-15T22:32:58Z
author_association: OWNER
body:
  Test failure:
  ```
  tests/test_canned_queries.py F
  >>>  captured stdout >>> 
  __enter__
  >>>  traceback >>> 

  canned_write_client = <datasette.utils.testing.TestClient object at 0x108ec5040>

      def test_insert(canned_write_client):
          response = canned_write_client.post(
              "/data/add_name",
              {"name": "Hello"},
              allow_redirects=False,
              csrftoken_from=True,
              cookies={"foo": "bar"},
          )
          assert 302 == response.status
  >       assert "/data/add_name?success" == response.headers["Location"]
  E       AssertionError: assert '/data/add_name?success' == '/data/add_name'
  E         - /data/add_name
  E         + /data/add_name?success
  E         ?               ++++++++

  /Users/simon/Dropbox/Development/datasette/tests/test_canned_queries.py:66: AssertionError
  ```
  No idea why this change would affect that test.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/936#issuecomment-674453772
issue_url: https://api.github.com/repos/simonw/datasette/issues/936
node_id: MDEyOklzc3VlQ29tbWVudDY3NDQ1Mzc3Mg==
created_at: 2020-08-15T22:35:29Z
updated_at: 2020-08-15T22:35:29Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=h1) Report
  > Merging [#936](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/13b3b51087964d5e1a8c1cdd2495e07bdbe176b8&el=desc) will **increase** coverage by `0.02%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/936/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main     #936      +/-   ##
  ==========================================
  + Coverage   84.02%   84.04%   +0.02%     
  ==========================================
    Files          28       28              
    Lines        3774     3774              
  ==========================================
  + Hits         3171     3172       +1     
  + Misses        603      602       -1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/936/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.18% <0.00%> (+0.18%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=footer). Last update [13b3b51...94a68b9](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/938#issuecomment-674544691
issue_url: https://api.github.com/repos/simonw/datasette/issues/938
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU0NDY5MQ==
created_at: 2020-08-16T16:03:52Z
updated_at: 2020-08-16T16:03:52Z
author_association: OWNER
body:
  Four plugin hooks need this extra `columns` argument:

  - `extra_css_urls(template, database, table, datasette)`
  - `extra_js_urls(template, database, table, datasette)`
  - `extra_body_script(template, database, table, view_name, datasette)`
  - `extra_template_vars(template, database, table, view_name, request, datasette)`

  It's weird that these take different arguments. I should fix that too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/939#issuecomment-674544875
issue_url: https://api.github.com/repos/simonw/datasette/issues/939
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU0NDg3NQ==
created_at: 2020-08-16T16:05:36Z
updated_at: 2020-08-16T16:05:36Z
author_association: OWNER
body:
  They should all take:

      template, database, table, view_name, request, datasette, columns

  `columns` is new, see #938.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/939#issuecomment-674544973
issue_url: https://api.github.com/repos/simonw/datasette/issues/939
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU0NDk3Mw==
created_at: 2020-08-16T16:06:34Z
updated_at: 2020-08-16T16:06:34Z
author_association: OWNER
body:
  They should also be next to each other in the documentation - right now they're a bit scattered in terms of page order:

  <img width="598" alt="Plugin_hooks_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/90338620-c5d3af00-df9f-11ea-908b-3f888947dd03.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/939#issuecomment-674545058
issue_url: https://api.github.com/repos/simonw/datasette/issues/939
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU0NTA1OA==
created_at: 2020-08-16T16:07:20Z
updated_at: 2020-08-16T16:07:20Z
author_association: OWNER
body:
  I'm going to implement this first as a single commit, then implement `columns` from #938 as a separate change.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/939#issuecomment-674547788
issue_url: https://api.github.com/repos/simonw/datasette/issues/939
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU0Nzc4OA==
created_at: 2020-08-16T16:31:08Z
updated_at: 2020-08-16T16:31:08Z
author_association: OWNER
body:
  Also: they should all be able to return `async def` inner functions. At the moment only `extra_template_vars` supports that pattern.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/938#issuecomment-674551826
issue_url: https://api.github.com/repos/simonw/datasette/issues/938
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU1MTgyNg==
created_at: 2020-08-16T17:07:57Z
updated_at: 2020-08-16T17:07:57Z
author_association: OWNER
body:
  extra_tenplate_vars should he documented first and should be the only one to document the arguments and the async return feature. Others should refer back to it rather than duplicating that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/938#issuecomment-674558631
issue_url: https://api.github.com/repos/simonw/datasette/issues/938
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU1ODYzMQ==
created_at: 2020-08-16T18:10:23Z
updated_at: 2020-08-16T18:10:23Z
author_association: OWNER
body:
  Documentation: https://github.com/simonw/datasette/blob/3a4c8ed36aa97211e46849d32a09f2f386f342dd/docs/plugin_hooks.rst#extra-template-vars-template-database-table-columns-view-name-request-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/941#issuecomment-674566290
issue_url: https://api.github.com/repos/simonw/datasette/issues/941
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU2NjI5MA==
created_at: 2020-08-16T19:18:43Z
updated_at: 2020-08-18T05:04:31Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/941?src=pr&el=h1) Report
  > Merging [#941](https://codecov.io/gh/simonw/datasette/pull/941?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/52eabb019d4051084b21524bd0fd9c2731126985&el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/941/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/941?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main     #941   +/-   ##
  =======================================
    Coverage   84.10%   84.10%           
  =======================================
    Files          28       28           
    Lines        3788     3788           
  =======================================
    Hits         3186     3186           
    Misses        602      602           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/941?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/941?src=pr&el=footer). Last update [52eabb0...f5a72e1](https://codecov.io/gh/simonw/datasette/pull/941?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-674566618
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU2NjYxOA==
created_at: 2020-08-16T19:20:58Z
updated_at: 2020-08-16T19:20:58Z
author_association: OWNER
body:
  I need to figure out how to build and push the Docker image on releases. Here's the Travis code for that: https://github.com/simonw/datasette/blob/52eabb019d4051084b21524bd0fd9c2731126985/.travis.yml#L38-L47

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/914#issuecomment-674578388
issue_url: https://api.github.com/repos/simonw/datasette/issues/914
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU3ODM4OA==
created_at: 2020-08-16T21:10:27Z
updated_at: 2020-08-16T21:10:27Z
author_association: OWNER
body:
  Demo of the fix: https://latest.datasette.io/fixtures/binary_data.json?_sort_desc=data&_shape=array&_nl=on

  ```
  {"rowid": 2, "data": {"$base64": true, "encoded": "FRwDx60F/g=="}}
  {"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-674589035
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU4OTAzNQ==
created_at: 2020-08-16T23:02:23Z
updated_at: 2020-08-16T23:02:23Z
author_association: OWNER
body:
  I'd like to set these up as different workflows that depend on each other, if that's possible.

  I want to start three test runs in parallel (on three different Python versions), then if all three pass kick off the PyPI push (without running more tests), then if that passes do the Docker build and push.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-674589321
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU4OTMyMQ==
created_at: 2020-08-16T23:04:34Z
updated_at: 2020-08-16T23:04:34Z
author_association: OWNER
body:
  https://docs.github.com/en/actions/getting-started-with-github-actions/core-concepts-for-github-actions#job

  > A set of steps that execute on the same runner. You can define the dependency rules for how jobs run in a workflow file. Jobs can run at the same time in parallel or run sequentially depending on the status of a previous job. For example, a workflow can have two sequential jobs that build and test code, where the test job is dependent on the status of the build job. If the build job fails, the test job will not run.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-674589472
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU4OTQ3Mg==
created_at: 2020-08-16T23:05:57Z
updated_at: 2020-08-16T23:05:57Z
author_association: OWNER
body:
  When I figure this out I'll update the https://github.com/simonw/datasette-plugin/blob/main/datasette-%7B%7Bcookiecutter.hyphenated%7D%7D/.github/workflows/publish.yml default workflow to do this - right now it runs the tests once on just a single version of Python as part of the package deploy to PyPI step.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-674590583
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY3NDU5MDU4Mw==
created_at: 2020-08-16T23:15:51Z
updated_at: 2020-08-18T05:04:43Z
author_association: OWNER
body:
  This example of jobs depending on each other and sharing data via artifacts looks relevant: https://docs.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts#passing-data-between-jobs-in-a-workflow

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-675259273
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDY3NTI1OTI3Mw==
created_at: 2020-08-18T05:28:32Z
updated_at: 2020-08-18T05:28:32Z
author_association: MEMBER
body:
  Oh that's interesting - i didn't realize "reviews" were a separate concept. I'd definitely accept a pull request adding those!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/47#issuecomment-675523053
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/47
node_id: MDEyOklzc3VlQ29tbWVudDY3NTUyMzA1Mw==
created_at: 2020-08-18T14:45:53Z
updated_at: 2020-08-18T14:45:53Z
author_association: MEMBER
body:
  ```
  % github-to-sqlite emojis emojis.db --fetch
    [########----------------------------]  397/1682   23%  00:03:43
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-675538586
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY3NTUzODU4Ng==
created_at: 2020-08-18T15:11:36Z
updated_at: 2020-08-18T15:11:36Z
author_association: OWNER
body:
  I tested this new publish pattern (running the tests in parallel before the deploy step) on `github-to-sqlite` - skipping the Docker step - and it worked: https://github.com/dogsheep/github-to-sqlite/actions/runs/213809864

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-675609109
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY3NTYwOTEwOQ==
created_at: 2020-08-18T17:21:51Z
updated_at: 2020-08-18T17:21:51Z
author_association: OWNER
body:
  Asked about this on the encode gitter here: https://gitter.im/encode/community?at=5f3c0dcaa8c17801765940c0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/873#issuecomment-675610275
issue_url: https://api.github.com/repos/simonw/datasette/issues/873
node_id: MDEyOklzc3VlQ29tbWVudDY3NTYxMDI3NQ==
created_at: 2020-08-18T17:24:05Z
updated_at: 2020-08-18T17:26:10Z
author_association: OWNER
body:
  Maybe I can do this with ASGI after all. Here's the output of `/-/asgi-scope` with `datasette-debug-asgi` installed:
  ```
  {'asgi': {'spec_version': '2.1', 'version': '3.0'},
   'client': ('127.0.0.1', 62035),
   'headers': [(b'host', b'127.0.0.1:62029'),
               (b'user-agent',
                b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:79.0) Gecko'
                b'/20100101 Firefox/79.0'),
               (b'accept',
                b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
                b'webp,*/*;q=0.8'),
               (b'accept-language', b'en-US,en;q=0.5'),
               (b'accept-encoding', b'gzip, deflate'),
               (b'dnt', b'1'),
               (b'connection', b'keep-alive'),
               (b'upgrade-insecure-requests', b'1'),
               (b'cache-control', b'max-age=0')],
   'http_version': '1.1',
   'method': 'GET',
   'path': '/-/asgi-scope',
   'query_string': b'',
   'raw_path': b'/-/asgi-scope',
   'root_path': '',
   'scheme': 'http',
   'server': ('127.0.0.1', 62029),
   'type': 'http'}
  ```
  That `'server': ('127.0.0.1', 62029)` bit has the correct port. Question is, can I access that programmatically on server startup?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-675715472
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: MDEyOklzc3VlQ29tbWVudDY3NTcxNTQ3Mg==
created_at: 2020-08-18T20:55:02Z
updated_at: 2020-08-18T20:55:02Z
author_association: OWNER
body:
  Could display these as tooltips on icons something like this (from the experimental `datasette-inspect-columns` plugin):

  <img width="500" alt="fixtures__facetable__15_rows_and_NOAA_tides_second_attempt_-_Jupyter_Notebook" src="https://user-images.githubusercontent.com/9599/90564416-37eff380-e15a-11ea-95e5-b556e9039a2a.png">

  This would need to take accessibility into account, and would need a different display for the mobile web layout. Need to consider how it will interact with the column menu suggested in #690.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-675718593
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: MDEyOklzc3VlQ29tbWVudDY3NTcxODU5Mw==
created_at: 2020-08-18T21:02:11Z
updated_at: 2020-08-18T21:02:24Z
author_association: OWNER
body:
  Easiest solution: if you provide column metadata it gets displayed above the table, something like on https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act

  <img width="500" alt="fivethirtyeight__antiquities-act_actions_under_antiquities_act__344_rows" src="https://user-images.githubusercontent.com/9599/90565187-57d3e700-e15b-11ea-89c8-0270e3040a50.png">

  HTML `title=` tooltips are also added to the table headers, which won't be visible on touch devices but that's OK because the information is visible on the page already.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-675720040
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: MDEyOklzc3VlQ29tbWVudDY3NTcyMDA0MA==
created_at: 2020-08-18T21:05:24Z
updated_at: 2020-08-18T21:05:24Z
author_association: OWNER
body:
  Is `columns` the right key for this in the table metadata block? I might want to use that for initial values for `?_col=` in #615.

  Alternative names:

  - `column_descriptions`
  - `column_info`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/268#issuecomment-675725464
issue_url: https://api.github.com/repos/simonw/datasette/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDY3NTcyNTQ2NA==
created_at: 2020-08-18T21:18:07Z
updated_at: 2020-08-18T21:18:35Z
author_association: OWNER
body:
  I want this on the table page - but that means that the table page will need to run a slightly more complex query since it needs access to a `rank` column to sort by - which it gets from running a join.

  BUT... that join needs to be constructed in a way that keeps existing filters, `?_where=` clauses etc intact.

  Here's a prototype using SQLite CTEs: https://register-of-members-interests.datasettes.com/regmem?sql=with+original+as+%28select+rowid%2C+*+from+items%29%0D%0Aselect%0D%0A++original.*%2C%0D%0A++items_fts.rank+as+items_fts_rank%0D%0Afrom%0D%0A++original+join+items_fts+on+original.rowid+%3D+items_fts.rowid%0D%0Awhere%0D%0A++items_fts+match+escape_fts%28%3Asearch%29%0D%0Aorder+by+items_fts_rank+desc+limit+10&search=hotel

  ```sql
  with original as (
    select
      rowid,
      *
    from
      items
  )
  select
    original.*,
    items_fts.rank as items_fts_rank
  from
    original
    join items_fts on original.rowid = items_fts.rowid
  where
    items_fts match escape_fts(:search)
  order by
    items_fts_rank desc
  limit
    10
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/915#issuecomment-675746544
issue_url: https://api.github.com/repos/simonw/datasette/issues/915
node_id: MDEyOklzc3VlQ29tbWVudDY3NTc0NjU0NA==
created_at: 2020-08-18T22:14:41Z
updated_at: 2020-08-18T22:14:41Z
author_association: OWNER
body:
  I'm actually pretty happy with how `datasette-graphql` works now - maybe the trick here is to redesign the JSON format in #782 such that it can be used as a documented interface by things like `datasette-graphql` and then ensure Datasette has a documented mechanism for dispatching internal requests.

  I just did a horrible hack here that simulates an internal request, so supporting them as a feature would definitely make sense: https://github.com/natbat/tidepools_near_me/commit/ec102c6da5a5d86f17628740d90b6365b671b5e1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675747878
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTc0Nzg3OA==
created_at: 2020-08-18T22:18:46Z
updated_at: 2020-08-18T22:19:12Z
author_association: OWNER
body:
  Could be as simple as `response = await datasette.get("/path/blah")` - which could also be re-used by the implementation of the `datasette --get /` CLI option introduced in #927.

  Bit weird calling it `.get()` since that clashes with Python's dictionary `.get()` method.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675748573
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTc0ODU3Mw==
created_at: 2020-08-18T22:20:52Z
updated_at: 2020-08-18T22:20:52Z
author_association: OWNER
body:
  Should it default to treating things as if they had the `.json` extension? There are use-cases for the non-JSON method, such as https://github.com/natbat/tidepools_near_me/commit/ec102c6da5a5d86f17628740d90b6365b671b5e1

  I think I'm OK with people having to add `.json` to their internal calls. Maybe they could use `format="json"`) as an optional parameter which would automatically handle the very weird edge-cases where you need to use `?_format=json` instead of `.json` (due to table names existing with a `.json` suffix).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675749076
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTc0OTA3Ng==
created_at: 2020-08-18T22:22:21Z
updated_at: 2020-08-18T22:22:21Z
author_association: OWNER
body:
  Alternative name possibilities:

  - `datasette.http_get(...)` - slightly misleading since it's not going over the HTTP protocol
  - `datasette.internal_get(...)` - the `internal_` might suggest its not an API for external use, which isn't true - it's for plugins
  - `datasette.get(...)` - clashes with `dict.get()` but I'm not at all sure that's a good reason not to use it

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675749319
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTc0OTMxOQ==
created_at: 2020-08-18T22:23:01Z
updated_at: 2020-08-18T22:23:01Z
author_association: OWNER
body:
  Actually no - `requests.get()` and `httpx.get()` prove that having a `.get()` method for an HTTP-related API isn't confusing to people at all.

  `datasette.get()` it is.

  (I'll probably add `datasette.post()` in the future too).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675750382
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTc1MDM4Mg==
created_at: 2020-08-18T22:26:15Z
updated_at: 2020-08-18T22:26:15Z
author_association: OWNER
body:
  Should internal requests executed in this way be handled by plugins that used the `asgi_wrapper()` hook?

  Hard to be sure one way or the other. I'm worried about logging middleware triggering twice - but actually anyone doing serious logging of their Datasette instance is probably doing it in a different layer (uvicorn logs or nginx proxy or whatever) so they wouldn't be affected. There aren't any ASGI logging middlewares out there that I've seen.

  Also: if you run into a situation where your stuff is breaking because `datasette.get()` is calling ASGI middleware twice you can fix it by running your ASGI middleware outside of the `asgi_wrapper` plugin hook mechanism.

  So I think it DOES execute `asgi_wrapper()` middleware.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675750845
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTc1MDg0NQ==
created_at: 2020-08-18T22:27:43Z
updated_at: 2020-08-18T22:27:43Z
author_association: OWNER
body:
  What about authentication checks etc? Won't they run twice?

  I think that's OK too, in fact it's desirable: think of the case of `datasette-graphql` where a bunch of different TableView calls are being made as part of the same GraphQL queries. Having those calls take advantage of finely grained per-table authentication and permission checks seems like a good feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675751719
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTc1MTcxOQ==
created_at: 2020-08-18T22:30:27Z
updated_at: 2020-08-18T22:30:27Z
author_association: OWNER
body:
  Right now calling `datasette.app()` instantiates an ASGI application - complete with a bunch of routes and wrappers - and returns that application object. Calling it twice instantiates another ASGI application.

  I think a single `Datasette` instance should only ever create a single ASGI app - so the `.app()` method should cache the ASGI app that it returns the first time and return the same application again on future calls.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675752436
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTc1MjQzNg==
created_at: 2020-08-18T22:32:44Z
updated_at: 2020-08-18T22:32:44Z
author_association: OWNER
body:
  One thing to consider here: Datasette's table and database name escaping rules can be a little bit convoluted.

  If a plugin wants to get back the first five rows of a table, it will need to construct a URL `/dbname/tablename?_size=5` - but it will need to know how to turn the database and table names into the correctly escaped `dbname` and `tablename` values.

  Here's how the `row.html` table handles that right now: https://github.com/simonw/datasette/blob/b21ed237ab940768574c834aa5a7130724bd3a2d/datasette/templates/row.html#L19-L23

  It would be an improvement to have this logic abstracted out somewhere and documented so plugins can use it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675753114
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTc1MzExNA==
created_at: 2020-08-18T22:34:55Z
updated_at: 2020-08-18T22:34:55Z
author_association: OWNER
body:
  Maybe allow this:

      response = await datasette.get("/{database}/{table}.json", database=database, table=table)

  This could cause problems if users ever need to pass literal `{` in their paths. Maybe allow this too:

      response = await datasette.get("/{database}/{table}.json", interpolate=False)

  Not convinced this is useful - it's a bit unintuitive.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675787416
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTc4NzQxNg==
created_at: 2020-08-19T00:42:38Z
updated_at: 2020-08-19T00:42:38Z
author_association: OWNER
body:
  I just realised that this mechanism is kind of like being able to use microservices - make API calls within your application - except that everything runs in the same process against SQLite databases so calls will be _lightning fast_.

  It also means that a plugin can add a new internal API to Datasette that's accessible to other plugins by registering a new route with `register_routes`!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675788203
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTc4ODIwMw==
created_at: 2020-08-19T00:46:08Z
updated_at: 2020-08-19T00:46:23Z
author_association: OWNER
body:
  Also fun: the inevitable plugin that exposes this to the template language - so Datasette templates can stitch together data from multiple other internal API calls. Fun way to take advantage of `async` support in Jinja.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/944#issuecomment-675829942
issue_url: https://api.github.com/repos/simonw/datasette/issues/944
node_id: MDEyOklzc3VlQ29tbWVudDY3NTgyOTk0Mg==
created_at: 2020-08-19T03:27:25Z
updated_at: 2020-08-19T03:27:25Z
author_association: OWNER
body:
  I created a template file called `templates/pages/museums/{slug}.html` and used the debugger to see if Jinja could see it. This worked:
  ```
  (Pdb) self.ds.jinja_env.list_templates()
  ['500.html', '_codemirror.html', '_codemirror_foot.html', '_description_source_license.html', '_footer.html',
  '_table.html', 'allow_debug.html', 'base.html', 'database.html', 'default:500.html', 'default:_codemirror.html',
  'default:_codemirror_foot.html', 'default:_description_source_license.html', 'default:_footer.html',
  'default:_table.html', 'default:allow_debug.html', 'default:base.html', 'default:database.html',
  'default:index.html', 'default:logout.html', 'default:messages_debug.html', 'default:patterns.html',
  'default:permissions_debug.html', 'default:query.html', 'default:row.html', 'default:show_json.html',
  'default:table.html', 'forbidden.html', 'index.html', 'logout.html', 'messages_debug.html',
  'pages/about.html', 'pages/museums/{slug}.html', 'patterns.html', 'permissions_debug.html',
  'query.html', 'row.html', 'show_json.html', 'table.html']
  ```
  The `pages/museums/{slug}.html` template is in that list.

  Here's the implementation of that `list_templates()` method - it does some filesystem walking so it may be a bit expensive to run it on every request: https://github.com/pallets/jinja/blob/ca8b0b0287e320fe1f4a74f36910ef7ae3303d99/src/jinja2/loaders.py#L197-L212

  But caching it would be pretty easy - either until the server is restarted or as an in-memory cache for a few seconds.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/944#issuecomment-675830678
issue_url: https://api.github.com/repos/simonw/datasette/issues/944
node_id: MDEyOklzc3VlQ29tbWVudDY3NTgzMDY3OA==
created_at: 2020-08-19T03:30:10Z
updated_at: 2020-08-19T03:30:10Z
author_association: OWNER
body:
  These templates will need a way to raise a 404 - so that if the template itself is deciding if the page exists (for example using `datasette-template-sql` or the proposed `datasette.get()` method from #943 or the `graphql()` template function in https://github.com/simonw/datasette-graphql/issues/50) it can return a regular 404 page.

  This can imitate the `custom_redirect()` function from https://docs.datasette.io/en/stable/custom_templates.html#custom-redirects:
  ```html+jinja
  {{ custom_redirect("https://github.com/simonw/datasette", 301) }}
  ```
  It could be as simple as this:
  ```
  {{ raise_404("Museum not found") }}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675884980
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTg4NDk4MA==
created_at: 2020-08-19T06:44:26Z
updated_at: 2020-08-19T06:44:26Z
author_association: OWNER
body:
  Need to decide what to do about JSON responses.

  When called from a template it's likely the intent will be to further loop through the JSON data returned. It would be annoying to have to run `json.loads` here.

  Maybe a `.get_json()` method then? Or even return a response that has `.json()` and `.text` similar to `httpx` - or just return an `httpx` response.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-675889551
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY3NTg4OTU1MQ==
created_at: 2020-08-19T06:56:06Z
updated_at: 2020-08-19T06:56:17Z
author_association: OWNER
body:
  I'm leaning towards defaulting to JSON as the requested format - you can pass `format="html"` if you want HTML.

  But weird that it's different from the web UI.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678474018
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/134
node_id: MDEyOklzc3VlQ29tbWVudDY3ODQ3NDAxOA==
created_at: 2020-08-21T20:16:20Z
updated_at: 2020-08-21T20:16:20Z
author_association: OWNER
body:
  Trickiest part of this is how to write a test for it.

  I'll do a `pytest.skipIf` that only executes the test if SpatiaLite is available.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678474928
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/134
node_id: MDEyOklzc3VlQ29tbWVudDY3ODQ3NDkyOA==
created_at: 2020-08-21T20:18:33Z
updated_at: 2020-08-21T20:18:33Z
author_association: OWNER
body:
  This should get me SpatiaLite in the GitHub Actions Ubuntu:
  ```
  apt install libsqlite3-mod-spatialite
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678475578
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/135
node_id: MDEyOklzc3VlQ29tbWVudDY3ODQ3NTU3OA==
created_at: 2020-08-21T20:20:05Z
updated_at: 2020-08-21T20:20:05Z
author_association: OWNER
body:
  https://github.com/simonw/cryptozoology/blob/2ad69168f3b78ebd90a2cbeea8136c9115e2a9b7/build_cryptids_database.py#L16-L22
  ```python
  try_these = (
      "mod_spatialite",
      "/usr/local/lib/mod_spatialite.dylib",
      "/usr/lib/x86_64-linux-gnu/mod_spatialite.so",
      "/usr/lib/x86_64-linux-gnu/libspatialite.so.5",
      "/usr/lib/x86_64-linux-gnu/libspatialite.so.7",
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678479741
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/135
node_id: MDEyOklzc3VlQ29tbWVudDY3ODQ3OTc0MQ==
created_at: 2020-08-21T20:30:37Z
updated_at: 2020-08-21T20:30:37Z
author_association: OWNER
body:
  Docs: https://github.com/simonw/sqlite-utils/blob/bf4c6b7c82fab6b2400e48424f8dac1ae2f0a2dc/docs/python-api.rst#finding-spatialite

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/136#issuecomment-678480969
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/136
node_id: MDEyOklzc3VlQ29tbWVudDY3ODQ4MDk2OQ==
created_at: 2020-08-21T20:33:45Z
updated_at: 2020-08-21T20:33:45Z
author_association: OWNER
body:
  I think this should initialize SpatiaLite against the current database if it has not been initialized already.

  Relevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126

  ```python
  def init_spatialite(db, lib):
      db.conn.enable_load_extension(True)
      db.conn.load_extension(lib)
      # Initialize SpatiaLite if not yet initialized
      if "spatial_ref_sys" in db.table_names():
          return
      db.conn.execute("select InitSpatialMetadata(1)")


  def ensure_table_has_geometry(db, table, table_srid):
      if "geometry" not in db[table].columns_dict:
          db.conn.execute(
              "SELECT AddGeometryColumn(?, 'geometry', ?, 'GEOMETRY', 2);",
              [table, table_srid],
          )
  ```
  Not sure if I should add a utility function or CLI command for that `ensure_table_has_geometry` bit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678497497
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/134
node_id: MDEyOklzc3VlQ29tbWVudDY3ODQ5NzQ5Nw==
created_at: 2020-08-21T21:06:26Z
updated_at: 2020-08-21T21:06:26Z
author_association: OWNER
body:
  Ended up needing two skipIfs:

  https://github.com/simonw/sqlite-utils/blob/7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d/tests/test_cli.py#L888-L893

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/138#issuecomment-678732667
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/138
node_id: MDEyOklzc3VlQ29tbWVudDY3ODczMjY2Nw==
created_at: 2020-08-23T05:46:10Z
updated_at: 2020-08-23T05:46:10Z
author_association: OWNER
body:
  Actually the `TEXT` column thing wasn't a `sqlite-utils` issue, it was unique to how `shapefile-to-spatialite` was creating the table when using the SpatiaLite extension.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/948#issuecomment-679333717
issue_url: https://api.github.com/repos/simonw/datasette/issues/948
node_id: MDEyOklzc3VlQ29tbWVudDY3OTMzMzcxNw==
created_at: 2020-08-24T19:55:59Z
updated_at: 2020-08-24T19:55:59Z
author_association: OWNER
body:
  CodeMirror 6 is in pre-release at the moment and is a complete rewrite. I'll stick with the 5.x series for now. https://github.com/codemirror/codemirror.next/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/948#issuecomment-679355426
issue_url: https://api.github.com/repos/simonw/datasette/issues/948
node_id: MDEyOklzc3VlQ29tbWVudDY3OTM1NTQyNg==
created_at: 2020-08-24T20:43:07Z
updated_at: 2020-08-24T20:43:07Z
author_association: OWNER
body:
  It would also be interesting to try out the SQL hint mode, which can autocomplete against tables and columns. This demo shows how to configure that: https://codemirror.net/mode/sql/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/949#issuecomment-679363710
issue_url: https://api.github.com/repos/simonw/datasette/issues/949
node_id: MDEyOklzc3VlQ29tbWVudDY3OTM2MzcxMA==
created_at: 2020-08-24T21:00:43Z
updated_at: 2020-08-24T21:00:43Z
author_association: OWNER
body:
  I think this requires three extra files from https://github.com/codemirror/CodeMirror/tree/5.57.0/addon/hint

  - `show-hint.css`
  - `show-hint.js`
  - `sql-hint.js`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/949#issuecomment-679367931
issue_url: https://api.github.com/repos/simonw/datasette/issues/949
node_id: MDEyOklzc3VlQ29tbWVudDY3OTM2NzkzMQ==
created_at: 2020-08-24T21:09:50Z
updated_at: 2020-08-24T21:09:50Z
author_association: OWNER
body:
  I'm attracted to this because of how good GraphiQL is for auto-completing queries. But I realize there's a problem here: GraphQL is designed to be autocomplete-friendly, but SQL is not. If you type `select ` and it doesn't know what's going in the `from` clause it can't give you good column autocomplete, for example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/950#issuecomment-680262437
issue_url: https://api.github.com/repos/simonw/datasette/issues/950
node_id: MDEyOklzc3VlQ29tbWVudDY4MDI2MjQzNw==
created_at: 2020-08-25T20:49:24Z
updated_at: 2023-08-23T21:34:09Z
author_association: OWNER
body:
  The alternative to this would be to use regular databases and control access to them using [Authentication and permissions](https://docs.datasette.io/en/stable/authentication.html). My concern there is that it's just too easy for someone to mess up their configuration, which would be really bad. I like the idea of a much stronger defense mechanism specifically designed for secrets that should not be exposed.

  Outside of secrets, passwords and tokens this mechanism could also be useful for the use-case of using Datasette to power websites - as seen on https://www.niche-museums.com/ and https://www.rockybeaches.com/ - maybe those sites don't want to expose their data through their API but still want to use `datasette-template-sql` and the `graphql()` template tag in `datasette-graphql` to render data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/950#issuecomment-680263427
issue_url: https://api.github.com/repos/simonw/datasette/issues/950
node_id: MDEyOklzc3VlQ29tbWVudDY4MDI2MzQyNw==
created_at: 2020-08-25T20:51:30Z
updated_at: 2020-08-25T20:52:13Z
author_association: OWNER
body:
  `datasette-graphql` currently dispatches requests through the `TableView` class, so if that couldn't access private databases then it would not be able to either. See also the concept for `datasette.get(...)` as an internal API in #943 - that might need to have a mechanism for also being able to query private databases, maybe `datasette.get(path, allow_private=True)`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/950#issuecomment-680263999
issue_url: https://api.github.com/repos/simonw/datasette/issues/950
node_id: MDEyOklzc3VlQ29tbWVudDY4MDI2Mzk5OQ==
created_at: 2020-08-25T20:52:47Z
updated_at: 2020-08-25T20:52:47Z
author_association: OWNER
body:
  Naming challenge: secret databases or private databases?

  I prefer private. But `datasette -p` is already taken by `--port`. `datasette -s` is currently available.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/950#issuecomment-680374196
issue_url: https://api.github.com/repos/simonw/datasette/issues/950
node_id: MDEyOklzc3VlQ29tbWVudDY4MDM3NDE5Ng==
created_at: 2020-08-26T00:43:50Z
updated_at: 2020-08-26T00:43:50Z
author_association: OWNER
body:
  The problem with the term "private" is that it could be confused with the concept of databases that aren't visible to the public due to the permissions system - the ones that are displayed with the padlock icon e.g. on https://datasette-auth-passwords-demo.datasette.io/

  <img width="595" alt="datasette-auth-passwords_demo__public__private" src="https://user-images.githubusercontent.com/9599/91241752-807b5400-e6fa-11ea-9f71-9da37c371b45.png">

  So I think "secret" is a better term for these.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682182178
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/139
node_id: MDEyOklzc3VlQ29tbWVudDY4MjE4MjE3OA==
created_at: 2020-08-27T20:46:18Z
updated_at: 2020-08-27T20:46:18Z
author_association: CONTRIBUTOR
body:
  > I tried changing the batch_size argument to the total number of records, but it seems only to effect the number of rows that are committed at a time, and has no influence on this problem.

  So the reason for this is that the `batch_size` for import is limited (of necessity) here: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L1048

  With regard to the issue of ignoring columns, however, I made a fork and hacked a temporary fix that looks like this:
  https://github.com/simonwiles/sqlite-utils/commit/3901f43c6a712a1a3efc340b5b8d8fd0cbe8ee63

  It doesn't seem to affect performance enormously (but I've not tested it thoroughly), and it now does what I need (and would expect, tbh), but it now fails the test here:
  https://github.com/simonw/sqlite-utils/blob/main/tests/test_create.py#L710-L716

  The existence of this test suggests that `insert_all()` is behaving as intended, of course.  It seems odd to me that this would be a desirable default behaviour (let alone the only behaviour), and its not very prominently flagged-up, either.

  @simonw is this something you'd be willing to look at a PR for?  I assume you wouldn't want to change the default behaviour at this point, but perhaps an option could be provided, or at least a bit more of a warning in the docs.  Are there oversights in the implementation that I've made?

  Would be grateful for your thoughts!  Thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682284908
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/139
node_id: MDEyOklzc3VlQ29tbWVudDY4MjI4NDkwOA==
created_at: 2020-08-28T02:11:40Z
updated_at: 2020-08-28T02:11:40Z
author_association: OWNER
body:
  This is deliberate behaviour, but I'm not at all attached to it - you're right in pointing out that it's actually pretty unexpected.

  I'd be happy to change this behaviour so if you pass `alter=True` and then use `.insert_all()` on more than 100 rows it works as you would expect, instead of silently ignoring new columns past the first 100 rows. I don't expect that anyone would be depending on the current behaviour (ignore new columns after the first 100) such that this should be considered a backwards incompatible change.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682285212
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/139
node_id: MDEyOklzc3VlQ29tbWVudDY4MjI4NTIxMg==
created_at: 2020-08-28T02:12:51Z
updated_at: 2020-08-28T02:12:51Z
author_association: OWNER
body:
  I'd be happy to accept a PR for this, provided it included updated unit tests that illustrate it working. I think this is a really good improvement.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/953#issuecomment-682312494
issue_url: https://api.github.com/repos/simonw/datasette/issues/953
node_id: MDEyOklzc3VlQ29tbWVudDY4MjMxMjQ5NA==
created_at: 2020-08-28T04:03:56Z
updated_at: 2020-08-28T04:03:56Z
author_association: OWNER
body:
  Documentation says that the old dictionary mechanism will be deprecated by 1.0:

  https://github.com/simonw/datasette/blob/799ecae94824640bdff21f86997f69844048d5c3/docs/plugin_hooks.rst#L460

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/954#issuecomment-682312736
issue_url: https://api.github.com/repos/simonw/datasette/issues/954
node_id: MDEyOklzc3VlQ29tbWVudDY4MjMxMjczNg==
created_at: 2020-08-28T04:05:01Z
updated_at: 2020-08-28T04:05:10Z
author_association: OWNER
body:
  >  It can also return a dictionary with the following keys. This format is **deprecated** as-of Datasette 0.49 and will be removed by Datasette 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682762911
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/139
node_id: MDEyOklzc3VlQ29tbWVudDY4Mjc2MjkxMQ==
created_at: 2020-08-28T15:54:57Z
updated_at: 2020-08-28T15:55:20Z
author_association: OWNER
body:
  Here's a suggested test update:
  ```diff
  diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
  index a8791c3..12fa2f2 100644
  --- a/sqlite_utils/db.py
  +++ b/sqlite_utils/db.py
  @@ -1074,6 +1074,13 @@ class Table(Queryable):
                   all_columns = list(sorted(all_columns))
                   if hash_id:
                       all_columns.insert(0, hash_id)
  +            else:
  +                all_columns += [
  +                    column
  +                    for record in chunk
  +                    for column in record
  +                    if column not in all_columns
  +                ]
               validate_column_names(all_columns)
               first = False
               # values is the list of insert data that is passed to the
  diff --git a/tests/test_create.py b/tests/test_create.py
  index a84eb8d..3a7fafc 100644
  --- a/tests/test_create.py
  +++ b/tests/test_create.py
  @@ -707,13 +707,15 @@ def test_insert_thousands_using_generator(fresh_db):
       assert 10000 == fresh_db["test"].count
   
   
  -def test_insert_thousands_ignores_extra_columns_after_first_100(fresh_db):
  +def test_insert_thousands_adds_extra_columns_after_first_100(fresh_db):
  +    # https://github.com/simonw/sqlite-utils/issues/139
       fresh_db["test"].insert_all(
           [{"i": i, "word": "word_{}".format(i)} for i in range(100)]
  -        + [{"i": 101, "extra": "This extra column should cause an exception"}]
  +        + [{"i": 101, "extra": "Should trigger ALTER"}],
  +        alter=True,
       )
       rows = fresh_db.execute_returning_dicts("select * from test where i = 101")
  -    assert [{"i": 101, "word": None}] == rows
  +    assert [{"i": 101, "word": None, "extra": "Should trigger ALTER"}] == rows
   
   
   def test_insert_ignore(fresh_db):
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682771226
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/139
node_id: MDEyOklzc3VlQ29tbWVudDY4Mjc3MTIyNg==
created_at: 2020-08-28T15:57:42Z
updated_at: 2020-08-28T15:57:42Z
author_association: OWNER
body:
  That pull request should update this section of the documentation too:

  > If you have more than one record to insert, the insert_all() method is a much more efficient way of inserting them. Just like insert() it will automatically detect the columns that should be created, but it will inspect the first batch of 100 items to help decide what those column types should be.

  https://github.com/simonw/sqlite-utils/blob/ea87c2b943fdd162c42a900ac0aea5ecc2f4b9d9/docs/python-api.rst#L393

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682815377
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/139
node_id: MDEyOklzc3VlQ29tbWVudDY4MjgxNTM3Nw==
created_at: 2020-08-28T16:14:58Z
updated_at: 2020-08-28T16:14:58Z
author_association: CONTRIBUTOR
body:
  Thanks!  And yeah, I had updating the docs on my list too :)  Will try to get to it this afternoon (budgeting time is fraught with uncertainty at the moment!).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/119#issuecomment-683146200
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/119
node_id: MDEyOklzc3VlQ29tbWVudDY4MzE0NjIwMA==
created_at: 2020-08-28T21:05:37Z
updated_at: 2020-08-28T21:05:37Z
author_association: OWNER
body:
  Maybe use `transform_table()` in #114 for this? Would be less efficient as it would copy the whole table but it would reduce library complexity a bit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172829
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/142
node_id: MDEyOklzc3VlQ29tbWVudDY4MzE3MjgyOQ==
created_at: 2020-08-28T22:27:05Z
updated_at: 2020-08-28T22:27:05Z
author_association: OWNER
body:
  Looks like it failed the "black" formatting test - possibly because there's a new release if black out. I'm going to merge despite that failure.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683173375
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/142
node_id: MDEyOklzc3VlQ29tbWVudDY4MzE3MzM3NQ==
created_at: 2020-08-28T22:29:02Z
updated_at: 2020-08-28T22:29:02Z
author_association: OWNER
body:
  Yeah I think that failure is actually because there's a brand new release of Black out and it subtly changes some of the formatting rules. I'll merge this and then run Black against the entire codebase.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/143#issuecomment-683175491
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/143
node_id: MDEyOklzc3VlQ29tbWVudDY4MzE3NTQ5MQ==
created_at: 2020-08-28T22:37:15Z
updated_at: 2020-08-28T22:37:15Z
author_association: OWNER
body:
  I'm going to start running black exclusively in the GitHub Actions workflow, rather than having it run by the unit tests themselves.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/139#issuecomment-683178570
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/139
node_id: MDEyOklzc3VlQ29tbWVudDY4MzE3ODU3MA==
created_at: 2020-08-28T22:48:51Z
updated_at: 2020-08-28T22:48:51Z
author_association: OWNER
body:
  Thanks @simonwiles, this is now released in 2.16.1: https://sqlite-utils.readthedocs.io/en/stable/changelog.html

reactions:
  {"total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/sqlite-utils/issues/144#issuecomment-683180581
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/144
node_id: MDEyOklzc3VlQ29tbWVudDY4MzE4MDU4MQ==
created_at: 2020-08-28T22:57:04Z
updated_at: 2020-08-28T22:57:04Z
author_association: OWNER
body:
  That worked! https://github.com/simonw/sqlite-utils/runs/1043640785?check_suite_focus=true

  <img width="1001" alt="Add_numpy_to_the_matrix__refs__144_·_simonw_sqlite-utils_cbc22ef" src="https://user-images.githubusercontent.com/9599/91621188-1b686e00-e947-11ea-906d-129ceadbb434.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/955#issuecomment-683185861
issue_url: https://api.github.com/repos/simonw/datasette/issues/955
node_id: MDEyOklzc3VlQ29tbWVudDY4MzE4NTg2MQ==
created_at: 2020-08-28T23:17:09Z
updated_at: 2020-08-28T23:17:09Z
author_association: OWNER
body:
  I released 0.49a0 which means I can update the main branches of those two plugins - I'll push a release of them once 0.49 is fully shipped.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/956#issuecomment-683212246
issue_url: https://api.github.com/repos/simonw/datasette/issues/956
node_id: MDEyOklzc3VlQ29tbWVudDY4MzIxMjI0Ng==
created_at: 2020-08-29T01:21:26Z
updated_at: 2020-08-29T01:21:26Z
author_association: OWNER
body:
  I added this but I have no idea if I got it right or not: https://github.com/simonw/datasette/blob/c36e287d71d68ecb2a45e9808eede15f19f931fb/.github/workflows/publish.yml#L58-L63

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/956#issuecomment-683212421
issue_url: https://api.github.com/repos/simonw/datasette/issues/956
node_id: MDEyOklzc3VlQ29tbWVudDY4MzIxMjQyMQ==
created_at: 2020-08-29T01:22:23Z
updated_at: 2020-08-29T01:22:23Z
author_association: OWNER
body:
  Here's the error message again:

  > invalid argument `"***/datasette::0.49a0"` for `"-t, --tag"` flag: invalid reference format

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/956#issuecomment-683212960
issue_url: https://api.github.com/repos/simonw/datasette/issues/956
node_id: MDEyOklzc3VlQ29tbWVudDY4MzIxMjk2MA==
created_at: 2020-08-29T01:25:34Z
updated_at: 2020-08-29T01:25:34Z
author_association: OWNER
body:
  So I guess this bit is wrong: https://github.com/simonw/datasette/blob/c36e287d71d68ecb2a45e9808eede15f19f931fb/.github/workflows/publish.yml#L71-L73

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/956#issuecomment-683213973
issue_url: https://api.github.com/repos/simonw/datasette/issues/956
node_id: MDEyOklzc3VlQ29tbWVudDY4MzIxMzk3Mw==
created_at: 2020-08-29T01:31:39Z
updated_at: 2020-08-29T01:31:39Z
author_association: OWNER
body:
  Here's how the old Travis mechanism worked: https://github.com/simonw/datasette/blob/52eabb019d4051084b21524bd0fd9c2731126985/.travis.yml#L41-L47

  So I was assuming that the eqivalent of `$REPO:$TRAVIS_TAG` in GitHub Actions is `$REPO::${GITHUB_REF#refs/tags/}`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/957#issuecomment-683355508
issue_url: https://api.github.com/repos/simonw/datasette/issues/957
node_id: MDEyOklzc3VlQ29tbWVudDY4MzM1NTUwOA==
created_at: 2020-08-29T23:54:01Z
updated_at: 2020-08-29T23:54:01Z
author_association: OWNER
body:
  Reviewing https://github.com/search?q=user%3Asimonw+%22from+datasette%22&type=Code I spotted these others:

  ```python
  # Various:
  from datasette.utils import path_with_replaced_args
  from datasette.plugins import pm
  from datasette.utils import QueryInterrupted
  from datasette.utils.asgi import Response, Forbidden, NotFound

  # datasette-publish-vercel:
  from datasette.publish.common import (
      add_common_publish_arguments_and_options,
      fail_if_publish_binary_not_installed
  )
  from datasette.utils import temporary_docker_directory

  # datasette-insert
  from datasette.utils import actor_matches_allow, sqlite3

  # obsolete: russian-ira-facebook-ads-datasette 
  from datasette.utils import TableFilter

  # simonw/museums
  from datasette.utils.asgi import asgi_send

  # datasette-media
  from datasette.utils.asgi import Response, asgi_send_file

  # datasette/tests/plugins/my_plugin.py
  from datasette.facets import Facet

  # datasette-graphql
  from datasette.views.table import TableView
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/957#issuecomment-683355598
issue_url: https://api.github.com/repos/simonw/datasette/issues/957
node_id: MDEyOklzc3VlQ29tbWVudDY4MzM1NTU5OA==
created_at: 2020-08-29T23:55:10Z
updated_at: 2020-08-29T23:55:34Z
author_association: OWNER
body:
  Of these I think I'm going to promote the following to being importable directly `from datasette`:

  - `from datasette.app import Datasette`
  - `from datasette.utils import QueryInterrupted`
  - `from datasette.utils.asgi import Response, Forbidden, NotFound`
  - `from datasette.utils import actor_matches_allow`

  All of the rest are infrequently used enough (or clearly named enough) that I'm happy to leave them as-is.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/957#issuecomment-683355993
issue_url: https://api.github.com/repos/simonw/datasette/issues/957
node_id: MDEyOklzc3VlQ29tbWVudDY4MzM1NTk5Mw==
created_at: 2020-08-30T00:02:11Z
updated_at: 2020-08-30T00:04:18Z
author_association: OWNER
body:
  I tried doing this and got this error:
  ```
  (datasette) datasette % pytest
  ==================================================================== test session starts =====================================================================
  platform darwin -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
  rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
  plugins: asyncio-0.14.0, timeout-1.4.2
  collected 1 item / 23 errors                                                                                                                                 

  =========================================================================== ERRORS ===========================================================================
  _____________________________________________________________ ERROR collecting tests/test_api.py _____________________________________________________________
  ImportError while importing test module '/Users/simon/Dropbox/Development/datasette/tests/test_api.py'.
  Hint: make sure your test modules/packages have valid Python names.
  Traceback:
  /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  tests/test_api.py:1: in <module>
      from datasette.plugins import DEFAULT_PLUGINS
  datasette/__init__.py:2: in <module>
      from .app import Datasette
  datasette/app.py:30: in <module>
      from .views.base import DatasetteError, ureg
  datasette/views/base.py:12: in <module>
      from datasette.plugins import pm
  datasette/plugins.py:26: in <module>
      mod = importlib.import_module(plugin)
  /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  datasette/publish/heroku.py:2: in <module>
      from datasette import hookimpl
  E   ImportError: cannot import name 'hookimpl' from partially initialized module 'datasette' (most likely due to a circular import) (/Users/simon/Dropbox/Development/datasette/datasette/__init__.py)
  ```
  That's with `datasette/__init__.py` looking like this:
  ```python
  from datasette.version import __version_info__, __version__  # noqa
  from .app import Datasette
  from .utils.asgi import Forbidden, NotFound, Response
  from .utils import actor_matches_allow, QueryInterrupted
  from .hookspecs import hookimpl  # noqa
  from .hookspecs import hookspec  # noqa


  __all__ = [
      "actor_matches_allow",
      "hookimpl",
      "hookspec",
      "QueryInterrupted",
      "Forbidden",
      "NotFound",
      "Response",
      "Datasette",
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/957#issuecomment-683356440
issue_url: https://api.github.com/repos/simonw/datasette/issues/957
node_id: MDEyOklzc3VlQ29tbWVudDY4MzM1NjQ0MA==
created_at: 2020-08-30T00:08:18Z
updated_at: 2020-08-30T00:10:26Z
author_association: OWNER
body:
  Annoyingly this seems to be the line that causes the circular import:
  ```python
  from .utils.asgi import Forbidden, NotFound, Response
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/957#issuecomment-683357092
issue_url: https://api.github.com/repos/simonw/datasette/issues/957
node_id: MDEyOklzc3VlQ29tbWVudDY4MzM1NzA5Mg==
created_at: 2020-08-30T00:15:51Z
updated_at: 2020-08-30T00:16:02Z
author_association: OWNER
body:
  Weirdly even removing this single `datasette` import from `utils/asgi.py` didn't fix the circular import:

  https://github.com/simonw/datasette/blob/44cf424a94a85b74552075272660bb96a7432661/datasette/utils/asgi.py#L1-L3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/145#issuecomment-683382252
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/145
node_id: MDEyOklzc3VlQ29tbWVudDY4MzM4MjI1Mg==
created_at: 2020-08-30T06:27:25Z
updated_at: 2020-08-30T06:27:52Z
author_association: CONTRIBUTOR
body:
  Note: had to adjust the test above because trying to exhaust a `SQLITE_MAX_VARIABLE_NUMBER` of 250000 in 99 records requires 2526 columns, and trips the ` "Rows can have a maximum of {} columns".format(SQLITE_MAX_VARS)` check even before it trips the default `SQLITE_MAX_COLUMN` value (2000).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/948#issuecomment-683445114
issue_url: https://api.github.com/repos/simonw/datasette/issues/948
node_id: MDEyOklzc3VlQ29tbWVudDY4MzQ0NTExNA==
created_at: 2020-08-30T17:06:39Z
updated_at: 2020-08-30T17:06:39Z
author_association: OWNER
body:
  Minifying using `npx`:
  ```
  npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js
  npx uglify-js codemirror-5.57.0-sql.js -o codemirror-5.57.0-sql.min.js
  npx clean-css-cli codemirror-5.57.0.css -o codemirror-5.57.0.min.css
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/948#issuecomment-683445704
issue_url: https://api.github.com/repos/simonw/datasette/issues/948
node_id: MDEyOklzc3VlQ29tbWVudDY4MzQ0NTcwNA==
created_at: 2020-08-30T17:11:58Z
updated_at: 2020-08-30T17:33:30Z
author_association: OWNER
body:
  One catch: this stripped the license information from the top of the JS.

  I fixed this by editing the license to be a single `/* ... */` block comment instead of multiple `//` lines and running this:

      npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/948#issuecomment-683448569
issue_url: https://api.github.com/repos/simonw/datasette/issues/948
node_id: MDEyOklzc3VlQ29tbWVudDY4MzQ0ODU2OQ==
created_at: 2020-08-30T17:39:09Z
updated_at: 2020-08-30T18:34:34Z
author_association: OWNER
body:
  So the steps needed are:

  - Download and extract latest CodeMirror zip file
  - Rename `lib/codemirror.js` to `codemirror-5.57.0.js`
  - Rename `lib/codemirror.css` to `codemirror-5.57.0.css`
  - Rename `mode/sql/sql.js` to `codemirror-5.57.0-sql.js`
  - Edit both JS files to make the top comment a `/* */` block
  - Minify JavaScript files like this:
    - `npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'`
    - `npx uglify-js codemirror-5.57.0-sql.js -o codemirror-5.57.0-sql.min.js --comments '/LICENSE/'`
  - Check that the LICENSE comment did indeed survive minification
  - Minify CSS file like this:
    - `npx clean-css-cli codemirror-5.57.0.css -o codemirror-5.57.0.min.css`
  - Edit the `_codemirror.html` template to reference the new files
  - `git rm` the old files, `git add` the new files

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/948#issuecomment-683448635
issue_url: https://api.github.com/repos/simonw/datasette/issues/948
node_id: MDEyOklzc3VlQ29tbWVudDY4MzQ0ODYzNQ==
created_at: 2020-08-30T17:39:54Z
updated_at: 2020-08-30T17:39:54Z
author_association: OWNER
body:
  I'll wait for this to deploy to https://latest.datasette.io/ and then test it in various desktop and mobile browsers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/948#issuecomment-683452613
issue_url: https://api.github.com/repos/simonw/datasette/issues/948
node_id: MDEyOklzc3VlQ29tbWVudDY4MzQ1MjYxMw==
created_at: 2020-08-30T18:16:28Z
updated_at: 2020-08-30T18:16:28Z
author_association: OWNER
body:
  I added documentation on how to upgrade CodeMirror for the future here: https://docs.datasette.io/en/latest/contributing.html#upgrading-codemirror

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/147#issuecomment-683528149
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/147
node_id: MDEyOklzc3VlQ29tbWVudDY4MzUyODE0OQ==
created_at: 2020-08-31T03:17:26Z
updated_at: 2020-08-31T03:17:26Z
author_association: OWNER
body:
  +1 to making this something that users can customize. An optional argument to the `Database` constructor would be a neat way to do this.

  I think there's a terrifying way that we could find this value... we could perform a binary search for it! Open up a memory connection and try running different bulk inserts against it and catch the exceptions - then adjust and try again.

  My hunch is that we could perform just 2 or 3 probes (maybe against carefully selected values) to find the highest value that works. If this process took less than a few ms to run I'd be happy to do it automatically when the class is instantiated (and let users disable that automatic proving by setting a value using the constructor argument).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/4#issuecomment-684395444
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDY4NDM5NTQ0NA==
created_at: 2020-09-01T06:00:03Z
updated_at: 2020-09-01T06:00:03Z
author_association: MEMBER
body:
  I ran `sqlite-utils optimize beta.db` against my test DB and the size reduced from 183M to 176M - and a 450ms search ran in 359ms. So not a huge improvement but still worthwhile.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/pocket-to-sqlite/issues/3#issuecomment-684424396
issue_url: https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3
node_id: MDEyOklzc3VlQ29tbWVudDY4NDQyNDM5Ng==
created_at: 2020-09-01T06:17:45Z
updated_at: 2020-09-01T06:17:45Z
author_association: MEMBER
body:
  It looks like I could ignore the `image` column and synthesize a unique key from the data in the `images` column using `$item_id/$image_id`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685115519
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2
node_id: MDEyOklzc3VlQ29tbWVudDY4NTExNTUxOQ==
created_at: 2020-09-01T20:31:57Z
updated_at: 2020-09-01T20:31:57Z
author_association: MEMBER
body:
  Actually this doesn't work: you can't turn on stemming for specific tables, because all of the content goes into a single `search_index` table which is configured the same way.

  So stemming needs to be a global option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685121074
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2
node_id: MDEyOklzc3VlQ29tbWVudDY4NTEyMTA3NA==
created_at: 2020-09-01T20:42:00Z
updated_at: 2020-09-01T20:42:00Z
author_association: MEMBER
body:
  Documentation at the bottom of the Usage section here: https://github.com/dogsheep/dogsheep-beta/blob/0.2/README.md#usage

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685895540
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7
node_id: MDEyOklzc3VlQ29tbWVudDY4NTg5NTU0MA==
created_at: 2020-09-02T17:46:44Z
updated_at: 2020-09-02T17:46:44Z
author_association: MEMBER
body:
  Some opet questions about this:

  - Should I restrict to two exclusive categories here, or should I have a generic category mechanism that can be expanded to more than two?
  - Should an item be able to exist in more than one category? Do I want to be able to mark an indexed item as both by-me and liked-by-me for example? This question is more interesting if the number of categories is greater than two.
  - How should this be modeled? Single column, multiple boolean columns, JSON array, m2m against separate table?
  - What's the best way to make this performant

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/8#issuecomment-685960072
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/8
node_id: MDEyOklzc3VlQ29tbWVudDY4NTk2MDA3Mg==
created_at: 2020-09-02T19:50:47Z
updated_at: 2020-09-02T19:50:47Z
author_association: MEMBER
body:
  This doesn't actually help, because the Datasette table view page doesn't then support adding the `where search_index_fts match :query` bit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-685961809
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3
node_id: MDEyOklzc3VlQ29tbWVudDY4NTk2MTgwOQ==
created_at: 2020-09-02T19:54:24Z
updated_at: 2020-09-02T19:54:24Z
author_association: MEMBER
body:
  This should implement search highlighting too, as seen on https://til.simonwillison.net/til/search?q=cloud

  <img width="1029" alt="TIL_search__cloud" src="https://user-images.githubusercontent.com/9599/92029959-32c6a300-ed1b-11ea-8b5e-971950980c38.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685962280
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7
node_id: MDEyOklzc3VlQ29tbWVudDY4NTk2MjI4MA==
created_at: 2020-09-02T19:55:26Z
updated_at: 2020-09-02T19:59:58Z
author_association: MEMBER
body:
  Relevant: https://charlesleifer.com/blog/a-tour-of-tagging-schemas-many-to-many-bitmaps-and-more/

  SQLite supports bitwise operators Binary AND (&) and Binary OR (|) - I could try those. Not sure how they interact with indexes though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685965516
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7
node_id: MDEyOklzc3VlQ29tbWVudDY4NTk2NTUxNg==
created_at: 2020-09-02T20:01:54Z
updated_at: 2020-09-02T20:01:54Z
author_association: MEMBER
body:
  Relevant post: https://sqlite.org/forum/forumpost/9f06fedaa5 - drh says:

  > Indexes are one-to-one. There is one entry in the index for each row in the table.
  >
  > You are asking for an index that is many-to-one - multiple index entries for each table row.
  >
  > A Full-Text Index is basically a many-to-one index. So if all of your array entries really are words, you could probably get this to work using a Full-Text Index.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966361
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7
node_id: MDEyOklzc3VlQ29tbWVudDY4NTk2NjM2MQ==
created_at: 2020-09-02T20:03:29Z
updated_at: 2020-09-02T20:03:41Z
author_association: MEMBER
body:
  I'm going to implement the first version of this as an indexed integer `category` column which has 1 for "about me" and 2 for "liked by me" - and space for other category numerals in the future, albeit a row can only belong to one category.

  I'll think about a full tagging system separately.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966707
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7
node_id: MDEyOklzc3VlQ29tbWVudDY4NTk2NjcwNw==
created_at: 2020-09-02T20:04:08Z
updated_at: 2020-09-02T20:04:08Z
author_association: MEMBER
body:
  I'll make `category` a foreign key to a `categories` table so Datasette can automatically show the `name` column.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686153967
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDY4NjE1Mzk2Nw==
created_at: 2020-09-03T00:17:16Z
updated_at: 2020-09-03T00:17:55Z
author_association: MEMBER
body:
  Maybe I can take advantage of https://sqlite.org/np1queryprob.html here - I could define a SQL query for fetching the "display" version of each item, and include a Jinja template fragment in the configuration as well. Maybe something like this:
  ```yaml
  photos.db:
      photos_with_apple_metadata:
          sql: |-
              select
                sha256 as key,
                'Photo in ' || coalesce(place_city, 'unknown') as title,
                (
                  select
                    group_concat(normalized_string, ' ')
                  from
                    labels
                  where
                    labels.uuid = photos_with_apple_metadata.uuid
                ) as search_1,
                date as timestamp,
                1 as category
              from
                photos_with_apple_metadata
          display_sql: |-
              select
                sha256, place_city, date
              from photos_with_apple_metadata
              where sha256 = :key
          display: |-
            <img src="https://photos.simonwillison.net/i/{{ display.sha256 }}.jpeg?w=600">
            <p>Taken in {{ display.place_city }} on {{ display.date }}</p>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686154486
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDY4NjE1NDQ4Ng==
created_at: 2020-09-03T00:18:54Z
updated_at: 2020-09-03T00:18:54Z
author_association: MEMBER
body:
  `display_sql` could be optional. If it's not defined, a `row` object is passed to the template which is the row that's stored in `search_index`. If `display_sql` IS defined then it's executed and the result is made available as a `display` object in addition to the `row` object.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686154627
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDY4NjE1NDYyNw==
created_at: 2020-09-03T00:19:22Z
updated_at: 2020-09-03T00:19:22Z
author_association: MEMBER
body:
  If this performs well enough (100 displayed items will be 100 extra `display_sql` calls) then I'll go with this as the design for the feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686158454
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDY4NjE1ODQ1NA==
created_at: 2020-09-03T00:32:42Z
updated_at: 2020-09-03T00:32:42Z
author_association: MEMBER
body:
  If this turns out to be too inefficient I could add a `display` text column to the `search_index` table which is designed to be populated with arbitrary JSON by the indexing query, which can then be used to render the template fragment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686163754
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDY4NjE2Mzc1NA==
created_at: 2020-09-03T00:46:21Z
updated_at: 2020-09-03T00:46:21Z
author_association: MEMBER
body:
  Challenge: the `dogsheep-beta.yml` configuration file that is passed to the `dogsheep-beta index` command needs to also be made available to Datasette itself, so that it can read the configuration.

  Let's say it can either be duplicated in the `plugins` configuration block of the `metadata.yml` OR you can do this in `metadata.yml`:

  ```yaml
  plugins:
    dogsheep-beta:
      config_file: dogsheep-beta.yml
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/13#issuecomment-686774592
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/13
node_id: MDEyOklzc3VlQ29tbWVudDY4Njc3NDU5Mg==
created_at: 2020-09-03T21:30:21Z
updated_at: 2020-09-03T21:30:21Z
author_association: MEMBER
body:
  This is partially supported: the custom search SQL we run doesn't escape them, but the `?_search` used to calculate facet counts does. So this is a bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/17#issuecomment-687880459
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17
node_id: MDEyOklzc3VlQ29tbWVudDY4Nzg4MDQ1OQ==
created_at: 2020-09-06T19:36:32Z
updated_at: 2020-09-06T19:36:32Z
author_association: MEMBER
body:
  At some point I may even want to support search types which are indexed from (and inflated from) more than one database file. I'm going to ignore that for the moment though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/148#issuecomment-688434226
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/148
node_id: MDEyOklzc3VlQ29tbWVudDY4ODQzNDIyNg==
created_at: 2020-09-07T16:50:33Z
updated_at: 2020-09-07T16:50:33Z
author_association: OWNER
body:
  This may be as easy as applying `textwrap.dedent()` to this: https://github.com/simonw/sqlite-utils/blob/0e62744da9a429093e3409575c1f881376b0361f/sqlite_utils/db.py#L778-L787

  I could apply that to a few other queries in that code as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688460729
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/149
node_id: MDEyOklzc3VlQ29tbWVudDY4ODQ2MDcyOQ==
created_at: 2020-09-07T18:06:44Z
updated_at: 2020-09-07T18:06:44Z
author_association: OWNER
body:
  First posted on SQLite forum here but I'm pretty sure this is a bug in how `sqlite-utils` created those tables: https://sqlite.org/forum/forumpost/51aada1b45

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688460865
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/149
node_id: MDEyOklzc3VlQ29tbWVudDY4ODQ2MDg2NQ==
created_at: 2020-09-07T18:07:14Z
updated_at: 2020-09-07T18:07:14Z
author_association: OWNER
body:
  Another likely culprit: `licenses` has a text primary key, so it's not using `rowid`:
  ```sql
  CREATE TABLE [licenses] (
     [key] TEXT PRIMARY KEY,
     [name] TEXT,
     [spdx_id] TEXT,
     [url] TEXT,
     [node_id] TEXT
  );
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688464181
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/149
node_id: MDEyOklzc3VlQ29tbWVudDY4ODQ2NDE4MQ==
created_at: 2020-09-07T18:19:54Z
updated_at: 2020-09-07T18:19:54Z
author_association: OWNER
body:
  Even though that table doesn't declare an integer primary key it does have a `rowid` column: https://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+%5Bkey%5D%2C+name%2C+spdx_id%2C+url%2C+node_id+from+licenses+order+by+%5Bkey%5D+limit+101

  | rowid | key | name | spdx_id | url | node_id |
  | --- | --- | --- | --- | --- | --- |
  | 9150 | apache-2.0 | Apache License 2.0 | Apache-2.0 | <https://api.github.com/licenses/apache-2.0> | MDc6TGljZW5zZTI= |
  | 112 | bsd-3-clause | BSD 3-Clause "New" or "Revised" License | BSD-3-Clause | <https://api.github.com/licenses/bsd-3-clause> | MDc6TGljZW5zZTU= |

  https://www.sqlite.org/rowidtable.html explains has this clue:

  >  If the rowid is not aliased by INTEGER PRIMARY KEY then it is not persistent and might change. In particular the VACUUM command will change rowids for tables that do not declare an INTEGER PRIMARY KEY. Therefore, applications should not normally access the rowid directly, but instead use an INTEGER PRIMARY KEY. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688479163
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/146
node_id: MDEyOklzc3VlQ29tbWVudDY4ODQ3OTE2Mw==
created_at: 2020-09-07T19:10:33Z
updated_at: 2020-09-07T19:11:57Z
author_association: CONTRIBUTOR
body:
  @simonw -- I've gone ahead updated the documentation to reflect the changes introduced in this PR.  IMO it's ready to merge now.

  In writing the documentation changes, I begin to wonder about the value and role of `batch_size` at all, tbh.  May I assume it was originally intended to prevent using the entire row set to determine columns and column types, and that this was a performance consideration?  If so, this PR entirely undermines its purpose.  I've been passing in excess of 500,000 rows at a time to `insert_all()` with these changes and although I'm sure the performance difference is measurable it's not really noticeable; given #145, I don't know that any performance advantages outweigh the problems doing it this way removes.  What do you think about just dropping the argument and defaulting to the maximum `batch_size` permissible given `SQLITE_MAX_VARS`?  Are there other reasons one might want to restrict `batch_size` that I've overlooked?  I could open a new issue to discuss/implement this.

  Of course the documentation will need to change again too if/when something is done about #147.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688480665
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/149
node_id: MDEyOklzc3VlQ29tbWVudDY4ODQ4MDY2NQ==
created_at: 2020-09-07T19:16:20Z
updated_at: 2020-09-07T19:16:20Z
author_association: OWNER
body:
  Aha! I have managed to replicate the bug:
  ```
  (github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses
   {"table": "licenses", "count": 7},
   {"table": "licenses_fts_data", "count": 35},
   {"table": "licenses_fts_idx", "count": 16},
   {"table": "licenses_fts_docsize", "count": 9151},
   {"table": "licenses_fts_config", "count": 1},
   {"table": "licenses_fts", "count": 7},
  (github-to-sqlite) /tmp % github-to-sqlite repos github.db dogsheep       
  (github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses
   {"table": "licenses", "count": 7},
   {"table": "licenses_fts_data", "count": 45},
   {"table": "licenses_fts_idx", "count": 26},
   {"table": "licenses_fts_docsize", "count": 9161},
   {"table": "licenses_fts_config", "count": 1},
   {"table": "licenses_fts", "count": 7},
  ```
  Note that the number of records in `licenses_fts_docsize` went from 9151 to 9161.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688481317
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/146
node_id: MDEyOklzc3VlQ29tbWVudDY4ODQ4MTMxNw==
created_at: 2020-09-07T19:18:55Z
updated_at: 2020-09-07T19:18:55Z
author_association: CONTRIBUTOR
body:
  Just force-pushed to update d042f9c with more formatting changes to satisfy `black==20.8b1` and pass the GitHub Actions "Test" workflow.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688481374
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/149
node_id: MDEyOklzc3VlQ29tbWVudDY4ODQ4MTM3NA==
created_at: 2020-09-07T19:19:08Z
updated_at: 2020-09-07T19:19:08Z
author_association: OWNER
body:
  reading through the code for `github-to-sqlite repos` - one of the things it does is calls `save_license` for each repo:

  https://github.com/dogsheep/github-to-sqlite/blob/39b2234253096bd579feed4e25104698b8ccd2ba/github_to_sqlite/utils.py#L259-L262

  ```python
  def save_license(db, license):
      if license is None:
          return None
      return db["licenses"].insert(license, pk="key", replace=True).last_pk
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688482055
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/149
node_id: MDEyOklzc3VlQ29tbWVudDY4ODQ4MjA1NQ==
created_at: 2020-09-07T19:21:42Z
updated_at: 2020-09-07T19:21:42Z
author_association: OWNER
body:
  Using `replace=True` there executes `INSERT OR REPLACE` - and Dan Kennedy (SQLite maintainer) on the SQLite forums said this:
  > Are you using "REPLACE INTO", or "UPDATE OR REPLACE" on the "licenses" table without having first executed "PRAGMA recursive_triggers = 1"? The docs note that delete triggers will not be fired in this case, which would explain things. Second paragraph under "REPLACE" here:
  >
  > https://www.sqlite.org/lang_conflict.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688482355
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/149
node_id: MDEyOklzc3VlQ29tbWVudDY4ODQ4MjM1NQ==
created_at: 2020-09-07T19:22:51Z
updated_at: 2020-09-07T19:22:51Z
author_association: OWNER
body:
  And the SQLite documentation says:
  > When the REPLACE conflict resolution strategy deletes rows in order to satisfy a constraint, [delete triggers](https://www.sqlite.org/lang_createtrigger.html) fire if and only if [recursive triggers](https://www.sqlite.org/pragma.html#pragma_recursive_triggers) are enabled.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499650
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/149
node_id: MDEyOklzc3VlQ29tbWVudDY4ODQ5OTY1MA==
created_at: 2020-09-07T20:24:35Z
updated_at: 2020-09-07T20:24:35Z
author_association: OWNER
body:
  This replicates the problem:
  ```
  (github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses
   {"table": "licenses", "count": 7},
   {"table": "licenses_fts_data", "count": 35},
   {"table": "licenses_fts_idx", "count": 16},
   {"table": "licenses_fts_docsize", "count": 9151},
   {"table": "licenses_fts_config", "count": 1},
   {"table": "licenses_fts", "count": 7},
  (github-to-sqlite) /tmp % github-to-sqlite repos github.db dogsheep       
  (github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses
   {"table": "licenses", "count": 7},
   {"table": "licenses_fts_data", "count": 45},
   {"table": "licenses_fts_idx", "count": 26},
   {"table": "licenses_fts_docsize", "count": 9161},
   {"table": "licenses_fts_config", "count": 1},
   {"table": "licenses_fts", "count": 7},
  ```
  Note how the number of rows in `licenses_fts_docsize` goes from 9151 to 9161.

  The number went up by ten. I used tracing from #151 to show that the following SQL executed ten times:
  ```
  INSERT OR REPLACE INTO [licenses] ([key], [name], [node_id], [spdx_id], [url]) VALUES 
      (?, ?, ?, ?, ?);
  ```
  Then I tried executing `PRAGMA recursive_triggers=on;` at the start of the script. This fixed the problem - running the script did not increase the number of rows in `licenses_fts_docsize`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499924
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/149
node_id: MDEyOklzc3VlQ29tbWVudDY4ODQ5OTkyNA==
created_at: 2020-09-07T20:25:40Z
updated_at: 2020-09-07T20:25:50Z
author_association: OWNER
body:
  https://www.sqlite.org/pragma.html#pragma_recursive_triggers says:

  > Prior to SQLite [version 3.6.18](https://www.sqlite.org/releaselog/3_6_18.html) (2009-09-11), recursive triggers were not supported. The behavior of SQLite was always as if this pragma was set to OFF. Support for recursive triggers was added in version 3.6.18 but was initially turned OFF by default, for compatibility. Recursive triggers may be turned on by default in future versions of SQLite.

  So I think the fix is to turn on `recursive_triggers` globally by default for `sqlite-utils`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/152#issuecomment-688500704
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/152
node_id: MDEyOklzc3VlQ29tbWVudDY4ODUwMDcwNA==
created_at: 2020-09-07T20:28:45Z
updated_at: 2020-09-07T21:17:48Z
author_association: OWNER
body:
  The principle reason to turn these on - at least so far - is that without it weird things happen where FTS tables (in particular `*_fts_docsize`) grow without limit over time, because calls to `INSERT OR REPLACE` against the parent table cause additional rows to be inserted into `*_fts_docsize` even if the row was replaced rather than being inserted.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688501064
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/149
node_id: MDEyOklzc3VlQ29tbWVudDY4ODUwMTA2NA==
created_at: 2020-09-07T20:30:15Z
updated_at: 2020-09-07T20:30:38Z
author_association: OWNER
body:
  The second challenge here is cleaning up all of those junk rows in existing `*_fts_docsize` tables. Doing that just to the demo database from https://github-to-sqlite.dogsheep.net/github.db dropped its size from 22MB to 16MB! Here's the SQL:
  ```sql
  DELETE FROM [licenses_fts_docsize] WHERE id NOT IN (
    SELECT rowid FROM [licenses_fts]);
  ```
  I can do that as part of the existing `table.optimize()` method, which optimizes FTS tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/153#issuecomment-688506015
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/153
node_id: MDEyOklzc3VlQ29tbWVudDY4ODUwNjAxNQ==
created_at: 2020-09-07T20:46:58Z
updated_at: 2020-09-07T20:46:58Z
author_association: OWNER
body:
  Writing a test for this will be a tiny bit tricky. I think I'll use a test that replicates the bug in #149.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688508510
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/146
node_id: MDEyOklzc3VlQ29tbWVudDY4ODUwODUxMA==
created_at: 2020-09-07T20:56:03Z
updated_at: 2020-09-07T20:56:24Z
author_association: OWNER
body:
  The problem with this approach is that it requires us to consume the entire iterator before we can start inserting rows into the table - here on line 1052:

  https://github.com/simonw/sqlite-utils/blob/bb131793feac16bc7181ab997568f941b0220ef2/sqlite_utils/db.py#L1047-L1054

  I designed the `.insert_all()` to avoid doing this, because I want to be able to pass it an iterator (or more likely a generator) that could produce potentially millions of records. Doing things one batch of 100 records at a time means that the Python process doesn't need to pull millions of records into memory at once.

  `db-to-sqlite` is one example of a tool that uses that characteristic, in https://github.com/simonw/db-to-sqlite/blob/63e4ee972f292de13bb11767c0fb64b35339d954/db_to_sqlite/cli.py#L94-L106

  So we need to solve this issue without consuming the entire iterator with a `records = list(records)` call.

  I think one way to do this is to execute each chunk one at a time and watch out for an exception that indicates that we sent too many parameters - then adjust the chunk size down and try again.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/153#issuecomment-688511161
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/153
node_id: MDEyOklzc3VlQ29tbWVudDY4ODUxMTE2MQ==
created_at: 2020-09-07T21:07:20Z
updated_at: 2020-09-07T21:07:29Z
author_association: OWNER
body:
  FTS4 uses a different column name here: https://datasette-sqlite-fts4.datasette.io/24ways-fts4/articles_fts_docsize

  ```
  CREATE TABLE 'articles_fts_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/154#issuecomment-688543128
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/154
node_id: MDEyOklzc3VlQ29tbWVudDY4ODU0MzEyOA==
created_at: 2020-09-07T23:43:10Z
updated_at: 2020-09-07T23:43:10Z
author_association: OWNER
body:
  Running this against the same file works:
  ```
  $ sqlite3 beta.db 
  SQLite version 3.31.1 2020-01-27 19:55:54
  Enter ".help" for usage hints.
  sqlite> PRAGMA journal_mode=wal;
  wal
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/154#issuecomment-688544156
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/154
node_id: MDEyOklzc3VlQ29tbWVudDY4ODU0NDE1Ng==
created_at: 2020-09-07T23:47:10Z
updated_at: 2020-09-07T23:47:10Z
author_association: OWNER
body:
  This is already covered in the tests though: https://github.com/simonw/sqlite-utils/blob/deb2eb013ff85bbc828ebc244a9654f0d9c3139e/tests/test_cli.py#L1300-L1328

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688573964
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/146
node_id: MDEyOklzc3VlQ29tbWVudDY4ODU3Mzk2NA==
created_at: 2020-09-08T01:55:07Z
updated_at: 2020-09-08T01:55:07Z
author_association: CONTRIBUTOR
body:
  Okay, I've rewritten this PR to preserve the batching behaviour but still fix #145, and rebased the branch to account for the `db.execute()` api change.  It's not terribly sophisticated -- if it attempts to insert a batch which has too many variables, the exception is caught, the batch is split in two and each half is inserted separately, and then it carries on as before with the same `batch_size`.  In the edge case where this gets triggered, subsequent batches will all be inserted in two groups too if they continue to have the same number of columns (which is presumably reasonably likely).  Do you reckon this is acceptable when set against the awkwardness of recalculating the `batch_size` on the fly?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/18#issuecomment-688622995
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18
node_id: MDEyOklzc3VlQ29tbWVudDY4ODYyMjk5NQ==
created_at: 2020-09-08T05:15:21Z
updated_at: 2020-09-08T05:15:21Z
author_association: MEMBER
body:
  Alternatively it could run as it does now but add a `DELETE FROM index1.search_index WHERE key not in (select key from ...)`.

  I'm not sure which would be more efficient.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/18#issuecomment-688623097
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18
node_id: MDEyOklzc3VlQ29tbWVudDY4ODYyMzA5Nw==
created_at: 2020-09-08T05:15:51Z
updated_at: 2020-09-08T05:15:51Z
author_association: MEMBER
body:
  I'm inclined to go with the first, simpler option. I have longer term plans for efficient incremental index updates based on clever trickery with triggers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/19#issuecomment-688625430
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19
node_id: MDEyOklzc3VlQ29tbWVudDY4ODYyNTQzMA==
created_at: 2020-09-08T05:24:50Z
updated_at: 2020-09-08T05:24:50Z
author_association: MEMBER
body:
  I thought about allowing tables to define a incremental indexing SQL query - maybe something that can return just records touched in the past hour, or records since a recorded "last indexed record" value.

  The problem with this is deletes - if you delete a record, how does the indexer know to remove it? See #18  - that's already caused problems.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/19#issuecomment-688626037
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19
node_id: MDEyOklzc3VlQ29tbWVudDY4ODYyNjAzNw==
created_at: 2020-09-08T05:27:07Z
updated_at: 2020-09-08T05:27:07Z
author_association: MEMBER
body:
  A really clever way to do this would be with triggers. The indexer script would add triggers to each of the database tables that it is indexing - each in their own database.

  Those triggers would then maintain a `_index_queue_` table. This table would record the primary key of rows that are added, modified or deleted. The indexer could then work by reading through the `_index_queue_` table, re-indexing (or deleting) just the primary keys listed there, and then emptying the queue once it has finished.

  This would add a small amount of overhead to insert/update/delete queries run against the table. My hunch is that the overhead would be miniscule, but I could still allow people to opt-out for tables that are so high traffic that this would matter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/155#issuecomment-689163158
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/155
node_id: MDEyOklzc3VlQ29tbWVudDY4OTE2MzE1OA==
created_at: 2020-09-08T22:10:27Z
updated_at: 2020-09-08T22:10:27Z
author_association: OWNER
body:
  For the command version:

      sqlite-utils rebuild-fts mydb.db

  This will rebuild all detected FTS tables. You can also specify one or more explicit tables:

      sqlite-utils rebuild-fts mydb.db dogs

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/153#issuecomment-689165985
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/153
node_id: MDEyOklzc3VlQ29tbWVudDY4OTE2NTk4NQ==
created_at: 2020-09-08T22:18:52Z
updated_at: 2020-09-08T22:18:52Z
author_association: OWNER
body:
  I've reverted this change again, because it turns out using the `rebuild` FTS mechanism is a better way of repairing this issue - see #155.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/155#issuecomment-689166404
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/155
node_id: MDEyOklzc3VlQ29tbWVudDY4OTE2NjQwNA==
created_at: 2020-09-08T22:20:03Z
updated_at: 2020-09-08T22:20:03Z
author_association: OWNER
body:
  I'm going to update `sqlite-utils optimize` to also take an optional list of tables, for consistency.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/146#issuecomment-689185393
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/146
node_id: MDEyOklzc3VlQ29tbWVudDY4OTE4NTM5Mw==
created_at: 2020-09-08T23:17:42Z
updated_at: 2020-09-08T23:17:42Z
author_association: OWNER
body:
  That seems like a reasonable approach to me, especially since this is going to be a pretty rare edge-case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/17#issuecomment-689226390
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17
node_id: MDEyOklzc3VlQ29tbWVudDY4OTIyNjM5MA==
created_at: 2020-09-09T00:36:07Z
updated_at: 2020-09-09T00:36:07Z
author_association: MEMBER
body:
  Alternative names:

  - type
  - record_type
  - doctype

  I think `type` is right. It matches what Elasticsearch used to call their equivalent of this (before they removed the feature!). https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/961#issuecomment-689635094
issue_url: https://api.github.com/repos/simonw/datasette/issues/961
node_id: MDEyOklzc3VlQ29tbWVudDY4OTYzNTA5NA==
created_at: 2020-09-09T15:23:24Z
updated_at: 2020-09-09T15:23:24Z
author_association: OWNER
body:
  Checks can include:

  - `facets:` lists columns that exist
  - `sort:` and `sort_desc:` columns
  - `fts_table` and `fts_pk` are valid

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/961#issuecomment-689635754
issue_url: https://api.github.com/repos/simonw/datasette/issues/961
node_id: MDEyOklzc3VlQ29tbWVudDY4OTYzNTc1NA==
created_at: 2020-09-09T15:24:31Z
updated_at: 2020-09-09T15:24:31Z
author_association: OWNER
body:
  I thought about checking that every database in the `databases:` section exists and ditto for `tables:` - but actually I think it's useful to be able to keep a `metadata.yml` around with configuration for databases or tables that aren't currently attached to Datasette.

  I could treat those as warnings and output a warning to standard out when the server starts instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/157#issuecomment-689850289
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/157
node_id: MDEyOklzc3VlQ29tbWVudDY4OTg1MDI4OQ==
created_at: 2020-09-09T22:14:19Z
updated_at: 2020-09-09T22:14:19Z
author_association: OWNER
body:
  This can accept four arguments: table, column, other_table, other_column:
  ```
  sqlite-utils add-foreign-keys calands.db \
    units_with_maps ACCESS_TYP ACCESS_TYP id \
    units_with_maps AGNCY_NAME AGNCY_NAME id \
    units_with_maps AGNCY_LEV AGNCY_LEV id
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/157#issuecomment-689850509
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/157
node_id: MDEyOklzc3VlQ29tbWVudDY4OTg1MDUwOQ==
created_at: 2020-09-09T22:14:49Z
updated_at: 2020-09-09T22:14:49Z
author_association: OWNER
body:
  It will call this method: https://github.com/simonw/sqlite-utils/blob/367082e787101fb90901ef3214804ab23a92ce46/sqlite_utils/db.py#L405-L411

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/50#issuecomment-690860653
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50
node_id: MDEyOklzc3VlQ29tbWVudDY5MDg2MDY1Mw==
created_at: 2020-09-11T04:04:08Z
updated_at: 2020-09-11T04:04:08Z
author_association: CONTRIBUTOR
body:
  There's probably a nicer way of doing (hence this is a comment rather than a PR), but this appears to fix it:
  ```diff
  --- a/twitter_to_sqlite/utils.py
  +++ b/twitter_to_sqlite/utils.py
  @@ -181,6 +181,7 @@ def fetch_timeline(
       args["tweet_mode"] = "extended"
       min_seen_id = None
       num_rate_limit_errors = 0
  +    seen_count = 0
       while True:
           if min_seen_id is not None:
               args["max_id"] = min_seen_id - 1
  @@ -208,6 +209,7 @@ def fetch_timeline(
               yield tweet
           min_seen_id = min(t["id"] for t in tweets)
           max_seen_id = max(t["id"] for t in tweets)
  +        seen_count += len(tweets)
           if last_since_id is not None:
               max_seen_id = max((last_since_id, max_seen_id))
               last_since_id = max_seen_id
  @@ -217,7 +219,9 @@ def fetch_timeline(
                   replace=True,
               )
           if stop_after is not None:
  -            break
  +            if seen_count >= stop_after:
  +                break
  +            args["count"] = min(args["count"], stop_after - seen_count)
           time.sleep(sleep)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-691323302
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDY5MTMyMzMwMg==
created_at: 2020-09-11T21:38:27Z
updated_at: 2020-09-11T21:40:04Z
author_association: OWNER
body:
  Another idea: the default output could be the list of dicts:
  ```json
  [
    {
      "pk1": "a",
      "pk2": "a",
      "pk3": "a",
      "content": "a-a-a"
    },
    ...
  ]
  ```
  BUT... I could include pagination information in the HTTP headers - as seen in the WordPress REST API or the GitHub API:

  ```
  ~ % curl -s -i 'https://api.github.com/repos/simonw/datasette/commits' | head -n 40
  HTTP/1.1 200 OK
  server: GitHub.com
  date: Fri, 11 Sep 2020 21:37:46 GMT
  content-type: application/json; charset=utf-8
  status: 200 OK
  cache-control: public, max-age=60, s-maxage=60
  vary: Accept, Accept-Encoding, Accept, X-Requested-With
  etag: W/"71c99379743513394e880c6306b66bf9"
  last-modified: Fri, 11 Sep 2020 21:32:54 GMT
  x-github-media-type: github.v3; format=json
  link: <https://api.github.com/repositories/107914493/commits?page=2>; rel="next", <https://api.github.com/repositories/107914493/commits?page=44>; rel="last"
  access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
  access-control-allow-origin: *
  strict-transport-security: max-age=31536000; includeSubdomains; preload
  x-frame-options: deny
  x-content-type-options: nosniff
  x-xss-protection: 1; mode=block
  referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
  content-security-policy: default-src 'none'
  X-Ratelimit-Limit: 60
  X-Ratelimit-Remaining: 55
  X-Ratelimit-Reset: 1599863850
  X-Ratelimit-Used: 5
  Accept-Ranges: bytes
  Content-Length: 118240
  X-GitHub-Request-Id: EC76:0EAD:313F40:5291A4:5F5BEE37

  [
    {
      "sha": "d02f6151dae073135a22d0123e8abdc6cbef7c50",
      "node_id": "MDY6Q29tbWl0MTA3OTE0NDkzOmQwMmY2MTUxZGFlMDczMTM1YTIyZDAxMjNlOGFiZGM2Y2JlZjdjNTA=",
      "commit": {
  ```
  Alternative shapes would provide the pagination information (and other extensions) in the JSON, e.g.:

  `/squirrels/squirrels.json?_shape=paginated`
  ```json
  {
    "rows": [
    {
      "pk1": "a",
      "pk2": "a",
      "pk3": "a",
      "content": "a-a-a"
    }
    ],
    "pagination": {
      "next": "234",
      "count": 442
    }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/50#issuecomment-691501132
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50
node_id: MDEyOklzc3VlQ29tbWVudDY5MTUwMTEzMg==
created_at: 2020-09-12T14:48:10Z
updated_at: 2020-09-12T14:48:10Z
author_association: NONE
body:
  This seems to be an issue even with larger values of `--stop_after`:

  ```
  $ twitter-to-sqlite favorites twitter.db --stop_after=2000
  Importing favorites  [####################################]  198
  $
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-691526416
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDY5MTUyNjQxNg==
created_at: 2020-09-12T18:16:36Z
updated_at: 2020-09-12T18:16:36Z
author_association: OWNER
body:
  I'm going to hack together a preview of this in a branch and deploy it somewhere so people can see what I've got planned. Much easier to evaluate a working prototype than static examples.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/680#issuecomment-691526635
issue_url: https://api.github.com/repos/simonw/datasette/issues/680
node_id: MDEyOklzc3VlQ29tbWVudDY5MTUyNjYzNQ==
created_at: 2020-09-12T18:18:50Z
updated_at: 2020-09-12T18:18:50Z
author_association: OWNER
body:
  I'm happy with the not-quite-automated way I'm doing this, so I'm going to close this issue.

  That's documented here https://docs.datasette.io/en/0.48/contributing.html#release-process - I use https://euangoddard.github.io/clipboard2markdown/ to create the GitHub releases markdown version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-691526719
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: MDEyOklzc3VlQ29tbWVudDY5MTUyNjcxOQ==
created_at: 2020-09-12T18:19:50Z
updated_at: 2020-09-12T18:19:50Z
author_association: OWNER
body:
  > Idea: `?_extra=sqllog` could output a lot of every individual SQL statement that was executed in order to generate the page - useful for seeing how foreign key expansion and faceting actually works.

  I built a version of that a while ago as the `?_trace=1` argument.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-691526878
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDY5MTUyNjg3OA==
created_at: 2020-09-12T18:21:41Z
updated_at: 2020-09-12T18:22:20Z
author_association: OWNER
body:
  Would it be so bad if the default format had a `"rows"` key containing the array of rows? Maybe it wouldn't. The reason I always use `?_shape=array` is because I want an array of objects, rather than an array of arrays that I have to match up again with their columns.

  A default format that's an object rather than array also gives something for the `?_extra=` parameter to add its extras to.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-691526975
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: MDEyOklzc3VlQ29tbWVudDY5MTUyNjk3NQ==
created_at: 2020-09-12T18:22:44Z
updated_at: 2020-09-12T18:22:44Z
author_association: OWNER
body:
  Are there any interesting use-cases for a plugin hook that allows plugins to define their own `?_extra=` blocks?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-691554088
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDY5MTU1NDA4OA==
created_at: 2020-09-12T21:39:03Z
updated_at: 2020-09-12T21:39:03Z
author_association: OWNER
body:
  Plan: release a new release of Datasette (probably 0.49) with the new JSON API design, but provide a plugin called something like `datasette-api-0-48` which runs as ASGI wrapping middleware and internally rewrites incoming requests to e.g. `/db/table.json` to behave if they have the `?_extra=` params on them necessary to produce the 0.48 version of the JSON.

  Anyone who has built applications against 0.48 can install that plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/880#issuecomment-691557429
issue_url: https://api.github.com/repos/simonw/datasette/issues/880
node_id: MDEyOklzc3VlQ29tbWVudDY5MTU1NzQyOQ==
created_at: 2020-09-12T21:59:39Z
updated_at: 2020-09-12T21:59:39Z
author_association: OWNER
body:
  What should happen when something does a POST to an extension that was registered by a plugin, e.g. `POST /db/table.atom` ?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/880#issuecomment-691557675
issue_url: https://api.github.com/repos/simonw/datasette/issues/880
node_id: MDEyOklzc3VlQ29tbWVudDY5MTU1NzY3NQ==
created_at: 2020-09-12T22:01:02Z
updated_at: 2020-09-12T22:01:11Z
author_association: OWNER
body:
  Maybe POST to `.json` doesn't actually make sense. I could instead support `POST /db/queryname` with an optional mechanism for requesting that the response to that POST be in a JSON format.

  Could be a `Accept: application/json` header with an option of including `"_accept": "json"` as a POST parameter instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/880#issuecomment-691558387
issue_url: https://api.github.com/repos/simonw/datasette/issues/880
node_id: MDEyOklzc3VlQ29tbWVudDY5MTU1ODM4Nw==
created_at: 2020-09-12T22:04:48Z
updated_at: 2020-09-12T22:04:48Z
author_association: OWNER
body:
  Is it safe to skip CSRF checks if the incoming request has `Accept: application/json` on it?

  I'm not sure that matters since `asgi-csrf` already won't reject requests that either have no cookies or are using a `Authorization: Bearer ...` header.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/519#issuecomment-691566247
issue_url: https://api.github.com/repos/simonw/datasette/issues/519
node_id: MDEyOklzc3VlQ29tbWVudDY5MTU2NjI0Nw==
created_at: 2020-09-12T22:48:53Z
updated_at: 2020-09-12T22:48:53Z
author_association: OWNER
body:
  I think I've figured out what to do about stability of the HTML and the default templates with respect to semantic versioning.

  I'm going to announce that the JSON API - including the variables made available to templates - should be considered stable according to semver. I will only break backwards compatibility at that level in a major version release.

  The template HTML (and default CSS) will not be considered a stable interface. They won't change on bug fix releases but they may change (albeit described in the release notes) on minor version bumps.

  Since the template inputs are stable, you can run your own copy of the previous version's templates if something breaks.

  This means users (and plugin authors) who make changes to the default Datasette UI will have to test their changes against every minor release. I think that's OK.

  If you write plugins that don't affect the Datasette HTML UI you will be able to expect stability across minor version releases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/944#issuecomment-691769222
issue_url: https://api.github.com/repos/simonw/datasette/issues/944
node_id: MDEyOklzc3VlQ29tbWVudDY5MTc2OTIyMg==
created_at: 2020-09-14T02:01:33Z
updated_at: 2020-09-14T02:01:33Z
author_association: OWNER
body:
  I'm going to cache the `list_templates()` result in memory. If you want to add a new template-defined route you will need to restart the server. I think that's acceptable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/944#issuecomment-691774262
issue_url: https://api.github.com/repos/simonw/datasette/issues/944
node_id: MDEyOklzc3VlQ29tbWVudDY5MTc3NDI2Mg==
created_at: 2020-09-14T02:24:08Z
updated_at: 2020-09-14T02:24:08Z
author_association: OWNER
body:
  Actually don't need `{{ raise_404("Museum not found") }}` because we already have `{{ custom_status(404) }}`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-691779361
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY5MTc3OTM2MQ==
created_at: 2020-09-14T02:45:04Z
updated_at: 2020-09-14T02:45:04Z
author_association: OWNER
body:
  Package deploys are still broken, just got this error trying to ship 0.49a1: https://github.com/simonw/datasette/actions/runs/253099665

  > The workflow is not valid. .github/workflows/publish.yml (Line: 61, Col: 9): Unexpected symbol: '"a"'. Located at position 24 within expression: !(contains(github.ref, "a") || contains(github.ref, "b"))

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-691779510
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY5MTc3OTUxMA==
created_at: 2020-09-14T02:45:53Z
updated_at: 2020-09-14T02:45:53Z
author_association: OWNER
body:
  This bit here: https://github.com/simonw/datasette/blob/c18117cf08ad67c704dab29e3cb3b88f1de4026b/.github/workflows/publish.yml#L58-L62

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-691781345
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY5MTc4MTM0NQ==
created_at: 2020-09-14T02:53:25Z
updated_at: 2020-09-14T02:53:49Z
author_association: OWNER
body:
  That worked: https://github.com/simonw/datasette/runs/1110040212?check_suite_focus=true ran and deployed https://pypi.org/project/datasette/0.49a1/ to PyPI but it skipped the push to Docker step because there was an "a" in the tag.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/880#issuecomment-691785692
issue_url: https://api.github.com/repos/simonw/datasette/issues/880
node_id: MDEyOklzc3VlQ29tbWVudDY5MTc4NTY5Mg==
created_at: 2020-09-14T03:10:11Z
updated_at: 2020-09-14T03:10:11Z
author_association: OWNER
body:
  Answer: no, it's [not safe](https://twitter.com/glenathan/status/1305081266065244162) to skip CSRF if there's an `Accept: application/json` header because of a nasty old `crossdomain.xml` Flash vulnerability: https://blog.appsecco.com/exploiting-csrf-on-json-endpoints-with-flash-and-redirects-681d4ad6b31b?gi=a5ee3d7a8235

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/944#issuecomment-691788478
issue_url: https://api.github.com/repos/simonw/datasette/issues/944
node_id: MDEyOklzc3VlQ29tbWVudDY5MTc4ODQ3OA==
created_at: 2020-09-14T03:21:45Z
updated_at: 2020-09-14T03:21:45Z
author_association: OWNER
body:
  Having tried this out I think it does need a `raise_404()` mechanism - which needs to be smart enough to trigger the default 404 handler without accidentally going into an infinite loop.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/965#issuecomment-692231257
issue_url: https://api.github.com/repos/simonw/datasette/issues/965
node_id: MDEyOklzc3VlQ29tbWVudDY5MjIzMTI1Nw==
created_at: 2020-09-14T18:25:04Z
updated_at: 2020-09-14T18:25:04Z
author_association: OWNER
body:
  In documenting this I realized that it's confusing that the default `500.html` template is often used for non-500 errors (404 for example). I think I'll rename that default template to `error.html` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/880#issuecomment-692271804
issue_url: https://api.github.com/repos/simonw/datasette/issues/880
node_id: MDEyOklzc3VlQ29tbWVudDY5MjI3MTgwNA==
created_at: 2020-09-14T19:41:37Z
updated_at: 2020-09-14T19:41:37Z
author_association: OWNER
body:
  Relevant code section: https://github.com/simonw/datasette/blob/1552ac931e4d2cf516caac3ceeab4fd24da1510a/datasette/views/database.py#L209-L232

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/880#issuecomment-692272860
issue_url: https://api.github.com/repos/simonw/datasette/issues/880
node_id: MDEyOklzc3VlQ29tbWVudDY5MjI3Mjg2MA==
created_at: 2020-09-14T19:43:47Z
updated_at: 2020-09-14T19:43:47Z
author_association: OWNER
body:
  I'm going to add support for POST content that is sent as a JSON document, in addition to the existing support for key=value encoded POST bodies.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/880#issuecomment-692298011
issue_url: https://api.github.com/repos/simonw/datasette/issues/880
node_id: MDEyOklzc3VlQ29tbWVudDY5MjI5ODAxMQ==
created_at: 2020-09-14T20:33:13Z
updated_at: 2020-09-14T20:33:13Z
author_association: OWNER
body:
  I'm going to support several ways of indicating that you would like a JSON response instead of getting a HTTP redirect from your writable canned query submission:

  - Use the `Accept: application/json` request header
  - Include `?_json=1` in the request query string
  - Include `"_json": 1` in the form submission (or the JSON body submission)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/880#issuecomment-692299770
issue_url: https://api.github.com/repos/simonw/datasette/issues/880
node_id: MDEyOklzc3VlQ29tbWVudDY5MjI5OTc3MA==
created_at: 2020-09-14T20:36:40Z
updated_at: 2020-09-14T20:36:40Z
author_association: OWNER
body:
  The JSON response will look like this:
  ```json
  {
    "ok": true,
    "message": "A message",
    "redirect": "/blah"
  }
  ```
  `"ok"` will be `true` if everything went right and `false` if there was an error.

  The `"message"` and `"redirect"` will be whatever was configured using the on_success_message - the message shown `on_success_message`, `on_success_redirect`, `on_error_message` and `on_error_redirect` settings, see https://docs.datasette.io/en/stable/sql_queries.html#writable-canned-queries

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/880#issuecomment-692324230
issue_url: https://api.github.com/repos/simonw/datasette/issues/880
node_id: MDEyOklzc3VlQ29tbWVudDY5MjMyNDIzMA==
created_at: 2020-09-14T21:28:15Z
updated_at: 2020-09-14T21:28:21Z
author_association: OWNER
body:
  Documentation here: https://docs.datasette.io/en/latest/sql_queries.html#json-api-for-writable-canned-queries

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-692331349
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY5MjMzMTM0OQ==
created_at: 2020-09-14T21:46:11Z
updated_at: 2020-09-14T21:46:11Z
author_association: OWNER
body:
  Just release Datasette 0.49 - which shipped to PyPI just fine but skipped the Docker step for some reason!

  https://github.com/simonw/datasette/runs/1114585275?check_suite_focus=true

  <img width="929" alt="Release_0_49_·_simonw_datasette_c024952" src="https://user-images.githubusercontent.com/9599/93141431-0571e180-f699-11ea-93c3-acaa68bd1272.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-692331919
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY5MjMzMTkxOQ==
created_at: 2020-09-14T21:47:39Z
updated_at: 2020-09-14T21:47:39Z
author_association: OWNER
body:
  I bet that's because the `github.ref` actually looks like this:

  `${GITHUB_REF#refs/tags/}`

  And the `refs/tags/` part has an `a` in it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-692332430
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY5MjMzMjQzMA==
created_at: 2020-09-14T21:48:59Z
updated_at: 2020-09-14T21:48:59Z
author_association: OWNER
body:
  So now I've released Datasette 0.49 but failed to push a new Docker image. This is bad, and I need to fix it.

  I'd like to push to Docker from GitHub Actions, so I think I'm going to create a one-off workflow task for doing that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-692336564
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY5MjMzNjU2NA==
created_at: 2020-09-14T21:59:40Z
updated_at: 2020-09-14T21:59:40Z
author_association: OWNER
body:
  Using https://github.com/marketplace/actions/debugging-with-tmate to manually submit a new build from within an interactive GitHub Actions session.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-692337397
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY5MjMzNzM5Nw==
created_at: 2020-09-14T22:01:56Z
updated_at: 2020-09-14T22:01:56Z
author_association: OWNER
body:
  I'm going to switch to using this logic to decide if I should ship to Docker: https://github.community/t/release-prerelease-action-triggers/17275/2

      if: "!github.event.release.prerelease"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-692339645
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY5MjMzOTY0NQ==
created_at: 2020-09-14T22:07:58Z
updated_at: 2020-09-14T22:07:58Z
author_association: OWNER
body:
  I shipped the Docker build manually by running the following in a tmate session:

      docker login
      # Typed my username and password interactively
      export REPO=datasetteproject/datasette 
      docker build -f Dockerfile -t $REPO:0.49 . 
      docker tag $REPO:0.49 $REPO:latest
      docker push $REPO 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/940#issuecomment-692340275
issue_url: https://api.github.com/repos/simonw/datasette/issues/940
node_id: MDEyOklzc3VlQ29tbWVudDY5MjM0MDI3NQ==
created_at: 2020-09-14T22:09:35Z
updated_at: 2020-09-14T22:09:35Z
author_association: OWNER
body:
  I'm going to cross my fingers and hope that this works - I don't want to leave this issue open until Datasette 0.50.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/967#issuecomment-692832113
issue_url: https://api.github.com/repos/simonw/datasette/issues/967
node_id: MDEyOklzc3VlQ29tbWVudDY5MjgzMjExMw==
created_at: 2020-09-15T16:34:53Z
updated_at: 2020-09-15T16:37:43Z
author_association: OWNER
body:
  This is so weird. In the test I wrote for this the following passed:

      response = magic_parameters_client.post("/data/runme_post?_json=1", {}, csrftoken_from=True)

  But without the `csrftoken_from=True` parameter it failed with the bindings error:

      response = magic_parameters_client.post("/data/runme_post?_json=1", {})

  Here's the test I wrote:

  ```python
  def test_magic_parameters_json_body(magic_parameters_client):
      magic_parameters_client.ds._metadata["databases"]["data"]["queries"]["runme_post"][
          "sql"
      ] = "insert into logs (line) values (:_header_host)"
      response = magic_parameters_client.post("/data/runme_post?_json=1", {}, csrftoken_from=True)
      assert response.status == 200
      assert response.json["ok"], response.json
      post_actual = magic_parameters_client.get(
          "/data/logs.json?_sort_desc=rowid&_shape=array"
      ).json[0]["line"]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/967#issuecomment-692834064
issue_url: https://api.github.com/repos/simonw/datasette/issues/967
node_id: MDEyOklzc3VlQ29tbWVudDY5MjgzNDA2NA==
created_at: 2020-09-15T16:38:21Z
updated_at: 2020-09-15T16:38:21Z
author_association: OWNER
body:
  So the mystery here is why does omitting `csrftoken_from=True` break the `MagicParameters` mechanism?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/967#issuecomment-692834670
issue_url: https://api.github.com/repos/simonw/datasette/issues/967
node_id: MDEyOklzc3VlQ29tbWVudDY5MjgzNDY3MA==
created_at: 2020-09-15T16:39:29Z
updated_at: 2020-09-15T16:39:29Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L222-L236

  This issue may not be about `_json=1` interacting with magic parameters after all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/967#issuecomment-692835066
issue_url: https://api.github.com/repos/simonw/datasette/issues/967
node_id: MDEyOklzc3VlQ29tbWVudDY5MjgzNTA2Ng==
created_at: 2020-09-15T16:40:12Z
updated_at: 2020-09-15T16:40:12Z
author_association: OWNER
body:
  Is the bug here that magic parameters are incompatible with CSRF-exempt requests (e.g. request with no cookies)?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/967#issuecomment-692927867
issue_url: https://api.github.com/repos/simonw/datasette/issues/967
node_id: MDEyOklzc3VlQ29tbWVudDY5MjkyNzg2Nw==
created_at: 2020-09-15T19:25:23Z
updated_at: 2020-09-15T19:25:23Z
author_association: OWNER
body:
  Hunch: I think the `asgi-csrf` middleware may be consuming the request body and failing to restore it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/967#issuecomment-692937150
issue_url: https://api.github.com/repos/simonw/datasette/issues/967
node_id: MDEyOklzc3VlQ29tbWVudDY5MjkzNzE1MA==
created_at: 2020-09-15T19:42:57Z
updated_at: 2020-09-15T19:42:57Z
author_association: OWNER
body:
  New (failing) test:
  ```python
  @pytest.mark.parametrize("use_csrf", [True, False])
  @pytest.mark.parametrize("return_json", [True, False])
  def test_magic_parameters_csrf_json(magic_parameters_client, use_csrf, return_json):
      magic_parameters_client.ds._metadata["databases"]["data"]["queries"]["runme_post"][
          "sql"
      ] = "insert into logs (line) values (:_header_host)"
      qs = ""
      if return_json:
          qs = "?_json=1"
      response = magic_parameters_client.post(
          "/data/runme_post{}".format(qs),
          {},
          csrftoken_from=use_csrf or None,
          allow_redirects=False,
      )
      if return_json:
          assert response.status == 200
          assert response.json["ok"], response.json
      else:
          assert response.status == 302
          messages = magic_parameters_client.ds.unsign(
              response.cookies["ds_messages"], "messages"
          )
          assert [["Query executed, 1 row affected", 1]] == messages
      post_actual = magic_parameters_client.get(
          "/data/logs.json?_sort_desc=rowid&_shape=array"
      ).json[0]["line"]
      assert post_actual == "localhost"
  ```
  It passes twice, fails twice - failures are for the ones where `use_csrf` is `False`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/967#issuecomment-692938935
issue_url: https://api.github.com/repos/simonw/datasette/issues/967
node_id: MDEyOklzc3VlQ29tbWVudDY5MjkzODkzNQ==
created_at: 2020-09-15T19:44:21Z
updated_at: 2020-09-15T19:44:41Z
author_association: OWNER
body:
  While I'm running the above test, in the rounds that work the `receive()` awaitable returns `{'type': 'http.request', 'body': b'csrftoken=IlpwUGlSMFVVa3Z3ZlVoamQi.uY2U1tF4i0M-5M6x34vnBCmJgr0'}`

  In the rounds that fails it returns `{'type': 'http.request'}`

  So it looks like the `csrftoken_from=True` parameter may be helping just by ensuring the `body` key is present and not missing. I wonder if it would work if a body of `b''` was present there?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/967#issuecomment-692940375
issue_url: https://api.github.com/repos/simonw/datasette/issues/967
node_id: MDEyOklzc3VlQ29tbWVudDY5Mjk0MDM3NQ==
created_at: 2020-09-15T19:47:09Z
updated_at: 2020-09-15T19:47:09Z
author_association: OWNER
body:
  Yes! The tests all pass if I update the test function to do this:
  ```python
      response = magic_parameters_client.post(
          "/data/runme_post{}".format(qs),
          {"ignore_me": "1"},
          csrftoken_from=use_csrf or None,
          allow_redirects=False,
      )
  ```
  So the bug only occurs if the POST body is completely empty.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/967#issuecomment-692945504
issue_url: https://api.github.com/repos/simonw/datasette/issues/967
node_id: MDEyOklzc3VlQ29tbWVudDY5Mjk0NTUwNA==
created_at: 2020-09-15T19:57:10Z
updated_at: 2020-09-15T19:57:10Z
author_association: OWNER
body:
  So the problem actually occurs when the `MagicParameters` class wraps an empty dictionary.

  Relevant code:

  https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L228-L236

  And:

  https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L364-L383

  I'm passing a special magic parameters dictionary for the Python `sqlite3` module to look up parameters in. When that dictionary is `{}` a `__len__` check is performed on that dictionary, the result comes back as 0 and as a result it assumes there are no parameters.

  I tracked down the relevant C code:

  https://github.com/python/cpython/blob/81715808716198471fbca0a3db42ac408468dbc5/Modules/_sqlite/statement.c#L218-L237

  ```c
      Py_BEGIN_ALLOW_THREADS
      num_params_needed = sqlite3_bind_parameter_count(self->st);
      Py_END_ALLOW_THREADS

      if (PyTuple_CheckExact(parameters) || PyList_CheckExact(parameters) || (!PyDict_Check(parameters) && PySequence_Check(parameters))) {
          /* parameters passed as sequence */
          if (PyTuple_CheckExact(parameters)) {
              num_params = PyTuple_GET_SIZE(parameters);
          } else if (PyList_CheckExact(parameters)) {
              num_params = PyList_GET_SIZE(parameters);
          } else {
              num_params = PySequence_Size(parameters);
          }
          if (num_params != num_params_needed) {
              PyErr_Format(pysqlite_ProgrammingError,
                           "Incorrect number of bindings supplied. The current "
                           "statement uses %d, and there are %zd supplied.",
                           num_params_needed, num_params);
              return;
          }
  ```

  It looks to me like this should fail if the number of keys known to be in the dictionary differs from the number of named parameters in the query. But if those numbers fail to match it still works as far as I can tell - it's only dictionary length of 0 that is causing the problems.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/967#issuecomment-692946616
issue_url: https://api.github.com/repos/simonw/datasette/issues/967
node_id: MDEyOklzc3VlQ29tbWVudDY5Mjk0NjYxNg==
created_at: 2020-09-15T19:59:21Z
updated_at: 2020-09-15T19:59:21Z
author_association: OWNER
body:
  I wish I could call https://www.sqlite.org/c3ref/bind_parameter_count.html and https://www.sqlite.org/c3ref/bind_parameter_name.html from Python.

  Might be possible to do that using `ctypes` - see this example code: https://mail.python.org/pipermail//pypy-commit/2013-February/071372.html

  ```python
              param_count = lib.sqlite3_bind_parameter_count(self.statement)
              for idx in range(1, param_count + 1):
                  param_name = lib.sqlite3_bind_parameter_name(self.statement,
                                                               idx)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/967#issuecomment-692951144
issue_url: https://api.github.com/repos/simonw/datasette/issues/967
node_id: MDEyOklzc3VlQ29tbWVudDY5Mjk1MTE0NA==
created_at: 2020-09-15T20:08:12Z
updated_at: 2020-09-15T20:08:12Z
author_association: OWNER
body:
  I think the easiest fix is for me to ensure that calls to `__len__` on the `MagicParameters` class always return at least 1.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/956#issuecomment-692955850
issue_url: https://api.github.com/repos/simonw/datasette/issues/956
node_id: MDEyOklzc3VlQ29tbWVudDY5Mjk1NTg1MA==
created_at: 2020-09-15T20:17:49Z
updated_at: 2020-09-15T20:17:49Z
author_association: OWNER
body:
  I think I've fixed this with recent changes I made as part of #941 - but I won't know until I release the next version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/956#issuecomment-692965022
issue_url: https://api.github.com/repos/simonw/datasette/issues/956
node_id: MDEyOklzc3VlQ29tbWVudDY5Mjk2NTAyMg==
created_at: 2020-09-15T20:36:34Z
updated_at: 2020-09-15T20:36:34Z
author_association: OWNER
body:
  https://hub.docker.com/r/datasetteproject/datasette/tags - 0.49.1 was successfully pushed to Docker Hub by https://github.com/simonw/datasette/runs/1119815175?check_suite_focus=true

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/888#issuecomment-692966625
issue_url: https://api.github.com/repos/simonw/datasette/issues/888
node_id: MDEyOklzc3VlQ29tbWVudDY5Mjk2NjYyNQ==
created_at: 2020-09-15T20:39:49Z
updated_at: 2020-09-15T20:39:49Z
author_association: OWNER
body:
  Thanks, I've fixed that now. It only affected the GitHub release notes - the ones at https://docs.datasette.io/en/stable/changelog.html#v0-45 had the correct links.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/877#issuecomment-692967733
issue_url: https://api.github.com/repos/simonw/datasette/issues/877
node_id: MDEyOklzc3VlQ29tbWVudDY5Mjk2NzczMw==
created_at: 2020-09-15T20:42:04Z
updated_at: 2020-09-15T20:42:04Z
author_association: OWNER
body:
  I'm not going to drop CSRF protection - it's still needed for older browsers - but I have relaxed the circumstances under which it is applied. It only applies to requests that include cookies for example, so API clients that don't send cookies don't need to worry about it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/891#issuecomment-692968792
issue_url: https://api.github.com/repos/simonw/datasette/issues/891
node_id: MDEyOklzc3VlQ29tbWVudDY5Mjk2ODc5Mg==
created_at: 2020-09-15T20:44:15Z
updated_at: 2020-09-15T20:44:15Z
author_association: OWNER
body:
  https://github.com/peter-wangxu/persist-queue/issues/74 warns that this might not work with PyPy.

  I could solve that with:
  ```python
  if hasattr(sqlite3, "enable_callback_tracebacks"):
      sqlite3.enable_callback_tracebacks(True)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/891#issuecomment-692998061
issue_url: https://api.github.com/repos/simonw/datasette/issues/891
node_id: MDEyOklzc3VlQ29tbWVudDY5Mjk5ODA2MQ==
created_at: 2020-09-15T21:49:03Z
updated_at: 2020-09-15T21:49:03Z
author_association: OWNER
body:
  I've been trying to figure out why this is an optional setting that defaults to off.

  I think it's because it writes directly to `stderr`, so the maintainers of `sqlite3` reasonably decided that people should be able to opt in to that rather than having weird stuff show up on `stderr` that they weren't expecting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/891#issuecomment-692999893
issue_url: https://api.github.com/repos/simonw/datasette/issues/891
node_id: MDEyOklzc3VlQ29tbWVudDY5Mjk5OTg5Mw==
created_at: 2020-09-15T21:53:36Z
updated_at: 2020-09-15T21:53:36Z
author_association: OWNER
body:
  Here's the commit (from 15 years ago) where `enable_callback_tracebacks` was first added: https://github.com/ghaering/pysqlite/commit/1e8bd36be93b7d7425910642b72e4152c77b0dfd

  > - Exceptions in callbacks lead to the query being aborted now instead of silently leading to generating values.
  > - Exceptions in callbacks can be echoed to stderr if you call the module level function enable_callback_tracebacks: enable_callback_tracebacks(1).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/891#issuecomment-693000522
issue_url: https://api.github.com/repos/simonw/datasette/issues/891
node_id: MDEyOklzc3VlQ29tbWVudDY5MzAwMDUyMg==
created_at: 2020-09-15T21:55:11Z
updated_at: 2020-09-15T21:55:11Z
author_association: OWNER
body:
  I'm going to turn this on. If people complain about it I can turn it off again (or make it a configuration setting).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/891#issuecomment-693001937
issue_url: https://api.github.com/repos/simonw/datasette/issues/891
node_id: MDEyOklzc3VlQ29tbWVudDY5MzAwMTkzNw==
created_at: 2020-09-15T21:58:56Z
updated_at: 2020-09-15T21:58:56Z
author_association: OWNER
body:
  Here's what that looks like:
  ```
  Traceback (most recent call last):
    File "/Users/simon/Dropbox/Development/datasette/plugins/sql_error.py", line 5, in oh_no_error
      return 100 / 0
  ZeroDivisionError: division by zero
  ERROR: conn=<sqlite3.Connection object at 0x10bce0030>, sql = 'select oh_no_error()', params = {}: user-defined function raised exception
  INFO:     127.0.0.1:54066 - "GET /data?sql=select+oh_no_error%28%29 HTTP/1.1" 400 Bad Request
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-693003652
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5MzAwMzY1Mg==
created_at: 2020-09-15T22:03:08Z
updated_at: 2020-09-15T22:03:08Z
author_association: OWNER
body:
  I'm not going to mess around with formats - you'll get back the exact response that a web client would receive.

  Question: what should the response object look like? e.g. if you do:

      response = await datasette.get("/db/table.json")

  What should `response` be?

  I could reuse the Datasette `Response` class from `datasette.utils.asgi`. This would work well for regular responses which just have a status code, some headers and a response body. It wouldn't be great for streaming responses though such as you get back from `?_stream=1` CSV exports.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-693004296
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5MzAwNDI5Ng==
created_at: 2020-09-15T22:04:54Z
updated_at: 2020-09-15T22:04:54Z
author_association: OWNER
body:
  So what should I do about streaming responses?

  I could deliberately ignore them - through an exception if you attempt to run `await datasette.get(...)` against a streaming URL.

  I could load the entire response into memory and return it as a wrapped object.

  I could support some kind of asynchronous iterator mechanism. This would be pretty elegant if I could decide the right syntax for it - it would allow plugins to take advantage of other internal URLs that return streaming content without needing to load that content entirely into memory in order to process it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-693004572
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5MzAwNDU3Mg==
created_at: 2020-09-15T22:05:39Z
updated_at: 2020-09-15T22:05:39Z
author_association: OWNER
body:
  Maybe these methods become the way most Datasette tests are written, replacing the existing `TestClient` mechanism?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-693004770
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5MzAwNDc3MA==
created_at: 2020-09-15T22:06:13Z
updated_at: 2020-09-15T22:06:13Z
author_association: OWNER
body:
  I'm tempted to create a `await datasette.request()` method which can take any HTTP verb - then have `datasette.get()` and `datasette.post()` as thin wrappers around it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-693005033
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5MzAwNTAzMw==
created_at: 2020-09-15T22:06:58Z
updated_at: 2020-09-15T22:10:58Z
author_association: OWNER
body:
  What if `datasette.get()` was an alias for `httpx.get()`, pre-configured to route to the correct application? And with some sugar that added `http://localhost/` to the beginning of the path if it was missing?

  This would make `httpx` a dependency of core Datasette, which I think is OK.

  It would also solve the return type problem: I would return whatever `httpx` returns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-693007512
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5MzAwNzUxMg==
created_at: 2020-09-15T22:13:30Z
updated_at: 2020-09-15T22:13:30Z
author_association: OWNER
body:
  I could solve streaming using something like this:
  ```python
  async with datasette.stream("GET", "/fixtures/compound_three_primary_keys.csv?_stream=on&_size=max") as response:
      async for chunk in response.aiter_bytes():
          print(chunk)
  ```
  Which would be a wrapper around `AsyncClient.stream(method, url, ...)` from https://www.python-httpx.org/async/#streaming-responses

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-693008540
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5MzAwODU0MA==
created_at: 2020-09-15T22:16:07Z
updated_at: 2020-09-15T22:16:07Z
author_association: OWNER
body:
  I think I can use `async with httpx.AsyncClient(base_url="http://localhost/") as client:` to ensure I don't need to use `http://localhost/` on every call.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-693009048
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5MzAwOTA0OA==
created_at: 2020-09-15T22:17:30Z
updated_at: 2020-09-22T14:37:00Z
author_association: OWNER
body:
  Maybe instead of implementing `datasette.get()` and `datasette.post()` and `datasette.request()` and `datasette.stream()` I could instead have a nested object called `datasette.client` which is a preconfigured `AsyncClient` instance.

  ```python
  response = await datasette.client.get("/")
  ```
  Or perhaps this should be a method in case I ever need to be able to `await` it:
  ```python
  response = await (await datasette.client()).get("/")
  ```
  This is a bit cosmetically ugly though, I'd rather avoid that if possible.

  Maybe I could get this working by returning an object from `.client()` which provides a `await obj.get()` method:
  ```python
  response = await datasette.client().get("/")
  ```
  I don't think there's any benefit to that over `await datasette.client.get()` though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-693010291
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5MzAxMDI5MQ==
created_at: 2020-09-15T22:20:55Z
updated_at: 2020-09-15T22:20:55Z
author_association: OWNER
body:
  Should I instantiate a single `Client` and reuse it for all internal requests, or can I instantiate a new `Client` for each request?

  https://www.python-httpx.org/advanced/#why-use-a-client says that the main benefit of a Client instance is HTTP connection pooling - which isn't an issue for these internal requests since they won't be using the HTTP protocol at all, they'll be calling the ASGI application directly.

  So I'm leaning towards instantiating a fresh client for every internal request. I'll run a microbenchmark to check that this doesn't have any unpleasant performance implications.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693199049
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/159
node_id: MDEyOklzc3VlQ29tbWVudDY5MzE5OTA0OQ==
created_at: 2020-09-16T06:20:26Z
updated_at: 2020-09-16T06:20:26Z
author_association: OWNER
body:
  See #121 - I need to think harder about how this all interacts with transactions.

  You can do this:

  ```python
  with db.conn:
      db["mytable"].delete_where()
  ```
  But that should be documented and maybe rethought.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693486183
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/159
node_id: MDEyOklzc3VlQ29tbWVudDY5MzQ4NjE4Mw==
created_at: 2020-09-16T15:34:13Z
updated_at: 2020-09-16T15:34:13Z
author_association: NONE
body:
  I appreciate the response, it's just unexpected.

  If I insert, it commits, if I update it commits, if I upsert it commits... if I delete.. it doesn't???
  Confused me...

  I did just db commit it... But it's confusing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693694343
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/159
node_id: MDEyOklzc3VlQ29tbWVudDY5MzY5NDM0Mw==
created_at: 2020-09-16T22:15:39Z
updated_at: 2020-09-16T22:15:39Z
author_association: OWNER
body:
  Independent of the transaction changes in #121 I may be able to check `self.conn.in_transaction` to see if a transaction is active and, if one is NOT active, execute the delete inside of one. https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.in_transaction

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693694968
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/159
node_id: MDEyOklzc3VlQ29tbWVudDY5MzY5NDk2OA==
created_at: 2020-09-16T22:17:19Z
updated_at: 2020-09-16T22:17:19Z
author_association: OWNER
body:
  That's strange... this test here doesn't manually commit a transaction and passes:

  https://github.com/simonw/sqlite-utils/blob/7805d53bcf11199bd1f2b07e05ae90151f9d0eb0/tests/test_delete.py#L17-L23

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693773191
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50
node_id: MDEyOklzc3VlQ29tbWVudDY5Mzc3MzE5MQ==
created_at: 2020-09-17T02:39:26Z
updated_at: 2020-09-17T02:39:26Z
author_association: MEMBER
body:
  I'm going to start with `github-to-sqlite get` and `github-to-sqlite post` - I may add `put` and suchlike later on.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/507#issuecomment-693775390
issue_url: https://api.github.com/repos/simonw/datasette/issues/507
node_id: MDEyOklzc3VlQ29tbWVudDY5Mzc3NTM5MA==
created_at: 2020-09-17T02:47:35Z
updated_at: 2020-09-17T02:47:35Z
author_association: OWNER
body:
  I have a pattern for creating screenshots using Puppeteer running in a GitHub Action now, see https://simonwillison.net/2020/Sep/3/weeknotes-airtable-screenshots-dogsheep/#weeknotes-2020-09-03-social-media-cards-tils

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693788032
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50
node_id: MDEyOklzc3VlQ29tbWVudDY5Mzc4ODAzMg==
created_at: 2020-09-17T03:35:22Z
updated_at: 2020-09-17T03:35:22Z
author_association: MEMBER
body:
  Documentation: https://github.com/dogsheep/github-to-sqlite/blob/b02bf135485c0a7a3768868967f45a6b5e515289/README.md#making-authenticated-api-calls

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693788387
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50
node_id: MDEyOklzc3VlQ29tbWVudDY5Mzc4ODM4Nw==
created_at: 2020-09-17T03:36:47Z
updated_at: 2020-09-17T03:36:58Z
author_association: MEMBER
body:
  Fun demo of the `--nl` option:

      github-to-sqlite get /users/simonw/repos --paginate --nl | sqlite-utils insert simonw.db repos - --nl

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693789129
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50
node_id: MDEyOklzc3VlQ29tbWVudDY5Mzc4OTEyOQ==
created_at: 2020-09-17T03:40:01Z
updated_at: 2020-09-17T03:40:01Z
author_association: MEMBER
body:
  Bug with endpoints that return dictionaries rather than arrays:
  ```
  github-to-sqlite get /users/simonw
  [
      "login",
      "id",
      "node_id",
      "avatar_url",
      "gravatar_id",
      "url",
      "html_url",
      "followers_url",
      "following_url",
      "gists_url",
      "starred_url",
      "subscriptions_url",
      "organizations_url",
      "repos_url",
      "events_url",
      "received_events_url",
      "type",
      "site_admin",
      "name",
      "company",
      "blog",
      "location",
      "email",
      "hireable",
      "bio",
      "twitter_username",
      "public_repos",
      "public_gists",
      "followers",
      "following",
      "created_at",
      "updated_at"
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551406
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDY5NDU1MTQwNg==
created_at: 2020-09-17T23:22:07Z
updated_at: 2020-09-17T23:22:07Z
author_association: MEMBER
body:
  Neat, I can debug this with the new `--pdb` option:

      datasette . --get '/-/beta?q=pycon&sort=oldest' --pdb

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551646
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDY5NDU1MTY0Ng==
created_at: 2020-09-17T23:22:48Z
updated_at: 2020-09-17T23:22:48Z
author_association: MEMBER
body:
  Looks like its happening in a Jinja fragment template for one of the results:
  ```
    /Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/__init__.py(169)process_results()
  -> output = compiled.render({**result, **{"json": json}})
    /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/asyncsupport.py(71)render()
  -> return original_render(self, *args, **kwargs)
    /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(1090)render()
  -> self.environment.handle_exception()
    /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(832)handle_exception()
  -> reraise(*rewrite_traceback_stack(source=source))
    /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/_compat.py(28)reraise()
  -> raise value.with_traceback(tb)
    <template>(5)top-level template code()
  > /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads()
  -> raise TypeError(f'the JSON object must be str, bytes or bytearray, '
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552393
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDY5NDU1MjM5Mw==
created_at: 2020-09-17T23:25:01Z
updated_at: 2020-09-17T23:25:17Z
author_association: MEMBER
body:
  Ran `locals()` In the debugger:
  `{'range': <class 'range'>, 'dict': <class 'dict'>, 'lipsum': <function generate_lorem_ipsum at 0x10aeff430>, 'cycler': <class 'jinja2.utils.Cycler'>, 'joiner': <class 'jinja2.utils.Joiner'>, 'namespace': <class 'jinja2.utils.Namespace'>, 'rank': -9.383801886431414, 'rowid': 14297, 'type': 'twitter.db/tweets', 'key': '312658917933076480', 'title': 'Tweet by @chrisstreeter', 'category': 2, 'timestamp': '2013-03-15T20:17:49+00:00', 'search_1': '@simonw are you at pycon? Would love to meet you.', 'display': {'avatar_url': 'https://pbs.twimg.com/profile_images/806275088597204993/38yLHfJi_normal.jpg', 'user_name': 'Chris Streeter', 'screen_name': 'chrisstreeter', 'followers_count': 280, 'tweet_id': 312658917933076480, 'created_at': '2013-03-15T20:17:49+00:00', 'full_text': '@simonw are you at pycon? Would love to meet you.', 'media_urls_2': '[]', 'media_urls': '[]'}, 'json': <module 'json' from '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py'>}`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552681
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDY5NDU1MjY4MQ==
created_at: 2020-09-17T23:25:54Z
updated_at: 2020-09-17T23:25:54Z
author_association: MEMBER
body:
  This is the template fragment it's rendering:
  ```html+jinja
              <div style="overflow: hidden;">
                <p>Tweet by <a href="https://twitter.com/{{ display.screen_name }}">@{{ display.screen_name }}</a> ({{ display.user_name }}, {{ "{:,}".format(display.followers_count or 0) }} followers)
                  on <a href="https://twitter.com/{{ display.screen_name }}/status/{{ display.tweet_id }}">{{ display.created_at }}</a></p>
                </p>
                <blockquote>{{ display.full_text }}</blockquote>
                {% if display.media_urls and json.loads(display.media_urls) %}
                  {% for url in json.loads(display.media_urls) %}
                    <img src="{{ url }}" style="height: 200px;">
                  {% endfor %}
                {% endif %}
              </div>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694553579
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDY5NDU1MzU3OQ==
created_at: 2020-09-17T23:28:37Z
updated_at: 2020-09-17T23:28:37Z
author_association: MEMBER
body:
  More investigation in pdb:
  ```
  (dogsheep-beta) dogsheep-beta % datasette . --get '/-/beta?q=pycon&sort=oldest' --pdb
  > /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads()
  -> raise TypeError(f'the JSON object must be str, bytes or bytearray, '
  (Pdb) list
  336  	        if s.startswith('\ufeff'):
  337  	            raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
  338  	                                  s, 0)
  339  	    else:
  340  	        if not isinstance(s, (bytes, bytearray)):
  341  ->	            raise TypeError(f'the JSON object must be str, bytes or bytearray, '
  342  	                            f'not {s.__class__.__name__}')
  343  	        s = s.decode(detect_encoding(s), 'surrogatepass')
  344  	
  345  	    if "encoding" in kw:
  346  	        import warnings
  (Pdb) bytes
  <class 'bytes'>
  (Pdb) locals()['s']
  Undefined
  (Pdb) type(locals()['s'])
  <class 'jinja2.runtime.Undefined'>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDY5NDU1NDU4NA==
created_at: 2020-09-17T23:31:25Z
updated_at: 2020-09-17T23:31:25Z
author_association: MEMBER
body:
  I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695109140
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25
node_id: MDEyOklzc3VlQ29tbWVudDY5NTEwOTE0MA==
created_at: 2020-09-18T22:12:20Z
updated_at: 2020-09-18T22:12:20Z
author_association: MEMBER
body:
  Documented here: https://github.com/dogsheep/dogsheep-beta/blob/534fc9689227eba70e69a45da0cee5820bbda9e1/README.md#datasette-plugin

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-695113871
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24
node_id: MDEyOklzc3VlQ29tbWVudDY5NTExMzg3MQ==
created_at: 2020-09-18T22:30:17Z
updated_at: 2020-09-18T22:30:17Z
author_association: MEMBER
body:
  I think I know what's going on here:

  https://github.com/dogsheep/dogsheep-beta/blob/0f1b951c5131d16f3c8559a8e4d79ed5c559e3cb/dogsheep_beta/__init__.py#L166-L171

  This is a logic bug - the `compiled` variable could be the template from the previous loop!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-695133768
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5NTEzMzc2OA==
created_at: 2020-09-19T00:06:56Z
updated_at: 2020-09-19T00:07:35Z
author_association: OWNER
body:
  [dogsheep-beta](https://github.com/dogsheep/dogsheep-beta) could do with this too. It currently [makes a call](https://github.com/dogsheep/dogsheep-beta/blob/ab36101bdae69b11af7c6bd7edee838d052e6ecf/dogsheep_beta/__init__.py#L216-L225) to `TableView` in a similar way to `datasette-graphql` in order to calculate facets.

  `dogsheep-beta` would benefit with a mechanism for changing the facet timeout setting during that call (as would `datasette-graphql`, see the [DatasetteSpecialConfig mechanism](https://github.com/simonw/datasette-graphql/blob/f9dc5c518b7cdc94b93873ef20069a7ea2882a95/datasette_graphql/utils.py#L516-L519) it uses).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-695698227
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5NTY5ODIyNw==
created_at: 2020-09-20T04:27:26Z
updated_at: 2020-09-20T04:28:26Z
author_association: OWNER
body:
  This is going to need #114 (the `transform_table()` method) in order to convert string columns into integer foreign key columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/160#issuecomment-695839557
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/160
node_id: MDEyOklzc3VlQ29tbWVudDY5NTgzOTU1Nw==
created_at: 2020-09-20T21:37:03Z
updated_at: 2020-09-20T21:37:03Z
author_association: OWNER
body:
  Should this support `ignore=True` as well? I'm tempted to skip that - I think `replace=True` is more useful because it implies "ignore if the options are already the same, but replace if they are different".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855646
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDY5NTg1NTY0Ng==
created_at: 2020-09-21T00:16:11Z
updated_at: 2020-09-21T00:16:11Z
author_association: MEMBER
body:
  Should I do this with offset/limit or should I do proper keyset pagination?

  I think keyset because then it will work well for the full search interface with no filters or search string.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855723
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDY5NTg1NTcyMw==
created_at: 2020-09-21T00:16:52Z
updated_at: 2020-09-21T00:17:53Z
author_association: MEMBER
body:
  It feels a bit weird to implement keyset pagination against results sorted by `rank` because the ranks could change substantially if the search index gets updated while the user is paginating.

  I may just ignore that though. If you want reliable pagination you can get it by sorting by date. Maybe it doesn't even make sense to offer pagination if you sort by relevance?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856967
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDY5NTg1Njk2Nw==
created_at: 2020-09-21T00:26:59Z
updated_at: 2020-09-21T00:26:59Z
author_association: MEMBER
body:
  It's a shame Datasette doesn't currently have an easy way to implement sorted-by-rank keyset-paginated using a TableView or QueryView. I'll have to do this using the custom SQL query constructed in the plugin: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L8-L43

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695875274
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDY5NTg3NTI3NA==
created_at: 2020-09-21T02:28:58Z
updated_at: 2020-09-21T02:28:58Z
author_association: MEMBER
body:
  Datasette's implementation is complex because it has to support compound primary keys: https://github.com/simonw/datasette/blob/a258339a935d8d29a95940ef1db01e98bb85ae63/datasette/utils/__init__.py#L88-L114 - but that's not something that's needed for dogsheep-beta.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695877627
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDY5NTg3NzYyNw==
created_at: 2020-09-21T02:42:29Z
updated_at: 2020-09-21T02:42:29Z
author_association: MEMBER
body:
  Fun twist: assuming `timestamp` is always stored as UTC, I need the interface to be timezone aware so I can see e.g. everything from 4th July 2020 in the San Francisco timezone definition of 4th July 2020.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879237
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDY5NTg3OTIzNw==
created_at: 2020-09-21T02:53:29Z
updated_at: 2020-09-21T02:53:29Z
author_association: MEMBER
body:
  If previous page ended at `2018-02-11T16:32:53+00:00`:
  ```sql
  select
    search_index.rowid,
    search_index.type,
    search_index.key,
    search_index.title,
    search_index.category,
    search_index.timestamp,
    search_index.search_1
  from
    search_index
   where 
    date("timestamp") = '2018-02-11'
   and timestamp < '2018-02-11T16:32:53+00:00'
  order by
    search_index.timestamp desc, rowid
  limit 41
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879531
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDY5NTg3OTUzMQ==
created_at: 2020-09-21T02:55:28Z
updated_at: 2020-09-21T02:55:54Z
author_association: MEMBER
body:
  Actually for the tie-breaker it should be something like https://latest.datasette.io/fixtures?sql=select+pk%2C+created%2C+planet_int%2C+on_earth%2C+state%2C+city_id%2C+neighborhood%2C+tags%2C+complex_array%2C+distinct_some_null+from+facetable+where+%28created+%3E+%3Ap1+or+%28created+%3D+%3Ap1+and+%28%28pk+%3E+%3Ap0%29%29%29%29+order+by+created%2C+pk+limit+11&p0=10&p1=2019-01-16+08%3A00%3A00
  ```sql
  where
    (
      created > :p1
      or (
        created = :p1
        and ((pk > :p0))
      )
    )
  ```
  But with `rowid` and `timestamp` in place of `pk` and `created`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/970#issuecomment-695895960
issue_url: https://api.github.com/repos/simonw/datasette/issues/970
node_id: MDEyOklzc3VlQ29tbWVudDY5NTg5NTk2MA==
created_at: 2020-09-21T04:36:45Z
updated_at: 2020-09-21T04:36:45Z
author_association: OWNER
body:
  I like this. It could work with the `--root` option too and automatically sign you in as the root user.

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 1, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/970#issuecomment-695896557
issue_url: https://api.github.com/repos/simonw/datasette/issues/970
node_id: MDEyOklzc3VlQ29tbWVudDY5NTg5NjU1Nw==
created_at: 2020-09-21T04:40:12Z
updated_at: 2020-09-21T04:40:12Z
author_association: OWNER
body:
  The Python standard library has a module for this: https://docs.python.org/3/library/webbrowser.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-696163452
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDY5NjE2MzQ1Mg==
created_at: 2020-09-21T14:46:10Z
updated_at: 2020-09-21T14:46:10Z
author_association: NONE
body:
  I'm currently using PostgREST to serve OpenAPI APIs off Postgresql databases. I would like to try out datasette once this becomes available on Postgres.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/971#issuecomment-696297601
issue_url: https://api.github.com/repos/simonw/datasette/issues/971
node_id: MDEyOklzc3VlQ29tbWVudDY5NjI5NzYwMQ==
created_at: 2020-09-21T18:41:07Z
updated_at: 2020-09-21T18:41:07Z
author_association: OWNER
body:
  How to detect it? Looks like it's visible in SQLite compile time options: https://latest.datasette.io/-/versions

  ```
          "compile_options": [
              "COMPILER=gcc-8.3.0",
              "ENABLE_COLUMN_METADATA",
              "ENABLE_DBSTAT_VTAB",
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/971#issuecomment-696297930
issue_url: https://api.github.com/repos/simonw/datasette/issues/971
node_id: MDEyOklzc3VlQ29tbWVudDY5NjI5NzkzMA==
created_at: 2020-09-21T18:41:47Z
updated_at: 2020-09-21T18:41:47Z
author_association: OWNER
body:
  https://www.sqlite.org/dbstat.html
  > The DBSTAT virtual table is an eponymous virtual table, meaning that is not necessary to run CREATE VIRTUAL TABLE to create an instance of the dbstat virtual table before using it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


