# issue_comments

html_url: https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623865250
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDYyMzg2NTI1MA==
created_at: 2020-05-05T05:38:16Z
updated_at: 2020-05-05T05:38:16Z
author_association: MEMBER
body:
  It looks like `groups.content_string` often has a null byte in it. I should clean this up as part of the import.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624278090
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17
node_id: MDEyOklzc3VlQ29tbWVudDYyNDI3ODA5MA==
created_at: 2020-05-05T20:06:01Z
updated_at: 2020-05-05T20:06:01Z
author_association: MEMBER
body:
  https://www.python.org/dev/peps/pep-0508/#environment-markers I think I want `sys_platform` of `darwin`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624278714
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17
node_id: MDEyOklzc3VlQ29tbWVudDYyNDI3ODcxNA==
created_at: 2020-05-05T20:07:19Z
updated_at: 2020-05-05T20:07:19Z
author_association: MEMBER
body:
  From https://hynek.me/articles/conditional-python-dependencies/ I think this will look like:
  ```python
  setup(
      # ...
      install_requires=[
          # ...
          "osxphotos>=0.28.13 ; sys_platform=='darwin'",
      ]
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624284539
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17
node_id: MDEyOklzc3VlQ29tbWVudDYyNDI4NDUzOQ==
created_at: 2020-05-05T20:20:05Z
updated_at: 2020-05-05T20:20:05Z
author_association: CONTRIBUTOR
body:
  FYI, I've got an [issue](https://github.com/RhetTbull/osxphotos/issues/25) to make osxphotos cross-platform but it's low on my priority list.  About 90% of the functionality could be done cross-platform but right now the MacOS specific stuff is embedded throughout and would take some work.  Though I try to minimize it, there's sprinklings of ObjC & Applescript throughout osxphotos.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/19#issuecomment-624406285
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/19
node_id: MDEyOklzc3VlQ29tbWVudDYyNDQwNjI4NQ==
created_at: 2020-05-06T02:10:03Z
updated_at: 2020-05-06T02:10:03Z
author_association: MEMBER
body:
  Most annoying part of this is the difficulty of actually showing a photo.

  Maybe I need to run a local proxy that I can link to? A custom Datasette plugin perhaps?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408220
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDYyNDQwODIyMA==
created_at: 2020-05-06T02:18:47Z
updated_at: 2020-05-06T02:18:47Z
author_association: MEMBER
body:
  The `apple_photos` table has an indexed `uuid` column and a `path` column which stores the full path to that photo file on disk.

  I can write a custom Datasette plugin which takes the `uuid` from the URL, looks up the path, then serves up a thumbnail of the jpeg or heic image file.

  I'll prototype this is a one-off plugin first, then package it on PyPI for other people to install.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408370
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDYyNDQwODM3MA==
created_at: 2020-05-06T02:19:27Z
updated_at: 2020-05-06T02:19:27Z
author_association: MEMBER
body:
  The plugin can be generalized: it can be configured to know how to take the URL path, look it up in ANY table (via a custom SQL query) to get a path on disk and then serve that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408738
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDYyNDQwODczOA==
created_at: 2020-05-06T02:21:05Z
updated_at: 2020-05-06T02:21:32Z
author_association: MEMBER
body:
  Here's rendering code from my hacked-together not-yet-released S3 image proxy:
  ```python
  from starlette.responses import Response
  from PIL import Image, ExifTags
  import pyheif

  for ORIENTATION_TAG in ExifTags.TAGS.keys():
      if ExifTags.TAGS[ORIENTATION_TAG] == "Orientation":
          break
      ...
      # Load it into Pillow
      if ext == "heic":
          heic = pyheif.read_heif(image_response.content)
          image = Image.frombytes(mode=heic.mode, size=heic.size, data=heic.data)
      else:
          image = Image.open(io.BytesIO(image_response.content))

      # Does EXIF tell us to rotate it?
      try:
          exif = dict(image._getexif().items())
          if exif[ORIENTATION_TAG] == 3:
              image = image.rotate(180, expand=True)
          elif exif[ORIENTATION_TAG] == 6:
              image = image.rotate(270, expand=True)
          elif exif[ORIENTATION_TAG] == 8:
              image = image.rotate(90, expand=True)
      except (AttributeError, KeyError, IndexError):
          pass

      # Resize based on ?w= and ?h=, if set
      width, height = image.size
      w = request.query_params.get("w")
      h = request.query_params.get("h")
      if w is not None or h is not None:
          if h is None:
              # Set h based on w
              w = int(w)
              h = int((float(height) / width) * w)
          elif w is None:
              h = int(h)
              # Set w based on h
              w = int((float(width) / height) * h)
          w = int(w)
          h = int(h)
          image.thumbnail((w, h))

      # ?bw= converts to black and white
      if request.query_params.get("bw"):
          image = image.convert("L")

      # ?q= sets the quality - defaults to 75
      quality = 75
      q = request.query_params.get("q")
      if q and q.isdigit() and 1 <= int(q) <= 100:
          quality = int(q)

      # Output as JPEG or PNG
      output_image = io.BytesIO()
      image_type = "JPEG"
      kwargs = {"quality": quality}
      if image.format == "PNG":
          image_type = "PNG"
          kwargs = {}

      image.save(output_image, image_type, **kwargs)
      return Response(
          output_image.getvalue(),
          media_type="image/jpeg",
          headers={"cache-control": "s-maxage={}, public".format(365 * 24 * 60 * 60)},
      )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/760#issuecomment-624729459
issue_url: https://api.github.com/repos/simonw/datasette/issues/760
node_id: MDEyOklzc3VlQ29tbWVudDYyNDcyOTQ1OQ==
created_at: 2020-05-06T15:47:44Z
updated_at: 2020-05-06T15:47:44Z
author_association: OWNER
body:
  `select * from pragma_table_info(tablename);` is currently disallowed for user-provided queries via a regex restriction - but could help here too.

  https://github.com/simonw/datasette/blob/d349d57cdf3d577afb62bdf784af342a4d5be660/datasette/utils/__init__.py#L174

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/761#issuecomment-624766424
issue_url: https://api.github.com/repos/simonw/datasette/issues/761
node_id: MDEyOklzc3VlQ29tbWVudDYyNDc2NjQyNA==
created_at: 2020-05-06T16:54:38Z
updated_at: 2020-05-06T17:01:02Z
author_association: OWNER
body:
  I could allow-list some other useful `pragma_x` tables too.

  SQLite calls these "pragma functions" - documented here: https://www.sqlite.org/pragma.html#pragfunc

  They sound safe:

  >  Table-valued functions exist only for PRAGMAs that return results and that have no side-effects. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/761#issuecomment-624767466
issue_url: https://api.github.com/repos/simonw/datasette/issues/761
node_id: MDEyOklzc3VlQ29tbWVudDYyNDc2NzQ2Ng==
created_at: 2020-05-06T16:56:40Z
updated_at: 2020-05-06T16:57:03Z
author_association: OWNER
body:
  The rationale for blocking `pragma` entirely from statements is that it can be used to change the state of the SQLite database, e.g. from https://www.sqlite.org/pragma.html :

  ```
  PRAGMA schema.application_id;
  PRAGMA schema.application_id = integer ;
  ```
  That second line is unsafe. I don't think it's possible to use the `pragma_table_x` variants to make writes in this way.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/761#issuecomment-624768744
issue_url: https://api.github.com/repos/simonw/datasette/issues/761
node_id: MDEyOklzc3VlQ29tbWVudDYyNDc2ODc0NA==
created_at: 2020-05-06T16:59:01Z
updated_at: 2020-05-06T16:59:01Z
author_association: OWNER
body:
  Maybe use a negative lookahead assertion? https://docs.python.org/3/library/re.html#index-20

  > `(?!...)`
  > 
  > Matches if `...` doesn’t match next. This is a negative lookahead assertion. For example, `Isaac (?!Asimov)` will match 'Isaac ' only if it’s not followed by 'Asimov'.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/761#issuecomment-624774928
issue_url: https://api.github.com/repos/simonw/datasette/issues/761
node_id: MDEyOklzc3VlQ29tbWVudDYyNDc3NDkyOA==
created_at: 2020-05-06T17:11:15Z
updated_at: 2020-05-06T17:11:15Z
author_association: OWNER
body:
  For the moment I'll allow-list the following:
  * `pragma_database_list`
  * `pragma_foreign_key_list`
  * `pragma_function_list`
  * `pragma_index_info`
  * `pragma_index_list`
  * `pragma_index_xinfo`
  * `pragma_page_count`
  * `pragma_max_page_count`
  * `pragma_page_size`
  * `pragma_schema_version`
  * `pragma_table_info`
  * `pragma_table_xinfo`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/761#issuecomment-624782775
issue_url: https://api.github.com/repos/simonw/datasette/issues/761
node_id: MDEyOklzc3VlQ29tbWVudDYyNDc4Mjc3NQ==
created_at: 2020-05-06T17:26:05Z
updated_at: 2020-05-06T17:26:05Z
author_association: OWNER
body:
  Some demos:
  * https://latest.datasette.io/fixtures?sql=select+*+from+pragma_database_list%28%29
  * https://latest.datasette.io/fixtures?sql=select+*+from+pragma_foreign_key_list%28%27complex_foreign_keys%27%29
  * https://latest.datasette.io/fixtures?sql=select+*+from+pragma_function_list%28%29
  * https://latest.datasette.io/fixtures?sql=select+*+from+pragma_index_info%28%27idx_compound_three_primary_keys_content%27%29
  * https://latest.datasette.io/fixtures?sql=select+*+from+pragma_index_list%28%27compound_three_primary_keys%27%29
  * https://latest.datasette.io/fixtures?sql=select+*+from+pragma_index_xinfo%28%27idx_compound_three_primary_keys_content%27%29
  * https://latest.datasette.io/fixtures?sql=select+*+from+pragma_page_count%28%29
  * https://latest.datasette.io/fixtures?sql=select+*+from+pragma_max_page_count%28%29
  * https://latest.datasette.io/fixtures?sql=select+*+from+pragma_page_size%28%29
  * https://latest.datasette.io/fixtures?sql=select+*+from+pragma_schema_version%28%29
  * https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_info%28%27complex_foreign_keys%27%29
  * https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_xinfo%28%27complex_foreign_keys%27%29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/761#issuecomment-624783996
issue_url: https://api.github.com/repos/simonw/datasette/issues/761
node_id: MDEyOklzc3VlQ29tbWVudDYyNDc4Mzk5Ng==
created_at: 2020-05-06T17:28:20Z
updated_at: 2020-05-06T17:28:20Z
author_association: OWNER
body:
  Interestingly https://latest.datasette.io/fixtures?sql=select+*+from+pragma_function_list() doesn't work, when it DOES work on my laptop. 

  `latest.datasette.io` currently runs SQLite `3.27.2` while my laptop runs `3.31.1`

  https://www.sqlite.org/changes.html#version_3_30_0 says that as-of 3.30.0:
  > The PRAGMA function_list, PRAGMA module_list, and PRAGMA pragma_list commands are now enabled in all builds by default. Disable them using -DSQLITE_OMIT_INTROSPECTION_PRAGMAS. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/760#issuecomment-624787678
issue_url: https://api.github.com/repos/simonw/datasette/issues/760
node_id: MDEyOklzc3VlQ29tbWVudDYyNDc4NzY3OA==
created_at: 2020-05-06T17:35:05Z
updated_at: 2020-05-06T17:35:05Z
author_association: OWNER
body:
  Potential recipe in https://github.com/simonw/til/blob/master/sqlite/list-all-columns-in-a-database.md

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/759#issuecomment-624795695
issue_url: https://api.github.com/repos/simonw/datasette/issues/759
node_id: MDEyOklzc3VlQ29tbWVudDYyNDc5NTY5NQ==
created_at: 2020-05-06T17:50:57Z
updated_at: 2020-05-06T17:52:07Z
author_association: OWNER
body:
  This was a deliberate change from #651.

  The `_search_colname=` alternative argument for doing this still works - compare these two:

  * https://latest.datasette.io/fixtures/searchable?_search=dog
  * https://latest.datasette.io/fixtures/searchable?_search_tex1=dog

  If you want to use advanced search syntax on those pages you can do so using the `&_searchmode=raw` option - I added better documentation for this the other day: https://datasette.readthedocs.io/en/latest/full_text_search.html#advanced-sqlite-search-queries

  Example: https://latest.datasette.io/fixtures/searchable?_search=text1:dog&_searchmode=raw

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/758#issuecomment-624797119
issue_url: https://api.github.com/repos/simonw/datasette/issues/758
node_id: MDEyOklzc3VlQ29tbWVudDYyNDc5NzExOQ==
created_at: 2020-05-06T17:53:46Z
updated_at: 2020-05-06T17:53:46Z
author_association: OWNER
body:
  It's interesting to hear from someone who's using this feature - I'm considering moving it out into a plugin #647.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/757#issuecomment-624798182
issue_url: https://api.github.com/repos/simonw/datasette/issues/757
node_id: MDEyOklzc3VlQ29tbWVudDYyNDc5ODE4Mg==
created_at: 2020-05-06T17:55:50Z
updated_at: 2020-05-06T17:55:50Z
author_association: OWNER
body:
  I'll definitely get that out this week!

  For the moment a trick I often use is to put a URL to the most recent commit in my `requirements.txt` - e.g. https://github.com/simonw/datasette/archive/0784f2ef9d3ff6dd9df05f54cb51de29a6d11764.zip

  This should be safe because nothing lands on Datasette master without the full unit test suite passing. But you're right, there's a bunch of stuff now that needs to go out in a release: https://github.com/simonw/datasette/compare/0.40...0784f2ef9d3ff6dd9df05f54cb51de29a6d11764

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/757#issuecomment-624798540
issue_url: https://api.github.com/repos/simonw/datasette/issues/757
node_id: MDEyOklzc3VlQ29tbWVudDYyNDc5ODU0MA==
created_at: 2020-05-06T17:56:34Z
updated_at: 2020-05-06T17:56:34Z
author_association: OWNER
body:
  Actually I'm going to put that release out today. I was hoping to finish #698 first but that shouldn't delay those other features any longer.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/757#issuecomment-624821090
issue_url: https://api.github.com/repos/simonw/datasette/issues/757
node_id: MDEyOklzc3VlQ29tbWVudDYyNDgyMTA5MA==
created_at: 2020-05-06T18:41:29Z
updated_at: 2020-05-06T18:41:29Z
author_association: OWNER
body:
  OK, I just released 0.41 with that and a bunch of other stuff: https://datasette.readthedocs.io/en/latest/changelog.html#v0-41

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/759#issuecomment-624860451
issue_url: https://api.github.com/repos/simonw/datasette/issues/759
node_id: MDEyOklzc3VlQ29tbWVudDYyNDg2MDQ1MQ==
created_at: 2020-05-06T20:03:01Z
updated_at: 2020-05-06T20:04:42Z
author_association: NONE
body:
  Thank you. Now it's ok with the url

  http://localhost:8001/index/summary?_search=language%3Aeng&_sort=title&_searchmode=raw

  But I'm not able to manage it in the metadata file. Here is mine (note that the sort column is taken into account)
  Here it is:

  ```
  {
      "databases": {
        "index": {
          "tables": {
              "summary": {
                  "sort": "title",
                  "searchmode": "raw"
              }
          }
        }
      }
    }

  ```
  Any idea ?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/760#issuecomment-624949809
issue_url: https://api.github.com/repos/simonw/datasette/issues/760
node_id: MDEyOklzc3VlQ29tbWVudDYyNDk0OTgwOQ==
created_at: 2020-05-06T23:49:06Z
updated_at: 2020-05-06T23:49:06Z
author_association: OWNER
body:
  ```sql
  select
    sqlite_master.name as table_name,
    table_info.*
  from
    sqlite_master
    join pragma_table_info(sqlite_master.name) as table_info
  order by
    sqlite_master.name,
    table_info.cid
  ```
  https://latest.datasette.io/fixtures?sql=select%0D%0A++sqlite_master.name+as+table_name%2C%0D%0A++table_info.*%0D%0Afrom%0D%0A++sqlite_master%0D%0A++join+pragma_table_info%28sqlite_master.name%29+as+table_info%0D%0Aorder+by%0D%0A++sqlite_master.name%2C%0D%0A++table_info.cid

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/761#issuecomment-625003013
issue_url: https://api.github.com/repos/simonw/datasette/issues/761
node_id: MDEyOklzc3VlQ29tbWVudDYyNTAwMzAxMw==
created_at: 2020-05-07T03:07:55Z
updated_at: 2020-05-07T03:09:05Z
author_association: OWNER
body:
  Here's another demo I built using this feature: https://observablehq.com/@simonw/datasette-table-diagram

  <img width="995" alt="Datasette_foreign_key_diagram___Simon_Willison___Observable" src="https://user-images.githubusercontent.com/9599/81250291-6c89d600-8fd5-11ea-859c-d99a0de0531a.png">

  Running on this query:

  ```sql
  select
    sqlite_master.name as table_from,
    fk_info.[from] as column_from,
    fk_info.[table] as table_to,
    fk_info.[to] as column_to
  from
    sqlite_master
    join pragma_foreign_key_list(sqlite_master.name) as fk_info
  order by
    sqlite_master.name
  ```
  See https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++sqlite_master.name+as+table_from%2C%0D%0A++fk_info.%5Bfrom%5D+as+column_from%2C%0D%0A++fk_info.%5Btable%5D+as+table_to%2C%0D%0A++fk_info.%5Bto%5D+as+column_to%0D%0Afrom%0D%0A++sqlite_master%0D%0A++join+pragma_foreign_key_list%28sqlite_master.name%29+as+fk_info%0D%0Aorder+by%0D%0A++sqlite_master.name

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-625060561
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYyNTA2MDU2MQ==
created_at: 2020-05-07T06:38:24Z
updated_at: 2020-05-07T06:38:24Z
author_association: NONE
body:
  Hi @simonw probably I could try to do it in Python for windows. I do not like to do these things in win enviroment.

  Because probably WSL Linux env (in which I do a lot of great things) is not an environment that will be tested for datasette.

  In win I shouldn't have any problems. Am I right?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-625061250
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYyNTA2MTI1MA==
created_at: 2020-05-07T06:40:09Z
updated_at: 2020-05-07T06:40:09Z
author_association: OWNER
body:
  Have a try and let me know what happens! I'd like this stuff to just work on Windows but I need to figure out how to get an environment working.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-625066073
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYyNTA2NjA3Mw==
created_at: 2020-05-07T06:53:09Z
updated_at: 2020-05-07T06:53:09Z
author_association: NONE
body:
  @simonw another error starting from Windows.

  I run

  ```
  datasette  publish heroku -n comunepa --template-dir template commissioniComunePalermo.db
  ```

  And I have

  ```
  Traceback (most recent call last):
    File "c:\python37\lib\runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "c:\python37\lib\runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\Scripts\datasette.exe\__main__.py", line 9, in <module>
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 782, in main
      rv = self.invoke(ctx)
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 53, in heroku
      line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines()
    File "c:\python37\lib\subprocess.py", line 395, in check_output
      **kwargs).stdout
    File "c:\python37\lib\subprocess.py", line 472, in run
      with Popen(*popenargs, **kwargs) as process:
    File "c:\python37\lib\subprocess.py", line 775, in __init__
      restore_signals, start_new_session)
    File "c:\python37\lib\subprocess.py", line 1178, in _execute_child
      startupinfo)
  FileNotFoundError: [WinError 2] The specified file could not be found
  ```


  [files.zip](https://github.com/simonw/datasette/files/4591173/files.zip)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-625066876
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYyNTA2Njg3Ng==
created_at: 2020-05-07T06:55:04Z
updated_at: 2020-05-07T06:55:04Z
author_association: OWNER
body:
  I have a Windows 10 gaming PC in the house: what steps can I take to get an environment in there that matches yours? I've not installed Python on Windows before.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-625083715
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYyNTA4MzcxNQ==
created_at: 2020-05-07T07:34:18Z
updated_at: 2020-05-07T07:34:18Z
author_association: NONE
body:
  In Windows I'm not very strong. I use debian (inside WSL).

  However these are the possible steps:

  - I have installed Python 3 for win (I have 3.7.3);
  - I have installed heroku cli for win64 and logged in;
  - I have installed datasette running `python -m pip install --upgrade --user datasette`.

  It's a very basic Python env that I do not use. This time only to reach my goal: try to publish using custom template

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-625091976
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYyNTA5MTk3Ng==
created_at: 2020-05-07T07:51:25Z
updated_at: 2020-05-07T07:51:25Z
author_association: NONE
body:
  I have installed `heroku plugins:install heroku-builds`, but I have the same error.

  Then I have removed from `datasette\publish\heroku.py`

  ```python
          # Check for heroku-builds plugin
          plugins = [
              line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines()
          ]
          if b"heroku-builds" not in plugins:
              click.echo(
                  "Publishing to Heroku requires the heroku-builds plugin to be installed."
              )
              click.confirm(
                  "Install it? (this will run `heroku plugins:install heroku-builds`)",
                  abort=True,
              )
              call(["heroku", "plugins:install", "heroku-builds"])
  ```

  And now I have

  ```
  Traceback (most recent call last):
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 210, in temporary_heroku_directory
      yield
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 96, in heroku
      list_output = check_output(["heroku", "apps:list", "--json"]).decode(
    File "c:\python37\lib\subprocess.py", line 395, in check_output
      **kwargs).stdout
    File "c:\python37\lib\subprocess.py", line 472, in run
      with Popen(*popenargs, **kwargs) as process:
    File "c:\python37\lib\subprocess.py", line 775, in __init__
      restore_signals, start_new_session)
    File "c:\python37\lib\subprocess.py", line 1178, in _execute_child
      startupinfo)
  FileNotFoundError: [WinError 2] The specified file could not be found

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "c:\python37\lib\runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "c:\python37\lib\runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\Scripts\datasette.exe\__main__.py", line 9, in <module>
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 782, in main
      rv = self.invoke(ctx)
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 120, in heroku
      call(["heroku", "builds:create", "-a", app_name, "--include-vcs-ignore"])
    File "c:\python37\lib\contextlib.py", line 130, in __exit__
      self.gen.throw(type, value, traceback)
    File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 213, in temporary_heroku_directory
      tmp.cleanup()
    File "c:\python37\lib\tempfile.py", line 809, in cleanup
      _shutil.rmtree(self.name)
    File "c:\python37\lib\shutil.py", line 513, in rmtree
      return _rmtree_unsafe(path, onerror)
    File "c:\python37\lib\shutil.py", line 401, in _rmtree_unsafe
      onerror(os.rmdir, path, sys.exc_info())
    File "c:\python37\lib\shutil.py", line 399, in _rmtree_unsafe
      os.rmdir(path)
  PermissionError: [WinError 32] Unable to access file. The file is being used by another process: 'C:\\Users\\aborr\\AppData\\Local\\Temp\\tmpkcxy8i_q'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/648#issuecomment-625286519
issue_url: https://api.github.com/repos/simonw/datasette/issues/648
node_id: MDEyOklzc3VlQ29tbWVudDYyNTI4NjUxOQ==
created_at: 2020-05-07T14:23:22Z
updated_at: 2020-05-07T14:28:33Z
author_association: NONE
body:
  Hi! I'm using datasette on this repository: https://github.com/chekos/RIPA-2018-datasette

  and on my local machine i can see an /about page i created but when i deploy to heroku i get a 404  (http://ripa-2018-db.herokuapp.com)

  <img width="629" alt="Screen Shot 2020-05-07 at 7 04 43 AM" src="https://user-images.githubusercontent.com/28694175/81303797-0af75500-9031-11ea-8a02-adee7a2374ea.png">


  I bumped datasette in my requirements file to 0.41 so I'm 100% what the issue is 🤔 

  Do you have any idea what could be the problem? 👀 

  EDIT: for context, I have a templates directory with a pages/about.html file in https://github.com/chekos/RIPA-2018-datasette/tree/master/datasette/templates

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/648#issuecomment-625321121
issue_url: https://api.github.com/repos/simonw/datasette/issues/648
node_id: MDEyOklzc3VlQ29tbWVudDYyNTMyMTEyMQ==
created_at: 2020-05-07T15:21:19Z
updated_at: 2020-05-07T15:21:19Z
author_association: NONE
body:
  It seems that heroku wasn't updating to 0.41 on deployment. 

  Had to add `--branch 0.41` and that solved it! Heroku caches dependencies 
  <img width="1331" alt="Screen Shot 2020-05-07 at 8 19 08 AM" src="https://user-images.githubusercontent.com/28694175/81312379-6f1f1680-903b-11ea-847c-4149268504a6.png">

  and (i think) because the `requirements.txt` doesn't specify the datasette version, it didn't update from 0.40 to 0.41 on heroku even though it was specified on my local requirements file as `datasette >= 0.41`

  These are the lines that gave me an idea on how to solve it:
  https://github.com/simonw/datasette/blob/182e5c8745c94576718315f7596ccc81e5e2417b/datasette/publish/heroku.py#L164-L186

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/685#issuecomment-625603221
issue_url: https://api.github.com/repos/simonw/datasette/issues/685
node_id: MDEyOklzc3VlQ29tbWVudDYyNTYwMzIyMQ==
created_at: 2020-05-08T03:00:23Z
updated_at: 2020-05-08T03:00:23Z
author_association: OWNER
body:
  Working with this in https://github.com/simonw/datasette-media/issues/2 made me really want to redesign this API: https://github.com/simonw/datasette-media/commit/be23ec3d05900b69a3d47bc1e0dd056c333125f4

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/685#issuecomment-625839046
issue_url: https://api.github.com/repos/simonw/datasette/issues/685
node_id: MDEyOklzc3VlQ29tbWVudDYyNTgzOTA0Ng==
created_at: 2020-05-08T14:20:43Z
updated_at: 2020-05-08T14:25:05Z
author_association: OWNER
body:
  The API design for the `.execute()` function is actually fine, I think it's more the API of the returned `Results` object that I want to improve.

  That object encapsulates the returned data, the named columns and whether or not the result was truncated.

      return Results(rows, truncated, cursor.description)

  The `rows` argument comes from either `rows = cursor.fetchmany(max_returned_rows + 1)` or `rows = cursor.fetchall()`. It's a Python list of `sqlite3.Row` objects.

  Here's the current class implementation:

  https://github.com/simonw/datasette/blob/0784f2ef9d3ff6dd9df05f54cb51de29a6d11764/datasette/utils/__init__.py#L54-L68

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/685#issuecomment-625841397
issue_url: https://api.github.com/repos/simonw/datasette/issues/685
node_id: MDEyOklzc3VlQ29tbWVudDYyNTg0MTM5Nw==
created_at: 2020-05-08T14:25:56Z
updated_at: 2020-05-08T14:27:15Z
author_association: OWNER
body:
  Maybe all I really want here is to add length and access-by-index methods to the Results class? So I don't have to do `results.rows[0][0]` to get at a single returned value.

  Also how about a `results.single_value()` method which returns a value only if there is a single row with a single column, otherwise raises an exception?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/685#issuecomment-625849497
issue_url: https://api.github.com/repos/simonw/datasette/issues/685
node_id: MDEyOklzc3VlQ29tbWVudDYyNTg0OTQ5Nw==
created_at: 2020-05-08T14:43:13Z
updated_at: 2020-05-08T14:43:13Z
author_association: OWNER
body:
  I'm going to add `.first()` and `.last()` methods too. These will return the first or last row, or `None` if the results are empty (rather than raising an `IndexError`).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/685#issuecomment-625865434
issue_url: https://api.github.com/repos/simonw/datasette/issues/685
node_id: MDEyOklzc3VlQ29tbWVudDYyNTg2NTQzNA==
created_at: 2020-05-08T15:16:43Z
updated_at: 2020-05-08T15:16:43Z
author_association: OWNER
body:
  Documentation for improved design: https://github.com/simonw/datasette/pull/763/files#diff-87703da03a02e0f8dae01f39abd6250f

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/685#issuecomment-625866419
issue_url: https://api.github.com/repos/simonw/datasette/issues/685
node_id: MDEyOklzc3VlQ29tbWVudDYyNTg2NjQxOQ==
created_at: 2020-05-08T15:18:55Z
updated_at: 2020-05-08T15:18:55Z
author_association: OWNER
body:
  This code should live somewhere other than `datasette/utils/__init__.py`. Especially the exceptions, since calling code needs to be able to sensibly import them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/685#issuecomment-625887196
issue_url: https://api.github.com/repos/simonw/datasette/issues/685
node_id: MDEyOklzc3VlQ29tbWVudDYyNTg4NzE5Ng==
created_at: 2020-05-08T16:03:16Z
updated_at: 2020-05-08T16:03:16Z
author_association: OWNER
body:
  [ Fun aside: I implemented and shipped this entire branch in my browser using the beta of GitHub's new [Codespaces](https://github.com/features/codespaces) ]

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/685#issuecomment-625889303
issue_url: https://api.github.com/repos/simonw/datasette/issues/685
node_id: MDEyOklzc3VlQ29tbWVudDYyNTg4OTMwMw==
created_at: 2020-05-08T16:07:55Z
updated_at: 2020-05-08T16:07:55Z
author_association: OWNER
body:
  New documentation is live here: https://datasette.readthedocs.io/en/latest/internals.html#database-class

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/685#issuecomment-625889673
issue_url: https://api.github.com/repos/simonw/datasette/issues/685
node_id: MDEyOklzc3VlQ29tbWVudDYyNTg4OTY3Mw==
created_at: 2020-05-08T16:08:40Z
updated_at: 2020-05-08T16:08:40Z
author_association: OWNER
body:
  I'm going to ship a release with just this change purely so I can start depending on it from my in-development https://github.com/simonw/datasette-media plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/685#issuecomment-625890190
issue_url: https://api.github.com/repos/simonw/datasette/issues/685
node_id: MDEyOklzc3VlQ29tbWVudDYyNTg5MDE5MA==
created_at: 2020-05-08T16:09:44Z
updated_at: 2020-05-08T16:09:56Z
author_association: OWNER
body:
  Re-opening this ticket because I forgot to document `execute_fn()` (the old `execute_against_connection_in_thread()` method).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-625947133
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDYyNTk0NzEzMw==
created_at: 2020-05-08T18:13:06Z
updated_at: 2020-05-08T18:13:06Z
author_association: MEMBER
body:
  `datasette-media` will be able to handle this once I implement https://github.com/simonw/datasette-media/issues/3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/619#issuecomment-626006493
issue_url: https://api.github.com/repos/simonw/datasette/issues/619
node_id: MDEyOklzc3VlQ29tbWVudDYyNjAwNjQ5Mw==
created_at: 2020-05-08T20:29:12Z
updated_at: 2020-05-08T20:29:12Z
author_association: NONE
body:
  just trying out datasette and quite like it, thanks! i found this issue annoying enough to have a go at a fix. have you any thoughts on a good approach? (i'm happy to dig in myself if you haven't thought about it yet, but wanted to check if you had an idea for how to fix when you raised the issue)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/254#issuecomment-626340387
issue_url: https://api.github.com/repos/simonw/datasette/issues/254
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM0MDM4Nw==
created_at: 2020-05-10T14:54:13Z
updated_at: 2020-05-10T14:54:13Z
author_association: NONE
body:
  This has now been resolved and is not present in current version of datasette. 

  Sample query @simonw mentioned now returns as expected. 

  https://aggreg8streams.tinyviking.ie/simplestreams?sql=select+*+from+cloudimage+where+%22content_id%22+%3D+%22com.ubuntu.cloud%3Areleased%3Adownload%22+order+by+id+limit+10

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388764
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM4ODc2NA==
created_at: 2020-05-10T20:58:52Z
updated_at: 2020-05-10T20:58:52Z
author_association: MEMBER
body:
  More from the debugger:
  ```
  > /Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py(614)place()
  -> self._place = PlaceInfo5(self._info["reverse_geolocation"])
  ```
  And:
  ```
  > /Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/utils.py(91)osxphoto_to_row()
  -> place = photo.place
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388837
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM4ODgzNw==
created_at: 2020-05-10T20:59:32Z
updated_at: 2020-05-10T20:59:32Z
author_association: MEMBER
body:
  So it appears it's possible for `photo.place` to raise that exception. A workaround could be to catch that and treat those photos as not having a place.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626390317
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM5MDMxNw==
created_at: 2020-05-10T21:11:24Z
updated_at: 2020-05-10T21:50:58Z
author_association: CONTRIBUTOR
body:
  Ugh....Yeah, I think easiest is to catch the exception and return no place as you suggest.  This particular bit of code involves un-archiving a serialized NSKeyedArchiver which uses an object table and it is certainly possible to create a circular reference that way.  Because this is happening in the decode, the circular reference must be in the original data.  Does Photos show valid reverse geolocation info for the photo in question? If so, Photos may be doing something beyond a simple decode of the binary plist.  For now, I'll push a patch to catch the exception.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626390456
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/110
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM5MDQ1Ng==
created_at: 2020-05-10T21:12:40Z
updated_at: 2020-05-10T21:12:40Z
author_association: OWNER
body:
  It definitely makes sense to me that this library should support `decimal.Decimal`.

  Here are the current supported types:

  https://github.com/simonw/sqlite-utils/blob/396bee92364fc3a88f6c76969366dd1c4c9c944d/sqlite_utils/db.py#L35-L55

  Should `decimal.Decimal` be stored as a `text` or `float` in SQLite I wonder?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626390822
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/110
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM5MDgyMg==
created_at: 2020-05-10T21:15:28Z
updated_at: 2020-05-10T21:15:28Z
author_association: OWNER
body:
  https://www.sqlite.org/datatype3.html#affinity_name_examples suggests that `DECIMAL(10,5)` should be mapped to the SQLite affinity of `NUMERIC` - which I've not worked with before.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391063
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/110
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM5MTA2Mw==
created_at: 2020-05-10T21:17:16Z
updated_at: 2020-05-10T21:17:16Z
author_association: OWNER
body:
  From https://www.sqlite.org/datatype3.html#type_affinity :
  > A column with NUMERIC affinity may contain values using all five storage classes. When text data is inserted into a NUMERIC column, the storage class of the text is converted to INTEGER or REAL (in order of preference) if the text is a well-formed integer or real literal, respectively. If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL. For conversions between TEXT and REAL storage classes, only the first 15 significant decimal digits of the number are preserved. If the TEXT value is not a well-formed integer or real literal, then the value is stored as TEXT.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391217
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/110
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM5MTIxNw==
created_at: 2020-05-10T21:18:28Z
updated_at: 2020-05-10T21:18:28Z
author_association: OWNER
body:
  `sqlite-utils` currently treats the SQLite NUMERIC concept as a float:

  https://github.com/simonw/sqlite-utils/blob/daf2a245aa4e0b0cf62a94c1232cfb858821803b/tests/test_column_affinity.py#L28-L30

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391307
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/110
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM5MTMwNw==
created_at: 2020-05-10T21:19:04Z
updated_at: 2020-05-10T21:19:04Z
author_association: OWNER
body:
  I'm going to set it up so that Python `decimal.Decimal` is treated in a FLOAT column, until someone convinces me otherwise!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395103
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM5NTEwMw==
created_at: 2020-05-10T21:51:36Z
updated_at: 2020-05-10T21:51:36Z
author_association: MEMBER
body:
  @RhetTbull I tried that workaround and it turns out I'm getting this error on ALL of my photos now!

  It's weird: a few day ago this wasn't happening. Now it's happening to everything. I'm not sure what I might have changed. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395209
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM5NTIwOQ==
created_at: 2020-05-10T21:52:42Z
updated_at: 2020-05-10T21:52:42Z
author_association: MEMBER
body:
  Aha! It looks like I accidentally installed the old bplist into the same environment:
  ```
  $ pip freeze | grep bpylist
  bpylist==0.1.4
  bpylist2==3.0.0
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395507
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM5NTUwNw==
created_at: 2020-05-10T21:54:45Z
updated_at: 2020-05-10T21:54:45Z
author_association: CONTRIBUTOR
body:
  @simonw does Photos show valid reverse geolocation info?  Are you sure you're using [bpylist2](https://github.com/xa4a/bpylist2) and not bpylist?  They're both unfortunately imported as "bpylist" so if you somehow got the wrong (original bpylist) version installed, it could be the issue. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395641
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM5NTY0MQ==
created_at: 2020-05-10T21:55:54Z
updated_at: 2020-05-10T21:55:54Z
author_association: CONTRIBUTOR
body:
  Did removing old bpylist solve the original problem or do you still have a photo that throws circular reference?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395781
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM5NTc4MQ==
created_at: 2020-05-10T21:57:09Z
updated_at: 2020-05-10T21:57:09Z
author_association: MEMBER
body:
  Yes, I just recreated my virtual environment from scratch and the error went away.

  The problem occurred when I ran `pip install datasette-bplist` in the same virtual environment - https://github.com/simonw/datasette-bplist/blob/master/setup.py depends on `bpylist` which is incompatible with `bpylist2`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626396379
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDYyNjM5NjM3OQ==
created_at: 2020-05-10T22:01:48Z
updated_at: 2020-05-10T22:01:48Z
author_association: CONTRIBUTOR
body:
  Frustrates me when package authors create a "drop in" replacement with the same import name...this kind of thing has bitten me more than once! Would've been nicer I think for bpylist2 to do "import bpylist2 as bpylist"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/111#issuecomment-626417220
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/111
node_id: MDEyOklzc3VlQ29tbWVudDYyNjQxNzIyMA==
created_at: 2020-05-11T00:46:04Z
updated_at: 2020-05-11T00:46:04Z
author_association: OWNER
body:
  Docs:
  * https://sqlite-utils.readthedocs.io/en/latest/cli.html#dropping-tables
  * https://sqlite-utils.readthedocs.io/en/latest/cli.html#dropping-views

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-626667235
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22
node_id: MDEyOklzc3VlQ29tbWVudDYyNjY2NzIzNQ==
created_at: 2020-05-11T12:20:34Z
updated_at: 2020-05-11T12:20:34Z
author_association: CONTRIBUTOR
body:
  @simonw FYI, osxphotos includes a built in ExifTool class that uses [exiftool](https://exiftool.org/) to read and write exif data.  It's not exposed yet in the docs because I really only use it right now in the osphotos command line interface to write tags when exporting.  In v0.28.16 (just pushed) I added an ExifTool.as_dict() method which will give you a dict with all the exif tags in a file.  For example:

  ```python
  import osxphotos
  photos = osxphotos.PhotosDB().photos()
  exiftool = osxphotos.exiftool.ExifTool(photos[0].path)
  exifdata = exiftool.as_dict()
  tags = exifdata["IPTC:Keywords"]
  ```

  Not as elegant perhaps as a python only implementation because ExifTool has to make subprocess calls to an external tool but exiftool is by far the best tool available for reading and writing EXIF data and it does support HEIC.

  As for implementation, ExifTool uses a singleton pattern so the first time you instantiate it, it spawns an IPC to exiftool but then keeps it open and uses the same process for any subsequent calls (even on different files). 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-626807487
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYyNjgwNzQ4Nw==
created_at: 2020-05-11T16:23:57Z
updated_at: 2020-05-11T16:24:59Z
author_association: NONE
body:
  `Authorization: bearer xxx` auth for API keys is a plus plus for me. Looked into just adding this into your `Flask`  logic but learned this project doesn't use flask. Interesting 🤔

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/764#issuecomment-626808805
issue_url: https://api.github.com/repos/simonw/datasette/issues/764
node_id: MDEyOklzc3VlQ29tbWVudDYyNjgwODgwNQ==
created_at: 2020-05-11T16:26:17Z
updated_at: 2020-05-11T16:27:27Z
author_association: OWNER
body:
  The keys here can be anything: https://packaging.python.org/guides/distributing-packages-using-setuptools/#project-urls

  So where do the icons come from? Turns out the PyPI site has special case rules for the icons here:  https://github.com/pypa/warehouse/blob/2f00f4a9f208546ff0ebb6a6e61439021ca60a43/warehouse/templates/packaging/detail.html#L16-L60

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/764#issuecomment-626810129
issue_url: https://api.github.com/repos/simonw/datasette/issues/764
node_id: MDEyOklzc3VlQ29tbWVudDYyNjgxMDEyOQ==
created_at: 2020-05-11T16:28:43Z
updated_at: 2020-05-11T16:28:43Z
author_association: OWNER
body:
  For Datasette I'll go with:
  * Documentation
  * Changelog
  * Live demo
  * Source code
  * Issues
  * CI

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/765#issuecomment-626874374
issue_url: https://api.github.com/repos/simonw/datasette/issues/765
node_id: MDEyOklzc3VlQ29tbWVudDYyNjg3NDM3NA==
created_at: 2020-05-11T18:25:43Z
updated_at: 2020-05-11T18:25:43Z
author_association: OWNER
body:
  Trickiness here is what tag to use. Do we use the tag of the installed copy of Datasette that is running `datasette publish`? That would mean you don't get the latest features in your deployed release.

  Could hit an API to figure out the most recent version? Bit odd.

  Could we output the version of Datasette that was deployed and tell people they can run `--latest` to force the most recent release?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-626941278
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22
node_id: MDEyOklzc3VlQ29tbWVudDYyNjk0MTI3OA==
created_at: 2020-05-11T20:25:58Z
updated_at: 2020-05-11T20:25:58Z
author_association: MEMBER
body:
  Interesting - do you know if there's anything the `exiftool` process handles that `ExifReader` doesn't?

  I'm actually just going to extract a subset of the EXIF data at first - since the original photo files will always be available I don't feel the need to get everything out for the first step.

  My plan is to use EXIF to help support photo collections that aren't in Apple Photos - I'm going to build a database table keyed by the `sha256` of each photo that extracts the camera make, lens, a few settings (ISO, aperture etc) and the GPS lat/lon.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-626943809
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYyNjk0MzgwOQ==
created_at: 2020-05-11T20:30:07Z
updated_at: 2020-05-11T20:31:18Z
author_association: OWNER
body:
  I implemented bearer tokens in a private project of mine as a one-off plugin. I'm going to extract that out into a installable plugin soon. For the moment, my `plugins/token_auth.py` file looks like this:
  ```python
  from datasette import hookimpl
  import secrets


  class TokenAuth:
      def __init__(
          self, app, secret, auth,
      ):
          self.app = app
          self.secret = secret
          self.auth = auth

      async def __call__(self, scope, receive, send):
          if scope.get("type") != "http":
              return await self.app(scope, receive, send)

          authorization = dict(scope.get("headers") or {}).get(b"authorization") or b""
          expected = "Bearer {}".format(self.secret).encode("utf8")

          if secrets.compare_digest(authorization, expected):
              scope = dict(scope, auth=self.auth)

          return await self.app(scope, receive, send)


  @hookimpl(trylast=True)
  def asgi_wrapper(datasette):
      config = datasette.plugin_config("token-auth") or {}
      secret = config.get("secret")
      auth = config.get("auth")

      def wrap_with_asgi_auth(app):
          return TokenAuth(app, secret=secret, auth=auth,)

      return wrap_with_asgi_auth
  ```
  Then I have the following in `metadata.json`:
  ```json
  {
      "plugins": {
          "token-auth": {
              "auth": {
                  "name": "token-bot"
              },
              "secret": {
                  "$env": "TOKEN_SECRET"
              }
          }
      }
  }
  ```
  And a `TOKEN_SECRET` environment variable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-626945281
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYyNjk0NTI4MQ==
created_at: 2020-05-11T20:32:33Z
updated_at: 2020-05-11T20:32:33Z
author_association: OWNER
body:
  I did have a bit of trouble with this one-off plugin getting it to load in the correct order - since I need authentication to work if EITHER the one-off plugin spots a token or my `datasette-auth-github` plugin authenticates the user.

  That's why I want authentication as a core Datasette concept - so plugins like these can easily play together in a predictable manner.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-627007458
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22
node_id: MDEyOklzc3VlQ29tbWVudDYyNzAwNzQ1OA==
created_at: 2020-05-11T22:51:52Z
updated_at: 2020-05-11T22:52:26Z
author_association: CONTRIBUTOR
body:
  I'm not familiar with `ExifReader`.  I wrote my own wrapper around `exiftool` because I wanted a simple way to write EXIF data when exporting photos (e.g. writing out to PersonInImage and keywords to IPTC:Keywords) and the existing python packages like [pyexiftool](https://github.com/smarnach/pyexiftool) didn't do quite what I wanted.  If all you're after is the camera and shot info, that's available in `ZEXTENDEDATTRIBUTES` table.  I've got an open issue [#11](https://github.com/RhetTbull/osxphotos/issues/11) to add this to osxphotos but it hasn't bubbled to the top of my backlog yet.  

  osxphotos will give you the location info: `PhotoInfo.location` returns a tuple of (lat, lon) though this info is in ZEXTENDEDATTRIBUTES too (though it might not be correct as I believe Photos creates this table at import and the user might have changed the location of a photo, e.g. if camera didn't have GPS).

  ```sql
  CREATE TABLE ZEXTENDEDATTRIBUTES (
    Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER, 
    Z_OPT INTEGER, ZFLASHFIRED INTEGER, 
    ZISO INTEGER, ZMETERINGMODE INTEGER, 
    ZSAMPLERATE INTEGER, ZTRACKFORMAT INTEGER, 
    ZWHITEBALANCE INTEGER, ZASSET INTEGER, 
    ZAPERTURE FLOAT, ZBITRATE FLOAT, ZDURATION FLOAT, 
    ZEXPOSUREBIAS FLOAT, ZFOCALLENGTH FLOAT, 
    ZFPS FLOAT, ZLATITUDE FLOAT, ZLONGITUDE FLOAT, 
    ZSHUTTERSPEED FLOAT, ZCAMERAMAKE VARCHAR, 
    ZCAMERAMODEL VARCHAR, ZCODEC VARCHAR, 
    ZLENSMODEL VARCHAR
  );
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/112#issuecomment-627035928
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/112
node_id: MDEyOklzc3VlQ29tbWVudDYyNzAzNTkyOA==
created_at: 2020-05-12T00:24:16Z
updated_at: 2020-05-12T00:24:51Z
author_association: OWNER
body:
  I would add `ignore=True/False` as a parameter here:
  https://github.com/simonw/sqlite-utils/blob/af3f81b540923f2cf04c76cfa81b0d811c0084bf/sqlite_utils/db.py#L731-L761

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/112#issuecomment-627036010
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/112
node_id: MDEyOklzc3VlQ29tbWVudDYyNzAzNjAxMA==
created_at: 2020-05-12T00:24:37Z
updated_at: 2020-05-12T00:24:37Z
author_association: OWNER
body:
  Here's how it works for `create_view`:
  https://github.com/simonw/sqlite-utils/blob/af3f81b540923f2cf04c76cfa81b0d811c0084bf/sqlite_utils/db.py#L327-L336

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/112#issuecomment-627036475
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/112
node_id: MDEyOklzc3VlQ29tbWVudDYyNzAzNjQ3NQ==
created_at: 2020-05-12T00:26:25Z
updated_at: 2020-05-12T00:26:48Z
author_association: OWNER
body:
  Question: if you use `ignore=True` should it still raise an error if one of the columns you referenced does not exist?

  That's tricky. I'm leaning towards "that's still an error".

  Are we ignoring an already existing foreign key, or ignoring all errors?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/112#issuecomment-627036749
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/112
node_id: MDEyOklzc3VlQ29tbWVudDYyNzAzNjc0OQ==
created_at: 2020-05-12T00:27:24Z
updated_at: 2020-05-12T00:27:24Z
author_association: OWNER
body:
  Here's an example of some code that would be cleaner with this mechanism: https://github.com/dogsheep/swarm-to-sqlite/blob/f4a82633da927cde672c9d9af92930bfca2e3ddf/swarm_to_sqlite/utils.py#L120-L143

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-628405453
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22
node_id: MDEyOklzc3VlQ29tbWVudDYyODQwNTQ1Mw==
created_at: 2020-05-14T05:59:53Z
updated_at: 2020-05-14T05:59:53Z
author_association: CONTRIBUTOR
body:
  I've added support for the above exif data to [v0.28.17](https://github.com/RhetTbull/osxphotos/releases/tag/v0.28.17) of osxphotos.  `PhotoInfo.exif_info` will return an `ExifInfo` [dataclass](https://docs.python.org/3/library/dataclasses.html) object with the following properties:

  ```python
      flash_fired: bool
      iso: int
      metering_mode: int
      sample_rate: int
      track_format: int
      white_balance: int
      aperture: float
      bit_rate: float
      duration: float
      exposure_bias: float
      focal_length: float
      fps: float
      latitude: float
      longitude: float
      shutter_speed: float
      camera_make: str
      camera_model: str
      codec: str
      lens_model: str
  ```

  It's not all the EXIF data available in most files but is the data Photos deems important to save.  Of course, you can get all the exif_data

  Note: this only works in Photos 5.  As best as I can tell, EXIF data is not stored in the database for earlier versions. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/729#issuecomment-629032425
issue_url: https://api.github.com/repos/simonw/datasette/issues/729
node_id: MDEyOklzc3VlQ29tbWVudDYyOTAzMjQyNQ==
created_at: 2020-05-15T05:17:09Z
updated_at: 2020-05-15T05:17:39Z
author_association: OWNER
body:
  Not sure the best way to do this. Here's an example using very subtle colour changes - but I'm worried it's not at all discoverable:

  <img width="673" alt="fixtures__sortable__201_rows" src="https://user-images.githubusercontent.com/9599/82014033-92942380-9630-11ea-846b-d117af7b3dde.png">

  That's with `#4c0077` for floats and `#666666` for integers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/729#issuecomment-629032619
issue_url: https://api.github.com/repos/simonw/datasette/issues/729
node_id: MDEyOklzc3VlQ29tbWVudDYyOTAzMjYxOQ==
created_at: 2020-05-15T05:17:53Z
updated_at: 2020-05-15T05:18:29Z
author_association: OWNER
body:
  I like the gray. The purple is a bit confusing.

  I could have both integers and floats show up as gray? The floating point should help show which is which.

  Maybe I could default floating point columns with integer values in them to `11.0` just to make it clear that they are floats?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/729#issuecomment-629033619
issue_url: https://api.github.com/repos/simonw/datasette/issues/729
node_id: MDEyOklzc3VlQ29tbWVudDYyOTAzMzYxOQ==
created_at: 2020-05-15T05:21:17Z
updated_at: 2020-05-15T05:21:30Z
author_association: OWNER
body:
  Turns out `1.0` and `2` stored in a float column already displays like that:
  ```
  echo '[{"foo": 1.0, "bar": 1}, {"foo": 2, "bar": 2}]' | sqlite-utils insert /tmp/db.db t -
  ```
  <img width="542" alt="db__t__2_rows" src="https://user-images.githubusercontent.com/9599/82014303-3382de80-9631-11ea-9d6b-9be1f178c8cb.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/729#issuecomment-629042437
issue_url: https://api.github.com/repos/simonw/datasette/issues/729
node_id: MDEyOklzc3VlQ29tbWVudDYyOTA0MjQzNw==
created_at: 2020-05-15T05:52:39Z
updated_at: 2020-05-15T05:52:39Z
author_association: OWNER
body:
  `type-pk` is a more accurate name than `type-link` - since people might assume that `type-link` refers to columns containing a link or a URL or similar.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/729#issuecomment-629050775
issue_url: https://api.github.com/repos/simonw/datasette/issues/729
node_id: MDEyOklzc3VlQ29tbWVudDYyOTA1MDc3NQ==
created_at: 2020-05-15T06:17:12Z
updated_at: 2020-05-15T06:17:12Z
author_association: OWNER
body:
  ![4F8D336A-ECEB-4C68-A859-C8A3DA546E9C](https://user-images.githubusercontent.com/9599/82017875-fae70300-9638-11ea-9cc2-3969299ae9a0.jpeg)

  I don't like how the column headers themselves are no longer black in mobile view.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/729#issuecomment-630323870
issue_url: https://api.github.com/repos/simonw/datasette/issues/729
node_id: MDEyOklzc3VlQ29tbWVudDYzMDMyMzg3MA==
created_at: 2020-05-18T17:20:02Z
updated_at: 2020-05-18T17:20:02Z
author_association: OWNER
body:
  Deployed here: https://covid-19.datasettes.com/covid/economist_excess_deaths - and it helped me spot a bug! The floating point numbers there were being treated as strings:

  <img width="818" alt="covid__economist_excess_deaths__2_559_rows" src="https://user-images.githubusercontent.com/9599/82241500-1f6d0480-98f1-11ea-9ce7-fa74bc449546.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/113#issuecomment-631084748
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/113
node_id: MDEyOklzc3VlQ29tbWVudDYzMTA4NDc0OA==
created_at: 2020-05-19T21:11:22Z
updated_at: 2020-05-19T21:11:22Z
author_association: OWNER
body:
  Here's how `twitter-to-sqlite` does it: https://github.com/dogsheep/twitter-to-sqlite/blob/6be0ffcee24504fa2964b0e526842b8cfce7567b/twitter_to_sqlite/utils.py#L506-L511

  ```python
                  attach_sql = """
                      ATTACH DATABASE '{}' AS [{}];
                  """.format(
                      str(pathlib.Path(filepath).resolve()), alias
                  )
                  db.conn.execute(attach_sql)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/23#issuecomment-631120771
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDYzMTEyMDc3MQ==
created_at: 2020-05-19T22:32:48Z
updated_at: 2020-05-19T22:32:48Z
author_association: MEMBER
body:
  Documentation: https://github.com/dogsheep/photos-to-sqlite/blob/e2fab012551eed05278040b5d57e7373a1b9a0bf/README.md#creating-a-subset-database

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631127454
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25
node_id: MDEyOklzc3VlQ29tbWVudDYzMTEyNzQ1NA==
created_at: 2020-05-19T22:48:00Z
updated_at: 2020-05-21T15:58:32Z
author_association: MEMBER
body:
  I built #23 to help with this.

      $ dogsheep-photos create-subset photos.db public.db \
          "select sha256 from apple_photos where albums like '%Public%'"

  And publish with Vercel:

      $ datasette publish now public.db --project dogsheep-photos \
          --about=dogsheep/dogsheep-photos \
          --about_url="https://github.com/dogsheep/dogsheep-photos" \
          --install=datasette-json-html \
          --install=datasette-cluster-map

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226953
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDYzMTIyNjk1Mw==
created_at: 2020-05-20T04:20:34Z
updated_at: 2020-05-20T04:20:34Z
author_association: MEMBER
body:
  Huh, it looks like Circle CI picked up the name change automatically. https://app.circleci.com/pipelines/github/dogsheep/dogsheep-photos

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227105
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDYzMTIyNzEwNQ==
created_at: 2020-05-20T04:21:06Z
updated_at: 2020-05-20T04:21:06Z
author_association: MEMBER
body:
  Then I just need to push a final photos-to-sqlite release that updates the README to tell people about the name change.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253136
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25
node_id: MDEyOklzc3VlQ29tbWVudDYzMTI1MzEzNg==
created_at: 2020-05-20T05:53:58Z
updated_at: 2020-05-20T05:53:58Z
author_association: MEMBER
body:
  Updated deploy command:
  ```
  datasette publish now public.db --project dogsheep-photos \
      --about=dogsheep/dogsheep-photos \
      --about_url="https://github.com/dogsheep/dogsheep-photos" \
      --install=datasette-json-html \
      --install=datasette-cluster-map \
      --title "Dogsheep Photos demo"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253852
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25
node_id: MDEyOklzc3VlQ29tbWVudDYzMTI1Mzg1Mg==
created_at: 2020-05-20T05:56:17Z
updated_at: 2020-05-21T22:26:16Z
author_association: MEMBER
body:
  I have a `deploy-demo.sh` script now:
  ```bash
  #!/bin/bash
  if [ -f public.db ]; then
    rm public.db
  fi
  pipenv run dogsheep-photos create-subset photos.db public.db \
      "select sha256 from apple_photos where albums like '%Public%'"
  pipenv run sqlite-utils create-view public.db photos_on_a_map \
      "select
        date,
        latitude,
        longitude,
        apple_photos.sha256,
        uploads.ext,
        json_object(
          'title',
          'Taken on ' || date,
          'image',
          'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=400',
          'link',
          'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=1200'
        ) as popup
      from
        apple_photos
        join uploads on apple_photos.sha256 = uploads.sha256
      where
        latitude is not null
      order by
        date desc" \
      --replace
  pipenv run datasette publish now public.db --project dogsheep-photos \
      --about=dogsheep/dogsheep-photos \
      --about_url="https://github.com/dogsheep/dogsheep-photos" \
      --install=datasette-json-html \
      --install=datasette-pretty-json \
      --install=datasette-cluster-map>=0.10 \
      --title "Dogsheep Photos demo"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-632233393
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYzMjIzMzM5Mw==
created_at: 2020-05-21T17:16:15Z
updated_at: 2020-05-21T17:17:43Z
author_association: OWNER
body:
  I just hit this bug myself, or at least a variant of it!

  https://github.com/simonw/museums/runs/697063068?check_suite_focus=true

  ```
  2020-05-21T17:14:02.2596471Z Traceback (most recent call last):
  2020-05-21T17:14:02.2599146Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py", line 605, in link_or_copy_directory
  2020-05-21T17:14:02.2599861Z     shutil.copytree(src, dst, copy_function=os.link)
  2020-05-21T17:14:02.2600377Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 554, in copytree
  2020-05-21T17:14:02.2600786Z     return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  2020-05-21T17:14:02.2601196Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 510, in _copytree
  2020-05-21T17:14:02.2601580Z     raise Error(errors)
  2020-05-21T17:14:02.2604565Z shutil.Error: [('/home/runner/work/museums/museums/templates/row-browse-museums.html', '/tmp/tmpq47xi96y/datasette/templates/row-browse-museums.html',
  "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/row-browse-museums.html' -> '/tmp/tmpq47xi96y/datasette/templates/row-browse-museums.html'"), ('/home/runner/work/museums/museums/templates/index.html', '/tmp/tmpq47xi96y/datasette/templates/index.html',
  "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/index.html' -> '/tmp/tmpq47xi96y/datasette/templates/index.html'"), ('/home/runner/work/museums/museums/templates/query-browse-search.html', '/tmp/tmpq47xi96y/datasette/templates/query-browse-search.html',
  "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/query-browse-search.html' -> '/tmp/tmpq47xi96y/datasette/templates/query-browse-search.html'"), ('/home/runner/work/museums/museums/templates/_analytics.html', '/tmp/tmpq47xi96y/datasette/templates/_analytics.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/_analytics.html' -> '/tmp/tmpq47xi96y/datasette/templates/_analytics.html'"), ('/home/runner/work/museums/museums/templates/_museum_card.html', '/tmp/tmpq47xi96y/datasette/templates/_museum_card.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/_museum_card.html' -> '/tmp/tmpq47xi96y/datasette/templates/_museum_card.html'"), ('/home/runner/work/museums/museums/templates/pages/about.html', '/tmp/tmpq47xi96y/datasette/templates/pages/about.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/pages/about.html' -> '/tmp/tmpq47xi96y/datasette/templates/pages/about.html'"), ('/home/runner/work/museums/museums/templates/pages/map.html', '/tmp/tmpq47xi96y/datasette/templates/pages/map.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/pages/map.html' -> '/tmp/tmpq47xi96y/datasette/templates/pages/map.html'")]
  2020-05-21T17:14:02.2605437Z 
  2020-05-21T17:14:02.2605797Z During handling of the above exception, another exception occurred:
  2020-05-21T17:14:02.2606102Z 
  2020-05-21T17:14:02.2606423Z Traceback (most recent call last):
  2020-05-21T17:14:02.2606817Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette", line 8, in <module>
  2020-05-21T17:14:02.2607189Z     sys.exit(cli())
  2020-05-21T17:14:02.2607907Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
  2020-05-21T17:14:02.2608347Z     return self.main(*args, **kwargs)
  2020-05-21T17:14:02.2609024Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
  2020-05-21T17:14:02.2609451Z     rv = self.invoke(ctx)
  2020-05-21T17:14:02.2610116Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
  2020-05-21T17:14:02.2610550Z     return _process_result(sub_ctx.command.invoke(sub_ctx))
  2020-05-21T17:14:02.2611451Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
  2020-05-21T17:14:02.2611989Z     return _process_result(sub_ctx.command.invoke(sub_ctx))
  2020-05-21T17:14:02.2612682Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
  2020-05-21T17:14:02.2613117Z     return ctx.invoke(self.callback, **ctx.params)
  2020-05-21T17:14:02.2613801Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
  2020-05-21T17:14:02.2614235Z     return callback(*args, **kwargs)
  2020-05-21T17:14:02.2614946Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/cloudrun.py", line 111, in cloudrun
  2020-05-21T17:14:02.2615371Z     with temporary_docker_directory(
  2020-05-21T17:14:02.2615769Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/contextlib.py", line 113, in __enter__
  2020-05-21T17:14:02.2616135Z     return next(self.gen)
  2020-05-21T17:14:02.2616957Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py", line 392, in temporary_docker_directory
  2020-05-21T17:14:02.2617403Z     link_or_copy_directory(
  2020-05-21T17:14:02.2618298Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py", line 607, in link_or_copy_directory
  2020-05-21T17:14:02.2619270Z     shutil.copytree(src, dst)
  2020-05-21T17:14:02.2619872Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 554, in copytree
  2020-05-21T17:14:02.2622179Z     return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  2020-05-21T17:14:02.2622609Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 455, in _copytree
  2020-05-21T17:14:02.2622878Z     os.makedirs(dst, exist_ok=dirs_exist_ok)
  2020-05-21T17:14:02.2623157Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/os.py", line 223, in makedirs
  2020-05-21T17:14:02.2623405Z     mkdir(name, mode)
  2020-05-21T17:14:02.2623978Z FileExistsError: [Errno 17] File exists: '/tmp/tmpq47xi96y/datasette/templates'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-632253073
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYzMjI1MzA3Mw==
created_at: 2020-05-21T17:54:55Z
updated_at: 2020-05-21T17:54:55Z
author_association: OWNER
body:
  I think this is the fix! I just landed it in master.

  @aborruso you can install Datasette master to test it out like this:

      pip install https://github.com/simonw/datasette/archive/cee671a58f417f827d1735b1abaa40716534ea67.zip

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-632305868
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYzMjMwNTg2OA==
created_at: 2020-05-21T19:43:23Z
updated_at: 2020-05-21T19:43:23Z
author_association: NONE
body:
  @simonw now I have

  ```
  Traceback (most recent call last):
    File "/home/aborruso/.local/bin/datasette", line 8, in <module>
      sys.exit(cli())
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 103, in heroku
      extra_metadata,
    File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
      return next(self.gen)
    File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 191, in temporary_heroku_directory
      os.path.join(tmp.name, "templates"),
    File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 605, in link_or_copy_directory
      shutil.copytree(src, dst, copy_function=os.link, dirs_exist_ok=True)
  TypeError: copytree() got an unexpected keyword argument 'dirs_exist_ok'
  ```

  Do I must open a new issue?

  Thank you

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/767#issuecomment-632398475
issue_url: https://api.github.com/repos/simonw/datasette/issues/767
node_id: MDEyOklzc3VlQ29tbWVudDYzMjM5ODQ3NQ==
created_at: 2020-05-21T23:36:00Z
updated_at: 2020-05-21T23:36:00Z
author_association: OWNER
body:
  Interesting. So this would take effect exclusively on pages that link to the queries, which I think just means this page: https://latest.datasette.io/fixtures

  How about this for the design: there's a new optional key in the metadata configuration for a query called `"fragment"` - used like this:

  ```json
  {
      "databases": {
          "fixtures": {
              "queries": {
                  "neighborhood_search": {
                      "sql": "\nselect neighborhood, facet_cities.name, state\nfrom facetable\n    join facet_cities\n        on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%'\norder by neighborhood;\n",
                      "title": "Search neighborhoods",
                      "description_html": "<b>Demonstrating</b> simple like search",
                      "name": "neighborhood_search",
                      "fragment": "sql-format"
                  }
              }
          }
      }
  }
  ```
  Because of the `"fragment": "sql-format"` bit there the link to that query from https://latest.datasette.io/fixtures would be to `https://latest.datasette.io/fixtures/neighborhood_search#sql-format`.

  Does that cover what you need here? Have I missed anything?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633629944
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDYzMzYyOTk0NA==
created_at: 2020-05-25T15:47:42Z
updated_at: 2020-05-25T15:47:42Z
author_association: MEMBER
body:
  I'll add a proper section to the README, but for the moment here's how I do this.

  First, install `datasette` and the `datasette-media` plugin.

  Create a `metadata.yaml` file with the following content:

  ```yaml
  plugins:
    datasette-media:
      photo:
        sql: |-
          select path as filepath, 200 as resize_height from apple_photos where uuid = :key
      photo-big:
        sql: |-
          select path as filepath, 1024 as resize_height from apple_photos where uuid = :key
  ```
  Now run `datasette -m metadata.yaml photos.db` - thumbnails will be served at http://127.0.0.1:8001/-/media/photo/F4469918-13F3-43D8-9EC1-734C0E6B60AD and larger sizes of the image at http://127.0.0.1:8001/-/media/photo-big/A8B02C7D-365E-448B-9510-69F80C26304D

  I also made myself two custom pages, one showing recent images and one showing random images.

  To do this, install the `datasette-template-sql` plugin and then create a `templates/pages` directory and add these files:

  `recent-photos.html`
  ```html
  <h1>Recent photos</h1>

  <div>
  {% for photo in sql("select * from apple_photos order by date desc limit 100") %}
      <img src="/-/media/photo/{{ photo['uuid'] }}">
  {% endfor %}
  </div>
  ```

  `random-photos.html`
  ```html
  <h1>Random photos</h1>

  <div>
  {% for photo in sql("with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 100") %}
      <img src="/-/media/photo/{{ photo['uuid'] }}">
  {% endfor %}
  </div>
  ```

  Now run `datasette -m metadata.yaml photos.db --template-dir=templates/`

  Visit http://127.0.0.1:8001/random-photos to see some random photos or http://127.0.0.1:8002/recent-photos for recent photos.

  This is using this mechanism: https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633643921
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDYzMzY0MzkyMQ==
created_at: 2020-05-25T16:29:44Z
updated_at: 2020-05-25T16:29:44Z
author_association: MEMBER
body:
  https://github.com/dogsheep/dogsheep-photos/blob/dc43fa8653cb9c7238a36f52239b91d1ec916d5c/README.md#serving-photos-locally-with-datasette-media

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633704127
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDYzMzcwNDEyNw==
created_at: 2020-05-25T20:14:22Z
updated_at: 2020-05-25T20:14:22Z
author_association: MEMBER
body:
  https://github.com/dogsheep/dogsheep-photos/blob/0.4.1/README.md#serving-photos-locally-with-datasette-media

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-634254943
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYzNDI1NDk0Mw==
created_at: 2020-05-26T20:15:11Z
updated_at: 2020-05-26T20:15:11Z
author_association: OWNER
body:
  Oh no! It looks like `dirs_exist_ok` is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-634283355
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYzNDI4MzM1NQ==
created_at: 2020-05-26T21:15:34Z
updated_at: 2020-05-26T21:15:34Z
author_association: NONE
body:
  > Oh no! It looks like `dirs_exist_ok` is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.

  Thank you very much

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-634395343
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYzNDM5NTM0Mw==
created_at: 2020-05-27T02:49:26Z
updated_at: 2020-05-27T02:49:26Z
author_association: OWNER
body:
  OK, here's a new branch you can try. Install it like this:

      pip install https://github.com/simonw/datasette/archive/shutil-backport.zip

  If it works for you I'll merge that branch into master.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/744#issuecomment-634446887
issue_url: https://api.github.com/repos/simonw/datasette/issues/744
node_id: MDEyOklzc3VlQ29tbWVudDYzNDQ0Njg4Nw==
created_at: 2020-05-27T06:01:28Z
updated_at: 2020-05-27T06:01:28Z
author_association: NONE
body:
  Dear @simonw thank you for your time, now IT WORKS!!!

  I hope that this edit to datasette code is not for an exceptional case (my PC configuration) and that it will be useful to other users. 

  Thank you again!!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634852196
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDg1MjE5Ng==
created_at: 2020-05-27T18:20:46Z
updated_at: 2020-05-27T18:20:46Z
author_association: OWNER
body:
  Here's the code that calls the renderers - this needs to be expanded to check for those extra optional arguments:

  https://github.com/simonw/datasette/blob/2d099ad9c657d2cab59de91cdb8bfed2da236ef6/datasette/views/base.py#L387-L398

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634853296
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDg1MzI5Ng==
created_at: 2020-05-27T18:22:46Z
updated_at: 2020-05-27T18:22:46Z
author_association: OWNER
body:
  While I'm doing this, another feature I would like is the ability for renderers to opt-in / opt-out of being displayed as options on the page.

  https://www.niche-museums.com/browse/museums for example shows a `atom` link because the `datasette-atom` plugin is installed... but clicking it will give you a 400 error because the correct columns are not present:

  <img width="1113" alt="browse__museums__102_rows" src="https://user-images.githubusercontent.com/9599/83057883-60110000-a00c-11ea-966f-8b2a7222f1b0.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634856748
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDg1Njc0OA==
created_at: 2020-05-27T18:28:32Z
updated_at: 2020-05-27T18:28:32Z
author_association: OWNER
body:
  Here's the code that passes a list of renderers to the template:

  https://github.com/simonw/datasette/blob/2d099ad9c657d2cab59de91cdb8bfed2da236ef6/datasette/views/base.py#L411-L423

  A renderer is currently defined as a two-key dictionary:
  ```python
  @hookimpl
  def register_output_renderer(datasette):
      return {
          'extension': 'test',
          'callback': render_test
      }
  ```
  I can add a third key, `"should_suggest"` which is a function that returns `True` or `False` for a given query. If that key is missing it is assumed to return `True`.

  One catch: what arguments should be passed to the `should_suggest(...)` function?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634857975
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDg1Nzk3NQ==
created_at: 2020-05-27T18:30:29Z
updated_at: 2020-05-27T18:30:29Z
author_association: OWNER
body:
  I'll use #770 for the `should_suggest` mechanism - this issue is for the extra arguments passed to the rendering callback.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634865620
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDg2NTYyMA==
created_at: 2020-05-27T18:44:06Z
updated_at: 2020-05-27T18:44:06Z
author_association: OWNER
body:
  The existing render callback takes the following arguments:

  > `args` - dictionary
  >     The GET parameters of the request
  > 
  > `data` - dictionary
  >    The data to be rendered
  > 
  > `view_name` - string
  > The name of the view where the renderer is being called. (`index`, `database`, `table`, and `row` are the most important ones.)

  The `data` argument is a bit of a problem, because it tightly couples plugins to a currently undocumented datastructure within Datasette. Here's how `datasette-atom` picks that apart for example: https://github.com/simonw/datasette-atom/blob/095941c23c81b70c4787cdeef873c556b573b5fa/datasette_atom/__init__.py#L15-L66 - it does things like access `data["query"]["sql"]` to figure out the SQL query that was used.

  I'm going to change the design of part of this ticket. I won't break the old `data` value just yet, but I'll mark it to be deprecated by Datasette 1.0.

  I think the only plugins using it right now are my `datasette-atom` and `datasette-ics` and @russss's [datasette-geo](https://github.com/russss/datasette-geo/blob/0d24c9fd782eeae8d136ee791143ee8cbf49ebdf/datasette_plugin_geo/__init__.py#L94-L101) so hopefully changing this won't cause any wider damage.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634879258
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDg3OTI1OA==
created_at: 2020-05-27T19:09:22Z
updated_at: 2020-05-28T01:33:45Z
author_association: OWNER
body:
  OK, the new design: your callback function can take any of the following arguments:

  - `datasette` - a Datasette instance
  - `columns` - the list of columns
  - `rows` - the list of rows (each one a SQLite `Row` object)
  - `sql` - the SQL query being executed
  - `query_name` - the name of the canned query, if this is one
  - `database` - the database name
  - `table` - the table or view name
  - `request` - the request object (to be documented in #706)
  - `view_name` - the name of the view

  We will also continue to support the existing `args` and `data` arguments, but these will be undocumented and will be deprecated in Datasette 1.0. UPDATE: Decided against this, see https://github.com/simonw/datasette/issues/581#issuecomment-634946197

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634879734
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDg3OTczNA==
created_at: 2020-05-27T19:10:17Z
updated_at: 2020-05-27T19:12:36Z
author_association: OWNER
body:
  The `should_suggest` callback will take the same arguments: https://github.com/simonw/datasette/issues/770#issuecomment-634880090

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/770#issuecomment-634880090
issue_url: https://api.github.com/repos/simonw/datasette/issues/770
node_id: MDEyOklzc3VlQ29tbWVudDYzNDg4MDA5MA==
created_at: 2020-05-27T19:10:57Z
updated_at: 2020-05-28T04:22:47Z
author_association: OWNER
body:
  This `can_render` callback should take the same arguments as the redesigned `render` (previously called `callback`): https://github.com/simonw/datasette/issues/581#issuecomment-634879258

  - `datasette` - a Datasette instance
  - `columns` - the list of columns
  - `rows` - the list of rows (each one a SQLite `Row` object)
  - `sql` - the SQL query being executed
  - `query_name` - the name of the canned query, if this is one
  - `database` - the database name
  - `table` - the table or view name
  - `request` - the request object (to be documented in #706)
  - `view_name` - the name of the view

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634881287
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDg4MTI4Nw==
created_at: 2020-05-27T19:13:09Z
updated_at: 2020-05-27T19:13:09Z
author_association: OWNER
body:
  I think I need a utility function for "call this function with this dictionary of arguments, but only pass the arguments which are inspected by the function".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634882112
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDg4MjExMg==
created_at: 2020-05-27T19:14:55Z
updated_at: 2020-05-27T19:14:55Z
author_association: OWNER
body:
  https://docs.python.org/3/library/inspect.html#introspecting-callables-with-the-signature-object

  > New in version 3.3.
  > 
  > The Signature object represents the call signature of a callable object and its return annotation. To retrieve a Signature object, use the [`signature()`](https://docs.python.org/3/library/inspect.html#inspect.signature "inspect.signature") function.
  > 
  > `inspect.``signature`(*callable*, ***, *follow_wrapped=True*)
  > 
  > Return a [`Signature`](https://docs.python.org/3/library/inspect.html#inspect.Signature "inspect.Signature") object for the given `callable`:
  > 
  > ```
  > >>> from inspect import signature
  > >>> def foo(a, *, b:int, **kwargs):
  > ...     pass
  > 
  > >>> sig = signature(foo)
  > 
  > >>> str(sig)
  > '(a, *, b:int, **kwargs)'
  > 
  > >>> str(sig.parameters['b'])
  > 'b:int'
  > 
  > >>> sig.parameters['b'].annotation
  > <class 'int'>
  > ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634882770
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDg4Mjc3MA==
created_at: 2020-05-27T19:16:19Z
updated_at: 2020-05-27T19:16:19Z
author_association: OWNER
body:
  ```
  In [1]: import inspect                                                                                                                                                             

  In [2]: def foo(view, sql, inspect): 
     ...:     pass 
     ...:                                                                                                                                                                            

  In [3]: inspect.signature(foo)                                                                                                                                                     
  Out[3]: <Signature (view, sql, inspect)>

  In [4]: inspect.signature(foo).parameters                                                                                                                                          
  Out[4]: 
  mappingproxy({'view': <Parameter "view">,
                'sql': <Parameter "sql">,
                'inspect': <Parameter "inspect">})

  In [5]: inspect.signature(foo).parameters.keys()                                                                                                                                   
  Out[5]: odict_keys(['view', 'sql', 'inspect'])

  In [6]: set(inspect.signature(foo).parameters.keys())                                                                                                                              
  Out[6]: {'inspect', 'sql', 'view'}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634888582
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDg4ODU4Mg==
created_at: 2020-05-27T19:23:23Z
updated_at: 2020-05-27T19:23:23Z
author_association: OWNER
body:
  Here's the function I just wrote for this:
  ```python
  def call_with_supported_arguments(fn, **kwargs):
      parameters = inspect.signature(fn).parameters.keys()
      call_with = []
      for parameter in parameters:
          if parameter not in kwargs:
              raise TypeError("{} requires parameters {}".format(fn, tuple(parameters)))
          call_with.append(kwargs[parameter])
      return fn(*call_with)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/771#issuecomment-634900776
issue_url: https://api.github.com/repos/simonw/datasette/issues/771
node_id: MDEyOklzc3VlQ29tbWVudDYzNDkwMDc3Ng==
created_at: 2020-05-27T19:44:25Z
updated_at: 2020-05-27T19:44:25Z
author_association: OWNER
body:
  This seems to work:
  ```diff
  diff --git a/tests/test_plugins.py b/tests/test_plugins.py
  index 8b6a6b4..e9a40aa 100644
  --- a/tests/test_plugins.py
  +++ b/tests/test_plugins.py
  @@ -7,7 +7,7 @@ from .fixtures import (
       TestClient as _TestClient,
   )  # noqa
   from datasette.app import Datasette
  -from datasette.plugins import get_plugins, DEFAULT_PLUGINS
  +from datasette.plugins import get_plugins, DEFAULT_PLUGINS, pm
   from datasette.utils import sqlite3
   import base64
   import json
  @@ -20,6 +20,21 @@ import pytest
   import urllib
   
   
  +def test_plugin_hooks_have_tests():
  +    "Every plugin hook should be referenced in this test module"
  +    hooks = [name for name in dir(pm.hook) if not name.startswith("_")]
  +    tests_in_this_module = [t for t in globals().keys() if t.startswith('test_')]
  +    untested = []
  +    for hook in hooks:
  +        ok = False
  +        for test in tests_in_this_module:
  +            if hook in test:
  +                ok = True
  +        if not ok:
  +            untested.append(hook)
  +    assert not untested, 'These plugin hooks are missing tests: {}'.format(untested)
  +
  +
   def test_plugins_dir_plugin_prepare_connection(app_client):
       response = app_client.get(
           "/fixtures.json?sql=select+convert_units(100%2C+'m'%2C+'ft')"
  ```
  Based on how the documentation unit tests work.

  Currently fails with:

      AssertionError: These plugin hooks are missing tests:
      ['prepare_jinja2_environment', 'publish_subcommand', 'register_facet_classes', 'register_output_renderer']

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/771#issuecomment-634915104
issue_url: https://api.github.com/repos/simonw/datasette/issues/771
node_id: MDEyOklzc3VlQ29tbWVudDYzNDkxNTEwNA==
created_at: 2020-05-27T20:14:32Z
updated_at: 2020-05-27T20:14:32Z
author_association: OWNER
body:
  ```
  $ pytest -k test_plugin_hooks_have_tests -vv
  ====================================== test session starts ======================================
  platform darwin -- Python 3.7.7, pytest-5.2.4, py-1.8.1, pluggy-0.13.1 -- /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/python
  cachedir: .pytest_cache
  rootdir: /Users/simon/Dropbox/Development/datasette, inifile: pytest.ini
  plugins: asyncio-0.10.0
  collected 486 items / 475 deselected / 11 selected                                              

  tests/test_plugins.py::test_plugin_hooks_have_tests[asgi_wrapper] XPASS                   [  9%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[extra_body_script] XPASS              [ 18%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[extra_css_urls] XPASS                 [ 27%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[extra_js_urls] XPASS                  [ 36%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[extra_template_vars] XPASS            [ 45%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[prepare_connection] XPASS             [ 54%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[prepare_jinja2_environment] XFAIL     [ 63%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[publish_subcommand] XFAIL             [ 72%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[register_facet_classes] XFAIL         [ 81%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[register_output_renderer] XFAIL       [ 90%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[render_cell] XPASS                    [100%]

  ========================= 475 deselected, 4 xfailed, 7 xpassed in 1.70s =========================

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634921101
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDkyMTEwMQ==
created_at: 2020-05-27T20:27:36Z
updated_at: 2020-05-27T20:27:36Z
author_association: OWNER
body:
  Actually passing the `request` object would be OK if I document it see https://github.com/simonw/datasette/issues/706

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/773#issuecomment-634940522
issue_url: https://api.github.com/repos/simonw/datasette/issues/773
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk0MDUyMg==
created_at: 2020-05-27T21:07:48Z
updated_at: 2020-05-27T21:07:48Z
author_association: OWNER
body:
  Remove this `xfail` decorator once they are all tested: https://github.com/simonw/datasette/blob/da87e963bff24e47878a5bc2025c8bfc63d4bc93/tests/test_plugins.py#L23-L28

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634943336
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk0MzMzNg==
created_at: 2020-05-27T21:13:04Z
updated_at: 2020-05-27T21:13:04Z
author_association: OWNER
body:
  Since I'm passing `request` I won't pass `scope` - if people want that they can access `request.scope`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634944832
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk0NDgzMg==
created_at: 2020-05-27T21:15:50Z
updated_at: 2020-05-27T21:16:28Z
author_association: OWNER
body:
  It bothers me that `query_name` here means the configured name of the canned query, but `view_name` means the name of the Datasette view class, NOT the name of an associated SQL view. That's in `table`.

  Can I come up with clearer names for these?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634946197
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk0NjE5Nw==
created_at: 2020-05-27T21:18:30Z
updated_at: 2020-05-27T21:18:30Z
author_association: OWNER
body:
  I'm going to break backwards compatibility directly here, without waiting for Datasette 1.0.

  The reason is that https://github.com/russss/datasette-geo hasn't been updated in 13 months so is already broken against current Datasette, and the other two plugins using this hook are owned by me so I can upgrade them myself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634946319
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk0NjMxOQ==
created_at: 2020-05-27T21:18:50Z
updated_at: 2020-05-27T21:18:50Z
author_association: OWNER
body:
  (I used GitHub code search to find code using this plugin hook: https://github.com/search?q=register_output_renderer&type=Code )

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/758#issuecomment-634950200
issue_url: https://api.github.com/repos/simonw/datasette/issues/758
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk1MDIwMA==
created_at: 2020-05-27T21:26:37Z
updated_at: 2020-05-27T21:26:37Z
author_association: OWNER
body:
  https://latest.datasette.io/.json currently returns:

  ```
  {
    "fixtures": {
      "name": "fixtures",
      "hash": "87b3f2c55dfb81ff1452dd306c2623fa5550b90982cfa32bad404c4d8bbedde2",
      "color": "87b3f2",
      "path": "/fixtures",
      "tables_and_views_truncated": [
  ```

  I published `fixtures.db` here like this:

      datasette publish cloudrun fixtures.db --service datasette-hash-urls --extra-options '--config hash_urls:1'

  https://datasette-hash-urls-j7hipcg4aq-uw.a.run.app/.json
  ```
  {
    "fixtures": {
      "name": "fixtures",
      "hash": "bda7daa889c23f9a8f06e46d7d280dd423c76275e9593c4c1cad7c53b19032fe",
      "color": "bda7da",
      "path": "/fixtures-bda7daa",
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/758#issuecomment-634951605
issue_url: https://api.github.com/repos/simonw/datasette/issues/758
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk1MTYwNQ==
created_at: 2020-05-27T21:29:19Z
updated_at: 2020-05-27T21:29:19Z
author_association: OWNER
body:
  But... https://datasette-hash-urls-j7hipcg4aq-uw.a.run.app/fixtures-bda7daa.json doesn't expose that hash:
  ```
  {
    "database": "fixtures",
    "size": 258048,
    "tables": [
      {
        "name": "123_starts_with_digits",
  ```
  Likewise https://datasette-hash-urls-j7hipcg4aq-uw.a.run.app/fixtures-bda7daa/complex_foreign_keys.json
  ```
  {
    "database": "fixtures",
    "table": "complex_foreign_keys",
    "is_view": false,
    "human_description_en": "",
    "rows": [
      [
        "1",
        "1",
        "2",
        "1"
      ]
    ],
  ```
  And https://datasette-hash-urls-j7hipcg4aq-uw.a.run.app/fixtures-bda7daa/complex_foreign_keys/1.json
  ```
  {
    "database": "fixtures",
    "table": "complex_foreign_keys",
    "rows": [
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634964294
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk2NDI5NA==
created_at: 2020-05-27T21:57:10Z
updated_at: 2020-05-27T21:57:10Z
author_association: OWNER
body:
  Right now a rendering callback returns the following:
  ```
  body - string or bytes, optional
      The response body, default empty
  content_type - string, optional
      The Content-Type header, default text/plain
  status_code - integer, optional
      The HTTP status code, default 200 
  ```
  I'm going to add an optional `headers` dictionary key, too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/706#issuecomment-634965148
issue_url: https://api.github.com/repos/simonw/datasette/issues/706
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk2NTE0OA==
created_at: 2020-05-27T21:59:07Z
updated_at: 2020-05-27T21:59:07Z
author_association: OWNER
body:
  This is the full current implementation of the request object:
  https://github.com/simonw/datasette/blob/9424687e9e94401438896116898a071702b09d40/datasette/utils/asgi.py#L15-L95

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/706#issuecomment-634973596
issue_url: https://api.github.com/repos/simonw/datasette/issues/706
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk3MzU5Ng==
created_at: 2020-05-27T22:19:02Z
updated_at: 2020-05-27T22:19:02Z
author_association: OWNER
body:
  New documentation can be seen here: https://github.com/simonw/datasette/blob/6d7cb02f00010d3cb4b4bac0460d41277652b80e/docs/internals.rst#request-object

  It's inspired me to reconsider the name of the `.raw_args` property, which isn't particularly clear.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/706#issuecomment-634974088
issue_url: https://api.github.com/repos/simonw/datasette/issues/706
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk3NDA4OA==
created_at: 2020-05-27T22:20:20Z
updated_at: 2020-05-27T22:20:20Z
author_association: OWNER
body:
  It looks like I inherited `.raw_args` from Sanic - I use it in a few places: https://github.com/search?q=user%3Asimonw+raw_args&type=Code

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/706#issuecomment-634974819
issue_url: https://api.github.com/repos/simonw/datasette/issues/706
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk3NDgxOQ==
created_at: 2020-05-27T22:22:20Z
updated_at: 2020-05-27T22:22:20Z
author_association: OWNER
body:
  What would a better name be?

  - `.simple_args`
  - `.kv_args`
  - `.pair_args`
  - `.dict_args`
  - `.args_dict`

  I dislike the last two the least.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/706#issuecomment-634975252
issue_url: https://api.github.com/repos/simonw/datasette/issues/706
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk3NTI1Mg==
created_at: 2020-05-27T22:23:26Z
updated_at: 2020-05-27T22:30:05Z
author_association: OWNER
body:
  I'm going to leave `.raw_args` in for the moment but deliberately not document it. I'll hope to phase it out entirely at a later date.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-634978388
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk3ODM4OA==
created_at: 2020-05-27T22:32:03Z
updated_at: 2020-05-27T22:32:03Z
author_association: OWNER
body:
  Request object is now documented: https://datasette.readthedocs.io/en/latest/internals.html#request-object

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/770#issuecomment-634980179
issue_url: https://api.github.com/repos/simonw/datasette/issues/770
node_id: MDEyOklzc3VlQ29tbWVudDYzNDk4MDE3OQ==
created_at: 2020-05-27T22:37:19Z
updated_at: 2020-05-27T22:37:19Z
author_association: OWNER
body:
  Can I come up with a better name than `should_suggest`?

  It's a check that sees if the current query is supported by the renderer plugin. Some options:

  - `can_render`
  - `supports_query`
  - `is_supported`

  I like `can_render`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/775#issuecomment-635023389
issue_url: https://api.github.com/repos/simonw/datasette/issues/775
node_id: MDEyOklzc3VlQ29tbWVudDYzNTAyMzM4OQ==
created_at: 2020-05-28T00:47:32Z
updated_at: 2020-05-28T00:47:32Z
author_association: OWNER
body:
  These:

  https://github.com/simonw/datasette/blob/4b96857f170e329a73186e703cc0d9ca4e8719cc/tests/fixtures.py#L337-L506

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/581#issuecomment-635055346
issue_url: https://api.github.com/repos/simonw/datasette/issues/581
node_id: MDEyOklzc3VlQ29tbWVudDYzNTA1NTM0Ng==
created_at: 2020-05-28T02:24:14Z
updated_at: 2020-05-28T02:24:14Z
author_association: OWNER
body:
  Updated documentation is here: https://datasette.readthedocs.io/en/latest/plugins.html#register-output-renderer-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/776#issuecomment-635056357
issue_url: https://api.github.com/repos/simonw/datasette/issues/776
node_id: MDEyOklzc3VlQ29tbWVudDYzNTA1NjM1Nw==
created_at: 2020-05-28T02:27:26Z
updated_at: 2020-05-28T02:27:26Z
author_association: OWNER
body:
  Example code from elsewhere:
  https://github.com/simonw/datasette/blob/52c4387c7d37c867104e3728cc1f4c4d1e100642/datasette/views/base.py#L416-L420

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/773#issuecomment-635069987
issue_url: https://api.github.com/repos/simonw/datasette/issues/773
node_id: MDEyOklzc3VlQ29tbWVudDYzNTA2OTk4Nw==
created_at: 2020-05-28T03:13:27Z
updated_at: 2020-05-28T03:13:27Z
author_association: OWNER
body:
  `register_output_renderer` test added in https://github.com/simonw/datasette/commit/52c4387c7d37c867104e3728cc1f4c4d1e100642#diff-56f7d7b4778bac73b6b655c02c8467aaR336

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/758#issuecomment-635102675
issue_url: https://api.github.com/repos/simonw/datasette/issues/758
node_id: MDEyOklzc3VlQ29tbWVudDYzNTEwMjY3NQ==
created_at: 2020-05-28T05:04:07Z
updated_at: 2020-05-28T05:04:07Z
author_association: OWNER
body:
  @clausjuhl do you have any thoughts on what would be most useful for you in these JSON responses? The full `/databasename-hash` path, just the 7 character hash, or something else?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/706#issuecomment-635131217
issue_url: https://api.github.com/repos/simonw/datasette/issues/706
node_id: MDEyOklzc3VlQ29tbWVudDYzNTEzMTIxNw==
created_at: 2020-05-28T06:20:44Z
updated_at: 2020-05-28T06:20:44Z
author_association: OWNER
body:
  Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#canned-queries-default-fragment

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/758#issuecomment-635195322
issue_url: https://api.github.com/repos/simonw/datasette/issues/758
node_id: MDEyOklzc3VlQ29tbWVudDYzNTE5NTMyMg==
created_at: 2020-05-28T08:23:27Z
updated_at: 2020-05-28T08:23:27Z
author_association: NONE
body:
  @simonw I would prefer just the 7 character hash. No need to make the urls any longer than they need to be :)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/780#issuecomment-635413437
issue_url: https://api.github.com/repos/simonw/datasette/issues/780
node_id: MDEyOklzc3VlQ29tbWVudDYzNTQxMzQzNw==
created_at: 2020-05-28T15:15:03Z
updated_at: 2020-05-28T15:15:03Z
author_association: OWNER
body:
  Also: I think I should add a `query=` parameter to help lookup metadata about a specific named canned query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/781#issuecomment-635468994
issue_url: https://api.github.com/repos/simonw/datasette/issues/781
node_id: MDEyOklzc3VlQ29tbWVudDYzNTQ2ODk5NA==
created_at: 2020-05-28T16:55:35Z
updated_at: 2020-05-28T16:55:35Z
author_association: OWNER
body:
  I think the right way to fix this is to modify the `scope["scheme"]` ASGI key before the Request object is constructed - otherwise that Request class will have to gain knowledge of Datasette's configuration mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/781#issuecomment-635471007
issue_url: https://api.github.com/repos/simonw/datasette/issues/781
node_id: MDEyOklzc3VlQ29tbWVudDYzNTQ3MTAwNw==
created_at: 2020-05-28T16:58:47Z
updated_at: 2020-05-28T16:58:47Z
author_association: OWNER
body:
  I'm inclined to do this at the earliest possible moment. I think that's probably in the `DatasetteRouter` class, which should see every `scope` first and be able to apply that setting to it. It already has some special case logic to deal with the `base_url` setting here:

  https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/datasette/app.py#L779-L789

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/780#issuecomment-635480948
issue_url: https://api.github.com/repos/simonw/datasette/issues/780
node_id: MDEyOklzc3VlQ29tbWVudDYzNTQ4MDk0OA==
created_at: 2020-05-28T17:12:45Z
updated_at: 2020-05-28T17:12:45Z
author_association: OWNER
body:
  This is a good opportunity to reconsider the design of this function and see if I'm happy with it as-is or if there are some improvements I want to make before adding it to the documented API.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/781#issuecomment-635494730
issue_url: https://api.github.com/repos/simonw/datasette/issues/781
node_id: MDEyOklzc3VlQ29tbWVudDYzNTQ5NDczMA==
created_at: 2020-05-28T17:39:54Z
updated_at: 2020-05-28T17:39:54Z
author_association: OWNER
body:
  https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fwww.niche-museums.com%2Fbrowse%2Ffeed.atom validates now!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/777#issuecomment-635513983
issue_url: https://api.github.com/repos/simonw/datasette/issues/777
node_id: MDEyOklzc3VlQ29tbWVudDYzNTUxMzk4Mw==
created_at: 2020-05-28T18:16:49Z
updated_at: 2020-05-28T18:16:49Z
author_association: NONE
body:
   think, because the given URL of the CSS file doesn't have any complete parameters after query 
  Try to complete the parameter 
  `<link rel="stylesheet" href="-/static/app.css?(whatever you want that shows a css file)">`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/774#issuecomment-635530876
issue_url: https://api.github.com/repos/simonw/datasette/issues/774
node_id: MDEyOklzc3VlQ29tbWVudDYzNTUzMDg3Ng==
created_at: 2020-05-28T18:50:18Z
updated_at: 2020-05-28T18:50:18Z
author_association: OWNER
body:
  How about moving this functionality to the request object itself?
  ```python
  q = request["q"]  # Raises KeyError if missing, otherwise returns first
  q = request.get("q", "default") # Returns first, or optional default or None
  facets = request.getlist("_facet")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/508#issuecomment-635532216
issue_url: https://api.github.com/repos/simonw/datasette/issues/508
node_id: MDEyOklzc3VlQ29tbWVudDYzNTUzMjIxNg==
created_at: 2020-05-28T18:53:02Z
updated_at: 2020-05-28T18:53:02Z
author_association: OWNER
body:
  I fixed this a while ago in #702: https://datasette.readthedocs.io/en/stable/metadata.html#setting-a-default-sort-order

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/351#issuecomment-635533807
issue_url: https://api.github.com/repos/simonw/datasette/issues/351
node_id: MDEyOklzc3VlQ29tbWVudDYzNTUzMzgwNw==
created_at: 2020-05-28T18:56:15Z
updated_at: 2020-05-28T18:56:15Z
author_association: OWNER
body:
  I'm not going to do this. I have a good enough solution now pasting the rendered HTML from the release notes into https://euangoddard.github.io/clipboard2markdown/ to get the Markdown.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/774#issuecomment-635702201
issue_url: https://api.github.com/repos/simonw/datasette/issues/774
node_id: MDEyOklzc3VlQ29tbWVudDYzNTcwMjIwMQ==
created_at: 2020-05-29T01:20:34Z
updated_at: 2020-05-29T01:20:34Z
author_association: OWNER
body:
  Or change `request.args` to behave more like `request.raw_args` - mainly to return a single value when you look things up by key.

  It's currently defined like this: https://github.com/simonw/datasette/blob/3c1a60589e14849344acd8aa6da0a60b40fbfc60/datasette/utils/__init__.py#L756-L766

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/774#issuecomment-636227927
issue_url: https://api.github.com/repos/simonw/datasette/issues/774
node_id: MDEyOklzc3VlQ29tbWVudDYzNjIyNzkyNw==
created_at: 2020-05-29T22:58:32Z
updated_at: 2020-05-29T22:58:32Z
author_association: OWNER
body:
  I think I want `request.args["key"]` to return the FIRST item for that key or raise a `KeyError` if none are found.

  Right now it returns the full list.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/774#issuecomment-636228656
issue_url: https://api.github.com/repos/simonw/datasette/issues/774
node_id: MDEyOklzc3VlQ29tbWVudDYzNjIyODY1Ng==
created_at: 2020-05-29T23:01:22Z
updated_at: 2020-05-29T23:01:22Z
author_association: OWNER
body:
  As far as I can tell the only code I've ever written that would break if I made this change is in `russian-ira-facebook-ads`:

  https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/e7106710abdd7bdcae035bedd8bdaba75ae56a12/plugins/target.py#L22

  https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/b8a22348c6b315ab94ddba69e8117dfdfd9573dc/plugins/regexp.py#L17

  That doesn't work against latest Datasette anyway, so I think I can safely make this change.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/774#issuecomment-636229764
issue_url: https://api.github.com/repos/simonw/datasette/issues/774
node_id: MDEyOklzc3VlQ29tbWVudDYzNjIyOTc2NA==
created_at: 2020-05-29T23:05:48Z
updated_at: 2020-05-29T23:05:48Z
author_association: OWNER
body:
  I'm going to rebuild `RequestParameters` to no longer subclass `dict`. I'll keep the following methods:

  - `__contains__()`
  - `__getitem__()` (with the new behaviour)
  - `keys()`
  - iterating iterates keys
  - `__len__`
  - `get`
  - `getlist`

  It won't support writing, so it will effectively be immutable after you have constructed it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/774#issuecomment-636234067
issue_url: https://api.github.com/repos/simonw/datasette/issues/774
node_id: MDEyOklzc3VlQ29tbWVudDYzNjIzNDA2Nw==
created_at: 2020-05-29T23:24:34Z
updated_at: 2020-05-29T23:24:34Z
author_association: OWNER
body:
  Updated documentation for `RequestParameters`: https://datasette.readthedocs.io/en/latest/internals.html#the-requestparameters-class

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/774#issuecomment-636234759
issue_url: https://api.github.com/repos/simonw/datasette/issues/774
node_id: MDEyOklzc3VlQ29tbWVudDYzNjIzNDc1OQ==
created_at: 2020-05-29T23:27:35Z
updated_at: 2020-05-29T23:27:35Z
author_association: OWNER
body:
  Oh dear... it looks like `.raw_args` is used in my TIL script, which has been copied by a few people!

  https://github.com/search?q=request+raw_args+datasette&type=Code

  I'll fix it in mine and file pull requests against other pieces before this code gets released.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-636322089
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDYzNjMyMjA4OQ==
created_at: 2020-05-30T12:08:43Z
updated_at: 2020-05-30T12:08:43Z
author_association: OWNER
body:
  Idea: use a chained API to define a complex transition and then execute it all at once. For example:
  ```python
  db["mytable"].transform().rename("col1", "col_1") \
    .change_type("col1", float) \
    .execute()
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/576#issuecomment-636329095
issue_url: https://api.github.com/repos/simonw/datasette/issues/576
node_id: MDEyOklzc3VlQ29tbWVudDYzNjMyOTA5NQ==
created_at: 2020-05-30T13:11:39Z
updated_at: 2020-05-30T13:25:10Z
author_association: OWNER
body:
  I need to document the `.databases` property - both how to get access to specific databases and how to access the first/only database.

  Idea: `datasette.get_database(name)` method (consistent with existing `.add_database()` and `.remove_database()` methods) - but `name` parameter is optional, returns first database if you omit that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/519#issuecomment-636329468
issue_url: https://api.github.com/repos/simonw/datasette/issues/519
node_id: MDEyOklzc3VlQ29tbWVudDYzNjMyOTQ2OA==
created_at: 2020-05-30T13:14:52Z
updated_at: 2020-05-30T13:21:03Z
author_association: OWNER
body:
  I've made a lot of progress towards this recently:

  - Started internals documentation #576
  - Improved design of the request object #706
  - Redesigned register_output_renderer plugin hook #581

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/519#issuecomment-636330023
issue_url: https://api.github.com/repos/simonw/datasette/issues/519
node_id: MDEyOklzc3VlQ29tbWVudDYzNjMzMDAyMw==
created_at: 2020-05-30T13:19:24Z
updated_at: 2020-05-30T13:19:24Z
author_association: OWNER
body:
  Goals for Datasette 1.0:

  - Generally signify confidence in the quality/stability of Datasette. I think I'm there already.
  - Plugin authors can have confidence that their plugins will work for the whole 1.x release cycle
  - Developers building against Datasette JSON APIs should have confidence in 1.x compatibility
  - Template authors and CSS theme authors should have that confidence too

  I think I'm very nearly there for the plugin API. The harder ones are JSON APIs and template authors.

  For JSON APIs: The default JSON just isn't right. I find myself using `?_shape=array` for almost everything I build.

  The template part is harder. I think I need to fully document the template variables for every view - and add protective unit tests that match that documentation.

  The CSS theme part is so hard it may not be possible. How do you guarantee stable HTML that won't break with custom CSS when you'll be adding new features during the 1.x release run? The pattern portfolio from #151 should hopefully help a lot there, but I still don't have a complete idea of what this entails or how feasible it is.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/576#issuecomment-636330843
issue_url: https://api.github.com/repos/simonw/datasette/issues/576
node_id: MDEyOklzc3VlQ29tbWVudDYzNjMzMDg0Mw==
created_at: 2020-05-30T13:26:38Z
updated_at: 2020-05-30T13:26:38Z
author_association: OWNER
body:
  There's a bunch of methods on that class which I could add a `_` prefix to and leave undocumented. `datasette.register_renderers()` should be `datasette._register_renderers()` for example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/576#issuecomment-636332083
issue_url: https://api.github.com/repos/simonw/datasette/issues/576
node_id: MDEyOklzc3VlQ29tbWVudDYzNjMzMjA4Mw==
created_at: 2020-05-30T13:36:55Z
updated_at: 2020-05-30T13:36:55Z
author_association: OWNER
body:
  Here's the current `Datasette` class as introspected using the webserver run by `pydoc -p 8000`:

  <img width="956" alt="Pydoc__module_datasette_app" src="https://user-images.githubusercontent.com/9599/83329718-f0835680-a23f-11ea-8cbf-1a1e5dafb7ec.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/576#issuecomment-636332183
issue_url: https://api.github.com/repos/simonw/datasette/issues/576
node_id: MDEyOklzc3VlQ29tbWVudDYzNjMzMjE4Mw==
created_at: 2020-05-30T13:37:51Z
updated_at: 2020-05-30T13:38:35Z
author_association: OWNER
body:
  **\_\_init\_\_**(self, files, immutables=None, cache_headers=True, cors=False, inspect_data=None, metadata=None, sqlite_extensions=None, template_dir=None, plugins_dir=None, static_mounts=None, memory=False, config=None, version_note=None, config_dir=None)

  `Initialize self.  See help(type(self)) for accurate signature.`

  **absolute_url**(self, request, path)

  **add_database**(self, name, db)

  **app**(self)

  `Returns an ASGI app function that serves the whole of [Datasette](http://localhost:8066/datasette.app.html#Datasette)`

  **app_css_hash**(self)

  **config**(self, key)

  **config_dict**(self)

  **connected_databases**(self)

  **execute**(self, db_name, sql, params=None, truncate=False, custom_time_limit=None, page_size=None, log_sql_errors=True)

  **expand_foreign_keys**(self, database, table, column, values)

  `Returns dict mapping (column, value) -> label`

  **get_canned_queries**(self, database_name)

  **get_canned_query**(self, database_name, query_name)

  **metadata**(self, key=None, database=None, table=None, fallback=True)

  `Looks up metadata, cascading backwards from specified level.\
  Returns None if metadata value is not found.`

  **plugin_config**(self, plugin_name, database=None, table=None, fallback=True)

  `Return config for plugin, falling back from specified database/table`

  **plugins**(self, show_all=False)

  **prepare_connection**(self, conn, database)

  **register_custom_units**(self)

  `Register any custom units defined in the metadata.json with Pint`

  **register_renderers**(self)

  `Register output renderers which output data in custom formats.`

  **remove_database**(self, name)

  **render_template**(self, templates, context=None, request=None, view_name=None)

  **table_metadata**(self, database, table)

  `Fetch table-specific metadata.`

  **threads**(self)

  **update_with_inherited_metadata**(self, metadata)

  **versions**(self)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/576#issuecomment-636340618
issue_url: https://api.github.com/repos/simonw/datasette/issues/576
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM0MDYxOA==
created_at: 2020-05-30T14:46:04Z
updated_at: 2020-05-30T18:41:41Z
author_association: OWNER
body:
  I should also think about the class properties (as opposed to methods) that are setup in the Datasette constructor. Many of these should be private, some should be documented.

  - cache_headers
  - cors
  - databases
  - executor
  - files
  - immutables
  - inspect_data
  - jinja_env
  - max_returned_rows
  - page_size
  - plugins_dir
  - renderers
  - sql_time_limit_ms
  - sqlite_extensions
  - sqlite_functions
  - static_mounts
  - template_dir
  - version_note

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/498#issuecomment-636360574
issue_url: https://api.github.com/repos/simonw/datasette/issues/498
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM2MDU3NA==
created_at: 2020-05-30T17:26:02Z
updated_at: 2020-05-30T17:26:02Z
author_association: OWNER
body:
  I released a plugin that implements an early version of this a while ago: https://github.com/simonw/datasette-search-all

  I'm still thinking about how a plugin / separate tool could work that builds a single FTS index over the content from multiple tables such that you can run relevance-calculated queries against those multiple tables.

  Since that's going to be a plugin / separate tool too, I'm closing this issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/37#issuecomment-636360861
issue_url: https://api.github.com/repos/simonw/datasette/issues/37
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM2MDg2MQ==
created_at: 2020-05-30T17:29:20Z
updated_at: 2020-05-30T17:29:20Z
author_association: OWNER
body:
  I'm not going to do this: 2.5 years later I have yet to run into anything that makes me think that JSON serialization performance is worth any extra work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-636369978
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM2OTk3OA==
created_at: 2020-05-30T18:50:31Z
updated_at: 2020-05-30T18:50:31Z
author_association: OWNER
body:
  Here's the default JSON at the moment: https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2

  ```json
  {
    "database": "fixtures",
    "table": "compound_three_primary_keys",
    "is_view": false,
    "human_description_en": "",
    "rows": [
      [
        "a",
        "a",
        "a",
        "a-a-a"
      ],
      [
        "a",
        "a",
        "b",
        "a-a-b"
      ]
    ],
    "truncated": false,
    "filtered_table_rows_count": 1001,
    "expanded_columns": [],
    "expandable_columns": [],
    "columns": [
      "pk1",
      "pk2",
      "pk3",
      "content"
    ],
    "primary_keys": [
      "pk1",
      "pk2",
      "pk3"
    ],
    "units": {},
    "query": {
      "sql": "select pk1, pk2, pk3, content from compound_three_primary_keys order by pk1, pk2, pk3 limit 3",
      "params": {}
    },
    "facet_results": {},
    "suggested_facets": [
      {
        "name": "pk1",
        "toggle_url": "http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_facet=pk1"
      },
      {
        "name": "pk2",
        "toggle_url": "http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_facet=pk2"
      },
      {
        "name": "pk3",
        "toggle_url": "http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_facet=pk3"
      }
    ],
    "next": "a,a,b",
    "next_url": "http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_next=a%2Ca%2Cb",
    "query_ms": 17.56119728088379,
    "source": "tests/fixtures.py",
    "source_url": "https://github.com/simonw/datasette/blob/master/tests/fixtures.py",
    "license": "Apache License 2.0",
    "license_url": "https://github.com/simonw/datasette/blob/master/LICENSE"
  }
  ```
  There's a lot of stuff in there. This increases the risk that future minor changes might break existing API consumers.

  It returns rows as a list of lists of values, and expects you to correlate these with the list of columns. I originally designed it like this because I thought this was a more efficient representation than repeating the column names in a dictionary for every row. With hindsight this was a bad optimization - I _always_ use `?shape=array` because it's more convenient, and gzip encoding of the response means there's no bandwidth saving. Users who want that efficiency should request it using a custom `?_shape=`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-636370064
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM3MDA2NA==
created_at: 2020-05-30T18:51:19Z
updated_at: 2020-05-30T18:51:19Z
author_association: OWNER
body:
  https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_shape=array returns this:
  ```json
  [
    {
      "pk1": "a",
      "pk2": "a",
      "pk3": "a",
      "content": "a-a-a"
    },
    {
      "pk1": "a",
      "pk2": "a",
      "pk3": "b",
      "content": "a-a-b"
    }
  ]
  ```
  There's one big problem with this format: it doesn't provide any space for pagination information.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636376209
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM3NjIwOQ==
created_at: 2020-05-30T19:53:28Z
updated_at: 2020-05-30T20:09:10Z
author_association: OWNER
body:
  I think there are two hooks here:

  `actor_from_request(datasette, request)` - returns `None` or a dictionary.

  - `datasette` is a Datasette instance - useful for things like reading plugin configuration or executing queries
  - `request` is a [Request object](https://datasette.readthedocs.io/en/latest/internals.html#request-object) - which means ASGI scope can be accessed as `request.scope`

  A non-None value means the request is authenticated in some way. The shape of that dictionary is entirely undefined.

  The second hook is for checking permissions. It can look something like this:

  `permission_allowed(actor, action, resource_type, resource_identifier)`

  - `actor` = the dictionary that was returned by `actor_from_scope`
  - `action` = a string representing the action to be performed, e.g. `edit-schema`
  - `resource_type` = a string representing the type of resource being acted on, e.g. `table`
  - `resource_identifier` = a string (or maybe tuple?) representing the specific resource, e.g. the table name

  I don't know if Datasette should provide default implementations of these hooks. It may be that leaving them completely up to plugins is the way to go.

  I think I need to prototype this quickly to start feeling for how well it might work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636376893
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM3Njg5Mw==
created_at: 2020-05-30T19:57:54Z
updated_at: 2020-05-30T20:09:05Z
author_association: OWNER
body:
  `auth_from_scope(datasette, scope)` needs to be able to return an awaitable which is then awaited - so it can execute database queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636377235
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM3NzIzNQ==
created_at: 2020-05-30T20:00:42Z
updated_at: 2020-05-30T20:01:35Z
author_association: OWNER
body:
  I'm changing `auth` to `actor` and updating the above [design comment](https://github.com/simonw/datasette/issues/699#issuecomment-636376209).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636377755
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM3Nzc1NQ==
created_at: 2020-05-30T20:04:23Z
updated_at: 2020-05-30T20:04:23Z
author_association: OWNER
body:
  My usage of the term `subject` here to mean "the thing I am checking I have permission to interact with, e.g. a database table" may be misleading. https://stackoverflow.com/questions/4989063/what-is-the-meaning-and-difference-between-subject-user-and-principal for example shows that JAAS (Java Authentication and Authorization Service) defines subject as "The purpose of the Subject is to represent the authenticated user".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636378121
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM3ODEyMQ==
created_at: 2020-05-30T20:06:47Z
updated_at: 2020-05-30T20:06:47Z
author_association: OWNER
body:
  In AWS IAM world the following terminology is used: https://aws.amazon.com/iam/features/manage-permissions/

  > Permissions are granted to IAM **entities** (users, groups, and roles) [...]
  > 
  > To assign permissions to a user, group, role, or resource, you create a policy that lets you specify:
  > 
  > * **Actions** – Which AWS service actions you allow. For example, you might allow a user to call the Amazon S3 ListBucket action. Any actions that you don't explicitly allow are denied.
  > * **Resources** – Which AWS resources you allow the action on. For example, what Amazon S3 buckets will you allow the user to perform the ListBucket action on? Users cannot access any resources that you do not explicitly grant permissions to.
  > * **Effect** – Whether to allow or deny access. Because access is denied by default, you typically write policies where the effect is to allow.
  > * **Conditions** – Which conditions must be present for the policy to take effect. For example, you might allow access only to the specific S3 buckets if the user is connecting from a specific IP range or has used multi-factor authentication at login.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636378228
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM3ODIyOA==
created_at: 2020-05-30T20:07:25Z
updated_at: 2020-05-30T20:07:25Z
author_association: OWNER
body:
  I like "actor" better than "entity" to mean "the user or API key that is authenticated for this request".

  I'm going to use "resource" instead of "subject" - updating the design comment again.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636379067
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM3OTA2Nw==
created_at: 2020-05-30T20:12:47Z
updated_at: 2020-05-30T20:40:42Z
author_association: OWNER
body:
  I could bake some permission checks into default Datasette, which are all treated as allow by default but can then be locked down by plugins. Maybe the following:

      permission_allowed(request.actor, "execute-sql", "database", "name-of-database")

  Checks that current user can execute arbitrary SQL queries against a specific database (or use the `?_where=` feature). Equivalent to current [allow_sql](https://datasette.readthedocs.io/en/0.43/config.html#allow-sql) setting.

      permission_allowed(request.actor, "download-database", "database", "name-of-database")

  Can the user download the database file? Like [allow_download](https://datasette.readthedocs.io/en/0.43/config.html#allow-download).

  Maybe one for [allow_csv_stream](https://datasette.readthedocs.io/en/0.43/config.html#allow-csv-stream) too.

  Having a permission check (defaulting to True) on every single "view" would be useful:

  - view_index
  - view_database
  - view_table
  - view_row
  - view_query
  - view_special (for `/-/versions` and so on)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636381732
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM4MTczMg==
created_at: 2020-05-30T20:32:11Z
updated_at: 2020-05-30T20:39:11Z
author_association: OWNER
body:
  I started sketching this out in the [authentication](https://github.com/simonw/datasette/tree/authentication) branch. Here's the documentation so far: https://github.com/simonw/datasette/blob/8871c20/docs/plugins.rst#actor_from_requestdatasette-request

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636388288
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM4ODI4OA==
created_at: 2020-05-30T21:34:50Z
updated_at: 2020-05-30T21:34:50Z
author_association: OWNER
body:
  Debugging permissions is going to be important. Optional tooling that supports the following would be useful:

  - Log every check to `permission_allowed` to the console - optionally with tracebacks showing where in the code the check was made
  - Log every check to the https://latest.datasette.io/?_trace=1 output
  - A tool that shows you exactly what permissions the current authenticated user/entity has
  - A tool showing all available permissions

  That last one is tricky if permissions are just strings that might be passed to `permission_allowed` - so maybe there needs to be a plugin hook that lets plugins register their permissions, such that they can be introspected later on? A `register_permission_actions()` hook that returns a list of permission action strings (or objects of some sort) perhaps.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636391331
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM5MTMzMQ==
created_at: 2020-05-30T22:08:21Z
updated_at: 2020-05-30T22:08:21Z
author_association: OWNER
body:
  I'm going to add an awaitable utility method to the Datasette class for checking permissions:

      await datasette.permission_allowed(actor, action, resource_type, resource_identifier)

  The second two arguments will be optional.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636392850
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM5Mjg1MA==
created_at: 2020-05-30T22:25:19Z
updated_at: 2020-05-30T22:25:19Z
author_association: OWNER
body:
  The branch is now usable! Next step: write some experimental plugins that exercise some real authentication use-cases with it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636393204
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM5MzIwNA==
created_at: 2020-05-30T22:29:44Z
updated_at: 2020-05-30T22:30:15Z
author_association: OWNER
body:
  Robust testing of permissions is really important. I should think about utilities I may be able to add to Datasette's unit testing tools that make it as easy as possible to confirm which permission checks were carried out on a specific HTTP request.

  That way I can set a good example that any Datasette plugin which makes permission checks can follow.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636395263
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjM5NTI2Mw==
created_at: 2020-05-30T22:54:09Z
updated_at: 2020-05-30T22:54:09Z
author_association: OWNER
body:
  Idea: add a `/-/actor.json` special page which JSON dumps out the current `request.scope["actor"]` - so you can easily test how your request has been authenticated.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636494374
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjQ5NDM3NA==
created_at: 2020-05-31T16:29:48Z
updated_at: 2020-05-31T16:29:48Z
author_association: OWNER
body:
  If Datasette were to support authentication out-of-the-box, without plugins (which makes more sense with writable canned queries, #698) what would that look like?

  Some options:

  - Jupyter notebook style: output a magic URL on the console with a one-time token to authenticate the user as an "admin"
  - Really simple password authentication - via an environment variable perhaps?
  - SQL based authentication: I was going to do this as a plugin, but maybe it should be default? A way of configuring a SQL query which can be used to authenticate a user based on their username and password.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636495005
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjQ5NTAwNQ==
created_at: 2020-05-31T16:35:10Z
updated_at: 2020-05-31T16:35:26Z
author_association: OWNER
body:
  I think I want to keep full username/password authentication against a database table as a plugin. I'll experiment with Jupyter-style URLs as a starting point.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636498770
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjQ5ODc3MA==
created_at: 2020-05-31T17:03:38Z
updated_at: 2020-05-31T17:03:38Z
author_association: OWNER
body:
  I'm going to draw the line here: default Datasette supports authentication but only for a single user account ("admin"). Plugins can then add support for multiple user accounts, social auth, SSO etc.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636498913
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjQ5ODkxMw==
created_at: 2020-05-31T17:04:50Z
updated_at: 2020-05-31T17:06:40Z
author_association: OWNER
body:
  This also means some writable canned queries can allow writes from unauthenticated users (for stuff like feedback forms), while others can require an authenticated user - all with core Datasette without any plugins needed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636499075
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjQ5OTA3NQ==
created_at: 2020-05-31T17:06:09Z
updated_at: 2020-05-31T17:06:09Z
author_association: OWNER
body:
  I believe that this plugin hook design is flexible enough that role-based permissions could be built on top of it as a separate plugin.

  Would be good to check that with a proof of concept though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636510303
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUxMDMwMw==
created_at: 2020-05-31T18:35:17Z
updated_at: 2020-05-31T18:35:17Z
author_association: OWNER
body:
  Keeping the structure of the actor dictionary completely undefined doesn't make sense if Datasette is going to ship with a default authentication mechanism for admin users.

  I'm going to define a small set of required keys for the actor dictionary, and enforce them in code.

  But which keys? I feel I need a unique key representing the identity of the actor, plus a key that can be displayed in the "You are logged in as X" navigation. Maybe these are the same key?

  So the single required key could be `id`. Problem is: is that a string or an integer? Some use-cases may call for an integer, which matches to how SQLite auto incrementing primary keys work. `admin` is a string.

  Maybe `id` is required, `name` is optional - but if `name` is present then the "You are logged in as..." uses that in preference to `id`. `id` has to be a string, and if you want to store integer IDs in your database you need to remember to convert them to a string in your `actor_from_request` implementation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636510398
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUxMDM5OA==
created_at: 2020-05-31T18:35:57Z
updated_at: 2020-05-31T18:36:05Z
author_association: OWNER
body:
  Again I will use exploratory prototyping to inform a decision on the minimum subset design for the `actor` dictionary.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636510647
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUxMDY0Nw==
created_at: 2020-05-31T18:37:39Z
updated_at: 2020-05-31T18:37:39Z
author_association: OWNER
body:
  Maybe the default single account should be called something other than `admin`? The problem with `admin` is that it sounds like more of a role - in larger installations one can expect multiple admins. `root` may be better since there's clearly only one root account. Bit of a technical term though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636510761
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUxMDc2MQ==
created_at: 2020-05-31T18:38:30Z
updated_at: 2020-05-31T18:38:30Z
author_association: OWNER
body:
  I quite like `root` - it supports the idea that best practice is to NOT do things as the root account, but to use a plugin to set up separate accounts for different purposes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/784#issuecomment-636510838
issue_url: https://api.github.com/repos/simonw/datasette/issues/784
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUxMDgzOA==
created_at: 2020-05-31T18:39:08Z
updated_at: 2020-05-31T18:39:08Z
author_association: OWNER
body:
  I'm calling this the `root` account now, for reasons discussed in these two comments: https://github.com/simonw/datasette/issues/699#issuecomment-636510647

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/784#issuecomment-636514974
issue_url: https://api.github.com/repos/simonw/datasette/issues/784
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUxNDk3NA==
created_at: 2020-05-31T19:12:48Z
updated_at: 2020-05-31T19:12:48Z
author_association: OWNER
body:
  For the first version of this I'm not going to use passwords at all. I'll implement this:

      $ datasette fixtures.db --root

  The `--root` option will cause Datasette to output a URL with a one-time-use token in it which, when clicked, will authenticate the user as the root account (by setting a signed cookie).

  Signed cookie means Datasette needs a secrets recipe. I'll open a new issue for that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/785#issuecomment-636515599
issue_url: https://api.github.com/repos/simonw/datasette/issues/785
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUxNTU5OQ==
created_at: 2020-05-31T19:17:43Z
updated_at: 2020-05-31T19:17:43Z
author_association: OWNER
body:
  I previously solved this for the `datasette-auth-existing-cookies` plugin as described in this issue: https://github.com/simonw/datasette-auth-existing-cookies/issues/1

  > Concrete plan: you have to pass a secret to the class constructor. The Datasette plugin (the code in `__init__.py`) uses the following in order of preference (first things are most preferred):
  > 
  > - A plugin configuration option called `cookie_secret` - which can be protected by this mechanism: https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values
  > - A JSON configuration file in the `user_state_dir` file, if it exists
  > - If that does not exist, a secret is generated and written to that JSON file
  > 
  > I originally planned to have separate support for an environment variable, but the existence of the [secret configuration values](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values) mechanism means this is already handled.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/785#issuecomment-636515671
issue_url: https://api.github.com/repos/simonw/datasette/issues/785
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUxNTY3MQ==
created_at: 2020-05-31T19:18:18Z
updated_at: 2020-05-31T19:18:18Z
author_association: OWNER
body:
  That `user_state_dir` solution may have been more trouble than it was worth though - I seem to remember it causing issues on some hosting providers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/785#issuecomment-636515763
issue_url: https://api.github.com/repos/simonw/datasette/issues/785
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUxNTc2Mw==
created_at: 2020-05-31T19:19:03Z
updated_at: 2020-05-31T19:19:13Z
author_association: OWNER
body:
  Maybe Datasette should have a `--secrets=path/to/secrets.json` command-line option for storing these?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/785#issuecomment-636537679
issue_url: https://api.github.com/repos/simonw/datasette/issues/785
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUzNzY3OQ==
created_at: 2020-05-31T22:09:23Z
updated_at: 2020-05-31T22:09:23Z
author_association: OWNER
body:
  I'm going to use https://github.com/pallets/itsdangerous for this.

  Annoyingly they're very close to release v2.0 which adds support for key rotation... but it's not quite out of pre-release yet. I'll go with 1.1.0 for the moment and upgrade to 2.0 as soon as that is out.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/785#issuecomment-636537921
issue_url: https://api.github.com/repos/simonw/datasette/issues/785
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUzNzkyMQ==
created_at: 2020-05-31T22:11:29Z
updated_at: 2020-05-31T22:11:29Z
author_association: OWNER
body:
  First version of cookie signing will use a secret that is either pulled from `DATASETTE_SECRET` environment variable or generated every time the server starts. I'll add a non-environment-variable based secret later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/785#issuecomment-636538298
issue_url: https://api.github.com/repos/simonw/datasette/issues/785
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUzODI5OA==
created_at: 2020-05-31T22:14:43Z
updated_at: 2020-05-31T22:15:01Z
author_association: OWNER
body:
  ... actually no I'll do it using a CLI option that can also be in an environment variable:

  https://click.palletsprojects.com/en/7.x/options/#values-from-environment-variables

  ```python
  @click.command()
  @click.option('--secret', envvar='DATASETTE_SECRET')
  def greet(secret):
      ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/785#issuecomment-636539295
issue_url: https://api.github.com/repos/simonw/datasette/issues/785
node_id: MDEyOklzc3VlQ29tbWVudDYzNjUzOTI5NQ==
created_at: 2020-05-31T22:24:14Z
updated_at: 2020-05-31T22:28:27Z
author_association: OWNER
body:
  I'll add two utility methods to the Datasette class:

  - `datasette.sign(value, "namespace")` - returns signed string
  - `datasette.unsign(signed, "namespace")` - returns value OR raises `BadSignature`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/785#issuecomment-636541630
issue_url: https://api.github.com/repos/simonw/datasette/issues/785
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU0MTYzMA==
created_at: 2020-05-31T22:45:07Z
updated_at: 2020-05-31T22:45:07Z
author_association: OWNER
body:
  Documentation for those new methods: https://github.com/simonw/datasette/blob/e28207e76ec3b26b2c396370fd3fb325a60bfd49/docs/internals.rst#signvalue-namespacedefault

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/785#issuecomment-636541827
issue_url: https://api.github.com/repos/simonw/datasette/issues/785
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU0MTgyNw==
created_at: 2020-05-31T22:46:34Z
updated_at: 2020-06-01T00:17:35Z
author_association: OWNER
body:
  This is nearly ready to close. I'm going to add documentation for `--secret` and the  `DATASETTE_SECRET` environment variable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/785#issuecomment-636541929
issue_url: https://api.github.com/repos/simonw/datasette/issues/785
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU0MTkyOQ==
created_at: 2020-05-31T22:47:17Z
updated_at: 2020-05-31T22:47:17Z
author_association: OWNER
body:
  I'll add a section about secrets to this page: https://datasette.readthedocs.io/en/latest/config.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/785#issuecomment-636553736
issue_url: https://api.github.com/repos/simonw/datasette/issues/785
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU1MzczNg==
created_at: 2020-06-01T00:18:40Z
updated_at: 2020-06-01T00:18:40Z
author_association: OWNER
body:
  That documentation: https://github.com/simonw/datasette/blob/c818de88a9c2683437875f788e325d911c8b767b/docs/config.rst#configuring-the-secret

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/784#issuecomment-636554258
issue_url: https://api.github.com/repos/simonw/datasette/issues/784
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU1NDI1OA==
created_at: 2020-06-01T00:21:33Z
updated_at: 2020-06-01T00:21:33Z
author_association: OWNER
body:
  The URL for this will be:

  `/-/auth-token?token=xxx`

  The token will be generated by Datasette on startup and will only be valid for a single request, at which point it will be used to set a signed `ds_actor` cookie and then redirect to the homepage.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636562658
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU2MjY1OA==
created_at: 2020-06-01T01:08:20Z
updated_at: 2020-06-01T01:08:54Z
author_association: OWNER
body:
  OK, the implementation in PR #783 is in a good state now - it implements the new plugin hooks with tests and documentation, plus it implements this:

      $ datasette . --root
      http://127.0.0.1:8001/-/auth-token?token=3ca9ee460a6451142389351d19b147bce27d2a785dfb6b5a74f82211be1ede49
      ...

  That URL, when clicked, will set a cookie for the `{"id": "root"}` user. The cookie is respected and used to populate `scope["actor"]`.

  I'm going to merge that pull request and continue working on this stuff on master.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/784#issuecomment-636565242
issue_url: https://api.github.com/repos/simonw/datasette/issues/784
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU2NTI0Mg==
created_at: 2020-06-01T01:18:20Z
updated_at: 2020-06-01T01:18:20Z
author_association: OWNER
body:
  I'm considering this done. I'm going to leave it to plugins to implement a web-based sign-in flow for accounts (at least for the moment).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636566433
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU2NjQzMw==
created_at: 2020-06-01T01:22:59Z
updated_at: 2020-06-01T01:22:59Z
author_association: OWNER
body:
  Some next steps:

  - Try out a branch of `datasette-auth-github` that builds on these new plugin hooks
  - Build a `datasette-api-tokens` plugin which implements `Authorization: bearer xxx` token support for API access
  - Maybe prototype up a `datasette-user-accounts` plugin which supports username/password accounts and allows an admin user to create/delete them
  - Do more work on writable canned queries in #698 and see what they look like if they take advantage of the permissions hook (to restrict some to only allowing authenticated users)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636566616
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU2NjYxNg==
created_at: 2020-06-01T01:23:48Z
updated_at: 2020-06-01T01:23:48Z
author_association: OWNER
body:
  https://latest.datasette.io/-/actor is now live (it returns `null` because there's no current way to sign into the `latest.datasette.io` site - not even with a fake `ds_actor` cookie because there's no way to know what that site's random secret is).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-636569917
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU2OTkxNw==
created_at: 2020-06-01T01:39:44Z
updated_at: 2020-06-01T01:39:44Z
author_association: OWNER
body:
  Idea for the authentication piece: I'll have the canned query code execute the following:

  ```python
  if await datasette.permission_allowed(
      request.scope.get("actor"),  "execute_query", "canned_query", query_name, default=True
  ):
  ```
  Then I'll add a default plugin to Datasette which implements that plugin hook, looks at the Datasette metadata for that query, and says "No" if the following (and `request.scope["actor"]` is empty):

  ```json
  {
      "databases": {
         "my-database": {
             "queries": {
                 "add_twitter_handle": {
                     "sql": "insert into twitter_handles (username) values (:username)",
                     "write": true,
                     "requires_actor": true
                 }
             }
         }
      }
  }
  ```
  I think I'll support this too:

  ```json
                     "allowed_actors": ["root"]
  ```
  So you can configure queries to only be available to specific `{"id": xxx}` actors.

  This will be the first time the new `permission_allowed` mechanism from #699 will be exercised in Datasette core.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636576252
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU3NjI1Mg==
created_at: 2020-06-01T02:11:40Z
updated_at: 2020-06-01T02:11:40Z
author_association: OWNER
body:
  Plugin idea: `datasette-allow-all` - really simple plugin which just says "yes" to every permission check.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-636576603
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU3NjYwMw==
created_at: 2020-06-01T02:13:26Z
updated_at: 2020-06-01T03:13:31Z
author_association: OWNER
body:
  Debugging tool idea: `/-/permissions` page which shows you the actor and lets you type in the strings for `action`, `resource_type` and `resource_identifier` - then shows you EVERY plugin hook that would have executed and what it would have said, plus when the chain would have terminated.

  Bonus: if you're logged in as the `root` user (or a user that matches some kind of permission check, maybe a check for `permissions_debug`) you get to see a rolling log of the last 30 permission checks and what the results were across the whole of Datasette. This should make figuring out permissions policies a whole lot easier.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/788#issuecomment-636598949
issue_url: https://api.github.com/repos/simonw/datasette/issues/788
node_id: MDEyOklzc3VlQ29tbWVudDYzNjU5ODk0OQ==
created_at: 2020-06-01T03:53:00Z
updated_at: 2020-06-01T03:53:00Z
author_association: OWNER
body:
  I can use a deque with a max length for this: https://docs.python.org/3/library/collections.html#deque-objects

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/786#issuecomment-636614062
issue_url: https://api.github.com/repos/simonw/datasette/issues/786
node_id: MDEyOklzc3VlQ29tbWVudDYzNjYxNDA2Mg==
created_at: 2020-06-01T05:02:18Z
updated_at: 2020-06-01T05:02:18Z
author_association: OWNER
body:
  The skeleton of this page now exists at https://datasette.readthedocs.io/en/latest/authentication.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/789#issuecomment-636616307
issue_url: https://api.github.com/repos/simonw/datasette/issues/789
node_id: MDEyOklzc3VlQ29tbWVudDYzNjYxNjMwNw==
created_at: 2020-06-01T05:11:03Z
updated_at: 2020-06-01T05:11:03Z
author_association: OWNER
body:
  Or I could get fancy and implement my own `pm.trace.root.setwriter` function which collects data that can be appended to the `?_trace=1` dump.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/788#issuecomment-636616638
issue_url: https://api.github.com/repos/simonw/datasette/issues/788
node_id: MDEyOklzc3VlQ29tbWVudDYzNjYxNjYzOA==
created_at: 2020-06-01T05:12:30Z
updated_at: 2020-06-01T05:12:30Z
author_association: OWNER
body:
  Looks like this (at the moment):

  <img width="789" alt="Debug_permissions" src="https://user-images.githubusercontent.com/9599/83377861-cf2f8100-a38b-11ea-9f7d-b6334c1711c0.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-636617140
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYzNjYxNzE0MA==
created_at: 2020-06-01T05:14:39Z
updated_at: 2020-06-01T05:14:39Z
author_association: OWNER
body:
  Here's the new `default_permissions.py` file I can add this permission check to: https://github.com/simonw/datasette/blob/dfdbdf378aba9afb66666f66b78df2f2069d2595/datasette/default_permissions.py#L1-L7

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-636906773
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNjkwNjc3Mw==
created_at: 2020-06-01T14:57:02Z
updated_at: 2020-06-01T14:58:14Z
author_association: OWNER
body:
  Actually I'm inclined to use cookies now, ala Django: https://docs.djangoproject.com/en/3.0/ref/contrib/messages/

  > This class stores the message data in a cookie (signed with a secret hash to prevent manipulation) to persist notifications across requests. Old messages are dropped if the cookie data size would exceed 2048 bytes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-636908972
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNjkwODk3Mg==
created_at: 2020-06-01T15:01:00Z
updated_at: 2020-06-01T15:01:00Z
author_association: OWNER
body:
  Setting messages just needs access to the response. Reading messages needs access to both request AND response, since it needs to clear the messages that are being displayed.

  That's if the messages are persisted exclusively in cookies - which makes sense for Django since it's designed to run as many different load-balanced processes.

  Since Datasette is a single process which can access an on-file database, maybe consider storing the flash messages within Datasette memory itself - a sort of session mechanism?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-636912730
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNjkxMjczMA==
created_at: 2020-06-01T15:08:13Z
updated_at: 2020-06-01T15:08:13Z
author_association: OWNER
body:
  I'm going to build the first version of this with signed cookies.

  I'm inclined to do this all on the request object, since it's the object representing the current request as it flows through the application. I need the ability to remember which messages were set and which need to be cleared, so I need to do that on something that is available for the lifetime of the request.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-636915499
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNjkxNTQ5OQ==
created_at: 2020-06-01T15:13:40Z
updated_at: 2020-06-01T15:13:40Z
author_association: OWNER
body:
  Maybe two utility functions:

  `add_message(request, message)` - adds a Flash message (will be set later)

  `read_and_clear_messages(request)` - reads messages and sets them to be cleared

  Problem: the `request` object isn't created at the very top of the stack - it's actually created within each view. So maybe I need to move its creation up to the top of the routing stuff so that the code that returns the response can see it?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-636916107
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNjkxNjEwNw==
created_at: 2020-06-01T15:14:30Z
updated_at: 2020-06-01T15:15:52Z
author_association: OWNER
body:
  Alternative: `datasette.add_message(message)` and `datasette.read_and_clear_messages()` - these would need some kind of dark magic to ensure that the message was associated with the current request flowing through the system though, since that `datasette` object is shared my multiple concurrent requests.

  Maybe use a request correlation ID that gets added to the scope? This is all getting a bit messy.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-636920304
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNjkyMDMwNA==
created_at: 2020-06-01T15:22:15Z
updated_at: 2020-06-01T15:22:15Z
author_association: OWNER
body:
  Here's how the Django stuff works: https://github.com/django/django/blob/master/django/contrib/messages/storage/base.py

  Notably the messages are mostly dealt with on the request object, with a piece of middleware that reads from the request and modifies the response (to set or clear cookies) right at the end: https://github.com/django/django/blob/master/django/contrib/messages/middleware.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-636922104
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNjkyMjEwNA==
created_at: 2020-06-01T15:25:39Z
updated_at: 2020-06-01T15:25:39Z
author_association: OWNER
body:
  What if I use a mutable key on `scope` to track messages for the duration of the request? Is that an OK thing to do?

  ASGI spec says this: https://asgi.readthedocs.io/en/latest/specs/main.html#middleware

  > ### Middleware
  > 
  > It is possible to have ASGI "middleware" - code that plays the role of both server and application, taking in a scope and the send/receive awaitables, potentially modifying them, and then calling an inner application.
  > 
  > When middleware is modifying the scope, it should make a copy of the scope object before mutating it and passing it to the inner application, as changes may leak upstream otherwise. In particular, you should not assume that the copy of the scope you pass down to the application is the one that it ends up using, as there may be other middleware in the way; thus, do not keep a reference to it and try to mutate it outside of the initial ASGI constructor callable that receives `scope`. Your one and only chance to add to it is before you hand control to the child application.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-636925354
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNjkyNTM1NA==
created_at: 2020-06-01T15:32:02Z
updated_at: 2020-06-01T15:32:02Z
author_association: OWNER
body:
  If `scope` had an immutable correlation ID I could use that with a dict somewhere mapping `correlation_id` to a messages object.

  The problem then is how do I know to clean up the memory used by that dictionary when the request flows out of the system? I guess the code that updates the cookies in the response could do that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-636959774
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNjk1OTc3NA==
created_at: 2020-06-01T16:15:33Z
updated_at: 2020-06-01T16:15:33Z
author_association: OWNER
body:
  It would be neat if this was driven by a method on `datasette` just because that's already the object passed to plugins as a documented API.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/791#issuecomment-636973355
issue_url: https://api.github.com/repos/simonw/datasette/issues/791
node_id: MDEyOklzc3VlQ29tbWVudDYzNjk3MzM1NQ==
created_at: 2020-06-01T16:33:33Z
updated_at: 2020-06-01T16:33:33Z
author_association: OWNER
body:
  A fun thing about this tutorial is that it can start with a classic, basic todo list - and then start growing all kinds of outlandish features to help demonstrate various Datasette plugins and approaches.

  Your TODOs on a map. URLs in TODOs that have been unfurled. Tag your TODOs and browse them with facets. Vega graphs showing your progress. Etc etc etc.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-636978065
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNjk3ODA2NQ==
created_at: 2020-06-01T16:42:59Z
updated_at: 2020-06-01T17:44:12Z
author_association: OWNER
body:
  `datasette.add_message(request, message, type=datasette.INFO)`

  Then later:

  `datasette.write_messages_to_response(request, response)`
  Writes the messages as cookies in the response.

  `datasette.fetch_and_clear_messages(request, response)`
  To display messages and clears them from the response.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-637009509
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNzAwOTUwOQ==
created_at: 2020-06-01T17:44:55Z
updated_at: 2020-06-01T17:46:18Z
author_association: OWNER
body:
  Problem with `datasette.fetch_and_clear_messages(request, response)` is that I want to call it from the template (so we only clear messages if they have been displayed) - but by that point in the code the `Response` object has not yet been created, so it can't have cookie set on it to clear the list of messages.

  Solution: call it `datasette.show_messages(request)` and have it update internal state on the `datasette` object such that a later call to `write_messages_to_response(request, response)` knows to clear the cookie.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-637066496
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNzA2NjQ5Ng==
created_at: 2020-06-01T19:48:20Z
updated_at: 2020-06-01T19:48:20Z
author_association: OWNER
body:
  I'm going to stash these on the `request` object after all, so the memory used for the messages gets automatically cleaned up at the end of the request.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-637699337
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNzY5OTMzNw==
created_at: 2020-06-02T17:34:47Z
updated_at: 2020-06-02T17:34:47Z
author_association: OWNER
body:
  I'm going to use a output renderer plugin to test this, since then my unit tests can run against custom code that both sets and displays messages.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-637708090
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNzcwODA5MA==
created_at: 2020-06-02T17:52:30Z
updated_at: 2020-06-02T17:52:30Z
author_association: OWNER
body:
  I need to make sure that any time cookies are set there's no cache-control header (or it is set to private).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-637790860
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNzc5MDg2MA==
created_at: 2020-06-02T20:34:15Z
updated_at: 2020-06-02T20:34:15Z
author_association: OWNER
body:
  The `/-/messages` debug tool will need CSRF protection or people will be able to add messages using a hidden form on another website.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-637793590
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNzc5MzU5MA==
created_at: 2020-06-02T20:40:02Z
updated_at: 2020-06-02T20:40:02Z
author_association: OWNER
body:
  From https://github.com/simonw/datasette/issues/698#issuecomment-621037724
  > Concept for displaying a success message:
  > 
  > <img width="662" alt="fixtures__compound_three_primary_keys__1_001_rows" src="https://user-images.githubusercontent.com/9599/80571201-0a1b4f00-89b1-11ea-8688-2e3e4304e475.png">
  > CSS:
  > 
  > ```css
  > .success {
  >     padding: 1em;
  >     border: 1px solid green;
  >     background-color: #c7fbc7;
  > }

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/790#issuecomment-637813616
issue_url: https://api.github.com/repos/simonw/datasette/issues/790
node_id: MDEyOklzc3VlQ29tbWVudDYzNzgxMzYxNg==
created_at: 2020-06-02T21:22:02Z
updated_at: 2020-06-02T21:22:02Z
author_association: OWNER
body:
  Debug tool is live here: https://latest.datasette.io/-/messages

  Documentation is here: https://github.com/simonw/datasette/blob/master/docs/internals.rst#add_messagerequest-message-message_typedatasetteinfo

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/793#issuecomment-637813993
issue_url: https://api.github.com/repos/simonw/datasette/issues/793
node_id: MDEyOklzc3VlQ29tbWVudDYzNzgxMzk5Mw==
created_at: 2020-06-02T21:22:50Z
updated_at: 2020-06-02T21:22:50Z
author_association: OWNER
body:
  This is a minor security issue with `master` at the moment, but I'll resolve this before I ship the next release.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-637819025
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDYzNzgxOTAyNQ==
created_at: 2020-06-02T21:34:31Z
updated_at: 2020-06-02T21:34:31Z
author_association: OWNER
body:
  I can close this issue once I've expanded out this page of documentation https://datasette.readthedocs.io/en/latest/authentication.html - and published at least one plugin and/or feature that takes advantage of this new mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/794#issuecomment-637832651
issue_url: https://api.github.com/repos/simonw/datasette/issues/794
node_id: MDEyOklzc3VlQ29tbWVudDYzNzgzMjY1MQ==
created_at: 2020-06-02T22:10:34Z
updated_at: 2020-06-02T22:10:34Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/-/plugins?all=1

  ```json
  [
    {
      "name": "datasette.facets",
      "static": false,
      "templates": false,
      "version": null,
      "hooks": [
        "register_facet_classes"
      ]
    },
    {
      "name": "datasette.publish.cloudrun",
      "static": false,
      "templates": false,
      "version": null,
      "hooks": [
        "publish_subcommand"
      ]
    },
    {
      "name": "datasette.actor_auth_cookie",
      "static": false,
      "templates": false,
      "version": null,
      "hooks": [
        "actor_from_request"
      ]
    },
    {
      "name": "datasette.default_permissions",
      "static": false,
      "templates": false,
      "version": null,
      "hooks": [
        "permission_allowed"
      ]
    },
    {
      "name": "datasette.sql_functions",
      "static": false,
      "templates": false,
      "version": null,
      "hooks": [
        "prepare_connection"
      ]
    },
    {
      "name": "datasette.publish.heroku",
      "static": false,
      "templates": false,
      "version": null,
      "hooks": [
        "publish_subcommand"
      ]
    }
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-637934813
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYzNzkzNDgxMw==
created_at: 2020-06-03T03:45:07Z
updated_at: 2020-06-03T03:45:07Z
author_association: OWNER
body:
  Some extra thoughts now that this is mostly working:

  - "Edit this row" is such an obvious use-case. Could I automatically support row editing where every column except the primary key can be updated?
  - It would be useful to be able to link to a query in a way that pre-populates various form fields. The "edit" interface could then be a link that pre-populates the form with all of the existing values.
  - Can the redirect URL be configured to include values from the form submission? So you could e.g. add a blog post with a unique slug and then redirect to that URL?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-638183337
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYzODE4MzMzNw==
created_at: 2020-06-03T13:05:03Z
updated_at: 2020-06-03T13:05:03Z
author_association: OWNER
body:
  One challenge with this feature is that it confuses the messaging about what Datasette does somewhat.

  Prior to shipping this, Datasette's core value proposition is as a way to publish read-only data.

  That changed a little [in 0.37 in February](https://datasette.readthedocs.io/en/stable/changelog.html#v0-37) when plugins gained the supported ability to execute writes, but there was no way of doing that without a plugin.

  With this feature, Datasette becomes a read-write database solution.

  I should update the documentation to help explain this. Essentially the message is that Datasette can be used in one of two "modes" - it can be used just for sharing/publishing data, or you can use it to collect and manage data, most likely still in collaboration with plugins for things like authentication.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/796#issuecomment-638188196
issue_url: https://api.github.com/repos/simonw/datasette/issues/796
node_id: MDEyOklzc3VlQ29tbWVudDYzODE4ODE5Ng==
created_at: 2020-06-03T13:13:27Z
updated_at: 2020-06-03T14:32:27Z
author_association: OWNER
body:
  "Query executed" is the default message, but it's pretty bland:

  <img width="805" alt="data__insert_into_names__name__values___name_" src="https://user-images.githubusercontent.com/9599/83640640-42d6b700-a561-11ea-9ed4-fe7815d24df2.png">

  How about letting queries define custom success messages in their metadata configuration?

  `"on_success_message"` and `"on_error_message"`

  How can the system tell if an "update" query was actually successful? Maybe I should expose `.rowcount` somehow, so I can report back on how many rows were updated.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/796#issuecomment-638205923
issue_url: https://api.github.com/repos/simonw/datasette/issues/796
node_id: MDEyOklzc3VlQ29tbWVudDYzODIwNTkyMw==
created_at: 2020-06-03T13:43:12Z
updated_at: 2020-06-03T13:43:12Z
author_association: OWNER
body:
  For `.json` mode (when the URL has a `.json` suffix) I'm going to handle both form-encoded and JSON encoded inputs and return a 200 status code with JSON that tells you if the query executed successfully and how many rows were affected.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/796#issuecomment-638206851
issue_url: https://api.github.com/repos/simonw/datasette/issues/796
node_id: MDEyOklzc3VlQ29tbWVudDYzODIwNjg1MQ==
created_at: 2020-06-03T13:44:29Z
updated_at: 2020-06-03T13:44:29Z
author_association: OWNER
body:
  Default message is now "Query executed, 1 row affected" as of https://github.com/simonw/datasette/pull/796/commits/f45c44ac8f3a9a2182d76c6bda44a06676499e4b

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/796#issuecomment-638238144
issue_url: https://api.github.com/repos/simonw/datasette/issues/796
node_id: MDEyOklzc3VlQ29tbWVudDYzODIzODE0NA==
created_at: 2020-06-03T14:32:54Z
updated_at: 2020-06-03T14:33:11Z
author_association: OWNER
body:
  I'm going to have `on_success_redirect` and `on_error_redirect` properties for specifying where the redirect should go too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/796#issuecomment-638240919
issue_url: https://api.github.com/repos/simonw/datasette/issues/796
node_id: MDEyOklzc3VlQ29tbWVudDYzODI0MDkxOQ==
created_at: 2020-06-03T14:37:22Z
updated_at: 2020-06-03T14:37:22Z
author_association: OWNER
body:
  I'm having trouble coming up with a canned SQL query that will only error with certain parameters, which I need in order to test out the `on_error_redirect` and `on_error_message` properties.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/796#issuecomment-638241366
issue_url: https://api.github.com/repos/simonw/datasette/issues/796
node_id: MDEyOklzc3VlQ29tbWVudDYzODI0MTM2Ng==
created_at: 2020-06-03T14:38:03Z
updated_at: 2020-06-03T14:38:31Z
author_association: OWNER
body:
  Maybe I need some kind of optional validation mechanism? SQLite lets me insert text or floating point numbers into integer columns right now, which feels wrong.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/796#issuecomment-638249652
issue_url: https://api.github.com/repos/simonw/datasette/issues/796
node_id: MDEyOklzc3VlQ29tbWVudDYzODI0OTY1Mg==
created_at: 2020-06-03T14:51:29Z
updated_at: 2020-06-03T14:51:51Z
author_association: OWNER
body:
  Consider this one:
  ```
                          "delete_name": {
                              "sql": "delete from names where rowid = :rowid",
                              "write": True,
                              "on_success_message": "Name deleted",
                          },
  ```
  If the user enters an invalid `rowid` the query will still execute without errors and hence the success message will still be displayed. Can I address this?

  Maybe allow an optional `"rowcount_expected": 1` property? And if that count isn't matched treat the query as an error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/796#issuecomment-638257697
issue_url: https://api.github.com/repos/simonw/datasette/issues/796
node_id: MDEyOklzc3VlQ29tbWVudDYzODI1NzY5Nw==
created_at: 2020-06-03T15:05:07Z
updated_at: 2020-06-03T15:05:07Z
author_association: OWNER
body:
  I'm going to document this, land it and then continue to work on the other pieces - CSRF protection and .json mode - in separate tickets.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-638266171
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYzODI2NjE3MQ==
created_at: 2020-06-03T15:18:49Z
updated_at: 2020-06-03T15:18:49Z
author_association: OWNER
body:
  Landed the work so far from #796! Here's the documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#writable-canned-queries

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/797#issuecomment-638461797
issue_url: https://api.github.com/repos/simonw/datasette/issues/797
node_id: MDEyOklzc3VlQ29tbWVudDYzODQ2MTc5Nw==
created_at: 2020-06-03T21:07:06Z
updated_at: 2020-06-03T21:07:06Z
author_association: OWNER
body:
  Docs here (search for "params"): https://datasette.readthedocs.io/en/latest/sql_queries.html#canned-queries-named-parameters

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/793#issuecomment-638462052
issue_url: https://api.github.com/repos/simonw/datasette/issues/793
node_id: MDEyOklzc3VlQ29tbWVudDYzODQ2MjA1Mg==
created_at: 2020-06-03T21:07:39Z
updated_at: 2020-06-03T21:07:39Z
author_association: OWNER
body:
  I need to land and release the fix for signing cookies in https://github.com/simonw/asgi-csrf/issues/2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/119#issuecomment-639047315
issue_url: https://api.github.com/repos/simonw/datasette/issues/119
node_id: MDEyOklzc3VlQ29tbWVudDYzOTA0NzMxNQ==
created_at: 2020-06-04T18:46:39Z
updated_at: 2020-06-04T18:46:39Z
author_association: OWNER
body:
  The OAuth dance needed for this is a pretty nasty barrier to plugin installation and configuration.

  I'm going to focus on making it easy to copy and paste data into sheets instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/684#issuecomment-639053707
issue_url: https://api.github.com/repos/simonw/datasette/issues/684
node_id: MDEyOklzc3VlQ29tbWVudDYzOTA1MzcwNw==
created_at: 2020-06-04T18:56:15Z
updated_at: 2020-06-04T18:56:15Z
author_association: OWNER
body:
  This documentation is live here: https://datasette.readthedocs.io/en/latest/internals.html#database-introspection

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/798#issuecomment-639249743
issue_url: https://api.github.com/repos/simonw/datasette/issues/798
node_id: MDEyOklzc3VlQ29tbWVudDYzOTI0OTc0Mw==
created_at: 2020-06-05T04:23:01Z
updated_at: 2020-06-05T04:23:01Z
author_association: OWNER
body:
  Needs unit tests.

  More importantly: needs very, very careful consideration of how this plays with HTTP caching.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/798#issuecomment-639269559
issue_url: https://api.github.com/repos/simonw/datasette/issues/798
node_id: MDEyOklzc3VlQ29tbWVudDYzOTI2OTU1OQ==
created_at: 2020-06-05T05:34:56Z
updated_at: 2020-06-05T05:35:23Z
author_association: OWNER
body:
  I don't want to set a cookie on a page response that is being cached.

  Right now the ASGI middleware will be doing exactly that, which is bad.

  But how do I get certainty that when you load a page with a form that will be CSRF protected you have been served the cookie?

  Maybe those pages should do something explicit to the request object indicating that the cookie is needed?

  That works for Datasette (since it has mutable request objects) but I'm not sure how it would work in the asgi-csrf pure ASGI middleware context.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/798#issuecomment-639269994
issue_url: https://api.github.com/repos/simonw/datasette/issues/798
node_id: MDEyOklzc3VlQ29tbWVudDYzOTI2OTk5NA==
created_at: 2020-06-05T05:36:35Z
updated_at: 2020-06-05T05:38:25Z
author_association: OWNER
body:
  Django docs on CSRF and caching: https://docs.djangoproject.com/en/3.0/ref/csrf/#caching

  > If the csrf_token template tag is used by a template (or the get_token function is called some other way), CsrfViewMiddleware will add a cookie and a Vary: Cookie header to the response. This means that the middleware will play well with the cache middleware if it is used as instructed

  So the cookie is only set for pages that included a hidden csrftoken form field! This could work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/799#issuecomment-639660667
issue_url: https://api.github.com/repos/simonw/datasette/issues/799
node_id: MDEyOklzc3VlQ29tbWVudDYzOTY2MDY2Nw==
created_at: 2020-06-05T17:43:08Z
updated_at: 2020-06-05T17:43:08Z
author_association: OWNER
body:
  This really needs a `MultiValueDict` ala Django: https://github.com/django/django/blob/24b82cd201e21060fbc02117dc16d1702877a1f3/django/utils/datastructures.py#L42

  Turns out I have one of these in Datasette already - `RequestParameters` from https://github.com/simonw/datasette/commit/81be31322a968d23cf57cee62b58df55433385e3

  The name isn't quite right though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/798#issuecomment-639685550
issue_url: https://api.github.com/repos/simonw/datasette/issues/798
node_id: MDEyOklzc3VlQ29tbWVudDYzOTY4NTU1MA==
created_at: 2020-06-05T18:20:34Z
updated_at: 2020-06-05T18:20:34Z
author_association: OWNER
body:
  I'm solving the compatibility with caching problem in this ticket: https://github.com/simonw/asgi-csrf/issues/7

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-639779403
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYzOTc3OTQwMw==
created_at: 2020-06-05T20:20:12Z
updated_at: 2020-06-05T20:20:12Z
author_association: OWNER
body:
  CSRF is done. Last step: figure out a smart way to integrate this with permissions and authentication.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-639784651
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYzOTc4NDY1MQ==
created_at: 2020-06-05T20:25:02Z
updated_at: 2020-06-05T20:25:02Z
author_association: OWNER
body:
  Idea: default is anyone can execute a query.

  Or you can specify the following:

  ```json

  {
      "databases": {
         "my-database": {
             "queries": {
                 "add_twitter_handle": {
                     "sql": "insert into twitter_handles (username) values (:username)",
                     "write": true,
                     "allow": {
                         "id": ["simon"],
                         "role": ["staff"]
                     }
                 }
             }
         }
      }
  }
  ```
  These get matched against the actor JSON. If any of the fields in any of the keys of `"allow"` match a key on the actor, the query is allowed.

  `"id": "*"` matches any actor with an `id` key.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-639785878
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYzOTc4NTg3OA==
created_at: 2020-06-05T20:25:55Z
updated_at: 2020-06-05T20:25:55Z
author_association: OWNER
body:
  I'd really like to support SQL query defined permissions too, mainly to set an example for how plugins could do something similar.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-639787304
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYzOTc4NzMwNA==
created_at: 2020-06-05T20:26:57Z
updated_at: 2020-06-05T20:26:57Z
author_association: OWNER
body:
  Idea: an `"allow_sql"` key with a SQL query that gets passed the actor JSON as `:actor` and can extract the relevant keys from it and return 1 or 0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-639788562
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDYzOTc4ODU2Mg==
created_at: 2020-06-05T20:27:49Z
updated_at: 2020-06-05T20:27:49Z
author_association: OWNER
body:
  There can be a detailed section explaining these different mechanisms on the authentication documentation page.

  I imagine they will end up applying to more than just canned queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/800#issuecomment-639803099
issue_url: https://api.github.com/repos/simonw/datasette/issues/800
node_id: MDEyOklzc3VlQ29tbWVudDYzOTgwMzA5OQ==
created_at: 2020-06-05T20:39:34Z
updated_at: 2020-06-05T20:39:34Z
author_association: OWNER
body:
  Maybe #801 (configuring permissions with a SQL query) is enough here - might not need this mechanism at all, since that mechanism covers it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/802#issuecomment-639895450
issue_url: https://api.github.com/repos/simonw/datasette/issues/802
node_id: MDEyOklzc3VlQ29tbWVudDYzOTg5NTQ1MA==
created_at: 2020-06-05T23:33:52Z
updated_at: 2020-06-05T23:33:52Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/033a1bb22c70a955d9fd1d3b4675a0e2e5c8b8cd/datasette/cli.py#L126-L129

  But I changed the `.plugins()` method to this:

  https://github.com/simonw/datasette/blob/033a1bb22c70a955d9fd1d3b4675a0e2e5c8b8cd/datasette/app.py#L628-L633

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/800#issuecomment-640090343
issue_url: https://api.github.com/repos/simonw/datasette/issues/800
node_id: MDEyOklzc3VlQ29tbWVudDY0MDA5MDM0Mw==
created_at: 2020-06-06T17:04:36Z
updated_at: 2020-06-06T17:04:36Z
author_association: OWNER
body:
  I like this mechanism better than the SQL query one. Constructing SQL queries that return true if a particular string is embedded inside a JSON list in a larger object is decidedly non-trivial.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/786#issuecomment-640099333
issue_url: https://api.github.com/repos/simonw/datasette/issues/786
node_id: MDEyOklzc3VlQ29tbWVudDY0MDA5OTMzMw==
created_at: 2020-06-06T18:21:36Z
updated_at: 2020-06-06T18:21:36Z
author_association: OWNER
body:
  This is done but currently lives in a branch, will close this issue when that branch lands: Implemented in this branch: https://github.com/simonw/datasette/blob/30a8132d58a89fed0e034e058b62fab5180fae0f/docs/authentication.rst

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/800#issuecomment-640099404
issue_url: https://api.github.com/repos/simonw/datasette/issues/800
node_id: MDEyOklzc3VlQ29tbWVudDY0MDA5OTQwNA==
created_at: 2020-06-06T18:22:10Z
updated_at: 2020-06-06T18:24:26Z
author_association: OWNER
body:
  Docs here: https://github.com/simonw/datasette/blob/d4c7b85f556230923d37ff327a068ed08aa9b62b/docs/authentication.rst#setting-permissions-for-canned-queries

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/800#issuecomment-640101625
issue_url: https://api.github.com/repos/simonw/datasette/issues/800
node_id: MDEyOklzc3VlQ29tbWVudDY0MDEwMTYyNQ==
created_at: 2020-06-06T18:40:09Z
updated_at: 2020-06-06T18:40:09Z
author_association: OWNER
body:
  Documentation for `actor_matches_allow`: https://github.com/simonw/datasette/blob/14f6b4d200f24940a795ddc0825319ab2891bde2/docs/authentication.rst#actor_matches_allow

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/800#issuecomment-640101762
issue_url: https://api.github.com/repos/simonw/datasette/issues/800
node_id: MDEyOklzc3VlQ29tbWVudDY0MDEwMTc2Mg==
created_at: 2020-06-06T18:41:20Z
updated_at: 2020-06-06T18:41:20Z
author_association: OWNER
body:
  Now the actual permission checks. I need these in two places: the code that generates the list of available queries on https://latest.datasette.io/fixtures#queries and the query page itself at https://latest.datasette.io/fixtures/pragma_cache_size

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/800#issuecomment-640102200
issue_url: https://api.github.com/repos/simonw/datasette/issues/800
node_id: MDEyOklzc3VlQ29tbWVudDY0MDEwMjIwMA==
created_at: 2020-06-06T18:45:11Z
updated_at: 2020-06-06T18:45:11Z
author_association: OWNER
body:
  In the code that's:

  https://github.com/simonw/datasette/blob/9c563d6aed072f14d3d25f58e84659f9caa1a243/datasette/views/database.py#L56-L64

  And:

  https://github.com/simonw/datasette/blob/9c563d6aed072f14d3d25f58e84659f9caa1a243/datasette/views/database.py#L98-L112

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/800#issuecomment-640103204
issue_url: https://api.github.com/repos/simonw/datasette/issues/800
node_id: MDEyOklzc3VlQ29tbWVudDY0MDEwMzIwNA==
created_at: 2020-06-06T18:52:56Z
updated_at: 2020-06-06T18:52:56Z
author_association: OWNER
body:
  I'm also going to add an indicator to the UI next to queries that you can only execute because you are signed in:

  <img width="502" alt="data" src="https://user-images.githubusercontent.com/9599/83952232-403bc200-a7ec-11ea-94dc-f9b87ad3fb67.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/804#issuecomment-640106202
issue_url: https://api.github.com/repos/simonw/datasette/issues/804
node_id: MDEyOklzc3VlQ29tbWVudDY0MDEwNjIwMg==
created_at: 2020-06-06T19:18:23Z
updated_at: 2020-06-06T19:18:43Z
author_association: OWNER
body:
  I broke this in #775 https://github.com/simonw/datasette/commit/446e5de65d1b9c6c877e38b0ef13bc9285c465a1

  Here's the now-broken code (I removed the `PLUGIN1` and `PLUGIN2` constants): https://github.com/simonw/datasette/blob/9c563d6aed072f14d3d25f58e84659f9caa1a243/tests/fixtures.py#L828-L835

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/804#issuecomment-640106569
issue_url: https://api.github.com/repos/simonw/datasette/issues/804
node_id: MDEyOklzc3VlQ29tbWVudDY0MDEwNjU2OQ==
created_at: 2020-06-06T19:21:41Z
updated_at: 2020-06-06T19:21:41Z
author_association: OWNER
body:
  I don't think this is fully documented either. Current partial documentation is on https://datasette.readthedocs.io/en/stable/contributing.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/699#issuecomment-640108763
issue_url: https://api.github.com/repos/simonw/datasette/issues/699
node_id: MDEyOklzc3VlQ29tbWVudDY0MDEwODc2Mw==
created_at: 2020-06-06T19:42:11Z
updated_at: 2020-06-06T19:42:11Z
author_association: OWNER
body:
  I landed canned query writes. This feature can now be considered complete: https://datasette.readthedocs.io/en/latest/authentication.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/698#issuecomment-640108942
issue_url: https://api.github.com/repos/simonw/datasette/issues/698
node_id: MDEyOklzc3VlQ29tbWVudDY0MDEwODk0Mg==
created_at: 2020-06-06T19:43:48Z
updated_at: 2020-06-06T19:43:48Z
author_association: OWNER
body:
  Landed - documentation is here: https://datasette.readthedocs.io/en/latest/sql_queries.html#writable-canned-queries

  See also https://datasette.readthedocs.io/en/latest/authentication.html#permissions-for-canned-queries

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/787#issuecomment-640111383
issue_url: https://api.github.com/repos/simonw/datasette/issues/787
node_id: MDEyOklzc3VlQ29tbWVudDY0MDExMTM4Mw==
created_at: 2020-06-06T20:04:20Z
updated_at: 2020-06-06T20:04:20Z
author_association: OWNER
body:
  I should let people running the 'publish' command set this explicitly if they want to, so they can re-deploy a published Datasette without invalidating every user's cookies.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/791#issuecomment-640116494
issue_url: https://api.github.com/repos/simonw/datasette/issues/791
node_id: MDEyOklzc3VlQ29tbWVudDY0MDExNjQ5NA==
created_at: 2020-06-06T20:50:41Z
updated_at: 2020-06-06T20:50:41Z
author_association: OWNER
body:
  I have a better idea: a feed reader! You can insert URLs to feeds, then have a command which fetches the latest entries from them into a separate table. Then implement favorites as a canned query, let you search your favorites, etc.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/805#issuecomment-640116842
issue_url: https://api.github.com/repos/simonw/datasette/issues/805
node_id: MDEyOklzc3VlQ29tbWVudDY0MDExNjg0Mg==
created_at: 2020-06-06T20:53:51Z
updated_at: 2020-06-06T20:53:51Z
author_association: OWNER
body:
  I'd like to illustrate writable canned queries without the risk of someone abusing and breaking it (or filling it with bad content).

  I don't want to have to monitor it, so an application that won't run out of disk space after a few months would be good too.

  Maybe a polling app? If I'm only tracking integer numbers of votes it shouldn't ever run out of space.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/805#issuecomment-640116970
issue_url: https://api.github.com/repos/simonw/datasette/issues/805
node_id: MDEyOklzc3VlQ29tbWVudDY0MDExNjk3MA==
created_at: 2020-06-06T20:55:03Z
updated_at: 2020-06-06T20:55:03Z
author_association: OWNER
body:
  Would be useful if I had a plugin that could authenticate users based on a secret environment variable (maybe for a password) - that way I could have an "admin" account on the Glitch app that is allowed to setup new polls, while anonymous users can only vote on them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/215#issuecomment-640118802
issue_url: https://api.github.com/repos/simonw/datasette/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDY0MDExODgwMg==
created_at: 2020-06-06T21:12:41Z
updated_at: 2020-06-06T21:12:41Z
author_association: OWNER
body:
  @clausjuhl your use-case there is now covered by custom pages from Datasette 0.41 https://datasette.readthedocs.io/en/stable/changelog.html#v0-41

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/215#issuecomment-640119259
issue_url: https://api.github.com/repos/simonw/datasette/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDY0MDExOTI1OQ==
created_at: 2020-06-06T21:16:46Z
updated_at: 2020-06-06T21:16:46Z
author_association: OWNER
body:
  I deprioritised this a while ago because the asgi_wrapper hook allowed me to set up new URL routes: https://datasette.readthedocs.io/en/0.43/plugins.html#asgi-wrapper-datasette

  But... those were pretty low level, for example this code here: https://github.com/simonw/datasette-auth-github/blob/6c971064f6f4e6857bade5c6b88842f9cdeca9d9/datasette_auth_github/github_auth.py#L104-L113

  Now that Datasette has a documented request object #706 and that object is used by things like the flash messages system (#790) - https://datasette.readthedocs.io/en/latest/internals.html#add-message-request-message-message-type-datasette-info - I find myself wanting to add views which get a request, as opposed to an ASGI scope.

  So I'm re-prioritising this, with the main need being a way for plugins to hook up their own view functions that can accept a request and return a response. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/215#issuecomment-640121036
issue_url: https://api.github.com/repos/simonw/datasette/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDY0MDEyMTAzNg==
created_at: 2020-06-06T21:34:03Z
updated_at: 2020-06-06T21:34:03Z
author_association: OWNER
body:
  I'll refactor existing code to register views using the same mechanism that plugins will have access to.

  Maybe plugins get to register their routes first? That would allow plugins to do things like entirely take over the / page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/215#issuecomment-640121917
issue_url: https://api.github.com/repos/simonw/datasette/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDY0MDEyMTkxNw==
created_at: 2020-06-06T21:42:58Z
updated_at: 2020-06-07T05:58:36Z
author_association: OWNER
body:
  I might use some dependency injection here, with `call_with_supported_arguments()` from https://github.com/simonw/datasette/commit/41a0cd7b6afe0397efbbf27ad822679fc574811a#diff-942305c83055fdc0ff5f4e7d6ab06b29

  Maybe a view function can take `request` and optionally also take `datasette`? Or `scope` or `receive` or `send`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/215#issuecomment-640122120
issue_url: https://api.github.com/repos/simonw/datasette/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDY0MDEyMjEyMA==
created_at: 2020-06-06T21:45:13Z
updated_at: 2020-06-06T21:45:52Z
author_association: OWNER
body:
  Stretch goal: make it easy for plugin views to implement formats, so they can produce HTML by default and .json or .csv etc as alternative outputs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-640135332
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0MDEzNTMzMg==
created_at: 2020-06-07T00:13:51Z
updated_at: 2020-06-07T00:13:51Z
author_association: OWNER
body:
  These should not be shipped as the latest version on Docker Hub. They also should not become the "stable" release on ReadTheDocs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640248669
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDI0ODY2OQ==
created_at: 2020-06-07T17:01:44Z
updated_at: 2020-06-07T17:01:44Z
author_association: OWNER
body:
  If the allow block at the database level forbids access this needs to cascade down to the table, query and row levels as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640270178
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDI3MDE3OA==
created_at: 2020-06-07T19:48:39Z
updated_at: 2020-06-07T19:48:39Z
author_association: OWNER
body:
  Testing pattern:
  ```python
  def test_canned_query_with_custom_metadata(app_client):
      response = app_client.get("/fixtures/neighborhood_search?text=town")
      assert_permissions_checked(
          app_client.ds,
          [
              "view-instance",
              ("view-database", "database", "fixtures"),
              ("view-query", "query", ("fixtures", "neighborhood_search")),
          ],
      )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640273945
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDI3Mzk0NQ==
created_at: 2020-06-07T20:19:15Z
updated_at: 2020-06-07T20:19:15Z
author_association: OWNER
body:
  I'm going to add a `test_permissions.py` module that checks for 403 errors against different patterns of the `actors` block at different levels in `metadata.json`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640274171
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDI3NDE3MQ==
created_at: 2020-06-07T20:21:14Z
updated_at: 2020-06-07T20:21:14Z
author_association: OWNER
body:
  Next step: fix this
  ```
  -            # TODO: fix this to use that permission check
  -            if not actor_matches_allow(
  -                request.scope.get("actor", None), metadata.get("allow")
  -            ):
  -                return Response("Permission denied", status=403)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/801#issuecomment-640277557
issue_url: https://api.github.com/repos/simonw/datasette/issues/801
node_id: MDEyOklzc3VlQ29tbWVudDY0MDI3NzU1Nw==
created_at: 2020-06-07T20:48:00Z
updated_at: 2020-06-07T20:48:00Z
author_association: OWNER
body:
  Now that I'm expanding permission checks to everything else too (#811), not just canned queries, I think it makes sense to re-prioritize this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/801#issuecomment-640277775
issue_url: https://api.github.com/repos/simonw/datasette/issues/801
node_id: MDEyOklzc3VlQ29tbWVudDY0MDI3Nzc3NQ==
created_at: 2020-06-07T20:49:40Z
updated_at: 2020-06-07T20:49:40Z
author_association: OWNER
body:
  I'm going to pass the entire actor object as a dictionary of available named query parameters. So if the actor looks like this:
  ```json
  {
      "id": "simonw",
      "roles": ["staff", "developer"]
  }
  ```
  Then the SQL query will be called like this:

  ```python
  conn.execute(sql, {
    "id": "simonw",
    "roles: '["staff", "developer"]',
  })
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/395#issuecomment-640280741
issue_url: https://api.github.com/repos/simonw/datasette/issues/395
node_id: MDEyOklzc3VlQ29tbWVudDY0MDI4MDc0MQ==
created_at: 2020-06-07T21:12:57Z
updated_at: 2020-06-07T21:12:57Z
author_association: OWNER
body:
  This is a pattern I like:
  ```python
      with make_app_client(
          template_dir=str(pathlib.Path(__file__).parent / "test_templates")
      ) as client:
          response = client.get("/-/metadata")
          assert response.status == 200
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640287967
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDI4Nzk2Nw==
created_at: 2020-06-07T22:16:10Z
updated_at: 2020-06-07T22:16:10Z
author_association: OWNER
body:
  The tests in test_permissions.py could check the .json variants and assert that permission checks were carried out too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640337951
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDMzNzk1MQ==
created_at: 2020-06-08T03:11:58Z
updated_at: 2020-06-08T03:11:58Z
author_association: OWNER
body:
  I'd like to be able to apply permissions for the ability to run a SQL query - but I'm not sure where the best place for that `"allow"` block to live would be.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640338151
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDMzODE1MQ==
created_at: 2020-06-08T03:12:41Z
updated_at: 2020-06-08T03:12:41Z
author_association: OWNER
body:
  Also need to expand the docs on https://datasette.readthedocs.io/en/latest/authentication.html to explain where you can put `allow` blocks to control access to the instance, database or table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640338347
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDMzODM0Nw==
created_at: 2020-06-08T03:13:23Z
updated_at: 2020-06-08T03:13:23Z
author_association: OWNER
body:
  Do row-level permissions even make sense? Might be a good idea to remove those until I have a good use-case for them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/801#issuecomment-640339117
issue_url: https://api.github.com/repos/simonw/datasette/issues/801
node_id: MDEyOklzc3VlQ29tbWVudDY0MDMzOTExNw==
created_at: 2020-06-08T03:16:16Z
updated_at: 2020-06-08T03:16:16Z
author_association: OWNER
body:
  I'm going to call this key `"allow_by_query"` - I think I need `allow_sql` for something else (for configuring if users are allowed to execute arbitrary SQL queries).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640339674
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDMzOTY3NA==
created_at: 2020-06-08T03:18:15Z
updated_at: 2020-06-08T03:18:15Z
author_association: OWNER
body:
  I should take these permissions into account when displaying a list of tables or a list of databases (like I do right now when displaying a list of queries).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/801#issuecomment-640339828
issue_url: https://api.github.com/repos/simonw/datasette/issues/801
node_id: MDEyOklzc3VlQ29tbWVudDY0MDMzOTgyOA==
created_at: 2020-06-08T03:18:47Z
updated_at: 2020-06-08T03:18:47Z
author_association: OWNER
body:
  Example. This will only allow users to access the `fixtures` database if the logged-in actor's ID value appears for a record in the `users` table which has `admin` = 1.
  ```json
  {
      "databases": {
          "fixtures": {
              "allow_by_query": "select * from users where id = :id and admin = 1"
          }
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640344950
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDM0NDk1MA==
created_at: 2020-06-08T03:36:49Z
updated_at: 2020-06-08T03:36:49Z
author_association: OWNER
body:
  Oh this is a bit awkward - should I be running per-table permission checks for every table that might be shown on the index page?

  <img width="841" alt="Datasette__fixtures__data" src="https://user-images.githubusercontent.com/9599/83990459-9862e900-a8fe-11ea-8a42-60cc3d6f7693.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640345115
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDM0NTExNQ==
created_at: 2020-06-08T03:37:33Z
updated_at: 2020-06-08T03:37:33Z
author_association: OWNER
body:
  Per-table permissions is pretty interesting for large installations though - an organization might have hundreds of CSV files imported into Datasette and then allow users to specify which exact users within that organization are allowed to see which CSV.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640348785
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDM0ODc4NQ==
created_at: 2020-06-08T03:51:50Z
updated_at: 2020-06-08T03:51:50Z
author_association: OWNER
body:
  New convention: the 🔒 icon is now shown next to resources that are private - that are visible to you now, but would not be visible to the anonymous user.

  <img width="823" alt="Datasette__fixtures__data" src="https://user-images.githubusercontent.com/9599/83991127-b3cef380-a900-11ea-9df3-8fcd4c10cc42.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640362879
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDM2Mjg3OQ==
created_at: 2020-06-08T04:42:28Z
updated_at: 2020-06-08T13:39:46Z
author_association: OWNER
body:
  I'm finding myself repeating this pattern a lot:
  ```python
          for table in table_counts:
              allowed = await self.ds.permission_allowed(
                  request.scope.get("actor"),
                  "view-table",
                  resource_type="table",
                  resource_identifier=(database, table),
                  default=True,
              )
              if not allowed:
                  continue
              private = not await self.ds.permission_allowed(
                  None,
                  "view-table",
                  resource_type="table",
                  resource_identifier=(database, table),
              )
  ```
  I use a similar pattern for lists of databases and lists of queries, and I'll be doing the same thing for lists of SQL views too.

  An abstraction around this would be useful.

  Idea:

  ```python
  visible, private = await check_visibility(
      self.ds, actor, "view-table", "table", (database, table)
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640365512
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDM2NTUxMg==
created_at: 2020-06-08T04:53:49Z
updated_at: 2020-06-08T04:53:49Z
author_association: OWNER
body:
  I really like the padlocks. I should include a screenshot in the documentation that illustrates them.

  <img width="620" alt="data" src="https://user-images.githubusercontent.com/9599/83993771-483d5400-a909-11ea-8e7b-85b54e40f900.png">

  Maybe I should figure out a way to have the https://latest.datasette.io/ demo illustrate both a logged-in and a logged-out state.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/811#issuecomment-640367128
issue_url: https://api.github.com/repos/simonw/datasette/issues/811
node_id: MDEyOklzc3VlQ29tbWVudDY0MDM2NzEyOA==
created_at: 2020-06-08T05:00:13Z
updated_at: 2020-06-08T05:00:49Z
author_association: OWNER
body:
  Should the padlock show up on tables that are private only because they inherited their privacy from their parent database or even the parent instance?

  Interesting question. If an instance is private, I'm not sure it makes sense to show padlocks on absolutely everything.

  Likewise, a list of tables shown on the database table with a padlock next to every single table (when the database itself is private) doesn't seem to add any useful information.

  I think "Show 🔒 in header on private database page" will resolve this for me. I'll always show the padlock in the header of a database/table page even if that privacy is inherited - but I won't do that for padlocks shown in the list of tables or list of databases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/814#issuecomment-640638057
issue_url: https://api.github.com/repos/simonw/datasette/issues/814
node_id: MDEyOklzc3VlQ29tbWVudDY0MDYzODA1Nw==
created_at: 2020-06-08T14:11:51Z
updated_at: 2020-06-08T14:12:12Z
author_association: OWNER
body:
  The only impact it has at all is on this code here:
  https://github.com/simonw/datasette/blob/cc218fa9be55842656d030545c308392e3736053/datasette/views/base.py#L515-L527

  That `ds.cache_headers` property looks like it needs rethinking too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/815#issuecomment-640656143
issue_url: https://api.github.com/repos/simonw/datasette/issues/815
node_id: MDEyOklzc3VlQ29tbWVudDY0MDY1NjE0Mw==
created_at: 2020-06-08T14:25:48Z
updated_at: 2020-06-08T14:26:45Z
author_association: OWNER
body:
  Will we need a request correlation ID for this? Multiple asyncio threads can write things to the `ds._permission_checks` deque at the same time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/815#issuecomment-640671241
issue_url: https://api.github.com/repos/simonw/datasette/issues/815
node_id: MDEyOklzc3VlQ29tbWVudDY0MDY3MTI0MQ==
created_at: 2020-06-08T14:38:04Z
updated_at: 2020-06-08T14:38:04Z
author_association: OWNER
body:
  Alternative to a correlation ID would be to use the existing `AsgiTracer` / `capture_traces` mechanism. That's probably smarter.

  It could even start logging SQL queries to an in-memory deque too, so a debug tool could show you queries executed by other requests!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/815#issuecomment-640672540
issue_url: https://api.github.com/repos/simonw/datasette/issues/815
node_id: MDEyOklzc3VlQ29tbWVudDY0MDY3MjU0MA==
created_at: 2020-06-08T14:40:22Z
updated_at: 2020-06-08T14:40:22Z
author_association: OWNER
body:
  Here's the current tracer mechanism. Note that it captures a stacktrace (which is expensive) - but only if the tracer system has been enabled for a request.

  https://github.com/simonw/datasette/blob/1c063fae9dba70f70244db010d55a18846640f07/datasette/tracer.py#L27-L51

  For permissions checks I want to ALWAYS track those calls, not just on requests that have opted in.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/815#issuecomment-640673405
issue_url: https://api.github.com/repos/simonw/datasette/issues/815
node_id: MDEyOklzc3VlQ29tbWVudDY0MDY3MzQwNQ==
created_at: 2020-06-08T14:41:55Z
updated_at: 2020-06-08T14:41:55Z
author_association: OWNER
body:
  I want to be able to display the HTTP path and verb - `GET /fixtures`, `POST /fixtures/myquery` etc. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/816#issuecomment-640815550
issue_url: https://api.github.com/repos/simonw/datasette/issues/816
node_id: MDEyOklzc3VlQ29tbWVudDY0MDgxNTU1MA==
created_at: 2020-06-08T19:06:44Z
updated_at: 2020-06-08T19:06:44Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/c7d145e016522dd6ee229d4d0b3ba79a7a8877c1/docs/plugins.rst#extra_template_varstemplate-database-table-view_name-request-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/813#issuecomment-640831842
issue_url: https://api.github.com/repos/simonw/datasette/issues/813
node_id: MDEyOklzc3VlQ29tbWVudDY0MDgzMTg0Mg==
created_at: 2020-06-08T19:27:47Z
updated_at: 2020-06-08T19:27:47Z
author_association: OWNER
body:
  This needs to be ready for Datasette 0.44 because without it the "view-table" permission is useless - it will protect the https://latest.datasette.io/fixtures/facetable page but will not prevent users from executing https://latest.datasette.io/fixtures?sql=select+*+from+facetable

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/813#issuecomment-640837908
issue_url: https://api.github.com/repos/simonw/datasette/issues/813
node_id: MDEyOklzc3VlQ29tbWVudDY0MDgzNzkwOA==
created_at: 2020-06-08T19:33:03Z
updated_at: 2020-06-08T19:33:03Z
author_association: OWNER
body:
  Don't forget to link to the `allow_sql` docs from the warning block here: https://github.com/simonw/datasette/blob/54370853828bdf87ca844fd0fc00900e0e2e659d/docs/authentication.rst#controlling-access-to-specific-tables-and-views

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/813#issuecomment-640916290
issue_url: https://api.github.com/repos/simonw/datasette/issues/813
node_id: MDEyOklzc3VlQ29tbWVudDY0MDkxNjI5MA==
created_at: 2020-06-08T22:16:39Z
updated_at: 2020-06-08T22:17:32Z
author_association: OWNER
body:
  Naming problem: Datasette already has a config option with this name:

      $ datasette serve data.db --config allow_sql:1

  https://datasette.readthedocs.io/en/stable/config.html#allow-sql

  It's confusing to have two things called `allow_sql` that do slightly different things.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/813#issuecomment-640916807
issue_url: https://api.github.com/repos/simonw/datasette/issues/813
node_id: MDEyOklzc3VlQ29tbWVudDY0MDkxNjgwNw==
created_at: 2020-06-08T22:18:09Z
updated_at: 2020-06-08T22:18:09Z
author_association: OWNER
body:
  I could retire the `--config allow_sql:0` option entirely, since the new `metadata.json` mechanism can be used to achieve the exact same thing.

  I'm going to do that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/806#issuecomment-640916991
issue_url: https://api.github.com/repos/simonw/datasette/issues/806
node_id: MDEyOklzc3VlQ29tbWVudDY0MDkxNjk5MQ==
created_at: 2020-06-08T22:18:45Z
updated_at: 2020-06-08T22:18:45Z
author_association: OWNER
body:
  Reminder for release notes: I removed `--config allow_sql:0` - see https://github.com/simonw/datasette/issues/813#issuecomment-640916807

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/493#issuecomment-640924482
issue_url: https://api.github.com/repos/simonw/datasette/issues/493
node_id: MDEyOklzc3VlQ29tbWVudDY0MDkyNDQ4Mg==
created_at: 2020-06-08T22:39:45Z
updated_at: 2020-06-08T22:39:45Z
author_association: OWNER
body:
  I'm definitely doing this rename, now that `metadata.json` is used for `allow` permissions configuration as well as-of #811.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/818#issuecomment-640929693
issue_url: https://api.github.com/repos/simonw/datasette/issues/818
node_id: MDEyOklzc3VlQ29tbWVudDY0MDkyOTY5Mw==
created_at: 2020-06-08T22:56:38Z
updated_at: 2020-06-08T22:56:38Z
author_association: OWNER
body:
  https://datasette.readthedocs.io/en/latest/plugins.html#permission-allowed-datasette-actor-action-resource has a couple of examples now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/813#issuecomment-640951947
issue_url: https://api.github.com/repos/simonw/datasette/issues/813
node_id: MDEyOklzc3VlQ29tbWVudDY0MDk1MTk0Nw==
created_at: 2020-06-09T00:09:56Z
updated_at: 2020-06-09T00:09:56Z
author_association: OWNER
body:
  Documentation: https://datasette.readthedocs.io/en/latest/authentication.html#controlling-the-ability-to-execute-arbitrary-sql

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/777#issuecomment-640955788
issue_url: https://api.github.com/repos/simonw/datasette/issues/777
node_id: MDEyOklzc3VlQ29tbWVudDY0MDk1NTc4OA==
created_at: 2020-06-09T00:23:26Z
updated_at: 2020-06-09T00:23:57Z
author_association: OWNER
body:
  Clue: https://latest.datasette.io/404 displays correctly but https://latest.datasette.io/fixtures/404 does not.

  That's because `<link rel="stylesheet" href="-/static/app.css?">` does the correct thing if you are on the root of the site but not if you are in a sub-directory.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/777#issuecomment-640957423
issue_url: https://api.github.com/repos/simonw/datasette/issues/777
node_id: MDEyOklzc3VlQ29tbWVudDY0MDk1NzQyMw==
created_at: 2020-06-09T00:29:03Z
updated_at: 2020-06-09T00:29:03Z
author_association: OWNER
body:
  Here's why:

  https://github.com/simonw/datasette/blob/49d6d2f7b0f6cb02e25022e1c9403811f1fa0a7c/datasette/app.py#L1024-L1029

  404 errors are rendered by looking for a template from `["404.html", "500.html"]`.

  `404.html` doesn't actually ship with Datasette (plugins or custom template directories can provide it). So the `500.html` template is used.

  That template extends `base.html`, which expects there to be `base_url` and `app_css_hash` variables. But as you can see in the excerpt above, those variables are not being passed to the template context when the error page is rendered.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/215#issuecomment-640960553
issue_url: https://api.github.com/repos/simonw/datasette/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDY0MDk2MDU1Mw==
created_at: 2020-06-09T00:41:09Z
updated_at: 2020-06-09T00:41:09Z
author_association: OWNER
body:
  I'm going to imitate `register_output_renderer` and `register_facet_classes` - both return a list of things to register.

  So I'll do this:

  ```python
  @hookspec
  def register_routes():
      "Register URL routes. Return a list of (regex, view_function) pairs"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/215#issuecomment-640971470
issue_url: https://api.github.com/repos/simonw/datasette/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDY0MDk3MTQ3MA==
created_at: 2020-06-09T01:19:44Z
updated_at: 2020-06-09T01:19:44Z
author_association: OWNER
body:
  I'll need to add documentation of the `Response` object (and `Response.html()` and `Response.text()` class methods - I should add `Response.json()` too) to the internals page https://datasette.readthedocs.io/en/stable/internals.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/215#issuecomment-640972952
issue_url: https://api.github.com/repos/simonw/datasette/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDY0MDk3Mjk1Mg==
created_at: 2020-06-09T01:24:52Z
updated_at: 2020-06-09T01:25:33Z
author_association: OWNER
body:
  WIP documentation: https://github.com/simonw/datasette/blob/770dedb21adfc706592e6b5cdf5e751a8720fdf9/docs/plugins.rst#register_routes

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/820#issuecomment-640982533
issue_url: https://api.github.com/repos/simonw/datasette/issues/820
node_id: MDEyOklzc3VlQ29tbWVudDY0MDk4MjUzMw==
created_at: 2020-06-09T02:00:21Z
updated_at: 2020-06-09T02:00:21Z
author_association: OWNER
body:
  In the case of registering API tokens it would be useful if the plugin could call a writable canned query which knows how to insert a randomly generated value. This could be achieved using a custom SQL function.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/822#issuecomment-641003237
issue_url: https://api.github.com/repos/simonw/datasette/issues/822
node_id: MDEyOklzc3VlQ29tbWVudDY0MTAwMzIzNw==
created_at: 2020-06-09T03:17:32Z
updated_at: 2020-06-09T03:17:32Z
author_association: OWNER
body:
  Currently querystring parameters are accessed through `request.args` and POST variables through `request.post_vars()`. Would be good to have a name that was somewhat consistent with those.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/818#issuecomment-641013524
issue_url: https://api.github.com/repos/simonw/datasette/issues/818
node_id: MDEyOklzc3VlQ29tbWVudDY0MTAxMzUyNA==
created_at: 2020-06-09T03:57:38Z
updated_at: 2020-06-09T04:00:24Z
author_association: OWNER
body:
  Problem with that is it's more of a `actor_from_request` opportunity than `permission_allowed`. You could use `actor_from_request` to authenticate API clients from their `Authorization:` header, then use the regular `"allow"` blocks in `metadata.json` to actually assign their permissions.

  The most interesting permissions plugin would be one that implements permissions against some kind of database schema, hence allowing admins to edit permissions through writable canned queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/805#issuecomment-641017721
issue_url: https://api.github.com/repos/simonw/datasette/issues/805
node_id: MDEyOklzc3VlQ29tbWVudDY0MTAxNzcyMQ==
created_at: 2020-06-09T04:16:28Z
updated_at: 2020-06-09T04:16:28Z
author_association: OWNER
body:
  Create `data.db` with:
  ```
  echo '{"emoji": "🐯", "score": 0}' | sqlite-utils insert data.db emojis --pk=emoji -
  echo '{"emoji": "🐺", "score": 0}' | sqlite-utils insert data.db emojis --pk=emoji -
  ```
  Then run Datasette with this `metadata.yaml`:
  ```yaml
  title: Datasette Poll
  databases:
    data:
      queries:
        vote:
          sql: |-
            update emojis set score = score + 1 where emoji = :emoji
          write: true
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/805#issuecomment-641017851
issue_url: https://api.github.com/repos/simonw/datasette/issues/805
node_id: MDEyOklzc3VlQ29tbWVudDY0MTAxNzg1MQ==
created_at: 2020-06-09T04:17:00Z
updated_at: 2020-06-09T04:17:00Z
author_association: OWNER
body:
  I can't get Datasette working on Glitch installed from a URL - I'm going to try this on Glitch once I've shipped the 0.44 release in #806.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/823#issuecomment-641025760
issue_url: https://api.github.com/repos/simonw/datasette/issues/823
node_id: MDEyOklzc3VlQ29tbWVudDY0MTAyNTc2MA==
created_at: 2020-06-09T04:48:40Z
updated_at: 2020-06-09T04:48:40Z
author_association: OWNER
body:
  I should assert that `"id"` exists and is a string in the code that calls the `actor_from_request` hook.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/806#issuecomment-641026726
issue_url: https://api.github.com/repos/simonw/datasette/issues/806
node_id: MDEyOklzc3VlQ29tbWVudDY0MTAyNjcyNg==
created_at: 2020-06-09T04:52:07Z
updated_at: 2020-06-09T04:52:07Z
author_association: OWNER
body:
  Changelog for this is going to be huge - 96 commits since 0.43 already! https://github.com/simonw/datasette/compare/0.43...master

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/823#issuecomment-641059221
issue_url: https://api.github.com/repos/simonw/datasette/issues/823
node_id: MDEyOklzc3VlQ29tbWVudDY0MTA1OTIyMQ==
created_at: 2020-06-09T06:23:51Z
updated_at: 2020-06-09T06:24:09Z
author_association: OWNER
body:
  I don't like the "id" requirement.

  I can think of plenty of situations where a unique ID might not be available:

  - auth against an external token - an email address or a phone number for example
  - auth using encrypted tokens - where decrypting the token tells you exactly what permissions that token should have, like in https://blog.thea.codes/building-a-stateless-api-proxy/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/825#issuecomment-641062164
issue_url: https://api.github.com/repos/simonw/datasette/issues/825
node_id: MDEyOklzc3VlQ29tbWVudDY0MTA2MjE2NA==
created_at: 2020-06-09T06:30:24Z
updated_at: 2020-06-09T14:05:33Z
author_association: OWNER
body:
  Idea: the anonymous actor could be passed to `actor_matches_allow()` as:

  ```json
  {"anonymous": true}
  ```

  Then allow blocks like this could be used to allow them:

  ```json
  {
      "plugins": {
          "datasette-upload-csvs": {
              "allow": {
                   "anonymous": true
              }
          }
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/825#issuecomment-641320947
issue_url: https://api.github.com/repos/simonw/datasette/issues/825
node_id: MDEyOklzc3VlQ29tbWVudDY0MTMyMDk0Nw==
created_at: 2020-06-09T14:06:46Z
updated_at: 2020-06-09T14:06:46Z
author_association: OWNER
body:
  I'm torn between `anonymous` and `anon` - because the latter is less typing, and I envisage people writing a lot of code like this:

  ```python
      if actor.get("anonymous"):
          # ...
  ```

  I'm going with `anonymous` because it's that tiny bit clearer than `anon`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/823#issuecomment-641353186
issue_url: https://api.github.com/repos/simonw/datasette/issues/823
node_id: MDEyOklzc3VlQ29tbWVudDY0MTM1MzE4Ng==
created_at: 2020-06-09T14:58:36Z
updated_at: 2020-06-09T14:58:36Z
author_association: OWNER
body:
  Docs now say:
  >  The actor dictionary can be any shape - the design of that data structure is left up to the plugins. A useful convention is to include an `"id"` string, as demonstrated by the "root" actor below.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/812#issuecomment-641353729
issue_url: https://api.github.com/repos/simonw/datasette/issues/812
node_id: MDEyOklzc3VlQ29tbWVudDY0MTM1MzcyOQ==
created_at: 2020-06-09T14:59:25Z
updated_at: 2020-06-09T14:59:25Z
author_association: OWNER
body:
  I'm going to figure this out by working with https://github.com/simonw/datasette-auth-github/issues/62

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/826#issuecomment-641359103
issue_url: https://api.github.com/repos/simonw/datasette/issues/826
node_id: MDEyOklzc3VlQ29tbWVudDY0MTM1OTEwMw==
created_at: 2020-06-09T15:08:07Z
updated_at: 2020-06-09T15:10:33Z
author_association: OWNER
body:
  I should probably add a utility function for setting that cookie - right now the only code that does that is here:

  https://github.com/simonw/datasette/blob/dfff34e1987976e72f58ee7b274952840b1f4b71/datasette/views/special.py#L63-L76

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/826#issuecomment-641360187
issue_url: https://api.github.com/repos/simonw/datasette/issues/826
node_id: MDEyOklzc3VlQ29tbWVudDY0MTM2MDE4Nw==
created_at: 2020-06-09T15:10:00Z
updated_at: 2020-06-09T15:11:24Z
author_association: OWNER
body:
  Also a good reminder that I need a `set_cookie()` function (#795) so I don't have to mess around with `SimpleCookie` directly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/795#issuecomment-641361311
issue_url: https://api.github.com/repos/simonw/datasette/issues/795
node_id: MDEyOklzc3VlQ29tbWVudDY0MTM2MTMxMQ==
created_at: 2020-06-09T15:11:50Z
updated_at: 2020-06-09T15:11:50Z
author_association: OWNER
body:
  Also: https://github.com/simonw/datasette/blob/dfff34e1987976e72f58ee7b274952840b1f4b71/datasette/views/special.py#L63-L76

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/825#issuecomment-641406944
issue_url: https://api.github.com/repos/simonw/datasette/issues/825
node_id: MDEyOklzc3VlQ29tbWVudDY0MTQwNjk0NA==
created_at: 2020-06-09T16:12:02Z
updated_at: 2020-06-09T17:19:19Z
author_association: OWNER
body:
  Alternative design: leave actor alone. Instead specify that allow blocks can look like this:

  ```json
  {
      "allow": {
          "unauthenticated": true
      }
  }
  ```

  I like this: the above block is very self-documenting.

  The `"id": "*"` mechanism means there is already precedent for allow keys with special meaning.

  **I'm going with this design.**

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/825#issuecomment-641412424
issue_url: https://api.github.com/repos/simonw/datasette/issues/825
node_id: MDEyOklzc3VlQ29tbWVudDY0MTQxMjQyNA==
created_at: 2020-06-09T16:22:07Z
updated_at: 2020-06-09T16:22:07Z
author_association: OWNER
body:
  When I implement this I should also document default allow vs default deny as a concept, and specify that default next to every documented permission.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/825#issuecomment-641449725
issue_url: https://api.github.com/repos/simonw/datasette/issues/825
node_id: MDEyOklzc3VlQ29tbWVudDY0MTQ0OTcyNQ==
created_at: 2020-06-09T17:02:31Z
updated_at: 2020-06-09T17:02:31Z
author_association: OWNER
body:
  Documented at the bottom of this section: https://github.com/simonw/datasette/blob/7633b9ab249b2dce5ee0b4fcf9542c13a1703ef0/docs/authentication.rst#defining-permissions-with-allow-blocks

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/825#issuecomment-641452563
issue_url: https://api.github.com/repos/simonw/datasette/issues/825
node_id: MDEyOklzc3VlQ29tbWVudDY0MTQ1MjU2Mw==
created_at: 2020-06-09T17:08:00Z
updated_at: 2020-06-09T17:08:00Z
author_association: OWNER
body:
  https://datasette.readthedocs.io/en/latest/authentication.html#defining-permissions-with-allow-blocks

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/804#issuecomment-641538799
issue_url: https://api.github.com/repos/simonw/datasette/issues/804
node_id: MDEyOklzc3VlQ29tbWVudDY0MTUzODc5OQ==
created_at: 2020-06-09T20:01:08Z
updated_at: 2020-06-09T20:01:08Z
author_association: OWNER
body:
      $ python tests/fixtures.py fixtures.db fixtures-metadata.json fixtures-plugins
      Test tables written to fixtures.db
      - metadata written to fixtures-metadata.json
      Wrote plugin: fixtures-plugins/register_output_renderer.py
      Wrote plugin: fixtures-plugins/view_name.py
      Wrote plugin: fixtures-plugins/my_plugin.py
      Wrote plugin: fixtures-plugins/messages_output_renderer.py
      Wrote plugin: fixtures-plugins/my_plugin_2.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/804#issuecomment-641538982
issue_url: https://api.github.com/repos/simonw/datasette/issues/804
node_id: MDEyOklzc3VlQ29tbWVudDY0MTUzODk4Mg==
created_at: 2020-06-09T20:01:30Z
updated_at: 2020-06-09T20:01:30Z
author_association: OWNER
body:
  Now fully documented here: https://datasette.readthedocs.io/en/latest/contributing.html#setting-up-a-development-environment

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/806#issuecomment-641602794
issue_url: https://api.github.com/repos/simonw/datasette/issues/806
node_id: MDEyOklzc3VlQ29tbWVudDY0MTYwMjc5NA==
created_at: 2020-06-09T21:55:45Z
updated_at: 2020-06-09T21:55:45Z
author_association: OWNER
body:
  Last-minute thought:

  Should I worry about calling permissions "actions", when I have an idea for a future plugin hook that allows plugins to add something I was going to call "actions" to database, table and row pages?

  Those actions would take the form of menu item commands that Do Something to the selected object. If I use "actions" to mean permission names, will I be able to find a good alternative name for these dynamic menu items?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/806#issuecomment-641604210
issue_url: https://api.github.com/repos/simonw/datasette/issues/806
node_id: MDEyOklzc3VlQ29tbWVudDY0MTYwNDIxMA==
created_at: 2020-06-09T21:59:33Z
updated_at: 2020-06-09T22:00:11Z
author_association: OWNER
body:
  AWS IAM uses action and resource terminology: https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction_access-management.html - I think that's where I got that language:

  > ```json
  > {
  >   "Version": "2012-10-17",
  >   "Statement": {
  >     "Effect": "Allow",
  >     "Action": "dynamodb:*",
  >     "Resource": "arn:aws:dynamodb:us-east-2:123456789012:table/Books"
  >   }
  > }
  > ```
  I'm going to stick with "action" in its current meaning.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/795#issuecomment-641616185
issue_url: https://api.github.com/repos/simonw/datasette/issues/795
node_id: MDEyOklzc3VlQ29tbWVudDY0MTYxNjE4NQ==
created_at: 2020-06-09T22:33:33Z
updated_at: 2020-06-09T22:33:33Z
author_association: OWNER
body:
  Documentation: https://datasette.readthedocs.io/en/latest/internals.html#setting-cookies-with-response-set-cookie

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/806#issuecomment-641634749
issue_url: https://api.github.com/repos/simonw/datasette/issues/806
node_id: MDEyOklzc3VlQ29tbWVudDY0MTYzNDc0OQ==
created_at: 2020-06-09T23:34:52Z
updated_at: 2020-06-09T23:34:52Z
author_association: OWNER
body:
  Preview of the release notes is now available here: https://datasette.readthedocs.io/en/latest/changelog.html#v0-44

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/806#issuecomment-641637696
issue_url: https://api.github.com/repos/simonw/datasette/issues/806
node_id: MDEyOklzc3VlQ29tbWVudDY0MTYzNzY5Ng==
created_at: 2020-06-09T23:46:00Z
updated_at: 2020-06-09T23:46:00Z
author_association: OWNER
body:
  The issues that should be referenced from this release are:

  #395, #519, #576, #699, #706, #774, #777, #781, #784, #788, #790, #797, #798, #800, #802, #804, #819, #822 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/828#issuecomment-641713087
issue_url: https://api.github.com/repos/simonw/datasette/issues/828
node_id: MDEyOklzc3VlQ29tbWVudDY0MTcxMzA4Nw==
created_at: 2020-06-10T04:28:17Z
updated_at: 2020-06-10T04:28:17Z
author_association: OWNER
body:
  Fixed. https://datasette.readthedocs.io/en/latest/changelog.html

  <img width="473" alt="Changelog_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/84226994-20c8c180-aa98-11ea-8b28-3d00d1254d9a.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-641889565
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDY0MTg4OTU2NQ==
created_at: 2020-06-10T09:49:34Z
updated_at: 2020-06-10T09:49:34Z
author_association: NONE
body:
  Hi,

  I came across this issue while looking for a way to spawn Datasette as a SQLite files viewer in JupyterLab. I found https://github.com/simonw/jupyterserverproxy-datasette-demo which seems to be the most up to date proof of concept, but it seems to be failing to list the available db (at least in the Binder demo, https://hub.gke.mybinder.org/user/simonw-jupyters--datasette-demo-uw4dmlnn/datasette/, I only have `:memory`).

  Does anyone tried to improve on this proof of concept to have a Datasette visualization for SQLite files?

  Thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-641908346
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDY0MTkwODM0Ng==
created_at: 2020-06-10T10:22:54Z
updated_at: 2020-06-10T10:22:54Z
author_association: CONTRIBUTOR
body:
  There's a working demo here: https://github.com/wragge/datasette-test

  And if you want something that's more than just proof-of-concept, here's a notebook which does some harvesting from web archives and then displays the results using Datasette: https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/explore_presentations.ipynb

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/829#issuecomment-642161210
issue_url: https://api.github.com/repos/simonw/datasette/issues/829
node_id: MDEyOklzc3VlQ29tbWVudDY0MjE2MTIxMA==
created_at: 2020-06-10T17:45:58Z
updated_at: 2020-06-10T17:45:58Z
author_association: OWNER
body:
  `itsdangerous` has this ability but you specify the max-age when you call unsign: https://itsdangerous.palletsprojects.com/en/1.1.x/timed/

  >     s.unsign(string, max_age=5)
  >     Traceback (most recent call last):
  >       ...
  >     itsdangerous.exc.SignatureExpired: Signature age 15 > 5 seconds

  I currently only decode the `ds_actor` cookie in one place: https://github.com/simonw/datasette/blob/d828abaddec0dce3ec4b4eeddc3a74384e52cf34/datasette/actor_auth_cookie.py#L5-L12

  If plugins want to be able to set their own policies on how long the `ds_actor` cookie should remain valid, how do I know to listen to them when decoding the cookie here?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/829#issuecomment-642174272
issue_url: https://api.github.com/repos/simonw/datasette/issues/829
node_id: MDEyOklzc3VlQ29tbWVudDY0MjE3NDI3Mg==
created_at: 2020-06-10T18:10:13Z
updated_at: 2020-06-10T18:10:13Z
author_association: OWNER
body:
  Some options:

  - Redesign the `ds_actor` cookie to be `{"expires_at": 1591811250, "actor": ...}` - check if it has expired in that default `actor_from_request` hook
  - Let plugins set an additional cookie of some sort
  - Expect plugins that care about this to set a cookie with a different name and implement their own `actor_from_request` against that

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/829#issuecomment-642175892
issue_url: https://api.github.com/repos/simonw/datasette/issues/829
node_id: MDEyOklzc3VlQ29tbWVudDY0MjE3NTg5Mg==
created_at: 2020-06-10T18:13:26Z
updated_at: 2020-06-10T18:13:26Z
author_association: OWNER
body:
  I'm going with `expires_at` - except to keep the cookies shorter the key will be called `e` and the actor will go in `a`, like this:

  ```json
  {
      "e": "1UuHoo",
      "a": {"id": "root"}
  }
  ```
  That `e` value is a base64 encoded expiry integer timestamp (again for a shorter cookie) - using https://pypi.org/project/python-baseconv/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/829#issuecomment-642178604
issue_url: https://api.github.com/repos/simonw/datasette/issues/829
node_id: MDEyOklzc3VlQ29tbWVudDY0MjE3ODYwNA==
created_at: 2020-06-10T18:18:36Z
updated_at: 2020-06-10T18:20:19Z
author_association: OWNER
body:
  Even shorter: encode an integer that is the difference between that expiry timestamp and a more recent epoch - June 1st 2020 will do.

  ```
  >>> import datetime, calendar
  >>> calendar.timegm(datetime.date(2020, 6, 1).timetuple())
  1590969600
  >>> import baseconv
  >>> baseconv.base62.encode(int(time.time() - 1590969600))
  '3XST'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/829#issuecomment-642217520
issue_url: https://api.github.com/repos/simonw/datasette/issues/829
node_id: MDEyOklzc3VlQ29tbWVudDY0MjIxNzUyMA==
created_at: 2020-06-10T19:41:35Z
updated_at: 2020-06-10T19:41:35Z
author_association: OWNER
body:
  I didn't bother with the alternative epoch - it only shaves off two or three bytes from the cookie.

  Documentation for the new `ds_actor` cookie shape is here: https://datasette.readthedocs.io/en/latest/authentication.html#the-ds-actor-cookie

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/818#issuecomment-642229899
issue_url: https://api.github.com/repos/simonw/datasette/issues/818
node_id: MDEyOklzc3VlQ29tbWVudDY0MjIyOTg5OQ==
created_at: 2020-06-10T20:07:36Z
updated_at: 2020-06-10T20:07:36Z
author_association: OWNER
body:
  New policy in 9f236c4 dictates that this should be in Milestone 0.44 after all:

  >  * **New plugin hooks** should only be shipped if accompanied by a separate release of a non-demo plugin that uses them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/818#issuecomment-642230499
issue_url: https://api.github.com/repos/simonw/datasette/issues/818
node_id: MDEyOklzc3VlQ29tbWVudDY0MjIzMDQ5OQ==
created_at: 2020-06-10T20:08:46Z
updated_at: 2020-06-10T20:09:26Z
author_association: OWNER
body:
  What's a simple but useful plugin I could release that exercises this hook?

  Ideally one which executes permission checks against the database somehow.

  I could do a simplest-possible implementation of the idea in #801 (allow-by-query).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/818#issuecomment-642231871
issue_url: https://api.github.com/repos/simonw/datasette/issues/818
node_id: MDEyOklzc3VlQ29tbWVudDY0MjIzMTg3MQ==
created_at: 2020-06-10T20:11:50Z
updated_at: 2020-06-10T20:11:50Z
author_association: OWNER
body:
  `datasette-permissions-sql`
  ```yaml
  plugins:
    datasette-permissions-sql:
      view-instance: |-
        select count(*) from users where admin = 1 and id = :id
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/831#issuecomment-642324847
issue_url: https://api.github.com/repos/simonw/datasette/issues/831
node_id: MDEyOklzc3VlQ29tbWVudDY0MjMyNDg0Nw==
created_at: 2020-06-10T23:50:55Z
updated_at: 2020-06-10T23:50:55Z
author_association: OWNER
body:
  Actually I'm not sure about this. If `"allow": null` means "no-one can do this", what's the allow block syntax for "everyone can do this"?

  It could be `"allow": {}` - but that's not intuitive because normally the allow block shows keys that need to match. `{}` suggests to me that no matches are possible.

  So I think I'm going to stick with the current mechanism, which is that `"allow": null` means "anyone can do this" and `"allow": {}` means "no-one can do this".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/832#issuecomment-642412017
issue_url: https://api.github.com/repos/simonw/datasette/issues/832
node_id: MDEyOklzc3VlQ29tbWVudDY0MjQxMjAxNw==
created_at: 2020-06-11T05:13:59Z
updated_at: 2020-06-11T05:13:59Z
author_association: OWNER
body:
  Relevant code:

  https://github.com/simonw/datasette/blob/ce4958018ede00fbdadf0c37a99889b6901bfb9b/datasette/views/table.py#L267-L272

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/818#issuecomment-642420375
issue_url: https://api.github.com/repos/simonw/datasette/issues/818
node_id: MDEyOklzc3VlQ29tbWVudDY0MjQyMDM3NQ==
created_at: 2020-06-11T05:40:07Z
updated_at: 2020-06-11T05:40:07Z
author_association: OWNER
body:
  https://github.com/simonw/datasette-permissions-sql is now released as a 0.1a here: https://pypi.org/project/datasette-permissions-sql/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/394#issuecomment-642522285
issue_url: https://api.github.com/repos/simonw/datasette/issues/394
node_id: MDEyOklzc3VlQ29tbWVudDY0MjUyMjI4NQ==
created_at: 2020-06-11T09:15:19Z
updated_at: 2020-06-11T09:15:19Z
author_association: NONE
body:
  Hi @wragge,

  This looks great, thanks for the share! I refactored it into a self-contained function, binding on a random available TCP port (multi-user context). I am using subprocess API directly since the `%run` magic was leaving defunct process behind :/

  ![image](https://user-images.githubusercontent.com/58298410/84367566-b5d0d500-abd4-11ea-96e2-f5c05a28e506.png)

  ```python
  import socket

  from signal import SIGINT
  from subprocess import Popen, PIPE

  from IPython.display import display, HTML
  from notebook.notebookapp import list_running_servers


  def get_free_tcp_port():
      """
      Get a free TCP port.
      """
      tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
      tcp.bind(('', 0))
      _, port = tcp.getsockname()
      tcp.close()
      return port


  def datasette(database):
      """
      Run datasette on an SQLite database.
      """
      # Get current running servers
      servers = list_running_servers()

      # Get the current base url
      base_url = next(servers)['base_url']

      # Get a free port
      port = get_free_tcp_port()

      # Create a base url for Datasette suing the proxy path
      proxy_url = f'{base_url}proxy/absolute/{port}/'

      # Display a link to Datasette
      display(HTML(f'<p><a href="{proxy_url}">View Datasette</a> (Click on the stop button to close the Datasette server)</p>'))

      # Launch Datasette
      with Popen(
          [
              'python', '-m', 'datasette', '--',
              database,
              '--port', str(port),
              '--config', f'base_url:{proxy_url}'
          ],
          stdout=PIPE,
          stderr=PIPE,
          bufsize=1,
          universal_newlines=True
      ) as p:
          print(p.stdout.readline(), end='')
          while True:
              try:
                  line = p.stderr.readline()
                  if not line:
                      break
                  print(line, end='')
                  exit_code = p.poll()
              except KeyboardInterrupt:
                  p.send_signal(SIGINT)
  ```

  Ideally, I'd like some extra magic to notify users when they are leaving the closing the notebook tab and make them terminate the running datasette processes. I'll be looking for it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/832#issuecomment-642741930
issue_url: https://api.github.com/repos/simonw/datasette/issues/832
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjc0MTkzMA==
created_at: 2020-06-11T15:35:53Z
updated_at: 2020-06-11T15:36:05Z
author_association: OWNER
body:
  May the fix here is to implement a `.check_permissions()` method which passes when the first permission passes?
  ```python
  await self.check_permissions(request, [
      ("view-table", (database, table)),
      ("view-database", database),
      "view-instance",
  ])
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/809#issuecomment-642745518
issue_url: https://api.github.com/repos/simonw/datasette/issues/809
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjc0NTUxOA==
created_at: 2020-06-11T15:38:51Z
updated_at: 2020-06-11T15:38:51Z
author_association: OWNER
body:
  The way to manually test this is to publish a database to each provider and then check that the `/-/messages` debug tool works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/809#issuecomment-642750790
issue_url: https://api.github.com/repos/simonw/datasette/issues/809
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjc1MDc5MA==
created_at: 2020-06-11T15:42:23Z
updated_at: 2020-06-11T15:42:23Z
author_association: OWNER
body:
      datasette publish heroku fixtures.db -n datasette-publish-secret --branch=master

  https://datasette-publish-secret.herokuapp.com/-/messages - Heroku works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/809#issuecomment-642754589
issue_url: https://api.github.com/repos/simonw/datasette/issues/809
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjc1NDU4OQ==
created_at: 2020-06-11T15:45:25Z
updated_at: 2020-06-11T15:45:25Z
author_association: OWNER
body:
      datasette publish cloudrun fixtures.db --service datasette-publish-secret --branch=master

  https://datasette-publish-secret-j7hipcg4aq-uw.a.run.app/-/messages

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/809#issuecomment-642772344
issue_url: https://api.github.com/repos/simonw/datasette/issues/809
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjc3MjM0NA==
created_at: 2020-06-11T16:01:15Z
updated_at: 2020-06-11T16:01:15Z
author_association: OWNER
body:
  ```
  datasette package fixtures.db --secret woot --branch master
  Sending build context to Docker daemon  260.6kB
  Step 1/9 : FROM python:3.8
  3.8: Pulling from library/python
  e9afc4f90ab0: Downloading [=======>                                           ]  7.195MB/50.39MB
  989e6b19a265: Downloading [============================>                      ]  4.475MB/7.812MB
  af14b6c2f878: Downloading [===========================>                       ]  5.422MB/9.996MB
  5573c4b30949: Waiting 
  11a88e764313: Waiting 
  ee776f0e36af: Waiting 
  513c90a1afc3: Waiting 
  df9b9e95bdb9: Waiting 
  86c9edb54464: Waiting 
  ...
  datasette package fixtures.db --secret woot --branch master
  docker run -p 8001:8001 a155798bd842
  ```
  This works too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/832#issuecomment-642795966
issue_url: https://api.github.com/repos/simonw/datasette/issues/832
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjc5NTk2Ng==
created_at: 2020-06-11T16:37:21Z
updated_at: 2020-06-11T16:37:21Z
author_association: OWNER
body:
  How would I document this? Probably in another section on https://datasette.readthedocs.io/en/latest/authentication.html#permissions

  But I'd also need to add documentation to the individual views stating what permissions are checked and in what order. I could do that on this page: https://datasette.readthedocs.io/en/latest/pages.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/833#issuecomment-642902208
issue_url: https://api.github.com/repos/simonw/datasette/issues/833
node_id: MDEyOklzc3VlQ29tbWVudDY0MjkwMjIwOA==
created_at: 2020-06-11T20:08:57Z
updated_at: 2020-06-11T20:08:57Z
author_association: OWNER
body:
  I'm tempted to add a `view-instance` check before routing any URLs, but that wouldn't be compatible with the idea in #832 that having `view-table` should be enough to view a table even if you don't pass `view-instance`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/833#issuecomment-642905424
issue_url: https://api.github.com/repos/simonw/datasette/issues/833
node_id: MDEyOklzc3VlQ29tbWVudDY0MjkwNTQyNA==
created_at: 2020-06-11T20:16:41Z
updated_at: 2020-06-11T20:16:41Z
author_association: OWNER
body:
  I'll add a new test in `test_permissions.py` which locks down an instance and  then loops through paths as the anonymous user making sure they aren't accessible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/832#issuecomment-642906681
issue_url: https://api.github.com/repos/simonw/datasette/issues/832
node_id: MDEyOklzc3VlQ29tbWVudDY0MjkwNjY4MQ==
created_at: 2020-06-11T20:19:47Z
updated_at: 2020-06-11T20:20:02Z
author_association: OWNER
body:
  So for the following:
  ```
  await self.check_permissions(request, [
      ("view-table", (database, table)),
      ("view-database", database),
      "view-instance",
  ])
  ```
  The logic is: if the first test returns `True`, you get access. If it returns `False` you are denied. If it says `None` then move on to the next check in the list and repeat.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/832#issuecomment-642907021
issue_url: https://api.github.com/repos/simonw/datasette/issues/832
node_id: MDEyOklzc3VlQ29tbWVudDY0MjkwNzAyMQ==
created_at: 2020-06-11T20:20:35Z
updated_at: 2020-06-11T20:20:35Z
author_association: OWNER
body:
  I think the new `.check_permissions()` should be a documented utility that is available to plugins.
   Maybe a method on `datasette`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/220#issuecomment-642944645
issue_url: https://api.github.com/repos/simonw/datasette/issues/220
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjk0NDY0NQ==
created_at: 2020-06-11T21:49:55Z
updated_at: 2020-06-11T21:49:55Z
author_association: OWNER
body:
  I'm OK with not implementing this - I've got used to the existing mechanism, and it doesn't frustrate me enough to work on this more.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/824#issuecomment-642951150
issue_url: https://api.github.com/repos/simonw/datasette/issues/824
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjk1MTE1MA==
created_at: 2020-06-11T22:00:17Z
updated_at: 2020-06-11T22:00:17Z
author_association: OWNER
body:
  I got this working: https://github.com/simonw/datasette-auth-github/pull/64

  Just one problem: it uses the existing `ds_actor` cookie, which means it doesn't actually exercise the `actor_from_request` plugin!

  It does use `register_routes` though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/824#issuecomment-642952962
issue_url: https://api.github.com/repos/simonw/datasette/issues/824
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjk1Mjk2Mg==
created_at: 2020-06-11T22:01:58Z
updated_at: 2020-06-11T22:01:58Z
author_association: OWNER
body:
  Alternative idea: a plugin that handles Bearer token authentication. Uses `metadata.json` with secret plugin values to map an incoming token to an actor dictionary, which can then be mapped to permissions.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/824#issuecomment-642953605
issue_url: https://api.github.com/repos/simonw/datasette/issues/824
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjk1MzYwNQ==
created_at: 2020-06-11T22:02:32Z
updated_at: 2020-06-11T22:02:32Z
author_association: OWNER
body:
  `datasette-auth-tokens` can be the name. I can get a simple initial version of it running pretty quickly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/833#issuecomment-642958225
issue_url: https://api.github.com/repos/simonw/datasette/issues/833
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjk1ODIyNQ==
created_at: 2020-06-11T22:15:32Z
updated_at: 2020-06-11T22:15:32Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/29c5ff493ad7918b8fc44ea7920b41530e56dd5d/tests/test_permissions.py#L327-L348

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/824#issuecomment-642991513
issue_url: https://api.github.com/repos/simonw/datasette/issues/824
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjk5MTUxMw==
created_at: 2020-06-12T00:11:50Z
updated_at: 2020-06-12T00:11:50Z
author_association: OWNER
body:
  Done: https://github.com/simonw/datasette-auth-tokens and https://pypi.org/project/datasette-auth-tokens/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/838#issuecomment-642993277
issue_url: https://api.github.com/repos/simonw/datasette/issues/838
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjk5MzI3Nw==
created_at: 2020-06-12T00:18:26Z
updated_at: 2020-06-12T00:18:50Z
author_association: OWNER
body:
  Have you tried this without the `ProxyPassReverse` directive? I'm worried that might be confusing Datasette.

  This is the test I used to ensure this feature works - it scrapes all of the links on a bunch of different pages. Could it be missing something here?

  https://github.com/simonw/datasette/blob/647c5ff0f3e8140f40d7f41f0874ce4e1f4df65c/tests/test_html.py#L1233-L1274

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/806#issuecomment-642998097
issue_url: https://api.github.com/repos/simonw/datasette/issues/806
node_id: MDEyOklzc3VlQ29tbWVudDY0Mjk5ODA5Nw==
created_at: 2020-06-12T00:26:00Z
updated_at: 2020-06-12T00:26:00Z
author_association: OWNER
body:
  OK, I'm ready to ship. Last check of the release notes, then I'll update the news section in the README and release 0.44!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/806#issuecomment-643000948
issue_url: https://api.github.com/repos/simonw/datasette/issues/806
node_id: MDEyOklzc3VlQ29tbWVudDY0MzAwMDk0OA==
created_at: 2020-06-12T00:34:21Z
updated_at: 2020-06-12T00:34:21Z
author_association: OWNER
body:
  I'm going to add https://github.com/simonw/datasette-auth-tokens and https://github.com/simonw/datasette-permissions-sql to the documentation and release notes in a few places.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/806#issuecomment-643010591
issue_url: https://api.github.com/repos/simonw/datasette/issues/806
node_id: MDEyOklzc3VlQ29tbWVudDY0MzAxMDU5MQ==
created_at: 2020-06-12T01:13:06Z
updated_at: 2020-06-12T01:13:06Z
author_association: OWNER
body:
  Tests are passing again: https://github.com/simonw/datasette/commit/9ae0d483ead93c0832142e5dc85959ae3c8f73ea

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/838#issuecomment-643083451
issue_url: https://api.github.com/repos/simonw/datasette/issues/838
node_id: MDEyOklzc3VlQ29tbWVudDY0MzA4MzQ1MQ==
created_at: 2020-06-12T06:04:14Z
updated_at: 2020-06-12T06:04:14Z
author_association: NONE
body:
  Hmm, I haven't tried removing `ProxyPassReverse`, but it doesn't touch the HTML, which is the issue I'm seeing. You can read the [documentation here](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse). `ProxyPassReverse` is a standard directive when proxying with Apache. I've used it dozens of times with other applications.

  Looking a little more at the code, I think the issue here is that the behaviour of `base_url` makes sense when Datasette is _mounted_ at a path within a larger application, but not when HTTP requests are being _proxied_ to it.

  In a _mount_ situation, it is perfectly fine to construct URLs reusing the domain and path from the request. In a _proxy_ situation, it never is, as the domain and path in the request are not the domain and path that the non-proxy client actually needs to use. That is, links which include the Apache → Datasette request origin, `localhost:8001`, instead of the browser → Apache request origin, `example.com`, will be broken.

  The tests you pointed to also reflect this in two ways:

  1. They strip a leading `http://localhost`, allowing such URLs in the facet links to pass, but inclusion of that in a proxy situation would mean the URL is broken.

  2. The test client emits direct ASGI events instead of actual proxied HTTP requests. The headers of these ASGI events don't reflect the way an HTTP proxy works; instead they pass through the original request path which contains `base_url`. This works because Datasette responds to requests equivalently at either `/…` or `/{base_url}/…`, which makes some sense in a _mount_ situation but is unconventional (albeit workable) for a proxied app.

  Apps that support being proxied automatically support being mounted, but apps that only support being mounted don't automatically support being proxied.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643393506
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDY0MzM5MzUwNg==
created_at: 2020-06-12T17:21:14Z
updated_at: 2020-06-12T17:21:14Z
author_association: MEMBER
body:
  I only install SQLite for this:

  https://github.com/dogsheep/github-to-sqlite/blob/c0d54e0260468be38152293df5abd775c068495d/.github/workflows/deploy-demo.yml#L77-L78

  I'm going to remove the need to install sqlite3 by making this possible with sqlite-utils: https://github.com/simonw/sqlite-utils/issues/115

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/115#issuecomment-643406939
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/115
node_id: MDEyOklzc3VlQ29tbWVudDY0MzQwNjkzOQ==
created_at: 2020-06-12T17:51:11Z
updated_at: 2020-06-12T17:51:11Z
author_association: OWNER
body:
  https://github.com/simonw/sqlite-utils/blob/03ee97d2258254581bea72842518904fc1cbe60f/tests/test_cli.py#L1112-L1128

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/840#issuecomment-643453128
issue_url: https://api.github.com/repos/simonw/datasette/issues/840
node_id: MDEyOklzc3VlQ29tbWVudDY0MzQ1MzEyOA==
created_at: 2020-06-12T19:43:15Z
updated_at: 2020-06-12T19:43:15Z
author_association: OWNER
body:
  I don't like how this often involves a logout link that can be maliciously activated.

  I'm going to use a CSRF protected form button styled to look like a link instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/840#issuecomment-643454625
issue_url: https://api.github.com/repos/simonw/datasette/issues/840
node_id: MDEyOklzc3VlQ29tbWVudDY0MzQ1NDYyNQ==
created_at: 2020-06-12T19:47:38Z
updated_at: 2020-06-12T19:47:53Z
author_association: OWNER
body:
  Another problem: what to display in the "you are logged in as", since we don't dictate an actor design.

  I'm going to use a includes template for this that can easily be over-ridden by administrators or by plugins.

  The default will look for the first available of the following keys:

  - display
  - name
  - username
  - login
  - id

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/834#issuecomment-643501064
issue_url: https://api.github.com/repos/simonw/datasette/issues/834
node_id: MDEyOklzc3VlQ29tbWVudDY0MzUwMTA2NA==
created_at: 2020-06-12T22:04:43Z
updated_at: 2020-06-12T22:04:43Z
author_association: OWNER
body:
  Another use-case for this: I want to use the `--root` option on Glitch but it gives me a 127.0.0.1 URL. Glitch has a `PROJECT_DOMAIN` environment variable which tells me the URL. A `datasette-glitch` plugin could use a `startup` hook to output the correct login URL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/805#issuecomment-643501428
issue_url: https://api.github.com/repos/simonw/datasette/issues/805
node_id: MDEyOklzc3VlQ29tbWVudDY0MzUwMTQyOA==
created_at: 2020-06-12T22:06:08Z
updated_at: 2020-06-12T22:06:08Z
author_association: OWNER
body:
  This needs the `startup` hook, see https://github.com/simonw/datasette/issues/834#issuecomment-643501064

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/834#issuecomment-643510240
issue_url: https://api.github.com/repos/simonw/datasette/issues/834
node_id: MDEyOklzc3VlQ29tbWVudDY0MzUxMDI0MA==
created_at: 2020-06-12T22:40:26Z
updated_at: 2020-06-12T22:40:26Z
author_association: OWNER
body:
  Another use-case: plugins that need their own database with the correct tables. They can write to the database on startup to create their tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/834#issuecomment-643648359
issue_url: https://api.github.com/repos/simonw/datasette/issues/834
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY0ODM1OQ==
created_at: 2020-06-13T16:47:29Z
updated_at: 2020-06-13T16:47:29Z
author_association: OWNER
body:
  Implementing this is proving surprisingly tricky, because of the need to be able to optionally `await` the returned value. It's a bit of a fiddle to get this to work within unit tests because they run in non-async functions - due to this cunning `async_to_sync` usage in the test client:

  https://github.com/simonw/datasette/blob/b906030235efbdff536405d66078f4868ce0d3bd/tests/fixtures.py#L115-L133

  I could switch to using `async def test_*` functions decorated with `@pytest.mark.asyncio` but I'd rather not re-engineer the entire test suite just for this one feature, so I'll try to find another way.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/841#issuecomment-643655108
issue_url: https://api.github.com/repos/simonw/datasette/issues/841
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY1NTEwOA==
created_at: 2020-06-13T17:43:15Z
updated_at: 2020-06-13T17:43:15Z
author_association: OWNER
body:
  Using https://pypi.org/project/pytest-cov/ and running `pytest --cov=datasette`:
  ```
  ---------- coverage: platform darwin, python 3.7.7-final-0 -----------
  Name                                 Stmts   Miss  Cover
  --------------------------------------------------------
  datasette/__init__.py                    3      0   100%
  datasette/__main__.py                    3      3     0%
  datasette/_version.py                  277    152    45%
  datasette/actor_auth_cookie.py          19      3    84%
  datasette/app.py                       499     27    95%
  datasette/cli.py                       157     45    71%
  datasette/database.py                  233     17    93%
  datasette/default_permissions.py        39      0   100%
  datasette/facets.py                    209     24    89%
  datasette/filters.py                   122      7    94%
  datasette/hookspecs.py                  19      0   100%
  datasette/inspect.py                    37     23    38%
  datasette/plugins.py                    34      6    82%
  datasette/publish/__init__.py            0      0   100%
  datasette/publish/cloudrun.py           55      2    96%
  datasette/publish/common.py             19      1    95%
  datasette/publish/heroku.py             95     13    86%
  datasette/renderer.py                   63      4    94%
  datasette/sql_functions.py               4      0   100%
  datasette/tracer.py                     85     16    81%
  datasette/utils/__init__.py            503     31    94%
  datasette/utils/asgi.py                253     25    90%
  datasette/utils/shutil_backport.py      44     40     9%
  datasette/version.py                     4      0   100%
  datasette/views/__init__.py              0      0   100%
  datasette/views/base.py                288     19    93%
  datasette/views/database.py            120      2    98%
  datasette/views/index.py                57      2    96%
  datasette/views/special.py              72     16    78%
  datasette/views/table.py               418     18    96%
  --------------------------------------------------------
  TOTAL                                 3731    496    87%
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/841#issuecomment-643656053
issue_url: https://api.github.com/repos/simonw/datasette/issues/841
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY1NjA1Mw==
created_at: 2020-06-13T17:50:34Z
updated_at: 2020-06-13T17:50:34Z
author_association: OWNER
body:
  Added a `.coveragerc` file:
  ```
  [run]
  omit = datasette/_version.py, datasette/utils/shutil_backport.py
  ```
  And ran again: `pytest --cov=datasette --cov-config=.coveragerc`
  ```
  Name                               Stmts   Miss  Cover
  ------------------------------------------------------
  datasette/__init__.py                  3      0   100%
  datasette/__main__.py                  3      3     0%
  datasette/actor_auth_cookie.py        19      3    84%
  datasette/app.py                     499     27    95%
  datasette/cli.py                     157     45    71%
  datasette/database.py                233     17    93%
  datasette/default_permissions.py      39      0   100%
  datasette/facets.py                  209     24    89%
  datasette/filters.py                 122      7    94%
  datasette/hookspecs.py                19      0   100%
  datasette/inspect.py                  37     23    38%
  datasette/plugins.py                  34      6    82%
  datasette/publish/__init__.py          0      0   100%
  datasette/publish/cloudrun.py         55      2    96%
  datasette/publish/common.py           19      1    95%
  datasette/publish/heroku.py           95     13    86%
  datasette/renderer.py                 63      4    94%
  datasette/sql_functions.py             4      0   100%
  datasette/tracer.py                   85     16    81%
  datasette/utils/__init__.py          503     31    94%
  datasette/utils/asgi.py              253     25    90%
  datasette/version.py                   4      0   100%
  datasette/views/__init__.py            0      0   100%
  datasette/views/base.py              288     19    93%
  datasette/views/database.py          120      2    98%
  datasette/views/index.py              57      2    96%
  datasette/views/special.py            72     16    78%
  datasette/views/table.py             418     18    96%
  ------------------------------------------------------
  TOTAL                               3410    304    91%
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/841#issuecomment-643657287
issue_url: https://api.github.com/repos/simonw/datasette/issues/841
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY1NzI4Nw==
created_at: 2020-06-13T18:01:39Z
updated_at: 2020-06-13T18:01:39Z
author_association: OWNER
body:
  Added `--cov-report html` and got this report: https://static.simonwillison.net/static/2020/htmlcov-issue-841/index.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/841#issuecomment-643658036
issue_url: https://api.github.com/repos/simonw/datasette/issues/841
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY1ODAzNg==
created_at: 2020-06-13T18:08:13Z
updated_at: 2020-06-13T18:08:13Z
author_association: OWNER
body:
  From digging through that report it looks like the majority stuff that isn't fully covered is corner-cases... which are the kind of things I really do want the tests to catch.

  I'm not entirely ready to commit to 100%, but I'm going to start digging through and seeing how close I can get. If I can get to 98% (I'm on 91% already) I may as well push all  the way to 100.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/841#issuecomment-643660427
issue_url: https://api.github.com/repos/simonw/datasette/issues/841
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY2MDQyNw==
created_at: 2020-06-13T18:29:30Z
updated_at: 2020-06-13T18:29:36Z
author_association: OWNER
body:
  This one looks easy enough to fix:

  <img width="757" alt="Coverage_for_datasette_actor_auth_cookie_py__84_" src="https://user-images.githubusercontent.com/9599/84576366-15f88000-ad69-11ea-906b-efa15d3735a4.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/841#issuecomment-643660757
issue_url: https://api.github.com/repos/simonw/datasette/issues/841
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY2MDc1Nw==
created_at: 2020-06-13T18:32:20Z
updated_at: 2020-06-13T18:32:20Z
author_association: OWNER
body:
  Looking at options for publishing coverage reports:

  * https://github.com/codecov/codecov-action
  * https://github.com/coveralls-clients/coveralls-python

  I'm going to try https://codecov.io/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/841#issuecomment-643661125
issue_url: https://api.github.com/repos/simonw/datasette/issues/841
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY2MTEyNQ==
created_at: 2020-06-13T18:35:30Z
updated_at: 2020-06-13T18:36:50Z
author_association: OWNER
body:
  I ran export CODECOV_TOKEN="f7935cad...", then ran this:
  ```
  datasette $ bash <(curl -s https://codecov.io/bash)  
    _____          _
   / ____|        | |
  | |     ___   __| | ___  ___ _____   __
  | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
  | |___| (_) | (_| |  __/ (_| (_) \ V /
   \_____\___/ \__,_|\___|\___\___/ \_/
                                Bash-20200602-f809a24


  x> No CI provider detected.
      Testing inside Docker? http://docs.codecov.io/docs/testing-with-docker
      Testing with Tox? https://docs.codecov.io/docs/python#section-testing-with-tox
      project root: .
  --> token set from env
      Yaml not found, that's ok! Learn more at http://docs.codecov.io/docs/codecov-yaml
  ==> Running gcov in . (disable via -X gcov)
  ==> Searching for coverage reports in:
      + .
      -> Found 1 reports
  ==> Detecting git/mercurial file structure
  ==> Reading reports
      + ./coverage.xml bytes=139174
  ==> Appending adjustments
      https://docs.codecov.io/docs/fixing-reports
      -> No adjustments found
  ==> Gzipping contents
  ==> Uploading reports
      url: https://codecov.io
      query: branch=master&commit=0e49842e227a0f1f69d48108c87d17fe0379e548&build=&build_url=&name=&tag=&slug=simonw%2Fdatasette&service=&flags=&pr=&job=
      -> Pinging Codecov
  https://codecov.io/upload/v4?package=bash-20200602-f809a24&token=secret&branch=master&commit=0e49842e227a0f1f69d48108c87d17fe0379e548&build=&build_url=&name=&tag=&slug=simonw%2Fdatasette&service=&flags=&pr=&job=
      -> Uploading
      -> View reports at https://codecov.io/github/simonw/datasette/commit/0e49842e227a0f1f69d48108c87d17fe0379e548
  ```
  But https://codecov.io/github/simonw/datasette/commit/0e49842e227a0f1f69d48108c87d17fe0379e548 is a 404, so it doesn't seem to have worked?

  UPDATE: It works now, took about 30 seconds before the report showed up at that URL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-643663005
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY2MzAwNQ==
created_at: 2020-06-13T18:51:57Z
updated_at: 2020-06-13T18:51:57Z
author_association: OWNER
body:
  Two potential designs:

  - `_actor_id`, `_request_ip`, `_now_timestamp` - so special reserved parameters
  - a SQL function: `update blah set up = special('ip')`

  I fee the first would be easier to implement.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/843#issuecomment-643676069
issue_url: https://api.github.com/repos/simonw/datasette/issues/843
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY3NjA2OQ==
created_at: 2020-06-13T20:45:29Z
updated_at: 2020-06-13T20:45:29Z
author_association: OWNER
body:
  I set up https://codecov.io/gh/simonw/datasette/settings and added a `CODECOV_TOKEN` to the GitHub Actions secrets for this repo.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/844#issuecomment-643681517
issue_url: https://api.github.com/repos/simonw/datasette/issues/844
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY4MTUxNw==
created_at: 2020-06-13T21:36:15Z
updated_at: 2020-06-13T21:36:15Z
author_association: OWNER
body:
  OK, this works now: https://codecov.io/gh/simonw/datasette/tree/1210d9f41841bdca450f85a2342cdb0ff339c1b4

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/841#issuecomment-643681747
issue_url: https://api.github.com/repos/simonw/datasette/issues/841
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY4MTc0Nw==
created_at: 2020-06-13T21:38:46Z
updated_at: 2020-06-13T21:38:46Z
author_association: OWNER
body:
  Closing this because I've researched feasibility. I may start a milestone in the future to help me get to 100%.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/846#issuecomment-643685207
issue_url: https://api.github.com/repos/simonw/datasette/issues/846
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY4NTIwNw==
created_at: 2020-06-13T22:18:01Z
updated_at: 2020-06-13T22:18:01Z
author_association: OWNER
body:
  This shows currently open files (after `pip install psutil`):
  ```
  import psutil
  psutil.Process().open_files()
  ```

  I ran it inside `pytest -x --pdb` and got this:
  ```
  > /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.7/site-packages/jinja2/utils.py(154)open_if_exists()
  -> return open(filename, mode)
  (Pdb) import psutil
  (Pdb) psutil.Process().open_files()
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp9uhx5d8x/fixtures.db', fd=10),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyfw44ica/fixtures.dot.db', fd=11),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyrg6g48b/fixtures.db', fd=12),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=13),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=14),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=15),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=16),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=17),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=18),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=19),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpng4lg84_/fixtures.db', fd=20),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp9uhx5d8x/fixtures.db', fd=21),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp9uhx5d8x/fixtures.db', fd=22),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp9uhx5d8x/fixtures.db', fd=23),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmph11oalw_/fixtures.db', fd=24),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyfw44ica/fixtures.dot.db', fd=25),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyfw44ica/fixtures.dot.db', fd=26),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyfw44ica/fixtures.dot.db', fd=27),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpiorb2bo9/fixtures.db', fd=28),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyrg6g48b/fixtures.db', fd=29),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyrg6g48b/fixtures.db', fd=30),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyrg6g48b/fixtures.db', fd=31),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprvyj5udv/fixtures.db', fd=32),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpng4lg84_/fixtures.db', fd=33),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpng4lg84_/fixtures.db', fd=34),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpng4lg84_/fixtures.db', fd=35),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpb_l6gmq0/fixtures.db', fd=36),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmph11oalw_/extra database.db', fd=40),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/fixtures.db', fd=41),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpiorb2bo9/fixtures.db', fd=42),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpiorb2bo9/fixtures.db', fd=43),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpiorb2bo9/fixtures.db', fd=44),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmph11oalw_/fixtures.db', fd=45),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmph11oalw_/fixtures.db', fd=52),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/fixtures.db', fd=53),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprvyj5udv/fixtures.db', fd=54),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprvyj5udv/fixtures.db', fd=55),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprvyj5udv/fixtures.db', fd=56),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoveuwqn6/fixtures.db', fd=57),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpb_l6gmq0/fixtures.db', fd=61),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpb_l6gmq0/fixtures.db', fd=62),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpb_l6gmq0/fixtures.db', fd=63),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp_j4h9mrn/fixtures.db', fd=64),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/fixtures.db', fd=65),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/fixtures.db', fd=66),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/extra database.db', fd=67),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/extra database.db', fd=68),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/fixtures.db', fd=69),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/extra database.db', fd=70)
   popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpub3eodj1/fixtures.db', fd=71)
   popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/fixtures.db', fd=72)
   popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo.db', fd=73)
   popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo.db', fd=74)
   popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/fixtures.db', fd=75)
   popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/fixtures.db', fd=76)
   popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo.db', fd=77)
   popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo-bar.db', fd=78)
   popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo-bar.db', fd=79)
   popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo-bar.db', fd=80)
   popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-4/config-dir0/immutable.db', fd=81),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoveuwqn6/fixtures.db', fd=82),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoveuwqn6/fixtures.db', fd=83),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoveuwqn6/fixtures.db', fd=84),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp44w5d5wo/fixtures.db', fd=85),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp_j4h9mrn/fixtures.db', fd=86),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp_j4h9mrn/fixtures.db', fd=87),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp_j4h9mrn/fixtures.db', fd=88),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpvu7h14uy/fixtures.db', fd=89),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-4/config-dir0/demo.db', fd=119),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-4/config-dir0/demo.db', fd=120),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-4/config-dir0/demo.db', fd=121),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0xcnrjag/fixtures.db', fd=122),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpub3eodj1/fixtures.db', fd=123),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpub3eodj1/fixtures.db', fd=124),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpub3eodj1/fixtures.db', fd=125),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfz8go8rk/fixtures.db', fd=126),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp44w5d5wo/fixtures.db', fd=127),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp44w5d5wo/fixtures.db', fd=128),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp44w5d5wo/fixtures.db', fd=129),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/fixtures.db', fd=130)
   popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpvu7h14uy/fixtures.db', fd=131),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpvu7h14uy/fixtures.db', fd=132),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpvo3cobk9/fixtures.db', fd=133),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp2t9txyir/fixtures.db', fd=134),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfz8go8rk/fixtures.db', fd=135),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfz8go8rk/fixtures.db', fd=136),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp7h3skv8b/fixtures.db', fd=137),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/fixtures.db', fd=138),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/fixtures.db', fd=139),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/fixtures.db', fd=140),
  popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/extra database.db', fd=141),
  ```
  So yeah, that's too many open files!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/846#issuecomment-643685669
issue_url: https://api.github.com/repos/simonw/datasette/issues/846
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY4NTY2OQ==
created_at: 2020-06-13T22:24:22Z
updated_at: 2020-06-13T22:24:22Z
author_association: OWNER
body:
  I tried this experiment:
  ```python
  import sqlite3, psutil
  def show_things():
      conn = sqlite3.connect("fixtures.db")
      tables = [r[0] for r in conn.execute("select * from sqlite_master").fetchall()]
      return tables
  print(psutil.Process().open_files())
  print(show_things())
  print(psutil.Process().open_files())
  ```
  To see if the connection would be automatically released when the `conn` variable was garbage collected at the end of the function... and it was correctly released - the two calls to `open_files()` showed that the file did not remain open.

  Likewise:
  ```
  In [11]: conn = sqlite3.connect("fixtures.db")                                                                                                                                                                    

  In [12]: psutil.Process().open_files()                                                                                                                                                                            
  Out[12]: 
  [popenfile(path='/Users/simon/.ipython/profile_default/history.sqlite', fd=4),
   popenfile(path='/Users/simon/.ipython/profile_default/history.sqlite', fd=5),
   popenfile(path='/Users/simon/Dropbox/Development/datasette/fixtures.db', fd=12)]

  In [13]: del conn                                                                                                                                                                                                 

  In [14]: psutil.Process().open_files()                                                                                                                                                                            
  Out[14]: 
  [popenfile(path='/Users/simon/.ipython/profile_default/history.sqlite', fd=4),
   popenfile(path='/Users/simon/.ipython/profile_default/history.sqlite', fd=5)]
  ```
  So presumably there's something about the way my pytest fixtures work that's causing the many different `Datasette()` instances and their underlying SQLite connections that I create not to be cleaned up later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/846#issuecomment-643698790
issue_url: https://api.github.com/repos/simonw/datasette/issues/846
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY5ODc5MA==
created_at: 2020-06-14T00:20:42Z
updated_at: 2020-06-14T00:20:42Z
author_association: OWNER
body:
  Released a new plugin, `datasette-psutil`, as a side-effect of this investigation: https://github.com/simonw/datasette-psutil

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/846#issuecomment-643699063
issue_url: https://api.github.com/repos/simonw/datasette/issues/846
node_id: MDEyOklzc3VlQ29tbWVudDY0MzY5OTA2Mw==
created_at: 2020-06-14T00:22:32Z
updated_at: 2020-06-14T00:22:32Z
author_association: OWNER
body:
  Idea: `num_sql_threads` (described as "Number of threads in the thread pool for executing SQLite queries") defaults to 3 - can I knock that down to 1 in the tests and open less connections as a result?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/847#issuecomment-643702715
issue_url: https://api.github.com/repos/simonw/datasette/issues/847
node_id: MDEyOklzc3VlQ29tbWVudDY0MzcwMjcxNQ==
created_at: 2020-06-14T01:03:30Z
updated_at: 2020-06-14T01:03:40Z
author_association: OWNER
body:
  Filed a related issue with some ideas against `coveragepy` here: https://github.com/nedbat/coveragepy/issues/999

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/847#issuecomment-643704565
issue_url: https://api.github.com/repos/simonw/datasette/issues/847
node_id: MDEyOklzc3VlQ29tbWVudDY0MzcwNDU2NQ==
created_at: 2020-06-14T01:26:56Z
updated_at: 2020-06-14T01:26:56Z
author_association: OWNER
body:
  On closer inspection, I don't know if there's that much useful stuff you can do with the data from `.coverage` on its own.

  Consider the following query against a `.coverage` run against Datasette itself:

  ```sql
  select file_id, context_id, numbits_to_nums(numbits) from line_bits
  ```
  <img width="1451" alt="_coverage__select_file_id__context_id__numbits_to_nums_numbits__from_line_bits" src="https://user-images.githubusercontent.com/9599/84582622-40ffc580-ada3-11ea-98d2-52bcde514a26.png">

  It looks like this tells me which lines of which files were executed during the test run. But... without the actual source code, I don't think I can calculate the coverage percentage for each file. I don't want to count comment lines or whitespace as untested for example, and I don't know how many lines were in the file.

  If I'm right that it's not possible to calculate percentage coverage from just the `.coverage` data then I'll need to do something a bit more involved - maybe parsing the `coverage.xml` report and loading that into my own schema?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/847#issuecomment-643704730
issue_url: https://api.github.com/repos/simonw/datasette/issues/847
node_id: MDEyOklzc3VlQ29tbWVudDY0MzcwNDczMA==
created_at: 2020-06-14T01:28:34Z
updated_at: 2020-06-14T01:28:34Z
author_association: OWNER
body:
  Here's the plugin that adds those custom SQLite functions:
  ```python
  from datasette import hookimpl
  from coverage.numbits import register_sqlite_functions


  @hookimpl
  def prepare_connection(conn):
      register_sqlite_functions(conn)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/848#issuecomment-643711117
issue_url: https://api.github.com/repos/simonw/datasette/issues/848
node_id: MDEyOklzc3VlQ29tbWVudDY0MzcxMTExNw==
created_at: 2020-06-14T03:05:55Z
updated_at: 2020-07-03T02:44:09Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=h1) Report
  > Merging [#848](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/57879dc8b346a435804a9e45ffaacbf2a0228bc6&el=desc) will **decrease** coverage by `0.60%`.
  > The diff coverage is `0.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/848/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##           master     #848      +/-   ##
  ==========================================
  - Coverage   83.42%   82.82%   -0.61%     
  ==========================================
    Files          27       26       -1     
    Lines        3632     3540      -92     
  ==========================================
  - Hits         3030     2932      -98     
  - Misses        602      608       +6     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `71.34% <0.00%> (-0.89%)` | :arrow_down: |
  | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `77.77% <0.00%> (-3.40%)` | :arrow_down: |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.58% <0.00%> (-1.58%)` | :arrow_down: |
  | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `90.90% <0.00%> (-0.42%)` | :arrow_down: |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.84% <0.00%> (-0.09%)` | :arrow_down: |
  | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <0.00%> (ø)` | |
  | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <0.00%> (ø)` | |
  | [datasette/default\_permissions.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `100.00% <0.00%> (ø)` | |
  | [datasette/default\_magic\_parameters.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWFnaWNfcGFyYW1ldGVycy5weQ==) | | |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.40% <0.00%> (+<0.01%)` | :arrow_up: |
  | ... and [2 more](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree-more) | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=footer). Last update [57879dc...0d100d1](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/849#issuecomment-644322234
issue_url: https://api.github.com/repos/simonw/datasette/issues/849
node_id: MDEyOklzc3VlQ29tbWVudDY0NDMyMjIzNA==
created_at: 2020-06-15T19:06:16Z
updated_at: 2020-06-15T19:06:16Z
author_association: OWNER
body:
  I'll make this change on a few of my other repos first to make sure I haven't missed any tricky edge-cases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/849#issuecomment-644384417
issue_url: https://api.github.com/repos/simonw/datasette/issues/849
node_id: MDEyOklzc3VlQ29tbWVudDY0NDM4NDQxNw==
created_at: 2020-06-15T20:55:23Z
updated_at: 2020-06-15T20:55:23Z
author_association: OWNER
body:
  I'm doing https://github.com/simonw/datasette-psutil first.

  In my local checkout:
  ```
  git branch -m master main
  git push -u origin main
  ```
  (Thanks, https://www.hanselman.com/blog/EasilyRenameYourGitDefaultBranchFromMasterToMain.aspx)

  Then in https://github.com/simonw/datasette-psutil/settings/branches I changed the default branch to `main`.

  <img width="586" alt="Branches" src="https://user-images.githubusercontent.com/9599/84704950-79bcad80-af0f-11ea-9baf-0dc44dfcb79e.png">

  Links to these docs: https://help.github.com/en/github/administering-a-repository/setting-the-default-branch

  That worked! https://github.com/simonw/datasette-psutil

  One catch, which I think will impact my most widely used repos the most (like datasette) - linking to a specific file now looks like this:

  https://github.com/simonw/datasette-psutil/blob/main/datasette_psutil/__init__.py

  The old https://github.com/simonw/datasette-psutil/blob/master/datasette_psutil/__init__.py link is presumably frozen in time?

  I've definitely got links spread around the web to my "most recent version of this code" that would use the `master` reference, which would need to be updated to `main` instead. Most of those are probably in the Datasette docs and on my blog though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/849#issuecomment-644384787
issue_url: https://api.github.com/repos/simonw/datasette/issues/849
node_id: MDEyOklzc3VlQ29tbWVudDY0NDM4NDc4Nw==
created_at: 2020-06-15T20:56:07Z
updated_at: 2020-06-15T20:56:19Z
author_association: OWNER
body:
  The big question is how this impacts existing CI configuration. `datasette-psutil` is configured to use Circle CI, what happens if I push a new commit?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/849#issuecomment-644584075
issue_url: https://api.github.com/repos/simonw/datasette/issues/849
node_id: MDEyOklzc3VlQ29tbWVudDY0NDU4NDA3NQ==
created_at: 2020-06-16T07:24:08Z
updated_at: 2020-06-16T07:24:08Z
author_association: OWNER
body:
  This guide is fantastic - I'll be following it closely: https://github.com/chancancode/branch-rename/blob/main/README.md - in particular the Action to mirror master and main for a while.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/690#issuecomment-644987083
issue_url: https://api.github.com/repos/simonw/datasette/issues/690
node_id: MDEyOklzc3VlQ29tbWVudDY0NDk4NzA4Mw==
created_at: 2020-06-16T20:11:35Z
updated_at: 2020-06-16T20:11:35Z
author_association: OWNER
body:
  Twitter conversation about drop-down menu solutions that are accessible, fast loading and use minimal JavaScript: https://twitter.com/simonw/status/1272974294545395712

  I _really_ like the approach taken by GitHub Primer, which builds on top of HTML `<summary>` `<details>` tags: https://primer.style/css/components/dropdown

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645030262
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTAzMDI2Mg==
created_at: 2020-06-16T21:51:01Z
updated_at: 2020-06-16T21:51:39Z
author_association: OWNER
body:
  File locking is interesting here. https://docs.aws.amazon.com/lambda/latest/dg/services-efs.html

  > Amazon EFS supports [file locking](https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html#consistency) to prevent corruption if multiple functions try to write to the same file system at the same time. Locking in Amazon EFS follows the NFS v4.1 protocol for advisory locking, and enables your applications to use both whole file and byte range locks. 

  SQLite can apparently work on NFS v4.1. I think I'd rather set things up so there's only ever one writer - so a Datasette instance could scale reads by running lots more lambda functions but only one function ever writes to a file at a time. Not sure if that's feasible with Lambda though - maybe by adding some additional shared state mechanism like Redis?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645031225
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTAzMTIyNQ==
created_at: 2020-06-16T21:53:25Z
updated_at: 2020-06-16T21:53:25Z
author_association: OWNER
body:
  Easier solution to this might be to have two functions - a "read-only" one which is allowed to scale as much as it likes, and a "write-only" one which can write to the database files but is limited to running a maximum of one Lambda instance. https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645032643
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTAzMjY0Mw==
created_at: 2020-06-16T21:57:10Z
updated_at: 2020-06-16T21:57:10Z
author_association: OWNER
body:
  https://docs.aws.amazon.com/efs/latest/ug/wt1-getting-started.html is an EFS walk-through using the AWS CLI tool instead of clicking around in their web interface.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645041663
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA0MTY2Mw==
created_at: 2020-06-16T22:21:44Z
updated_at: 2020-06-16T22:21:44Z
author_association: OWNER
body:
  https://github.com/jordaneremieff/mangum looks like the best way to run an ASGI app on Lambda at the moment.

  ```python
  from mangum import Mangum

  async def app(scope, receive, send):
      await send(
          {
              "type": "http.response.start",
              "status": 200,
              "headers": [[b"content-type", b"text/plain; charset=utf-8"]],
          }
      )
      await send({"type": "http.response.body", "body": b"Hello, world!"})


  handler = Mangum(app)
  ``` 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645042625
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA0MjYyNQ==
created_at: 2020-06-16T22:24:26Z
updated_at: 2020-06-16T22:24:26Z
author_association: OWNER
body:
  From https://mangum.io/adapter/

  > The AWS Lambda handler `event` and `context` arguments are made available to an ASGI application in the ASGI connection scope.
  > 
  > ```
  > scope['aws.event']
  > scope['aws.context']
  > ```
  I can use https://github.com/simonw/datasette-debug-asgi to see that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645045055
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA0NTA1NQ==
created_at: 2020-06-16T22:31:49Z
updated_at: 2020-06-16T22:31:49Z
author_association: OWNER
body:
  It looks like SAM - AWS Serverless Application Model - is the currently recommended way to deploy Python apps to Lambda from the command-line: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-hello-world.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645047703
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA0NzcwMw==
created_at: 2020-06-16T22:40:19Z
updated_at: 2020-06-16T22:40:19Z
author_association: OWNER
body:
  Installed SAM:
  ```
  brew tap aws/tap
  brew install aws-sam-cli
  sam --version
  SAM CLI, version 0.52.0
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645048062
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA0ODA2Mg==
created_at: 2020-06-16T22:41:33Z
updated_at: 2020-06-16T22:41:33Z
author_association: OWNER
body:
  ```
  simon@Simons-MacBook-Pro /tmp % sam init

  	SAM CLI now collects telemetry to better understand customer needs.

  	You can OPT OUT and disable telemetry collection by setting the
  	environment variable SAM_CLI_TELEMETRY=0 in your shell.
  	Thanks for your help!

  	Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html

  Which template source would you like to use?
  	1 - AWS Quick Start Templates
  	2 - Custom Template Location
  Choice: 1

  Which runtime would you like to use?
  	1 - nodejs12.x
  	2 - python3.8
  	3 - ruby2.7
  	4 - go1.x
  	5 - java11
  	6 - dotnetcore3.1
  	7 - nodejs10.x
  	8 - python3.7
  	9 - python3.6
  	10 - python2.7
  	11 - ruby2.5
  	12 - java8
  	13 - dotnetcore2.1
  Runtime: 2

  Project name [sam-app]: datasette-proof-of-concept

  Cloning app templates from https://github.com/awslabs/aws-sam-cli-app-templates.git

  AWS quick start application templates:
  	1 - Hello World Example
  	2 - EventBridge Hello World
  	3 - EventBridge App from scratch (100+ Event Schemas)
  	4 - Step Functions Sample App (Stock Trader)
  Template selection: 1

  -----------------------
  Generating application:
  -----------------------
  Name: datasette-proof-of-concept
  Runtime: python3.8
  Dependency Manager: pip
  Application Template: hello-world
  Output Directory: .

  Next steps can be found in the README file at ./datasette-proof-of-concept/README.md
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645050948
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA1MDk0OA==
created_at: 2020-06-16T22:51:30Z
updated_at: 2020-06-16T22:52:30Z
author_association: OWNER
body:
  ```
  simon@Simons-MacBook-Pro datasette-proof-of-concept % sam build --use-container
  Starting Build inside a container
  Building function 'HelloWorldFunction'

  Fetching lambci/lambda:build-python3.8 Docker container image..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
  Mounting /private/tmp/datasette-proof-of-concept/hello_world as /tmp/samcli/source:ro,delegated inside runtime container

  Build Succeeded

  Built Artifacts  : .aws-sam/build
  Built Template   : .aws-sam/build/template.yaml

  Commands you can use next
  =========================
  [*] Invoke Function: sam local invoke
  [*] Deploy: sam deploy --guided
      
  Running PythonPipBuilder:ResolveDependencies
  Running PythonPipBuilder:CopySource
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645051370
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA1MTM3MA==
created_at: 2020-06-16T22:53:05Z
updated_at: 2020-06-16T22:53:05Z
author_association: OWNER
body:
  ```
  simon@Simons-MacBook-Pro datasette-proof-of-concept % sam local invoke
  Invoking app.lambda_handler (python3.8)

  Fetching lambci/lambda:python3.8 Docker container image....................................................................................................................................................................................................................................
  Mounting /private/tmp/datasette-proof-of-concept/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
  START RequestId: 4616ab43-6882-1627-e5e3-5a29730d52f9 Version: $LATEST
  END RequestId: 4616ab43-6882-1627-e5e3-5a29730d52f9
  REPORT RequestId: 4616ab43-6882-1627-e5e3-5a29730d52f9	Init Duration: 140.84 ms	Duration: 2.49 ms	Billed Duration: 100 ms	Memory Size: 128 MBMax Memory Used: 25 MB	

  {"statusCode":200,"body":"{\"message\": \"hello world\"}"}
  simon@Simons-MacBook-Pro datasette-proof-of-concept % sam local invoke
  Invoking app.lambda_handler (python3.8)

  Fetching lambci/lambda:python3.8 Docker container image......
  Mounting /private/tmp/datasette-proof-of-concept/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
  START RequestId: 3189df2f-e9c0-1be4-b9ac-f329c5fcd067 Version: $LATEST
  END RequestId: 3189df2f-e9c0-1be4-b9ac-f329c5fcd067
  REPORT RequestId: 3189df2f-e9c0-1be4-b9ac-f329c5fcd067	Init Duration: 87.22 ms	Duration: 2.34 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 25 MB	

  {"statusCode":200,"body":"{\"message\": \"hello world\"}"}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645051972
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA1MTk3Mg==
created_at: 2020-06-16T22:55:04Z
updated_at: 2020-06-16T22:55:04Z
author_association: OWNER
body:
  ```
  simon@Simons-MacBook-Pro datasette-proof-of-concept % sam deploy --guided

  Configuring SAM deploy
  ======================

  	Looking for samconfig.toml :  Not found

  	Setting default arguments for 'sam deploy'
  	=========================================
  	Stack Name [sam-app]: datasette-proof-of-concept
  	AWS Region [us-east-1]: 
  	#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
  	Confirm changes before deploy [y/N]: y
  	#SAM needs permission to be able to create roles to connect to the resources in your template
  	Allow SAM CLI IAM role creation [Y/n]: y
  	HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
  	Save arguments to samconfig.toml [Y/n]: y
  Error: Failed to create managed resources: Unable to locate credentials
  ```
  I need to get my AWS credentials sorted. I'm going to follow https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html and https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-set-up-credentials.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645053923
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA1MzkyMw==
created_at: 2020-06-16T23:01:49Z
updated_at: 2020-06-16T23:01:49Z
author_association: OWNER
body:
  I used https://console.aws.amazon.com/billing/home?#/account and activated "IAM user/role access to billing information" - what a puzzling first step!

  I created a new user with AWS console access (which means access to the web UI) called `simon-administrator` and set a password. I created an `Administrators` group with `AdministratorAccess`.

  <img width="1002" alt="Banners_and_Alerts_and_IAM_Management_Console" src="https://user-images.githubusercontent.com/9599/84836494-abef0d80-afea-11ea-8cb9-14bee79439db.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645054206
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA1NDIwNg==
created_at: 2020-06-16T23:02:54Z
updated_at: 2020-06-16T23:04:59Z
author_association: OWNER
body:
  I think I need to sign in to the AWS console with this new `simon-administrator` account and create IAM credentials for it.

  ... for which I needed my root "account ID" - a 12 digit number - to use on the IAM login form.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645055200
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA1NTIwMA==
created_at: 2020-06-16T23:05:48Z
updated_at: 2020-06-16T23:05:48Z
author_association: OWNER
body:
  Logged in as `simon-administrator` I'm using https://console.aws.amazon.com/iam/home?region=us-east-2#/security_credentials to create credentials:

  <img width="1079" alt="Banners_and_Alerts_and_IAM_Management_Console" src="https://user-images.githubusercontent.com/9599/84836752-3c2d5280-afeb-11ea-8314-e44bf689aba1.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645056636
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA1NjYzNg==
created_at: 2020-06-16T23:10:22Z
updated_at: 2020-06-16T23:10:22Z
author_association: OWNER
body:
  Clicking that button generated me an access key ID / access key secret pair. Dropping those into `~/.aws/credentials` using this format:
  ```
  [default]
  aws_access_key_id = your_access_key_id
  aws_secret_access_key = your_secret_access_key
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645058617
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA1ODYxNw==
created_at: 2020-06-16T23:17:09Z
updated_at: 2020-06-16T23:17:09Z
author_association: OWNER
body:
  OK, `sam deploy --guided` now works!
  ```
  simon@Simons-MacBook-Pro datasette-proof-of-concept % sam deploy --guided

  Configuring SAM deploy
  ======================

  	Looking for samconfig.toml :  Not found

  	Setting default arguments for 'sam deploy'
  	=========================================
  	Stack Name [sam-app]: datasette-proof-of-concept
  	AWS Region [us-east-1]: 
  	#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
  	Confirm changes before deploy [y/N]: 
  	#SAM needs permission to be able to create roles to connect to the resources in your template
  	Allow SAM CLI IAM role creation [Y/n]: 
  	HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
  	Save arguments to samconfig.toml [Y/n]: 

  	Looking for resources needed for deployment: Not found.
  	Creating the required resources...
  	Successfully created!

  		Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-1ksajo4h62s07
  		A different default S3 bucket can be set in samconfig.toml

  	Saved arguments to config file
  	Running 'sam deploy' for future deployments will use the parameters saved above.
  	The above parameters can be changed by modifying samconfig.toml
  	Learn more about samconfig.toml syntax at 
  	https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

  	Deploying with following values
  	===============================
  	Stack name                 : datasette-proof-of-concept
  	Region                     : us-east-1
  	Confirm changeset          : False
  	Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-1ksajo4h62s07
  	Capabilities               : ["CAPABILITY_IAM"]
  	Parameter overrides        : {}

  Initiating deployment
  =====================
  Uploading to datasette-proof-of-concept/0c208b5656a7aeb6186d49bebc595237  535344 / 535344.0  (100.00%)
  HelloWorldFunction may not have authorization defined.
  Uploading to datasette-proof-of-concept/14bd9ce3e21f9c88634d13c0c9b377e4.template  1147 / 1147.0  (100.00%)

  Waiting for changeset to be created..

  CloudFormation stack changeset
  ---------------------------------------------------------------------------------------------------------------------------------------------------------
  Operation                                           LogicalResourceId                                   ResourceType                                      
  ---------------------------------------------------------------------------------------------------------------------------------------------------------
  + Add                                               HelloWorldFunctionHelloWorldPermissionProd          AWS::Lambda::Permission                           
  + Add                                               HelloWorldFunctionRole                              AWS::IAM::Role                                    
  + Add                                               HelloWorldFunction                                  AWS::Lambda::Function                             
  + Add                                               ServerlessRestApiDeployment47fc2d5f9d               AWS::ApiGateway::Deployment                       
  + Add                                               ServerlessRestApiProdStage                          AWS::ApiGateway::Stage                            
  + Add                                               ServerlessRestApi                                   AWS::ApiGateway::RestApi                          
  ---------------------------------------------------------------------------------------------------------------------------------------------------------

  Changeset created successfully. arn:aws:cloudformation:us-east-1:462092780466:changeSet/samcli-deploy1592349262/d685f2de-87c1-4b8e-b13a-67b94f8fc928


  2020-06-16 16:14:29 - Waiting for stack create/update to complete

  CloudFormation events from changeset
  ---------------------------------------------------------------------------------------------------------------------------------------------------------
  ResourceStatus                         ResourceType                           LogicalResourceId                      ResourceStatusReason                 
  ---------------------------------------------------------------------------------------------------------------------------------------------------------
  CREATE_IN_PROGRESS                     AWS::IAM::Role                         HelloWorldFunctionRole                 -                                    
  CREATE_IN_PROGRESS                     AWS::IAM::Role                         HelloWorldFunctionRole                 Resource creation Initiated          
  CREATE_COMPLETE                        AWS::IAM::Role                         HelloWorldFunctionRole                 -                                    
  CREATE_IN_PROGRESS                     AWS::Lambda::Function                  HelloWorldFunction                     Resource creation Initiated          
  CREATE_IN_PROGRESS                     AWS::Lambda::Function                  HelloWorldFunction                     -                                    
  CREATE_COMPLETE                        AWS::Lambda::Function                  HelloWorldFunction                     -                                    
  CREATE_IN_PROGRESS                     AWS::ApiGateway::RestApi               ServerlessRestApi                      Resource creation Initiated          
  CREATE_IN_PROGRESS                     AWS::ApiGateway::RestApi               ServerlessRestApi                      -                                    
  CREATE_COMPLETE                        AWS::ApiGateway::RestApi               ServerlessRestApi                      -                                    
  CREATE_IN_PROGRESS                     AWS::Lambda::Permission                HelloWorldFunctionHelloWorldPermissi   -                                    
                                                                                onProd                                                                      
  CREATE_IN_PROGRESS                     AWS::ApiGateway::Deployment            ServerlessRestApiDeployment47fc2d5f9   -                                    
                                                                                d                                                                           
  CREATE_COMPLETE                        AWS::ApiGateway::Deployment            ServerlessRestApiDeployment47fc2d5f9   -                                    
                                                                                d                                                                           
  CREATE_IN_PROGRESS                     AWS::ApiGateway::Deployment            ServerlessRestApiDeployment47fc2d5f9   Resource creation Initiated          
                                                                                d                                                                           
  CREATE_IN_PROGRESS                     AWS::Lambda::Permission                HelloWorldFunctionHelloWorldPermissi   Resource creation Initiated          
                                                                                onProd                                                                      
  CREATE_IN_PROGRESS                     AWS::ApiGateway::Stage                 ServerlessRestApiProdStage             -                                    
  CREATE_COMPLETE                        AWS::ApiGateway::Stage                 ServerlessRestApiProdStage             -                                    
  CREATE_IN_PROGRESS                     AWS::ApiGateway::Stage                 ServerlessRestApiProdStage             Resource creation Initiated          
  CREATE_COMPLETE                        AWS::Lambda::Permission                HelloWorldFunctionHelloWorldPermissi   -                                    
                                                                                onProd                                                                      
  CREATE_COMPLETE                        AWS::CloudFormation::Stack             datasette-proof-of-concept             -                                    
  ---------------------------------------------------------------------------------------------------------------------------------------------------------

  CloudFormation outputs from deployed stack
  ---------------------------------------------------------------------------------------------------------------------------------------------------------
  Outputs                                                                                                                                                 
  ---------------------------------------------------------------------------------------------------------------------------------------------------------
  Key                 HelloWorldFunctionIamRole                                                                                                           
  Description         Implicit IAM Role created for Hello World function                                                                                  
  Value               arn:aws:iam::462092780466:role/datasette-proof-of-concept-HelloWorldFunctionRole-8MIDNIV5ECA6                                       

  Key                 HelloWorldApi                                                                                                                       
  Description         API Gateway endpoint URL for Prod stage for Hello World function                                                                    
  Value               https://q7lymja3sj.execute-api.us-east-1.amazonaws.com/Prod/hello/                                                                  

  Key                 HelloWorldFunction                                                                                                                  
  Description         Hello World Lambda Function ARN                                                                                                     
  Value               arn:aws:lambda:us-east-1:462092780466:function:datasette-proof-of-concept-HelloWorldFunction-QTF78ZEUDCB                            
  ---------------------------------------------------------------------------------------------------------------------------------------------------------

  Successfully created/updated stack - datasette-proof-of-concept in us-east-1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645058947
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA1ODk0Nw==
created_at: 2020-06-16T23:18:18Z
updated_at: 2020-06-16T23:18:18Z
author_association: OWNER
body:
  https://q7lymja3sj.execute-api.us-east-1.amazonaws.com/Prod/hello/

  That's a pretty ugly URL. I'm not sure how to get rid of the `/Prod/` prefix on it. Might have to use the `base_url` setting to get something working: https://datasette.readthedocs.io/en/stable/config.html#base-url                                                      

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645059663
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA1OTY2Mw==
created_at: 2020-06-16T23:20:46Z
updated_at: 2020-06-16T23:20:46Z
author_association: OWNER
body:
  I added an exclamation mark to hello world and ran `sam deploy` again. https://q7lymja3sj.execute-api.us-east-1.amazonaws.com/Prod/hello/ still shows the old message.

  Running `sam build --use-container` first and then `sam deploy` did the right thing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645060598
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA2MDU5OA==
created_at: 2020-06-16T23:24:01Z
updated_at: 2020-06-16T23:24:01Z
author_association: OWNER
body:
  I changed `requirements.txt` to this:
  ```
  datasette
  mangum
  ```
  And `app.py` to this:
  ```python
  from datasette.app import Datasette
  from mangum import Mangum


  datasette = Datasette([], memory=True)
  lambda_handler = Mangum(datasette.app())
  ```
  But then when I ran `sam build --use-container` I got this:
  ```
  simon@Simons-MacBook-Pro datasette-proof-of-concept % sam build --use-container
  Starting Build inside a container
  Building function 'HelloWorldFunction'

  Fetching lambci/lambda:build-python3.8 Docker container image......
  Mounting /private/tmp/datasette-proof-of-concept/hello_world as /tmp/samcli/source:ro,delegated inside runtime container

  Build Failed
  Running PythonPipBuilder:ResolveDependencies
  Error: PythonPipBuilder:ResolveDependencies - {uvloop==0.14.0(wheel)}
  ```
  `uvloop` isn't actually necessary for this project, since it's used by `uvicorn` which isn't needed if Lambda is serving ASGI traffic directly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645061088
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA2MTA4OA==
created_at: 2020-06-16T23:25:41Z
updated_at: 2020-06-16T23:25:41Z
author_association: OWNER
body:
  Someone else ran into this problem: https://github.com/iwpnd/fastapi-aws-lambda-example/issues/1

  So I need to be able to pip install MOST of Datasette, but skip `uvicorn`. Tricky. I'll try installing a custom fork?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645062266
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA2MjI2Ng==
created_at: 2020-06-16T23:30:12Z
updated_at: 2020-06-16T23:33:12Z
author_association: OWNER
body:
  OK, changed `requirements.txt` to this:
  ```
  https://github.com/simonw/datasette/archive/no-uvicorn.zip
  mangum
  ```
  No `sam build --use-container` runs without errors. Ran `sam deploy` too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645063058
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA2MzA1OA==
created_at: 2020-06-16T23:32:57Z
updated_at: 2020-06-16T23:32:57Z
author_association: OWNER
body:
  https://q7lymja3sj.execute-api.us-east-1.amazonaws.com/Prod/hello/ is now giving me a 500 internal server error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645063386
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA2MzM4Ng==
created_at: 2020-06-16T23:34:07Z
updated_at: 2020-06-16T23:34:07Z
author_association: OWNER
body:
  Tried `sam local invoke`:
  ```
  simon@Simons-MacBook-Pro datasette-proof-of-concept % sam local invoke
  Invoking app.lambda_handler (python3.8)

  Fetching lambci/lambda:python3.8 Docker container image......
  Mounting /private/tmp/datasette-proof-of-concept/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
  START RequestId: 7c04480b-5d42-168e-dec0-4e8bf34fa596 Version: $LATEST
  [INFO]	2020-06-16T23:33:27.24Z	7c04480b-5d42-168e-dec0-4e8bf34fa596	Waiting for application startup.
  [INFO]	2020-06-16T23:33:27.24Z	7c04480b-5d42-168e-dec0-4e8bf34fa596	LifespanCycleState.STARTUP:  'lifespan.startup.complete' event received from application.
  [INFO]	2020-06-16T23:33:27.24Z	7c04480b-5d42-168e-dec0-4e8bf34fa596	Application startup complete.
  [INFO]	2020-06-16T23:33:27.24Z	7c04480b-5d42-168e-dec0-4e8bf34fa596	Waiting for application shutdown.
  [INFO]	2020-06-16T23:33:27.24Z	7c04480b-5d42-168e-dec0-4e8bf34fa596	LifespanCycleState.SHUTDOWN:  'lifespan.shutdown.complete' event received from application.
  [ERROR] KeyError: 'requestContext'
  Traceback (most recent call last):
    File "/var/task/mangum/adapter.py", line 110, in __call__
      return self.handler(event, context)
    File "/var/task/mangum/adapter.py", line 130, in handler
      if "eventType" in event["requestContext"]:
  END RequestId: 7c04480b-5d42-168e-dec0-4e8bf34fa596
  REPORT RequestId: 7c04480b-5d42-168e-dec0-4e8bf34fa596	Init Duration: 1120.76 ms	Duration: 7.08 ms	Billed Duration: 100 ms	Memory Size: 128 MBMax Memory Used: 47 MB	

  {"errorType":"KeyError","errorMessage":"'requestContext'","stackTrace":["  File \"/var/task/mangum/adapter.py\", line 110, in __call__\n    return self.handler(event, context)\n","  File \"/var/task/mangum/adapter.py\", line 130, in handler\n    if \"eventType\" in event[\"requestContext\"]:\n"]}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645064332
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA2NDMzMg==
created_at: 2020-06-16T23:37:34Z
updated_at: 2020-06-16T23:37:34Z
author_association: OWNER
body:
  Just realized Colin Dellow reported an issue with Datasette and Mangum back in April - #719 - and has in fact been working on https://github.com/code402/datasette-lambda for a while!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/236#issuecomment-645066486
issue_url: https://api.github.com/repos/simonw/datasette/issues/236
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA2NjQ4Ng==
created_at: 2020-06-16T23:45:45Z
updated_at: 2020-06-16T23:45:45Z
author_association: OWNER
body:
  Hi Colin,

  Sorry I didn't see this sooner! I've just started digging into this myself, to try and play with the new EFS Lambda support: #850.

  Yes, uvloop is only needed because of uvicorn. I have a branch here that removes that dependency just for trying out Lambda: https://github.com/simonw/datasette/tree/no-uvicorn - so you can run `pip install https://github.com/simonw/datasette/archive/no-uvicorn.zip` to get that.

  I'm going to try out your `datasette-lambda` project next - really excited to see how far you've got with it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/236#issuecomment-645067611
issue_url: https://api.github.com/repos/simonw/datasette/issues/236
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA2NzYxMQ==
created_at: 2020-06-16T23:50:12Z
updated_at: 2020-06-16T23:50:59Z
author_association: OWNER
body:
  As for your other questions:

  > 1. I assume the goal is to have a CORS-friendly HTTPS endpoint that hosts the datasette service + user's db.

  Yes, exactly. I know this will limit the size of database that can be deployed (since Lambda has a 50MB total package limit as far as I can tell) but there are plenty of interesting databases that are small enough to fit there.

  The new EFS support for Lambda means that theoretically the size of database is now unlimited, which is really interesting. That's what got me inspired to take a look at a proof of concept in #850.

  > 2. If that's the goal, I think Lambda alone is insufficient. Lambda provides the compute fabric, but not the HTTP routing. You'd also need to add Application Load Balancer or API Gateway to provide an HTTP endpoint that routes to the lambda function.
  > 
  > Do you have a preference between ALB or API GW? ALB has better economics at scale, but has a minimum monthly cost. API GW has worse per-request economics, but scales to zero when no requests are happening.

  I personally like scale-to-zero because many of my projects are likely to receive very little traffic. So API GW first, and maybe ALB as an option later on for people operating at scale?

  > 3. Does Datasette have any native components, or is it all pure python? If it has native bits, they'll likely need to be recompiled to work on Amazon Linux 2.

  As you've found, the only native component is uvloop which is only needed if uvicorn is being used to serve requests.

  > 4. There are a few disparate services that need to be wired together to expose a Python service securely to the web. If I was doing this outside of the datasette publish system, I'd use an AWS CloudFormation template. Even within datasette, I think it still makes sense to use a CloudFormation template and just have the publish plugin invoke it (via the standard `aws` cli) with user-specified parameters. Does that sound reasonable to you?

  For the eventual "datasette publish lambda" command I want whatever results in the smallest amount of inconvenience for users. I've been trying out Amazon SAM in #850 and it requires users to run Docker on their machines, which is a pretty huge barrier to entry! I don't have much experience with CloudFormation but it's probably a better bet, especially if you can "pip install" the dependencies needed to deploy with it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/850#issuecomment-645068128
issue_url: https://api.github.com/repos/simonw/datasette/issues/850
node_id: MDEyOklzc3VlQ29tbWVudDY0NTA2ODEyOA==
created_at: 2020-06-16T23:52:16Z
updated_at: 2020-06-16T23:52:16Z
author_association: OWNER
body:
  https://aws.amazon.com/blogs/compute/announcing-http-apis-for-amazon-api-gateway/ looks very important here: AWS HTTP APIs were introduced in December 2019 and appear to be a third of the price of API Gateway.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/851#issuecomment-645293374
issue_url: https://api.github.com/repos/simonw/datasette/issues/851
node_id: MDEyOklzc3VlQ29tbWVudDY0NTI5MzM3NA==
created_at: 2020-06-17T10:32:02Z
updated_at: 2020-06-17T10:32:28Z
author_association: CONTRIBUTOR
body:
  Welp, I'm an idiot.

  Turns out I had a sneaky comma `,` after `sql` key:
  ```
  ... (:name, :url),
  ```
  which tells sqlite to expect another `values(...)` list.

  Correcting the SQL solved the issue. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47
node_id: MDEyOklzc3VlQ29tbWVudDY0NTUxMjEyNw==
created_at: 2020-06-17T17:24:22Z
updated_at: 2020-06-17T17:24:22Z
author_association: MEMBER
body:
  That means your version of SQLite is old enough that it doesn't support the FTS5 extension.

  Could you share what operating system you're running, and what the output is that you get from running this?

      python -c 'import sqlite3; print(sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0])'

  I can teach this tool to fall back on FTS4 if FTS5 isn't available.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645515103
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47
node_id: MDEyOklzc3VlQ29tbWVudDY0NTUxNTEwMw==
created_at: 2020-06-17T17:30:01Z
updated_at: 2020-06-17T17:30:01Z
author_association: NONE
body:
  It's the one with python3.7::

      >>> sqlite3.sqlite_version
      '3.11.0'

   
  On Wed, Jun 17, 2020 at 10:24 -0700, Simon Willison wrote:

  > That means your version of SQLite is old enough that it doesn't support the FTS5 extension.
  > 
  > Could you share what operating system you're running, and what the output is that you get from running this?
  > 
  >     python -c 'import sqlite3; print(sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0])'
  > 
  > I can teach this tool to fall back on FTS4 if FTS5 isn't available.
  > 
  > -- 
  > You are receiving this because you authored the thread.
  > Reply to this email directly or view it on GitHub:
  > https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645599881
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47
node_id: MDEyOklzc3VlQ29tbWVudDY0NTU5OTg4MQ==
created_at: 2020-06-17T20:13:48Z
updated_at: 2020-06-17T20:13:48Z
author_association: MEMBER
body:
  I've now figured out how to compile specific SQLite versions to help replicate this problem: https://github.com/simonw/til/blob/master/sqlite/ld-preload.md

  Next step: replicate the problem!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/852#issuecomment-645785830
issue_url: https://api.github.com/repos/simonw/datasette/issues/852
node_id: MDEyOklzc3VlQ29tbWVudDY0NTc4NTgzMA==
created_at: 2020-06-18T05:37:00Z
updated_at: 2020-06-18T05:37:00Z
author_association: OWNER
body:
  The easiest way to do this would be with a new plugin hook:

      def canned_queries(datasette, database):
          """Return a list of canned query definitions
          or an awaitable function that returns them"

  Another approach would be to make the whole of `metadata.json` customizable by plugins.

  I think I like the dedicated `canned_queries` option better. I'm not happy with the way metadata keeps growing - see #493 - so adding a dedicated hook would be more future proof against other changes I might make to the metadata mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/853#issuecomment-646140022
issue_url: https://api.github.com/repos/simonw/datasette/issues/853
node_id: MDEyOklzc3VlQ29tbWVudDY0NjE0MDAyMg==
created_at: 2020-06-18T16:21:53Z
updated_at: 2020-06-18T16:21:53Z
author_association: OWNER
body:
  I have a test that demonstrates this working, but also demonstrates that the CSRF protection from #798 makes this really tricky to work with. I'd like to improve that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/835#issuecomment-646172200
issue_url: https://api.github.com/repos/simonw/datasette/issues/835
node_id: MDEyOklzc3VlQ29tbWVudDY0NjE3MjIwMA==
created_at: 2020-06-18T16:57:45Z
updated_at: 2020-06-18T16:57:45Z
author_association: OWNER
body:
  I think there are a couple of steps to this one.

  The nature of CSRF is that it's about hijacking existing authentication credentials. If your Datasette site runs without any authentication plugins at all CSRF protection isn't actually useful.

  Some POST endpoints should be able to opt-out of CSRF protection entirely. A writable canned query that accepts anonymous poll submissions for example might determine that CSRF is not needed.

  If a plugin adds `Authorization: Bearer xxx` token support that plugin should also be able to specify that CSRF protection can be skipped. https://github.com/simonw/datasette-auth-tokens could do this.

  This means I need two new mechanisms:

  - A way for wrapped views to indicate "actually don't CSRF protect me". I'm not sure how feasible this is without a major redesign, since the decision to return a 403 forbidden status is made before the wrapped function has even been called.
  - A way for authentication plugins like `datasette-auth-tokens` to say "CSRF protection is not needed for this request". This is a bit tricky too, since right now the `actor_from_request` hook doesn't have a channel for information other than returning the actor dictionary.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/835#issuecomment-646204308
issue_url: https://api.github.com/repos/simonw/datasette/issues/835
node_id: MDEyOklzc3VlQ29tbWVudDY0NjIwNDMwOA==
created_at: 2020-06-18T17:32:41Z
updated_at: 2020-06-18T17:32:41Z
author_association: OWNER
body:
  The only way I can think of for a view to opt-out of CSRF protection is for them to be able to reconfigure the `asgi-csrf` middleware to skip specific URL patterns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/835#issuecomment-646209520
issue_url: https://api.github.com/repos/simonw/datasette/issues/835
node_id: MDEyOklzc3VlQ29tbWVudDY0NjIwOTUyMA==
created_at: 2020-06-18T17:39:30Z
updated_at: 2020-06-18T17:40:53Z
author_association: OWNER
body:
  `datasette-auth-tokens` could switch to using `asgi_wrapper` instead of `actor_from_request` - then it could add a `scope["skip_csrf"] = True` scope property to indicate that CSRF should not be protected.

  Since `asgi_wrapper` wraps the CSRF protection middleware changes made to the `scope` by an `asgi_wrapper` will be visible to the CSRF middleware:

  https://github.com/simonw/datasette/blob/d2aef9f7ef30fa20b1450cd181cf803f44fb4e21/datasette/app.py#L877-L888

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/835#issuecomment-646214158
issue_url: https://api.github.com/repos/simonw/datasette/issues/835
node_id: MDEyOklzc3VlQ29tbWVudDY0NjIxNDE1OA==
created_at: 2020-06-18T17:48:45Z
updated_at: 2020-06-18T17:48:45Z
author_association: OWNER
body:
  I wonder if it's safe to generically say "Don't do CSRF protection on any request that includes a `Authorization: Bearer...` header - because it's not possible for a regular browser to send that header since the format is different from the header used in browser-based HTTP basic auth?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/835#issuecomment-646216934
issue_url: https://api.github.com/repos/simonw/datasette/issues/835
node_id: MDEyOklzc3VlQ29tbWVudDY0NjIxNjkzNA==
created_at: 2020-06-18T17:54:14Z
updated_at: 2020-06-18T17:54:14Z
author_association: OWNER
body:
  > if you did Origin based CSRF checks, then could the absence of an Origin header be used?
  https://twitter.com/cnorthwood/status/1273674392757829632

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/835#issuecomment-646217766
issue_url: https://api.github.com/repos/simonw/datasette/issues/835
node_id: MDEyOklzc3VlQ29tbWVudDY0NjIxNzc2Ng==
created_at: 2020-06-18T17:55:54Z
updated_at: 2020-06-18T17:56:04Z
author_association: OWNER
body:
  Idea: a mechanism where the `asgi_csrf()` can take an optional `should_protect()` callback function which gets called with the `scope` and decides if the current request should be protected or not. It can then look at headers and paths and suchlike and make its own decisions. Datasette could then provide a `should_protect()` callback which can interact with plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-646238702
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY0NjIzODcwMg==
created_at: 2020-06-18T18:39:07Z
updated_at: 2020-06-18T18:39:07Z
author_association: OWNER
body:
  It would be nice if Datasette didn't have to do any additional work to find e.g. `_request_ip` if that parameter turned out not to be used by the query.

  Could I do this with a custom class that implements `__getitem__()` and then gets passed as SQLite arguments?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-646242172
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI0MjE3Mg==
created_at: 2020-06-18T18:46:06Z
updated_at: 2020-06-18T18:53:31Z
author_association: OWNER
body:
  Yes that can work - and using `__missing__` (new in Python 3) is nicer because then the regular dictionary gets checked first:
  ```python
  import sqlite3

  conn = sqlite3.connect(":memory:")


  class Magic(dict):
      def __missing__(self, key):
          return key.upper()


  conn.execute("select :name", Magic()).fetchall()
  ```
  Outputs:
  ```
  [('NAME',)]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-646246062
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI0NjA2Mg==
created_at: 2020-06-18T18:54:41Z
updated_at: 2020-06-18T18:54:41Z
author_association: OWNER
body:
  The `_actor_id` param makes this a bit trickier, because we can't just say "if you see an unknown parameter called X call this function" - our magic parameter logic isn't adding single parameters, it might add a whole family of them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-646264051
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI2NDA1MQ==
created_at: 2020-06-18T19:32:13Z
updated_at: 2020-06-18T19:32:37Z
author_association: OWNER
body:
  If every magic parameter has a prefix and suffix, like `_request_ip` and `_actor_id`, then plugins could register a function for a prefix. Register a function to `_actor` and `actor("id")`will be called for `_actor_id`.

  But does it make sense for every magic parameter to be of form `_a_b`? I think so.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-646270702
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI3MDcwMg==
created_at: 2020-06-18T19:47:19Z
updated_at: 2020-06-24T18:48:48Z
author_association: OWNER
body:
  Brainstorming more potential magic parameters:

  * `_actor_id`
  * `_actor_name`
  * `_request_ip`
  * `_request_user_agent`
  * `_cookie_cookiename`
  * `_signedcookie_cookiename` - reading signed cookies would be cool, not sure how to specify namespace though, maybe always use the same one? Or have the namespace come last, `_signedcookie_cookiename_mynamespace`. Might not need special signed cookie support since `actor` is already usually from a signed cookie.
  * `_timestamp_unix` (not happy with these names yet)
  * `_timestamp_localtime`
  * `_timestamp_datetime`
  * `_timestamp_utc`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/842#issuecomment-646271834
issue_url: https://api.github.com/repos/simonw/datasette/issues/842
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI3MTgzNA==
created_at: 2020-06-18T19:49:41Z
updated_at: 2020-06-24T18:49:22Z
author_association: OWNER
body:
  But then what kind of magic parameters might plugins want to add?

  Here's a crazy idea: `_scrapedcontent_url` - it would look for the `url` column on the data being inserted, scrape the content from it and insert that. This does suggest that the magic resolving function `scrapedcontent()` would need to optionally be sent the full row dictionary being inserted too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646273035
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI3MzAzNQ==
created_at: 2020-06-18T19:52:28Z
updated_at: 2020-06-18T19:52:28Z
author_association: OWNER
body:
  I'd like this soon, because I want to start experimenting with things like #852 and #842 without shipping those plugin hooks in a full stable release.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646276150
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI3NjE1MA==
created_at: 2020-06-18T19:59:17Z
updated_at: 2020-06-18T19:59:17Z
author_association: OWNER
body:
  Relevant PEP: https://www.python.org/dev/peps/pep-0440/

  Django's implementation dates back 8 years: https://github.com/django/django/commit/40f0ecc56a23d35c2849f8e79276f6d8931412d1

  From the PEP:

  > Implicit pre-release number
  >
  > Pre releases allow omitting the numeral in which case it is implicitly assumed to be 0. The normal form for this is to include the 0 explicitly. This allows versions such as 1.2a which is normalized to 1.2a0.

  I'm going to habitually include the 0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646277155
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI3NzE1NQ==
created_at: 2020-06-18T20:01:31Z
updated_at: 2020-06-18T20:01:31Z
author_association: OWNER
body:
  I thought I might have to update a regex (my CircleCI configs won't match on `a0`, [example](https://github.com/simonw/datasette-publish-now/blob/420f349b278857f62183d8e9835d64f116758be7/.circleci/config.yml#L22)) but it turns out Travis is currently configured to treat ALL tags as potential releases:

  https://github.com/simonw/datasette/blob/6151c25a5a8d566c109af296244b9267c536bd9a/.travis.yml#L21-L35

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646277680
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI3NzY4MA==
created_at: 2020-06-18T20:02:42Z
updated_at: 2020-06-18T20:02:42Z
author_association: OWNER
body:
  So I think if I push a tag of `0.45a0` everything might just work - Travis will build it, push the build to PyPI, PyPI won't treat it as a stable release.

  Except... I don't want to push alphas as Docker images - so I need to fix this code:

  https://github.com/simonw/datasette/blob/6151c25a5a8d566c109af296244b9267c536bd9a/.travis.yml#L34-L43

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646278801
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI3ODgwMQ==
created_at: 2020-06-18T20:05:18Z
updated_at: 2020-06-18T20:05:18Z
author_association: OWNER
body:
  Travis conditions documentation: https://docs.travis-ci.com/user/conditions-v1

  These look useful:
  ```
  branch =~ /^(one|two)-three$/
  (tag =~ ^v) AND (branch = master)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646279428
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI3OTQyOA==
created_at: 2020-06-18T20:06:43Z
updated_at: 2020-06-18T20:06:43Z
author_association: OWNER
body:
  I'm going to try this on a separate repository so I don't accidentally publish a Datasette release I didn't mean to publish!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/835#issuecomment-646288146
issue_url: https://api.github.com/repos/simonw/datasette/issues/835
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI4ODE0Ng==
created_at: 2020-06-18T20:26:22Z
updated_at: 2020-06-18T20:26:31Z
author_association: OWNER
body:
  Useful tip from Carlton Gibson: https://twitter.com/carltongibson/status/1273680590672453632

  > DRF makes ALL views CSRF exempt and then enforces CSRF if you're using Session auth only. 
  >
  > View: https://github.com/encode/django-rest-framework/blob/e18e40d6ae42457f60ca9c68054ad40d15ba8433/rest_framework/views.py#L144
  > Auth: https://github.com/encode/django-rest-framework/blob/e18e40d6ae42457f60ca9c68054ad40d15ba8433/rest_framework/authentication.py#L130

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646290171
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI5MDE3MQ==
created_at: 2020-06-18T20:30:48Z
updated_at: 2020-06-18T20:30:48Z
author_association: OWNER
body:
  OK, I just shipped 0.2a0 of `datasette-render-images` -  https://pypi.org/project/datasette-render-images/ has no indication of that:

  <img width="1330" alt="Banners_and_Alerts_and_datasette-render-images_·_PyPI" src="https://user-images.githubusercontent.com/9599/85068682-acf97980-b167-11ea-863d-a169bd64c25a.png">

  But this page does: https://pypi.org/project/datasette-render-images/#history

  <img width="1382" alt="Banners_and_Alerts_and_datasette-render-images_·_PyPI" src="https://user-images.githubusercontent.com/9599/85068736-c1d60d00-b167-11ea-801b-ebfc4fddbbed.png">

  And https://pypi.org/project/datasette-render-images/0.2a0/ exists.

  In a fresh virtual environment `pip install datasette-render-images` gets 0.1.

  `pip install datasette-render-images==0.2a0` gets 0.2a0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646291309
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI5MTMwOQ==
created_at: 2020-06-18T20:33:31Z
updated_at: 2020-06-18T20:33:31Z
author_association: OWNER
body:
  One more experiment: I'm going to ship `datasette-render-images` 0.2 and see if that works correctly - including printing out the new debug section I put in the Travis config here: https://github.com/simonw/datasette-render-images/blob/6b5f22dab75ca364f671f5597556d2665a251bd8/.travis.yml#L35-L39 - which should demonstrate if my conditional for pushing to Docker Hub will work or not.

  In the alpha releasing run on Travis that echo statement did NOT execute: https://travis-ci.com/github/simonw/datasette-render-images/builds/172116625

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646292578
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI5MjU3OA==
created_at: 2020-06-18T20:36:22Z
updated_at: 2020-06-18T20:36:22Z
author_association: OWNER
body:
  https://travis-ci.com/github/simonw/datasette-render-images/builds/172118541 demonstrates that the alpha/beta conditional is working as intended:

  <img width="1066" alt="Banners_and_Alerts_and_Build__13_-_simonw_datasette-render-images_-_Travis_CI" src="https://user-images.githubusercontent.com/9599/85069271-b0413500-b168-11ea-8a40-7fe2ff0389b2.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646293029
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI5MzAyOQ==
created_at: 2020-06-18T20:37:28Z
updated_at: 2020-06-18T20:37:46Z
author_association: OWNER
body:
  Here's the Read The Docs documentation on versioned releases: https://docs.readthedocs.io/en/stable/versions.html

  It looks like they do the right thing:

  > We in fact are parsing your tag names against the rules given by PEP 440. This spec allows “normal” version numbers like 1.4.2 as well as pre-releases. An alpha version or a release candidate are examples of pre-releases and they look like this: 2.0a1.
  > 
  > We only consider non pre-releases for the stable version of your documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646293670
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjI5MzY3MA==
created_at: 2020-06-18T20:38:50Z
updated_at: 2020-06-18T20:38:50Z
author_association: OWNER
body:
  https://pypi.org/project/datasette-render-images/#history worked:

  <img width="904" alt="Banners_and_Alerts_and_datasette-render-images_·_PyPI" src="https://user-images.githubusercontent.com/9599/85069459-02825600-b169-11ea-9747-cb24e26888d1.png">

  I'm now confident enough that I'll make these changes and ship an alpha of Datasette itself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646302909
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjMwMjkwOQ==
created_at: 2020-06-18T21:00:02Z
updated_at: 2020-06-18T21:00:02Z
author_association: OWNER
body:
  Alpha release is running through Travis now: https://travis-ci.org/github/simonw/datasette/builds/699864168

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646303240
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjMwMzI0MA==
created_at: 2020-06-18T21:00:41Z
updated_at: 2020-06-18T21:00:41Z
author_association: OWNER
body:
  New documentation about the alpha/beta releases: https://datasette.readthedocs.io/en/latest/contributing.html#contributing-alpha-beta

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/835#issuecomment-646307083
issue_url: https://api.github.com/repos/simonw/datasette/issues/835
node_id: MDEyOklzc3VlQ29tbWVudDY0NjMwNzA4Mw==
created_at: 2020-06-18T21:09:35Z
updated_at: 2020-06-18T21:09:35Z
author_association: OWNER
body:
  So maybe one really easy fix here is to disable CSRF checks entirely for any request that doesn't have any cookies? Also suggested here: https://twitter.com/mrkurt/status/1273682965168603137

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/835#issuecomment-646308467
issue_url: https://api.github.com/repos/simonw/datasette/issues/835
node_id: MDEyOklzc3VlQ29tbWVudDY0NjMwODQ2Nw==
created_at: 2020-06-18T21:12:50Z
updated_at: 2020-06-18T21:12:50Z
author_association: OWNER
body:
  Problem there is Login CSRF attacks: https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#login-csrf - I still want to perform CSRF checks on login forms, even though the user may not yet have any cookies.

  Maybe I can turn off CSRF checks for cookie-free requests but allow login forms to specifically opt back in to CSRF protection?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646319315
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjMxOTMxNQ==
created_at: 2020-06-18T21:38:56Z
updated_at: 2020-06-18T21:38:56Z
author_association: OWNER
body:
  This worked!

  https://pypi.org/project/datasette/#history

  <img width="924" alt="Banners_and_Alerts_and_datasette_·_PyPI" src="https://user-images.githubusercontent.com/9599/85074185-fd290980-b170-11ea-86b9-eebf8a30e94e.png">

  https://github.com/simonw/datasette/releases/tag/0.45a0 is my manually created GitHub prerelease.

  https://datasette.readthedocs.io/en/latest/changelog.html#a0-2020-06-18 has the release notes.

  A shame Read The Docs doesn't seem to build the docs for these releases -it's not showing the tag in the releases pane here:

  <img width="323" alt="Changelog_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/85074331-39f50080-b171-11ea-99b9-c70f8a89727b.png">

  Also the new tag isn't an option in the Build menu on https://readthedocs.org/projects/datasette/builds/

  Not a big problem though since the "latest" tag on Read The Docs will still carry the in-development documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/807#issuecomment-646320237
issue_url: https://api.github.com/repos/simonw/datasette/issues/807
node_id: MDEyOklzc3VlQ29tbWVudDY0NjMyMDIzNw==
created_at: 2020-06-18T21:41:16Z
updated_at: 2020-06-18T21:41:16Z
author_association: OWNER
body:
  https://pypi.org/project/datasette/0.45a0/ is the release on PyPI.

  And in a fresh virtual environment:

  ```
  $ pip install datasette==0.45a0
  ...
  $ datasette --version
  datasette, version 0.45a0
  ```
  But running `pip install datasette` still gets 0.44.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/852#issuecomment-646329456
issue_url: https://api.github.com/repos/simonw/datasette/issues/852
node_id: MDEyOklzc3VlQ29tbWVudDY0NjMyOTQ1Ng==
created_at: 2020-06-18T22:07:09Z
updated_at: 2020-06-18T22:07:37Z
author_association: OWNER
body:
  It would be neat if the queries returned by this hook could be restricted to specific users. I think I can do that by returning an "allow" block as part of the query.

  But... what if we allow users to save private queries and we might have thousands of users each with hundreds of saved queries?

  For that case it would be good if the plugin hook could take an optional `actor` parameter.

  This would also allow us to dynamically generate a canned query for "return the bookmarks belonging to this actor" or similar!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/852#issuecomment-646350530
issue_url: https://api.github.com/repos/simonw/datasette/issues/852
node_id: MDEyOklzc3VlQ29tbWVudDY0NjM1MDUzMA==
created_at: 2020-06-18T23:13:57Z
updated_at: 2020-06-18T23:14:11Z
author_association: OWNER
body:
  ```python
  @hookspec
  def canned_queries(datasette, database, actor):
      "Return a dictionary of canned query definitions or an awaitable function that returns them"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/852#issuecomment-646396499
issue_url: https://api.github.com/repos/simonw/datasette/issues/852
node_id: MDEyOklzc3VlQ29tbWVudDY0NjM5NjQ5OQ==
created_at: 2020-06-19T02:15:49Z
updated_at: 2020-06-19T02:15:58Z
author_association: OWNER
body:
  Released an alpha preview in https://github.com/simonw/datasette/releases/tag/0.45a1

  Wrote about this here: https://simonwillison.net/2020/Jun/19/datasette-alphas/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/852#issuecomment-646396690
issue_url: https://api.github.com/repos/simonw/datasette/issues/852
node_id: MDEyOklzc3VlQ29tbWVudDY0NjM5NjY5MA==
created_at: 2020-06-19T02:16:24Z
updated_at: 2020-06-19T02:16:24Z
author_association: OWNER
body:
  Documentation: https://datasette.readthedocs.io/en/latest/plugins.html#canned-queries-datasette-database-actor

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/849#issuecomment-646686493
issue_url: https://api.github.com/repos/simonw/datasette/issues/849
node_id: MDEyOklzc3VlQ29tbWVudDY0NjY4NjQ5Mw==
created_at: 2020-06-19T15:04:51Z
updated_at: 2020-06-19T15:04:51Z
author_association: OWNER
body:
  https://twitter.com/jaffathecake/status/1273983493006077952 concerns what happens to open pull requests - they will automatically close when you remove `master` unless you repoint them to `main` first.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/852#issuecomment-646760805
issue_url: https://api.github.com/repos/simonw/datasette/issues/852
node_id: MDEyOklzc3VlQ29tbWVudDY0Njc2MDgwNQ==
created_at: 2020-06-19T17:07:45Z
updated_at: 2020-06-19T17:07:45Z
author_association: OWNER
body:
  Plugin idea: `datasette-saved-queries` - it uses the `startup` hook to initialize a `saved_queries` table, then uses the `canned_queries` hook to add a writable canned query for saving records to that table.

  Then it returns any queries from that table as additional canned queries.

  Bonus idea: it could write the user's actor_id to a column if they are signed in, and provide a link to see "just my saved queries" in that case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/852#issuecomment-646905073
issue_url: https://api.github.com/repos/simonw/datasette/issues/852
node_id: MDEyOklzc3VlQ29tbWVudDY0NjkwNTA3Mw==
created_at: 2020-06-20T00:21:34Z
updated_at: 2020-06-20T00:22:28Z
author_association: OWNER
body:
  New repo: https://github.com/simonw/datasette-saved-queries - which I created using the new cookiecutter template at https://github.com/simonw/datasette-plugin

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/855#issuecomment-646928638
issue_url: https://api.github.com/repos/simonw/datasette/issues/855
node_id: MDEyOklzc3VlQ29tbWVudDY0NjkyODYzOA==
created_at: 2020-06-20T03:09:41Z
updated_at: 2020-06-20T03:09:41Z
author_association: OWNER
body:
  I've shipped the cookiecutter template and used it to build https://github.com/simonw/datasette-saved-queries - it's ready to add to the official documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/642#issuecomment-646930059
issue_url: https://api.github.com/repos/simonw/datasette/issues/642
node_id: MDEyOklzc3VlQ29tbWVudDY0NjkzMDA1OQ==
created_at: 2020-06-20T03:19:57Z
updated_at: 2020-06-20T03:19:57Z
author_association: OWNER
body:
  @psychemedia sorry I missed your comment before.

  Niche Museums is definitely the best example of custom templates at the moment: https://github.com/simonw/museums/tree/master/templates

  I want to comprehensively document the variables made available to custom templates before shipping Datasette 1.0 - just filed that as #857.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/642#issuecomment-646930160
issue_url: https://api.github.com/repos/simonw/datasette/issues/642
node_id: MDEyOklzc3VlQ29tbWVudDY0NjkzMDE2MA==
created_at: 2020-06-20T03:20:25Z
updated_at: 2020-06-20T03:20:25Z
author_association: OWNER
body:
  Shipped this today! https://github.com/simonw/datasette-plugin is a cookiecutter template for creating new plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/687#issuecomment-646938984
issue_url: https://api.github.com/repos/simonw/datasette/issues/687
node_id: MDEyOklzc3VlQ29tbWVudDY0NjkzODk4NA==
created_at: 2020-06-20T04:22:25Z
updated_at: 2020-06-20T04:23:02Z
author_association: OWNER
body:
  I think I want the "Plugin hooks" page to be top-level, parallel to "Plugins" and "Internals for Plugins". It's the page of documentation refer to most often so I don't want to have to click down a hierarchy from the side navigation to find it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/859#issuecomment-647135713
issue_url: https://api.github.com/repos/simonw/datasette/issues/859
node_id: MDEyOklzc3VlQ29tbWVudDY0NzEzNTcxMw==
created_at: 2020-06-21T14:30:02Z
updated_at: 2020-06-21T14:30:02Z
author_association: CONTRIBUTOR
body:
  Oops, the same method is called from both index and database pages. But removing select count queries speed up the page load quite a bit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/357#issuecomment-647189045
issue_url: https://api.github.com/repos/simonw/datasette/issues/357
node_id: MDEyOklzc3VlQ29tbWVudDY0NzE4OTA0NQ==
created_at: 2020-06-21T22:19:58Z
updated_at: 2020-06-21T22:19:58Z
author_association: OWNER
body:
  I'm going to take this in a different direction.

  I'm not happy with how `metadata.(json|yaml)` keeps growing new features. Rather than having a single plugin hook for all of `metadata.json` I'm going to split out the feature that shows actual real metadata for tables and databases - `source`, `license` etc - into its own plugin-powered mechanism.

  So I'm going to close this ticket and spin up a new one for that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/860#issuecomment-647189535
issue_url: https://api.github.com/repos/simonw/datasette/issues/860
node_id: MDEyOklzc3VlQ29tbWVudDY0NzE4OTUzNQ==
created_at: 2020-06-21T22:25:16Z
updated_at: 2020-06-21T22:25:27Z
author_association: OWNER
body:
  This is also relevant to #639, and may mean I can close that ticket in place of this one. I'm going to get this at least to a proof-of-concept stage first though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


