# issue_comments

html_url: https://github.com/simonw/datasette/issues/1067#issuecomment-719956184
issue_url: https://api.github.com/repos/simonw/datasette/issues/1067
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk1NjE4NA==
created_at: 2020-10-31T16:26:09Z
updated_at: 2020-10-31T16:26:09Z
author_association: OWNER
body:
  Should the hook provide an indication that it's running on a different type of page? I think yes for queries. Not sure about views  - they behave very much like tables, and the plugin can always introspect to see if something is a view if it needs to.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1067#issuecomment-719959419
issue_url: https://api.github.com/repos/simonw/datasette/issues/1067
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk1OTQxOQ==
created_at: 2020-10-31T16:53:42Z
updated_at: 2020-10-31T16:53:42Z
author_association: OWNER
body:
  For the 0.51 release I'm going to add tests that show this works on view pages. I won't implement it for query pages.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1067#issuecomment-719961701
issue_url: https://api.github.com/repos/simonw/datasette/issues/1067
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk2MTcwMQ==
created_at: 2020-10-31T17:11:59Z
updated_at: 2020-10-31T17:11:59Z
author_association: OWNER
body:
  It bothers me that these aren't visible in any public demos. Maybe `latest.datasette.io` should include the `my_plugins.py` and `my_plugins2.py` plugins?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1074#issuecomment-719963074
issue_url: https://api.github.com/repos/simonw/datasette/issues/1074
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk2MzA3NA==
created_at: 2020-10-31T17:23:48Z
updated_at: 2020-10-31T17:23:48Z
author_association: OWNER
body:
  Needs a way to login as root, seeing as several plugins only show extra content if the user is logged in as root.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1074#issuecomment-719965426
issue_url: https://api.github.com/repos/simonw/datasette/issues/1074
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk2NTQyNg==
created_at: 2020-10-31T17:45:00Z
updated_at: 2020-10-31T17:45:00Z
author_association: OWNER
body:
  This is working. Go to https://latest.datasette.io/login-as-root and click the button, then visit this page to see extra content added by plugins: https://latest.datasette.io/fixtures/compound_three_primary_keys

  ![latest-plugins](https://user-images.githubusercontent.com/9599/97786052-19e53d00-1b66-11eb-8268-b452e08965e3.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1067#issuecomment-719966176
issue_url: https://api.github.com/repos/simonw/datasette/issues/1067
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk2NjE3Ng==
created_at: 2020-10-31T17:51:31Z
updated_at: 2020-10-31T17:51:31Z
author_association: OWNER
body:
  Demo:

  - https://latest.datasette.io/fixtures/facetable?_bot=1
  - https://latest.datasette.io/fixtures/simple_view?_bot=1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1074#issuecomment-719977864
issue_url: https://api.github.com/repos/simonw/datasette/issues/1074
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk3Nzg2NA==
created_at: 2020-10-31T19:35:01Z
updated_at: 2020-10-31T19:35:01Z
author_association: OWNER
body:
  These plugins were not designed to be actually hosted online, so they do some nasty things like linking to the made-up `plugin-example.com` domain. I should fix that.

  <img width="853" alt="Datasette_Fixtures__fixtures" src="https://user-images.githubusercontent.com/9599/97788348-7b60d800-1b75-11eb-8519-a58e52108841.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1075#issuecomment-719980742
issue_url: https://api.github.com/repos/simonw/datasette/issues/1075
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk4MDc0Mg==
created_at: 2020-10-31T19:58:57Z
updated_at: 2020-10-31T19:58:57Z
author_association: OWNER
body:
  Sample traceback:
  ```
      <link rel="stylesheet" href="{{ urls.static('app.css') }}?{{ app_css_hash }}">
  /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/jinja2/asyncsupport.py:173: in auto_await
      if inspect.isawaitable(value):
  /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/inspect.py:226: in isawaitable
      isinstance(object, collections.abc.Awaitable))
  /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/abc.py:139: in __instancecheck__
      return _abc_instancecheck(cls, instance)
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

  cls = <class 'collections.abc.Awaitable'>
  subclass = <bound method PrefixedUrlString.__getattribute__.<locals>.method of '/-/static/app.css'>

      def __subclasscheck__(cls, subclass):
          """Override for issubclass(subclass, cls)."""
  >       return _abc_subclasscheck(cls, subclass)
  E       TypeError: issubclass() arg 1 must be a class
  ```
  This is within Jinja. It looks like Jinja really doesn't like methods that return non-string objects like `PrefixedUrlString`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1075#issuecomment-719981173
issue_url: https://api.github.com/repos/simonw/datasette/issues/1075
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk4MTE3Mw==
created_at: 2020-10-31T20:02:30Z
updated_at: 2020-10-31T20:03:45Z
author_association: OWNER
body:
  I wonder how Jinja's `Markup()` class works? It uses https://pypi.org/project/MarkupSafe/

  It's a subclass of `str`, defined here: https://github.com/pallets/markupsafe/blob/master/src/markupsafe/__init__.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1075#issuecomment-719983240
issue_url: https://api.github.com/repos/simonw/datasette/issues/1075
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk4MzI0MA==
created_at: 2020-10-31T20:18:49Z
updated_at: 2020-10-31T20:18:49Z
author_association: OWNER
body:
  Here's the core problem:
  ```
  (Pdb) isinstance('bob', collections.abc.Awaitable)
  False
  (Pdb) isinstance(PrefixedUrlString('bob'), collections.abc.Awaitable)
  *** TypeError: issubclass() arg 1 must be a class
  ```
  For some reason `isinstance()` does not like being handed an instance of PrefixedUrlString.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1075#issuecomment-719983484
issue_url: https://api.github.com/repos/simonw/datasette/issues/1075
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk4MzQ4NA==
created_at: 2020-10-31T20:20:28Z
updated_at: 2020-10-31T20:20:28Z
author_association: OWNER
body:
  It looks like this is specific to the way `PrefixedUrlString` is built.
  ```
  (Pdb) class Weird(str): pass
  (Pdb) isinstance(Weird('bob'), collections.abc.Awaitable)
  False
  ```
  So subclassing strings doesn't trigger this bug, but something about `PrefixedUrlString` causes the problem.

  Here's the current `PrefixedUrlString` implementation:

  https://github.com/simonw/datasette/blob/bf18b9ba175a7b25fb8b765847397dd6efb8bb7b/datasette/utils/__init__.py#L1015-L1035

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1075#issuecomment-719983565
issue_url: https://api.github.com/repos/simonw/datasette/issues/1075
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk4MzU2NQ==
created_at: 2020-10-31T20:21:03Z
updated_at: 2020-10-31T20:21:03Z
author_association: OWNER
body:
  Here's the output of `dir(str)`:

  `['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'capitalize', 'casefold', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index', 'isalnum', 'isalpha', 'isascii', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1075#issuecomment-719983750
issue_url: https://api.github.com/repos/simonw/datasette/issues/1075
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk4Mzc1MA==
created_at: 2020-10-31T20:22:29Z
updated_at: 2020-10-31T20:22:29Z
author_association: OWNER
body:
  I bet this is because I'm mucking around with one of those `__` methods. I'll try just doing the non-underscore methods instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719986698
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk4NjY5OA==
created_at: 2020-10-31T20:48:17Z
updated_at: 2020-10-31T20:48:17Z
author_association: OWNER
body:
  Here's the `datasette-edit-templates` plugin WIP I had before removing the hook: https://github.com/simonw/datasette-edit-templates/tree/82855c2612b84bc09c48fca885f831633a0d1552

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/838#issuecomment-719986904
issue_url: https://api.github.com/repos/simonw/datasette/issues/838
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk4NjkwNA==
created_at: 2020-10-31T20:50:41Z
updated_at: 2020-10-31T20:50:41Z
author_association: OWNER
body:
  OK, this should be working now. You can use the `datasette.urls.static_plugins()` method to generate the correct URLs in the `extra_css_urls` plugin hook: https://docs.datasette.io/en/latest/internals.html#datasette-urls

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1027#issuecomment-719988113
issue_url: https://api.github.com/repos/simonw/datasette/issues/1027
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk4ODExMw==
created_at: 2020-10-31T21:03:37Z
updated_at: 2020-10-31T21:03:37Z
author_association: OWNER
body:
  On my Mac, I run:

      datasette . -p 8009 --config base_url:/datasette-prefix/

  Then I edited `/usr/local/etc/httpd/httpd.conf` and add this section:
  ```
  LoadModule proxy_module lib/httpd/modules/mod_proxy.so
  LoadModule proxy_http_module lib/httpd/modules/mod_proxy_http.so
  ProxyPass          /datasette-prefix/    http://localhost:8009/datasette-prefix/
  ```
  I ran Apache in the foreground like so:

      apachectl -X

  Now hitting http://localhost:8081/datasette-prefix/fixtures/compound_three_primary_keys worked!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1048#issuecomment-719996693
issue_url: https://api.github.com/repos/simonw/datasette/issues/1048
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk5NjY5Mw==
created_at: 2020-10-31T22:32:09Z
updated_at: 2022-07-10T16:22:48Z
author_association: OWNER
body:
  The `row_path` part of these really isn't very user friendly, since you need to properly URL-encode the values. The safest way to do so is by calling this obscure, undocumented utility function: https://github.com/simonw/datasette/blob/f0bd2d05f5f7832df4879822afb99d2096c00d48/datasette/utils/__init__.py#L84-L98

  This feels like it should be improved before I turn it into a documented API.

  (Note that this API deals with a `row` that is a potentially-nested dictionary - not with a `sqlite3.Row` object.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/838#issuecomment-720354227
issue_url: https://api.github.com/repos/simonw/datasette/issues/838
node_id: MDEyOklzc3VlQ29tbWVudDcyMDM1NDIyNw==
created_at: 2020-11-02T09:33:58Z
updated_at: 2020-11-02T09:33:58Z
author_association: CONTRIBUTOR
body:
  Thanks; just a note that the `datasette.urls.static(path)` and `datasette.urls.static_plugins(plugin_name, path)` items both seem to be repeated and appear in the docs twice?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/596#issuecomment-720689653
issue_url: https://api.github.com/repos/simonw/datasette/issues/596
node_id: MDEyOklzc3VlQ29tbWVudDcyMDY4OTY1Mw==
created_at: 2020-11-02T19:53:36Z
updated_at: 2020-11-02T19:53:47Z
author_association: OWNER
body:
  In #998 I implemented a horizontal scrollbar for these tables, which is a big improvement - demo here: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1080#issuecomment-720695174
issue_url: https://api.github.com/repos/simonw/datasette/issues/1080
node_id: MDEyOklzc3VlQ29tbWVudDcyMDY5NTE3NA==
created_at: 2020-11-02T20:05:26Z
updated_at: 2020-11-02T20:05:26Z
author_association: OWNER
body:
  URL design:

  `/database/table/-/facet/colname`

  And for other types of facet (to be supported later):

  `/database/table/-/facet/colname?_type=m2m`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1080#issuecomment-720696827
issue_url: https://api.github.com/repos/simonw/datasette/issues/1080
node_id: MDEyOklzc3VlQ29tbWVudDcyMDY5NjgyNw==
created_at: 2020-11-02T20:08:49Z
updated_at: 2020-11-02T20:13:56Z
author_association: OWNER
body:
  Implementing pagination for facets will be interesting. Would be easier if I had a nicer reusable internal pagination mechanism, which is also needed for #856 (pagination of canned queries).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1080#issuecomment-720697226
issue_url: https://api.github.com/repos/simonw/datasette/issues/1080
node_id: MDEyOklzc3VlQ29tbWVudDcyMDY5NzIyNg==
created_at: 2020-11-02T20:09:38Z
updated_at: 2020-11-02T20:09:38Z
author_association: OWNER
body:
  Maybe this ends up being code that defers to a simulated canned query, rendered using the existing `query.html` template.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1080#issuecomment-720698577
issue_url: https://api.github.com/repos/simonw/datasette/issues/1080
node_id: MDEyOklzc3VlQ29tbWVudDcyMDY5ODU3Nw==
created_at: 2020-11-02T20:12:26Z
updated_at: 2020-11-02T20:12:26Z
author_association: OWNER
body:
  For regular column faceting, here's the query that is used:

  https://github.com/simonw/datasette/blob/13d1228d80c91d382a05b1a9549ed02c300ef851/datasette/facets.py#L196-L204

  Since it uses `order by count desc, value` maybe those values could be used to implement cursor-based pagination.

  That wouldn't be robust in the face of changing data, but I'm not sure it's possible to implement paginated faceting in a way that survives ongoing changes to the underlying data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1080#issuecomment-720699160
issue_url: https://api.github.com/repos/simonw/datasette/issues/1080
node_id: MDEyOklzc3VlQ29tbWVudDcyMDY5OTE2MA==
created_at: 2020-11-02T20:13:42Z
updated_at: 2020-11-02T20:13:42Z
author_association: OWNER
body:
  Also relevant to this issue: #830 - redesigning the facet plugin hook in preparation for Datasette 1.0. And #972 supporting faceting against arbitrary queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/830#issuecomment-720700065
issue_url: https://api.github.com/repos/simonw/datasette/issues/830
node_id: MDEyOklzc3VlQ29tbWVudDcyMDcwMDA2NQ==
created_at: 2020-11-02T20:15:36Z
updated_at: 2020-11-02T20:15:36Z
author_association: OWNER
body:
  #427 had a bunch of ambitious plans for faceting that I haven't realized yet:

  > Think of all of the potential kinds of facets:
  > 
  > * `?_facet_array=tags` where tags is a JSON array of values
  > * `_facet_date=datetimecol` - faceted by date part of a datetime
  > * `_facet_bins=numeric_column` - can I do some kind of fancy binning here? Might need to take an argument
  > * `?_facet_bins=numeric_column:5` - could be a way to take an argument. We’ll ignore columns with a : in their name.
  > * `?_facet_json=jsoncol:jsonpath` - could use a JSON path to extract out something to facet on?
  > * `?_facet_percentile=numericcolumn` - could this work?
  > * `?_facet_function=column:sqlfunctionname` - maybe this could be interesting? Would allow for e.g. facet by soundex
  > * `?_facet_prefix=column:prefix` - facet by terms but only if they start with a specific prefix
  > * `?_facet_substring=column:3,6` - facet by a substr(column, 3, 6)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/596#issuecomment-720741903
issue_url: https://api.github.com/repos/simonw/datasette/issues/596
node_id: MDEyOklzc3VlQ29tbWVudDcyMDc0MTkwMw==
created_at: 2020-11-02T21:44:45Z
updated_at: 2020-11-02T21:44:45Z
author_association: NONE
body:
  Hi & thanks for the note @simonw! I wish I had more time to play with (and contribute to) datasette. I know you don't need me to tell you that it's super cool :-)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721319602
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/192
node_id: MDEyOklzc3VlQ29tbWVudDcyMTMxOTYwMg==
created_at: 2020-11-03T19:05:05Z
updated_at: 2020-11-03T19:05:05Z
author_association: OWNER
body:
  Relevant example using a SQLite CTE: https://github.com/simonw/datasette/issues/268#issuecomment-675725464

  CTEs were added in SQLite 3.8.3 in 2014-02-03 so they should be safe to use. If someone tries to run `sqlite-utils search` no an older version of SQLite they'll get an error, which I think is OK.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/195#issuecomment-721397665
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/195
node_id: MDEyOklzc3VlQ29tbWVudDcyMTM5NzY2NQ==
created_at: 2020-11-03T22:02:57Z
updated_at: 2020-11-03T22:02:57Z
author_association: OWNER
body:
  Documentation so far: https://github.com/simonw/sqlite-utils/blob/6cadc6103ff1ba58c6409ce7fba74259e72965d9/docs/cli.rst#executing-searches

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721420539
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/192
node_id: MDEyOklzc3VlQ29tbWVudDcyMTQyMDUzOQ==
created_at: 2020-11-03T23:05:53Z
updated_at: 2020-11-03T23:05:53Z
author_association: OWNER
body:
  Just realized there's a problem with the SQL I am using here: joining to get `rank` in this way only works against FTS5 tables, it doesn't work against FTS4.

   https://github.com/simonw/sqlite-utils/blob/c8a900df59efd34f394c863c0adff9912f1bf1d7/sqlite_utils/db.py#L1303-L1319

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721420907
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/192
node_id: MDEyOklzc3VlQ29tbWVudDcyMTQyMDkwNw==
created_at: 2020-11-03T23:07:01Z
updated_at: 2020-11-03T23:07:01Z
author_association: OWNER
body:
  I could depend on my `sqlite-fts4` library to solve this: https://github.com/simonw/sqlite-fts4

  Or I could say that only `FTS5` is supported for ranked searches - but still support `.search()` against FTS4 just without the option to sort by relevance.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721453779
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/192
node_id: MDEyOklzc3VlQ29tbWVudDcyMTQ1Mzc3OQ==
created_at: 2020-11-04T00:59:24Z
updated_at: 2020-11-04T00:59:36Z
author_association: OWNER
body:
  FTS5 was added in SQLite 3.9.0 in 2015-10-14 - so about a year after CTEs, which means CTEs will always be safe to use with FTS5 queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1082#issuecomment-721545090
issue_url: https://api.github.com/repos/simonw/datasette/issues/1082
node_id: MDEyOklzc3VlQ29tbWVudDcyMTU0NTA5MA==
created_at: 2020-11-04T06:47:15Z
updated_at: 2020-11-04T06:47:15Z
author_association: OWNER
body:
  I've run into a similar problem with Google Cloud Run: beyond a certain size of database file I find myself needing to run instances there with more RAM assigned to them.

  I haven't yet figured out a method to estimate the amount of RAM that will be needed to successfully serve a database file of a specific size- I've been using trial and error.

  5GB is quite a big database file, so it doesn't surprise me that it may need a bigger instance. I recommend trying it on a 1GB or 2GB of RAM Digital Ocean instance (their default is 512MB) and see if that works.

  Let me know what you find out!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1082#issuecomment-721547177
issue_url: https://api.github.com/repos/simonw/datasette/issues/1082
node_id: MDEyOklzc3VlQ29tbWVudDcyMTU0NzE3Nw==
created_at: 2020-11-04T06:52:30Z
updated_at: 2020-11-04T06:53:16Z
author_association: NONE
body:
  I think I tried the same db size on the following scenarios in Digital Ocean:
  1. Basic ($5/month) with 512MB RAM
  2. Basic ($10/month) with 1GB RAM
  3. Pro ($12/month) with 1GB RAM

  All such attempts conked out with "out of memory" errors

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/268#issuecomment-721896822
issue_url: https://api.github.com/repos/simonw/datasette/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDcyMTg5NjgyMg==
created_at: 2020-11-04T18:23:29Z
updated_at: 2020-11-04T18:23:29Z
author_association: OWNER
body:
  Worth noting that joining to get the rank works for FTS5 but not for FTS4 - see comment here: https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721420539

  Easiest solution would be to only support sort-by-rank for FTS5 tables. Alternative would be to depend on https://github.com/simonw/sqlite-fts4

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1083#issuecomment-721926827
issue_url: https://api.github.com/repos/simonw/datasette/issues/1083
node_id: MDEyOklzc3VlQ29tbWVudDcyMTkyNjgyNw==
created_at: 2020-11-04T19:23:42Z
updated_at: 2020-11-04T19:23:42Z
author_association: OWNER
body:
  https://latest.datasette.io/fixtures/sortable#export has advanced export options, but https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+order+by+pk1%2C+pk2+limit+101 does not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1083#issuecomment-721927254
issue_url: https://api.github.com/repos/simonw/datasette/issues/1083
node_id: MDEyOklzc3VlQ29tbWVudDcyMTkyNzI1NA==
created_at: 2020-11-04T19:24:34Z
updated_at: 2020-11-04T19:24:34Z
author_association: OWNER
body:
  Related: #856 - if it's possible to paginate correctly configured canned query then the CSV option to "stream all rows" could work for queries as well as tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1082#issuecomment-721931504
issue_url: https://api.github.com/repos/simonw/datasette/issues/1082
node_id: MDEyOklzc3VlQ29tbWVudDcyMTkzMTUwNA==
created_at: 2020-11-04T19:32:47Z
updated_at: 2020-11-04T19:35:44Z
author_association: OWNER
body:
  I wonder if setting a soft memory limit within Datasette would help here: https://www.sqlite.org/malloc.html#_setting_memory_usage_limits 

  >  If attempts are made to allocate more memory than specified by the soft heap limit, then SQLite will first attempt to free cache memory before continuing with the allocation request.

  https://www.sqlite.org/pragma.html#pragma_soft_heap_limit

  > **PRAGMA soft_heap_limit**
  > **PRAGMA soft_heap_limit=N**
  > 
  > This pragma invokes the [sqlite3_soft_heap_limit64()](https://www.sqlite.org/c3ref/hard_heap_limit64.html) interface with the argument N, if N is specified and is a non-negative integer. The soft_heap_limit pragma always returns the same integer that would be returned by the [sqlite3_soft_heap_limit64](https://www.sqlite.org/c3ref/hard_heap_limit64.html)(-1) C-language function.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/192#issuecomment-722054264
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/192
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA1NDI2NA==
created_at: 2020-11-05T00:44:39Z
updated_at: 2020-11-05T00:44:39Z
author_association: OWNER
body:
  I want `.search()` to work against both FTS5 and FTS4 tables - but sort by rank should only work for FTS5.

  This means I need to be able to introspect and tell if a table is FTS4 or FTS5.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722055104
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA1NTEwNA==
created_at: 2020-11-05T00:47:34Z
updated_at: 2020-11-05T00:47:34Z
author_association: OWNER
body:
  This is surprisingly difficult. I need to parse the `CREATE VIRTUAL TABLE` statement, which will look something like this:

  ```sql
  CREATE VIRTUAL TABLE "global-power-plants_fts" USING FTS5 ("name", content="global-power-plants")
  ```

  The problem is I need to be able to handle various different quoting formats for the table name (`mytable` v.s. `"mytable"` v.s. `[mytable]`) plus I need to look out for `CREATE TABLE IF NOT EXISTS`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722056576
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA1NjU3Ng==
created_at: 2020-11-05T00:52:42Z
updated_at: 2020-11-05T00:52:42Z
author_association: OWNER
body:
  I could use a parsing library like https://parsy.readthedocs.io/en/latest/tutorial.html for this - or `pyparsing` which has a SQLite example here: https://github.com/pyparsing/pyparsing/blob/master/examples/select_parser.py

  I'd rather not add a new dependency for this though so I'm going to see if I can get something that's good-enough just using a regular expression.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722057392
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA1NzM5Mg==
created_at: 2020-11-05T00:55:31Z
updated_at: 2020-11-05T00:55:51Z
author_association: OWNER
body:
  https://sqlite.org/lang_keywords.html says:

  > There are four ways of quoting keywords in SQLite:
  >
  > **'keyword'** A keyword in single quotes is a string literal.
  > **"keyword"** A keyword in double-quotes is an identifier.
  > **[keyword]** A keyword enclosed in square brackets is an identifier. This is not standard SQL. This quoting mechanism is used by MS Access and SQL Server and is included in SQLite for compatibility.
  > **\`keyword\`** A keyword enclosed in grave accents (ASCII code 96) is an identifier. This is not standard SQL. This quoting mechanism is used by MySQL and is included in SQLite for compatibility.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722057923
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA1NzkyMw==
created_at: 2020-11-05T00:57:22Z
updated_at: 2020-11-05T00:57:22Z
author_association: OWNER
body:
  Then https://sqlite.org/lang_expr.html#literal_values_constants_ says:

  >  A string constant is formed by enclosing the string in single quotes ('). A single quote within the string can be encoded by putting two single quotes in a row - as in Pascal. C-style escapes using the backslash character are not supported because they are not standard SQL. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722058598
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA1ODU5OA==
created_at: 2020-11-05T00:59:58Z
updated_at: 2020-11-05T00:59:58Z
author_association: OWNER
body:
  That two-in-a-row thing works for `"` too:

  https://latest.datasette.io/fixtures?sql=select+%22foo%22%2C+%27bar%27%2C+%22foo%22%22and%22%2C+%27bar%27%27and%27

  <img width="501" alt="fixtures__select__foo____bar____foo__and____bar__and_" src="https://user-images.githubusercontent.com/9599/98184381-1da6f580-1ebf-11eb-9396-beb4586d69ca.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722062082
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA2MjA4Mg==
created_at: 2020-11-05T01:10:51Z
updated_at: 2020-11-05T01:10:51Z
author_association: OWNER
body:
  I confirmed all three of these are valid syntax for creating tables:
  ```
  ~ % sqlite3 tmp.db
  SQLite version 3.28.0 2019-04-15 14:49:49
  Enter ".help" for usage hints.
  sqlite> create table 'foo''and' (id int);
  sqlite> create table "bar""and" (id int);
  sqlite> create table [baz] (id int);
  sqlite> create table `bant` (id int);
  sqlite> .schema
  CREATE TABLE IF NOT EXISTS 'foo''and' (id int);
  CREATE TABLE IF NOT EXISTS "bar""and" (id int);
  CREATE TABLE [baz] (id int);
  CREATE TABLE `bant` (id int);
  sqlite> select * from sqlite_master;
  table|foo'and|foo'and|2|CREATE TABLE 'foo''and' (id int)
  table|bar"and|bar"and|3|CREATE TABLE "bar""and" (id int)
  table|baz|baz|4|CREATE TABLE [baz] (id int)
  table|bant|bant|5|CREATE TABLE `bant` (id int)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722062449
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA2MjQ0OQ==
created_at: 2020-11-05T01:12:14Z
updated_at: 2020-11-05T01:12:14Z
author_association: OWNER
body:
  Good news: I don't think I have to deal with `foo.tablename`, because that doesn't get reflected in the `sqlite_master` table:
  ```
  sqlite> attach 'foo.db' as foo;
  sqlite> create table foo.`bant` (id int);
  sqlite> select * from foo.sqlite_master;
  table|bant|bant|2|CREATE TABLE `bant` (id int)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722064258
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA2NDI1OA==
created_at: 2020-11-05T01:18:07Z
updated_at: 2020-11-05T01:21:31Z
author_association: OWNER
body:
  ```
  In [8]: r = re.compile(r"""'[^']*(?:''[^']*)*'""")
  In [9]: r.match("'fo'o'")
  Out[9]: <re.Match object; span=(0, 4), match="'fo'">
  In [10]: r.match("'fo''o'")
  Out[10]: <re.Match object; span=(0, 7), match="'fo''o'">
  ```

  `'[^']*(?:''[^']*)*'`

  This matches a single quote, then 0+ not-single-quotes, then 0+ (either 0+ not-single quotes or a double single quote), then a single quote.

  Unrolling the loop technique described here: http://www.softec.lu/site/RegularExpressions/UnrollingTheLoop

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722070569
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA3MDU2OQ==
created_at: 2020-11-05T01:38:40Z
updated_at: 2020-11-05T01:38:40Z
author_association: OWNER
body:
  I'm going to try `re.VERBOSE` to see if I can make this readable with comments. https://docs.python.org/3/howto/regex.html
  ```python
  charref = re.compile(r"""
   &[#]                # Start of a numeric entity reference
   (
       0[0-7]+         # Octal form
     | [0-9]+          # Decimal form
     | x[0-9a-fA-F]+   # Hexadecimal form
   )
   ;                   # Trailing semicolon
  """, re.VERBOSE)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722078286
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA3ODI4Ng==
created_at: 2020-11-05T02:04:18Z
updated_at: 2020-11-05T02:04:18Z
author_association: OWNER
body:
  I think this might be it:
  ```python
  create_virtual_table_re = re.compile(r"""
  \s*CREATE\s+VIRTUAL\s+TABLE\s+ # CREATE VIRTUAL TABLE
  (
     '(?P<squoted_table>[^']*(?:''[^']*)*)' | # single quoted name
     "(?P<dquoted_table>[^"]*(?:""[^"]*)*)" | # double quoted name
     `(?P<backtick_table>[^`]+)` |            # `backtick` quoted name
     \[(?P<squarequoted_table>[^\]]+)\]       # [...] quoted name
  )
  \s+(IF\s+NOT\s+EXISTS\s+)?     # IF NOT EXISTS (optional)
  USING\s+(?P<using>\w+)
  """, re.VERBOSE | re.IGNORECASE)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722082497
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA4MjQ5Nw==
created_at: 2020-11-05T02:18:08Z
updated_at: 2020-11-05T02:18:08Z
author_association: OWNER
body:
  I'm missing the case where a table has no quotes around it at all - `create virtual table foo using fts5`

  So I need to know how to create a regex for a SQLite identifier. https://www.sqlite.org/draft/tokenreq.html seems to be the only available documentation for that.

  > 
  > ### Identifier tokens
  > 
  > Identifiers follow the usual rules with the exception that SQLite allows the dollar-sign symbol in the interior of an identifier. The dollar-sign is for compatibility with Microsoft SQL-Server and is not part of the SQL standard.
  > 
  > > **H41130:** SQLite shall recognize as an ID token any sequence of characters that begins with an ALPHABETIC character and continue with zero or more ALPHANUMERIC characters and/or "$" (u0024) characters and which is not a keyword token.
  > 
  > Identifiers can be arbitrary character strings within square brackets. This feature is also for compatibility with Microsoft SQL-Server and not a part of the SQL standard.
  > 
  > > **H41140:** SQLite shall recognize as an ID token any sequence of non-zero characters that begins with "[" (u005b) and continuing through the first "]" (u005d) character.
  > 
  > The standard way of quoting SQL identifiers is to use double-quotes.
  > 
  > > **H41150:** SQLite shall recognize as an ID token any sequence of characters that begins with a double-quote (u0022), is followed by zero or more non-zero characters and/or pairs of double-quotes (u0022) and terminates with a double-quote (u0022) that is not part of a pair.
  > 
  > MySQL allows identifiers to be quoted using the grave accent character. SQLite supports this for interoperability.
  > 
  > > **H41160:** SQLite shall recognize as an ID token any sequence of characters that begins with a grave accent (u0060), is followed by zero or more non-zero characters and/or pairs ofgrave accents (u0060) and terminates with a grave accent (u0022) that is not part of a pair.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722082759
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA4Mjc1OQ==
created_at: 2020-11-05T02:18:58Z
updated_at: 2020-11-05T02:18:58Z
author_association: OWNER
body:
  More from that document, describing `ALPHANUMERIC`:

  > **ALPHABETIC**
  > 
  > Any of the characters in the range u0041 through u005a (letters "A" through "Z") or in the range u0061 through u007a (letters "a" through "z") or the character u005f ("_") or any other character larger than u007f.
  > 
  > **NUMERIC**
  > 
  > Any of the characters in the range u0030 through u0039 (digits "0" through "9")
  > 
  > **ALPHANUMERIC**
  > 
  > Any character which is either ALPHABETIC or NUMERIC

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722084213
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA4NDIxMw==
created_at: 2020-11-05T02:23:37Z
updated_at: 2020-11-05T02:23:37Z
author_association: OWNER
body:
  So...

  ALPHABETIC: `[\u0041-\u005a\u0061-\u0071\u007f-\uffff\u005f]`

  NUMERIC: `[\u0030-\u0039]`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722084593
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA4NDU5Mw==
created_at: 2020-11-05T02:24:47Z
updated_at: 2020-11-05T02:24:47Z
author_association: OWNER
body:
  And an identifier is "ALPHABETIC character and continue with zero or more ALPHANUMERIC characters and/or "$" (u0024) characters"

  So...

      [\u0041-\u005a\u0061-\u0071\u007f-\uffff\u005f][\u0041-\u005a\u0061-\u0071\u007f-\uffff\u005f\u0030-\u0039\u0024]+

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722086105
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/196
node_id: MDEyOklzc3VlQ29tbWVudDcyMjA4NjEwNQ==
created_at: 2020-11-05T02:29:50Z
updated_at: 2020-11-05T03:39:58Z
author_association: OWNER
body:
  The finished monster:

  ```python
  _virtual_table_using_re = re.compile(r"""
  ^ # Start of string
  \s*CREATE\s+VIRTUAL\s+TABLE\s+ # CREATE VIRTUAL TABLE
  (
      '(?P<squoted_table>[^']*(?:''[^']*)*)' | # single quoted name
      "(?P<dquoted_table>[^"]*(?:""[^"]*)*)" | # double quoted name
      `(?P<backtick_table>[^`]+)`            | # `backtick` quoted name
      \[(?P<squarequoted_table>[^\]]+)\]     | # [...] quoted name
      (?P<identifier>                          # SQLite non-quoted identifier
          [A-Za-z_\u0080-\uffff]  # \u0080-\uffff = "any character larger than u007f"
          [A-Za-z_\u0080-\uffff0-9\$]* # zero-or-more alphanemuric or $
      )
  )
  \s+(IF\s+NOT\s+EXISTS\s+)?      # IF NOT EXISTS (optional)
  USING\s+(?P<using>\w+)          # e.g. USING FTS5
  """, re.VERBOSE | re.IGNORECASE)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/195#issuecomment-722542895
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/195
node_id: MDEyOklzc3VlQ29tbWVudDcyMjU0Mjg5NQ==
created_at: 2020-11-05T18:01:33Z
updated_at: 2020-11-05T18:01:33Z
author_association: OWNER
body:
  Latest test failure:
  ```
  114  ->	        assert [("racoons are biting trash pandas", "USA", "bar")] == table.search(
  115  	            "bite", order="rowid"
  116  	        )
  117  	
  118  	
  119  	def test_optimize_fts(fresh_db):
  (Pdb) table.search("bite")
  [(2, 'racoons are biting trash pandas', 'USA', 'bar', -9.641434262948206e-07)]
  ```
  The problem here is that the `table.search()` method now behaves differently for FTS4 v.s. FTS5 tables.

  With FTS4 you get back just the table columns.

  With FTS5 you also get back the `rowid` as the first column and the `rank` score as the last column.

  This is weird. It also makes me question whether having `.search()` return a list of tuples is the right API design.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/197#issuecomment-722545442
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/197
node_id: MDEyOklzc3VlQ29tbWVudDcyMjU0NTQ0Mg==
created_at: 2020-11-05T18:05:33Z
updated_at: 2020-11-05T18:05:33Z
author_association: OWNER
body:
  This is likely to result in a 3.0 release due to a backwards-incompatible change to the current `.search()` method - #194

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/198#issuecomment-722849539
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/198
node_id: MDEyOklzc3VlQ29tbWVudDcyMjg0OTUzOQ==
created_at: 2020-11-06T05:39:17Z
updated_at: 2020-11-06T05:39:17Z
author_association: OWNER
body:
  I'd have to copy almost all of the code in https://github.com/simonw/sqlite-fts4/blob/master/sqlite_fts4/__init__.py so I think I will add it as a dependency instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/198#issuecomment-722852262
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/198
node_id: MDEyOklzc3VlQ29tbWVudDcyMjg1MjI2Mg==
created_at: 2020-11-06T05:41:58Z
updated_at: 2020-11-06T05:41:58Z
author_association: OWNER
body:
  Example query (from the tests):
  ```sql
  select c0, c1, rank_bm25(matchinfo(search, 'pcnalx')) as bm25
          from search where search match ?
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723143633
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/198
node_id: MDEyOklzc3VlQ29tbWVudDcyMzE0MzYzMw==
created_at: 2020-11-06T15:33:12Z
updated_at: 2020-11-06T15:33:12Z
author_association: OWNER
body:
  Here's the FTS5 query:
  ```sql
  with original as (
      select
          rowid,
          *
      from [global-power-plants]
  )
  select
      original.*,
      [global-power-plants_fts].rank as rank
  from
      [original]
      join [global-power-plants_fts] on [original].rowid = [global-power-plants_fts].rowid
  where
      [global-power-plants_fts] match :query
  order by
      rank desc
  limit 20
  ```
  The equivalent using `rank_bm25()` for FTS4 would be:
  ```sql
  with original as (
      select
          rowid,
          *
      from [global-power-plants]
  )
  select
      original.*,
      rank_bm25(matchinfo([global-power-plants_fts], 'pcnalx')) as rank
  from
      [original]
      join [global-power-plants_fts] on [original].rowid = [global-power-plants_fts].rowid
  where
      [global-power-plants_fts] match :query
  order by
      rank desc
  limit 20
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723144893
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/198
node_id: MDEyOklzc3VlQ29tbWVudDcyMzE0NDg5Mw==
created_at: 2020-11-06T15:35:45Z
updated_at: 2020-11-06T15:35:45Z
author_association: OWNER
body:
  Here's a demo of that rank query. I had to sort by `rank` instead of `rank desc` - need to double-check that:

  https://datasette-sqlite-fts4.datasette.io/24ways-fts4?sql=with+original+as+(%0D%0A++++select%0D%0A++++++++rowid%2C%0D%0A++++++++*%0D%0A++++from+[articles]%0D%0A)%0D%0Aselect%0D%0A++++original.*%2C%0D%0A++++rank_bm25(matchinfo([articles_fts]%2C+%27pcnalx%27))+as+rank%0D%0Afrom%0D%0A++++[original]%0D%0A++++join+[articles_fts]+on+[original].rowid+%3D+[articles_fts].rowid%0D%0Awhere%0D%0A++++[articles_fts]+match+%3Aquery%0D%0Aorder+by%0D%0A++++rank%0D%0Alimit+20&query=jquery+maps

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723145383
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/198
node_id: MDEyOklzc3VlQ29tbWVudDcyMzE0NTM4Mw==
created_at: 2020-11-06T15:36:47Z
updated_at: 2020-11-06T15:36:47Z
author_association: OWNER
body:
  Should I register the custom `rank_bm25` SQLite function for every connection, or should I register it against the connection just the first time the user attempts an FTS4 search? I think I'd rather register it only if it is needed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/199#issuecomment-723147463
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/199
node_id: MDEyOklzc3VlQ29tbWVudDcyMzE0NzQ2Mw==
created_at: 2020-11-06T15:41:00Z
updated_at: 2020-11-06T15:41:00Z
author_association: OWNER
body:
  Something like this:
  ```
  @db.register_function(replace=True)
  def my_function(a):
      return a.upper()
  ```
  If `replace=True` then this function will be registered even if a `my_function` of arity 1 has already been registered previously.

  It defaults to `False` though which means the Database object tracks what functions and arities have been registered in the past and silently ignores any new attempts to register the same name/arity.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/197#issuecomment-723148310
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/197
node_id: MDEyOklzc3VlQ29tbWVudDcyMzE0ODMxMA==
created_at: 2020-11-06T15:42:43Z
updated_at: 2020-11-06T15:42:43Z
author_association: OWNER
body:
  Having `.search()` return tuples when `.rows_where()` returns dictionaries just feels like bad API design to me - it's inconsistent. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/195#issuecomment-723148906
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/195
node_id: MDEyOklzc3VlQ29tbWVudDcyMzE0ODkwNg==
created_at: 2020-11-06T15:43:51Z
updated_at: 2020-11-06T15:43:51Z
author_association: OWNER
body:
  Thanks to #198 (introducing a `rank_bm25()` custom function for FTS4) this feature will be able to offer relevance search for both FTS5 AND FTS4 tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/194#issuecomment-723234493
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/194
node_id: MDEyOklzc3VlQ29tbWVudDcyMzIzNDQ5Mw==
created_at: 2020-11-06T18:32:34Z
updated_at: 2020-11-06T18:32:34Z
author_association: OWNER
body:
  The breaking changes will be:

  - `table.search()` now returns a generator that produces dictionaries, similar to `table.rows_where()`
  - The `-c` shortcut no longer works, use `--csv` instead.
  - The `-f` shortcut no longer works, use `--fmt` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/192#issuecomment-723348614
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/192
node_id: MDEyOklzc3VlQ29tbWVudDcyMzM0ODYxNA==
created_at: 2020-11-06T23:42:38Z
updated_at: 2020-11-06T23:42:38Z
author_association: OWNER
body:
  This is a bit surprising:
  ```
  (sqlite-utils) sqlite-utils % sqlite-utils search 24ways-fts4.db articles maps -c title      
  [{"rowid": 41, "title": "What Is Vagrant and Why Should I Care?", "rank": -1.9252039178908076},
   {"rowid": 298, "title": "First Steps in VR", "rank": -1.9945466378736434},
   {"rowid": 43, "title": "Content Production Planning", "rank": -2.1928058363046143},
   {"rowid": 100, "title": "Moo'y Christmas", "rank": -2.2698482999851675},
   {"rowid": 91, "title": "Infinite Canvas: Moving Beyond the Page", "rank": -2.290928999035195},
   {"rowid": 175, "title": "Front-End Code Reusability with CSS and JavaScript", "rank": -2.498731782924352},
   {"rowid": 209, "title": "Feeding the Audio Graph", "rank": -2.619968930100356},
   {"rowid": 296, "title": "Animation in Design Systems", "rank": -2.62060151817201},
   {"rowid": 118, "title": "Ghosts On The Internet", "rank": -2.7224894534521087},
   {"rowid": 77, "title": "Colour Accessibility", "rank": -2.7389782859427343},
   {"rowid": 245, "title": "Web Content Accessibility Guidelines 2.1\u2014for People Who Haven\u2019t Read the Update", "rank": -2.9750992611162888},
   {"rowid": 56, "title": "Helping VIPs Care About Performance", "rank": -3.0819662908932535},
   {"rowid": 109, "title": "Geotag Everywhere with Fire Eagle", "rank": -3.1371975973877277},
   {"rowid": 203, "title": "Jobs-to-Be-Done in Your UX Toolbox", "rank": -3.2416719461682733},
   {"rowid": 276, "title": "Your jQuery: Now With 67% Less Suck", "rank": -3.4947916564653028},
   {"rowid": 58, "title": "Beyond the Style Guide", "rank": -3.7508321464447905},
   {"rowid": 225, "title": "Good Ideas Grow on Paper", "rank": -4.120077674716844},
   {"rowid": 168, "title": "Unobtrusively Mapping Microformats with jQuery", "rank": -4.662224207228984},
   {"rowid": 27, "title": "Putting Design on the Map", "rank": -5.667327088267961},
   {"rowid": 220, "title": "Finding Your Way with Static Maps", "rank": -9.952534352591737}]
  ```
  I requested just `-c title` but also got back `rowid` and `rank`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/194#issuecomment-723356020
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/194
node_id: MDEyOklzc3VlQ29tbWVudDcyMzM1NjAyMA==
created_at: 2020-11-07T00:16:06Z
updated_at: 2020-11-07T00:16:06Z
author_association: OWNER
body:
  I'm going to release this as an alpha first and sit on it for a few days, since I don't want to ship any mistakes that would result in having to bump straight to 4.0!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/192#issuecomment-723357117
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/192
node_id: MDEyOklzc3VlQ29tbWVudDcyMzM1NzExNw==
created_at: 2020-11-07T00:21:05Z
updated_at: 2020-11-07T00:21:05Z
author_association: OWNER
body:
  I'm going to have it only output the exact `-c` columns you requested (if you requested any). Add `--rank` to specify the rank column, since you may not know what its name is.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/200#issuecomment-723357855
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/200
node_id: MDEyOklzc3VlQ29tbWVudDcyMzM1Nzg1NQ==
created_at: 2020-11-07T00:24:37Z
updated_at: 2020-11-07T00:24:37Z
author_association: OWNER
body:
  ```
  (sqlite-utils) sqlite-utils % sqlite-utils rows 24ways-fts4.db articles -c title -c author | head -n 3
  [{"title": "Why Bother with Accessibility?", "author": "Laura Kalbag"},
   {"title": "Levelling Up", "author": "Ashley Baxter"},
   {"title": "Project Hubs: A Home Base for Design Projects", "author": "Brad Frost"},
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/197#issuecomment-723360842
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/197
node_id: MDEyOklzc3VlQ29tbWVudDcyMzM2MDg0Mg==
created_at: 2020-11-07T00:40:55Z
updated_at: 2020-11-07T00:40:55Z
author_association: OWNER
body:
  The `order=` parameter should be called `order_by` for consistency with this:
  ```python
  for row in db["dogs"].rows_where("age > 1", order_by="age"):
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/197#issuecomment-723365651
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/197
node_id: MDEyOklzc3VlQ29tbWVudDcyMzM2NTY1MQ==
created_at: 2020-11-07T01:06:32Z
updated_at: 2020-11-07T01:06:32Z
author_association: OWNER
body:
  Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#searching-with-table-search

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/192#issuecomment-723637930
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/192
node_id: MDEyOklzc3VlQ29tbWVudDcyMzYzNzkzMA==
created_at: 2020-11-08T17:06:56Z
updated_at: 2020-11-08T17:06:56Z
author_association: OWNER
body:
  This looks pretty good now!
  ```
  % sqlite-utils search 24ways.db articles simon -c title -c author -t 
  title                                                                          author
  -----------------------------------------------------------------------------  ------------------
  Don't be eval()                                                                Simon Willison
  DOM Scripting Your Way to Better Blockquotes                                   Jeremy Keith
  Swooshy Curly Quotes Without Images                                            Simon Collison
  The Articulate Web Designer of Tomorrow                                        Simon Collison
  Writing Responsible JavaScript                                                 Drew McLellan
  Going Nuts with CSS Transitions                                                Natalie Downe
  Managing a Mind                                                                Christopher Murphy
  Design Systems                                                                 Laura Kalbag
  Bringing Your Code to the Streets                                              Ruth John
  Taming Complexity                                                              Simon Collison
  Unobtrusively Mapping Microformats with jQuery                                 Simon Willison
  Crafting the Front-end                                                         Ben Bodien
  Develop Your Naturalist Superpowers with Observable Notebooks and iNaturalist  Natalie Downe
  Fast Autocomplete Search for Your Website                                      Simon Willison
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/268#issuecomment-723740546
issue_url: https://api.github.com/repos/simonw/datasette/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDcyMzc0MDU0Ng==
created_at: 2020-11-09T04:01:50Z
updated_at: 2020-11-09T04:01:50Z
author_association: OWNER
body:
  I should depend on `sqlite-fts4` - I'm doing that in `sqlite-utils` now and it works great: https://github.com/simonw/sqlite-utils/issues/198

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/168#issuecomment-723829147
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/168
node_id: MDEyOklzc3VlQ29tbWVudDcyMzgyOTE0Nw==
created_at: 2020-11-09T07:43:30Z
updated_at: 2020-11-09T07:43:30Z
author_association: OWNER
body:
  Yeah whatever process the have in place is working great without any extra intervention: they upgraded to 3.0 four hours ago.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1086#issuecomment-725729857
issue_url: https://api.github.com/repos/simonw/datasette/issues/1086
node_id: MDEyOklzc3VlQ29tbWVudDcyNTcyOTg1Nw==
created_at: 2020-11-11T23:55:39Z
updated_at: 2020-11-11T23:55:39Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/foreign_key_references?_facet=foreign_key_with_blank_label

  <img width="772" alt="fixtures__foreign_key_references__2_rows" src="https://user-images.githubusercontent.com/9599/98877868-53575b80-2436-11eb-8669-f24eeb8d0f3c.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1085#issuecomment-725731685
issue_url: https://api.github.com/repos/simonw/datasette/issues/1085
node_id: MDEyOklzc3VlQ29tbWVudDcyNTczMTY4NQ==
created_at: 2020-11-12T00:01:18Z
updated_at: 2020-11-12T00:01:18Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1085?src=pr&el=h1) Report
  > Merging [#1085](https://codecov.io/gh/simonw/datasette/pull/1085?src=pr&el=desc) (51e7651) into [main](https://codecov.io/gh/simonw/datasette/commit/2a981e2ac1d13125973904b777d00ea75e8df4e6?el=desc) (2a981e2) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1085/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1085?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1085   +/-   ##
  =======================================
    Coverage   91.38%   91.38%           
  =======================================
    Files          30       30           
    Lines        3785     3785           
  =======================================
    Hits         3459     3459           
    Misses        326      326           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1085?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1085?src=pr&el=footer). Last update [2a981e2...51e7651](https://codecov.io/gh/simonw/datasette/pull/1085?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1088#issuecomment-725829903
issue_url: https://api.github.com/repos/simonw/datasette/issues/1088
node_id: MDEyOklzc3VlQ29tbWVudDcyNTgyOTkwMw==
created_at: 2020-11-12T04:33:14Z
updated_at: 2020-11-12T04:33:14Z
author_association: OWNER
body:
  I'm suspicious of this code: https://github.com/simonw/datasette/blob/e8e0a6f284ca953b2980186c4356594c07bd1929/datasette/views/table.py#L1032-L1045

  This code uses a different exception: https://github.com/simonw/datasette/blob/e8e0a6f284ca953b2980186c4356594c07bd1929/datasette/views/table.py#L658-L663

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1088#issuecomment-725830716
issue_url: https://api.github.com/repos/simonw/datasette/issues/1088
node_id: MDEyOklzc3VlQ29tbWVudDcyNTgzMDcxNg==
created_at: 2020-11-12T04:35:38Z
updated_at: 2020-11-12T04:35:38Z
author_association: OWNER
body:
  I'm going to fix this without a test, because writing a test for this is a bit fiddly and it's a very minor bug. If it comes back again I'll do the work to test for it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1089#issuecomment-726127465
issue_url: https://api.github.com/repos/simonw/datasette/issues/1089
node_id: MDEyOklzc3VlQ29tbWVudDcyNjEyNzQ2NQ==
created_at: 2020-11-12T14:54:11Z
updated_at: 2020-11-12T14:54:11Z
author_association: OWNER
body:
  Suggested list to look out for from that PR:

  - simply/simple
  - easy/easier/easiest
  - obvious/obviously
  - just
  - merely
  - straightforward
  - ridiculous

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/865#issuecomment-726385422
issue_url: https://api.github.com/repos/simonw/datasette/issues/865
node_id: MDEyOklzc3VlQ29tbWVudDcyNjM4NTQyMg==
created_at: 2020-11-12T22:40:14Z
updated_at: 2020-11-12T22:40:14Z
author_association: NONE
body:
  Just tested with the latest Docker image, and it works pretty much everywhere!  THANK YOU!

  I did notice that if I try to export json or csv, the base is not applied.  Not sure if I should reopen this issue or open a new one.

  To see this, go here: https://corpora.tika.apache.org/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES

  Click/hover over json or CSV and you'll see that the 'datasette' base is not included.

  Again, many thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-726413829
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDcyNjQxMzgyOQ==
created_at: 2020-11-12T23:52:50Z
updated_at: 2020-11-12T23:54:16Z
author_association: OWNER
body:
  Hmm... it's not just the `.csv` and `.json` export links - it's the column headings (which can be clicked to change the sort order) as well. Here's an extract of the HTML from that page:
  ```html
  <p class="export-links">This data as 
    <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json">json</a>, 
    <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.csv?_size=max">CSV</a> (
    <a href="#export">advanced</a>)
  </p>
  <div class="table-wrapper">
    <table class="rows-and-columns">
      <thead>
        <tr>
          <th class="col-Link" scope="col" data-column="Link" data-column-type="" data-column-not-null="0" data-is-pk="0">
            Link
          </th>
          <th class="col-rowid" scope="col" data-column="rowid" data-column-type="integer" data-column-not-null="0" data-is-pk="1">
            <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort_desc=rowid" rel="nofollow">rowid&nbsp;▼</a>
          </th>
          <th class="col-PARSE_EXCEPTION_ID" scope="col" data-column="PARSE_EXCEPTION_ID" data-column-type="INTEGER" data-column-not-null="0" data-is-pk="0">
            <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort=PARSE_EXCEPTION_ID" rel="nofollow">PARSE_EXCEPTION_ID</a>
          </th>
          <th class="col-PARSE_EXCEPTION_DESCRIPTION" scope="col" data-column="PARSE_EXCEPTION_DESCRIPTION" data-column-type="VARCHAR(128)" data-column-not-null="0" data-is-pk="0">
            <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort=PARSE_EXCEPTION_DESCRIPTION" rel="nofollow">PARSE_EXCEPTION_DESCRIPTION</a>
          </th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td class="col-Link type-pk">
            <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/1">1</a>
          </td>
          <td class="col-rowid type-int">1</td>
          <td class="col-PARSE_EXCEPTION_ID type-int">0</td>
          <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">RUNTIME</td>
        </tr>
        <tr>
          <td class="col-Link type-pk">
            <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/2">2</a>
          </td>
          <td class="col-rowid type-int">2</td>
          <td class="col-PARSE_EXCEPTION_ID type-int">1</td>
          <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">ENCRYPTION</td>
        </tr>
        <tr>
          <td class="col-Link type-pk">
            <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/3">3</a>
          </td>
          <td class="col-rowid type-int">3</td>
          <td class="col-PARSE_EXCEPTION_ID type-int">2</td>
          <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">ACCESS_PERMISSION</td>
        </tr>
        <tr>
          <td class="col-Link type-pk">
            <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/4">4</a>
          </td>
          <td class="col-rowid type-int">4</td>
          <td class="col-PARSE_EXCEPTION_ID type-int">3</td>
          <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">UNSUPPORTED_VERSION</td>
        </tr>
      </tbody>
    </table>
  </div>
  <div id="export" class="advanced-export">
    <h3>Advanced export</h3>
    <p>JSON shape:
      <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json">default</a>,
      <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json?_shape=array">array</a>,
      <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json?_shape=array&amp;_nl=on">newline-delimited</a>
    </p>
    <form action="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.csv" method="get">
      <p>
        CSV options:            
        <label>
          <input type="checkbox" name="_dl"> download file
          </label>
          <input type="submit" value="Export CSV">
            <input type="hidden" name="_size" value="max">
            </p>
          </form>
        </div>
  ```
  But here's something _really_ weird - the links to the individual rows DO include the `/datasette/` prefix:
  ```html
  <td class="col-Link type-pk">
      <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/2">2</a>
  </td>
  ```
  The navigation bar on that page is correct too:
  ```html
  <p class="crumbs">
          <a href="/datasette/">home</a> /
          <a href="/datasette/corpora-metadata">corpora-metadata</a>
  </p>
  ```
  I've also been unable to replicate this in my own local environment, running `datasette fixtures.db --config base_url:/datasette/`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-726415019
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDcyNjQxNTAxOQ==
created_at: 2020-11-12T23:56:23Z
updated_at: 2020-11-12T23:56:23Z
author_association: OWNER
body:
  @tballison is there any chance you're running any custom templates in that installation? I'm really confused as to why I can't replicate the bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-726415991
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDcyNjQxNTk5MQ==
created_at: 2020-11-12T23:59:34Z
updated_at: 2020-11-12T23:59:34Z
author_association: OWNER
body:
  The sort headers are generated by this template code: https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/templates/_table.html#L11-L15

  The export links use this code:

  https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/templates/table.html#L134

  https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/templates/table.html#L180-L201

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-726416330
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDcyNjQxNjMzMA==
created_at: 2020-11-13T00:00:43Z
updated_at: 2020-11-13T00:00:43Z
author_association: OWNER
body:
  Here's where `url_csv` comes from: https://github.com/simonw/datasette/blob/11eb1e026f3d84cb771f8d6e204939cbaee130cd/datasette/views/base.py#L542-L545

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/749#issuecomment-726417847
issue_url: https://api.github.com/repos/simonw/datasette/issues/749
node_id: MDEyOklzc3VlQ29tbWVudDcyNjQxNzg0Nw==
created_at: 2020-11-13T00:05:14Z
updated_at: 2020-11-13T00:05:14Z
author_association: OWNER
body:
  https://cloud.google.com/blog/products/serverless/cloud-run-now-supports-http-grpc-server-streaming indicates this limit should no longer apply:

  > With this addition, Cloud Run can now ... Send responses larger than the previous 32 MB limit

  But I'm still getting errors from Cloud Run attempting to download `.db` files larger than 32 MB.

  I filed a question in their issue tracker about that here: https://issuetracker.google.com/issues/173038375

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/268#issuecomment-726419027
issue_url: https://api.github.com/repos/simonw/datasette/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDcyNjQxOTAyNw==
created_at: 2020-11-13T00:09:04Z
updated_at: 2020-11-13T00:09:04Z
author_association: OWNER
body:
  Part of the challenge here is that this is the first time the `TableView` will have had a complete rewrite of the SQL it is going to execute. That SQL is currently constructed here: https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/views/table.py#L628-L636

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-726798745
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDcyNjc5ODc0NQ==
created_at: 2020-11-13T14:35:22Z
updated_at: 2020-11-13T14:35:22Z
author_association: NONE
body:
  I'm starting this with docker like so:

  `docker run --name datasette -d -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/file_profiles.db --config sql_time_limit_ms:120000 --config max_returned_rows:100000 --config base_url:/datasette/ --config cache_size_kb:50000`

  I'm not doing any templating or anything else custom.

  Apropos of nothing, I swapped out a simpler db, so this query should now work:

  https://corpora.tika.apache.org/datasette/file_profiles?sql=select%0D%0A++*%0D%0Afrom%0D%0A++file_profiles+fp%0D%0Alimit%0D%0A++10

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-726801731
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDcyNjgwMTczMQ==
created_at: 2020-11-13T14:40:56Z
updated_at: 2020-11-13T14:40:56Z
author_association: NONE
body:
  My headers aren't clickable/sortable with custom sql, but I think that's by design.

  In the default view, https://corpora.tika.apache.org/datasette/file_profiles/file_profiles, ah, y, now I see that the headers should be sortable, but you're right the base_url is not applied.

  base_url works with "View and Edit SQL" and with "(advanced)"

  As you point out, does not work with the export csv, json, other or with the "Next page" navigational button at the bottom.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-727233553
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDcyNzIzMzU1Mw==
created_at: 2020-11-14T16:46:52Z
updated_at: 2020-11-14T16:46:52Z
author_association: OWNER
body:
  @tballison could I see the section of your Apache config that configures the proxying to `/datasette/`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1097#issuecomment-727655018
issue_url: https://api.github.com/repos/simonw/datasette/issues/1097
node_id: MDEyOklzc3VlQ29tbWVudDcyNzY1NTAxOA==
created_at: 2020-11-15T23:18:18Z
updated_at: 2020-11-15T23:18:18Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=h1) Report
  > Merging [#1097](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=desc) (e89211d) into [main](https://codecov.io/gh/simonw/datasette/commit/5eb8e9bf250b26e30b017d39a392c33973997656?el=desc) (5eb8e9b) will **not change** coverage.
  > The diff coverage is `84.61%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1097/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1097   +/-   ##
  =======================================
    Coverage   91.38%   91.38%           
  =======================================
    Files          30       30           
    Lines        3785     3785           
  =======================================
    Hits         3459     3459           
    Misses        326      326           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `73.63% <0.00%> (ø)` | |
  | [datasette/inspect.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2luc3BlY3QucHk=) | `36.11% <ø> (ø)` | |
  | [datasette/publish/common.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3B1Ymxpc2gvY29tbW9uLnB5) | `94.73% <ø> (ø)` | |
  | [datasette/tracer.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3RyYWNlci5weQ==) | `81.60% <0.00%> (ø)` | |
  | [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.16% <ø> (ø)` | |
  | [datasette/publish/heroku.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3B1Ymxpc2gvaGVyb2t1LnB5) | `87.12% <50.00%> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.46% <66.66%> (ø)` | |
  | [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `94.35% <77.77%> (ø)` | |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.01% <86.20%> (ø)` | |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.92% <92.30%> (ø)` | |
  | ... and [9 more](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree-more) | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=footer). Last update [5eb8e9b...e89211d](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1098#issuecomment-727656208
issue_url: https://api.github.com/repos/simonw/datasette/issues/1098
node_id: MDEyOklzc3VlQ29tbWVudDcyNzY1NjIwOA==
created_at: 2020-11-15T23:26:14Z
updated_at: 2020-11-15T23:26:14Z
author_association: OWNER
body:
  Schema for that broken example:
  ```sql
  CREATE TABLE generators_eia860 (
  	id INTEGER NOT NULL, 
  	plant_id_eia INTEGER, 
  	generator_id TEXT, 
  	report_date DATE, 
  	operational_status_code TEXT, 
  	operational_status TEXT, 
  	ownership_code TEXT, 
  	utility_id_eia INTEGER, 
  	capacity_mw FLOAT, 
  	summer_capacity_mw FLOAT, 
  	winter_capacity_mw FLOAT, 
  	energy_source_code_1 TEXT, 
  	energy_source_code_2 TEXT, 
  	energy_source_code_3 TEXT, 
  	energy_source_code_4 TEXT, 
  	energy_source_code_5 TEXT, 
  	energy_source_code_6 TEXT, 
  	fuel_type_code_pudl TEXT, 
  	multiple_fuels BOOLEAN, 
  	deliver_power_transgrid BOOLEAN, 
  	syncronized_transmission_grid BOOLEAN, 
  	turbines_num INTEGER, 
  	planned_modifications BOOLEAN, 
  	planned_net_summer_capacity_uprate_mw FLOAT, 
  	planned_net_winter_capacity_uprate_mw FLOAT, 
  	planned_uprate_date DATE, 
  	planned_net_summer_capacity_derate_mw FLOAT, 
  	planned_net_winter_capacity_derate_mw FLOAT, 
  	planned_derate_date DATE, 
  	planned_new_prime_mover_code TEXT, 
  	planned_energy_source_code_1 TEXT, 
  	planned_repower_date DATE, 
  	other_planned_modifications BOOLEAN, 
  	other_modifications_date DATE, 
  	planned_retirement_date DATE, 
  	carbon_capture BOOLEAN, 
  	startup_source_code_1 TEXT, 
  	startup_source_code_2 TEXT, 
  	startup_source_code_3 TEXT, 
  	startup_source_code_4 TEXT, 
  	technology_description TEXT, 
  	turbines_inverters_hydrokinetics TEXT, 
  	time_cold_shutdown_full_load_code TEXT, 
  	planned_new_capacity_mw FLOAT, 
  	cofire_fuels BOOLEAN, 
  	switch_oil_gas BOOLEAN, 
  	nameplate_power_factor FLOAT, 
  	minimum_load_mw FLOAT, 
  	uprate_derate_during_year BOOLEAN, 
  	uprate_derate_completed_date DATE, 
  	current_planned_operating_date DATE, 
  	summer_estimated_capability_mw FLOAT, 
  	winter_estimated_capability_mw FLOAT, 
  	retirement_date DATE, 
  	PRIMARY KEY (id), 
  	FOREIGN KEY(plant_id_eia, generator_id) REFERENCES generators_entity_eia (plant_id_eia, generator_id), 
  	FOREIGN KEY(utility_id_eia) REFERENCES utilities_entity_eia (utility_id_eia), 
  	CHECK (multiple_fuels IN (0, 1)), 
  	CHECK (deliver_power_transgrid IN (0, 1)), 
  	CHECK (syncronized_transmission_grid IN (0, 1)), 
  	CHECK (planned_modifications IN (0, 1)), 
  	CHECK (other_planned_modifications IN (0, 1)), 
  	CHECK (carbon_capture IN (0, 1)), 
  	CHECK (cofire_fuels IN (0, 1)), 
  	CHECK (switch_oil_gas IN (0, 1)), 
  	CHECK (uprate_derate_during_year IN (0, 1))
  );
  ```
  https://pudl-datasette-xl7xwcpe2a-uc.a.run.app/pudl/generators_entity_eia is:
  ```sql
  CREATE TABLE generators_entity_eia (
  	plant_id_eia INTEGER NOT NULL, 
  	generator_id TEXT NOT NULL, 
  	prime_mover_code TEXT, 
  	duct_burners BOOLEAN, 
  	operating_date DATE, 
  	topping_bottoming_code TEXT, 
  	solid_fuel_gasification BOOLEAN, 
  	pulverized_coal_tech BOOLEAN, 
  	fluidized_bed_tech BOOLEAN, 
  	subcritical_tech BOOLEAN, 
  	supercritical_tech BOOLEAN, 
  	ultrasupercritical_tech BOOLEAN, 
  	stoker_tech BOOLEAN, 
  	other_combustion_tech BOOLEAN, 
  	bypass_heat_recovery BOOLEAN, 
  	rto_iso_lmp_node_id TEXT, 
  	rto_iso_location_wholesale_reporting_id TEXT, 
  	associated_combined_heat_power BOOLEAN, 
  	original_planned_operating_date DATE, 
  	operating_switch TEXT, 
  	previously_canceled BOOLEAN, 
  	PRIMARY KEY (plant_id_eia, generator_id), 
  	FOREIGN KEY(plant_id_eia) REFERENCES plants_entity_eia (plant_id_eia), 
  	CHECK (duct_burners IN (0, 1)), 
  	CHECK (solid_fuel_gasification IN (0, 1)), 
  	CHECK (pulverized_coal_tech IN (0, 1)), 
  	CHECK (fluidized_bed_tech IN (0, 1)), 
  	CHECK (subcritical_tech IN (0, 1)), 
  	CHECK (supercritical_tech IN (0, 1)), 
  	CHECK (ultrasupercritical_tech IN (0, 1)), 
  	CHECK (stoker_tech IN (0, 1)), 
  	CHECK (other_combustion_tech IN (0, 1)), 
  	CHECK (bypass_heat_recovery IN (0, 1)), 
  	CHECK (associated_combined_heat_power IN (0, 1)), 
  	CHECK (previously_canceled IN (0, 1))
  );
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-728262974
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDcyODI2Mjk3NA==
created_at: 2020-11-16T19:05:08Z
updated_at: 2020-11-16T19:05:08Z
author_association: OWNER
body:
  I have a hunch that there may be some extra configuration in play here - could Apache itself be rewriting some of the links using [mod_proxy_html](https://httpd.apache.org/docs/2.4/mod/mod_proxy_html.html)?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-729018386
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDcyOTAxODM4Ng==
created_at: 2020-11-17T15:48:58Z
updated_at: 2020-11-17T15:48:58Z
author_association: NONE
body:
  I don't think we are, but I'll check with Maruan.

  I think this is the relevant part of our config?

  ```
    Alias "/base/" "/usr/share/corpora/"
    <Directory "/usr/share/corpora/">
      Options +Indexes -Multiviews
      AllowOverride None
    </Directory>

    ProxyPreserveHost On

    ProxyPass /datasette http://0.0.0.0:8001
    ProxyPassReverse /datasette http://0.0.0.0:8001

  </VirtualHost>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/52#issuecomment-729484478
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/52
node_id: MDEyOklzc3VlQ29tbWVudDcyOTQ4NDQ3OA==
created_at: 2020-11-18T07:12:45Z
updated_at: 2020-11-18T07:12:45Z
author_association: NONE
body:
  I'm so sorry that you already have `--since_id` option and that's enough for the case I've mentioned. Thank you for this excellent tool!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/511#issuecomment-730893729
issue_url: https://api.github.com/repos/simonw/datasette/issues/511
node_id: MDEyOklzc3VlQ29tbWVudDczMDg5MzcyOQ==
created_at: 2020-11-20T06:35:13Z
updated_at: 2020-11-20T06:35:13Z
author_association: NONE
body:
  Trying to run on Windows today, I get an error from the utils/asgi.py module.  

  It's trying `from os import EX_CANTCREAT` which is Unix-only.  I commented this line out, and (so far) it's working. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1094#issuecomment-731260091
issue_url: https://api.github.com/repos/simonw/datasette/issues/1094
node_id: MDEyOklzc3VlQ29tbWVudDczMTI2MDA5MQ==
created_at: 2020-11-20T16:11:29Z
updated_at: 2020-11-20T16:11:29Z
author_association: NONE
body:
  I can confirm this issue, running version 0.51.1 under Windows.

  Fixed by commenting out the following line near the top of datasette\utils\asgi.py :

  `#from os import EX_CANTCREAT`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1084#issuecomment-731644064
issue_url: https://api.github.com/repos/simonw/datasette/issues/1084
node_id: MDEyOklzc3VlQ29tbWVudDczMTY0NDA2NA==
created_at: 2020-11-21T22:10:15Z
updated_at: 2020-11-21T22:10:15Z
author_association: OWNER
body:
  Another example of this bug: https://datasette-graphql-demo.datasette.io/github/users

  <img width="350" alt="github__users__4_823_rows_and_Comparing_1_1___main_·_simonw_datasette-graphql" src="https://user-images.githubusercontent.com/9599/99888639-3889a180-2c03-11eb-83b9-41f2eb12287e.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1084#issuecomment-731652991
issue_url: https://api.github.com/repos/simonw/datasette/issues/1084
node_id: MDEyOklzc3VlQ29tbWVudDczMTY1Mjk5MQ==
created_at: 2020-11-21T23:34:22Z
updated_at: 2020-11-21T23:34:22Z
author_association: OWNER
body:
  Fixed by positioning the menu relative to the `<div class="page-header">` element rather than the cog icon:

  <img width="966" alt="data" src="https://user-images.githubusercontent.com/9599/99889950-02eab580-2c0f-11eb-85dc-dd4bc840a92e.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1084#issuecomment-731653083
issue_url: https://api.github.com/repos/simonw/datasette/issues/1084
node_id: MDEyOklzc3VlQ29tbWVudDczMTY1MzA4Mw==
created_at: 2020-11-21T23:35:07Z
updated_at: 2020-11-21T23:35:07Z
author_association: OWNER
body:
  I got to use CSS `calc()` for this: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/static/app.css#L367-L371

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/860#issuecomment-731657486
issue_url: https://api.github.com/repos/simonw/datasette/issues/860
node_id: MDEyOklzc3VlQ29tbWVudDczMTY1NzQ4Ng==
created_at: 2020-11-22T00:25:34Z
updated_at: 2020-11-22T00:25:34Z
author_association: OWNER
body:
  There are three layers of metadata: table, database and instance.

  Currently the metadata fields are (ignoring not-quite-metadata like `sort` and `sort_desc`):

  - `title`
  - `description` (or `description_html`)
  - `about` / `about_url`
  - `source` / `source_url`
  - `license` / `license_url`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/860#issuecomment-731657660
issue_url: https://api.github.com/repos/simonw/datasette/issues/860
node_id: MDEyOklzc3VlQ29tbWVudDczMTY1NzY2MA==
created_at: 2020-11-22T00:27:32Z
updated_at: 2020-11-22T00:31:54Z
author_association: OWNER
body:
  Open question: how should cascading work? If a table is missing a field but the database or instance has it, should that value cascade down to the table?

  It feels like `license` should definitely cascade: if an instance lists a certain `license` that should absolutely filter through to all databases and tables.

  But... should the other fields cascade? Cascading `description` doesn't feel right at all, and neither does `title`.

  What about `about` and `about_url` and `source` and `source_url`? I'm a bit torn on whether they should cascade or not. I'm leaning towards cascading them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/860#issuecomment-731658059
issue_url: https://api.github.com/repos/simonw/datasette/issues/860
node_id: MDEyOklzc3VlQ29tbWVudDczMTY1ODA1OQ==
created_at: 2020-11-22T00:31:47Z
updated_at: 2020-11-22T00:33:48Z
author_association: OWNER
body:
  Documented behaviour right now, for metadata set at the instance level, is: https://docs.datasette.io/en/stable/metadata.html

  > The above metadata will be displayed on the index page of your Datasette-powered site. The source and license information will also be included in the footer of every page served by Datasette.
  >
  > ...
  > 
  > Metadata at the top level of the JSON will be shown on the index page and in the footer on every page of the site. The license and source is expected to apply to all of your data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1096#issuecomment-732542285
issue_url: https://api.github.com/repos/simonw/datasette/issues/1096
node_id: MDEyOklzc3VlQ29tbWVudDczMjU0MjI4NQ==
created_at: 2020-11-24T02:16:22Z
updated_at: 2020-11-24T02:16:22Z
author_association: OWNER
body:
  I'd like to implement this by first extending the `register_output_renderer()` hook to support streaming huge responses, then switching CSV to use the plugin hook in addition to TSV using it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1101#issuecomment-732543700
issue_url: https://api.github.com/repos/simonw/datasette/issues/1101
node_id: MDEyOklzc3VlQ29tbWVudDczMjU0MzcwMA==
created_at: 2020-11-24T02:20:30Z
updated_at: 2020-11-24T02:20:30Z
author_association: OWNER
body:
  Current design: https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette

  ```python
  @hookimpl
  def register_output_renderer(datasette):
      return {
          "extension": "test",
          "render": render_demo,
          "can_render": can_render_demo,  # Optional
      }
  ```
  Where `render_demo` looks something like this:
  ```python
  async def render_demo(datasette, columns, rows):
      db = datasette.get_database()
      result = await db.execute("select sqlite_version()")
      first_row = " | ".join(columns)
      lines = [first_row]
      lines.append("=" * len(first_row))
      for row in rows:
          lines.append(" | ".join(row))
      return Response(
          "\n".join(lines),
          content_type="text/plain; charset=utf-8",
          headers={"x-sqlite-version": result.first()[0]}
      )
  ```
  Meanwhile here's where the CSV streaming mode is implemented: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/views/base.py#L297-L380

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1101#issuecomment-732544590
issue_url: https://api.github.com/repos/simonw/datasette/issues/1101
node_id: MDEyOklzc3VlQ29tbWVudDczMjU0NDU5MA==
created_at: 2020-11-24T02:22:55Z
updated_at: 2020-11-24T02:22:55Z
author_association: OWNER
body:
  The trick I'm using here is to follow the `next_url` in order to paginate through all of the matching results. The loop calls the `data()` method multiple times, once for each page of results: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/views/base.py#L304-L307

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/860#issuecomment-733175454
issue_url: https://api.github.com/repos/simonw/datasette/issues/860
node_id: MDEyOklzc3VlQ29tbWVudDczMzE3NTQ1NA==
created_at: 2020-11-24T19:06:07Z
updated_at: 2020-11-24T19:06:07Z
author_association: OWNER
body:
  I see two ways this plugin hook could work. It could be asked about a specific instance, database or table and return the full metadata for that object. OR it could ask for a specific metadata field - e.g. `source_url` for table X, and return that.

  The more finely grained one would allow plugins to implement their own cascading rules pretty easily. Is there a reason it would be better for the hook to return an entire block of JSON for a specific table or database?

  I also need to decide if this hook is just going to be about source/license/about displayed metadata, or if it will include the functionality that has been sneaking into `metadata.json` over time - stuff like page size, default sort order or default facets.

  Perhaps I should split those out into a "configuration" concept first, after renaming `--config` to `--setting` in #992.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/992#issuecomment-733175965
issue_url: https://api.github.com/repos/simonw/datasette/issues/992
node_id: MDEyOklzc3VlQ29tbWVudDczMzE3NTk2NQ==
created_at: 2020-11-24T19:07:13Z
updated_at: 2020-11-24T19:07:13Z
author_association: OWNER
body:
  This is blocking progress on other metadata tickets like #860 because I want to split the concept of concrete metadata (source, license, etc) from configuration that currently lives in metadata (default sort order, default facets).

  I'm going to solve this next to unblock that stuff.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/992#issuecomment-733176252
issue_url: https://api.github.com/repos/simonw/datasette/issues/992
node_id: MDEyOklzc3VlQ29tbWVudDczMzE3NjI1Mg==
created_at: 2020-11-24T19:07:49Z
updated_at: 2020-11-24T19:07:49Z
author_association: OWNER
body:
  I'm going to keep `--config` for the moment but show a deprecation warning that it will be gone in Datasette 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/992#issuecomment-733180289
issue_url: https://api.github.com/repos/simonw/datasette/issues/992
node_id: MDEyOklzc3VlQ29tbWVudDczMzE4MDI4OQ==
created_at: 2020-11-24T19:16:30Z
updated_at: 2020-11-24T19:16:30Z
author_association: OWNER
body:
  Need to figure out the `--setting foo bar` alternative for this `--config foo:bar` logic: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/cli.py#L31-L63

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1105#issuecomment-733190827
issue_url: https://api.github.com/repos/simonw/datasette/issues/1105
node_id: MDEyOklzc3VlQ29tbWVudDczMzE5MDgyNw==
created_at: 2020-11-24T19:38:02Z
updated_at: 2020-11-24T19:38:02Z
author_association: OWNER
body:
  I'd like to redirect https://docs.datasette.io/en/stable/config.html to a new https://docs.datasette.io/en/stable/settings.html page too. I can use https://docs.readthedocs.io/en/stable/user-defined-redirects.html for that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1107#issuecomment-733244471
issue_url: https://api.github.com/repos/simonw/datasette/issues/1107
node_id: MDEyOklzc3VlQ29tbWVudDczMzI0NDQ3MQ==
created_at: 2020-11-24T21:29:59Z
updated_at: 2020-11-24T21:29:59Z
author_association: OWNER
body:
  I ran `rg '.config\(' datasette-*/` in my top-level directory:
  ```
  datasette-sentry/test_datasette_sentry.py:    def plugin_config(self, name):
  datasette-sentry/datasette_sentry.py:    config = datasette.plugin_config("datasette-sentry") or {}
  datasette-render-markdown/datasette_render_markdown/__init__.py:        datasette.plugin_config(
  datasette-render-images/datasette_render_images.py:        plugin_config = datasette.plugin_config("datasette-render-images") or {}
  datasette-render-html/datasette_render_html.py:    config = datasette.plugin_config(
  datasette-render-timestamps/datasette_render_timestamps/__init__.py:        datasette.plugin_config(
  datasette-permissions-sql/datasette_permissions_sql/__init__.py:        for rule in datasette.plugin_config("datasette-permissions-sql") or []:
  datasette-mask-columns/datasette_mask_columns/__init__.py:            datasette.plugin_config("datasette-mask-columns", database=database) or {}
  datasette-mask-columns/datasette_mask_columns/__init__.py:    masks = datasette.plugin_config("datasette-mask-columns", database=database) or {}
  datasette-media/datasette_media/__init__.py:    plugin_config = datasette.plugin_config("datasette-media") or {}
  datasette-insert/datasette_insert/__init__.py:    plugin_config = datasette.plugin_config("datasette-insert") or {}
  datasette-indieauth/datasette_indieauth/__init__.py:    plugin_config = datasette.plugin_config("datasette-indieauth") or {}
  datasette-graphql/datasette_graphql/__init__.py:    config = datasette.plugin_config("datasette-graphql") or {}
  datasette-graphql/datasette_graphql/__init__.py:    config = datasette.plugin_config("datasette-graphql") or {}
  datasette-graphql/datasette_graphql/utils.py:            auto_camelcase=(datasette.plugin_config("datasette-graphql") or {}).get(
  datasette-graphql/datasette_graphql/utils.py:    table_plugin_config = datasette.plugin_config(
  datasette-graphql/datasette_graphql/utils.py:    def config(self, key):
  datasette-graphql/datasette_graphql/utils.py:        return self.__wrapped__.config(key)
  datasette-init/datasette_init/__init__.py:        config = datasette.plugin_config("datasette-init")
  datasette-edit-templates/datasette_edit_templates/__init__.py:    plugin_config = datasette.plugin_config("datasette-edit-templates") or {}
  datasette-cors/datasette_cors.py:    config = datasette.plugin_config("datasette-cors") or {}
  datasette-cluster-map-old/build/lib/datasette_cluster_map/__init__.py:        datasette.plugin_config("datasette-cluster-map", database=database, table=table)
  datasette-cluster-map/datasette_cluster_map/__init__.py:        datasette.plugin_config("datasette-cluster-map", database=database, table=table)
  datasette-cluster-map/datasette_cluster_map/__init__.py:        datasette.plugin_config("datasette-cluster-map", database=database, table=table)
  datasette-cluster-map/tests/test_cluster_map.py:async def test_plugin_config(db_path, config, table, expected_fragments):
  datasette-configure-asgi/datasette_configure_asgi.py:        configs = datasette.plugin_config("datasette-configure-asgi") or []
  datasette-configure-asgi/test_datasette_configure_asgi.py:    def plugin_config(self, name):
  datasette-cluster-map-old/datasette_cluster_map/__init__.py:        datasette.plugin_config("datasette-cluster-map", database=database, table=table)
  datasette-auth-passwords/datasette_auth_passwords/__init__.py:    config = datasette.plugin_config("datasette-auth-passwords") or {}
  datasette-auth-github/datasette_auth_github/views.py:def verify_config(config):
  datasette-auth-github/datasette_auth_github/views.py:    config = datasette.plugin_config("datasette-auth-github")
  datasette-auth-github/datasette_auth_github/views.py:    verify_config(config)
  datasette-auth-github/datasette_auth_github/views.py:    config = datasette.plugin_config("datasette-auth-github")
  datasette-auth-github/datasette_auth_github/views.py:    verify_config(config)
  datasette-atom/datasette_atom/__init__.py:        plugin_config = datasette.plugin_config("datasette-atom")
  datasette-auth-google/datasette_auth_google/__init__.py:    config = datasette.plugin_config("datasette-auth-github") or {}
  datasette-auth-existing-cookies/test_datasette_auth_existing_cookies.py:    def plugin_config(self, name):
  datasette-auth-passwords/build/lib/datasette_auth_passwords/__init__.py:    config = datasette.plugin_config("datasette-auth-passwords") or {}
  datasette-annotate/datasette_annotate/utils.py:    plugin_config = datasette.plugin_config("datasette-annotate") or {}
  datasette-auth-existing-cookies/datasette_auth_existing_cookies/__init__.py:    config = datasette.plugin_config("datasette-auth-existing-cookies") or {}
  datasette-auth-simple/datasette_auth_simple/__init__.py:datasette.plugin_config("datasette-auth-simple")
  datasette-auth-tokens/datasette_auth_tokens/__init__.py:        config = datasette.plugin_config("datasette-auth-tokens") or {}
  datasette-block-robots/datasette_block_robots/__init__.py:    config = datasette.plugin_config("datasette-block-robots") or {}
  datasette-block-robots/datasette_block_robots/__init__.py:    config = datasette.plugin_config("datasette-block-robots") or {}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1107#issuecomment-733245097
issue_url: https://api.github.com/repos/simonw/datasette/issues/1107
node_id: MDEyOklzc3VlQ29tbWVudDczMzI0NTA5Nw==
created_at: 2020-11-24T21:31:10Z
updated_at: 2020-11-24T21:31:10Z
author_association: OWNER
body:
  Most of these use `plugin_config` which is unaffected. It looks like the only code I need to worry about is this trick in `datasette-graphl`: https://github.com/simonw/datasette-graphql/blob/483c9a9e203bb90365def3df8b8f01dda1e75865/datasette_graphql/utils.py#L456-L460

  ```python
  class DatasetteSpecialConfig(wrapt.ObjectProxy):
      def config(self, key):
          if key == "suggest_facets":
              return False
          return self.__wrapped__.config(key)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1106#issuecomment-733245596
issue_url: https://api.github.com/repos/simonw/datasette/issues/1106
node_id: MDEyOklzc3VlQ29tbWVudDczMzI0NTU5Ng==
created_at: 2020-11-24T21:32:11Z
updated_at: 2020-11-24T21:32:11Z
author_association: OWNER
body:
  https://docs.datasette.io/en/latest/settings.html is now live - need to redirect https://docs.datasette.io/en/latest/config.html to it using the ReadTheDocs redirects interface.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1106#issuecomment-733247101
issue_url: https://api.github.com/repos/simonw/datasette/issues/1106
node_id: MDEyOklzc3VlQ29tbWVudDczMzI0NzEwMQ==
created_at: 2020-11-24T21:35:29Z
updated_at: 2020-11-24T21:36:04Z
author_association: OWNER
body:
  <img width="861" alt="Edit_Redirects___Read_the_Docs" src="https://user-images.githubusercontent.com/9599/100153884-a3341a80-2e59-11eb-8d8c-7027c714b9cf.png">

  https://docs.datasette.io/en/latest/config.html isn't redirecting though, even after I tried running a rebuild of the `latest` version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1106#issuecomment-733248437
issue_url: https://api.github.com/repos/simonw/datasette/issues/1106
node_id: MDEyOklzc3VlQ29tbWVudDczMzI0ODQzNw==
created_at: 2020-11-24T21:38:50Z
updated_at: 2020-11-24T21:38:50Z
author_association: OWNER
body:
  I used an "exact redirect" instead and it worked:

  <img width="654" alt="Edit_Redirects___Read_the_Docs" src="https://user-images.githubusercontent.com/9599/100154418-60bf0d80-2e5a-11eb-92bf-527a14edb5c5.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1107#issuecomment-733257071
issue_url: https://api.github.com/repos/simonw/datasette/issues/1107
node_id: MDEyOklzc3VlQ29tbWVudDczMzI1NzA3MQ==
created_at: 2020-11-24T21:59:32Z
updated_at: 2020-11-24T21:59:32Z
author_association: OWNER
body:
  I'm going to make this a documented method in https://docs.datasette.io/en/latest/internals.html#datasette-class

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/860#issuecomment-733287416
issue_url: https://api.github.com/repos/simonw/datasette/issues/860
node_id: MDEyOklzc3VlQ29tbWVudDczMzI4NzQxNg==
created_at: 2020-11-24T23:15:44Z
updated_at: 2020-11-24T23:15:44Z
author_association: OWNER
body:
  I'm going to go with a plugin hook (and Datasette method) that returns individual values - so you ask it for e.g. the `license_url` for a specific table and it returns a string or `None`.

  The default plugin hook implementation that ships with Datasette will then implement cascading lookups against `metadata.json` - but other plugins will be able to provide their own implementations, which should make it easy to build a plugin that lets you keep metadata in a database file and edit it interactively.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/860#issuecomment-733287619
issue_url: https://api.github.com/repos/simonw/datasette/issues/860
node_id: MDEyOklzc3VlQ29tbWVudDczMzI4NzYxOQ==
created_at: 2020-11-24T23:16:21Z
updated_at: 2020-11-24T23:16:21Z
author_association: OWNER
body:
  I'll also allow any key to be looked up - so if users want to invent their own metadata keys other than the default `license_url` etc they can do so.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/860#issuecomment-733288522
issue_url: https://api.github.com/repos/simonw/datasette/issues/860
node_id: MDEyOklzc3VlQ29tbWVudDczMzI4ODUyMg==
created_at: 2020-11-24T23:18:47Z
updated_at: 2020-11-24T23:18:47Z
author_association: OWNER
body:
  In #942 I want to add support for per-column metadata - which means this new lookup mechanism will need to be able to answer the question "what description is available for this column".

  So what should the `.metadata()` method look like? A couple of options:

  - `datasette.metadata("description", table=x, database=y)` - can take optional `column=` too.
  - `datasette.table_metadata("description", table=x, database=y)` and `datasette.database_metadata("description", database=y)` and so on - multiple methods for the different types of metadata.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/860#issuecomment-733288841
issue_url: https://api.github.com/repos/simonw/datasette/issues/860
node_id: MDEyOklzc3VlQ29tbWVudDczMzI4ODg0MQ==
created_at: 2020-11-24T23:19:47Z
updated_at: 2020-11-24T23:20:24Z
author_association: OWNER
body:
  Here's what I have today - it's an undocumented `datasette.metadata()` method that returns a full JSON dictionary of values OR a single value if the optional `key=` argument is provided: https://github.com/simonw/datasette/blob/f2e2bfcdd9ad4891f3f66c9104c09943d943ffe4/datasette/app.py#L357-L388

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1110#issuecomment-733432768
issue_url: https://api.github.com/repos/simonw/datasette/issues/1110
node_id: MDEyOklzc3VlQ29tbWVudDczMzQzMjc2OA==
created_at: 2020-11-25T03:04:00Z
updated_at: 2020-11-25T03:04:00Z
author_association: OWNER
body:
  Design:

      datasette publish cloudrun blah.db --apt-get-install ripgrep --install datasette-ripgrep

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1110#issuecomment-733432916
issue_url: https://api.github.com/repos/simonw/datasette/issues/1110
node_id: MDEyOklzc3VlQ29tbWVudDczMzQzMjkxNg==
created_at: 2020-11-25T03:04:29Z
updated_at: 2020-11-25T03:04:29Z
author_association: OWNER
body:
  Already have a pattern for extra packages in the form of the `--spatialite` feature: https://github.com/simonw/datasette/blob/f2e2bfcdd9ad4891f3f66c9104c09943d943ffe4/datasette/utils/__init__.py#L50-L55

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1112#issuecomment-735279355
issue_url: https://api.github.com/repos/simonw/datasette/issues/1112
node_id: MDEyOklzc3VlQ29tbWVudDczNTI3OTM1NQ==
created_at: 2020-11-28T19:21:09Z
updated_at: 2020-11-28T19:21:09Z
author_association: CONTRIBUTOR
body:
  (Even more annoying is that I see my editor leaked an extra delete space at the end of the line. I'm happy to rebuild this to be less annoying, but you probably don't want the changelog update either way)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1112#issuecomment-735279733
issue_url: https://api.github.com/repos/simonw/datasette/issues/1112
node_id: MDEyOklzc3VlQ29tbWVudDczNTI3OTczMw==
created_at: 2020-11-28T19:24:28Z
updated_at: 2020-11-28T19:24:28Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1112?src=pr&el=h1) Report
  > Merging [#1112](https://codecov.io/gh/simonw/datasette/pull/1112?src=pr&el=desc) (1a30fc2) into [main](https://codecov.io/gh/simonw/datasette/commit/37d18a5bce08c9ee53c080f613bae84fc2ccc853?el=desc) (37d18a5) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1112/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1112?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1112   +/-   ##
  =======================================
    Coverage   91.44%   91.44%           
  =======================================
    Files          30       30           
    Lines        3833     3833           
  =======================================
    Hits         3505     3505           
    Misses        328      328           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1112?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1112?src=pr&el=footer). Last update [37d18a5...1a30fc2](https://codecov.io/gh/simonw/datasette/pull/1112?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/493#issuecomment-735281577
issue_url: https://api.github.com/repos/simonw/datasette/issues/493
node_id: MDEyOklzc3VlQ29tbWVudDczNTI4MTU3Nw==
created_at: 2020-11-28T19:39:53Z
updated_at: 2020-11-28T19:39:53Z
author_association: CONTRIBUTOR
body:
  I was confused by `--config` and I tried passing the json from datasette-ripgrep into `config.json` just as a wild guess. 

  A short term solution might be pointing out in plugins that their snippet json can go in `metadata.json` at least makes it easier to search for config options or to know where to start if someone is new. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1113#issuecomment-735304071
issue_url: https://api.github.com/repos/simonw/datasette/issues/1113
node_id: MDEyOklzc3VlQ29tbWVudDczNTMwNDA3MQ==
created_at: 2020-11-28T23:23:31Z
updated_at: 2020-11-28T23:23:31Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/bbde835a1fec01458e8d00929e7bab6d6a5ba948/datasette/views/table.py#L1013-L1026 - looks like I fixed this in #1088.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1111#issuecomment-735320736
issue_url: https://api.github.com/repos/simonw/datasette/issues/1111
node_id: MDEyOklzc3VlQ29tbWVudDczNTMyMDczNg==
created_at: 2020-11-29T02:46:23Z
updated_at: 2020-11-29T02:46:23Z
author_association: OWNER
body:
  This is a really useful bug report, thanks! I agree: more aggressive timeouts on table counts sounds like the right solution here.

  I've learned that avoiding `count(*)` is crucial for handling these larger databases. Datasette has been getting better about that over time but there are still some edge-cases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/123#issuecomment-735440555
issue_url: https://api.github.com/repos/simonw/datasette/issues/123
node_id: MDEyOklzc3VlQ29tbWVudDczNTQ0MDU1NQ==
created_at: 2020-11-29T19:12:30Z
updated_at: 2020-11-29T19:12:30Z
author_association: NONE
body:
  [datasette-connectors](https://github.com/pytables/datasette-connectors) provides an API for making connectors for any file based database. For example, [datasette-pytables](https://github.com/pytables/datasette-pytables) is a connector for HDF5 files, so now is possible to use this type of files with Datasette.

  It'd be nice if Datasette coud provide that API directly, for other file formats and for urls too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1098#issuecomment-735442226
issue_url: https://api.github.com/repos/simonw/datasette/issues/1098
node_id: MDEyOklzc3VlQ29tbWVudDczNTQ0MjIyNg==
created_at: 2020-11-29T19:28:04Z
updated_at: 2020-11-29T19:28:04Z
author_association: OWNER
body:
  For the moment I'm going to solve this by teaching Datasette's internal introspection methods - in particular these ones - to ignore compound foreign keys entirely:

  https://github.com/simonw/datasette/blob/e800ffcf7cc6a915eb554b369c654f87162575e5/datasette/utils/__init__.py#L470-L505

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1098#issuecomment-735444698
issue_url: https://api.github.com/repos/simonw/datasette/issues/1098
node_id: MDEyOklzc3VlQ29tbWVudDczNTQ0NDY5OA==
created_at: 2020-11-29T19:50:14Z
updated_at: 2020-11-29T19:50:14Z
author_association: OWNER
body:
  Demo of the fix: https://latest.datasette.io/fixtures/foreign_key_references (the compound foreign key columns do not link to anything)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1099#issuecomment-735444858
issue_url: https://api.github.com/repos/simonw/datasette/issues/1099
node_id: MDEyOklzc3VlQ29tbWVudDczNTQ0NDg1OA==
created_at: 2020-11-29T19:51:58Z
updated_at: 2020-11-29T19:51:58Z
author_association: OWNER
body:
  My fix in deb0be4ae56f191f121239b29e83dd53b62d6305 for #1098 was to have Datasette deliberately pretend that compound foreign keys don't exist: https://github.com/simonw/datasette/blob/deb0be4ae56f191f121239b29e83dd53b62d6305/datasette/utils/__init__.py#L470-L495

  This workaround will need to be rethought to implement real support for them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1115#issuecomment-735447198
issue_url: https://api.github.com/repos/simonw/datasette/issues/1115
node_id: MDEyOklzc3VlQ29tbWVudDczNTQ0NzE5OA==
created_at: 2020-11-29T20:12:49Z
updated_at: 2020-11-29T20:12:49Z
author_association: OWNER
body:
  ```
  $ datasette ../shapefile-to-sqlite/nps-spatialite.db
  Usage: datasette serve [OPTIONS] [FILES]...

  Error: It looks like you're trying to load a SpatiaLite database without first loading the SpatiaLite module.

  Try adding the --load-extension=spatialite option.

  Read more: https://docs.datasette.io/en/stable/spatialite.html
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1114#issuecomment-735447635
issue_url: https://api.github.com/repos/simonw/datasette/issues/1114
node_id: MDEyOklzc3VlQ29tbWVudDczNTQ0NzYzNQ==
created_at: 2020-11-29T20:16:32Z
updated_at: 2020-11-29T20:17:11Z
author_association: OWNER
body:
  The new Docker container is pushed to Docker Hub too. Here's it in action:

  ```
  % docker pull datasetteproject/datasette
  % docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette --version
  datasette, version 0.52.1
  % docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/nps-spatialite.db --load-extension=spatialite
  INFO:     Started server process [1]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735465708
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54
node_id: MDEyOklzc3VlQ29tbWVudDczNTQ2NTcwOA==
created_at: 2020-11-29T22:08:46Z
updated_at: 2020-11-29T22:08:46Z
author_association: MEMBER
body:
  Demo:
  - https://github-to-sqlite.dogsheep.net/github/steps?_facet=repo
  - https://github-to-sqlite.dogsheep.net/github/workflows
  - https://github-to-sqlite.dogsheep.net/github/jobs

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482546
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDczNTQ4MjU0Ng==
created_at: 2020-11-30T00:22:02Z
updated_at: 2020-11-30T00:22:02Z
author_association: MEMBER
body:
  As for reviews... here's the output of `github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews --accept 'application/vnd.github.v3+json'`

  ```json
  [
      {
          "id": 503368921,
          "node_id": "MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTAzMzY4OTIx",
          "user": {
              "login": "simonw",
              "id": 9599,
              "node_id": "MDQ6VXNlcjk1OTk=",
              "avatar_url": "https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4",
              "gravatar_id": "",
              "url": "https://api.github.com/users/simonw",
              "html_url": "https://github.com/simonw",
              "followers_url": "https://api.github.com/users/simonw/followers",
              "following_url": "https://api.github.com/users/simonw/following{/other_user}",
              "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}",
              "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}",
              "subscriptions_url": "https://api.github.com/users/simonw/subscriptions",
              "organizations_url": "https://api.github.com/users/simonw/orgs",
              "repos_url": "https://api.github.com/users/simonw/repos",
              "events_url": "https://api.github.com/users/simonw/events{/privacy}",
              "received_events_url": "https://api.github.com/users/simonw/received_events",
              "type": "User",
              "site_admin": false
          },
          "body": "",
          "state": "CHANGES_REQUESTED",
          "html_url": "https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-503368921",
          "pull_request_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48",
          "author_association": "MEMBER",
          "_links": {
              "html": {
                  "href": "https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-503368921"
              },
              "pull_request": {
                  "href": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"
              }
          },
          "submitted_at": "2020-10-06T21:28:40Z",
          "commit_id": "4f33b850bd37829262dd29e1c520afffebedc19c"
      },
      {
          "id": 513118561,
          "node_id": "MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTEzMTE4NTYx",
          "user": {
              "login": "adamjonas",
              "id": 755825,
              "node_id": "MDQ6VXNlcjc1NTgyNQ==",
              "avatar_url": "https://avatars1.githubusercontent.com/u/755825?v=4",
              "gravatar_id": "",
              "url": "https://api.github.com/users/adamjonas",
              "html_url": "https://github.com/adamjonas",
              "followers_url": "https://api.github.com/users/adamjonas/followers",
              "following_url": "https://api.github.com/users/adamjonas/following{/other_user}",
              "gists_url": "https://api.github.com/users/adamjonas/gists{/gist_id}",
              "starred_url": "https://api.github.com/users/adamjonas/starred{/owner}{/repo}",
              "subscriptions_url": "https://api.github.com/users/adamjonas/subscriptions",
              "organizations_url": "https://api.github.com/users/adamjonas/orgs",
              "repos_url": "https://api.github.com/users/adamjonas/repos",
              "events_url": "https://api.github.com/users/adamjonas/events{/privacy}",
              "received_events_url": "https://api.github.com/users/adamjonas/received_events",
              "type": "User",
              "site_admin": false
          },
          "body": "",
          "state": "COMMENTED",
          "html_url": "https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513118561",
          "pull_request_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48",
          "author_association": "CONTRIBUTOR",
          "_links": {
              "html": {
                  "href": "https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513118561"
              },
              "pull_request": {
                  "href": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"
              }
          },
          "submitted_at": "2020-10-20T20:45:05Z",
          "commit_id": "4f33b850bd37829262dd29e1c520afffebedc19c"
      },
      {
          "id": 513127529,
          "node_id": "MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTEzMTI3NTI5",
          "user": {
              "login": "adamjonas",
              "id": 755825,
              "node_id": "MDQ6VXNlcjc1NTgyNQ==",
              "avatar_url": "https://avatars1.githubusercontent.com/u/755825?v=4",
              "gravatar_id": "",
              "url": "https://api.github.com/users/adamjonas",
              "html_url": "https://github.com/adamjonas",
              "followers_url": "https://api.github.com/users/adamjonas/followers",
              "following_url": "https://api.github.com/users/adamjonas/following{/other_user}",
              "gists_url": "https://api.github.com/users/adamjonas/gists{/gist_id}",
              "starred_url": "https://api.github.com/users/adamjonas/starred{/owner}{/repo}",
              "subscriptions_url": "https://api.github.com/users/adamjonas/subscriptions",
              "organizations_url": "https://api.github.com/users/adamjonas/orgs",
              "repos_url": "https://api.github.com/users/adamjonas/repos",
              "events_url": "https://api.github.com/users/adamjonas/events{/privacy}",
              "received_events_url": "https://api.github.com/users/adamjonas/received_events",
              "type": "User",
              "site_admin": false
          },
          "body": "",
          "state": "COMMENTED",
          "html_url": "https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513127529",
          "pull_request_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48",
          "author_association": "CONTRIBUTOR",
          "_links": {
              "html": {
                  "href": "https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513127529"
              },
              "pull_request": {
                  "href": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"
              }
          },
          "submitted_at": "2020-10-20T20:57:33Z",
          "commit_id": "3a0d5c498f9faae4e40aab204cd01b965a4f61f3"
      }
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483604
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDczNTQ4MzYwNA==
created_at: 2020-11-30T00:26:50Z
updated_at: 2020-11-30T00:26:50Z
author_association: MEMBER
body:
  It seems like there's a lot missing from that - those aren't particularly interesting given the data that is returned.

  From the docs at https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#reviews it looks like each review consists of multiple comments, and the comments are where the useful material is - https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#list-comments-for-a-pull-request-review

  `github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews/503368921/comments --accept 'application/vnd.github.v3+json'`

  ```json
  [
      {
          "id": 500603838,
          "node_id": "MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwMzgzOA==",
          "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500603838",
          "pull_request_review_id": 503368921,
          "diff_hunk": "@@ -0,0 +1,370 @@\n+[\n+    {\n+        \"url\": \"https://api.github.com/repos/simonw/datasette/pulls/571\",\n+        \"id\": 313384926,\n+        \"node_id\": \"MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2\",\n+        \"html_url\": \"https://github.com/simonw/datasette/pull/571\",\n+        \"diff_url\": \"https://github.com/simonw/datasette/pull/571.diff\",\n+        \"patch_url\": \"https://github.com/simonw/datasette/pull/571.patch\",\n+        \"issue_url\": \"https://api.github.com/repos/simonw/datasette/issues/571\",\n+        \"number\": 571,\n+        \"state\": \"closed\",\n+        \"locked\": false,\n+        \"title\": \"detect_fts now works with alternative table escaping\",\n+        \"user\": {\n+          \"login\": \"simonw\",\n+          \"id\": 9599,\n+          \"node_id\": \"MDQ6VXNlcjk1OTk=\",\n+          \"avatar_url\": \"https://avatars0.githubusercontent.com/u/9599?v=4\",\n+          \"gravatar_id\": \"\",\n+          \"url\": \"https://api.github.com/users/simonw\",\n+          \"html_url\": \"https://github.com/simonw\",\n+          \"followers_url\": \"https://api.github.com/users/simonw/followers\",\n+          \"following_url\": \"https://api.github.com/users/simonw/following{/other_user}\",\n+          \"gists_url\": \"https://api.github.com/users/simonw/gists{/gist_id}\",\n+          \"starred_url\": \"https://api.github.com/users/simonw/starred{/owner}{/repo}\",\n+          \"subscriptions_url\": \"https://api.github.com/users/simonw/subscriptions\",\n+          \"organizations_url\": \"https://api.github.com/users/simonw/orgs\",\n+          \"repos_url\": \"https://api.github.com/users/simonw/repos\",\n+          \"events_url\": \"https://api.github.com/users/simonw/events{/privacy}\",\n+          \"received_events_url\": \"https://api.github.com/users/simonw/received_events\",\n+          \"type\": \"User\",\n+          \"site_admin\": false\n+        },\n+        \"body\": \"Fixes #570\",\n+        \"created_at\": \"2019-09-03T00:23:39Z\",\n+        \"updated_at\": \"2019-09-03T00:32:28Z\",\n+        \"closed_at\": \"2019-09-03T00:32:28Z\",\n+        \"merged_at\": \"2019-09-03T00:32:28Z\",\n+        \"merge_commit_sha\": \"2dc5c8dc259a0606162673d394ba8cc1c6f54428\",\n+        \"assignee\": null,\n+        \"assignees\": [\n+\n+        ],\n+        \"requested_reviewers\": [\n+\n+        ],\n+        \"requested_teams\": [\n+\n+        ],\n+        \"labels\": [\n+\n+        ],\n+        \"milestone\": null,\n+        \"draft\": false,\n+        \"commits_url\": \"https://api.github.com/repos/simonw/datasette/pulls/571/commits\",\n+        \"review_comments_url\": \"https://api.github.com/repos/simonw/datasette/pulls/571/comments\",\n+        \"review_comment_url\": \"https://api.github.com/repos/simonw/datasette/pulls/comments{/number}\",\n+        \"comments_url\": \"https://api.github.com/repos/simonw/datasette/issues/571/comments\",\n+        \"statuses_url\": \"https://api.github.com/repos/simonw/datasette/statuses/a85239f69261c10f1a9f90514c8b5d113cb94585\",\n+        \"head\": {\n+          \"label\": \"simonw:detect-fts\",\n+          \"ref\": \"detect-fts\",\n+          \"sha\": \"a85239f69261c10f1a9f90514c8b5d113cb94585\",\n+          \"user\": {\n+            \"login\": \"simonw\",\n+            \"id\": 9599,\n+            \"node_id\": \"MDQ6VXNlcjk1OTk=\",\n+            \"avatar_url\": \"https://avatars0.githubusercontent.com/u/9599?v=4\",\n+            \"gravatar_id\": \"\",\n+            \"url\": \"https://api.github.com/users/simonw\",\n+            \"html_url\": \"https://github.com/simonw\",\n+            \"followers_url\": \"https://api.github.com/users/simonw/followers\",\n+            \"following_url\": \"https://api.github.com/users/simonw/following{/other_user}\",\n+            \"gists_url\": \"https://api.github.com/users/simonw/gists{/gist_id}\",\n+            \"starred_url\": \"https://api.github.com/users/simonw/starred{/owner}{/repo}\",\n+            \"subscriptions_url\": \"https://api.github.com/users/simonw/subscriptions\",\n+            \"organizations_url\": \"https://api.github.com/users/simonw/orgs\",\n+            \"repos_url\": \"https://api.github.com/users/simonw/repos\",\n+            \"events_url\": \"https://api.github.com/users/simonw/events{/privacy}\",\n+            \"received_events_url\": \"https://api.github.com/users/simonw/received_events\",\n+            \"type\": \"User\",\n+            \"site_admin\": false\n+          },\n+          \"repo\": {\n+            \"id\": 107914493,\n+            \"node_id\": \"MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=\",\n+            \"name\": \"datasette\",\n+            \"full_name\": \"simonw/datasette\",\n+            \"private\": false,\n+            \"owner\": {\n+              \"login\": \"simonw\",\n+              \"id\": 9599,\n+              \"node_id\": \"MDQ6VXNlcjk1OTk=\",\n+              \"avatar_url\": \"https://avatars0.githubusercontent.com/u/9599?v=4\",\n+              \"gravatar_id\": \"\",\n+              \"url\": \"https://api.github.com/users/simonw\",\n+              \"html_url\": \"https://github.com/simonw\",\n+              \"followers_url\": \"https://api.github.com/users/simonw/followers\",\n+              \"following_url\": \"https://api.github.com/users/simonw/following{/other_user}\",\n+              \"gists_url\": \"https://api.github.com/users/simonw/gists{/gist_id}\",\n+              \"starred_url\": \"https://api.github.com/users/simonw/starred{/owner}{/repo}\",\n+              \"subscriptions_url\": \"https://api.github.com/users/simonw/subscriptions\",\n+              \"organizations_url\": \"https://api.github.com/users/simonw/orgs\",\n+              \"repos_url\": \"https://api.github.com/users/simonw/repos\",\n+              \"events_url\": \"https://api.github.com/users/simonw/events{/privacy}\",\n+              \"received_events_url\": \"https://api.github.com/users/simonw/received_events\",\n+              \"type\": \"User\",\n+              \"site_admin\": false\n+            },\n+            \"html_url\": \"https://github.com/simonw/datasette\",\n+            \"description\": \"An open source multi-tool for exploring and publishing data\",\n+            \"fork\": false,\n+            \"url\": \"https://api.github.com/repos/simonw/datasette\",\n+            \"forks_url\": \"https://api.github.com/repos/simonw/datasette/forks\",\n+            \"keys_url\": \"https://api.github.com/repos/simonw/datasette/keys{/key_id}\",\n+            \"collaborators_url\": \"https://api.github.com/repos/simonw/datasette/collaborators{/collaborator}\",\n+            \"teams_url\": \"https://api.github.com/repos/simonw/datasette/teams\",\n+            \"hooks_url\": \"https://api.github.com/repos/simonw/datasette/hooks\",\n+            \"issue_events_url\": \"https://api.github.com/repos/simonw/datasette/issues/events{/number}\",\n+            \"events_url\": \"https://api.github.com/repos/simonw/datasette/events\",\n+            \"assignees_url\": \"https://api.github.com/repos/simonw/datasette/assignees{/user}\",\n+            \"branches_url\": \"https://api.github.com/repos/simonw/datasette/branches{/branch}\",\n+            \"tags_url\": \"https://api.github.com/repos/simonw/datasette/tags\",\n+            \"blobs_url\": \"https://api.github.com/repos/simonw/datasette/git/blobs{/sha}\",\n+            \"git_tags_url\": \"https://api.github.com/repos/simonw/datasette/git/tags{/sha}\",\n+            \"git_refs_url\": \"https://api.github.com/repos/simonw/datasette/git/refs{/sha}\",\n+            \"trees_url\": \"https://api.github.com/repos/simonw/datasette/git/trees{/sha}\",\n+            \"statuses_url\": \"https://api.github.com/repos/simonw/datasette/statuses/{sha}\",\n+            \"languages_url\": \"https://api.github.com/repos/simonw/datasette/languages\",\n+            \"stargazers_url\": \"https://api.github.com/repos/simonw/datasette/stargazers\",\n+            \"contributors_url\": \"https://api.github.com/repos/simonw/datasette/contributors\",\n+            \"subscribers_url\": \"https://api.github.com/repos/simonw/datasette/subscribers\",\n+            \"subscription_url\": \"https://api.github.com/repos/simonw/datasette/subscription\",\n+            \"commits_url\": \"https://api.github.com/repos/simonw/datasette/commits{/sha}\",\n+            \"git_commits_url\": \"https://api.github.com/repos/simonw/datasette/git/commits{/sha}\",\n+            \"comments_url\": \"https://api.github.com/repos/simonw/datasette/comments{/number}\",\n+            \"issue_comment_url\": \"https://api.github.com/repos/simonw/datasette/issues/comments{/number}\",\n+            \"contents_url\": \"https://api.github.com/repos/simonw/datasette/contents/{+path}\",\n+            \"compare_url\": \"https://api.github.com/repos/simonw/datasette/compare/{base}...{head}\",\n+            \"merges_url\": \"https://api.github.com/repos/simonw/datasette/merges\",\n+            \"archive_url\": \"https://api.github.com/repos/simonw/datasette/{archive_format}{/ref}\",\n+            \"downloads_url\": \"https://api.github.com/repos/simonw/datasette/downloads\",\n+            \"issues_url\": \"https://api.github.com/repos/simonw/datasette/issues{/number}\",\n+            \"pulls_url\": \"https://api.github.com/repos/simonw/datasette/pulls{/number}\",\n+            \"milestones_url\": \"https://api.github.com/repos/simonw/datasette/milestones{/number}\",\n+            \"notifications_url\": \"https://api.github.com/repos/simonw/datasette/notifications{?since,all,participating}\",\n+            \"labels_url\": \"https://api.github.com/repos/simonw/datasette/labels{/name}\",\n+            \"releases_url\": \"https://api.github.com/repos/simonw/datasette/releases{/id}\",\n+            \"deployments_url\": \"https://api.github.com/repos/simonw/datasette/deployments\",\n+            \"created_at\": \"2017-10-23T00:39:03Z\",\n+            \"updated_at\": \"2020-07-27T20:42:15Z\",\n+            \"pushed_at\": \"2020-07-26T01:21:05Z\",\n+            \"git_url\": \"git://github.com/simonw/datasette.git\",\n+            \"ssh_url\": \"git@github.com:simonw/datasette.git\",\n+            \"clone_url\": \"https://github.com/simonw/datasette.git\",\n+            \"svn_url\": \"https://github.com/simonw/datasette\",\n+            \"homepage\": \"http://datasette.readthedocs.io/\",\n+            \"size\": 3487,\n+            \"stargazers_count\": 3642,\n+            \"watchers_count\": 3642,\n+            \"language\": \"Python\",\n+            \"has_issues\": true,\n+            \"has_projects\": false,\n+            \"has_downloads\": true,\n+            \"has_wiki\": true,\n+            \"has_pages\": false,\n+            \"forks_count\": 206,\n+            \"mirror_url\": null,\n+            \"archived\": false,\n+            \"disabled\": false,\n+            \"open_issues_count\": 190,\n+            \"license\": {\n+              \"key\": \"apache-2.0\",\n+              \"name\": \"Apache License 2.0\",\n+              \"spdx_id\": \"Apache-2.0\",\n+              \"url\": \"https://api.github.com/licenses/apache-2.0\",\n+              \"node_id\": \"MDc6TGljZW5zZTI=\"\n+            },\n+            \"forks\": 206,\n+            \"open_issues\": 190,\n+            \"watchers\": 3642,\n+            \"default_branch\": \"master\"\n+          }\n+        },\n+        \"base\": {\n+          \"label\": \"simonw:master\",\n+          \"ref\": \"master\",\n+          \"sha\": \"f04deebec4f3842f7bd610cd5859de529f77d50e\",\n+          \"user\": {\n+            \"login\": \"simonw\",\n+            \"id\": 9599,\n+            \"node_id\": \"MDQ6VXNlcjk1OTk=\",\n+            \"avatar_url\": \"https://avatars0.githubusercontent.com/u/9599?v=4\",\n+            \"gravatar_id\": \"\",\n+            \"url\": \"https://api.github.com/users/simonw\",\n+            \"html_url\": \"https://github.com/simonw\",\n+            \"followers_url\": \"https://api.github.com/users/simonw/followers\",\n+            \"following_url\": \"https://api.github.com/users/simonw/following{/other_user}\",\n+            \"gists_url\": \"https://api.github.com/users/simonw/gists{/gist_id}\",\n+            \"starred_url\": \"https://api.github.com/users/simonw/starred{/owner}{/repo}\",\n+            \"subscriptions_url\": \"https://api.github.com/users/simonw/subscriptions\",\n+            \"organizations_url\": \"https://api.github.com/users/simonw/orgs\",\n+            \"repos_url\": \"https://api.github.com/users/simonw/repos\",\n+            \"events_url\": \"https://api.github.com/users/simonw/events{/privacy}\",\n+            \"received_events_url\": \"https://api.github.com/users/simonw/received_events\",\n+            \"type\": \"User\",\n+            \"site_admin\": false\n+          },\n+          \"repo\": {\n+            \"id\": 107914493,\n+            \"node_id\": \"MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=\",\n+            \"name\": \"datasette\",\n+            \"full_name\": \"simonw/datasette\",\n+            \"private\": false,\n+            \"owner\": {\n+              \"login\": \"simonw\",\n+              \"id\": 9599,\n+              \"node_id\": \"MDQ6VXNlcjk1OTk=\",\n+              \"avatar_url\": \"https://avatars0.githubusercontent.com/u/9599?v=4\",\n+              \"gravatar_id\": \"\",\n+              \"url\": \"https://api.github.com/users/simonw\",\n+              \"html_url\": \"https://github.com/simonw\",\n+              \"followers_url\": \"https://api.github.com/users/simonw/followers\",\n+              \"following_url\": \"https://api.github.com/users/simonw/following{/other_user}\",\n+              \"gists_url\": \"https://api.github.com/users/simonw/gists{/gist_id}\",\n+              \"starred_url\": \"https://api.github.com/users/simonw/starred{/owner}{/repo}\",\n+              \"subscriptions_url\": \"https://api.github.com/users/simonw/subscriptions\",\n+              \"organizations_url\": \"https://api.github.com/users/simonw/orgs\",\n+              \"repos_url\": \"https://api.github.com/users/simonw/repos\",\n+              \"events_url\": \"https://api.github.com/users/simonw/events{/privacy}\",\n+              \"received_events_url\": \"https://api.github.com/users/simonw/received_events\",\n+              \"type\": \"User\",\n+              \"site_admin\": false\n+            },\n+            \"html_url\": \"https://github.com/simonw/datasette\",\n+            \"description\": \"An open source multi-tool for exploring and publishing data\",\n+            \"fork\": false,\n+            \"url\": \"https://api.github.com/repos/simonw/datasette\",\n+            \"forks_url\": \"https://api.github.com/repos/simonw/datasette/forks\",\n+            \"keys_url\": \"https://api.github.com/repos/simonw/datasette/keys{/key_id}\",\n+            \"collaborators_url\": \"https://api.github.com/repos/simonw/datasette/collaborators{/collaborator}\",\n+            \"teams_url\": \"https://api.github.com/repos/simonw/datasette/teams\",\n+            \"hooks_url\": \"https://api.github.com/repos/simonw/datasette/hooks\",\n+            \"issue_events_url\": \"https://api.github.com/repos/simonw/datasette/issues/events{/number}\",\n+            \"events_url\": \"https://api.github.com/repos/simonw/datasette/events\",\n+            \"assignees_url\": \"https://api.github.com/repos/simonw/datasette/assignees{/user}\",\n+            \"branches_url\": \"https://api.github.com/repos/simonw/datasette/branches{/branch}\",\n+            \"tags_url\": \"https://api.github.com/repos/simonw/datasette/tags\",\n+            \"blobs_url\": \"https://api.github.com/repos/simonw/datasette/git/blobs{/sha}\",\n+            \"git_tags_url\": \"https://api.github.com/repos/simonw/datasette/git/tags{/sha}\",\n+            \"git_refs_url\": \"https://api.github.com/repos/simonw/datasette/git/refs{/sha}\",\n+            \"trees_url\": \"https://api.github.com/repos/simonw/datasette/git/trees{/sha}\",\n+            \"statuses_url\": \"https://api.github.com/repos/simonw/datasette/statuses/{sha}\",\n+            \"languages_url\": \"https://api.github.com/repos/simonw/datasette/languages\",\n+            \"stargazers_url\": \"https://api.github.com/repos/simonw/datasette/stargazers\",\n+            \"contributors_url\": \"https://api.github.com/repos/simonw/datasette/contributors\",\n+            \"subscribers_url\": \"https://api.github.com/repos/simonw/datasette/subscribers\",\n+            \"subscription_url\": \"https://api.github.com/repos/simonw/datasette/subscription\",\n+            \"commits_url\": \"https://api.github.com/repos/simonw/datasette/commits{/sha}\",\n+            \"git_commits_url\": \"https://api.github.com/repos/simonw/datasette/git/commits{/sha}\",\n+            \"comments_url\": \"https://api.github.com/repos/simonw/datasette/comments{/number}\",\n+            \"issue_comment_url\": \"https://api.github.com/repos/simonw/datasette/issues/comments{/number}\",\n+            \"contents_url\": \"https://api.github.com/repos/simonw/datasette/contents/{+path}\",\n+            \"compare_url\": \"https://api.github.com/repos/simonw/datasette/compare/{base}...{head}\",\n+            \"merges_url\": \"https://api.github.com/repos/simonw/datasette/merges\",\n+            \"archive_url\": \"https://api.github.com/repos/simonw/datasette/{archive_format}{/ref}\",\n+            \"downloads_url\": \"https://api.github.com/repos/simonw/datasette/downloads\",\n+            \"issues_url\": \"https://api.github.com/repos/simonw/datasette/issues{/number}\",\n+            \"pulls_url\": \"https://api.github.com/repos/simonw/datasette/pulls{/number}\",\n+            \"milestones_url\": \"https://api.github.com/repos/simonw/datasette/milestones{/number}\",\n+            \"notifications_url\": \"https://api.github.com/repos/simonw/datasette/notifications{?since,all,participating}\",\n+            \"labels_url\": \"https://api.github.com/repos/simonw/datasette/labels{/name}\",\n+            \"releases_url\": \"https://api.github.com/repos/simonw/datasette/releases{/id}\",\n+            \"deployments_url\": \"https://api.github.com/repos/simonw/datasette/deployments\",\n+            \"created_at\": \"2017-10-23T00:39:03Z\",\n+            \"updated_at\": \"2020-07-27T20:42:15Z\",\n+            \"pushed_at\": \"2020-07-26T01:21:05Z\",\n+            \"git_url\": \"git://github.com/simonw/datasette.git\",\n+            \"ssh_url\": \"git@github.com:simonw/datasette.git\",\n+            \"clone_url\": \"https://github.com/simonw/datasette.git\",\n+            \"svn_url\": \"https://github.com/simonw/datasette\",\n+            \"homepage\": \"http://datasette.readthedocs.io/\",\n+            \"size\": 3487,\n+            \"stargazers_count\": 3642,\n+            \"watchers_count\": 3642,\n+            \"language\": \"Python\",\n+            \"has_issues\": true,\n+            \"has_projects\": false,\n+            \"has_downloads\": true,\n+            \"has_wiki\": true,\n+            \"has_pages\": false,\n+            \"forks_count\": 206,\n+            \"mirror_url\": null,\n+            \"archived\": false,\n+            \"disabled\": false,\n+            \"open_issues_count\": 190,\n+            \"license\": {\n+              \"key\": \"apache-2.0\",\n+              \"name\": \"Apache License 2.0\",\n+              \"spdx_id\": \"Apache-2.0\",\n+              \"url\": \"https://api.github.com/licenses/apache-2.0\",\n+              \"node_id\": \"MDc6TGljZW5zZTI=\"\n+            },\n+            \"forks\": 206,\n+            \"open_issues\": 190,\n+            \"watchers\": 3642,\n+            \"default_branch\": \"master\"\n+          }\n+        },\n+        \"_links\": {\n+          \"self\": {\n+            \"href\": \"https://api.github.com/repos/simonw/datasette/pulls/571\"\n+          },\n+          \"html\": {\n+            \"href\": \"https://github.com/simonw/datasette/pull/571\"\n+          },\n+          \"issue\": {\n+            \"href\": \"https://api.github.com/repos/simonw/datasette/issues/571\"\n+          },\n+          \"comments\": {\n+            \"href\": \"https://api.github.com/repos/simonw/datasette/issues/571/comments\"\n+          },\n+          \"review_comments\": {\n+            \"href\": \"https://api.github.com/repos/simonw/datasette/pulls/571/comments\"\n+          },\n+          \"review_comment\": {\n+            \"href\": \"https://api.github.com/repos/simonw/datasette/pulls/comments{/number}\"\n+          },\n+          \"commits\": {\n+            \"href\": \"https://api.github.com/repos/simonw/datasette/pulls/571/commits\"\n+          },\n+          \"statuses\": {\n+            \"href\": \"https://api.github.com/repos/simonw/datasette/statuses/a85239f69261c10f1a9f90514c8b5d113cb94585\"\n+          }\n+        },\n+        \"author_association\": \"OWNER\",\n+        \"active_lock_reason\": null,\n+        \"merged\": true,\n+        \"mergeable\": null,\n+        \"rebaseable\": null,\n+        \"mergeable_state\": \"unknown\",\n+        \"merged_by\": {",
          "path": "tests/pull_requests.json",
          "position": 342,
          "original_position": 342,
          "commit_id": "3a0d5c498f9faae4e40aab204cd01b965a4f61f3",
          "user": {
              "login": "simonw",
              "id": 9599,
              "node_id": "MDQ6VXNlcjk1OTk=",
              "avatar_url": "https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4",
              "gravatar_id": "",
              "url": "https://api.github.com/users/simonw",
              "html_url": "https://github.com/simonw",
              "followers_url": "https://api.github.com/users/simonw/followers",
              "following_url": "https://api.github.com/users/simonw/following{/other_user}",
              "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}",
              "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}",
              "subscriptions_url": "https://api.github.com/users/simonw/subscriptions",
              "organizations_url": "https://api.github.com/users/simonw/orgs",
              "repos_url": "https://api.github.com/users/simonw/repos",
              "events_url": "https://api.github.com/users/simonw/events{/privacy}",
              "received_events_url": "https://api.github.com/users/simonw/received_events",
              "type": "User",
              "site_admin": false
          },
          "body": "Running this should create a `merged_by` column on the `pull_requests` table which is a foreign key to the `users` table.",
          "created_at": "2020-10-06T21:22:47Z",
          "updated_at": "2020-10-20T20:56:33Z",
          "html_url": "https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500603838",
          "pull_request_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48",
          "author_association": "MEMBER",
          "_links": {
              "self": {
                  "href": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500603838"
              },
              "html": {
                  "href": "https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500603838"
              },
              "pull_request": {
                  "href": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"
              }
          },
          "original_commit_id": "4f33b850bd37829262dd29e1c520afffebedc19c"
      },
      {
          "id": 500606198,
          "node_id": "MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwNjE5OA==",
          "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606198",
          "pull_request_review_id": 503368921,
          "diff_hunk": "@@ -0,0 +1,124 @@\n+from github_to_sqlite import utils\n+import pytest\n+import pathlib\n+import sqlite_utils\n+from sqlite_utils.db import ForeignKey\n+import json\n+\n+\n+@pytest.fixture\n+def pull_requests():\n+    return json.load(open(pathlib.Path(__file__).parent / \"pull_requests.json\"))\n+\n+\n+@pytest.fixture\n+def db(pull_requests):\n+    db = sqlite_utils.Database(memory=True)\n+    db[\"repos\"].insert(\n+        {\"id\": 1},\n+        pk=\"id\",\n+        columns={\"organization\": int, \"topics\": str, \"name\": str, \"description\": str},\n+    )\n+    utils.save_pull_requests(db, pull_requests, {\"id\": 1})\n+    return db\n+\n+\n+def test_tables(db):\n+    assert {\"pull_requests\", \"users\", \"repos\", \"milestones\"} == set(\n+        db.table_names()\n+    )\n+    assert {\n+        ForeignKey(\n+            table=\"pull_requests\", column=\"repo\", other_table=\"repos\", other_column=\"id\"\n+        ),\n+        ForeignKey(\n+            table=\"pull_requests\",\n+            column=\"milestone\",\n+            other_table=\"milestones\",\n+            other_column=\"id\",\n+        ),\n+        ForeignKey(\n+            table=\"pull_requests\", column=\"assignee\", other_table=\"users\", other_column=\"id\"\n+        ),\n+        ForeignKey(\n+            table=\"pull_requests\", column=\"user\", other_table=\"users\", other_column=\"id\"\n+        ),\n+    } == set(db[\"pull_requests\"].foreign_keys)\n+\n+\n+def test_pull_requests(db):\n+    pull_request_rows = list(db[\"pull_requests\"].rows)\n+    assert [\n+        {\n+            'id': 313384926,\n+            'node_id': 'MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2',\n+            'number': 571,\n+            'state': 'closed',\n+            'locked': 0,\n+            'title': 'detect_fts now works with alternative table escaping',\n+            'user': 9599,\n+            'body': 'Fixes #570',\n+            'created_at': '2019-09-03T00:23:39Z',\n+            'updated_at': '2019-09-03T00:32:28Z',\n+            'closed_at': '2019-09-03T00:32:28Z',\n+            'merged_at': '2019-09-03T00:32:28Z',\n+            'merge_commit_sha': '2dc5c8dc259a0606162673d394ba8cc1c6f54428',\n+            'assignee': None,\n+            'milestone': None,\n+            'draft': 0,\n+            'head': 'a85239f69261c10f1a9f90514c8b5d113cb94585',\n+            'base': 'f04deebec4f3842f7bd610cd5859de529f77d50e',\n+            'author_association': 'OWNER',\n+            'merged': 1,\n+            'mergeable': None,\n+            'rebaseable': None,\n+            'mergeable_state': 'unknown',\n+            'merged_by': '{\"login\": \"simonw\", \"id\": 9599, \"node_id\": \"MDQ6VXNlcjk1OTk=\", \"avatar_url\": \"https://avatars0.githubusercontent.com/u/9599?v=4\", \"gravatar_id\": \"\", \"url\": \"https://api.github.com/users/simonw\", \"html_url\": \"https://github.com/simonw\", \"followers_url\": \"https://api.github.com/users/simonw/followers\", \"following_url\": \"https://api.github.com/users/simonw/following{/other_user}\", \"gists_url\": \"https://api.github.com/users/simonw/gists{/gist_id}\", \"starred_url\": \"https://api.github.com/users/simonw/starred{/owner}{/repo}\", \"subscriptions_url\": \"https://api.github.com/users/simonw/subscriptions\", \"organizations_url\": \"https://api.github.com/users/simonw/orgs\", \"repos_url\": \"https://api.github.com/users/simonw/repos\", \"events_url\": \"https://api.github.com/users/simonw/events{/privacy}\", \"received_events_url\": \"https://api.github.com/users/simonw/received_events\", \"type\": \"User\", \"site_admin\": false}',",
          "path": "tests/test_pull_requests.py",
          "position": null,
          "original_position": 76,
          "commit_id": "3a0d5c498f9faae4e40aab204cd01b965a4f61f3",
          "user": {
              "login": "simonw",
              "id": 9599,
              "node_id": "MDQ6VXNlcjk1OTk=",
              "avatar_url": "https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4",
              "gravatar_id": "",
              "url": "https://api.github.com/users/simonw",
              "html_url": "https://github.com/simonw",
              "followers_url": "https://api.github.com/users/simonw/followers",
              "following_url": "https://api.github.com/users/simonw/following{/other_user}",
              "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}",
              "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}",
              "subscriptions_url": "https://api.github.com/users/simonw/subscriptions",
              "organizations_url": "https://api.github.com/users/simonw/orgs",
              "repos_url": "https://api.github.com/users/simonw/repos",
              "events_url": "https://api.github.com/users/simonw/events{/privacy}",
              "received_events_url": "https://api.github.com/users/simonw/received_events",
              "type": "User",
              "site_admin": false
          },
          "body": "See above - this should be 9599, an integer reference to the row in the users table.",
          "created_at": "2020-10-06T21:27:43Z",
          "updated_at": "2020-10-20T20:56:33Z",
          "html_url": "https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606198",
          "pull_request_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48",
          "author_association": "MEMBER",
          "_links": {
              "self": {
                  "href": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606198"
              },
              "html": {
                  "href": "https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606198"
              },
              "pull_request": {
                  "href": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"
              }
          },
          "original_commit_id": "4f33b850bd37829262dd29e1c520afffebedc19c"
      },
      {
          "id": 500606665,
          "node_id": "MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwNjY2NQ==",
          "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606665",
          "pull_request_review_id": 503368921,
          "diff_hunk": "@@ -0,0 +1,124 @@\n+from github_to_sqlite import utils\n+import pytest\n+import pathlib\n+import sqlite_utils\n+from sqlite_utils.db import ForeignKey\n+import json\n+\n+\n+@pytest.fixture\n+def pull_requests():\n+    return json.load(open(pathlib.Path(__file__).parent / \"pull_requests.json\"))\n+\n+\n+@pytest.fixture\n+def db(pull_requests):\n+    db = sqlite_utils.Database(memory=True)\n+    db[\"repos\"].insert(\n+        {\"id\": 1},\n+        pk=\"id\",\n+        columns={\"organization\": int, \"topics\": str, \"name\": str, \"description\": str},\n+    )\n+    utils.save_pull_requests(db, pull_requests, {\"id\": 1})\n+    return db\n+\n+\n+def test_tables(db):\n+    assert {\"pull_requests\", \"users\", \"repos\", \"milestones\"} == set(\n+        db.table_names()\n+    )\n+    assert {\n+        ForeignKey(\n+            table=\"pull_requests\", column=\"repo\", other_table=\"repos\", other_column=\"id\"\n+        ),\n+        ForeignKey(\n+            table=\"pull_requests\",\n+            column=\"milestone\",\n+            other_table=\"milestones\",\n+            other_column=\"id\",\n+        ),\n+        ForeignKey(\n+            table=\"pull_requests\", column=\"assignee\", other_table=\"users\", other_column=\"id\"\n+        ),\n+        ForeignKey(\n+            table=\"pull_requests\", column=\"user\", other_table=\"users\", other_column=\"id\"\n+        ),\n+    } == set(db[\"pull_requests\"].foreign_keys)\n+\n+\n+def test_pull_requests(db):\n+    pull_request_rows = list(db[\"pull_requests\"].rows)\n+    assert [\n+        {\n+            'id': 313384926,",
          "path": "tests/test_pull_requests.py",
          "position": null,
          "original_position": 53,
          "commit_id": "3a0d5c498f9faae4e40aab204cd01b965a4f61f3",
          "user": {
              "login": "simonw",
              "id": 9599,
              "node_id": "MDQ6VXNlcjk1OTk=",
              "avatar_url": "https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4",
              "gravatar_id": "",
              "url": "https://api.github.com/users/simonw",
              "html_url": "https://github.com/simonw",
              "followers_url": "https://api.github.com/users/simonw/followers",
              "following_url": "https://api.github.com/users/simonw/following{/other_user}",
              "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}",
              "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}",
              "subscriptions_url": "https://api.github.com/users/simonw/subscriptions",
              "organizations_url": "https://api.github.com/users/simonw/orgs",
              "repos_url": "https://api.github.com/users/simonw/repos",
              "events_url": "https://api.github.com/users/simonw/events{/privacy}",
              "received_events_url": "https://api.github.com/users/simonw/received_events",
              "type": "User",
              "site_admin": false
          },
          "body": "Minor detail: I use Black for this repo, which requires double quotes - running \"black .\" in the root directory (with the latest version of Black) should handle this for you.",
          "created_at": "2020-10-06T21:28:31Z",
          "updated_at": "2020-10-20T20:56:33Z",
          "html_url": "https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606665",
          "pull_request_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48",
          "author_association": "MEMBER",
          "_links": {
              "self": {
                  "href": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606665"
              },
              "html": {
                  "href": "https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606665"
              },
              "pull_request": {
                  "href": "https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"
              }
          },
          "original_commit_id": "4f33b850bd37829262dd29e1c520afffebedc19c"
      }
  ]
  ```
  That's a lot more interesting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483820
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDczNTQ4MzgyMA==
created_at: 2020-11-30T00:27:47Z
updated_at: 2020-11-30T00:27:47Z
author_association: MEMBER
body:
  So it looks like anything that pulls reviews needs to pull each review, then for each one pull the comments.

  I'm going to consider this blocked on smarter rate limit handling in #51.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-735484186
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51
node_id: MDEyOklzc3VlQ29tbWVudDczNTQ4NDE4Ng==
created_at: 2020-11-30T00:29:31Z
updated_at: 2020-11-30T00:29:31Z
author_association: MEMBER
body:
  This just caused a failure in deploying the demo: https://github.com/dogsheep/github-to-sqlite/runs/1471304407?check_suite_focus=true

  ```
    File "/opt/hostedtoolcache/Python/3.8.6/x64/bin/github-to-sqlite", line 33, in <module>
      sys.exit(load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')())
    File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 142, in issue_comments
      for comment in utils.fetch_issue_comments(repo, token, issue):
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 380, in fetch_issue_comments
      for comments in paginate(url, headers):
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 472, in paginate
      raise GitHubError.from_response(response)
  github_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403)
  Error: Process completed with exit code 1.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/53#issuecomment-735485677
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/53
node_id: MDEyOklzc3VlQ29tbWVudDczNTQ4NTY3Nw==
created_at: 2020-11-30T00:36:09Z
updated_at: 2020-11-30T00:36:09Z
author_association: MEMBER
body:
  Given rate limits (see #51) this command might be better implemented by running a `git clone` into a temporary directory - doing so would retrieve all of the files in one go.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/263#issuecomment-735960132
issue_url: https://api.github.com/repos/simonw/datasette/issues/263
node_id: MDEyOklzc3VlQ29tbWVudDczNTk2MDEzMg==
created_at: 2020-11-30T18:25:17Z
updated_at: 2020-11-30T18:25:17Z
author_association: OWNER
body:
  Fixing this would unblock this issue for switching `datasette-graphql` to using `datasette.client` internally: https://github.com/simonw/datasette-graphql/issues/61

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1116#issuecomment-735992106
issue_url: https://api.github.com/repos/simonw/datasette/issues/1116
node_id: MDEyOklzc3VlQ29tbWVudDczNTk5MjEwNg==
created_at: 2020-11-30T19:27:10Z
updated_at: 2020-11-30T19:27:10Z
author_association: OWNER
body:
  I'm treating this as a bug - these columns should definitely be visible in Datasette.

  I created my own test database using SQLite from Homebrew like this:
  ```
  /usr/local/Cellar/sqlite/3.33.0/bin/sqlite3 deeds.db << EOF
  CREATE TABLE deeds (
      body TEXT,
      id INT GENERATED ALWAYS AS (json_extract(body, '$.id')) STORED,
      consideration INT GENERATED ALWAYS AS (json_extract(body, '$.consideration')) STORED
  );
  INSERT INTO deeds (body) VALUES ('{
      "id": 1,
      "consideration": "This is the consideration"
  }');
  EOF
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1116#issuecomment-735993935
issue_url: https://api.github.com/repos/simonw/datasette/issues/1116
node_id: MDEyOklzc3VlQ29tbWVudDczNTk5MzkzNQ==
created_at: 2020-11-30T19:30:44Z
updated_at: 2020-11-30T19:32:15Z
author_association: OWNER
body:
  It looks like `PRAGMA table_info` skips "hidden" columns: https://www.sqlite.org/pragma.html#pragma_table_info

  But `PRAGMA table_xinfo` does not: https://www.sqlite.org/pragma.html#pragma_table_xinfo

  Compare https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_info%28%27searchable%27%29 to https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_xinfo%28%27searchable%27%29 - the `xinfo` one has an additional `hidden` column.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1116#issuecomment-735995695
issue_url: https://api.github.com/repos/simonw/datasette/issues/1116
node_id: MDEyOklzc3VlQ29tbWVudDczNTk5NTY5NQ==
created_at: 2020-11-30T19:34:15Z
updated_at: 2020-11-30T19:34:15Z
author_association: OWNER
body:
  Generated column support was added in SQLite 3.31.0, so any unit tests I write for this should use skipIf to only run on that version or later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1116#issuecomment-736004383
issue_url: https://api.github.com/repos/simonw/datasette/issues/1116
node_id: MDEyOklzc3VlQ29tbWVudDczNjAwNDM4Mw==
created_at: 2020-11-30T19:51:51Z
updated_at: 2020-11-30T19:51:51Z
author_association: OWNER
body:
  This change will also have an impact on how hidden virtual FTS tables are displayed, since apparently those have some hidden columns: https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_xinfo%28%27searchable_fts%27%29

  | cid | name | type | notnull | dflt_value | pk | hidden |
  | --- | --- | --- | --- | --- | --- | --- |
  | 0 | text1 |   | 0 |   | 0 | 0 |
  | 1 | text2 |   | 0 |   | 0 | 0 |
  | 2 | name with . and spaces |   | 0 |   | 0 | 0 |
  | 3 | searchable_fts |   | 0 |   | 0 | 1 |
  | 4 | docid |   | 0 |   | 0 | 1 |
  | 5 | __langid |   | 0 |   | 0 | 1 |

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1116#issuecomment-736005833
issue_url: https://api.github.com/repos/simonw/datasette/issues/1116
node_id: MDEyOklzc3VlQ29tbWVudDczNjAwNTgzMw==
created_at: 2020-11-30T19:54:39Z
updated_at: 2020-11-30T19:54:39Z
author_association: NONE
body:
  @simonw thanks for investigating so quickly.  If it is undesirable to change that hidden behavior, maybe something like this is a suitable workaround:
  ```
  SELECT * FROM pragma_table_xinfo('deeds') where hidden in (0,2);
  0|body|TEXT|0||0|0
  1|id|INT GENERATED ALWAYS|0||0|2
  2|consideration|INT GENERATED ALWAYS|0||0|2
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1116#issuecomment-736014372
issue_url: https://api.github.com/repos/simonw/datasette/issues/1116
node_id: MDEyOklzc3VlQ29tbWVudDczNjAxNDM3Mg==
created_at: 2020-11-30T20:08:48Z
updated_at: 2020-11-30T20:08:48Z
author_association: OWNER
body:
  Ouch, the tests pass on my laptop but failed in CI: https://github.com/simonw/datasette/actions/runs/392367997

  Lots of failures look like this:
  ```
  ERROR: conn=<sqlite3.Connection object at 0x7f44f0494030>, sql = 'select rowid,  from facetable order by rowid limit 51', params = {}: near "from": syntax error
  ```
  Note the `select rowid, from...` - so it looks like invalid SQL queries are being constructed maybe due to mis-detecting columns somehow.

  I wonder why it didn't fail on my laptop?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1117#issuecomment-736018609
issue_url: https://api.github.com/repos/simonw/datasette/issues/1117
node_id: MDEyOklzc3VlQ29tbWVudDczNjAxODYwOQ==
created_at: 2020-11-30T20:17:31Z
updated_at: 2020-11-30T20:17:31Z
author_association: OWNER
body:
  I need to replicate these failures on my laptop. My hunch is that this is down to the version of SQLite available to Python.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1117#issuecomment-736023089
issue_url: https://api.github.com/repos/simonw/datasette/issues/1117
node_id: MDEyOklzc3VlQ29tbWVudDczNjAyMzA4OQ==
created_at: 2020-11-30T20:26:27Z
updated_at: 2020-11-30T20:26:27Z
author_association: OWNER
body:
  On my laptop:
  ```
  platform darwin -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
  SQLite: 3.33.0
  ```
  In CI they are all SQLite: 3.22.0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1117#issuecomment-736028726
issue_url: https://api.github.com/repos/simonw/datasette/issues/1117
node_id: MDEyOklzc3VlQ29tbWVudDczNjAyODcyNg==
created_at: 2020-11-30T20:37:50Z
updated_at: 2020-11-30T20:37:50Z
author_association: OWNER
body:
  This kind of problem is why I have a `tmate` workflow:

  <img width="1086" alt="Actions_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/100660845-72dff680-3307-11eb-89dc-211d226f68dd.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1117#issuecomment-736029337
issue_url: https://api.github.com/repos/simonw/datasette/issues/1117
node_id: MDEyOklzc3VlQ29tbWVudDczNjAyOTMzNw==
created_at: 2020-11-30T20:39:06Z
updated_at: 2020-11-30T20:39:06Z
author_association: OWNER
body:
  Here's the problem: https://www.sqlite.org/changes.html#version_3_26_0

  > ### 2018-12-01 (3.26.0)
  > 
  > -  Added [PRAGMA table_xinfo](https://www.sqlite.org/pragma.html#pragma_table_xinfo) that works just like [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that it also shows [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables.

  CI is running 3.22.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1117#issuecomment-736067475
issue_url: https://api.github.com/repos/simonw/datasette/issues/1117
node_id: MDEyOklzc3VlQ29tbWVudDczNjA2NzQ3NQ==
created_at: 2020-11-30T21:28:22Z
updated_at: 2020-11-30T21:28:22Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=h1) Report
  > Merging [#1117](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=desc) (ccdf2c6) into [main](https://codecov.io/gh/simonw/datasette/commit/dea3c508b39528e566d711c38a467b3d372d220b?el=desc) (dea3c50) will **decrease** coverage by `0.00%`.
  > The diff coverage is `95.23%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1117/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1117      +/-   ##
  ==========================================
  - Coverage   91.48%   91.48%   -0.01%     
  ==========================================
    Files          30       31       +1     
    Lines        3841     3852      +11     
  ==========================================
  + Hits         3514     3524      +10     
  - Misses        327      328       +1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1117/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.10% <87.50%> (-0.20%)` | :arrow_down: |
  | [datasette/utils/sqlite.py](https://codecov.io/gh/simonw/datasette/pull/1117/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL3NxbGl0ZS5weQ==) | `100.00% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=footer). Last update [dea3c50...ccdf2c6](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1117#issuecomment-736088949
issue_url: https://api.github.com/repos/simonw/datasette/issues/1117
node_id: MDEyOklzc3VlQ29tbWVudDczNjA4ODk0OQ==
created_at: 2020-11-30T22:15:58Z
updated_at: 2020-11-30T22:23:19Z
author_association: NONE
body:
  I just deployed this and its working great.

  ~In a very unscientific benchmark my response times went from around 22-25ms to 33-36ms, but I didn't even dig enough to confirm the latency is related to the change.  It's on a VPS, so maybe the load changed.~ I don't see any difference in performance.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1120#issuecomment-736135125
issue_url: https://api.github.com/repos/simonw/datasette/issues/1120
node_id: MDEyOklzc3VlQ29tbWVudDczNjEzNTEyNQ==
created_at: 2020-12-01T00:22:36Z
updated_at: 2020-12-01T00:22:36Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=h1) Report
  > Merging [#1120](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=desc) (ddad8db) into [main](https://codecov.io/gh/simonw/datasette/commit/461670a0b87efa953141b449a9a261919864ceb3?el=desc) (461670a) will **increase** coverage by `0.00%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1120/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1120   +/-   ##
  =======================================
    Coverage   91.48%   91.49%           
  =======================================
    Files          31       31           
    Lines        3852     3856    +4     
  =======================================
  + Hits         3524     3528    +4     
    Misses        328      328           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1120/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.10% <100.00%> (ø)` | |
  | [datasette/utils/sqlite.py](https://codecov.io/gh/simonw/datasette/pull/1120/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL3NxbGl0ZS5weQ==) | `100.00% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=footer). Last update [461670a...ddad8db](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1119#issuecomment-736142201
issue_url: https://api.github.com/repos/simonw/datasette/issues/1119
node_id: MDEyOklzc3VlQ29tbWVudDczNjE0MjIwMQ==
created_at: 2020-12-01T00:41:14Z
updated_at: 2020-12-01T00:41:14Z
author_association: OWNER
body:
  On my laptop:

  <img width="1123" alt="fixtures__generated_columns__1_row" src="https://user-images.githubusercontent.com/9599/100681839-8734eb00-3329-11eb-8aea-813bdc18efa1.png">

  https://latest.datasette.io/-/versions is running SQLite 3.27.2 at the moment so it won't show that table until it gets to 3.31.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-736173084
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: MDEyOklzc3VlQ29tbWVudDczNjE3MzA4NA==
created_at: 2020-12-01T02:20:58Z
updated_at: 2020-12-01T02:20:58Z
author_association: NONE
body:
  Are there common patterns for storing column-based metadata inside SQLite itself? I know Postgres allows "comment" fields, which this is kind of trying to replicate. Should the `units` and `description` and possibly other per-column metadata fields be combined into a single (tabular?) structure, that would be displayed above the data on the table / query results page?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1122#issuecomment-736318377
issue_url: https://api.github.com/repos/simonw/datasette/issues/1122
node_id: MDEyOklzc3VlQ29tbWVudDczNjMxODM3Nw==
created_at: 2020-12-01T08:47:33Z
updated_at: 2020-12-01T08:47:33Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1122?src=pr&el=h1) Report
  > Merging [#1122](https://codecov.io/gh/simonw/datasette/pull/1122?src=pr&el=desc) (94ea22f) into [main](https://codecov.io/gh/simonw/datasette/commit/a970276b9999687b96c5e11ea1c817d814f5d267?el=desc) (a970276) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1122/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1122?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1122   +/-   ##
  =======================================
    Coverage   91.49%   91.49%           
  =======================================
    Files          31       31           
    Lines        3856     3856           
  =======================================
    Hits         3528     3528           
    Misses        328      328           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1122?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1122?src=pr&el=footer). Last update [a970276...94ea22f](https://codecov.io/gh/simonw/datasette/pull/1122?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1111#issuecomment-736322290
issue_url: https://api.github.com/repos/simonw/datasette/issues/1111
node_id: MDEyOklzc3VlQ29tbWVudDczNjMyMjI5MA==
created_at: 2020-12-01T08:54:47Z
updated_at: 2020-12-01T08:54:47Z
author_association: CONTRIBUTOR
body:
  Somewhat related: https://github.com/simonw/datasette/issues/859
  I fixed the issue with forking and disabling the counts for hidden tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-737402392
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: MDEyOklzc3VlQ29tbWVudDczNzQwMjM5Mg==
created_at: 2020-12-02T18:08:55Z
updated_at: 2020-12-02T18:08:55Z
author_association: OWNER
body:
  SQLite does let you add comments in your CREATE TABLE statements:
  ```sql
  CREATE TABLE something (
      id integer primary key, -- integer primary key
      created text -- created date as ISO datetime
  );
  ```
  But the only mechanism for reading those back is to retrieve that `CREATE TABLE` block of SQL from the `sqlite_master` table and run a parser against it.

  I've so far resisted adding a SQL syntax parser to Datasette for complexity reasons - though I'm increasingly thinking I'll need to do it at some point.

  I think I'll leave this to plugins. I'm definitely going to build a plugin that lets you store metadata for tables and columns in a SQLite database table, which will then support interactively editing metadata through a UI.

  A plugin which extracts column comments from the SQLite CREATE TABLE comments would be feasible too, if I design the plugin hooks well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-737428262
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: MDEyOklzc3VlQ29tbWVudDczNzQyODI2Mg==
created_at: 2020-12-02T18:55:21Z
updated_at: 2020-12-02T18:55:21Z
author_association: NONE
body:
  Are you thinking that those metadata tables would be added to the SQLite DB by Datasette, when you tell it to wrap up the database, with the metadata coming from the `metadata.json`? Would it be easy to allow the prepopulation of those tables in the database itself? We've been struggling with the best way to make sure that the data is always accompanied by metadata, and baking it all into the database itself would be nice, since then we wouldn't need to worry about separately distributing different files in different contexts.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-737463116
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: MDEyOklzc3VlQ29tbWVudDczNzQ2MzExNg==
created_at: 2020-12-02T20:02:10Z
updated_at: 2020-12-02T20:03:01Z
author_association: OWNER
body:
  My idea is that if you installed my proposed plugin you wouldn't need `metadata.json` at all - your metadata would instead live in a table in the connected SQLite database files - either one table per database (so the metadata can live in the same place as the data) or maybe also in a dedicated separate database file, for if you want to add metadata to an otherwise read-only database.

  The plugin would then provide a UI for editing that metadata - maybe by configuring some writable canned queries or maybe something more custom than that. Or you could edit the metadata by manually editing the SQLite database file (or loading data into it using a tool like [yaml-to-sqlite](https://github.com/simonw/yaml-to-sqlite)).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/749#issuecomment-737563699
issue_url: https://api.github.com/repos/simonw/datasette/issues/749
node_id: MDEyOklzc3VlQ29tbWVudDczNzU2MzY5OQ==
created_at: 2020-12-02T23:45:42Z
updated_at: 2020-12-02T23:45:42Z
author_association: OWNER
body:
  I asked about this on Twitter - https://twitter.com/steren/status/1334281184965140483

  > You simply need to send the `Transfer-Encoding: chunked` header.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/749#issuecomment-737580084
issue_url: https://api.github.com/repos/simonw/datasette/issues/749
node_id: MDEyOklzc3VlQ29tbWVudDczNzU4MDA4NA==
created_at: 2020-12-03T00:31:14Z
updated_at: 2020-12-03T00:31:14Z
author_association: OWNER
body:
  This works!

  ```
  /tmp % wget 'https://covid-19.datasettes.com/covid.db'
  --2020-12-02 16:28:02--  https://covid-19.datasettes.com/covid.db
  Resolving covid-19.datasettes.com (covid-19.datasettes.com)... 172.217.5.83
  Connecting to covid-19.datasettes.com (covid-19.datasettes.com)|172.217.5.83|:443... connected.
  HTTP request sent, awaiting response... 200 OK
  Length: unspecified [application/octet-stream]
  Saving to: ‘covid.db’

  covid.db                [              <=>   ] 306.42M  3.27MB/s    in 98s     

  2020-12-02 16:29:40 (3.13 MB/s) - ‘covid.db’ saved [321306624]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1100#issuecomment-737581719
issue_url: https://api.github.com/repos/simonw/datasette/issues/1100
node_id: MDEyOklzc3VlQ29tbWVudDczNzU4MTcxOQ==
created_at: 2020-12-03T00:35:23Z
updated_at: 2020-12-03T00:35:23Z
author_association: OWNER
body:
  Replicated this against the live demo as well:
  ```
  /tmp % curl -XOPTIONS https://latest.datasette.io/fixtures.json
  {"ok": false, "error": "object Response can't be used in 'await' expression", "status": 500, "title": null}%                                                                                                                                  /tmp % 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1123#issuecomment-737586248
issue_url: https://api.github.com/repos/simonw/datasette/issues/1123
node_id: MDEyOklzc3VlQ29tbWVudDczNzU4NjI0OA==
created_at: 2020-12-03T00:47:37Z
updated_at: 2020-12-03T00:47:37Z
author_association: OWNER
body:
  Affected tests:
  ```
  FAILED tests/test_plugins.py::test_hook_table_actions[facetable] - AssertionE...
  FAILED tests/test_plugins.py::test_hook_table_actions[simple_view] - Assertio...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1121#issuecomment-737591281
issue_url: https://api.github.com/repos/simonw/datasette/issues/1121
node_id: MDEyOklzc3VlQ29tbWVudDczNzU5MTI4MQ==
created_at: 2020-12-03T01:03:18Z
updated_at: 2020-12-03T01:03:18Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures?_bot=1

  <img width="729" alt="fixtures" src="https://user-images.githubusercontent.com/9599/100950004-43c0b500-34c0-11eb-918c-aa959376461f.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1124#issuecomment-738209642
issue_url: https://api.github.com/repos/simonw/datasette/issues/1124
node_id: MDEyOklzc3VlQ29tbWVudDczODIwOTY0Mg==
created_at: 2020-12-03T18:39:19Z
updated_at: 2020-12-03T18:39:19Z
author_association: OWNER
body:
  The CSS files are in the expected location:
  ```
  [ec2-user@ip-172-31-30-7 ~]$ find /home/ec2-user/.local/pipx/venvs/datasette | grep css
  /home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css
  /home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/codemirror-5.57.0.min.css
  ```
  Wow it's running an ANCIENT version of SQLite:
  ```
  [ec2-user@ip-172-31-30-7 ~]$ datasette --get /-/versions.json
  {"python": {"version": "3.7.9", "full": "3.7.9 (default, Aug 27 2020, 21:58:41) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)]"}, "datasette": {"version": "0.52.2"}, "asgi": "3.0", "uvicorn": "0.12.3", "sqlite": {"version": "3.7.17", "fts_versions": ["FTS4", "FTS3"], "extensions": {}, "compile_options": ["DISABLE_DIRSYNC", "ENABLE_COLUMN_METADATA", "ENABLE_FTS3", "ENABLE_RTREE", "ENABLE_UNLOCK_NOTIFY", "SECURE_DELETE", "TEMP_STORE=1", "THREADSAFE=1"]}}
  ```
  http://www.sqlite.org/releaselog/3_7_17.html - SQLite Release 3.7.17 On 2013-05-20

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1124#issuecomment-738211152
issue_url: https://api.github.com/repos/simonw/datasette/issues/1124
node_id: MDEyOklzc3VlQ29tbWVudDczODIxMTE1Mg==
created_at: 2020-12-03T18:42:12Z
updated_at: 2020-12-03T18:42:12Z
author_association: OWNER
body:
  Added a line to print out `app_root` from https://github.com/simonw/datasette/blob/e048791a9a2686f47d81a2c8aa88aa1966d82521/datasette/app.py#L848-L853
  ```
  app_root =  /home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1124#issuecomment-738211776
issue_url: https://api.github.com/repos/simonw/datasette/issues/1124
node_id: MDEyOklzc3VlQ29tbWVudDczODIxMTc3Ng==
created_at: 2020-12-03T18:43:21Z
updated_at: 2020-12-03T18:43:21Z
author_association: OWNER
body:
  I'm suspicious of this code here:https://github.com/simonw/datasette/blob/e048791a9a2686f47d81a2c8aa88aa1966d82521/datasette/utils/asgi.py#L284-L307

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1124#issuecomment-738213342
issue_url: https://api.github.com/repos/simonw/datasette/issues/1124
node_id: MDEyOklzc3VlQ29tbWVudDczODIxMzM0Mg==
created_at: 2020-12-03T18:46:22Z
updated_at: 2020-12-03T21:40:51Z
author_association: OWNER
body:
  I replaced that function with this code:
  ```python
  def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):
      async def inner_static(request, send):
          path = request.scope["url_route"]["kwargs"]["path"]
          print("path =", path)
          try:
              full_path = (Path(root_path) / path).resolve().absolute()
          except FileNotFoundError as e:
              print("FileNotFoundError:", e)
              await asgi_send_html(send, "404", 404)
              return
          if full_path.is_dir():
              await asgi_send_html(send, "403: Directory listing is not allowed", 403)
              return
          # Ensure full_path is within root_path to avoid weird "../" tricks
          try:
              print("full_path={}, root_path={}".format(full_path, root_path))
              full_path.relative_to(root_path)
          except ValueError as e:
              print("  ValueError:", e)
              await asgi_send_html(send, "404", 404)
              return
          try:
              await asgi_send_file(send, full_path, chunk_size=chunk_size)
          except FileNotFoundError:
              await asgi_send_html(send, "404", 404)
              return

      return inner_static
  ```
  Edited using `vi /home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/utils/asgi.py`

  The output shows me what the bug is:
  ```
  $ datasette --get /-/static/app.css --pdb
  app_root =  /home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages
  path = app.css
  full_path=/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css, root_path=/home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages/datasette/static
    ValueError: '/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css' does not start with '/home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages/datasette/static'
  404
  ```
  `  ValueError: '/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css' does not start with '/home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages/datasette/static'`

  One is `../lib/python3.7/..` and the other is `../lib64/python3.7/..` - there's clearly some kind of symlink in play here which I'm not taking into account.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1124#issuecomment-738215487
issue_url: https://api.github.com/repos/simonw/datasette/issues/1124
node_id: MDEyOklzc3VlQ29tbWVudDczODIxNTQ4Nw==
created_at: 2020-12-03T18:50:26Z
updated_at: 2020-12-03T21:41:25Z
author_association: OWNER
body:
  This fix works - calling `.resolve()` on the `root_path` before the comparison to ensure symlinks are resolved:
  ```python
          # Ensure full_path is within root_path to avoid weird "../" tricks
          try:
              print("full_path={}, root_path={}".format(full_path, root_path))
              full_path.relative_to(root_path.resolve())
          except ValueError as e:
              print("  ValueError:", e)
              await asgi_send_html(send, "404", 404)
              return
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1124#issuecomment-738215686
issue_url: https://api.github.com/repos/simonw/datasette/issues/1124
node_id: MDEyOklzc3VlQ29tbWVudDczODIxNTY4Ng==
created_at: 2020-12-03T18:50:48Z
updated_at: 2020-12-03T21:42:02Z
author_association: OWNER
body:
  I'm going to punt on writing a unit test for this (not sure how I'd simulate those symlinks) - I'll manually test it and push out a dot release instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1124#issuecomment-738220067
issue_url: https://api.github.com/repos/simonw/datasette/issues/1124
node_id: MDEyOklzc3VlQ29tbWVudDczODIyMDA2Nw==
created_at: 2020-12-03T18:58:17Z
updated_at: 2020-12-03T18:58:17Z
author_association: OWNER
body:
  I tested this by running:

      pipx uninstall datasette
      pipx install 'https://github.com/simonw/datasette/archive/6b4c55efea3e9d34d92cbe5f0066553ad9b14071.zip'

  To replace that version of Datasette (in the correct virtual environment) with this patch. It worked!

  ```
  [ec2-user@ip-172-31-30-7 ~]$ datasette --get /-/static/app.css
  /* Reset and Page Setup ==================================================== */
  ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1124#issuecomment-738224865
issue_url: https://api.github.com/repos/simonw/datasette/issues/1124
node_id: MDEyOklzc3VlQ29tbWVudDczODIyNDg2NQ==
created_at: 2020-12-03T19:01:52Z
updated_at: 2020-12-03T19:01:52Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/runs/1494631261
  ```
  /home/runner/work/datasette/datasette/tests/test_html.py:81: AssertionError
  ----------------------------- Captured stderr call -----------------------------
  Traceback (most recent call last):
    File "/home/runner/work/datasette/datasette/datasette/app.py", line 1039, in route_path
      response = await view(request, send)
    File "/home/runner/work/datasette/datasette/datasette/utils/asgi.py", line 297, in inner_static
      full_path.relative_to(root_path.resolve())
  AttributeError: 'str' object has no attribute 'resolve'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1124#issuecomment-738313399
issue_url: https://api.github.com/repos/simonw/datasette/issues/1124
node_id: MDEyOklzc3VlQ29tbWVudDczODMxMzM5OQ==
created_at: 2020-12-03T21:10:54Z
updated_at: 2020-12-03T21:10:54Z
author_association: OWNER
body:
  Confirmed that installing a fresh copy of Datasette 0.52.3 on that server works correctly as expected.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1125#issuecomment-738548693
issue_url: https://api.github.com/repos/simonw/datasette/issues/1125
node_id: MDEyOklzc3VlQ29tbWVudDczODU0ODY5Mw==
created_at: 2020-12-04T03:52:51Z
updated_at: 2020-12-04T03:52:51Z
author_association: OWNER
body:
  That didn't work - https://latest.datasette.io/-/versions isn't showing the package.

  I bet that's because I'm actually installing `pysqlite3-binary` here: https://github.com/simonw/datasette/blob/e2fea36540e952d8d72c1bd0af7144b85b7a4671/.github/workflows/deploy-latest.yml#L57

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1125#issuecomment-738550588
issue_url: https://api.github.com/repos/simonw/datasette/issues/1125
node_id: MDEyOklzc3VlQ29tbWVudDczODU1MDU4OA==
created_at: 2020-12-04T04:01:10Z
updated_at: 2020-12-04T04:01:10Z
author_association: OWNER
body:
  Urgh, figuring out the version of `pysqlite3` is WAY harder than I expected. The `getversion` module looks like the smartest attempt at solving this problem generally, but I'd like to avoid adding another dependency just for this: https://github.com/smarie/python-getversion

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1125#issuecomment-738554392
issue_url: https://api.github.com/repos/simonw/datasette/issues/1125
node_id: MDEyOklzc3VlQ29tbWVudDczODU1NDM5Mg==
created_at: 2020-12-04T04:16:57Z
updated_at: 2020-12-04T04:16:57Z
author_association: OWNER
body:
  https://latest.datasette.io/-/versions now shows this:
  ```json
  {
      "python": {
          "version": "3.8.6",
          "full": "3.8.6 (default, Nov 18 2020, 13:49:49) \n[GCC 8.3.0]"
      },
      "datasette": {
          "version": "0.52.3",
          "note": "49d8fc056844d5a537d6cfd96dab0dd5686fe718"
      },
      "asgi": "3.0",
      "uvicorn": "0.12.3",
      "sqlite": {
          "version": "3.33.0",
          "fts_versions": [
              "FTS5",
              "FTS4",
              "FTS3"
          ],
          "extensions": {
              "json1": null
          },
          "compile_options": []
      },
      "pysqlite3": "0.4.4"
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1128#issuecomment-738613497
issue_url: https://api.github.com/repos/simonw/datasette/issues/1128
node_id: MDEyOklzc3VlQ29tbWVudDczODYxMzQ5Nw==
created_at: 2020-12-04T07:17:12Z
updated_at: 2020-12-04T07:17:12Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=h1) Report
  > Merging [#1128](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=desc) (7004c3b) into [main](https://codecov.io/gh/simonw/datasette/commit/49d8fc056844d5a537d6cfd96dab0dd5686fe718?el=desc) (49d8fc0) will **decrease** coverage by `0.00%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1128/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1128      +/-   ##
  ==========================================
  - Coverage   91.42%   91.42%   -0.01%     
  ==========================================
    Files          31       31              
    Lines        3873     3872       -1     
  ==========================================
  - Hits         3541     3540       -1     
    Misses        332      332              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1128/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `92.13% <ø> (-0.04%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=footer). Last update [49d8fc0...7004c3b](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1130#issuecomment-738620153
issue_url: https://api.github.com/repos/simonw/datasette/issues/1130
node_id: MDEyOklzc3VlQ29tbWVudDczODYyMDE1Mw==
created_at: 2020-12-04T07:34:48Z
updated_at: 2020-12-04T07:34:48Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1130?src=pr&el=h1) Report
  > Merging [#1130](https://codecov.io/gh/simonw/datasette/pull/1130?src=pr&el=desc) (8d4c69c) into [main](https://codecov.io/gh/simonw/datasette/commit/49d8fc056844d5a537d6cfd96dab0dd5686fe718?el=desc) (49d8fc0) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1130/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1130?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1130   +/-   ##
  =======================================
    Coverage   91.42%   91.42%           
  =======================================
    Files          31       31           
    Lines        3873     3873           
  =======================================
    Hits         3541     3541           
    Misses        332      332           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1130?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1130?src=pr&el=footer). Last update [49d8fc0...8d4c69c](https://codecov.io/gh/simonw/datasette/pull/1130?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1130#issuecomment-738897582
issue_url: https://api.github.com/repos/simonw/datasette/issues/1130
node_id: MDEyOklzc3VlQ29tbWVudDczODg5NzU4Mg==
created_at: 2020-12-04T17:03:30Z
updated_at: 2020-12-04T17:03:30Z
author_association: OWNER
body:
  I deployed this to https://datasette-issue-1129.vercel.app/ (using `datasette publish vercel fixtures.db --branch 8d4c69c6fb0ef741a19070f5172017ea3522e83c --about_url https://github.com/simonw/datasette/issues/1129 --about datasette/issues/1129 --project datasette-issue-1129`) - weirdly, on Mobile Safari the footer appears just below the visible window:

  ![RPReplay_Final1607100726](https://user-images.githubusercontent.com/9599/101191950-336b2000-360f-11eb-8f14-ed83bd86515c.gif)

  I've seen other problems with fixed footers on Mobile Safari too: at Eventbrite this was a really nasty problem for us to figure out: https://www.eventbrite.com/engineering/mobile-safari-why/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/188#issuecomment-738905376
issue_url: https://api.github.com/repos/simonw/datasette/issues/188
node_id: MDEyOklzc3VlQ29tbWVudDczODkwNTM3Ng==
created_at: 2020-12-04T17:18:34Z
updated_at: 2020-12-04T17:18:34Z
author_association: OWNER
body:
  This is likely to be covered by plugin hooks: #860 for the metadata and after investigating in #1042 it looks like the existing `prepare_jinja2_environment` hook may already be enough to load templates from the database.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1130#issuecomment-738907852
issue_url: https://api.github.com/repos/simonw/datasette/issues/1130
node_id: MDEyOklzc3VlQ29tbWVudDczODkwNzg1Mg==
created_at: 2020-12-04T17:22:29Z
updated_at: 2020-12-04T17:31:25Z
author_association: CONTRIBUTOR
body:
  EDIT: I misunderstood the problem. This seems like a fix better suited for Safari. But I don't have any Apple device to test it.

  ```css
  body {
    min-height: 100vh;
    min-height: -webkit-fill-available;
  }
  html {
    height: -webkit-fill-available;
  }
  ```
  https://css-tricks.com/css-fix-for-100vh-in-mobile-webkit/

  ---

  It's actually not that difficult to fix.
  Well, this is actually a workaround to keep viewport in place.

  I usually put a transition (forgot to do it here) that keeps page from resizing.

  ```css
  .container {
    min-height: 100vh;
    transition: height 10000s steps(0);
  }
  ```

  `steps()` function prevents excessive layout calculations, and lets the page snap back into place (10000s ~= 3h later) in a single step.
  This fix also prevents page from jumping around when the keyboard pops up and down.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1131#issuecomment-739083472
issue_url: https://api.github.com/repos/simonw/datasette/issues/1131
node_id: MDEyOklzc3VlQ29tbWVudDczOTA4MzQ3Mg==
created_at: 2020-12-05T00:01:12Z
updated_at: 2020-12-05T00:01:12Z
author_association: OWNER
body:
  Here's why: https://github.com/simonw/datasette/blob/37f87b5e52e7f8ddd1c4ffcf368bd7a62a406a6d/datasette/database.py#L158-L163

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1131#issuecomment-739083673
issue_url: https://api.github.com/repos/simonw/datasette/issues/1131
node_id: MDEyOklzc3VlQ29tbWVudDczOTA4MzY3Mw==
created_at: 2020-12-05T00:02:10Z
updated_at: 2020-12-05T00:02:10Z
author_association: OWNER
body:
  https://clig.dev/#the-basics

  > **Send messaging to stderr**. Log messages, errors, and so on should all be sent to stderr. This means that when commands are piped together, these messages are displayed to the user and not fed into the next command.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1131#issuecomment-739414118
issue_url: https://api.github.com/repos/simonw/datasette/issues/1131
node_id: MDEyOklzc3VlQ29tbWVudDczOTQxNDExOA==
created_at: 2020-12-05T20:48:33Z
updated_at: 2020-12-05T20:48:33Z
author_association: OWNER
body:
  Oddly enough, I tried fixing this with `sys.stderr.write("{}\n".format(e))` - but my Click `CLIRunner` tests failed because `result.stderr` was an empty string. Adding `sys.stderr.flush()` to the code that output errors fixed that issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1132#issuecomment-740228858
issue_url: https://api.github.com/repos/simonw/datasette/issues/1132
node_id: MDEyOklzc3VlQ29tbWVudDc0MDIyODg1OA==
created_at: 2020-12-07T22:50:36Z
updated_at: 2020-12-07T22:50:36Z
author_association: OWNER
body:
  Documented here: https://docs.datasette.io/en/latest/json_api.html#column-filter-arguments

  Demo: https://latest.datasette.io/fixtures/facetable?tags__arraynotcontains=tag2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/815#issuecomment-740385032
issue_url: https://api.github.com/repos/simonw/datasette/issues/815
node_id: MDEyOklzc3VlQ29tbWVudDc0MDM4NTAzMg==
created_at: 2020-12-08T05:26:09Z
updated_at: 2020-12-08T05:26:16Z
author_association: OWNER
body:
  Sure! It's a bit of a fiddle one - I've not found an approach that I like, but I also haven't thought about it since June. I'd love to see what you come up with!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1133#issuecomment-740850920
issue_url: https://api.github.com/repos/simonw/datasette/issues/1133
node_id: MDEyOklzc3VlQ29tbWVudDc0MDg1MDkyMA==
created_at: 2020-12-08T18:55:59Z
updated_at: 2020-12-08T18:55:59Z
author_association: OWNER
body:
  Inspiration was this script: https://gist.github.com/simonw/f6e3cd29fde5d15ea9cd746c942046ba - which pipes output through `tail -n +2` to strip off the headers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-741992106
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDc0MTk5MjEwNg==
created_at: 2020-12-09T19:19:54Z
updated_at: 2020-12-09T20:27:45Z
author_association: OWNER
body:
  Could you try removing the `ProxyPassReverse /datasette http://0.0.0.0:8001` line?

  My hunch is that `ProxyPassReverse` is rewriting some of the links in the HTML (or maybe in the HTTP headers) in a way that breaks things.

  Normally you would need `ProxyPassReverse` to compensate for the underlying application being unable to rewrite its links - but Datasette's `base_url` setting causes Datasette to rewrite all of the links for you, so `ProxyPassReverse` should be unneccessary.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-742001510
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDc0MjAwMTUxMA==
created_at: 2020-12-09T19:36:42Z
updated_at: 2020-12-09T19:38:04Z
author_association: NONE
body:
  I don't think this fixes it:

  ```
  grep -R datasette .
  ./sites-available/000-default.conf:        ProxyPass /datasette http://127.0.0.1:8001/
  ./sites-available/000-default.conf:        #ProxyPassReverse /datasette http://127.0.0.1:8001/
  ./sites-available/corpora-le-ssl.conf:  ProxyPass /datasette http://0.0.0.0:8001
  ./sites-available/corpora-le-ssl.conf:  #ProxyPassReverse /datasette http://0.0.0.0:8001
  ./sites-enabled/corpora-le-ssl.conf:  ProxyPass /datasette http://0.0.0.0:8001
  ./sites-enabled/corpora-le-ssl.conf:  #ProxyPassReverse /datasette http://0.0.0.0:8001
  ```

  And I confirmed that I actually restarted the server. :rofl: 

  https://corpora.tika.apache.org/datasette/file_profiles

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1136#issuecomment-742008087
issue_url: https://api.github.com/repos/simonw/datasette/issues/1136
node_id: MDEyOklzc3VlQ29tbWVudDc0MjAwODA4Nw==
created_at: 2020-12-09T19:48:56Z
updated_at: 2020-12-09T19:48:56Z
author_association: OWNER
body:
  I think I'm going to create a branch called `0.52.x` that starts with `8ae0f9f7f0d644b0161165a1084f53acd2786f7c` and then tag the release from there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1136#issuecomment-742009101
issue_url: https://api.github.com/repos/simonw/datasette/issues/1136
node_id: MDEyOklzc3VlQ29tbWVudDc0MjAwOTEwMQ==
created_at: 2020-12-09T19:50:53Z
updated_at: 2020-12-09T19:50:53Z
author_association: OWNER
body:
  My concern is if this will  break anything about CI. I don't think it will - the code that deploys the latest `main` to https://latest.datasette.io/ should be unaffected, and the checkout code in `publish.yml` should check out the correct code based on the tag used for that release.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1136#issuecomment-742009294
issue_url: https://api.github.com/repos/simonw/datasette/issues/1136
node_id: MDEyOklzc3VlQ29tbWVudDc0MjAwOTI5NA==
created_at: 2020-12-09T19:51:18Z
updated_at: 2020-12-09T19:51:18Z
author_association: OWNER
body:
  Likewise, Read The Docs publishes as stable the docs from the latest tagged release, so I would expect that to work fine as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-742010306
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDc0MjAxMDMwNg==
created_at: 2020-12-09T19:53:18Z
updated_at: 2020-12-09T19:59:52Z
author_association: NONE
body:
  I can't imagine this helps (esp. given your point about potential rewrites), but you can see that /datasette/ was correctly added to the sql form, but not to the "export-links"

   <img width="484" alt="Screen Shot 2020-12-09 at 2 51 09 PM" src="https://user-images.githubusercontent.com/6739646/101680055-234baa00-3a2e-11eb-8650-2b369bc6f031.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1136#issuecomment-742014366
issue_url: https://api.github.com/repos/simonw/datasette/issues/1136
node_id: MDEyOklzc3VlQ29tbWVudDc0MjAxNDM2Ng==
created_at: 2020-12-09T20:00:35Z
updated_at: 2020-12-09T20:00:35Z
author_association: OWNER
body:
  Actually I'll start from 0.52.4 and then cherry-pick the fixes.

      git branch 0.52.x 0.52.4
      git checkout 0.52.x
      git cherry-pick COMMIT

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1136#issuecomment-742017622
issue_url: https://api.github.com/repos/simonw/datasette/issues/1136
node_id: MDEyOklzc3VlQ29tbWVudDc0MjAxNzYyMg==
created_at: 2020-12-09T20:06:47Z
updated_at: 2020-12-09T20:06:47Z
author_association: OWNER
body:
  Then I can ship the release directly from that branch, creating the tag as part of the release process:

  <img width="1057" alt="New_release_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/101681442-ffc93500-3a16-11eb-8801-3a0e357a2bc1.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1136#issuecomment-742022222
issue_url: https://api.github.com/repos/simonw/datasette/issues/1136
node_id: MDEyOklzc3VlQ29tbWVudDc0MjAyMjIyMg==
created_at: 2020-12-09T20:15:24Z
updated_at: 2020-12-09T20:15:51Z
author_association: OWNER
body:
  Used this procedure for the first time for 0.52.5 - deploy run here: https://github.com/simonw/datasette/actions/runs/411465648 - PyPI release here: https://pypi.org/project/datasette/0.52.5/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1136#issuecomment-742023111
issue_url: https://api.github.com/repos/simonw/datasette/issues/1136
node_id: MDEyOklzc3VlQ29tbWVudDc0MjAyMzExMQ==
created_at: 2020-12-09T20:17:02Z
updated_at: 2020-12-09T20:17:02Z
author_association: OWNER
body:
  Documentation for this procedure is now here: https://docs.datasette.io/en/latest/contributing.html#releasing-bug-fixes-from-a-branch

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1134#issuecomment-742024588
issue_url: https://api.github.com/repos/simonw/datasette/issues/1134
node_id: MDEyOklzc3VlQ29tbWVudDc0MjAyNDU4OA==
created_at: 2020-12-09T20:19:59Z
updated_at: 2020-12-09T20:20:33Z
author_association: OWNER
body:
  https://byraadsarkivet.aarhus.dk/db/cases?_searchmode=raw&_search=sundhedsfrem%2A is an absolutely beautiful example of a themed Datasette! Very excited to show this to people.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1134#issuecomment-742260116
issue_url: https://api.github.com/repos/simonw/datasette/issues/1134
node_id: MDEyOklzc3VlQ29tbWVudDc0MjI2MDExNg==
created_at: 2020-12-10T05:57:17Z
updated_at: 2020-12-10T05:57:17Z
author_association: NONE
body:
  Hi Simon

  Thank you for the quick fix! And glad you like our use of Datasette (launches 1. january 2021). It's a site that currently (more to come) makes all minutes and their annexes from Aarhus City Council and the major committees (1997-2019) available to the public. So we're putting Datasette to good use :)

reactions:
  {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 2, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/sqlite-utils/issues/205#issuecomment-742737794
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/205
node_id: MDEyOklzc3VlQ29tbWVudDc0MjczNzc5NA==
created_at: 2020-12-10T19:18:22Z
updated_at: 2020-12-10T19:18:22Z
author_association: OWNER
body:
  Yup, it looks like you're using a window function that was added in SQLite 3.25.0: https://www.sqlite.org/changes.html#version_3_25_0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/998#issuecomment-743080047
issue_url: https://api.github.com/repos/simonw/datasette/issues/998
node_id: MDEyOklzc3VlQ29tbWVudDc0MzA4MDA0Nw==
created_at: 2020-12-11T09:25:09Z
updated_at: 2020-12-11T09:25:09Z
author_association: CONTRIBUTOR
body:
  Hello Simon,
  I have a similar problem with horizontal scrollbar display with Datasette version 0.51 and superior for a table with more than 30 rows. With Datasette 0.50, the horizontal scrollbar is displayed, if I upgrade Datasette to 0.51 and superior, the horizontal scrollbar disappears.

  Datasette 0.50: horizontal scrollbar

  ![2020-12-11 10_23_28-CN=Microsoft Windows, O=Microsoft Corporation, L=Redmond, S=Washington, C=US](https://user-images.githubusercontent.com/6371750/101885620-a5f17800-3b9a-11eb-8870-654e7d4372ca.png)

  Datasette 0.51 and superior: no horizontal scrollbar

  ![2020-12-11 10_24_55-CN=Microsoft Windows, O=Microsoft Corporation, L=Redmond, S=Washington, C=US](https://user-images.githubusercontent.com/6371750/101885782-dfc27e80-3b9a-11eb-9d55-6c9a56227bf2.png)

  Thanks,

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/207#issuecomment-743701422
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/207
node_id: MDEyOklzc3VlQ29tbWVudDc0MzcwMTQyMg==
created_at: 2020-12-12T04:37:14Z
updated_at: 2020-12-12T04:38:25Z
author_association: OWNER
body:
  Prototype:
  ```python
  from collections import namedtuple

  ColumnDetails = namedtuple("ColumnDetails", ("column", "num_null", "num_blank", "num_distinct", "most_common", "least_common"))

  def analyze_column(db, table, column, values=10):
      num_null = db.execute("select count(*) from [{}] where [{}] is null".format(table, column)).fetchone()[0]
      num_blank = db.execute("select count(*) from [{}] where [{}] = ''".format(table, column)).fetchone()[0]
      num_distinct = db.execute("select count(distinct [{}]) from [{}]".format(column, table)).fetchone()[0]
      most_common = None
      least_common = None
      if num_distinct != 1:
          most_common = [(r[0], r[1]) for r in db.execute(
              "select [{}], count(*) from [{}] group by [{}] order by count(*) desc limit ".format(column, table, column, values)
          ).fetchall()]
          if num_distinct <= values:
              # No need to run the query if it will just return the results in revers order
              least_common = most_common[::-1]
          else:
              least_common = [(r[0], r[1]) for r in db.execute(
                  "select [{}], count(*) from [{}] group by [{}] order by count(*) limit {}".format(column, table, column, values)
              ).fetchall()]
      return ColumnDetails(column, num_null, num_blank, num_distinct, most_common, least_common)


  def analyze_table(db, table):
      for column in db[table].columns:
          details = analyze_column(db, table, column.name)
          print(details)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/207#issuecomment-743701599
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/207
node_id: MDEyOklzc3VlQ29tbWVudDc0MzcwMTU5OQ==
created_at: 2020-12-12T04:38:52Z
updated_at: 2020-12-12T04:39:07Z
author_association: OWNER
body:
  I'll add a `table.analyze_column(column)` method which is used by the CLI tool - with a note that this is an unstable interface which may change in the future.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/207#issuecomment-743701697
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/207
node_id: MDEyOklzc3VlQ29tbWVudDc0MzcwMTY5Nw==
created_at: 2020-12-12T04:39:51Z
updated_at: 2020-12-12T04:39:51Z
author_association: OWNER
body:
  CLI could be:

      sqlite-utils analyze-tables

  To analyze all tables or:

      sqlite-utils analyze-tables table1 table2

  To analyze specific tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743707969
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/208
node_id: MDEyOklzc3VlQ29tbWVudDc0MzcwNzk2OQ==
created_at: 2020-12-12T05:42:26Z
updated_at: 2020-12-12T05:43:06Z
author_association: OWNER
body:
  Should truncate values in the least/most common JSON array to a sensible length, otherwise you end up with stuff like this:
  ```json
  [
      [
          "b'\\x00\\x05barry\\x03\\x01\\x02\\x00\\x00\\x03cat\\x03\\x01\\x03\\x00\\x00\\x03dog\\x08\\x01\\x01\\x01\\x03\\x00\\x01\\x03\\x00\\x00\\x07panther\\x05\\x01\\x01\\x02\\x02\\x00\\x01\\x03uma\\x05\\x02\\x01\\x02\\x02\\x00\\x00\\x04sara\\x05\\x02\\x01\\x01\\x02\\x00\\x00\\x05terry\\x08\\x01\\x01\\x01\\x02\\x00\\x01\\x02\\x00\\x00\\x06weasel\\x05\\x02\\x01\\x01\\x03\\x00'",
          1
      ]
  ]
  ```
  This example also shows that binary values (like those in `_fts` tables) look a bit weird, but I think I'm OK with that since binary data can't be represented neatly in JSON anyway.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708080
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/208
node_id: MDEyOklzc3VlQ29tbWVudDc0MzcwODA4MA==
created_at: 2020-12-12T05:43:45Z
updated_at: 2020-12-12T05:43:45Z
author_association: OWNER
body:
  CLI output looks like this at the moment, which is bad:
  ```
   % sqlite-utils analyze-tables ../datasette/fixtures.db facetable
  1/10: ColumnDetails(table='facetable', column='pk', total_rows=15, num_null=0, num_blank=0, num_distinct=15, most_common=None, least_common=None)
  2/10: ColumnDetails(table='facetable', column='created', total_rows=15, num_null=0, num_blank=0, num_distinct=4, most_common=[('2019-01-17 08:00:00', 4), ('2019-01-15 08:00:00', 4), ('2019-01-14 08:00:00', 4), ('2019-01-16 08:00:00', 3)], least_common=[('2019-01-16 08:00:00', 3), ('2019-01-14 08:00:00', 4), ('2019-01-15 08:00:00', 4), ('2019-01-17 08:00:00', 4)])
  3/10: ColumnDetails(table='facetable', column='planet_int', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[(1, 14), (2, 1)], least_common=[(2, 1), (1, 14)])
  4/10: ColumnDetails(table='facetable', column='on_earth', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[(1, 14), (0, 1)], least_common=[(0, 1), (1, 14)])
  5/10: ColumnDetails(table='facetable', column='state', total_rows=15, num_null=0, num_blank=0, num_distinct=3, most_common=[('CA', 10), ('MI', 4), ('MC', 1)], least_common=[('MC', 1), ('MI', 4), ('CA', 10)])
  6/10: ColumnDetails(table='facetable', column='city_id', total_rows=15, num_null=0, num_blank=0, num_distinct=4, most_common=[(1, 6), (3, 4), (2, 4), (4, 1)], least_common=[(4, 1), (2, 4), (3, 4), (1, 6)])
  7/10: ColumnDetails(table='facetable', column='neighborhood', total_rows=15, num_null=0, num_blank=0, num_distinct=14, most_common=[('Downtown', 2), ('Tenderloin', 1), ('SOMA', 1), ('Mission', 1), ('Mexicantown', 1), ('Los Feliz', 1), ('Koreatown', 1), ('Hollywood', 1), ('Hayes Valley', 1), ('Greektown', 1)], least_common=[('Arcadia Planitia', 1), ('Bernal Heights', 1), ('Corktown', 1), ('Dogpatch', 1), ('Greektown', 1), ('Hayes Valley', 1), ('Hollywood', 1), ('Koreatown', 1), ('Los Feliz', 1), ('Mexicantown', 1)])
  8/10: ColumnDetails(table='facetable', column='tags', total_rows=15, num_null=0, num_blank=0, num_distinct=3, most_common=[('[]', 13), ('["tag1", "tag3"]', 1), ('["tag1", "tag2"]', 1)], least_common=[('["tag1", "tag2"]', 1), ('["tag1", "tag3"]', 1), ('[]', 13)])
  9/10: ColumnDetails(table='facetable', column='complex_array', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[('[]', 14), ('[{"foo": "bar"}]', 1)], least_common=[('[{"foo": "bar"}]', 1), ('[]', 14)])
  10/10: ColumnDetails(table='facetable', column='distinct_some_null', total_rows=15, num_null=13, num_blank=0, num_distinct=2, most_common=[(None, 13), ('two', 1), ('one', 1)], least_common=[('one', 1), ('two', 1), (None, 13)])
  (sqlite-utils) sqlite-utils % 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708169
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/208
node_id: MDEyOklzc3VlQ29tbWVudDc0MzcwODE2OQ==
created_at: 2020-12-12T05:44:46Z
updated_at: 2020-12-12T05:44:46Z
author_association: OWNER
body:
  If there are less than ten values is it worth outputting them twice, once in `most_common` and then in reverse in `least_common`? Feels redundant - I think I should leave `least_common` empty in that case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708325
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/208
node_id: MDEyOklzc3VlQ29tbWVudDc0MzcwODMyNQ==
created_at: 2020-12-12T05:46:27Z
updated_at: 2020-12-12T05:46:27Z
author_association: OWNER
body:
  It would be neat if you could optionally specify a subset of columns to analyze, using `-c` or `--column`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708524
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/208
node_id: MDEyOklzc3VlQ29tbWVudDc0MzcwODUyNA==
created_at: 2020-12-12T05:48:20Z
updated_at: 2020-12-12T05:48:32Z
author_association: OWNER
body:
  ```
  % sqlite-utils analyze-tables ../datasette/fixtures.db facetable --column pk
  1/1: ColumnDetails(table='facetable', column='pk', total_rows=15, num_null=0, num_blank=0, num_distinct=15, most_common=None, least_common=None)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1142#issuecomment-743913004
issue_url: https://api.github.com/repos/simonw/datasette/issues/1142
node_id: MDEyOklzc3VlQ29tbWVudDc0MzkxMzAwNA==
created_at: 2020-12-12T22:17:46Z
updated_at: 2020-12-12T22:17:46Z
author_association: OWNER
body:
  You're actually choosing between two options here: the 100 rows you can see on the screen, or the x,000 rows that match the current query.

  Maybe a radio box would be more obvious?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743956666
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/208
node_id: MDEyOklzc3VlQ29tbWVudDc0Mzk1NjY2Ng==
created_at: 2020-12-13T05:44:49Z
updated_at: 2020-12-13T05:44:49Z
author_association: OWNER
body:
  Example output:
  ```
  % sqlite-utils analyze-tables github.db tags             
  tags.repo: (1/3)

    Total rows: 261
    Null rows: 0
    Blank rows: 0

    Distinct values: 14

    Most common:
      88: 107914493
      75: 140912432
      27: 206156866
      21: 207052882
      17: 197431109
      8: 197882382
      5: 256834907
      5: 205429375
      4: 248903544
      3: 206202864

    Least common:
      1: 209590345
      2: 206649770
      2: 303218369
      3: 206202864
      3: 213286752
      4: 248903544
      5: 205429375
      5: 256834907
      8: 197882382
      17: 197431109

  tags.name: (2/3)

    Total rows: 261
    Null rows: 0
    Blank rows: 0

    Distinct values: 175

    Most common:
      10: 0.2
      9: 0.1
      7: 0.3
      6: 0.4
      5: 0.7
      5: 0.5
      5: 0.1a
      4: 0.9
      4: 0.8
      4: 0.6

    Least common:
      1: 0.1.1
      1: 0.11.1
      1: 0.1a2
      1: 0.20.1
      1: 0.21.1
      1: 0.21.2
      1: 0.21.3
      1: 0.22
      1: 0.22.1
      1: 0.23

  tags.sha: (3/3)

    Total rows: 261
    Null rows: 0
    Blank rows: 0

    Distinct values: 261
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/207#issuecomment-743966801
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/207
node_id: MDEyOklzc3VlQ29tbWVudDc0Mzk2NjgwMQ==
created_at: 2020-12-13T07:25:23Z
updated_at: 2020-12-13T07:25:23Z
author_association: OWNER
body:
  CLI documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#analyzing-tables

  Python library documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#analyzing-a-column

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1142#issuecomment-743998792
issue_url: https://api.github.com/repos/simonw/datasette/issues/1142
node_id: MDEyOklzc3VlQ29tbWVudDc0Mzk5ODc5Mg==
created_at: 2020-12-13T12:14:06Z
updated_at: 2020-12-13T12:14:06Z
author_association: NONE
body:
  Agreed, it would definitely provide better controls. However, I do feel it makes for a bit of inconsistent UX for the 'Advanced export' section, with links to download for JSON, checkboxes and radio buttons + button to download for CSV. Do you think this example makes the UX a bit nicer/consistent?

  ![Screenshot 2020-12-13 at 5 38 43 PM](https://user-images.githubusercontent.com/6622733/102011444-1dc1cd00-3d6a-11eb-9e38-5af198161e80.png)

  I could give it a try if you'd like but I've never contributed to an actual project!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1031#issuecomment-744003454
issue_url: https://api.github.com/repos/simonw/datasette/issues/1031
node_id: MDEyOklzc3VlQ29tbWVudDc0NDAwMzQ1NA==
created_at: 2020-12-13T12:52:56Z
updated_at: 2020-12-13T12:52:56Z
author_association: NONE
body:
  Please let me know if there's anything I can do to help get this merged.

  This is causing problems for me because it means when I build my Docker image my databases aren't considered immutable, which I would like them to be so that a download link is produced.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-744066249
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc0NDA2NjI0OQ==
created_at: 2020-12-13T20:47:52Z
updated_at: 2020-12-13T20:47:52Z
author_association: OWNER
body:
  @yozlet just spotted this comment. Wow that is interesting!

  With the right plugin hooks on the page (see also #987) one relatively simple way to do that could be with bookmarklets - users could install bookmarklets which, when executed against a Datasette page in their browser, use the existing JavaScript plugin integration points to add all kinds of functionality.

  Doing full sandboxing is certainly daunting, but it looks like Figma figured it out so TIL it's technically feasible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1143#issuecomment-744249157
issue_url: https://api.github.com/repos/simonw/datasette/issues/1143
node_id: MDEyOklzc3VlQ29tbWVudDc0NDI0OTE1Nw==
created_at: 2020-12-14T07:53:15Z
updated_at: 2020-12-14T07:53:15Z
author_association: OWNER
body:
  Does this plugin do everything you need? https://github.com/simonw/datasette-cors

  I'm open to arguments as to why this should be in core rather than in a plugin - I'm on the fence about that at the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1142#issuecomment-744251252
issue_url: https://api.github.com/repos/simonw/datasette/issues/1142
node_id: MDEyOklzc3VlQ29tbWVudDc0NDI1MTI1Mg==
created_at: 2020-12-14T07:56:38Z
updated_at: 2020-12-14T07:56:38Z
author_association: OWNER
body:
  That's a really solid design for this! I'd be very happy to review a pull request - you should be able to implement this with just template edits and some CSS changes I think.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-744461856
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDc0NDQ2MTg1Ng==
created_at: 2020-12-14T14:04:57Z
updated_at: 2020-12-14T14:04:57Z
author_association: NONE
body:
  I'm looking into using datasette with a database with spatialite geometry columns, and came across this issue. Has there been any progress on this since 2018?

  In one of my tables I'm just storing lat/lon points in a spatialite point geometry, and I've managed to make datasette-cluster-map display the points by extracting the lat and lon in SQL - using something like `select ... ST_X(location) as longitude, ST_Y(location) as latitude from Blah`. Something more 'built-in' would be great though - particularly for the tables I have that store more complex geometries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1145#issuecomment-744475543
issue_url: https://api.github.com/repos/simonw/datasette/issues/1145
node_id: MDEyOklzc3VlQ29tbWVudDc0NDQ3NTU0Mw==
created_at: 2020-12-14T14:26:25Z
updated_at: 2020-12-14T14:26:25Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1145?src=pr&el=h1) Report
  > Merging [#1145](https://codecov.io/gh/simonw/datasette/pull/1145?src=pr&el=desc) (a8588f9) into [main](https://codecov.io/gh/simonw/datasette/commit/0c616f732cee79db80cad830917666f41b344262?el=desc) (0c616f7) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1145/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1145?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1145   +/-   ##
  =======================================
    Coverage   91.41%   91.41%           
  =======================================
    Files          31       31           
    Lines        3881     3881           
  =======================================
    Hits         3548     3548           
    Misses        333      333           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1145?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1145?src=pr&el=footer). Last update [0c616f7...a8588f9](https://codecov.io/gh/simonw/datasette/pull/1145?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1144#issuecomment-744489028
issue_url: https://api.github.com/repos/simonw/datasette/issues/1144
node_id: MDEyOklzc3VlQ29tbWVudDc0NDQ4OTAyOA==
created_at: 2020-12-14T14:47:11Z
updated_at: 2020-12-14T14:47:11Z
author_association: NONE
body:
  Thanks for opening the issue, @simonw. Let me elaborate on my Tweets.

  [datasette-chartjs](https://github.com/MarkusH/datasette-chartjs) provides drop down lists to pick the chart visualization (e.g. bar, line, doughnut, pie, ...) as well as the column used for the "x axis" (e.g. time).

  A user can change the values on-demand. The chart will be redrawn w/o querying the database again.

  However, if a user wants to change the underlying query, they will use the SQL field provided by datasette or any of the other datasette built-in features to amend a query. In order to maintain a user's selections for the plugin, datasette-chartjs copies some parts of [datasette-vega](https://github.com/simonw/datasette-vega) which persist the chosen visualization and column in the hash part of a URL (the stuff behind the `#`). The plugin load the config from the hash upon initialization on the next page and use it accordingly.

  Additionally, datasette-vega and datasette-chartjs need to make sure to include the hash in all links and forms that cause a reload of the page. This is, such that the config persists between clicks.

  This ticket is about moving thes parts into datasette that provide the functionality to do so. This includes:

  1. a way to load config options with a given prefix from the current URL hash
  1. a way to update the current URL hash with a new config value or a bunch of config options
  1. updating all necessary links and forms on the current page to include the URL hash whenever its updated
  1. to prevent leaking config options to external pages, only "internal" links should be updated

  There's another, optional, feature that we might want to think about during the design phase: the scope of the config. Links within a datasette instance have 1 of 3 scopes:

  1. global, for the whole datasette project
  1. database, for all tables in a database
  1. table, only for a table within a database

  When updating the links and forms as pointed out in 3. above, it might be worth considering which links need to be updated. I could imagine a plugin that wants to persist some setting across all tables within a database but another setting only within a table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1142#issuecomment-744522099
issue_url: https://api.github.com/repos/simonw/datasette/issues/1142
node_id: MDEyOklzc3VlQ29tbWVudDc0NDUyMjA5OQ==
created_at: 2020-12-14T15:37:47Z
updated_at: 2020-12-14T15:37:47Z
author_association: NONE
body:
  Alright I could give it a try! This might be a stupid question, can you tell me how to run the server from my fork? So that I can test the changes?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1142#issuecomment-744563209
issue_url: https://api.github.com/repos/simonw/datasette/issues/1142
node_id: MDEyOklzc3VlQ29tbWVudDc0NDU2MzIwOQ==
created_at: 2020-12-14T16:41:11Z
updated_at: 2020-12-14T16:41:11Z
author_association: OWNER
body:
  To check out and start the server:

      /tmp % git clone git@github.com:nitinpaul/datasette
      Cloning into 'datasette'...
      remote: Enumerating objects: 124, done.
      # ...
      datasette % python3 -m venv venv
      datasette % source venv/bin/activate
      (venv) datasette % pip install -e '.[test]'
      Obtaining file:///private/tmp/datasette
      Collecting asgiref<3.4.0,>=3.2.10
      Using cached asgiref-3.3.1-py3-none-any.whl (19 kB)
      # ...
      (venv) datasette % datasette
      INFO:     Started server process [24002]
      INFO:     Waiting for application startup.
      INFO:     Application startup complete.
      INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)

  And to run the tests:

      (venv) datasette % pytest
      ======================================================================== test session starts ========================================================================
      platform darwin -- Python 3.9.1, pytest-6.1.2, py-1.10.0, pluggy-0.13.1
      SQLite: 3.34.0
      rootdir: /private/tmp/datasette, configfile: pytest.ini
      plugins: asyncio-0.14.0, timeout-1.4.2
      collected 841 items                                                                                                                                                 

      tests/test_package.py ..                                                                                                                                      [  0%]

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1142#issuecomment-744576894
issue_url: https://api.github.com/repos/simonw/datasette/issues/1142
node_id: MDEyOklzc3VlQ29tbWVudDc0NDU3Njg5NA==
created_at: 2020-12-14T17:03:13Z
updated_at: 2020-12-14T17:03:13Z
author_association: OWNER
body:
  I'm not sure about the radio boxes for JSON, just because you can't right-click on a radio box and copy it to your clipboard like you can with links. Worth trying it out though.

  The radio boxes for that CSV option are definitely the right way to go.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1143#issuecomment-744618787
issue_url: https://api.github.com/repos/simonw/datasette/issues/1143
node_id: MDEyOklzc3VlQ29tbWVudDc0NDYxODc4Nw==
created_at: 2020-12-14T18:15:00Z
updated_at: 2020-12-15T02:21:53Z
author_association: NONE
body:
  From a quick look at the README, it does seem to do everything I need, thanks!

  I think the argument for inclusion in core is to lower the chances of unwanted data access. A local server can be accessed by anybody who can make an HTTP request to your computer regardless of CORS rules, but the default `*` rule additionally opens up access to the local instance to any website you visit while it is running. 

  That's probably not what people typically intend, particularly when the data is of a sensitive nature. A default of requiring the user to specify the origin (allowing `*` but encouraging a narrower scope) would solve this problem entirely, I think.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1143#issuecomment-744756861
issue_url: https://api.github.com/repos/simonw/datasette/issues/1143
node_id: MDEyOklzc3VlQ29tbWVudDc0NDc1Njg2MQ==
created_at: 2020-12-14T22:40:28Z
updated_at: 2020-12-14T22:40:28Z
author_association: OWNER
body:
  That's a very convincing argument. I'm keen on making sure Datasette is "secure by default" so you're right, encouraging finely grains CORS rules in core rather than leaving that to a plugin sounds like the right call.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1143#issuecomment-744757558
issue_url: https://api.github.com/repos/simonw/datasette/issues/1143
node_id: MDEyOklzc3VlQ29tbWVudDc0NDc1NzU1OA==
created_at: 2020-12-14T22:42:10Z
updated_at: 2020-12-14T22:42:10Z
author_association: OWNER
body:
  This may involve a breaking change to the CLI settings interface, so I'm adding this to the 1.0 milestone.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1142#issuecomment-745162571
issue_url: https://api.github.com/repos/simonw/datasette/issues/1142
node_id: MDEyOklzc3VlQ29tbWVudDc0NTE2MjU3MQ==
created_at: 2020-12-15T09:22:58Z
updated_at: 2020-12-15T09:22:58Z
author_association: NONE
body:
  You're right, probably more straightforward to have the links for JSON. I was imagining to toggle the `href` for the 'Export JSON' link (button) to the selected shape, but it'll probably be needlessly complex in the end.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746734412
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58
node_id: MDEyOklzc3VlQ29tbWVudDc0NjczNDQxMg==
created_at: 2020-12-16T17:58:56Z
updated_at: 2020-12-16T17:58:56Z
author_association: MEMBER
body:
  I'm going to rewrite those `<a href="#filtering-tables">` links to `<a href="#user-content-filtering-tables">` - but only if a corresponding `id="user-content-filtering-tables"` element exists.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746735889
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58
node_id: MDEyOklzc3VlQ29tbWVudDc0NjczNTg4OQ==
created_at: 2020-12-16T17:59:50Z
updated_at: 2020-12-16T17:59:50Z
author_association: MEMBER
body:
  I don't want to add a full HTML parser (like BeautifulSoup) as a dependency for this feature. Since the HTML comes from a single, trusted source (GitHub) I could probably handle this using [regular expressions](https://stackoverflow.com/a/1732454).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1143#issuecomment-746827083
issue_url: https://api.github.com/repos/simonw/datasette/issues/1143
node_id: MDEyOklzc3VlQ29tbWVudDc0NjgyNzA4Mw==
created_at: 2020-12-16T18:56:07Z
updated_at: 2020-12-16T18:56:07Z
author_association: OWNER
body:
  I think the right way to do this is to support multiple optional `--cors-origin=` pattern values, like you suggested.

reactions:
  {"total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747029636
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29
node_id: MDEyOklzc3VlQ29tbWVudDc0NzAyOTYzNg==
created_at: 2020-12-16T21:14:03Z
updated_at: 2020-12-16T21:14:03Z
author_association: MEMBER
body:
  I think I can do this as a cunning trick in `display_sql`. Consider this example query: https://til.simonwillison.net/tils?sql=select%0D%0A++path%2C%0D%0A++snippet%28til_fts%2C+-1%2C+%27b4de2a49c8%27%2C+%278c94a2ed4b%27%2C+%27...%27%2C+60%29+as+snippet%0D%0Afrom%0D%0A++til%0D%0A++join+til_fts+on+til.rowid+%3D+til_fts.rowid%0D%0Awhere%0D%0A++til_fts+match+escape_fts%28%3Aq%29%0D%0A++and+path+%3D+%27asgi_lifespan-test-httpx.md%27%0D%0A&q=pytest

  ```sql
  select
    path,
    snippet(til_fts, -1, 'b4de2a49c8', '8c94a2ed4b', '...', 60) as snippet
  from
    til
    join til_fts on til.rowid = til_fts.rowid
  where
    til_fts match escape_fts(:q)
    and path = 'asgi_lifespan-test-httpx.md'
  ```
  The `and path = 'asgi_lifespan-test-httpx.md'` bit means we only get back a specific document - but the snippet highlighting is applied to it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747030964
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29
node_id: MDEyOklzc3VlQ29tbWVudDc0NzAzMDk2NA==
created_at: 2020-12-16T21:14:54Z
updated_at: 2020-12-16T21:14:54Z
author_association: MEMBER
body:
  To do this I'll need the search term to be passed to the `display_sql` SQL query:
  https://github.com/dogsheep/dogsheep-beta/blob/4890ec87b5e2ec48940f32c9ad1f5aae25c75a4d/dogsheep_beta/__init__.py#L164-L171

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/675#issuecomment-747059277
issue_url: https://api.github.com/repos/simonw/datasette/issues/675
node_id: MDEyOklzc3VlQ29tbWVudDc0NzA1OTI3Nw==
created_at: 2020-12-16T21:43:52Z
updated_at: 2020-12-16T21:43:52Z
author_association: OWNER
body:
  It turns out I need this for a couple of projects:

  - [datasette-ripgrep](https://github.com/simonw/datasette-ripgrep) needs to ship a whole bunch of source code files up in a known location. I worked around this with a nasty hack involving `--static` but it would be better if I wasn't doing that.
  - [dogsheep-beta](https://github.com/dogsheep/dogsheep-beta) uses an additional `dogsheep-beta.yml` configuration file in the project root (a sibling to `metadata.yml`) which needs to be included when publishing - see https://github.com/simonw/datasette.io/issues/21#issuecomment-747058067

  I want this for `datasette publish cloudrun`, not just for `datasette package`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1148#issuecomment-747062909
issue_url: https://api.github.com/repos/simonw/datasette/issues/1148
node_id: MDEyOklzc3VlQ29tbWVudDc0NzA2MjkwOQ==
created_at: 2020-12-16T21:51:54Z
updated_at: 2020-12-16T21:51:54Z
author_association: OWNER
body:
  This is a really frustrating bug with Vercel: https://github.com/simonw/datasette-publish-vercel/issues/28

  `+` characters in URLs get translated into spaces before they get to Datasette. They know about the bug and said they were working on a fix a few months ago, but looks like it's still a problem.

  A workaround is to avoid `+` and use `-` instead - I think this SQL query does the same thing as yours:

  https://aws-partners-singapore.vercel.app/partners?sql=select%0D%0A++A.launch_rank%2C%0D%0A++A.partner_info%0D%0Afrom%0D%0A++summary+A%0D%0A++INNER+JOIN+summary+B+ON+A.launch_rank+%3E%3D+B.launch_rank+-+3%0D%0A++AND+A.launch_rank+-4+%3C%3D+B.launch_rank%0D%0AWHERE%0D%0A++B.%22partner_info%22+LIKE+%27%25Palo+Alto%25%27

  ```sql
  select
    A.launch_rank,
    A.partner_info
  from
    summary A
    INNER JOIN summary B ON A.launch_rank >= B.launch_rank - 3
    AND A.launch_rank -4 <= B.launch_rank
  WHERE
    B."partner_info" LIKE '%Palo Alto%'
  ```
  I've been moving projects from Vercel to Cloud Run when they run into this, but that's not a great situation to be in.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/675#issuecomment-747066629
issue_url: https://api.github.com/repos/simonw/datasette/issues/675
node_id: MDEyOklzc3VlQ29tbWVudDc0NzA2NjYyOQ==
created_at: 2020-12-16T21:59:58Z
updated_at: 2020-12-16T22:00:48Z
author_association: OWNER
body:
  Note that `datasette publish cloudrun` uses a working directory of `/app` - so users will need to copy their files into `/app` if that's where they need to live.

  https://github.com/simonw/datasette/blob/17cbbb1f7f230b39650afac62dd16476626001b5/datasette/utils/__init__.py#L348-L357

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/675#issuecomment-747067864
issue_url: https://api.github.com/repos/simonw/datasette/issues/675
node_id: MDEyOklzc3VlQ29tbWVudDc0NzA2Nzg2NA==
created_at: 2020-12-16T22:02:55Z
updated_at: 2020-12-16T22:02:55Z
author_association: OWNER
body:
  But since we're already running `COPY . /app` anything that's made it into the temporary directory will get copied into `/app`.

  But... I feel the usability of the command will be better if users can use absolute paths on the `target` side:

      datasette publish cloudrun my.db --cp dogsheep-beta.yml /app

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/675#issuecomment-747068624
issue_url: https://api.github.com/repos/simonw/datasette/issues/675
node_id: MDEyOklzc3VlQ29tbWVudDc0NzA2ODYyNA==
created_at: 2020-12-16T22:04:42Z
updated_at: 2020-12-16T22:04:42Z
author_association: OWNER
body:
  I can't just use `COPY /path/to/blah.yml /app` in the `Dockerfile` because it runs on the Google Cloud Build servers, not on the user's laptop - so I need to first copy the files they specify to that temporary directory that gets uploaded to the cloud, then rewrite the `COPY` lines in the `Dockerfile` to copy from there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/675#issuecomment-747070709
issue_url: https://api.github.com/repos/simonw/datasette/issues/675
node_id: MDEyOklzc3VlQ29tbWVudDc0NzA3MDcwOQ==
created_at: 2020-12-16T22:09:15Z
updated_at: 2020-12-16T22:09:15Z
author_association: OWNER
body:
  The other way this could work is passing a single argument - the file (or directory) to be copied in - and assuming it should always go in the `/app` root. Something like:

      datasette publish cloudrun my.db --include src/ --include dogsheep-beta.yml

  Which would add `/app/src/...` and `/app/dogsheep-beta.yml`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/issues/2#issuecomment-747126777
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2
node_id: MDEyOklzc3VlQ29tbWVudDc0NzEyNjc3Nw==
created_at: 2020-12-17T00:36:52Z
updated_at: 2020-12-17T00:36:52Z
author_association: MEMBER
body:
  The memory profiler tricks I used in https://github.com/dogsheep/healthkit-to-sqlite/issues/7 could help figure out what's going on here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/issues/2#issuecomment-747130908
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2
node_id: MDEyOklzc3VlQ29tbWVudDc0NzEzMDkwOA==
created_at: 2020-12-17T00:47:04Z
updated_at: 2020-12-17T00:47:43Z
author_association: NONE
body:
  it looks like almost all of the memory consumption is coming from `json.load()`.

  another direction here may be to use the new "Semantic Location History" data which is already broken down by year and month.

  it also provides much more interesting data, such as estimated address, form of travel, etc.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1149#issuecomment-747207487
issue_url: https://api.github.com/repos/simonw/datasette/issues/1149
node_id: MDEyOklzc3VlQ29tbWVudDc0NzIwNzQ4Nw==
created_at: 2020-12-17T05:05:08Z
updated_at: 2020-12-17T05:05:08Z
author_association: OWNER
body:
  I think what I want is for it to be easy to reuse portions of Datasette's CSS - the bit that styles the cog menu for example - without pulling in the whole thing.

  I tried linking in the `<link rel="stylesheet" href="/-/static/app.css">` stylesheet and the page broke, wildly:

  <img width="1030" alt="content__categories__3_rows_and_Datasette_search__pytest" src="https://user-images.githubusercontent.com/9599/102446222-2290b600-3fe2-11eb-9d3a-e32ca72d1dac.png">

  That's because Datasette's [built-in CSS](https://github.com/simonw/datasette/blob/0.53/datasette/static/app.css) applies styles directly to a whole bunch of different tags - `body`, `header`, `footer` etc - which means that if you import that stylesheet it can play havoc with the site you have already built.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1149#issuecomment-747207787
issue_url: https://api.github.com/repos/simonw/datasette/issues/1149
node_id: MDEyOklzc3VlQ29tbWVudDc0NzIwNzc4Nw==
created_at: 2020-12-17T05:06:16Z
updated_at: 2020-12-17T05:06:16Z
author_association: OWNER
body:
  So, an idea: what if Datasette's default CSS applied only to elements with classes - or maybe to childen of a `body class="datasette"` element? In such a way that you could write your own custom HTML that reused elements of Datasette's CSS - the cog menu styling for example - but only on an opt-in basis?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/741#issuecomment-747208543
issue_url: https://api.github.com/repos/simonw/datasette/issues/741
node_id: MDEyOklzc3VlQ29tbWVudDc0NzIwODU0Mw==
created_at: 2020-12-17T05:09:03Z
updated_at: 2020-12-17T05:09:03Z
author_association: OWNER
body:
  I really like this in `datasette-publish-vercel` - I'm definitely going to bring this to the other publish implementations as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/461#issuecomment-747734273
issue_url: https://api.github.com/repos/simonw/datasette/issues/461
node_id: MDEyOklzc3VlQ29tbWVudDc0NzczNDI3Mw==
created_at: 2020-12-17T22:14:46Z
updated_at: 2020-12-17T22:14:46Z
author_association: OWNER
body:
  I've been thinking about this a bunch. For Datasette to be useful as a private repository of data (Datasette Library, #417) it's crucial that it can handle a much, much larger number of databases.

  This makes me worry about how many connections (and open file handles) it makes sense to have open at one time.

  I realize now that this is much less of a problem for private instances. Public instances on the internet could get traffic to any database at any time, so connections could easily get out of control. A private instance with only a few users could instead get away with only opening connections to databases in "active use".

  This does however make it even more important for Datasette to maintain a cached set of metadata about the tables - which is also needed to power this feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747754082
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc1NDA4Mg==
created_at: 2020-12-17T23:04:13Z
updated_at: 2020-12-17T23:04:13Z
author_association: OWNER
body:
  Pages that need a list of all databases - the index page and /-/databases for example - could trigger a "check for new directories in the configured directories" scan.

  That scan would run at most once every 5 (n) seconds - the check is triggered if it’s run more recently than that it doesn’t run.

  Hopefully this means it could be done as a blocking operation, rather than trying to run it in a thread.

  When it runs it scans for *.db or *.sqlite files (maybe one or two other extensions) that it hasn’t seen before. It also checks that the existing list of known database files still exists.

  If it finds any new ones it connects to them once to run `.schema`. It also runs `PRAGMA schema_version` on each known database so that it can compare the schema version number to the last one it saw. That's how it detects if there are new tables or if the cached schema needs to be updated.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747754229
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc1NDIyOQ==
created_at: 2020-12-17T23:04:38Z
updated_at: 2020-12-17T23:04:38Z
author_association: OWNER
body:
  Open question: will this work for hundreds of database files, or is the overhead of connecting to each of 100 databases in turn to run `PRAGMA schema_version` too high?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747764712
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc2NDcxMg==
created_at: 2020-12-17T23:16:31Z
updated_at: 2020-12-17T23:16:31Z
author_association: OWNER
body:
  Quick micro-benchmark, run against a folder with 46 database files adding up to 1.4GB total:
  ```python
  import pathlib, sqlite3, time

  paths = list(pathlib.Path(".").glob('*.db'))

  def schema_version(path):
      db = sqlite3.connect(path)
      version = db.execute("PRAGMA schema_version").fetchall()[0]
      db.close()
      return version

  def all():
      versions = {}
      for path in paths:
          versions[path.name] = schema_version(path)
      return versions

  start = time.time(); all(); print(time.time() - start)
  # 0.012346982955932617
  ```
  So that's 12ms.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747766310
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc2NjMxMA==
created_at: 2020-12-17T23:20:49Z
updated_at: 2020-12-17T23:20:49Z
author_association: OWNER
body:
  I tried against my entire `~/Development/Dropbox` folder - deeply nested with 381 SQLite database files in sub-folders - and it took 25s! But it turned out 23.9s of that was the call to `pathlib.Path("/Users/simon/Dropbox/Development").glob('**/*.db')`.

  So it looks like connecting to a SQLite database file and getting the schema version is extremely fast. Scanning directories is slower.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747767055
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc2NzA1NQ==
created_at: 2020-12-17T23:22:41Z
updated_at: 2020-12-17T23:22:41Z
author_association: OWNER
body:
  It's just recursion that's expensive. I created 380 empty SQLite databases in a folder and timed `list(pathlib.Path("/tmp").glob("*.db"));` and it took 0.002s.

  So maybe I tell users that all SQLite databases have to be in the root folder.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747767598
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc2NzU5OA==
created_at: 2020-12-17T23:24:03Z
updated_at: 2020-12-17T23:24:03Z
author_association: OWNER
body:
  I'm going to assume that even the heaviest user will have trouble going beyond a few hundred database files, so this is fine.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747768112
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc2ODExMg==
created_at: 2020-12-17T23:25:21Z
updated_at: 2020-12-17T23:25:21Z
author_association: OWNER
body:
  Next challenge: figure out how to use the `Database` class from https://github.com/simonw/datasette/blob/0.53/datasette/database.py for an in-memory database which persists data for the duration of the lifetime of the server, and allows access to that in-memory database from multiple threads in a way that lets them see each other's changes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1151#issuecomment-747769830
issue_url: https://api.github.com/repos/simonw/datasette/issues/1151
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc2OTgzMA==
created_at: 2020-12-17T23:29:08Z
updated_at: 2020-12-17T23:29:08Z
author_association: OWNER
body:
  https://sqlite.org/inmemorydb.html

  >  The database ceases to exist as soon as the database connection is closed. Every :memory: database is distinct from every other. So, opening two database connections each with the filename ":memory:" will create two independent in-memory databases.
  >
  > [...]
  >
  > The special `":memory:"` filename also works when using URI filenames. For example:
  >
  >      rc = sqlite3_open("file::memory:", &db);
  >
  > [...]
  >
  >  However, the same in-memory database can be opened by two or more database connections as follows:
  >
  >      rc = sqlite3_open("file::memory:?cache=shared", &db);
  >
  > [...]
  > If two or more distinct but shareable in-memory databases are needed in a single process, then the mode=memory query parameter can be used with a URI filename to create a named in-memory database:
  >
  >     rc = sqlite3_open("file:memdb1?mode=memory&cache=shared", &db);

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1151#issuecomment-747770581
issue_url: https://api.github.com/repos/simonw/datasette/issues/1151
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc3MDU4MQ==
created_at: 2020-12-17T23:31:18Z
updated_at: 2020-12-17T23:32:07Z
author_association: OWNER
body:
  This works in `ipython`:
  ```
  In [1]: import sqlite3

  In [2]: c1 = sqlite3.connect("file:datasette?mode=memory&cache=shared", uri=True)

  In [3]: c2 = sqlite3.connect("file:datasette?mode=memory&cache=shared", uri=True)

  In [4]: c1.executescript("CREATE TABLE hello (world TEXT)")
  Out[4]: <sqlite3.Cursor at 0x1104addc0>

  In [5]: c1.execute("select * from sqlite_master").fetchall()
  Out[5]: [('table', 'hello', 'hello', 2, 'CREATE TABLE hello (world TEXT)')]

  In [6]: c2.execute("select * from sqlite_master").fetchall()
  Out[6]: [('table', 'hello', 'hello', 2, 'CREATE TABLE hello (world TEXT)')]

  In [7]: c3 = sqlite3.connect("file:datasette?mode=memory&cache=shared", uri=True)

  In [9]: c3.execute("select * from sqlite_master").fetchall()
  Out[9]: [('table', 'hello', 'hello', 2, 'CREATE TABLE hello (world TEXT)')]

  In [10]: c4 = sqlite3.connect("file:datasette?mode=memory", uri=True)

  In [11]: c4.execute("select * from sqlite_master").fetchall()
  Out[11]: []
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1151#issuecomment-747774855
issue_url: https://api.github.com/repos/simonw/datasette/issues/1151
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc3NDg1NQ==
created_at: 2020-12-17T23:42:34Z
updated_at: 2020-12-17T23:42:34Z
author_association: OWNER
body:
  This worked as a prototype:
  ```diff
  diff --git a/datasette/database.py b/datasette/database.py
  index 412e0c5..a90e617 100644
  --- a/datasette/database.py
  +++ b/datasette/database.py
  @@ -24,11 +24,12 @@ connections = threading.local()
   
   
   class Database:
  -    def __init__(self, ds, path=None, is_mutable=False, is_memory=False):
  +    def __init__(self, ds, path=None, is_mutable=False, is_memory=False, uri=None):
           self.ds = ds
           self.path = path
           self.is_mutable = is_mutable
           self.is_memory = is_memory
  +        self.uri = uri
           self.hash = None
           self.cached_size = None
           self.cached_table_counts = None
  @@ -46,6 +47,8 @@ class Database:
                   }
   
       def connect(self, write=False):
  +        if self.uri:
  +            return sqlite3.connect(self.uri, uri=True, check_same_thread=False)
           if self.is_memory:
               return sqlite3.connect(":memory:")
           # mode=ro or immutable=1?
  ```
  Then in `ipython`:
  ```
  from datasette.app import Datasette
  from datasette.database import Database
  ds = Datasette([])
  db = Database(ds, uri="file:datasette?mode=memory&cache=shared", is_memory=True)
  await db.execute_write("create table foo (bar text)")
  await db.table_names()
  # Outputs ["foo"]
  db2 = Database(ds, uri="file:datasette?mode=memory&cache=shared", is_memory=True)
  await db2.table_names()
  # Also outputs ["foo"]
  ```

  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1151#issuecomment-747775245
issue_url: https://api.github.com/repos/simonw/datasette/issues/1151
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc3NTI0NQ==
created_at: 2020-12-17T23:43:41Z
updated_at: 2020-12-17T23:56:27Z
author_association: OWNER
body:
  I'm going to add an argument to the `Database()` constructor which means "connect to named in-memory database called X".

  ```python
  db = Database(ds, memory_name="datasette")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1151#issuecomment-747775792
issue_url: https://api.github.com/repos/simonw/datasette/issues/1151
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc3NTc5Mg==
created_at: 2020-12-17T23:45:20Z
updated_at: 2020-12-17T23:45:20Z
author_association: OWNER
body:
  Do I use the current `is_memory=` boolean anywhere at the moment?

  https://ripgrep.datasette.io/-/ripgrep?pattern=is_memory - doesn't look like it.

  I may remove that feature, since it's not actually useful, and replace it with a mechanism for creating shared named memory databases instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1151#issuecomment-747779056
issue_url: https://api.github.com/repos/simonw/datasette/issues/1151
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc3OTA1Ng==
created_at: 2020-12-17T23:55:57Z
updated_at: 2020-12-17T23:55:57Z
author_association: OWNER
body:
  Wait I do use it - if you run `datasette --memory` - which is useful for trying things out in SQL that doesn't need to run against a table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1151#issuecomment-747784199
issue_url: https://api.github.com/repos/simonw/datasette/issues/1151
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzc4NDE5OQ==
created_at: 2020-12-18T00:09:36Z
updated_at: 2020-12-18T00:09:36Z
author_association: OWNER
body:
  Is it possible to connect to a memory database in read-only mode?

  `file:foo?mode=memory&cache=shared&mode=ro` isn't valid because it features `mode=` more than once.

  https://stackoverflow.com/a/40548682 suggests using `PRAGMA query_only` on the connection instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1151#issuecomment-747801084
issue_url: https://api.github.com/repos/simonw/datasette/issues/1151
node_id: MDEyOklzc3VlQ29tbWVudDc0NzgwMTA4NA==
created_at: 2020-12-18T01:01:26Z
updated_at: 2020-12-18T01:01:26Z
author_association: OWNER
body:
  I tested this with a one-off plugin and it worked!

  ```python
  from datasette import hookimpl
  from datasette.database import Database


  @hookimpl
  def startup(datasette):
      datasette.add_database("statistics", Database(
          datasette,
          memory_name="statistics"
      ))
  ```
  This created a `/statistics` database when I ran `datasette` - and if I installed https://github.com/simonw/datasette-write I could then create tables in it which persisted until I restarted the server.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1151#issuecomment-747801751
issue_url: https://api.github.com/repos/simonw/datasette/issues/1151
node_id: MDEyOklzc3VlQ29tbWVudDc0NzgwMTc1MQ==
created_at: 2020-12-18T01:03:39Z
updated_at: 2020-12-18T01:03:39Z
author_association: OWNER
body:
  This feature is illustrated by the tests: https://github.com/simonw/datasette/blob/5e9895c67f08e9f42acedd3d6d29512ac446e15f/tests/test_internals_database.py#L469-L496

  I added new documentation for the `Datasette()` constructor here as well: https://docs.datasette.io/en/latest/internals.html#database-ds-path-none-is-mutable-false-is-memory-false-memory-name-none

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747803268
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0NzgwMzI2OA==
created_at: 2020-12-18T01:08:40Z
updated_at: 2020-12-18T01:08:40Z
author_association: OWNER
body:
  Next step: design a schema for the in-memory database table that exposes all of the tables.

  I want to support things like:

  - Show me all of the tables
  - Show me the columns in a table
  - Show me all tables that contain a `tags` column
  - Show me the indexes
  - Show me every table configured for full-text search

  Maybe a starting point would be to build concrete tables using the results of things like `PRAGMA foreign_key_list(table)` and `PRAGMA table_xinfo(table)` - note though that `table_xinfo` is SQLite 3.26.0 or higher, as shown here: https://github.com/simonw/datasette/blob/5e9895c67f08e9f42acedd3d6d29512ac446e15f/datasette/utils/__init__.py#L563-L579

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747804254
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0NzgwNDI1NA==
created_at: 2020-12-18T01:12:13Z
updated_at: 2020-12-18T01:12:13Z
author_association: OWNER
body:
  Prototype: https://latest.datasette.io/fixtures?sql=select+%27facetable%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27facetable%27%29%0D%0Aunion%0D%0Aselect+%27searchable%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27searchable%27%29%0D%0Aunion%0D%0Aselect+%27compound_three_primary_keys%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27compound_three_primary_keys%27%29

  ```sql
  select 'facetable' as 'table', * from pragma_table_xinfo('facetable')
  union
  select 'searchable' as 'table', * from pragma_table_xinfo('searchable')
  union
  select 'compound_three_primary_keys' as 'table', * from pragma_table_xinfo('compound_three_primary_keys')
  ```


  table | cid | name | type | notnull | dflt_value | pk | hidden
  -- | -- | -- | -- | -- | -- | -- | --
  compound_three_primary_keys | 0 | pk1 | varchar(30) | 0 |   | 1 | 0
  compound_three_primary_keys | 1 | pk2 | varchar(30) | 0 |   | 2 | 0
  compound_three_primary_keys | 2 | pk3 | varchar(30) | 0 |   | 3 | 0
  compound_three_primary_keys | 3 | content | text | 0 |   | 0 | 0
  facetable | 0 | pk | integer | 0 |   | 1 | 0
  facetable | 1 | created | text | 0 |   | 0 | 0
  facetable | 2 | planet_int | integer | 0 |   | 0 | 0
  facetable | 3 | on_earth | integer | 0 |   | 0 | 0
  facetable | 4 | state | text | 0 |   | 0 | 0
  facetable | 5 | city_id | integer | 0 |   | 0 | 0
  facetable | 6 | neighborhood | text | 0 |   | 0 | 0
  facetable | 7 | tags | text | 0 |   | 0 | 0
  facetable | 8 | complex_array | text | 0 |   | 0 | 0
  facetable | 9 | distinct_some_null |   | 0 |   | 0 | 0
  searchable | 0 | pk | integer | 0 |   | 1 | 0
  searchable | 1 | text1 | text | 0 |   | 0 | 0
  searchable | 2 | text2 | text | 0 |   | 0 | 0
  searchable | 3 | name with . and spaces | text | 0 |   | 0 | 0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747805275
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0NzgwNTI3NQ==
created_at: 2020-12-18T01:15:27Z
updated_at: 2020-12-18T01:16:17Z
author_association: OWNER
body:
  This query uses a join to pull foreign key information for every table: https://latest.datasette.io/fixtures?sql=with+tables+as+%28%0D%0A++select%0D%0A++++name%0D%0A++from%0D%0A++++sqlite_master%0D%0A++where%0D%0A++++type+%3D+%27table%27%0D%0A%29%0D%0Aselect%0D%0A++tables.name+as+%27table%27%2C%0D%0A++foo.*%0D%0Afrom%0D%0A++tables%0D%0A++join+pragma_foreign_key_list%28tables.name%29+foo

  ```sql
  with tables as (
    select
      name
    from
      sqlite_master
    where
      type = 'table'
  )
  select
    tables.name as 'table',
    foo.*
  from
    tables
    join pragma_foreign_key_list(tables.name) foo
  ```

  Same query for `pragma_table_xinfo`: https://latest.datasette.io/fixtures?sql=with+tables+as+%28%0D%0A++select%0D%0A++++name%0D%0A++from%0D%0A++++sqlite_master%0D%0A++where%0D%0A++++type+%3D+%27table%27%0D%0A%29%0D%0Aselect%0D%0A++tables.name+as+%27table%27%2C%0D%0A++foo.*%0D%0Afrom%0D%0A++tables%0D%0A++join+pragma_table_xinfo%28tables.name%29+foo

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747807289
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0NzgwNzI4OQ==
created_at: 2020-12-18T01:22:05Z
updated_at: 2020-12-18T01:22:05Z
author_association: OWNER
body:
  Here's a simpler query pattern (not using CTEs so should work on older versions of SQLite) - this one lists all indexes for all tables:
  ```sql
  select
    sqlite_master.name as 'table',
    indexes.*
  from
    sqlite_master
    join pragma_index_list(sqlite_master.name) indexes
  where
    sqlite_master.type = 'table'
  ```
  https://latest.datasette.io/fixtures?sql=select%0D%0A++sqlite_master.name+as+%27table%27%2C%0D%0A++indexes.*%0D%0Afrom%0D%0A++sqlite_master%0D%0A++join+pragma_index_list%28sqlite_master.name%29+indexes%0D%0Awhere%0D%0A++sqlite_master.type+%3D+%27table%27

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747807891
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0NzgwNzg5MQ==
created_at: 2020-12-18T01:23:59Z
updated_at: 2020-12-18T01:23:59Z
author_association: OWNER
body:
  https://www.sqlite.org/pragma.html#pragfunc says:
  >  * This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported. 
  >  * The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747809670
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0NzgwOTY3MA==
created_at: 2020-12-18T01:29:30Z
updated_at: 2020-12-18T01:29:30Z
author_association: OWNER
body:
  I've been rediscovering the pattern I already documented in this TIL: https://github.com/simonw/til/blob/main/sqlite/list-all-columns-in-a-database.md#better-alternative-using-a-join

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747833639
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0NzgzMzYzOQ==
created_at: 2020-12-18T02:49:40Z
updated_at: 2020-12-18T03:52:12Z
author_association: OWNER
body:
  I'm going to use five tables to start off with:

  - `databases` - a list of databases. Each one has a `name`, `path` (if it's on disk), `is_memory`, `schema_version`
  - `tables` - a list of tables. Each row is `database_name`, `table_name`, `sql` (the create table statement) - may add more tables in the future, in particular maybe a `last_row_count` to cache results of counting the rows.
  - `columns` - a list of columns. It's the output of `pragma_table_xinfo` with the `database_name` and `table_name` columns added at the beginning.
  - `foreign_keys` - a list of foreign keys - `pragma_foreign_key_list` output plus `database_name` and `table_name`.
  - `indexes` - a list of indexes - `pragma_table_xinfo` output plus `database_name` and `table_name`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747834113
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0NzgzNDExMw==
created_at: 2020-12-18T02:51:13Z
updated_at: 2020-12-18T02:51:20Z
author_association: OWNER
body:
  SQLite uses `indexes` rather than `indices` as the plural, so I'll go with that: https://sqlite.org/lang_createindex.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747834462
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0NzgzNDQ2Mg==
created_at: 2020-12-18T02:52:19Z
updated_at: 2020-12-18T02:52:26Z
author_association: OWNER
body:
  Maintaining this database will be the responsibility of a subclass of `Database` called `_SchemaDatabase` which will be managed by the `Datasette` instance.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747834762
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0NzgzNDc2Mg==
created_at: 2020-12-18T02:53:22Z
updated_at: 2020-12-18T02:53:22Z
author_association: OWNER
body:
  I think I'm going to have to build this without using the `pragma_x()` SQL functions as they were only added in 3.16 in 2017-01-02 and I've seen plenty of Datasette instances running on older versions of SQLite.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747847180
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzg0NzE4MA==
created_at: 2020-12-18T03:35:15Z
updated_at: 2020-12-18T03:35:15Z
author_association: OWNER
body:
  Simpler implementation idea: a Datasette method `.refresh_schemas()` which loops through all known databases, checks their schema version and updates the in-memory schemas database if they have changed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747847405
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzg0NzQwNQ==
created_at: 2020-12-18T03:36:04Z
updated_at: 2020-12-18T03:36:04Z
author_association: OWNER
body:
  I could have another table that stores the combined rows from `sqlite_máster` on every connected database so I have a copy of the schema SQL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747861357
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzg2MTM1Nw==
created_at: 2020-12-18T04:32:52Z
updated_at: 2020-12-18T04:32:52Z
author_association: OWNER
body:
  I need to figure out how this will interact with Datasette permissions.

  If some tables are private, but others are public, should users be able to see the private tables listed in the schema metadata?

  If not, how can that mechanism work?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747861556
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzg2MTU1Ng==
created_at: 2020-12-18T04:33:45Z
updated_at: 2020-12-18T04:33:45Z
author_association: OWNER
body:
  One solution on permissions: if Datasette had an efficient way of saying "list the tables that this user has access to" I could use that as a filter any time the user views the schema information. The implementation could be tricky though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747862001
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzg2MjAwMQ==
created_at: 2020-12-18T04:35:34Z
updated_at: 2020-12-18T04:35:34Z
author_association: OWNER
body:
  I do need to solve the permissions problem properly though, because one of the goals of this system is to provide a paginated, searchable list of databases and tables for the homepage of the instance - #991.

  As such, the homepage will need to be able to display only the tables and databases that the user has permission to view.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747864080
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzg2NDA4MA==
created_at: 2020-12-18T04:43:29Z
updated_at: 2020-12-18T04:43:29Z
author_association: OWNER
body:
  I may be overthinking that problem. Many queries are fast in SQLite. If a Datasette instance has 1,000 connected tables will even that be a performance problem for permission checks? I should benchmark to find out.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747864831
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzg2NDgzMQ==
created_at: 2020-12-18T04:46:18Z
updated_at: 2020-12-18T04:46:18Z
author_association: OWNER
body:
  The homepage currently performs a massive flurry of permission checks - one for each, database, table and view: https://github.com/simonw/datasette/blob/0.53/datasette/views/index.py#L21-L75

  A paginated version of this is a little daunting as the permission checks would have to be carried out in every single table just to calculate the count that will be paginated.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1152#issuecomment-747891854
issue_url: https://api.github.com/repos/simonw/datasette/issues/1152
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzg5MTg1NA==
created_at: 2020-12-18T06:14:09Z
updated_at: 2020-12-18T06:14:15Z
author_association: OWNER
body:
  This is a classic challenge in permissions systems. If I want Datasette to be able to handle thousands of tables I need a reasonable solution for it.

  Twitter conversation: https://twitter.com/simonw/status/1339791768842248192

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1152#issuecomment-747892731
issue_url: https://api.github.com/repos/simonw/datasette/issues/1152
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzg5MjczMQ==
created_at: 2020-12-18T06:16:29Z
updated_at: 2020-12-18T06:16:29Z
author_association: OWNER
body:
  One enormous advantage I have is that after #1150 I will have a database table full of databases and tables that I can execute queries against.

  This means I could calculate visible tables using SQL where clauses, which should be easily fast enough even against ten thousand plus tables.

  The catch is the permissions hooks. Since I haven't hit Datasette 1.0 yet maybe I should redesign those hooks to work against the new in-memory database schema stuff?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1152#issuecomment-747893423
issue_url: https://api.github.com/repos/simonw/datasette/issues/1152
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzg5MzQyMw==
created_at: 2020-12-18T06:18:24Z
updated_at: 2020-12-18T06:18:24Z
author_association: OWNER
body:
  What would Datasette's permission hooks look like if they all dealt with sets of items rather than individual items? So plugins could return a set of items that the user has permission to access, or even a WHERE clause?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-747893704
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzg5MzcwNA==
created_at: 2020-12-18T06:19:13Z
updated_at: 2020-12-18T06:19:13Z
author_association: OWNER
body:
  I'm not going to block this issue on permissions - I will tackle the efficient bulk permissions problem in #1152.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1152#issuecomment-747919782
issue_url: https://api.github.com/repos/simonw/datasette/issues/1152
node_id: MDEyOklzc3VlQ29tbWVudDc0NzkxOTc4Mg==
created_at: 2020-12-18T07:27:01Z
updated_at: 2020-12-18T07:27:01Z
author_association: OWNER
body:
  Perhaps this can be solved by keeping the existing plugin hooks and adding new, optional ones for bulk lookups.

  If your plugin doesn't implement the bulk lookup hooks Datasette will do an inefficient loop through everything checking permissions on each one.

  If you DO implement it you can speed things up dramatically.

  Not sure if this would solve the homepage problem though, where you might need to run 1,000 table permission checks. That's more a case where you want to think in terms of a SQL where clause.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1152#issuecomment-747920087
issue_url: https://api.github.com/repos/simonw/datasette/issues/1152
node_id: MDEyOklzc3VlQ29tbWVudDc0NzkyMDA4Nw==
created_at: 2020-12-18T07:27:58Z
updated_at: 2020-12-18T07:28:30Z
author_association: OWNER
body:
  I want to keep the existing `metadata.json` "allow" blocks mechanism working. Note that if you have 1,000 tables and a permissions policy you won't be using "allow" blocks, you'll be using a more sophisticated permissions plugin instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1152#issuecomment-747920515
issue_url: https://api.github.com/repos/simonw/datasette/issues/1152
node_id: MDEyOklzc3VlQ29tbWVudDc0NzkyMDUxNQ==
created_at: 2020-12-18T07:29:21Z
updated_at: 2020-12-22T23:57:29Z
author_association: OWNER
body:
  Could I solve this using a configured canned query against the `_internal` tables with the actor's properties as inputs?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1152#issuecomment-747920852
issue_url: https://api.github.com/repos/simonw/datasette/issues/1152
node_id: MDEyOklzc3VlQ29tbWVudDc0NzkyMDg1Mg==
created_at: 2020-12-18T07:30:22Z
updated_at: 2020-12-18T07:30:22Z
author_association: OWNER
body:
  Redefining all Datasette permissions in terms of SQL queries that return the set of databases and tables that the user is allowed to interact with does feel VERY Datasette-y.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/493#issuecomment-747966232
issue_url: https://api.github.com/repos/simonw/datasette/issues/493
node_id: MDEyOklzc3VlQ29tbWVudDc0Nzk2NjIzMg==
created_at: 2020-12-18T09:19:41Z
updated_at: 2020-12-18T09:19:41Z
author_association: OWNER
body:
  Is there any reason to keep `--setting` rather than moving those items into a `configure.json` file with all the configuration options that currently live in `metadata.json`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1152#issuecomment-748206874
issue_url: https://api.github.com/repos/simonw/datasette/issues/1152
node_id: MDEyOklzc3VlQ29tbWVudDc0ODIwNjg3NA==
created_at: 2020-12-18T17:03:00Z
updated_at: 2020-12-22T23:58:04Z
author_association: OWNER
body:
  Another permissions thought: what if ALL Datasette permissions were default-deny, and plugins could only grant permission to things, not block permission?

  Right now a plugin can reply `False` to block, `True` to allow or `None` for "I have no opinion on this, ask someone else" - but even I'm confused by the interactions between block and allow and I implemented the system!

  If everything in Datasette was default-deny then the user could use `--public-view` as an option when starting the server to default-allow view actions.

  More importantly: plugins could return SQL statements that select a list of databases/tables the user is allowed access to. These could then be combined with `UNION` to create a full list of available resources.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-748260118
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0ODI2MDExOA==
created_at: 2020-12-18T18:54:12Z
updated_at: 2020-12-18T18:54:12Z
author_association: OWNER
body:
  I'm going to tidy this up and land it. A couple of additional decisions:

  - The database will be called `/_schemas`
  - By default it will only be visible to `root` - thus avoiding having to solve the permissions problem with regards to users seeing schemas for tables that are otherwise invisible to them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-748260875
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0ODI2MDg3NQ==
created_at: 2020-12-18T18:55:12Z
updated_at: 2020-12-18T18:55:12Z
author_association: OWNER
body:
  I'm going to move the code into a `utils/schemas.py` module, to avoid further extending the `Datasette` class definition and to make it more easily testable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/493#issuecomment-748305976
issue_url: https://api.github.com/repos/simonw/datasette/issues/493
node_id: MDEyOklzc3VlQ29tbWVudDc0ODMwNTk3Ng==
created_at: 2020-12-18T20:34:39Z
updated_at: 2020-12-18T20:34:39Z
author_association: CONTRIBUTOR
body:
  I can't keep up with the renaming contexts, but I like having the ability to run datasette+ datasette-ripgrep against different configs: 

  ```shell
  datasette serve --metadata=./metadata.json
  ```

  I have one for all of my code and one per client who has lots of code. So as long as I can point to datasette to something, it's easy to work with. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-748351350
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0ODM1MTM1MA==
created_at: 2020-12-18T22:32:13Z
updated_at: 2020-12-18T22:32:13Z
author_association: OWNER
body:
  Getting all the tests to pass is tricky because this adds a whole extra database to Datasette - and there's various code that loops through `ds.databases` as part of the tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-748354841
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc0ODM1NDg0MQ==
created_at: 2020-12-18T22:43:49Z
updated_at: 2020-12-18T22:43:49Z
author_association: OWNER
body:
  For a demo, visit https://latest.datasette.io/login-as-root and then hit https://latest.datasette.io/_schemas

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1155#issuecomment-748356492
issue_url: https://api.github.com/repos/simonw/datasette/issues/1155
node_id: MDEyOklzc3VlQ29tbWVudDc0ODM1NjQ5Mg==
created_at: 2020-12-18T22:49:32Z
updated_at: 2020-12-22T01:13:05Z
author_association: OWNER
body:
  There's some messy code that needs fixing here. The `datasette.databases` dictionary right now has a key that corresponds to the `/_internal` URL in the path, and a value that's a `Database()` object. BUT... the `Database()` object doesn't know what its key is.

  While fixing this I should fix the issue where Datasette gets confused by multiple databases with the same stem: https://github.com/simonw/datasette/issues/509

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/509#issuecomment-748356637
issue_url: https://api.github.com/repos/simonw/datasette/issues/509
node_id: MDEyOklzc3VlQ29tbWVudDc0ODM1NjYzNw==
created_at: 2020-12-18T22:50:03Z
updated_at: 2020-12-18T22:50:03Z
author_association: OWNER
body:
  Related problem caused by the new `_schemas` database - if a user attempts to open their own `_schemas.db` file it will fail. I'd like to open and mount that as `/_schemas_` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1155#issuecomment-748367922
issue_url: https://api.github.com/repos/simonw/datasette/issues/1155
node_id: MDEyOklzc3VlQ29tbWVudDc0ODM2NzkyMg==
created_at: 2020-12-18T23:15:24Z
updated_at: 2020-12-18T23:15:24Z
author_association: OWNER
body:
  The code for building up that `.databases` dictionary is a bit convoluted. Here's the code that adds a `:memory:` database if the user specified `--memory` OR if there are no files to be attached:

  https://github.com/simonw/datasette/blob/ebc7aa287c99fe6114b79aeab8efb8d4489a6182/datasette/app.py#L221-L241

  I'm not sure why I wrote it this way, instead of just calling `.add_database(":memory:", Database(..., is_memory=True)`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1155#issuecomment-748368384
issue_url: https://api.github.com/repos/simonw/datasette/issues/1155
node_id: MDEyOklzc3VlQ29tbWVudDc0ODM2ODM4NA==
created_at: 2020-12-18T23:17:00Z
updated_at: 2020-12-18T23:17:00Z
author_association: OWNER
body:
  Here's the commit where I added it. https://github.com/simonw/datasette/commit/9743e1d91b5f0a2b3c1c0bd6ffce8739341f43c4 - I didn't yet have the `.add_database()` mechanism. Today the `MEMORY` object bit is no longer needed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1155#issuecomment-748368660
issue_url: https://api.github.com/repos/simonw/datasette/issues/1155
node_id: MDEyOklzc3VlQ29tbWVudDc0ODM2ODY2MA==
created_at: 2020-12-18T23:18:04Z
updated_at: 2020-12-19T01:12:00Z
author_association: OWNER
body:
  A `Database` should have a `.name` which is unique across the Datasette instance and is used in the URL. The `path` should be optional, only set for file databases. A new `.memory_name` property can be used for shared memory databases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1155#issuecomment-748368938
issue_url: https://api.github.com/repos/simonw/datasette/issues/1155
node_id: MDEyOklzc3VlQ29tbWVudDc0ODM2ODkzOA==
created_at: 2020-12-18T23:19:04Z
updated_at: 2020-12-18T23:19:04Z
author_association: OWNER
body:
  `Database` internal class is documented here: https://docs.datasette.io/en/latest/internals.html#database-class

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1155#issuecomment-748397998
issue_url: https://api.github.com/repos/simonw/datasette/issues/1155
node_id: MDEyOklzc3VlQ29tbWVudDc0ODM5Nzk5OA==
created_at: 2020-12-19T01:28:18Z
updated_at: 2020-12-19T01:28:18Z
author_association: OWNER
body:
  `datasette-graphql` returns an error due to this issue:

  <img width="878" alt="GraphiQL_and_fixtures__compound_three_primary_keys__1_001_rows" src="https://user-images.githubusercontent.com/9599/102677257-61e01380-4156-11eb-8f88-fe26b9c4a40b.png">

  On the console:
  ```
  INFO:     127.0.0.1:63116 - "POST /graphql/_internal HTTP/1.1" 500 Internal Server Error
  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/promise/promise.py", line 844, in handle_future_result
      resolve(future.result())
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/datasette_graphql/utils.py", line 603, in resolve_table
      data, _, _ = await view.data(
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 304, in data
      db = self.ds.databases[database]
  graphql.error.located_error.GraphQLLocatedError: ':memory:c6dd5abe1a757a7de00d99b699175bd33d9a575f05b5751bf856b8656fb07edd'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426501
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31
node_id: MDEyOklzc3VlQ29tbWVudDc0ODQyNjUwMQ==
created_at: 2020-12-19T06:12:22Z
updated_at: 2020-12-19T06:12:22Z
author_association: MEMBER
body:
  I deliberately added support for advanced FTS in https://github.com/dogsheep/dogsheep-beta/commit/cbb2491b85d7ff416d6d429b60109e6c2d6d50b9 for #13 but that's the cause of this bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426581
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31
node_id: MDEyOklzc3VlQ29tbWVudDc0ODQyNjU4MQ==
created_at: 2020-12-19T06:13:17Z
updated_at: 2020-12-19T06:13:17Z
author_association: MEMBER
body:
  One fix for this could be to try running the raw query, but if it throws an error run it again with the query escaped.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426663
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31
node_id: MDEyOklzc3VlQ29tbWVudDc0ODQyNjY2Mw==
created_at: 2020-12-19T06:14:06Z
updated_at: 2020-12-19T06:14:06Z
author_association: MEMBER
body:
  Looks like I already do that here: https://github.com/dogsheep/dogsheep-beta/blob/9ba4401017ac24ffa3bc1db38e0910ea49de7616/dogsheep_beta/__init__.py#L141-L146

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426877
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31
node_id: MDEyOklzc3VlQ29tbWVudDc0ODQyNjg3Nw==
created_at: 2020-12-19T06:16:11Z
updated_at: 2020-12-19T06:16:11Z
author_association: MEMBER
body:
  Here's why:

      if "fts5" in str(e):

  But the error being raised here is:

      sqlite3.OperationalError: no such column: to

  I'm going to attempt the escaped on on every error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436115
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15
node_id: MDEyOklzc3VlQ29tbWVudDc0ODQzNjExNQ==
created_at: 2020-12-19T07:43:38Z
updated_at: 2020-12-19T07:47:36Z
author_association: NONE
body:
  Hey Simon! I really enjoy datasette so far, just started trying it out today following your iPhone photos [example](https://simonwillison.net/2020/May/21/dogsheep-photos/). 

  I am not sure if you had run into this or not, but it seems like they might have changed one of the column names from
  `ZGENERICASSET` to `ZASSET`. Should I open a PR? 

  Would change:
  - [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L209-L213)
  - [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L238)
  - [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L240)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-748436195
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21
node_id: MDEyOklzc3VlQ29tbWVudDc0ODQzNjE5NQ==
created_at: 2020-12-19T07:44:32Z
updated_at: 2020-12-19T07:44:49Z
author_association: NONE
body:
  I have also run into this a bit, would it be possible to post your `requirements.txt` so I can try and reproduce your [blog post](https://simonwillison.net/2020/May/21/dogsheep-photos/)?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/53#issuecomment-748436453
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/53
node_id: MDEyOklzc3VlQ29tbWVudDc0ODQzNjQ1Mw==
created_at: 2020-12-19T07:47:01Z
updated_at: 2020-12-19T07:47:01Z
author_association: NONE
body:
  I think this should probably be closed as won't fix.

  Attempting to make a patch for this I realized that the since_id would limit to tweets posted since that since_id, not when it was favorited.  So favoriting something in the older would be missed if you used `--since` with a cron script

  Better to just use `--stop_after` set to a couple hundred

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436779
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15
node_id: MDEyOklzc3VlQ29tbWVudDc0ODQzNjc3OQ==
created_at: 2020-12-19T07:49:00Z
updated_at: 2020-12-19T07:49:00Z
author_association: CONTRIBUTOR
body:
  @nickvazz ZGENERICASSET changed to ZASSET in Big Sur. Here's a list of other changes to the schema in Big Sur: https://github.com/RhetTbull/osxphotos/wiki/Changes-in-Photos-6---Big-Sur

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748562288
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15
node_id: MDEyOklzc3VlQ29tbWVudDc0ODU2MjI4OA==
created_at: 2020-12-20T04:44:22Z
updated_at: 2020-12-20T04:44:22Z
author_association: CONTRIBUTOR
body:
  @nickvazz @simonw I opened a [PR](https://github.com/dogsheep/dogsheep-photos/pull/31) that replaces the SQL for `ZCOMPUTEDASSETATTRIBUTES` to use osxphotos which now exposes all this data and has been updated for Big Sur.  I did regression tests to confirm the extracted data is identical, with one exception which should not affect operation: the old code pulled data from `ZCOMPUTEDASSETATTRIBUTES` for missing photos while the main loop ignores missing photos and does not add them to `apple_photos`.  The new code does not add rows to the `apple_photos_scores` table for missing photos.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-748562330
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31
node_id: MDEyOklzc3VlQ29tbWVudDc0ODU2MjMzMA==
created_at: 2020-12-20T04:45:08Z
updated_at: 2020-12-20T04:45:08Z
author_association: CONTRIBUTOR
body:
  Fixes the issue mentioned here: https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436115

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1156#issuecomment-749158111
issue_url: https://api.github.com/repos/simonw/datasette/issues/1156
node_id: MDEyOklzc3VlQ29tbWVudDc0OTE1ODExMQ==
created_at: 2020-12-21T19:33:45Z
updated_at: 2020-12-21T19:33:45Z
author_association: OWNER
body:
  One reason for this change: it means I can use that database for more stuff. I've been thinking about moving metadata storage there for example, which fits a database called `_internal` but not one called `_schemas`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1155#issuecomment-749176936
issue_url: https://api.github.com/repos/simonw/datasette/issues/1155
node_id: MDEyOklzc3VlQ29tbWVudDc0OTE3NjkzNg==
created_at: 2020-12-21T20:18:15Z
updated_at: 2020-12-21T20:18:15Z
author_association: OWNER
body:
  Fun query:
  ```sql
  select table_name, group_concat(name, ', ') from columns group by database_name, table_name
  ```
  https://latest.datasette.io/_internal?sql=select+table_name%2C+group_concat%28name%2C+%27%2C+%27%29+from+columns+group+by+database_name%2C+table_name

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1157#issuecomment-749179460
issue_url: https://api.github.com/repos/simonw/datasette/issues/1157
node_id: MDEyOklzc3VlQ29tbWVudDc0OTE3OTQ2MA==
created_at: 2020-12-21T20:24:19Z
updated_at: 2020-12-21T20:24:19Z
author_association: OWNER
body:
  Three places to fix:

  https://github.com/simonw/datasette/blob/dcdfb2c301341d45b66683e3e3be72f9c7585b2f/datasette/tracer.py#L40-L42

  https://github.com/simonw/datasette/blob/dcdfb2c301341d45b66683e3e3be72f9c7585b2f/datasette/utils/__init__.py#L139-L152

  https://github.com/simonw/datasette/blob/dcdfb2c301341d45b66683e3e3be72f9c7585b2f/datasette/views/base.py#L460-L461

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1155#issuecomment-749723557
issue_url: https://api.github.com/repos/simonw/datasette/issues/1155
node_id: MDEyOklzc3VlQ29tbWVudDc0OTcyMzU1Nw==
created_at: 2020-12-22T19:08:27Z
updated_at: 2020-12-22T19:08:27Z
author_association: OWNER
body:
  I'm going to have the `.add_database()` method select the name used in the path, de-duping against any existing names. It will then set database.name to that so that the database has access to its own name.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/509#issuecomment-749749948
issue_url: https://api.github.com/repos/simonw/datasette/issues/509
node_id: MDEyOklzc3VlQ29tbWVudDc0OTc0OTk0OA==
created_at: 2020-12-22T20:03:10Z
updated_at: 2020-12-22T20:03:10Z
author_association: OWNER
body:
  If you open multiple files with the same filename, e.g. like this:

      datasette fixtures.db templates/fixtures.db plugins/fixtures.db

  You'll now get this:

  <img width="702" alt="Datasette__fixtures__fixtures_2__fixtures_3" src="https://user-images.githubusercontent.com/9599/102928494-a4069f00-444d-11eb-9c1e-382a4e27bb51.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1152#issuecomment-749750995
issue_url: https://api.github.com/repos/simonw/datasette/issues/1152
node_id: MDEyOklzc3VlQ29tbWVudDc0OTc1MDk5NQ==
created_at: 2020-12-22T20:05:30Z
updated_at: 2020-12-22T20:05:30Z
author_association: OWNER
body:
  #1150 is landed now, which means there's a new, hidden `_internal` SQLite in-memory database containing all of the tables and databases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1099#issuecomment-749771231
issue_url: https://api.github.com/repos/simonw/datasette/issues/1099
node_id: MDEyOklzc3VlQ29tbWVudDc0OTc3MTIzMQ==
created_at: 2020-12-22T20:54:25Z
updated_at: 2020-12-22T20:54:25Z
author_association: OWNER
body:
  https://latest.datasette.io/_internal/foreign_keys (use https://latest.datasette.io/login-as-root first) is now a compound foreign key table:
  ```sql
  CREATE TABLE foreign_keys (
          "database_name" TEXT,
          "table_name" TEXT,
          "id" INTEGER,
          "seq" INTEGER,
          "table" TEXT,
          "from" TEXT,
          "to" TEXT,
          "on_update" TEXT,
          "on_delete" TEXT,
          "match" TEXT,
          PRIMARY KEY (database_name, table_name, id, seq),
          FOREIGN KEY (database_name) REFERENCES databases(database_name),
          FOREIGN KEY (database_name, table_name) REFERENCES tables(database_name, table_name)
      );
  ```
  Currently the `database_name` column becomes a link (because it's a single foreign key) but the `table_name` one remains a non-link:

  <img width="1079" alt="_internal__foreign_keys__24_rows" src="https://user-images.githubusercontent.com/9599/102932110-87ba3080-4454-11eb-9ad5-b70a65129588.png">

  My original idea for compound foreign keys was to turn both of those columns into links, but that doesn't fit here because `database_name` is already part of a different foreign key.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1099#issuecomment-749845797
issue_url: https://api.github.com/repos/simonw/datasette/issues/1099
node_id: MDEyOklzc3VlQ29tbWVudDc0OTg0NTc5Nw==
created_at: 2020-12-23T00:13:29Z
updated_at: 2020-12-23T00:14:25Z
author_association: OWNER
body:
  Also need to solve displaying these links in the opposite direction:

  https://latest.datasette.io/_internal/tables/fixtures,facet_cities

  <img width="617" alt="_internal__tables" src="https://user-images.githubusercontent.com/9599/102944742-84cd3900-4470-11eb-95a0-bfaf00b17e82.png">

  That page should link to lists of records in columns, foreign_keys and indexes - like this example: https://latest.datasette.io/fixtures/roadside_attractions/1

  <img width="712" alt="fixtures__roadside_attractions" src="https://user-images.githubusercontent.com/9599/102944816-c1009980-4470-11eb-9dbd-50d0ba4fd137.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1158#issuecomment-750373496
issue_url: https://api.github.com/repos/simonw/datasette/issues/1158
node_id: MDEyOklzc3VlQ29tbWVudDc1MDM3MzQ5Ng==
created_at: 2020-12-23T16:26:06Z
updated_at: 2020-12-23T16:26:06Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=h1) Report
  > Merging [#1158](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=desc) (37ce72f) into [main](https://codecov.io/gh/simonw/datasette/commit/90eba4c3ca569c57e96bce314e7ac8caf67d884e?el=desc) (90eba4c) will **not change** coverage.
  > The diff coverage is `87.50%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1158/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1158   +/-   ##
  =======================================
    Coverage   91.55%   91.55%           
  =======================================
    Files          32       32           
    Lines        3930     3930           
  =======================================
    Hits         3598     3598           
    Misses        332      332           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `77.41% <ø> (ø)` | |
  | [datasette/facets.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2ZhY2V0cy5weQ==) | `89.04% <ø> (ø)` | |
  | [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `94.35% <ø> (ø)` | |
  | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <ø> (ø)` | |
  | [datasette/inspect.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2luc3BlY3QucHk=) | `36.11% <ø> (ø)` | |
  | [datasette/renderer.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3JlbmRlcmVyLnB5) | `94.02% <ø> (ø)` | |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.01% <50.00%> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.85% <100.00%> (ø)` | |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.11% <100.00%> (ø)` | |
  | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `92.13% <100.00%> (ø)` | |
  | ... and [1 more](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree-more) | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=footer). Last update [90eba4c...37ce72f](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1158#issuecomment-750373603
issue_url: https://api.github.com/repos/simonw/datasette/issues/1158
node_id: MDEyOklzc3VlQ29tbWVudDc1MDM3MzYwMw==
created_at: 2020-12-23T16:26:21Z
updated_at: 2020-12-23T16:26:21Z
author_association: OWNER
body:
  Did you use a tool to find these or are they all from manual code inspection?

  They look good - I particularly like the set/dict  literals.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1158#issuecomment-750389683
issue_url: https://api.github.com/repos/simonw/datasette/issues/1158
node_id: MDEyOklzc3VlQ29tbWVudDc1MDM4OTY4Mw==
created_at: 2020-12-23T17:02:50Z
updated_at: 2020-12-23T17:02:50Z
author_association: CONTRIBUTOR
body:
  The dict/set suggestion comes from `pyupgrade --py36-plus`, but then had to `black` the change.

  The rest comes from PyCharm's Inspect code function. I reviewed all the suggestions and fixed a thing or two, such as leading/trailing spaces in the docstrings or turned around the chained conditions.

  Then I tried to convert all `os.path/glob/open` to `Path`, but there were some local test issues, so I'll have to start over in smaller chunks if you want to have that too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1159#issuecomment-750849460
issue_url: https://api.github.com/repos/simonw/datasette/issues/1159
node_id: MDEyOklzc3VlQ29tbWVudDc1MDg0OTQ2MA==
created_at: 2020-12-24T11:07:35Z
updated_at: 2020-12-24T11:29:21Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=h1) Report
  > Merging [#1159](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=desc) (c820abd) into [main](https://codecov.io/gh/simonw/datasette/commit/a882d679626438ba0d809944f06f239bcba8ee96?el=desc) (a882d67) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1159/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1159   +/-   ##
  =======================================
    Coverage   91.55%   91.55%           
  =======================================
    Files          32       32           
    Lines        3930     3930           
  =======================================
    Hits         3598     3598           
    Misses        332      332           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=footer). Last update [a882d67...c820abd](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/417#issuecomment-751127384
issue_url: https://api.github.com/repos/simonw/datasette/issues/417
node_id: MDEyOklzc3VlQ29tbWVudDc1MTEyNzM4NA==
created_at: 2020-12-24T22:56:48Z
updated_at: 2020-12-24T22:56:48Z
author_association: NONE
body:
  Instead of scanning the directory every 10s, have you considered listening for the native system events to notify you of updates?

  I think python has a nice module to do this for you called [watchdog](https://pypi.org/project/watchdog/)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/417#issuecomment-751127485
issue_url: https://api.github.com/repos/simonw/datasette/issues/417
node_id: MDEyOklzc3VlQ29tbWVudDc1MTEyNzQ4NQ==
created_at: 2020-12-24T22:58:05Z
updated_at: 2020-12-24T22:58:05Z
author_association: OWNER
body:
  That's a great idea. I'd ruled that out because working with the different operating system versions of those is tricky, but if `watchdog` can handle those differences for me this could be a really good option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-751375487
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59
node_id: MDEyOklzc3VlQ29tbWVudDc1MTM3NTQ4Nw==
created_at: 2020-12-26T17:08:44Z
updated_at: 2020-12-26T17:08:44Z
author_association: CONTRIBUTOR
body:
  Hi @simonw,  do I need to do anything else for this PR to be considered to be included? I've tried using this project and it is quite nice to be able to explore a repository, but noticed that a couple commands don't allow you to use authorization from the environment variable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1150#issuecomment-751476406
issue_url: https://api.github.com/repos/simonw/datasette/issues/1150
node_id: MDEyOklzc3VlQ29tbWVudDc1MTQ3NjQwNg==
created_at: 2020-12-27T14:51:39Z
updated_at: 2020-12-27T14:51:39Z
author_association: NONE
body:
  I like the idea of _internal, it's a nice way to get a data catalog quickly. I wonder if this trick applies to db other than SQLite.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/417#issuecomment-751504136
issue_url: https://api.github.com/repos/simonw/datasette/issues/417
node_id: MDEyOklzc3VlQ29tbWVudDc1MTUwNDEzNg==
created_at: 2020-12-27T19:02:06Z
updated_at: 2020-12-27T19:02:06Z
author_association: NONE
body:
  Very much looking forward to seeing this functionality come together. This is probably out-of-scope for an initial release, but in the future it could be useful to also think of how to run this is a container'ized context. For example, an immutable datasette container that points to an S3 bucket of SQLite DBs or CSVs. Or an immutable datasette container pointing to a NFS volume elsewhere on a Kubernetes cluster.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/675#issuecomment-751826621
issue_url: https://api.github.com/repos/simonw/datasette/issues/675
node_id: MDEyOklzc3VlQ29tbWVudDc1MTgyNjYyMQ==
created_at: 2020-12-28T18:48:51Z
updated_at: 2020-12-28T18:48:51Z
author_association: OWNER
body:
  I could make `--include` work if I also had a mechanism for running some shell commands inside the container at the end of the build - which users could then use to move files into the correct place.

      datasette publish cloudrun my.db --include src/ --exec 'mv /app/src/config.yml /etc/conf/config.yml'

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/675#issuecomment-751826749
issue_url: https://api.github.com/repos/simonw/datasette/issues/675
node_id: MDEyOklzc3VlQ29tbWVudDc1MTgyNjc0OQ==
created_at: 2020-12-28T18:49:21Z
updated_at: 2020-12-28T18:49:21Z
author_association: OWNER
body:
  That `--exec` could help solve all sorts of other problems too, like needing to `apt-get install` extra packages or download files from somewhere using `wget`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-751925934
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MTkyNTkzNA==
created_at: 2020-12-29T02:40:13Z
updated_at: 2020-12-29T20:25:57Z
author_association: OWNER
body:
  Basic command design:

      datasette insert data.db blah.csv

  The options can include:

  - `--format` to specify the exact format - without this it will be guessed based on the filename
  - `--table` to specify the table (otherwise the filename is used)
  - `--pk` to specify one or more primary key columns
  - `--replace` to specify that existing rows with a matching primary key should be replaced
  - `--upsert` to specify that existing matching rows should be upserted
  - `--ignore` to ignore matching rows
  - `--alter` to alter the table to add missing columns
  - `--type column type` to specify the type of a column - useful when working with CSV or TSV files

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-751926095
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MTkyNjA5NQ==
created_at: 2020-12-29T02:41:15Z
updated_at: 2020-12-29T02:41:15Z
author_association: OWNER
body:
  The UI can live at `/-/insert` and be available by default to the `root` user only. It can offer the following:

  - Upload a file and have the import type detected (equivalent to `datasette insert data.db thatfile.csv`)
  - Copy and paste the data to be inserted into a textarea
  - API equivalents of these

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-751926218
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MTkyNjIxOA==
created_at: 2020-12-29T02:41:57Z
updated_at: 2020-12-29T02:41:57Z
author_association: OWNER
body:
  Other names I considered:

  - `datasette load`
  - `datasette import` - I decided to keep this name available for any future work that might involve plugins that help import data from APIs as opposed to inserting it from files

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-751926437
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MTkyNjQzNw==
created_at: 2020-12-29T02:43:21Z
updated_at: 2020-12-29T02:43:37Z
author_association: OWNER
body:
  Default formats to support:

  - CSV
  - TSV
  - JSON and newline-delimited JSON
  - YAML

  Each of these will be implemented as a default plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-751943837
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MTk0MzgzNw==
created_at: 2020-12-29T04:40:30Z
updated_at: 2020-12-29T04:40:30Z
author_association: OWNER
body:
  The `insert` command should also accept URLs - anything starting with `http://` or `https://`.

  It should accept more than one file name at a time for bulk inserts.

  if using a URL that URL will be passed to the method that decides if a plugin implementation can handle the import or not. This will allow plugins to register themselves for specific websites.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-751945094
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MTk0NTA5NA==
created_at: 2020-12-29T04:48:11Z
updated_at: 2020-12-29T04:48:11Z
author_association: OWNER
body:
  It would be pretty cool if you could launch Datasette directly against an insert-compatible file or URL without first having to load it into a SQLite database file.

  Or imagine being able to tail a log file and like that directly into a new Datasette process, which then runs a web server with the UI while simultaneously continuing to load new entries from that log into the in-memory SQLite database that it is serving...

  Not quite sure what that CLI interface would look like. Maybe treat that as a future stretch goal for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-751946262
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MTk0NjI2Mg==
created_at: 2020-12-29T04:56:12Z
updated_at: 2020-12-29T04:56:32Z
author_association: OWNER
body:
  Potential design for this: a `datasette memory` command which takes most of the same arguments as `datasette serve` but starts an in-memory database and treats the command arguments as things that should be inserted into that in-memory database.

      tail -f access.log | datasette memory - \
        --format clf -p 8002 -o

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-751947991
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MTk0Nzk5MQ==
created_at: 2020-12-29T05:06:50Z
updated_at: 2020-12-29T05:07:03Z
author_association: OWNER
body:
  Given the URL option could it be possible for plugins to "subscribe" to URLs that keep on streaming?

      datasette insert db.db https://example.con/streaming-api \
        --format api-stream

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/417#issuecomment-752098906
issue_url: https://api.github.com/repos/simonw/datasette/issues/417
node_id: MDEyOklzc3VlQ29tbWVudDc1MjA5ODkwNg==
created_at: 2020-12-29T14:34:30Z
updated_at: 2020-12-29T14:34:50Z
author_association: CONTRIBUTOR
body:
  FWIW, I had a look at `watchdog` for a `datasette` powered Jupyter notebook search tool: https://github.com/ouseful-testing/nbsearch/blob/main/nbsearch/nbwatchdog.py

  Not a production thing, just an experiment trying to explore what might be possible...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752203909
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjIwMzkwOQ==
created_at: 2020-12-29T18:54:19Z
updated_at: 2020-12-29T18:54:19Z
author_association: OWNER
body:
  More thoughts on this: the key mechanism that populates the tables needs to be an `aysnc def` method of some sort so that it can run as part of the async loop in core Datasette - for importing from web uploads.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752208036
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjIwODAzNg==
created_at: 2020-12-29T19:06:35Z
updated_at: 2020-12-29T19:06:35Z
author_association: OWNER
body:
  If I'm going to execute 1000s of writes in an `async def` operation it may make sense to break that up into smaller chunks, so as not to block the event loop for too long.

  https://stackoverflow.com/a/36648102 and https://github.com/python/asyncio/issues/284 confirm that `await asyncio.sleep(0)` is the recommended way of doing this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752236520
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjIzNjUyMA==
created_at: 2020-12-29T20:48:51Z
updated_at: 2020-12-29T20:48:51Z
author_association: OWNER
body:
  It would be neat if `datasette insert` could accept a `--plugins-dir` option which allowed one-off format plugins to be registered. Bit tricky to implement since the `--format` Click option will already be populated by that plugin hook call.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752257666
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjI1NzY2Ng==
created_at: 2020-12-29T22:09:18Z
updated_at: 2020-12-29T22:09:18Z
author_association: OWNER
body:
  ### Figuring out the API design

  I want to be able to support different formats, and be able to parse them into tables either streaming or in one go depending on if the format supports that.

  Ideally I want to be able to pull the first 1,024 bytes for the purpose of detecting the format, then replay those bytes again later. I'm considering this a stretch goal though.

  CSV is easy to parse as a stream - here’s [how sqlite-utils does it](https://github.com/simonw/sqlite-utils/blob/f1277f638f3a54a821db6e03cb980adad2f2fa35/sqlite_utils/cli.py#L630):

          dialect = "excel-tab" if tsv else "excel"
          with file_progress(json_file, silent=silent) as json_file:
              reader = csv_std.reader(json_file, dialect=dialect)
              headers = next(reader)
              docs = (dict(zip(headers, row)) for row in reader)

  Problem: using `db.insert_all()` could block for a long time on a big set of rows. Probably easiest to batch the records before calling `insert_all()` and then run a batch at a time using a `db.execute_write_fn()` call.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752259345
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjI1OTM0NQ==
created_at: 2020-12-29T22:11:54Z
updated_at: 2020-12-29T22:11:54Z
author_association: OWNER
body:
  Important detail from https://docs.python.org/3/library/csv.html#csv.reader

  > If *csvfile* is a file object, it should be opened with `newline=''`. [1]
  >
  > [...]
  >
  > If `newline=''` is not specified, newlines embedded inside quoted fields will not be interpreted correctly, and on platforms that use `\r\n` linendings on write an extra `\r` will be added. It should always be safe to specify `newline=''`, since the csv module does its own ([universal](https://docs.python.org/3/glossary.html#term-universal-newlines)) newline handling.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752265600
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjI2NTYwMA==
created_at: 2020-12-29T22:39:56Z
updated_at: 2020-12-29T22:39:56Z
author_association: OWNER
body:
  Does it definitely make sense to break this operation up into the code that turns the incoming format into a iterator of dictionaries, then the code that inserts those into the database using `sqlite-utils`?

  That seems right for simple imports, where the incoming file represents a sequence of records in a single table. But what about more complex formats? What if a format needs to be represented as multiple tables?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752266076
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjI2NjA3Ng==
created_at: 2020-12-29T22:42:23Z
updated_at: 2020-12-29T22:42:59Z
author_association: OWNER
body:
  Aside: maybe `datasette insert` works against simple files, but a later mechanism called `datasette import` allows plugins to register sub-commands, like `datasette import github ...` or `datasette import jira ...` or whatever.

  This would be useful for import mechanisms that are likely to need their own custom set of command-line options unique to that source.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752267905
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjI2NzkwNQ==
created_at: 2020-12-29T22:52:09Z
updated_at: 2020-12-29T22:52:09Z
author_association: OWNER
body:
  What's the simplest thing that could possible work? I think it's `datasette insert blah.db data.csv` - no URL handling, no other formats.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752273306
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjI3MzMwNg==
created_at: 2020-12-29T23:19:15Z
updated_at: 2020-12-29T23:19:15Z
author_association: OWNER
body:
  It would be nice if this abstraction could support progress bars as well. These won't necessarily work for every format - or they might work for things loaded from files but not things loaded over URLs (if the `content-length` HTTP header is missing) - but if they ARE possible it would be good to provide them - both for the CLI interface and the web insert UI.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752273873
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjI3Mzg3Mw==
created_at: 2020-12-29T23:22:30Z
updated_at: 2020-12-29T23:22:30Z
author_association: OWNER
body:
  How much of this should I get done in a branch before merging into `main`?

  The challenge here is the plugin hook design: ideally I don't want an incomplete plugin hook design in `main` since that could be a blocker for a release.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752274078
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjI3NDA3OA==
created_at: 2020-12-29T23:23:39Z
updated_at: 2020-12-29T23:23:39Z
author_association: OWNER
body:
  If I design this right I can ship a full version of the command-line `datasette insert` command in a release without doing any work at all on the Web UI version of it - that UI can then come later, without needing any changes to be made to the plugin hook.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752274509
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjI3NDUwOQ==
created_at: 2020-12-29T23:26:02Z
updated_at: 2020-12-29T23:26:02Z
author_association: OWNER
body:
  The documentation for this plugin hook is going to be pretty detailed, since it involves writing custom classes.

  I'll stick it all on the existing hooks page for the moment, but I should think about breaking up the plugin hook documentation into a page-per-hook in the future.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-752275611
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MjI3NTYxMQ==
created_at: 2020-12-29T23:32:04Z
updated_at: 2020-12-29T23:32:04Z
author_association: OWNER
body:
  If I can get this working for CSV, TSV, JSON and JSON-NL that should be enough to exercise the API design pretty well across both streaming and non-streaming formats.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/987#issuecomment-752696499
issue_url: https://api.github.com/repos/simonw/datasette/issues/987
node_id: MDEyOklzc3VlQ29tbWVudDc1MjY5NjQ5OQ==
created_at: 2020-12-30T17:21:08Z
updated_at: 2020-12-30T17:21:08Z
author_association: OWNER
body:
  More generally, I need to document certain areas of the page that JavaScript plugins are invited to append their content to - such that plugin authors can use them and feel confident that future changes to the Datasette templates won't break their plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/987#issuecomment-752697279
issue_url: https://api.github.com/repos/simonw/datasette/issues/987
node_id: MDEyOklzc3VlQ29tbWVudDc1MjY5NzI3OQ==
created_at: 2020-12-30T17:23:27Z
updated_at: 2020-12-30T17:23:32Z
author_association: OWNER
body:
  Related problem: right now if you're writing custom template it's not at all obvious how to write them such that visualization plugins like `datasette-cluster-map` will continue to work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/987#issuecomment-752714747
issue_url: https://api.github.com/repos/simonw/datasette/issues/987
node_id: MDEyOklzc3VlQ29tbWVudDc1MjcxNDc0Nw==
created_at: 2020-12-30T18:23:08Z
updated_at: 2020-12-30T18:23:20Z
author_association: OWNER
body:
  In terms of "places to put your plugin content", the simplest solution I can think of is something like this:
  ```html
  <div id="plugin-content-pre-table"></div>
  ```
  Alternative designs:

  - A documented JavaScript function that returns the CSS selector where plugins should put their content
  - A documented JavaScript function that returns a DOM node where plugins should put their content. This would allow the JavaScript to create the element if it does not already exist (though it wouldn't be obvious WHERE that element should be created)
  - Documented JavaScript functions for things like "append this node/HTML to the place-where-plugins-go"

  I think the original option - an empty `<div>` with a known `id` attribute - is the right one to go with here. It's the simplest, it's very easy for custom template authors to understand and it acknowledges that plugins may have all kinds of extra crazy stuff they want to do - like checking in that div to see if another plugin has written to it already, for example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752715236
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MjcxNTIzNg==
created_at: 2020-12-30T18:24:54Z
updated_at: 2020-12-30T18:24:54Z
author_association: OWNER
body:
  I think I'm going to try building a very lightweight clone of the core API design of Pluggy - not the advanced features, just the idea that plugins can register and a call to `plugin.nameOfHook()` will return the concatenated results of all of the registered hooks.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752715412
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MjcxNTQxMg==
created_at: 2020-12-30T18:25:31Z
updated_at: 2020-12-30T18:25:31Z
author_association: OWNER
body:
  I'm going to introduce a global `datasette` object which holds all the documented JavaScript API for plugin authors.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752721069
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MjcyMTA2OQ==
created_at: 2020-12-30T18:46:10Z
updated_at: 2020-12-30T18:46:10Z
author_association: OWNER
body:
  Pluggy does dependency injection by introspecting the named arguments to the Python function, which I really like.

  That's tricker in JavaScript. It looks like the only way to introspect a function is to look at the `.toString()` representation of it and parse the `(parameter, list)` using a regular expression.

  Even more challenging: JavaScript developers love minifying their code, and minification can shorten the function parameter names.

  From https://code-maven.com/dependency-injection-in-angularjs it looks like Angular.js does dependency injection and solves this by letting you optionally provide a separate list of the arguments your function uses:

  ```javascript
     angular.module('DemoApp', [])
     .controller('DemoController', ['$scope', '$log', function($scope, $log) {
         $scope.message = "Hello World";
         $log.debug('logging hello');
     }]);
  ```

  I can copy that approach: I'll introspect by default, but provide a documented mechanism for explicitly listing your parameter names so that if you know your plugin code will be minified you can use that instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752721840
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MjcyMTg0MA==
created_at: 2020-12-30T18:48:53Z
updated_at: 2020-12-30T18:51:51Z
author_association: OWNER
body:
  Potential design:
  ```javascript
  datasette.plugins.register('column_actions', function(database, table, column, actor) {
      /* ... *l
  })
  ```
  Or if you want to be explicit to survive minification:
  ```javascript
  datasette.plugins.register('column_actions', function(database, table, column, actor) {
      /* ... *l
  }, ['database', 'table', 'column', 'actor'])
  ```
  I'm making that list of parameter names an optional third argument to the `register()` function. If that argument isn't passed, introspection will be used to figure out the parameter names.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752722863
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MjcyMjg2Mw==
created_at: 2020-12-30T18:52:39Z
updated_at: 2020-12-30T18:52:39Z
author_association: OWNER
body:
  Then to call the plugins:
  ```javascript
  datasette.plugins.call('column_actions', {database: 'database', table: 'table'})
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752729035
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MjcyOTAzNQ==
created_at: 2020-12-30T19:15:56Z
updated_at: 2020-12-30T19:16:44Z
author_association: OWNER
body:
  The `column_actions` hook is the obvious first place to try this out. What are some demo plugins I could build for it?

  - Word cloud for this column
  - Count values (essentially linking to the SQL query for that column, as an extended version of the facet counts) - would be great if this could include pagination somehow, via #856.
  - Extract this column into a separate table
  - Add an index to this column

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752742669
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc0MjY2OQ==
created_at: 2020-12-30T20:07:05Z
updated_at: 2020-12-30T20:07:18Z
author_association: OWNER
body:
  Initial prototype:
  ```javascript
  window.datasette = {};
  window.datasette.plugins = (function() {
      var registry = {};

      function extractParameters(fn) {
          var match = /\((.*)\)/.exec(fn.toString());
          if (match && match[1].trim()) {
              return match[1].split(',').map(s => s.trim());
          } else {
              return [];
          }
      }

      function register(hook, fn, parameters) {
          parameters = parameters || extractParameters(fn);
          if (!registry[hook]) {
              registry[hook] = [];
          }
          registry[hook].push([fn, parameters]);
      }

      function call(hook, args) {
          args = args || {};
          var implementations = registry[hook] || [];
          var results = [];
          implementations.forEach(([fn, parameters]) => {
              /* Call with the correct arguments */
              var callWith = parameters.map(parameter => args[parameter]);
              var result = fn.apply(fn, callWith);
              if (result) {
                  results.push(result);
              }
          });
          return results;
      }
      return {
          register: register,
          _registry: registry,
          call: call
      };
  })();
  ```
  Usage example:
  ```javascript
  datasette.plugins.register('numbers', (a, b) => a + b)
  datasette.plugins.register('numbers', (a, b) => a * b)
  datasette.plugins.call('numbers', {a: 4, b: 6})
  /* Returns [10, 24] */
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752744195
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc0NDE5NQ==
created_at: 2020-12-30T20:12:26Z
updated_at: 2020-12-30T20:12:26Z
author_association: OWNER
body:
  This implementation doesn't have an equivalent of "hookspecs" which can identify if a registered plugin implementation matches a known signature. I should add that, it will provide a better developer experience if someone has a typo.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752744311
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc0NDMxMQ==
created_at: 2020-12-30T20:12:50Z
updated_at: 2020-12-30T20:13:02Z
author_association: OWNER
body:
  This could work to define a plugin hook:
  ```javascript
  datasette.plugins.define('numbers', ['a' ,'b'])
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752747169
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc0NzE2OQ==
created_at: 2020-12-30T20:24:07Z
updated_at: 2020-12-30T20:24:07Z
author_association: OWNER
body:
  This version adds `datasette.plugins.define()` plus extra validation of both `.register()` and `.call()`:
  ```javascript
  window.datasette = {};
  window.datasette.plugins = (function() {
      var registry = {};
      var definitions = {};

      function extractParameters(fn) {
          var match = /\((.*)\)/.exec(fn.toString());
          if (match && match[1].trim()) {
              return match[1].split(',').map(s => s.trim());
          } else {
              return [];
          }
      }

      function define(hook, parameters) {
          definitions[hook] = parameters || [];
      }

      function isSubSet(a, b) {
          return a.every(parameter => b.includes(parameter))
      }

      function register(hook, fn, parameters) {
          parameters = parameters || extractParameters(fn);
          if (!definitions[hook]) {
              throw new Error('"' + hook + '" is not a defined plugin hook');
          }
          if (!definitions[hook]) {
              throw new Error('"' + hook + '" is not a defined plugin hook');
          }
          /* Check parameters is a subset of definitions[hook] */
          var validParameters = definitions[hook];
          if (!isSubSet(parameters, validParameters)) {
              throw new Error('"' + hook + '" valid parameters are ' + JSON.stringify(validParameters));
          }
          if (!registry[hook]) {
              registry[hook] = [];
          }
          registry[hook].push([fn, parameters]);
      }

      function call(hook, args) {
          args = args || {};
          if (!definitions[hook]) {
              throw new Error('"' + hook + '" hook has not been defined');
          }
          if (!isSubSet(Object.keys(args), definitions[hook])) {
              throw new Error('"' + hook + '" valid arguments are ' + JSON.stringify(definitions[hook]));
          }
          
          var implementations = registry[hook] || [];
          var results = [];
          implementations.forEach(([fn, parameters]) => {
              /* Call with the correct arguments */
              var callWith = parameters.map(parameter => args[parameter]);
              var result = fn.apply(fn, callWith);
              if (result) {
                  results.push(result);
              }
          });
          return results;
      }
      return {
          define: define,
          register: register,
          _registry: registry,
          call: call
      };
  })();
  ```
  Usage:
  ```javascript
  datasette.plugins.define('numbers', ['a', 'b'])
  datasette.plugins.register('numbers', (a, b) => a + b)
  datasette.plugins.register('numbers', (a, b) => a * b)
  datasette.plugins.call('numbers', {a: 4, b: 6})
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752747999
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc0Nzk5OQ==
created_at: 2020-12-30T20:27:00Z
updated_at: 2020-12-30T20:27:00Z
author_association: OWNER
body:
  I need to decide how this code is going to be loaded. Putting it in a blocking `<script>` element in the head would work, but I'd rather not block loading of the rest of the page. Using a `<script async>` method would be nicer, but then I have to worry about plugins attempting to register themselves before the page has fully loaded.

  Running it through https://javascript-minifier.com/ produces this, which is 855 characters - so maybe I could inline that into the header of the page?

  `window.datasette={},window.datasette.plugins=function(){var r={},n={};function e(r,n){return r.every(r=>n.includes(r))}return{define:function(r,e){n[r]=e||[]},register:function(t,i,o){if(o=o||function(r){var n=/\((.*)\)/.exec(r.toString());return n&&n[1].trim()?n[1].split(",").map(r=>r.trim()):[]}(i),!n[t])throw new Error('"'+t+'" is not a defined plugin hook');if(!n[t])throw new Error('"'+t+'" is not a defined plugin hook');var a=n[t];if(!e(o,a))throw new Error('"'+t+'" valid parameters are '+JSON.stringify(a));r[t]||(r[t]=[]),r[t].push([i,o])},_registry:r,call:function(t,i){if(i=i||{},!n[t])throw new Error('"'+t+'" hook has not been defined');if(!e(Object.keys(i),n[t]))throw new Error('"'+t+'" valid arguments are '+JSON.stringify(n[t]));var o=r[t]||[],a=[];return o.forEach(([r,n])=>{var e=n.map(r=>i[r]),t=r.apply(r,e);t&&a.push(t)}),a}}}();`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752748496
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc0ODQ5Ng==
created_at: 2020-12-30T20:28:48Z
updated_at: 2020-12-30T20:28:48Z
author_association: OWNER
body:
  If I'm going to minify it I'll need to figure out a build step in Datasette itself so that I can easily work on that minified version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752749189
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc0OTE4OQ==
created_at: 2020-12-30T20:31:28Z
updated_at: 2020-12-30T20:31:28Z
author_association: OWNER
body:
  Using raw string exceptions, `throw '"' + hook + '" hook has not been defined';`, knocks it down to 795 characters.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752750551
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc1MDU1MQ==
created_at: 2020-12-30T20:36:38Z
updated_at: 2020-12-30T20:37:48Z
author_association: OWNER
body:
  This version minifies to 702 characters:
  ```javascript
  window.datasette = window.datasette || {};
  window.datasette.plugins = (() => {
      var registry = {};
      var definitions = {};
      var stringify = JSON.stringify;

      function extractParameters(fn) {
          var match = /\((.*)\)/.exec(fn.toString());
          if (match && match[1].trim()) {
              return match[1].split(',').map(s => s.trim());
          } else {
              return [];
          }
      }

      function isSubSet(a, b) {
          return a.every(parameter => b.includes(parameter))
      }

      return {
          _registry: registry,
          define: (hook, parameters) => {
              definitions[hook] = parameters || [];
          },
          register: (hook, fn, parameters) => {
              parameters = parameters || extractParameters(fn);
              if (!definitions[hook]) {
                  throw '"' + hook + '" is not a defined hook';
              }
              /* Check parameters is a subset of definitions[hook] */
              var validParameters = definitions[hook];
              if (!isSubSet(parameters, validParameters)) {
                  throw '"' + hook + '" valid args are ' + stringify(validParameters);
              }
              if (!registry[hook]) {
                  registry[hook] = [];
              }
              registry[hook].push([fn, parameters]);
          },
          
          call: (hook, args) => {
              args = args || {};
              if (!definitions[hook]) {
                  throw '"' + hook + '" hook is not defined';
              }
              if (!isSubSet(Object.keys(args), definitions[hook])) {
                  throw '"' + hook + '" valid args: ' + stringify(definitions[hook]);
              }
              
              var implementations = registry[hook] || [];
              var results = [];
              implementations.forEach(([fn, parameters]) => {
                  /* Call with the correct arguments */
                  var callWith = parameters.map(parameter => args[parameter]);
                  var result = fn.apply(fn, callWith);
                  if (result) {
                      results.push(result);
                  }
              });
              return results;
          }       
      };
  })();
  ```
  Or 701 characters using https://skalman.github.io/UglifyJS-online/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752751490
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc1MTQ5MA==
created_at: 2020-12-30T20:40:04Z
updated_at: 2020-12-30T21:34:22Z
author_association: OWNER
body:
  This one is 683 bytes with Uglify - I like how https://skalman.github.io/UglifyJS-online/ shows you the minified character count as you edit the script:
  ```javascript
  window.datasette = window.datasette || {};
  window.datasette.plugins = (() => {
      var registry = {};
      var definitions = {};
      var stringify = JSON.stringify;

      function extractParameters(fn) {
          var match = /\((.*)\)/.exec(fn.toString());
          if (match && match[1].trim()) {
              return match[1].split(',').map(s => s.trim());
          } else {
              return [];
          }
      }

      function isSubSet(a, b) {
          return a.every(parameter => b.includes(parameter))
      }

      return {
          _r: registry,
          define: (hook, parameters) => {
              definitions[hook] = parameters || [];
          },
          register: (hook, fn, parameters) => {
              parameters = parameters || extractParameters(fn);
              if (!definitions[hook]) {
                  throw 'Hook "' + hook + '" not defined';
              }
              /* Check parameters is a subset of definitions[hook] */
              var validParameters = definitions[hook];
              if (!isSubSet(parameters, validParameters)) {
                  throw '"' + hook + '" valid args: ' + stringify(validParameters);
              }
              if (!registry[hook]) {
                  registry[hook] = [];
              }
              registry[hook].push([fn, parameters]);
          },
          
          call: (hook, args) => {
              args = args || {};
              if (!definitions[hook]) {
                  throw '"' + hook + '" hook not defined';
              }
              if (!isSubSet(Object.keys(args), definitions[hook])) {
                  throw '"' + hook + '" valid args: ' + stringify(definitions[hook]);
              }
              
              var implementations = registry[hook] || [];
              var results = [];
              implementations.forEach(([fn, parameters]) => {
                  /* Call with the correct arguments */
                  var callWith = parameters.map(parameter => args[parameter]);
                  var result = fn.apply(fn, callWith);
                  if (result) {
                      results.push(result);
                  }
              });
              return results;
          }       
      };
  })();
  ```
  `window.datasette=window.datasette||{},window.datasette.plugins=(()=>{var t={},r={},e=JSON.stringify;function i(t,r){return t.every(t=>r.includes(t))}return{_r:t,define:(t,e)=>{r[t]=e||[]},register:(a,n,o)=>{if(o=o||function(t){var r=/\((.*)\)/.exec(t.toString());return r&&r[1].trim()?r[1].split(",").map(t=>t.trim()):[]}(n),!r[a])throw'Hook "'+a+'" not defined';var d=r[a];if(!i(o,d))throw'"'+a+'" valid args: '+e(d);t[a]||(t[a]=[]),t[a].push([n,o])},call:(a,n)=>{if(n=n||{},!r[a])throw'"'+a+'" hook not defined';if(!i(Object.keys(n),r[a]))throw'"'+a+'" valid args: '+e(r[a]);var o=t[a]||[],d=[];return o.forEach(([t,r])=>{var e=r.map(t=>n[t]),i=t.apply(t,e);i&&d.push(i)}),d}}})();`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1164#issuecomment-752756612
issue_url: https://api.github.com/repos/simonw/datasette/issues/1164
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc1NjYxMg==
created_at: 2020-12-30T20:59:54Z
updated_at: 2020-12-30T20:59:54Z
author_association: OWNER
body:
  I tried a few different pure-Python JavaScript minifying libraries and none of them produced results as good as https://www.npmjs.com/package/uglify-js for the plugin code I'm considering in #983.

  So I think I'll need to rely on a Node.js tool for this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1164#issuecomment-752757075
issue_url: https://api.github.com/repos/simonw/datasette/issues/1164
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc1NzA3NQ==
created_at: 2020-12-30T21:01:27Z
updated_at: 2020-12-30T21:01:27Z
author_association: OWNER
body:
  I don't want Datasette contributors to need a working Node.js install to run the tests or work on Datasette unless they are explicitly working on the JavaScript.

  I think I'm going to do this with a unit test that runs only if `upglify-js` is available on the path and confirms that the `*.min.js` version of each script in the repository correctly matches the results from running `uglify-js` against it.

  That way if anyone checks in a change to JavaScript but forgets to run the minifier the tests will fail in CI.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752758802
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc1ODgwMg==
created_at: 2020-12-30T21:07:33Z
updated_at: 2020-12-30T21:10:10Z
author_association: OWNER
body:
  Removing the `datasette.plugin.define()` method and associated error handling reduces the uglified version from 683 bytes to 380 bytes. I think the error checking is worth the extra 303 bytes per page load, even if it's only really needed for a better developer experience.
  ```javascript
  window.datasette = window.datasette || {};
  window.datasette.plugins = (() => {
      var registry = {};

      function extractParameters(fn) {
          var match = /\((.*)\)/.exec(fn.toString());
          if (match && match[1].trim()) {
              return match[1].split(',').map(s => s.trim());
          } else {
              return [];
          }
      }
      return {
          register: (hook, fn, parameters) => {
              parameters = parameters || extractParameters(fn);
              if (!registry[hook]) {
                  registry[hook] = [];
              }
              registry[hook].push([fn, parameters]);
          },
          
          call: (hook, args) => {
              args = args || {};
              var implementations = registry[hook] || [];
              var results = [];
              implementations.forEach(([fn, parameters]) => {
                  /* Call with the correct arguments */
                  var callWith = parameters.map(parameter => args[parameter]);
                  var result = fn.apply(fn, callWith);
                  if (result) {
                      results.push(result);
                  }
              });
              return results;
          }
      };
  })();
  ```
  `window.datasette=window.datasette||{},window.datasette.plugins=(()=>{var t={};return{register:(r,a,e)=>{e=e||function(t){var r=/\((.*)\)/.exec(t.toString());return r&&r[1].trim()?r[1].split(",").map(t=>t.trim()):[]}(a),t[r]||(t[r]=[]),t[r].push([a,e])},call:(r,a)=>{a=a||{};var e=t[r]||[],i=[];return e.forEach(([t,r])=>{var e=r.map(t=>a[t]),n=t.apply(t,e);n&&i.push(n)}),i}}})();`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752759885
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc1OTg4NQ==
created_at: 2020-12-30T21:11:52Z
updated_at: 2020-12-30T21:14:00Z
author_association: OWNER
body:
  262 bytes if I remove the parameter introspection code, instead requiring plugin authors to specify the arguments they take:
  ```javascript
  window.datasette = window.datasette || {};
  window.datasette.plugins = (() => {
      var registry = {};
      return {
          register: (hook, fn, parameters) => {
              if (!registry[hook]) {
                  registry[hook] = [];
              }
              registry[hook].push([fn, parameters]);
          },
          call: (hook, args) => {
              args = args || {};
              var results = [];
              (registry[hook] || []).forEach(([fn, parameters]) => {
                  /* Call with the correct arguments */
                  var callWith = parameters.map(parameter => args[parameter]);
                  var result = fn.apply(fn, callWith);
                  if (result) {
                      results.push(result);
                  }
              });
              return results;
          }
      };
  })();
  ```
  `window.datasette=window.datasette||{},window.datasette.plugins=(()=>{var a={};return{register:(t,e,r)=>{a[t]||(a[t]=[]),a[t].push([e,r])},call:(t,e)=>{e=e||{};var r=[];return(a[t]||[]).forEach(([a,t])=>{var s=t.map(a=>e[a]),d=a.apply(a,s);d&&r.push(d)}),r}}})();`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752760054
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc2MDA1NA==
created_at: 2020-12-30T21:12:36Z
updated_at: 2020-12-30T21:14:05Z
author_association: OWNER
body:
  I gotta admit that 262 byte version is pretty tempting, if it's going to end up in the `<head>` of every single page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752760815
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc2MDgxNQ==
created_at: 2020-12-30T21:15:41Z
updated_at: 2020-12-30T21:15:41Z
author_association: OWNER
body:
  I'm going to write a few example plugins and try them out against the longer and shorter versions of the script, to get a better feel for how useful the longer versions with the error handling and explicit definition actually are.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752767500
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc2NzUwMA==
created_at: 2020-12-30T21:42:07Z
updated_at: 2020-12-30T21:42:07Z
author_association: OWNER
body:
  Another option: have both "dev" and "production" versions of the plugin mechanism script. Make it easy to switch between the two. Build JavaScript unit tests that exercise the "production" APIs against the development version, and have extra tests that just work against the features in the development version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1164#issuecomment-752768652
issue_url: https://api.github.com/repos/simonw/datasette/issues/1164
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc2ODY1Mg==
created_at: 2020-12-30T21:46:29Z
updated_at: 2020-12-30T21:46:29Z
author_association: OWNER
body:
  Running https://skalman.github.io/UglifyJS-online/ against https://github.com/simonw/datasette/blob/0.53/datasette/static/table.js knocks it down from 7810 characters to 4643.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1164#issuecomment-752769452
issue_url: https://api.github.com/repos/simonw/datasette/issues/1164
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc2OTQ1Mg==
created_at: 2020-12-30T21:50:16Z
updated_at: 2020-12-30T21:50:16Z
author_association: OWNER
body:
  If I implement this I can automate the CodeMirror minification and remove the bit about running `uglify-js` against it from the documentation here: https://docs.datasette.io/en/0.53/contributing.html#upgrading-codemirror

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752770133
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc3MDEzMw==
created_at: 2020-12-30T21:53:45Z
updated_at: 2020-12-30T21:54:22Z
author_association: OWNER
body:
  FixMyStreet inlines some JavaScript, and it's always a good idea to copy what they're doing when it comes to web performance: https://github.com/mysociety/fixmystreet/blob/23e9564b58a86b783ce47f3c0bf837cbd4fe7282/templates/web/base/common_header_tags.html#L19-L25

  Note `var fixmystreet=fixmystreet||{};` which is shorter - https://twitter.com/dracos/status/1344399909794045954

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752770488
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc3MDQ4OA==
created_at: 2020-12-30T21:55:35Z
updated_at: 2020-12-30T21:58:26Z
author_association: OWNER
body:
  This one minifies to 241:
  ```javascript
  var datasette = datasette || {};
  datasette.plugins = (() => {
      var registry = {};
      return {
          register: (hook, fn, parameters) => {
              if (!registry[hook]) {
                  registry[hook] = [];
              }
              registry[hook].push([fn, parameters]);
          },
          call: (hook, args) => {
              args = args || {};
              var results = [];
              (registry[hook] || []).forEach(([fn, parameters]) => {
                  /* Call with the correct arguments */
                  var result = fn.apply(fn, parameters.map(parameter => args[parameter]));
                  if (result) {
                      results.push(result);
                  }
              });
              return results;
          }
      };
  })();
  ```
  `var datasette=datasette||{};datasette.plugins=(()=>{var a={};return{register:(t,r,e)=>{a[t]||(a[t]=[]),a[t].push([r,e])},call:(t,r)=>{r=r||{};var e=[];return(a[t]||[]).forEach(([a,t])=>{var s=a.apply(a,t.map(a=>r[a]));s&&e.push(s)}),e}}})();`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752773508
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc3MzUwOA==
created_at: 2020-12-30T22:10:08Z
updated_at: 2020-12-30T22:11:34Z
author_association: OWNER
body:
  https://twitter.com/dracos/status/1344402639476424706 points out that plugins returning 0 will be ignored.

  This should probably check for `result !== undefined` instead - knocks the size up to 250.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1165#issuecomment-752777744
issue_url: https://api.github.com/repos/simonw/datasette/issues/1165
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc3Nzc0NA==
created_at: 2020-12-30T22:30:24Z
updated_at: 2020-12-30T22:30:24Z
author_association: OWNER
body:
  https://www.valentinog.com/blog/jest/ was useful.

  I created a `static/__tests__` folder and added this file as `plugins.spec.js`:

  ```javascript
  const datasette = require("../plugins.js");

  describe("Datasette Plugins", () => {
    test("it should have datasette.plugins", () => {
      expect(!!datasette.plugins).toEqual(true);
    });
    test("registering a plugin should work", () => {
      datasette.plugins.register("numbers", (a, b) => a + b, ["a", "b"]);
      var result = datasette.plugins.call("numbers", { a: 1, b: 2 });
      expect(result).toEqual([3]);
      datasette.plugins.register("numbers", (a, b) => a * b, ["a", "b"]);
      var result2 = datasette.plugins.call("numbers", { a: 1, b: 2 });
      expect(result2).toEqual([3, 2]);
    });
  });
  ```

  In `static/plugins.js` I put this:
  ```javascript
  var datasette = datasette || {};
  datasette.plugins = (() => {
      var registry = {};
      return {
          register: (hook, fn, parameters) => {
              if (!registry[hook]) {
                  registry[hook] = [];
              }
              registry[hook].push([fn, parameters]);
          },
          call: (hook, args) => {
              args = args || {};
              var results = [];
              (registry[hook] || []).forEach(([fn, parameters]) => {
                  /* Call with the correct arguments */
                  var result = fn.apply(fn, parameters.map(parameter => args[parameter]));
                  if (result !== undefined) {
                      results.push(result);
                  }
              });
              return results;
          }
      };
  })();

  module.exports = datasette;
  ```
  Note the `module.exports` line at the end.

  Then inside `static/` I ran the following command:

  ```
  % npx jest -c '{}'
   PASS  __tests__/plugins.spec.js
    Datasette Plugins
      ✓ it should have datasette.plugins (3 ms)
      ✓ registering a plugin should work (1 ms)

  Test Suites: 1 passed, 1 total
  Tests:       2 passed, 2 total
  Snapshots:   0 total
  Time:        1.163 s
  Ran all test suites.
  ```
  The `-c {}` was necessary because I didn't have a Jest configuration or a `package.json`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1165#issuecomment-752779820
issue_url: https://api.github.com/repos/simonw/datasette/issues/1165
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjc3OTgyMA==
created_at: 2020-12-30T22:40:28Z
updated_at: 2020-12-30T22:40:28Z
author_association: OWNER
body:
  I don't know if Jest on the command-line is the right tool for this. It works for the `plugins.js` script but I'm increasingly going to want to start adding tests for browser JavaScript features - like the https://github.com/simonw/datasette/blob/0.53/datasette/static/table.js script - which will need to run in a browser.

  So maybe I should just find a browser testing solution and figure out how to run that under CI in GitHub Actions. Maybe https://www.cypress.io/ ?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1165#issuecomment-752828851
issue_url: https://api.github.com/repos/simonw/datasette/issues/1165
node_id: MDEyOklzc3VlQ29tbWVudDc1MjgyODg1MQ==
created_at: 2020-12-31T03:19:38Z
updated_at: 2020-12-31T03:19:38Z
author_association: OWNER
body:
  I got Cypress working! I added the `datasette.plugins` code to the table template and ran a test called `plugins.spec.js` using the following:
  ```javascript
  context('datasette.plugins API', () => {
      beforeEach(() => {
        cy.visit('/fixtures/compound_three_primary_keys')
      });
      it('should exist', () => {
          let datasette;
          cy.window().then(win => {
              datasette = win.datasette;
          }).then(() => {
              expect(datasette).to.exist;
              expect(datasette.plugins).to.exist;
          });
      });
      it('should register and execute plugins', () => {
          let datasette;
          cy.window().then(win => {
              datasette = win.datasette;
          }).then(() => {
              expect(datasette.plugins.call('numbers')).to.deep.equal([]);
              // Register a plugin
              datasette.plugins.register("numbers", (a, b) => a + b, ['a', 'b']);
              var result = datasette.plugins.call("numbers", {a: 1, b: 2});
              expect(result).to.deep.equal([3]);
              // Second plugin
              datasette.plugins.register("numbers", (a, b) => a * b, ['a', 'b']);
              var result2 = datasette.plugins.call("numbers", {a: 1, b: 2});
              expect(result2).to.deep.equal([3, 2]);
          });
      });
  });
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1165#issuecomment-752839433
issue_url: https://api.github.com/repos/simonw/datasette/issues/1165
node_id: MDEyOklzc3VlQ29tbWVudDc1MjgzOTQzMw==
created_at: 2020-12-31T04:29:40Z
updated_at: 2020-12-31T04:29:40Z
author_association: OWNER
body:
  Important to absorb the slightly bizarre assertion syntax from Chai - docs here https://www.chaijs.com/api/bdd/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1165#issuecomment-752846267
issue_url: https://api.github.com/repos/simonw/datasette/issues/1165
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjg0NjI2Nw==
created_at: 2020-12-31T05:10:41Z
updated_at: 2020-12-31T05:13:14Z
author_association: OWNER
body:
  https://github.com/PostHog/posthog/tree/master/cypress/integration has some useful examples, linked from this article: https://posthog.com/blog/cypress-end-to-end-tests

  Also useful: their workflow https://github.com/PostHog/posthog/blob/master/.github/workflows/e2e.yml

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752882797
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjg4Mjc5Nw==
created_at: 2020-12-31T08:07:59Z
updated_at: 2020-12-31T15:04:32Z
author_association: NONE
body:
  If you're using arrow functions, you can presumably use default parameters, not much difference in support. That would save you 9 bytes. But OTOH you need `"use strict";` to use arrow functions etc, and that's 13 bytes.

  Your latest 250-byte one, with use strict, gzips to 199 bytes. The following might be 292 bytes, but compresses to 204, basically the same, and works in any browser (well, IE9+) at all:

  `var datasette=datasette||{};datasette.plugins=function(){var d={};return{register:function(b,c,e){d[b]||(d[b]=[]);d[b].push([c,e])},call:function(b,c){c=c||{};var e=[];(d[b]||[]).forEach(function(a){a=a[0].apply(a[0],a[1].map(function(a){return c[a]}));void 0!==a&&e.push(a)});return e}}}();`

  Source for that is below; I replaced the [fn,parameters] because closure-compiler includes a polyfill for that, and I ran `closure-compiler --language_out ECMASCRIPT3`:

  ```js
  var datasette = datasette || {};
  datasette.plugins = (() => {
      var registry = {};
      return {
          register: (hook, fn, parameters) => {
              if (!registry[hook]) {
                  registry[hook] = [];
              }
              registry[hook].push([fn, parameters]);
          },
          call: (hook, args) => {
              args = args || {};
              var results = [];
              (registry[hook] || []).forEach((data) => {
                  /* Call with the correct arguments */
                  var result = data[0].apply(data[0], data[1].map(parameter => args[parameter]));
                  if (result !== undefined) {
                      results.push(result);
                  }
              });
              return results;
          }
      };
  })();
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-752888552
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1Mjg4ODU1Mg==
created_at: 2020-12-31T08:33:11Z
updated_at: 2020-12-31T08:34:27Z
author_association: NONE
body:
  If you could say that all hook functions had to accept one options parameter (and could use object destructuring if they wished to only see a subset), you could have this, which minifies (to all-browser-JS) to 200 bytes, gzips to 146, and works practically the same:

  ```js
  var datasette = datasette || {};
  datasette.plugins = (() => {
      var registry = {};
      return {
          register: (hook, fn) => {
              registry[hook] = registry[hook] || [];
              registry[hook].push(fn);
          },
          call: (hook, args) => {
              var results = (registry[hook] || []).map(fn => fn(args||{}));
              return results;
          }
      };
  })();
  ```

  `var datasette=datasette||{};datasette.plugins=function(){var b={};return{register:function(a,c){b[a]=b[a]||[];b[a].push(c)},call:function(a,c){return(b[a]||[]).map(function(a){return a(c||{})})}}}();`

  Called the same, definitions tiny bit different:

  ```js
  datasette.plugins.register('numbers', ({a, b}) => a + b)
  datasette.plugins.register('numbers', o => o.a * o.b)
  datasette.plugins.call('numbers', {a: 4, b: 6})
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1165#issuecomment-753033121
issue_url: https://api.github.com/repos/simonw/datasette/issues/1165
node_id: MDEyOklzc3VlQ29tbWVudDc1MzAzMzEyMQ==
created_at: 2020-12-31T19:33:47Z
updated_at: 2020-12-31T19:33:47Z
author_association: NONE
body:
  Sorry to go on about it, but it's my only example ;) And thought it might be of interest/use. Here is FixMyStreet's Cypress workflow https://github.com/mysociety/fixmystreet/blob/master/.github/workflows/cypress.yml with the master script that sets up server etc at https://github.com/mysociety/fixmystreet/blob/master/bin/browser-tests (that has features such as working inside/outside Vagrant, and can do JS code coverage) and then the tests are at https://github.com/mysociety/fixmystreet/tree/master/.cypress/cypress/integration

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1166#issuecomment-753193475
issue_url: https://api.github.com/repos/simonw/datasette/issues/1166
node_id: MDEyOklzc3VlQ29tbWVudDc1MzE5MzQ3NQ==
created_at: 2020-12-31T21:33:00Z
updated_at: 2020-12-31T21:33:00Z
author_association: OWNER
body:
  I want a CI check that confirms that files conform to prettier - but only `datasette/static/*.js` files that are not already  minified.

  This seems to do the job:

      npx prettier --check 'datasette/static/*[!.min].js'

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1166#issuecomment-753195905
issue_url: https://api.github.com/repos/simonw/datasette/issues/1166
node_id: MDEyOklzc3VlQ29tbWVudDc1MzE5NTkwNQ==
created_at: 2020-12-31T21:34:46Z
updated_at: 2020-12-31T21:34:46Z
author_association: OWNER
body:
  This action looks good - tag 3.2 is equivalent to this commit hash: https://github.com/creyD/prettier_action/tree/bb361e2979cff283ca7684908deac8f95400e779

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1166#issuecomment-753197957
issue_url: https://api.github.com/repos/simonw/datasette/issues/1166
node_id: MDEyOklzc3VlQ29tbWVudDc1MzE5Nzk1Nw==
created_at: 2020-12-31T21:36:14Z
updated_at: 2020-12-31T21:36:14Z
author_association: OWNER
body:
  Maybe not that action actually - I wanted to use a pre-built action to avoid installing Prettier every time, but that's what it seems to do: https://github.com/creyD/prettier_action/blob/bb361e2979cff283ca7684908deac8f95400e779/entrypoint.sh#L28-L37

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1166#issuecomment-753200580
issue_url: https://api.github.com/repos/simonw/datasette/issues/1166
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIwMDU4MA==
created_at: 2020-12-31T21:38:06Z
updated_at: 2020-12-31T21:38:06Z
author_association: OWNER
body:
  I think this should work:
  ```
  - uses: actions/cache@v2
    with:
      path: ~/.npm
      key: ${{ runner.os }}-node-${{ hashFiles('**/prettier.yml' }}
  ```
  I'll use the `prettier.yml` workflow that I'm about to create as the cache key for the NPM cache.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1166#issuecomment-753209192
issue_url: https://api.github.com/repos/simonw/datasette/issues/1166
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIwOTE5Mg==
created_at: 2020-12-31T21:44:22Z
updated_at: 2020-12-31T21:44:22Z
author_association: OWNER
body:
  Tests passed in https://github.com/simonw/datasette/runs/1631677726?check_suite_focus=true

  I'm going to try submitting a pull request with badly formatted JavaScript to see if it gets caught.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1166#issuecomment-753211535
issue_url: https://api.github.com/repos/simonw/datasette/issues/1166
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIxMTUzNQ==
created_at: 2020-12-31T21:46:04Z
updated_at: 2020-12-31T21:46:04Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/runs/1631682372?check_suite_focus=true failed!

  <img width="693" alt="Trying_out_bad_formatting__refs__1166_·_simonw_datasette_8087091" src="https://user-images.githubusercontent.com/9599/103426449-841e5c00-4b6e-11eb-95e6-26432a7b27dd.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753215545
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIxNTU0NQ==
created_at: 2020-12-31T22:05:41Z
updated_at: 2020-12-31T22:05:41Z
author_association: OWNER
body:
  Using object destructuring like that is a great idea. I'm going to play with your version  - it's delightfully succinct.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753215761
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIxNTc2MQ==
created_at: 2020-12-31T22:07:31Z
updated_at: 2020-12-31T22:07:31Z
author_association: OWNER
body:
  I think I need to keep the mechanism whereby a plugin can return `undefined` in order to indicate that it has nothing to say for that specific item - that's borrowed from Pluggy and I've used it a bunch in my Python plugins. That makes the code a bit longer.

  I'll write some example plugins to help me decide if the filtering-out-of-undefined mechanism is needed or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/987#issuecomment-753217127
issue_url: https://api.github.com/repos/simonw/datasette/issues/987
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIxNzEyNw==
created_at: 2020-12-31T22:16:46Z
updated_at: 2020-12-31T22:16:46Z
author_association: OWNER
body:
  I'm going to use `class="plugin-content-pre-table"` rather than `id=` - just because I still want to be able to display all of this stuff on the single https://latest.datasette.io/-/patterns page so duplicate IDs are best avoided.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753217714
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIxNzcxNA==
created_at: 2020-12-31T22:21:33Z
updated_at: 2020-12-31T22:21:33Z
author_association: OWNER
body:
  Eventually I'd like to provide a whole bunch of other `datasette.X` utility functions that plugins can use - things like `datasette.addTabbedContentPane()` or similar.

  But I don't want to inline those into the page.

  So... I think the basic plugin system remains inline - maybe from an inlined file called `plugins-bootstrap.js`. Then a separate `plugins.js` contains the rest of the API functionality.

  If a plugin wants to take advantage of those APIs, maybe it registers itself using `datasette.plugins.register('load', () => ...)` - that `load` hook can then be fired once the bulkier plugin code has been loaded.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753217917
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIxNzkxNw==
created_at: 2020-12-31T22:23:29Z
updated_at: 2020-12-31T22:23:36Z
author_association: OWNER
body:
  If I'm going to do that, it would be good if subsequent plugins that register against the `load` event are executed straight away. That's a bit of a weird edge-case in plugin world - it would involve the bulkier code that gets loaded redefining how `datasette.plugins.register` works to special-case the `'load'` hook.

  Maybe the tiny bootstrap code could define a `datasette.plugins.onload(callbackFunction)` method which gets upgraded later into something that fires straight away? Would add more bytes though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753218817
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIxODgxNw==
created_at: 2020-12-31T22:32:25Z
updated_at: 2020-12-31T22:32:25Z
author_association: NONE
body:
  Amazing work! And you've put in far more work than I'd expect to reduce the payload (which is admirable).

  So, to add a plugin with the current design, it goes in (a) the template or (b) a bookmarklet, right?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753219407
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIxOTQwNw==
created_at: 2020-12-31T22:38:45Z
updated_at: 2020-12-31T22:39:10Z
author_association: OWNER
body:
  You'll be able to add JavaScript plugins using a bunch of different mechanisms:

  - In a custom template, dropping the code in to a `<script>` block
  - A bookmarklet that injects an extra script (I'm really excited to try this out)
  - A separate `script.js` file that's loaded into Datasette using the `"extra_js_urls"` metadata option, documented here: https://docs.datasette.io/en/stable/custom_templates.html#custom-css-and-javascript
  - A plugin you can install, like `datasette-vega` or `datasette-cluster-map` - since plugins can bundle their own script files that then get loaded on pages via this hook: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-js-urls-template-database-table-columns-view-name-request-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753219521
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIxOTUyMQ==
created_at: 2020-12-31T22:39:52Z
updated_at: 2020-12-31T22:39:52Z
author_association: OWNER
body:
  For inlining the `plugins.min.js` file into the Jinja templates I could use the trick described here: https://stackoverflow.com/a/41404611 - which adds a `{{ include_file('file.txt') }}` function to Jinja.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1164#issuecomment-753220412
issue_url: https://api.github.com/repos/simonw/datasette/issues/1164
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIyMDQxMg==
created_at: 2020-12-31T22:47:36Z
updated_at: 2020-12-31T22:47:36Z
author_association: OWNER
body:
  I'm trying to minify `table.js` and I ran into a problem:

      Uglification failed. Unexpected character '`'

  It turns out `uglify-js` doesn't support ES6 syntax!

  But `uglify-es` does:

      npm install uglify-es

  Annoyingly it looks like `uglify-es` uses the same CLI command, `uglifyjs`. So after installing it this seemed to work:

      npx uglifyjs table.js --source-map -o table.min.js

  I really don't like how `npx uglifyjs` could mean different things depending on which package was installed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1164#issuecomment-753220665
issue_url: https://api.github.com/repos/simonw/datasette/issues/1164
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIyMDY2NQ==
created_at: 2020-12-31T22:49:36Z
updated_at: 2020-12-31T22:49:36Z
author_association: OWNER
body:
  I started with a 7K `table.js` file.

  `npx uglifyjs table.js --source-map -o table.min.js` gave me a 5.6K `table.min.js` file. 

  `npx uglifyjs table.js --source-map -o table.min.js --compress --mangle` gave me 4.5K.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1164#issuecomment-753221362
issue_url: https://api.github.com/repos/simonw/datasette/issues/1164
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIyMTM2Mg==
created_at: 2020-12-31T22:55:57Z
updated_at: 2020-12-31T22:55:57Z
author_association: OWNER
body:
  I had to add this as the first line in `table.min.js` for the source mapping to work:
  ```
  //# sourceMappingURL=/-/static/table.min.js.map
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753221646
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIyMTY0Ng==
created_at: 2020-12-31T22:58:47Z
updated_at: 2020-12-31T22:58:47Z
author_association: OWNER
body:
  https://github.com/mishoo/UglifyJS/issues/1905#issuecomment-300485490 says:

  > `sourceMappingURL` aren't added by default in `3.x` due to one of the feature requests not to - some users are putting them within HTTP response headers instead.
  > 
  > So the command line for that would be:
  > 
  > ```js
  > $ uglifyjs main.js -cmo main.min.js --source-map url=main.min.js.map
  > ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1166#issuecomment-753224351
issue_url: https://api.github.com/repos/simonw/datasette/issues/1166
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIyNDM1MQ==
created_at: 2020-12-31T23:23:29Z
updated_at: 2020-12-31T23:23:29Z
author_association: OWNER
body:
  I should configure the action to only run if changes have been made within the `datasette/static` directory.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753224999
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzIyNDk5OQ==
created_at: 2020-12-31T23:29:36Z
updated_at: 2020-12-31T23:29:36Z
author_association: NONE
body:
  I have yet to build Datasette plugin and am unfamiliar with Pluggy. Since browsers have event handling builtin Datasette could communicate with plugins through it. Handlers register as listeners for custom Datasette events and Datasette's JS can then trigger said events.

  I was also wondering if you had looked at Javascript Modules for JS plugins? With services like Skypack (https://www.skypack.dev) NPM libraries can be loaded directly into browser, no build step needed. Same goes for local JS if you adhere to ES Module spec. 

  If minification is required then tools such as Snowpack (https://www.snowpack.dev) could fit better. It uses https://github.com/evanw/esbuild for bundling and minification.

  On plugins you'd simply:

  ```javascript
  import {register} from '/assets/js/datasette'
  register.on({'click' : my_func})
  ```

  In Datasette HTML pages' head you'd merely import these files as modules one by one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753366024
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzM2NjAyNA==
created_at: 2021-01-01T18:48:34Z
updated_at: 2021-01-01T18:48:34Z
author_association: OWNER
body:
  Also: in #188 I proposed bundling metadata in the SQLite database itself alongside the data. This is a great way of ensuring metadata travels with the data when it is downloaded as a SQLite `.db` file. But how would that play with the idea of an in-memory `_metadata` table? Could that table perhaps offer views that join data across multiple attached physical databases?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753388809
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzM4ODgwOQ==
created_at: 2021-01-01T21:47:51Z
updated_at: 2021-01-01T21:47:51Z
author_association: OWNER
body:
  A database that exposes metadata will have the same restriction as the new `_internal` database that exposes columns and tables, in that it needs to take permissions into account. A user should not be able to view metadata for tables that they are not able to see.

  As such, I'd rather bundle any metadata tables into the existing `_internal` database so I don't have to solve that permissions problem in two places.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753389477
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzM4OTQ3Nw==
created_at: 2021-01-01T21:49:57Z
updated_at: 2021-01-01T21:49:57Z
author_association: OWNER
body:
  What if metadata was stored in a JSON text column in the existing `_internal` tables? This would allow for users to invent additional metadata fields in the future beyond the current `license`, `license_url` etc fields - without needing a schema change.

  The downside of JSON columns generally is that they're harder to run indexed queries against. For metadata I don't think that matters - even with 10,000 tables each with their own metadata a SQL query asking for e.g. "everything that has Apache 2 as the license" would return in just a few ms.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753389938
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzM4OTkzOA==
created_at: 2021-01-01T21:54:15Z
updated_at: 2021-01-01T21:54:15Z
author_association: OWNER
body:
  So what if the `databases`, `tables` and `columns` tables in `_internal` each grew a new `metadata` text column?

  These columns could be populated by Datasette on startup through reading the `metadata.json` file. But how would plugins interact with them?

reactions:
[cell repeats an earlier one in this file: 112 characters]


