# issues

node_id: MDU6SXNzdWU1OTQxNjg3NTg=
title: extra_template_vars() sending wrong view_name for index
state: closed
created_at: 2020-04-04T23:57:09Z
updated_at: 2020-04-05T20:04:08Z
closed_at: 2020-04-05T18:28:48Z
author_association: OWNER
type: issue
state_reason: completed
body:
  See https://github.com/simonw/museums/issues/20#issuecomment-609103663 - at some point between 286ed286b68793532c2a38436a08343b45cfbc91 and current master (e0e7a0facfc935a835cd73c720bc46661462f0b1 today) a bug was introduced where the `extra_template_vars(request, view_name)` plugin hook started being passed `None` instead of `index` for the `view_name` parameter on the site index page.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/716/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1OTQxODk1Mjc=
title: See if I can get Datasette working on Zeit Now v2
state: closed
created_at: 2020-04-05T00:56:48Z
updated_at: 2020-04-06T22:47:22Z
closed_at: 2020-04-06T22:47:21Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I thought this was impossible because AWS Lambda doesn't ship the `sqlite3` standard library module... but apparenttly that's not the case on Now v2 any more!

  https://now-2-python-versions-ks69olzpi.now.sh/api

  ```
   _________________________________________________________________________________________________________________________________________________________________ 
  / Hello from Python from a ZEIT Now Serverless Function! Version is 3.6.10 (default, Mar 10 2020, 22:54:43)                                                       \
  \ [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)], sqlite3 module = <module 'sqlite3' from '/var/lang/lib/python3.6/sqlite3/__init__.py'>, sqlite3 version = [('3.7.17',)] /
   ----------------------------------------------------------------------------------------------------------------------------------------------------------------- 
       \   ^__^
        \  (oo)\_______
           (__)\       )\/\
             ||----w |
             ||     ||
  ```
  That's from shipping this code as `api/index.py`:
  ```python
  from http.server import BaseHTTPRequestHandler
  from cowpy import cow
  import sys


  try:
      import sqlite3
  except ImportError:
      sqlite3 = None


  class handler(BaseHTTPRequestHandler):
      def do_GET(self):
          self.send_response(200)
          self.send_header("Content-type", "text/plain")
          self.end_headers()
          message = cow.Cowacter().milk(
              "Hello from Python from a ZEIT Now Serverless Function! Version is {}, sqlite3 module = {}, sqlite3 version = {}".format(
                  sys.version, sqlite3, sqlite3.connect(":memory:").execute("select sqlite_version()").fetchall()
              )
          )
          self.wfile.write(message.encode())
          return
  ```
  Now v2 supports ASGI so this might be possible without too much work: https://zeit.co/docs/runtimes#advanced-usage/advanced-python-usage/asynchronous-server-gateway-interface

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/717/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1OTQyMzcwMTU=
title: Plugin idea: datasette-redirects
state: open
created_at: 2020-04-05T03:41:38Z
updated_at: 2023-08-30T22:17:31Z
author_association: OWNER
type: issue
state_reason: reopened
body:
  I just had to write a one-off custom plugin to redirect niche-musems.com to www.niche-museums.com (https://github.com/simonw/museums/issues/21) - it would be great if this kind of thing could be handled by a configurable plugin.

  https://github.com/simonw/museums/blob/6b1faf00c463b2228860d4d62d104b11935e01b1/plugins/redirect_www.py

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/718/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mzk5MTY2NDMz
title: asgi: check raw_path is not None
state: closed
created_at: 2020-04-05T16:53:58Z
updated_at: 2020-05-04T17:14:26Z
closed_at: 2020-05-04T17:14:26Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/719
type: pull
body:
  The ASGI spec
  (https://asgi.readthedocs.io/en/latest/specs/www.html#http) seems to imply that `None` is a valid value, so we need to check the value itself, not just whether the key is present.

  In particular, the [mangum](https://github.com/erm/mangum) adapter passes `None` for this key's value. This change permits mangum to be used to front datasette in Amazon API Gateway + AWS Lambda deployments.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/719/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDAwNTc4OTc5
title: Update beautifulsoup4 requirement from ~=4.8.1 to >=4.8.1,<4.10.0
state: closed
created_at: 2020-04-08T01:24:38Z
updated_at: 2020-05-04T17:14:51Z
closed_at: 2020-05-04T17:14:46Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/720
type: pull
body:
  Updates the requirements on [beautifulsoup4](http://www.crummy.com/software/BeautifulSoup/bs4/) to permit the latest version.


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  **Note:** This repo was added to Dependabot recently, so you'll receive a maximum of 5 PRs for your first few update runs. Once an update run creates fewer than 5 PRs we'll remove that limit.

  You can always request more updates by clicking `Bump now` in your [Dependabot dashboard](https://app.dependabot.com).

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/720/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDAwNTc5MDc3
title: Update pytest requirement from ~=5.2.2 to >=5.2.2,<5.5.0
state: closed
created_at: 2020-04-08T01:25:04Z
updated_at: 2020-05-04T17:13:49Z
closed_at: 2020-05-04T17:13:41Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/721
type: pull
body:
  Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/releases">pytest's releases</a>.</em></p>
  <blockquote>
  <h2>5.4.1</h2>
  <h1>pytest 5.4.1 (2020-03-13)</h1>
  <h2>Bug Fixes</h2>
  <ul>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6909">#6909</a>: Revert the change introduced by <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/pull/6330">#6330</a>, which required all arguments to <code>@pytest.mark.parametrize</code> to be explicitly defined in the function signature.</p>
  <p>The intention of the original change was to remove what was expected to be an unintended/surprising behavior, but it turns out many people relied on it, so the restriction has been reverted.</p>
  </li>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6910">#6910</a>: Fix crash when plugins return an unknown stats while using the <code>--reportlog</code> option.</p>
  </li>
  </ul>
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst">pytest's changelog</a>.</em></p>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pytest-dev/pytest/commit/3d0f3baa2bb89257dfff25ae6ebabd565287240e"><code>3d0f3ba</code></a> Preparing release version 5.4.1</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/b9e2cd0a81e043bdb409e7078db9af4c4d38da54"><code>b9e2cd0</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6914">#6914</a> from nicoddemus/revert-6330</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/a84fcbf5b2fc007a1630a1ddd60b0a2438b61642"><code>a84fcbf</code></a> Revert &quot;[parametrize] enforce explicit argnames declaration (<a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6330">#6330</a>)&quot;</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/59c1bfada7de3b3d239f5d7cfff04ce460c9aecd"><code>59c1bfa</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6913">#6913</a> from nicoddemus/backport-6910</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/3267f64724085c8f4be4c8d6eee6726cd7b20c14"><code>3267f64</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6910">#6910</a> from nicoddemus/resultlog-logreport</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/c9fd1bdbd6d25917d33b27511d8ebc2a65538899"><code>c9fd1bd</code></a> Preparing release version 5.4.0</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/93aa988e0152392c233acac1e24bf6db8a2772c7"><code>93aa988</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6901">#6901</a> from RonnyPfannschmidt/regendoc-fix-simple</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/7996724f23760275a9cba9daad5af99c6da91f2c"><code>7996724</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6902">#6902</a> from RoyalTS/filterwarnings-docfix</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/90ee8a75995fb69bcfaae258afdb6694b3aa620d"><code>90ee8a7</code></a> docfix</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/378a75ddf602278fb2049a743b2ca3237722603d"><code>378a75d</code></a> run and fix tox -e regen to prepare 5.4</li>
  <li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest/compare/5.2.2...5.4.1">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  **Note:** This repo was added to Dependabot recently, so you'll receive a maximum of 5 PRs for your first few update runs. Once an update run creates fewer than 5 PRs we'll remove that limit.

  You can always request more updates by clicking `Bump now` in your [Dependabot dashboard](https://app.dependabot.com).

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/721/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDAwNTc5MTM2
title: Update jinja2 requirement from ~=2.10.3 to >=2.10.3,<2.12.0
state: closed
created_at: 2020-04-08T01:25:24Z
updated_at: 2020-05-04T17:13:26Z
closed_at: 2020-05-04T17:13:16Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/722
type: pull
body:
  Updates the requirements on [jinja2](https://github.com/pallets/jinja) to permit the latest version.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/pallets/jinja/releases">jinja2's releases</a>.</em></p>
  <blockquote>
  <h2>2.11.1</h2>
  <p>This fixes an issue in async environment when indexing the result of an attribute lookup, like <code>{{ data.items[1:] }}</code>.</p>
  <ul>
  <li>Changes: <a href="https://jinja.palletsprojects.com/en/2.11.x/changelog/#version-2-11-1">https://jinja.palletsprojects.com/en/2.11.x/changelog/#version-2-11-1</a></li>
  </ul>
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/pallets/jinja/blob/master/CHANGES.rst">jinja2's changelog</a>.</em></p>
  <blockquote>
  <h1>Version 2.11.1</h1>
  <p>Released 2020-01-30</p>
  <ul>
  <li>Fix a bug that prevented looking up a key after an attribute (<code>{{ data.items[1:] }}</code>) in an async template. 1141</li>
  </ul>
  <h1>Version 2.11.0</h1>
  <p>Released 2020-01-27</p>
  <ul>
  <li>Drop support for Python 2.6, 3.3, and 3.4. This will be the last version to support Python 2.7 and 3.5.</li>
  <li>Added a new <code>ChainableUndefined</code> class to support getitem and getattr on an undefined object. 977</li>
  <li>Allow <code>{%+</code> syntax (with NOP behavior) when <code>lstrip_blocks</code> is disabled. 748</li>
  <li>Added a <code>default</code> parameter for the <code>map</code> filter. 557</li>
  <li>Exclude environment globals from meta.find_undeclared_variables. 931</li>
  <li>Float literals can be written with scientific notation, like 2.56e-3. 912, 922</li>
  <li>Int and float literals can be written with the '_' separator for legibility, like 12_345. 923</li>
  <li>Fix a bug causing deadlocks in <code>LRUCache.setdefault</code>. 1000</li>
  <li>The <code>trim</code> filter takes an optional string of characters to trim. 828</li>
  <li>A new <code>jinja2.ext.debug</code> extension adds a <code>{% debug %}</code> tag to quickly dump the current context and available filters and tests. 174, 798, 983</li>
  <li>Lexing templates with large amounts of whitespace is much faster. 857, 858</li>
  <li>Parentheses around comparisons are preserved, so <code>{{ 2 * (3 &lt; 5) }}</code> outputs &quot;2&quot; instead of &quot;False&quot;. 755, 938</li>
  <li>Add new <code>boolean</code>, <code>false</code>, <code>true</code>, <code>integer</code> and <code>float</code> tests. 824</li>
  <li>The environment's <code>finalize</code> function is only applied to the output of expressions (constant or not), not static template data. 63</li>
  <li>When providing multiple paths to <code>FileSystemLoader</code>, a template can have the same name as a directory. 821</li>
  <li>Always return Undefined when omitting the <code>else</code> clause in a <code>{{ 'foo' if bar }}</code> expression, regardless of the environment's <code>undefined</code> class. Omitting the <code>else</code> clause is a valid shortcut and should not raise an error when using StrictUndefined. 710, 1079</li>
  <li>Fix behavior of <code>loop</code> control variables such as <code>length</code> and <code>revindex0</code> when looping over a generator. 459, 751, 794, 993</li>
  <li>Async support is only loaded the first time an environment enables it, in order to avoid a slow initial import. 765</li>
  <li>In async environments, the <code>|map</code> filter will await the filter call if needed. 913</li>
  <li>In for loops that access <code>loop</code> attributes, the iterator is not advanced ahead of the current iteration unless <code>length</code>, <code>revindex</code>, <code>nextitem</code>, or <code>last</code> are accessed. This makes it less likely to break <code>groupby</code> results. 555, 1101</li>
  <li>In async environments, the <code>loop</code> attributes <code>length</code> and <code>revindex</code> work for async iterators. 1101</li>
  <li>In async environments, values from attribute/property access will be awaited if needed. 1101</li>
  <li>~loader.PackageLoader doesn't depend on setuptools or pkg_resources. 970</li>
  <li><code>PackageLoader</code> has limited support for 420 namespace packages. 1097</li>
  <li>Support os.PathLike objects in ~loader.FileSystemLoader and ~loader.ModuleLoader. 870</li>
  <li>~nativetypes.NativeTemplate correctly handles quotes between expressions. <code>&quot;'{{ a }}', '{{ b }}'&quot;</code> renders as the tuple <code>('1', '2')</code> rather than the string <code>'1, 2'</code>. 1020</li>
  <li>Creating a ~nativetypes.NativeTemplate directly creates a ~nativetypes.NativeEnvironment instead of a default Environment. 1091</li>
  <li>After calling <code>LRUCache.copy()</code>, the copy's queue methods point to the correct queue. 843</li>
  <li>Compiling templates always writes UTF-8 instead of defaulting to the system encoding. 889</li>
  <li><code>|wordwrap</code> filter treats existing newlines as separate paragraphs to be wrapped individually, rather than creating short intermediate lines. 175</li>
  <li>Add <code>break_on_hyphens</code> parameter to <code>|wordwrap</code> filter. 550</li>
  <li>Cython compiled functions decorated as context functions will be passed the context. 1108</li>
  <li>When chained comparisons of constants are evaluated at compile time, the result follows Python's behavior of returning <code>False</code> if any comparison returns <code>False</code>, rather than only the last one. 1102</li>
  <li>Tracebacks for exceptions in templates show the correct line numbers and source for Python &gt;= 3.7. 1104</li>
  <li>Tracebacks for template syntax errors in Python 3 no longer show internal compiler frames. 763</li>
  <li>Add a <code>DerivedContextReference</code> node that can be used by extensions to get the current context and local variables such as <code>loop</code>. 860</li>
  <li>Constant folding during compilation is applied to some node types that were previously overlooked. 733</li>
  <li><code>TemplateSyntaxError.source</code> is not empty when raised from an included template. 457</li>
  </ul>
  </tr></table> ... (truncated)
  </blockquote>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pallets/jinja/commit/b85283e8464d56ef129fe8969f9abbb77be31be0"><code>b85283e</code></a> release version 2.11.1</li>
  <li><a href="https://github.com/pallets/jinja/commit/3d5bfc6109bcc682f5266b118180e8b5fb775ae9"><code>3d5bfc6</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/jinja/issues/1143">#1143</a> from pallets/bugfix/attribute-access</li>
  <li><a href="https://github.com/pallets/jinja/commit/d61c1ea70bfc98eb6a0dc690e2f0a1527fd80f10"><code>d61c1ea</code></a> add changelog</li>
  <li><a href="https://github.com/pallets/jinja/commit/15d7e615f4dc81424ad15b852ca69d1d98129139"><code>15d7e61</code></a> Added regression test for slicing of attributes</li>
  <li><a href="https://github.com/pallets/jinja/commit/05dee9b87023930f8131cbc0815b56a23e973c3d"><code>05dee9b</code></a> Fix attribute access in async code. Fixes <a href="https://github-redirect.dependabot.com/pallets/jinja/issues/1141">#1141</a></li>
  <li><a href="https://github.com/pallets/jinja/commit/bbdafe33ce9f47e3cbfb9415619e354349f11243"><code>bbdafe3</code></a> release version 2.11.0</li>
  <li><a href="https://github.com/pallets/jinja/commit/9ff27f64a92171c912e403bd66c392ec1a70533a"><code>9ff27f6</code></a> add python 3.8 classifier, clean up changelog</li>
  <li><a href="https://github.com/pallets/jinja/commit/d312609ccd561498d2963d47f0330715e8f24982"><code>d312609</code></a> isolate bytecode cache tests</li>
  <li><a href="https://github.com/pallets/jinja/commit/984997913cc2eb1a00e1b8787c3d3ada5dd359c7"><code>9849979</code></a> import Markup from markupsafe, fix flake8 import warnings</li>
  <li><a href="https://github.com/pallets/jinja/commit/c6d864cf8975415bd4ffee17f29e37f95969c8c9"><code>c6d864c</code></a> increment bytecode cache version</li>
  <li>Additional commits viewable in <a href="https://github.com/pallets/jinja/compare/2.10.3...2.11.1">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  **Note:** This repo was added to Dependabot recently, so you'll receive a maximum of 5 PRs for your first few update runs. Once an update run creates fewer than 5 PRs we'll remove that limit.

  You can always request more updates by clicking `Bump now` in your [Dependabot dashboard](https://app.dependabot.com).

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/722/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1OTc2NzE1MTg=
state: closed
created_at: 2020-04-10T03:19:40Z
updated_at: 2021-09-28T04:38:44Z
closed_at: 2020-04-13T03:29:15Z
author_association: OWNER
type: issue
state_reason: completed
title:
  Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/98/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1OTgwMTM5NjU=
title: --plugin-secret over-rides existing metadata.json plugin config
state: closed
created_at: 2020-04-10T17:56:30Z
updated_at: 2020-04-16T04:58:12Z
closed_at: 2020-04-10T18:34:21Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This means if you use `--plugin-secret` at all (with e.g. `publish cloudrun`) any existing plugin configuration in your `metadata.json` will be ignored.

  https://github.com/simonw/datasette/blob/af9cd4ca64652fae262e6f7b5d201f6e0adc989b/datasette/publish/cloudrun.py#L98-L109

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/724/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1OTg2NDAyMzQ=
title: .upsert_all() should maybe error if dictionaries passed to it do not have the same keys
state: closed
created_at: 2020-04-13T03:02:25Z
updated_at: 2020-04-13T03:05:20Z
closed_at: 2020-04-13T03:05:04Z
author_association: OWNER
type: issue
state_reason: completed
body:
  While investigating #98 I stumbled across this:
  ```
      def test_upsert_compound_primary_key(fresh_db):
          table = fresh_db["table"]
          table.upsert_all(
              [
                  {"species": "dog", "id": 1, "name": "Cleo", "age": 4},
                  {"species": "cat", "id": 1, "name": "Catbag"},
              ],
              pk=("species", "id"),
          )
          table.upsert_all(
              [
                  {"species": "dog", "id": 1, "age": 5},
                  {"species": "dog", "id": 2, "name": "New Dog", "age": 1},
              ],
              pk=("species", "id"),
          )
  >       assert [
              {"species": "dog", "id": 1, "name": "Cleo", "age": 5},
              {"species": "cat", "id": 1, "name": "Catbag", "age": None},
              {"species": "dog", "id": 2, "name": "New Dog", "age": 1},
          ] == list(table.rows)
  E       AssertionError: assert [{'age': 5, '...cies': 'dog'}] == [{'age': 5, '...cies': 'dog'}]
  E         At index 0 diff: {'species': 'dog', 'id': 1, 'name': 'Cleo', 'age': 5} != {'species': 'dog', 'id': 1, 'name': None, 'age': 5}
  E         Full diff:
  E         - [{'age': 5, 'id': 1, 'name': 'Cleo', 'species': 'dog'},
  E         ?                              ^^^ --
  E         + [{'age': 5, 'id': 1, 'name': None, 'species': 'dog'},
  E         ?                              ^^^
  E         {'age': None, 'id': 1, 'name': 'Catbag', 'species': 'cat'},
  E         {'age': 1, 'id': 2, 'name': 'New Dog', 'species': 'dog'}]
  ```
  If you run `.upsert_all()` with multiple dictionaries it doesn't quite have the effect you might expect.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/99/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDAyNjQ1OTg0
title: Update aiofiles requirement from ~=0.4.0 to >=0.4,<0.6
state: closed
created_at: 2020-04-13T13:32:47Z
updated_at: 2020-05-04T18:16:54Z
closed_at: 2020-05-04T16:17:49Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/725
type: pull
body:
  Updates the requirements on [aiofiles](https://github.com/Tinche/aiofiles) to permit the latest version.
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/Tinche/aiofiles/commit/9a2141e6c51013995ed8b72b121e4e0a27af77b5"><code>9a2141e</code></a> 0.5.0</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/479b7eea2c54a9daadb26d1073770e5a50f9ca27"><code>479b7ee</code></a> Update README</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/6c247a2cfd1e12f8359aeff04bfe3510f5ef0dc2"><code>6c247a2</code></a> Modernize tests</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/eec75d3f425d53306b6aaa8a6005344e56350312"><code>eec75d3</code></a> Switch to async def wherever possible</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/786c3e9293a37f63ae1c72b115498d038b5828a0"><code>786c3e9</code></a> Prepare for 3.8</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/145107595fe72b877005fb4ebf9d77ffb754689a"><code>1451075</code></a> Update README.rst</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/5db1e380e307d0f96177d18db778f82ea0ef87ce"><code>5db1e38</code></a> Add several async os functions</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/a60f19bb7b6bd5b063f853d8cae2c41a6878841f"><code>a60f19b</code></a> Add async remove function</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/9cf2ac8f9d43446ecdf71c95d05ad6a95bec7377"><code>9cf2ac8</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/Tinche/aiofiles/issues/53">#53</a> from graingert/patch-1</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/b88912c6a7a13597b8c5ac93093d7eef86ab1ebb"><code>b88912c</code></a> <strong>all</strong> should be a List[str]</li>
  <li>Additional commits viewable in <a href="https://github.com/Tinche/aiofiles/compare/v0.4.0...v0.5.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/725/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1OTk3NzYzNDU=
title: Feature idea: github-to-sqlite everything ...
state: open
created_at: 2020-04-14T18:34:00Z
updated_at: 2020-04-14T18:34:00Z
author_association: MEMBER
type: issue
body:
  At the moment if you want to pull all your repos, issues, issues comments etc you have to do it with a sequence of separate commands.

  Consider adding a `everything` or `all` command which fetches everything that the tool knows how to fetch, and is designed to be run on a cron in a way that fetches just new stuff each time.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/24/reactions", "total_count": 7, "+1": 7, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDAxMjA0Mzk=
title: Foreign key : case of a link to the associated row not displayed
state: closed
created_at: 2020-04-15T08:31:27Z
updated_at: 2020-04-27T22:05:47Z
closed_at: 2020-04-27T22:05:46Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Hello,
  I use Datasette to publish tsv files linked together by foreign keys declared thanks to sqlite-utils. In one table, [prelib_personne](http://crbc-dataset.huma-num.fr/prelib/prelib_personne), the foreign keys are properly noticed by a link to the associated row (for instance ville_naissance_id is properly linked to prelib_ville). But every link to the foreign key prelib_oeuvre.id fails. For instance, [prelib_ecritoeuvre](http://crbc-dataset.huma-num.fr/prelib/prelib_ecritoeuvre) has links to prelib_personne but none to prelib_oeuvre. In despite of the schema:

  CREATE TABLE "prelib_ecritoeuvre" (
  "id" INTEGER,
    "fonction_id" INTEGER,
    "oeuvre_id" INTEGER,
    "personne_id" INTEGER
    ,PRIMARY KEY ([id]),
     FOREIGN KEY(fonction_id) REFERENCES prelib_fonctionecritoeuvre(id),
     FOREIGN KEY(personne_id) REFERENCES prelib_personne(id),
     FOREIGN KEY(oeuvre_id) REFERENCES prelib_oeuvre(id)
  ); 

  Would you have any clue to investigate the reason of this problem?
  Thanks,

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/726/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDA1ODMyNzE=
title: Custom CSS class on body for styling canned queries
state: closed
created_at: 2020-04-15T20:57:32Z
updated_at: 2020-04-15T21:14:58Z
closed_at: 2020-04-15T21:07:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://latest.datasette.io/fixtures/neighborhood_search is a canned query page.

  One of the templates scanned is `query-fixtures-neighborhood_search.html`

  BUT... the body CSS class just looks like this:
  ```html
  <body class="query db-fixtures">
  ```
  I would be useful if that included a class that can be used to style that specific canned query page.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/727/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDEyNjUwMjM=
title: Improvements to demo instance
state: closed
created_at: 2020-04-16T17:26:55Z
updated_at: 2020-04-16T18:07:12Z
closed_at: 2020-04-16T18:07:12Z
author_association: MEMBER
body: - [x] Demo should pull issue-comments as well
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDEyNzE2MTI=
title: Topics are missing from repositories
state: closed
created_at: 2020-04-16T17:36:32Z
updated_at: 2020-04-16T17:41:11Z
closed_at: 2020-04-16T17:41:11Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I'm sure this used to work, but right now repositories are fetched without their topics.

  https://developer.github.com/v3/repos/ says you need to send a custom `Accept` header of `application/vnd.github.mercy-preview+json` to get topics.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDEzMzAyNzc=
title: Repos have a big blob of JSON in the organization column
state: closed
created_at: 2020-04-16T18:43:14Z
updated_at: 2020-04-18T00:19:16Z
closed_at: 2020-04-18T00:18:52Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  e.g. https://github-to-sqlite.dogsheep.net/github/repos

  ![github__repos__11_rows_where_sorted_by_updated_at_descending](https://user-images.githubusercontent.com/9599/79494124-5640b980-7fd7-11ea-99a2-17ffbd82f9ce.png)

  This appears to be obsolete because the `owner` column already links to that record, albeit in the `users` table with `type` set to `Organization`: https://github-to-sqlite.dogsheep.net/github/users/53015001

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDEzMzM2MzQ=
title: Pull repository contributors
state: closed
created_at: 2020-04-16T18:46:40Z
updated_at: 2020-04-18T15:05:10Z
closed_at: 2020-04-18T15:05:10Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://developer.github.com/v3/repos/#list-contributors

  `GET /repos/:owner/:repo/contributors`

  Not sure if this should be a separate command or should be part of the existing `repos` command. I'm leaning towards a new `contributors` command.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDEzNTg2NDk=
title: Mechanism for forcing column-type, over-riding auto-detection
state: closed
created_at: 2020-04-16T19:12:52Z
updated_at: 2020-04-17T23:53:32Z
closed_at: 2020-04-17T23:53:32Z
author_association: OWNER
type: issue
state_reason: completed
body:
  As seen in https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406 - there's a problem where you insert a record with a `None` value for a column and that column is created as `TEXT` - but actually you intended it to be an `INT` (as later examples will demonstrate).

  Some kind of mechanism for over-riding the detected types of columns would be useful here.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/100/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDEzOTIzMTg=
title: README should include an example of CLI data insertion
state: closed
created_at: 2020-04-16T19:45:37Z
updated_at: 2020-04-17T23:59:49Z
closed_at: 2020-04-17T23:59:49Z
author_association: OWNER
body: Maybe using `curl` from the GitHub API.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/101/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDIxNzM1ODk=
title: Error running user-timeline with --sql and --ids together
state: closed
created_at: 2020-04-17T19:02:06Z
updated_at: 2020-04-17T23:34:40Z
closed_at: 2020-04-17T23:34:40Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  ```
  $ twitter-to-sqlite user-timeline tweets.db --sql='select id from users' --ids
  Traceback (most recent call last):
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite", line 11, in <module>
      load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')()
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py", line 284, in user_timeline
      "@{:" + str(max(len(identifier) for identifier in identifiers)) + "}"
    File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py", line 284, in <genexpr>
      "@{:" + str(max(len(identifier) for identifier in identifiers)) + "}"
  TypeError: object of type 'int' has no len()
  ```
  But this DID work - casting to strings:
  ```
  $ twitter-to-sqlite user-timeline tweets.db --sql='select "" || id from users' --ids
  ... this worked ...
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/42/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDIxNzY4NzA=
title: "twitter-to-sqlite lists" command for retrieving a user's owned lists
state: closed
created_at: 2020-04-17T19:08:59Z
updated_at: 2020-04-17T23:48:28Z
closed_at: 2020-04-17T23:30:39Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://developer.twitter.com/en/docs/accounts-and-users/create-manage-lists/api-reference/get-lists-ownerships

  `https://api.twitter.com/1.1/lists/ownerships.json `

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/43/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDIxODE1ODE=
title: tweet["source"] can be an empty string
state: closed
created_at: 2020-04-17T19:18:26Z
updated_at: 2020-04-17T22:01:44Z
closed_at: 2020-04-17T22:01:44Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Got this excepion:
  ```
    File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py", line 641, in extract_and_save_source
      details = m.groupdict()
  AttributeError: 'NoneType' object has no attribute 'groupdict'
  ```
  I traced it back to this tweet: https://twitter.com/osder/status/578712651393576960
  ```
  (Pdb) source_re
  re.compile('<a href="(?P<url>.*?)".*?>(?P<name>.*?)</a>')
  (Pdb) locals()['source']
  ''
  (Pdb) u
  > /Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py(393)save_tweets()
  -> tweet["source"] = extract_and_save_source(db, tweet["source"])
  (Pdb) tweet
  {'created_at': '2015-03-20T00:20:22+00:00', 'id': 578712651393576960, 'full_text': '@osder', 'truncated': False, 'display_text_range': [0, 6], 'source': '', 'in_reply_to_status_id': 578712521382715392, 'in_reply_to_user_id': 1545741, 'in_reply_to_screen_name': 'osder', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 0, 'favorite_count': 0, 'favorited': False, 'retweeted': False, 'lang': 'und', 'user': 1545741}
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/44/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDI1MzMzMDA=
title: Import photo metadata from Apple Photos into SQLite
state: open
created_at: 2020-04-18T19:23:26Z
updated_at: 2020-05-04T02:41:40Z
author_association: MEMBER
body: Faces, albums, locations, that kind of thing.
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDI1MzMzNTI=
title: Ability to convert HEIC images to JPEG
state: closed
created_at: 2020-04-18T19:23:43Z
updated_at: 2020-04-28T16:47:21Z
closed_at: 2020-04-28T16:47:21Z
author_association: MEMBER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDI1MzM0ODE=
title: Import EXIF data into SQLite - lens used, ISO, aperture etc
state: open
created_at: 2020-04-18T19:24:31Z
updated_at: 2021-10-05T12:38:24Z
author_association: MEMBER
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDI1MzM1Mzk=
title: Upload all my photos to a secure S3 bucket
state: closed
created_at: 2020-04-18T19:24:50Z
updated_at: 2020-04-18T21:58:11Z
closed_at: 2020-04-18T21:57:13Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  - [x] Create a bucket with bucket credentials
  - [x] Programmatically upload some recent photos to it (from a notebook)
  - [x] Turn this into a script

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDI1NTE2Mzg=
title: photos-to-sqlite s3-auth command
state: closed
created_at: 2020-04-18T21:05:25Z
updated_at: 2020-04-18T21:08:44Z
closed_at: 2020-04-18T21:08:44Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Modeled on `github-to-sqlite auth` - prompts the user for their S3 credentials and saves them to `auth.json`.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDI1NjkzMTU=
title: Can't store an array or dictionary containing a bytes value
state: closed
created_at: 2020-04-18T22:49:21Z
updated_at: 2020-05-01T20:45:45Z
closed_at: 2020-05-01T20:45:45Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  In [1]: import sqlite_utils                                                     

  In [2]: db = sqlite_utils.Database(memory=True)                                 

  In [3]: db["t"].insert({"id": 1, "data": {"foo": b"bytes"}})                    
  ---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  <ipython-input-3-a8ab1f72c72c> in <module>
  ----> 1 db["t"].insert({"id": 1, "data": {"foo": b"bytes"}})

  ~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert(self, record, pk, foreign_keys, column_order, not_null, defaults, hash_id, alter, ignore, replace, extracts, conversions, columns)
      950             extracts=extracts,
      951             conversions=conversions,
  --> 952             columns=columns,
      953         )
      954 

  ~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, conversions, columns, upsert)
     1052                 for key in all_columns:
     1053                     value = jsonify_if_needed(
  -> 1054                         record.get(key, None if key != hash_id else _hash(record))
     1055                     )
     1056                     if key in extracts:

  ~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in jsonify_if_needed(value)
     1318 def jsonify_if_needed(value):
     1319     if isinstance(value, (dict, list, tuple)):
  -> 1320         return json.dumps(value)
     1321     elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)):
     1322         return value.isoformat()

  /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
      229         cls is None and indent is None and separators is None and
      230         default is None and not sort_keys and not kw):
  --> 231         return _default_encoder.encode(obj)
      232     if cls is None:
      233         cls = JSONEncoder

  /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in encode(self, o)
      197         # exceptions aren't as detailed.  The list call should be roughly
      198         # equivalent to the PySequence_Fast that ''.join() would do.
  --> 199         chunks = self.iterencode(o, _one_shot=True)
      200         if not isinstance(chunks, (list, tuple)):
      201             chunks = list(chunks)

  /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in iterencode(self, o, _one_shot)
      255                 self.key_separator, self.item_separator, self.sort_keys,
      256                 self.skipkeys, _one_shot)
  --> 257         return _iterencode(o, 0)
      258 
      259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

  /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in default(self, o)
      177 
      178         """
  --> 179         raise TypeError(f'Object of type {o.__class__.__name__} '
      180                         f'is not JSON serializable')
      181 

  TypeError: Object of type bytes is not JSON serializable
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/102/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDI1NzU1NzU=
title: Add progress bar to upload command
state: closed
created_at: 2020-04-18T23:32:41Z
updated_at: 2020-04-19T00:15:24Z
closed_at: 2020-04-19T00:15:24Z
author_association: MEMBER
body: Upload was added in #4
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDI1ODU0OTc=
title: Integrate image content hashing
state: open
created_at: 2020-04-19T00:36:58Z
updated_at: 2021-08-26T02:01:01Z
author_association: MEMBER
type: issue
body:
  To spot duplicate images (where the file content differs such that the sha256 is no longer a match) it would be useful to calculate and store perceptual hashes of some sort.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7/reactions", "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDI2MTkzMzA=
title: Use raise_for_status() everywhere
state: open
created_at: 2020-04-19T04:38:28Z
updated_at: 2020-04-19T04:39:22Z
author_association: MEMBER
type: issue
body:
  I keep seeing errors which I think are caused by authentication or rate limit problems but which appear to be unexpected JSON responses - presumably because they are actually an error message.

  Recent example: https://github.com/simonw/jsk-fellows-on-twitter/runs/598892575

  Using `response.raise_for_status()` everywhere will make these errors less confusing.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDA2MDY3MDE5
title: Update mergedeep requirement from ~=1.1.1 to >=1.1.1,<1.4.0
state: closed
created_at: 2020-04-20T13:33:23Z
updated_at: 2020-05-04T16:45:58Z
closed_at: 2020-05-04T16:45:49Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/728
type: pull
body:
  Updates the requirements on [mergedeep](https://github.com/clarketm/mergedeep) to permit the latest version.
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/clarketm/mergedeep/commit/3d6e7b4c9c25b64eba0834da6f54c8dd309767ea"><code>3d6e7b4</code></a> v1.3.0 - support additive merging of <code>Counter</code> types</li>
  <li><a href="https://github.com/clarketm/mergedeep/commit/56a258a0b58366aa205322476db1b3f84aac65d9"><code>56a258a</code></a> v1.2.1 - tidy docs and variable names</li>
  <li><a href="https://github.com/clarketm/mergedeep/commit/61ab21382ba20d7973a848c942eb36bd33bc7cd0"><code>61ab213</code></a> v1.2.0 - support both TYPESAFE_REPLACE and TYPESAFE_ADDITIVE merge strategies...</li>
  <li><a href="https://github.com/clarketm/mergedeep/commit/b331bb5f1580d8290dcf5c6e7690fd3b9708b5cd"><code>b331bb5</code></a> cleanup Makefile</li>
  <li><a href="https://github.com/clarketm/mergedeep/commit/6f577bf29750c073f6764ffbea29ab6d6759a8d3"><code>6f577bf</code></a> officially label support for python3.8</li>
  <li><a href="https://github.com/clarketm/mergedeep/commit/84faf37d2da81d4356a4ade4c015fc3964262585"><code>84faf37</code></a> use pipenv for managing dev dependencies</li>
  <li><a href="https://github.com/clarketm/mergedeep/commit/3a8761a736c8cb7a324bd9e6f4d50a58898a3ba9"><code>3a8761a</code></a> Update README.md</li>
  <li>See full diff in <a href="https://github.com/clarketm/mergedeep/compare/v1.1.1...v1.3.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/728/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDMyOTU5NzA=
title: Visually distinguish integer and text columns
state: closed
created_at: 2020-04-20T14:47:26Z
updated_at: 2020-05-18T17:20:02Z
closed_at: 2020-05-15T18:16:56Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It would be useful if I could tell from looking at the table page if a column was a integer or a text (or a float I guess?). This is particularly important for knowing if it safe to sort by that column.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/729/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDM2MTcwMTM=
title: Milestones should have foreign key to creator and repo
state: closed
created_at: 2020-04-21T00:20:44Z
updated_at: 2020-04-21T00:43:58Z
closed_at: 2020-04-21T00:43:58Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://github-to-sqlite.dogsheep.net/github/milestones

  Creator is an integer but not a foreign key to users

  Repo is missing entirely!

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDM2MTgyNDQ=
title: Issues milestone column is the wrong type
state: closed
created_at: 2020-04-21T00:24:34Z
updated_at: 2020-04-21T00:45:23Z
closed_at: 2020-04-21T00:36:22Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://github-to-sqlite.dogsheep.net/github/issues?milestone=2857392

  ![2A4C1185-2434-4F29-9EA0-3246E2F03F77](https://user-images.githubusercontent.com/9599/79811760-b7e08b00-832b-11ea-9ad7-684a6ae097a6.jpeg)

  It is TEXT when it should be an INTEGER - which is why the foreign key label is not correctly displayed.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDM2MjQ4NjI=
title: Issue and milestone should have foreign key to repo
state: closed
created_at: 2020-04-21T00:46:24Z
updated_at: 2020-04-22T01:20:19Z
closed_at: 2020-04-22T01:20:19Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Currently the `repo` column on those tables is a string `simonw/datasette` rather than an ID referencing a row in `repos`.

  _Originally posted by @simonw in https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275_

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDA2Njc3MjA1
title: Update pytest-asyncio requirement from ~=0.10.0 to >=0.10,<0.12
state: closed
created_at: 2020-04-21T13:32:35Z
updated_at: 2020-05-04T13:27:24Z
closed_at: 2020-05-04T13:27:23Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/730
type: pull
body:
  Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version.
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/1026c39495a963ff3e5fee7da2ae9f3a5d21fb83"><code>1026c39</code></a> 0.11.0</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/ab2b14048a691479fa9f8811aaa558018c6db6e3"><code>ab2b140</code></a> Test on Python 3.8, drop 3.3 and 3.4</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/6397a2255e3e9ef858439b164018438a8106f454"><code>6397a22</code></a> plugin: Use pytest 5.4.0 new Function API</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/21a0f9476be84ca0c84af60057f0f24c5fb2fd71"><code>21a0f94</code></a> Replace yield_fixture() by fixture()</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/964b295ba280a6e217159706279b67f8f4cbb5f4"><code>964b295</code></a> Added min hypothesis version so that bugfix for <a href="https://github.com/Hypothesis">https://github.com/Hypothesis</a>...</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/4a11a206fbcf88ee18cbed2d01041e61c20a9a48"><code>4a11a20</code></a> Add max supported pytest version to &lt; 5.4.0 to prevent fails until <a href="https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/141">#141</a> is fi...</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/b3055940db49cc17e36b66631e3d863e15fe34e4"><code>b305594</code></a> Change event_loop to module scope in hypothesis tests, fixing <a href="https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/145">#145</a>.</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/d5a0f4789e7fecb58d509409e2c537b206c4fde2"><code>d5a0f47</code></a> Enable test_subprocess to be run on win, by changing to ProactorEventLoop in ...</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/d07cd2d447cf313446c4e00e25a35cb2adcb2c63"><code>d07cd2d</code></a> Fix required pytest version</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/86cd9a6fd248e1dc0a7de9c6d60dbf3773aa1fa6"><code>86cd9a6</code></a> Handle BaseExceptions from loop.run_until_complete (<a href="https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/126">#126</a>)</li>
  <li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest-asyncio/compare/v0.10.0...v0.11.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/730/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDQyMjIyOTU=
title: Issue comments don't appear to populate issues foreign key
state: closed
created_at: 2020-04-21T19:17:32Z
updated_at: 2020-04-22T01:17:44Z
closed_at: 2020-04-22T01:17:44Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101

  <img width="774" alt="Screen Shot 2020-04-21 at 12 16 57 PM" src="https://user-images.githubusercontent.com/9599/79904752-0b091b00-83ca-11ea-9b15-b2893f05dfdc.png">

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDUxMTAwMTU=
title: Option to automatically configure based on directory layout
state: closed
created_at: 2020-04-22T22:17:47Z
updated_at: 2020-04-27T16:32:44Z
closed_at: 2020-04-27T16:30:26Z
author_association: OWNER
type: issue
state_reason: completed
body:
  My Datasette projects increasingly take on the following structure:

  - `metadata.json` with the metadata
  - One or more `something.db` database files
  - A `templates/` folder with some custom templates
  - A `plugins/` folder with some custom plugins

  Then I have to run Datasette like this:

      datasette *.db -m metadata.json --template-dir=templates --plugins-dir=plugins

  It would be really interesting if Datasette had a special mode where you could point it at a directory with the above layout and it would automatically configure itself based on the contents.

  Maybe even allow `datasette serve` to detect if it was passed a single argument that's a directory, not a file, and kick in to "directory layout configuration mode" in that case:

      datasette .

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/731/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDUxNDc2Mzg=
title: Should I have used MD5 instead of SHA256?
state: closed
created_at: 2020-04-23T00:02:08Z
updated_at: 2020-04-23T00:03:35Z
closed_at: 2020-04-23T00:03:35Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html

  > Objects created by the PUT Object, POST Object, or Copy operation, or through the AWS Management Console, and are encrypted by SSE-S3 or plaintext, have ETags that are an MD5 digest of their object data.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDA3OTI5MTI4
title: Update janus requirement from ~=0.4.0 to >=0.4,<0.6
state: closed
created_at: 2020-04-23T13:43:45Z
updated_at: 2020-05-04T16:48:14Z
closed_at: 2020-05-04T16:48:04Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/734
type: pull
body:
  Updates the requirements on [janus](https://github.com/aio-libs/janus) to permit the latest version.
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/aio-libs/janus/blob/master/CHANGES.rst">janus's changelog</a>.</em></p>
  <blockquote>
  <h1>0.5.0 (2020-04-23)</h1>
  <ul>
  <li>Remove explicit loop arguments and forbid creating queues outside event loops <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/246">#246</a></li>
  </ul>
  <h1>0.4.0 (2018-07-28)</h1>
  <ul>
  <li>Add <code>py.typed</code> macro <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/89">#89</a></li>
  <li>Drop python 3.4 support and fix minimal version python3.5.3 <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/88">#88</a></li>
  <li>Add property with that indicates if queue is closed <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/86">#86</a></li>
  </ul>
  <h1>0.3.2 (2018-07-06)</h1>
  <ul>
  <li>Fixed python 3.7 support <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/97">#97</a></li>
  </ul>
  <h1>0.3.1 (2018-01-30)</h1>
  <ul>
  <li>Fixed bug with join() in case tasks are added by sync_q.put() <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/75">#75</a></li>
  </ul>
  <h1>0.3.0 (2017-02-21)</h1>
  <ul>
  <li>Expose unfinished_tasks property <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/34">#34</a></li>
  </ul>
  <h1>0.2.4 (2016-12-05)</h1>
  <ul>
  <li>Restore tarball deploying</li>
  </ul>
  <h1>0.2.3 (2016-07-12)</h1>
  <ul>
  <li>Fix exception type</li>
  </ul>
  <h1>0.2.2 (2016-07-11)</h1>
  <ul>
  <li>Update asyncio.async() to use asyncio.ensure_future() <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/6">#6</a></li>
  </ul>
  <h1>0.2.1 (2016-03-24)</h1>
  <ul>
  <li>Fix python setup.py test command <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/4">#4</a></li>
  </ul>
  <h1>0.2.0 (2015-09-20)</h1>
  </tr></table> ... (truncated)
  </blockquote>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/aio-libs/janus/commit/8e89b451f6718d705027a60f10ea59b900643512"><code>8e89b45</code></a> Bump to 0.5.0</li>
  <li><a href="https://github.com/aio-libs/janus/commit/ec8592b91254971473b508313fb91b01623f13d7"><code>ec8592b</code></a> Fix up Python 3.8 loop argument warnings (<a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/246">#246</a>)</li>
  <li><a href="https://github.com/aio-libs/janus/commit/2543af654ba55b7f0ec5e376d14e05cf777d3ca9"><code>2543af6</code></a> Bump coverage from 5.0.4 to 5.1</li>
  <li><a href="https://github.com/aio-libs/janus/commit/03d1b3620393296d7c10d8bf9f67eda2edcbb735"><code>03d1b36</code></a> Bump tox from 3.14.5 to 3.14.6</li>
  <li><a href="https://github.com/aio-libs/janus/commit/8219c38e58a84f5ed7e995ac7f331e7d998dad5f"><code>8219c38</code></a> Bump coverage from 5.0.3 to 5.0.4</li>
  <li><a href="https://github.com/aio-libs/janus/commit/85ec71dda401be9a2df21cd457e20f64109f67c2"><code>85ec71d</code></a> Bump pytest from 5.4.0 to 5.4.1</li>
  <li><a href="https://github.com/aio-libs/janus/commit/3b974c95d41ac470e8f85d713d1797bbe284c383"><code>3b974c9</code></a> Bump pytest from 5.3.5 to 5.4.0</li>
  <li><a href="https://github.com/aio-libs/janus/commit/282dc123842e71d66aae77461db973430ae78a6d"><code>282dc12</code></a> Bump mypy from 0.761 to 0.770</li>
  <li><a href="https://github.com/aio-libs/janus/commit/1364fb3a7766481bb099d24db9efce2044e00141"><code>1364fb3</code></a> Bump tox from 3.14.4 to 3.14.5</li>
  <li><a href="https://github.com/aio-libs/janus/commit/dc519bb4bf4e4bb1fa3972c919b9be9ea6e33aa3"><code>dc519bb</code></a> Bump tox from 3.14.3 to 3.14.4</li>
  <li>Additional commits viewable in <a href="https://github.com/aio-libs/janus/compare/v0.4.0...v0.5.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/734/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDU4MDYzODY=
title: Error when I click on "View and edit SQL"
state: closed
created_at: 2020-04-23T19:31:32Z
updated_at: 2020-04-28T06:10:20Z
closed_at: 2020-04-27T19:00:30Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi,
  when I do it [here](https://my-database.now.sh/commissioniComunePalermo/youtube), I have "unrecognized token: "["" error.

  Is it normal?

  Thank you

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/735/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDU5MzgwNjM=
title: upload command should be resumable, should only upload photos not already uploaded
state: closed
created_at: 2020-04-23T23:31:08Z
updated_at: 2020-04-23T23:39:14Z
closed_at: 2020-04-23T23:39:14Z
author_association: MEMBER
body: Follow on from #4.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDYwMjgyNzI=
title: Speed up hashing step using threads
state: closed
created_at: 2020-04-24T04:20:08Z
updated_at: 2020-04-24T04:32:35Z
closed_at: 2020-04-24T04:32:35Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  This TODO from the code:

  https://github.com/dogsheep/photos-to-sqlite/blob/2e7f2c67cc18b02c75bb64992a05b0196e507252/photos_to_sqlite/cli.py#L82-L90

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDYwMzI5NTA=
title: Try running S3 uploads in a thread pool
state: closed
created_at: 2020-04-24T04:34:31Z
updated_at: 2020-04-24T16:45:41Z
closed_at: 2020-04-24T16:45:41Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Since #10 provided such a speedup, can the same thing be done for the actual uploads?

  http://ls.pwd.io/2013/06/parallel-s3-uploads-using-boto-and-threads-in-python/ suggests it can really help performance.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/11/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDYwMzMxMDQ=
title: If less than 500MB, show size in MB not GB
state: open
created_at: 2020-04-24T04:35:01Z
updated_at: 2020-04-24T04:35:25Z
author_association: MEMBER
body:
  Just saw this:
  ```
  Uploading 0.05 GB
  ```

type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDY3MjA2NzQ=
title: strange behavior using accented characters
state: closed
created_at: 2020-04-25T08:34:51Z
updated_at: 2020-04-28T06:09:28Z
closed_at: 2020-04-27T18:59:16Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi,
  when I search `incompatibilità` [here](https://my-database.now.sh/commissioniComunePalermo/youtube), using full text search, it becomes `incompatibilitÃÂ ` and I have no result.

  If I encode the `à` char in the URL (`incompatibilit%C3%A0`) I have the right result.

  ![image](https://user-images.githubusercontent.com/30607/80275201-00a79380-86e0-11ea-865e-f7e1474e8098.png)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/736/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDA5MTIzODk3
title: Custom pages mechanism, refs #648
state: closed
created_at: 2020-04-26T17:31:41Z
updated_at: 2020-04-26T18:46:43Z
closed_at: 2020-04-26T18:46:43Z
author_association: OWNER
pull_request: simonw/datasette/pulls/737
type: pull
body:
  Refs #648. TODO:
  - [x] Pass a `view_name` to `render_template()`
  - [x] Mechanism for custom status code / headers / redirect
  - [x] Documentation

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/737/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDcwODY3ODA=
title: Pass a request object to custom page templates
state: closed
created_at: 2020-04-26T18:57:48Z
updated_at: 2020-04-26T19:01:54Z
closed_at: 2020-04-26T19:01:54Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Follow-up to #648. I'm not passing a request object to `.render_template()` at the moment, which breaks any other custom plugins using e.g. `extra_template_vars()` that were expecting to be able to access the request.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/738/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDA5MTUzODcw
title: Configuration directory mode
state: closed
created_at: 2020-04-26T20:37:46Z
updated_at: 2020-04-27T16:30:25Z
closed_at: 2020-04-27T16:30:25Z
author_association: OWNER
pull_request: simonw/datasette/pulls/739
type: pull
body:
  Refs #731

  TODO:

  - [x] Decide how to combine explicit command-line options with items detected from the directory structure
  - [x] Add unit tests
  - [x] Implement `inspect-data.json` mechanism for populating `immutables`
  - [x] Add documentation

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/739/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDcyMTEwNTg=
title: Don't throw 500 error on attempted directory browse
state: closed
created_at: 2020-04-27T03:50:11Z
updated_at: 2020-04-27T18:29:15Z
closed_at: 2020-04-27T18:29:15Z
author_association: OWNER
type: issue
state_reason: completed
body:
  <img width="725" alt="Error_500" src="https://user-images.githubusercontent.com/9599/80332338-76eee780-87ff-11ea-85b3-32704d715cd9.png">

  This should be a 403 error instead, because the `--static` mechanism doesn't allow directory browsing.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/740/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDcyMjMxMzY=
title: Replace "datasette publish --extra-options" with "--setting"
state: open
created_at: 2020-04-27T04:29:04Z
updated_at: 2022-05-12T19:21:16Z
author_association: OWNER
type: issue
body:
  See https://github.com/simonw/datasette-publish-now/issues/9#issuecomment-618155764 - the `--extra-options` mechanism is in practice just used to set `--config` options in data that you publish, but that means you end up with pretty messy looking commands:

      datasette publish my.db --extra-options="--config default_page_size:50 --config sql_time_limit_ms:3500"

  A neater design would be to support `--config` as an option for `datasette publish` directly:

      datasette publish my.db --config default_page_size:50 --config sql_time_limit_ms:3500

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/741/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDcyNDM5NDA=
title: Speed up tests with scope="session"?
state: closed
created_at: 2020-04-27T05:23:54Z
updated_at: 2020-04-27T18:24:53Z
closed_at: 2020-04-27T18:24:53Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Tests are pretty slow - could I speed them up with pytest `scope="session"` on some of the fixtures?

  Eg https://travis-ci.org/github/simonw/datasette/jobs/679940036 ran 452 tests in 3m53s - the `test_html` ones seem particularly slow.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/742/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDc3NzA1OTU=
title: escape_fts() does not correctly escape * wildcards
state: closed
created_at: 2020-04-27T18:48:53Z
updated_at: 2020-04-27T19:11:30Z
closed_at: 2020-04-27T19:11:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Spotted in #732. This should not return any results... but it does:

  https://latest.datasette.io/fixtures/searchable?_search=bar%2A&_trace=1

  <img width="694" alt="fixtures__searchable__1_row_where_where_search_matches__bar__" src="https://user-images.githubusercontent.com/9599/80408992-f66ecc00-887c-11ea-852f-08dd9d115206.png">

  The query from trace is:
  ```
        "sql": "select count(*) from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search))",
        "params": {
          "search": "bar*"
        }
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/743/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDc4ODgzNjc=
title: Also upload movie files
state: open
created_at: 2020-04-27T22:11:25Z
updated_at: 2020-04-28T00:39:45Z
author_association: MEMBER
type: issue
body:
  The `upload` command currently only handles static images:

  https://github.com/dogsheep/photos-to-sqlite/blob/d939455af00e07866686457ee2fcb9b2d1b7194e/photos_to_sqlite/utils.py#L26-L33

  Need to cover movies taken by my phone and DSLR too.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDgwNTg4OTA=
title: link_or_copy_directory() error - Invalid cross-device link
state: closed
created_at: 2020-04-28T06:26:45Z
updated_at: 2020-05-28T14:32:53Z
closed_at: 2020-05-27T06:01:28Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi,
  when I run

  ```
  datasette  publish heroku -n myapp --template-dir ./template mydb.db
  ```

  I have this error

  ```
  Traceback (most recent call last):
    File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 607, in link_or_copy_directory
      shutil.copytree(src, dst, copy_function=os.link)
    File "/usr/lib/python3.7/shutil.py", line 365, in copytree
      raise Error(errors)
  shutil.Error: [('/myfolder/youtubeComunePalermo/processing/./template/base.html', '/tmp/tmps9_4mzc4/templates/base.html', "[Errno 18] Invalid cross-device link: '/myfolder/youtubeComunePalermo/processing/./template/base.html' -> '/tmp/tmps9_4mzc4/templates/base.html'"), ('/myfolder/youtubeComunePalermo/processing/./template/index.html', '/tmp/tmps9_4mzc4/templates/index.html', "[Errno 18] Invalid cross-device link: '/myfolder/youtubeComunePalermo/processing/./template/index.html' -> '/tmp/tmps9_4mzc4/templates/index.html'")]

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/aborruso/.local/bin/datasette", line 8, in <module>
      sys.exit(cli())
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 103, in heroku
      extra_metadata,
    File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
      return next(self.gen)
    File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 191, in temporary_heroku_directory
      os.path.join(tmp.name, "templates"),
    File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 609, in link_or_copy_directory
      shutil.copytree(src, dst)
    File "/usr/lib/python3.7/shutil.py", line 321, in copytree
      os.makedirs(dst)
    File "/usr/lib/python3.7/os.py", line 221, in makedirs
      mkdir(name, mode)
  FileExistsError: [Errno 17] File exists: '/tmp/tmps9_4mzc4/templates'
  ```

  I'm attaching my very basic template folder.

  Thank you 

  [template.zip](https://github.com/simonw/datasette/files/4543751/template.zip)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/744/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDg1MTI3NDc=
title: Annotate photos using the Google Cloud Vision API
state: open
created_at: 2020-04-28T18:09:03Z
updated_at: 2020-04-28T18:19:06Z
author_association: MEMBER
type: issue
body:
  It can detect faces, run OCR, do image labeling (it knows what a lemur is!) and do object localization where it identifies objects and returns bounding polygons for them.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14/reactions", "total_count": 3, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDg2MTMwMzM=
title: Extract the hash-URL mechanism out into a plugin
state: closed
created_at: 2020-04-28T21:00:38Z
updated_at: 2020-10-23T19:47:18Z
closed_at: 2020-10-23T19:47:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  0.28 in May 2019 made this feature not-the-default: https://datasette.readthedocs.io/en/stable/changelog.html#v0-28 - see #418

  I've not felt the need to use it myself since. I think I should move it into a plugin.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/745/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDEwNDY5Mjcy
title: shutil.Error, not OSError
state: closed
created_at: 2020-04-29T03:30:51Z
updated_at: 2020-04-29T07:07:24Z
closed_at: 2020-04-29T07:07:23Z
author_association: OWNER
pull_request: simonw/datasette/pulls/746
body: Refs #744
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/746/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MDk5NTAwOTA=
title: Fall back to authentication via ENV
state: closed
created_at: 2020-04-30T12:58:14Z
updated_at: 2020-05-02T18:46:10Z
closed_at: 2020-05-02T18:45:37Z
author_association: NONE
type: issue
state_reason: completed
body:
  Would you accept a PR that falls back to looking for an environment variable for the GitHub token? Specifically a change here:
  https://github.com/dogsheep/github-to-sqlite/blob/c34d5a18bfc41fa08755ba3d5cf9fe09ff204238/github_to_sqlite/cli.py#L271

  I'd like to use `github-to-sqlite` in a GitHub Action workflow and this would be simpler than trying to fill out the prompt or generate a file with sensitive content.

  Wanted to check first, I'm happy to submit a PR with tests and updates to the docs. 

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTAxOTIxNTI=
title: Directory configuration mode should support metadata.yaml
state: closed
created_at: 2020-04-30T16:05:30Z
updated_at: 2020-04-30T19:04:19Z
closed_at: 2020-04-30T19:04:19Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Refs #739 - `metadata.yml` or `metadata.yaml` should be detected in the same way as `metadata.json` is.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/747/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTAyODQ0NzE=
title: Error running 'search' for the first time
state: closed
created_at: 2020-04-30T18:11:20Z
updated_at: 2020-04-30T18:11:58Z
closed_at: 2020-04-30T18:11:58Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  ```
  % twitter-to-sqlite search infodemic.db '#infodemic'
  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/bin/twitter-to-sqlite", line 11, in <module>
      load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')()
    File "/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/Users/simon/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py", line 867, in search
      for tweet in tweets:
    File "/Users/simon/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py", line 165, in fetch_timeline
      [since_type_id, since_key],
  sqlite3.OperationalError: no such table: since_ids
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/46/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTAzNDI1NzU=
title: ?_searchmode=raw should be documented on full-text search page
state: closed
created_at: 2020-04-30T19:50:06Z
updated_at: 2020-04-30T21:06:12Z
closed_at: 2020-04-30T21:06:12Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It's currently documented here: https://datasette.readthedocs.io/en/stable/json_api.html#special-table-arguments

  But it should also be described here:  https://datasette.readthedocs.io/en/stable/full_text_search.html#the-table-view-api

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/748/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTA0MDg5MDg=
title: Command for retrieving dependents for a repo
state: closed
created_at: 2020-04-30T21:47:51Z
updated_at: 2020-05-03T15:53:01Z
closed_at: 2020-05-03T15:53:01Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I really, really want to start grabbing this data: https://github.com/simonw/datasette/network/dependents

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTA1MTE0NTA=
title: Create index on issue_comments(user) and other foreign keys
state: closed
created_at: 2020-05-01T02:06:56Z
updated_at: 2020-05-02T18:26:24Z
closed_at: 2020-05-02T18:26:24Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  ```
  create index issue_comments_user on issue_comments(user)
  ```
  I'm sure there are other user columns that could benefit from an index.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTA1MTc0NzI=
state: closed
created_at: 2020-05-01T02:26:14Z
updated_at: 2020-05-14T00:18:57Z
closed_at: 2020-05-14T00:18:57Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
title:
  sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns

body:
  If using insert_all to put in 1000 rows of data with varying number of columns, it comes up with this message `sqlite3.OperationalError: too many SQL variables` if the number of columns is larger in later records (past the first row)

  I've reduced `SQLITE_MAX_VARS` by 100 to 899 at the top of `db.py` to add wiggle room, so that if the column count increases it wont go past SQLite's batch limit as calculated by this line of code based on the count of the first row's dict keys

          batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/103/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTA4MjkyMjc=
title: Cloud Run fails to serve database files larger than 32MB
state: closed
created_at: 2020-05-01T16:06:46Z
updated_at: 2020-12-03T00:31:15Z
closed_at: 2020-12-03T00:31:14Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://cloud.google.com/run/quotas lists the maximum response size as 32MB.

  I spotted a bug where attempting to download a database file larger than that from a Cloud Run deployment (in this case it was https://github-to-sqlite.dogsheep.net/github.db after I [accidentally increased the size of that database](https://github.com/dogsheep/github-to-sqlite/commit/630bdba68a23c0ac453e015518ef0bf41107a952)) returned a 500 error because of this.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/749/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTA4NDI5MjY=
title: Add view for better display of dependent repos
state: closed
created_at: 2020-05-01T16:33:44Z
updated_at: 2020-05-02T16:50:31Z
closed_at: 2020-05-02T16:30:11Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  ```sql
  select
    repos.full_name as repo,
    'https://github.com/' || repos2.full_name as dependent,
    repos2.created_at as dependent_repo_created,
    repos2.updated_at as dependent_repo_updated,
    repos2.stargazers_count as dependent_repo_stars,
    repos2.watchers_count as dependent_repo_watchers
  from
    dependents
    join repos as repos2 on dependents.dependent = repos2.id
    join repos on dependents.repo = repos.id
  order by
    repos2.created_at desc
  ```
  https://dogsheep.simonwillison.net/github?sql=select%0D%0A++repos.full_name+as+repo%2C%0D%0A++%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos2.full_name+as+dependent%2C%0D%0A++repos2.created_at+as+dependent_repo_created%2C%0D%0A++repos2.updated_at+as+dependent_repo_updated%2C%0D%0A++repos2.stargazers_count+as+dependent_repo_stars%2C%0D%0A++repos2.watchers_count+as+dependent_repo_watchers%0D%0Afrom%0D%0A++dependents%0D%0A++join+repos+as+repos2+on+dependents.dependent+%3D+repos2.id%0D%0A++join+repos+on+dependents.repo+%3D+repos.id%0D%0Aorder+by%0D%0A++repos2.created_at+desc

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTA4NDMxMzY=
title: Mechanism for creating views if they don't yet exist
state: closed
created_at: 2020-05-01T16:34:10Z
updated_at: 2020-05-02T16:19:47Z
closed_at: 2020-05-02T16:19:31Z
author_association: MEMBER
body: Needed for #36 #10 #12
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTA4NTMzOTM=
title: --schema option to "sqlite-utils tables"
state: closed
created_at: 2020-05-01T16:55:49Z
updated_at: 2020-05-01T17:12:37Z
closed_at: 2020-05-01T17:12:37Z
author_association: OWNER
body: Adds output showing the table schema.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/104/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTA4NTM1NzY=
title: "sqlite-utils views" command
state: closed
created_at: 2020-05-01T16:56:11Z
updated_at: 2020-05-01T20:40:07Z
closed_at: 2020-05-01T20:38:36Z
author_association: OWNER
body: Similar to `sqlite-utils tables`. See also #104.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/105/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTEyMTY4NjI=
title: create_view(..., ignore=True, replace=True) parameters
state: closed
created_at: 2020-05-02T15:45:21Z
updated_at: 2020-05-02T16:04:51Z
closed_at: 2020-05-02T16:02:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Two new parameters which specify what should happen if the view already exists. I want this for https://github.com/dogsheep/github-to-sqlite/issues/37

  Here's the current `create_view()` implementation:

  https://github.com/simonw/sqlite-utils/blob/b4d953d3ccef28bb81cea40ca165a647b59971fa/sqlite_utils/db.py#L325-L332

  `ignore=True` will not do anything if the view exists already.

  `replace=True` will drop and redefine the view - but only if its SQL definition differs, otherwise it will be left alone.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/106/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTEyMjI5Njg=
title: sqlite-utils create-view CLI command
state: closed
created_at: 2020-05-02T16:15:13Z
updated_at: 2020-05-03T15:36:58Z
closed_at: 2020-05-03T15:36:37Z
author_association: OWNER
body: Can go with #27 - `sqlite-utils create-table`.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/107/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTEyNTIyNDQ=
title: Add notlike table filter
state: closed
created_at: 2020-05-02T18:54:36Z
updated_at: 2020-05-02T19:10:44Z
closed_at: 2020-05-02T19:10:44Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I found myself wanting that for applying the opposite of this: https://github-to-sqlite.dogsheep.net/github/dependent_repos?dependent__like=%25simonw%2F%25&_sort_desc=dependent_stars

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/750/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTEyODQ0ODE=
title: [Feature Request] Support Repo Name in Search 🥺
state: closed
created_at: 2020-05-02T22:08:51Z
updated_at: 2020-05-03T02:34:32Z
closed_at: 2020-05-02T23:15:11Z
author_association: NONE
type: issue
state_reason: completed
body:
  ## Description
  Per your [v2.2 release tweet](https://twitter.com/simonw/status/1256700238099693568) I played with the demo, but the output did not match my expectations.

  ## Expected Behavior
  Expected a search query for "twitter" contained within the `repo` column to return non-zero results.

  ## Actual Behavior
  😭  [0 rows where repo contains "twitter" sorted by starred_at descending](https://github-to-sqlite.dogsheep.net/github/stars?repo__contains=twitter&_sort_desc=starred_at) 

  ## Best Explanation
  Per the table schema (see appendix) `repo` is of type `INTEGER` which built from `repo_id` and does not expose the repo name in search.

  ## Desired Behavior
  Given that searching for "206156866" is less intuitive than "twitter",  it would be great to support this via extending the search capabilities or by adding an additional column.

  ✅ 104 rows where repo contains "twitter"
  ❌  [104 rows where repo contains "206156866" sorted by starred_at descending](https://github-to-sqlite.dogsheep.net/github/stars?repo__contains=206156866&_sort_desc=starred_at) 

  ## Appendix
  ```
  CREATE TABLE [stars] (
     [user] INTEGER REFERENCES [users]([id]),
     [repo] INTEGER REFERENCES [repos]([id]),
     [starred_at] TEXT,
     PRIMARY KEY ([user], [repo])
  );
  CREATE INDEX [idx_stars_repo]
                  ON [stars] ([repo]);
  CREATE INDEX [idx_stars_user]
                  ON [stars] ([user]);
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTEzMjY3MDE=
title: Documentation unit tests for CLI commands
state: closed
created_at: 2020-05-03T03:58:42Z
updated_at: 2020-05-03T04:13:57Z
closed_at: 2020-05-03T04:13:57Z
author_association: OWNER
body: Have a test that ensures all CLI commands are documented.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/108/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTE1NDA3OTc=
title: Ability to set custom default _size on a per-table basis
state: closed
created_at: 2020-05-04T00:13:03Z
updated_at: 2020-05-28T05:00:22Z
closed_at: 2020-05-28T05:00:20Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I have some tables where I'd like the default page size to be 10, without affecting the rest of my Datasette instance.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/751/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTE4MzUyODU=
title: Non-utf8 encoding in exceptionhandlers and custom-pages
state: closed
created_at: 2020-05-04T12:24:42Z
updated_at: 2020-05-04T17:42:20Z
closed_at: 2020-05-04T17:42:20Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi Simon.

  Whenever a response is not piped through a router-view, the template is encoded in latin-1 (I think). This is especially a problem (for me) with the new custom_pages-functionality, but also problematic with the 404- and 500-handlers. Thanks!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/752/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDEyOTUxMTkx
title: Update pytest-asyncio requirement from ~=0.10.0 to >=0.10,<0.13
state: closed
created_at: 2020-05-04T13:27:19Z
updated_at: 2020-05-04T17:41:01Z
closed_at: 2020-05-04T17:40:49Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/753
type: pull
body:
  Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version.
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/b8e2a45e152a196cef7fdb6ddcf3e2d67a0f01ca"><code>b8e2a45</code></a> 0.12.0</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/06580c6413e905c5fddcc57c2895e9d3d2913847"><code>06580c6</code></a> Update changelog</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/b45de23c0de0acc843c3f871985ba16fdfb1d5fd"><code>b45de23</code></a> Fixed failing test case, 'test_asyncio_marker_without_loop'.</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/238ccedde8a196c16007a8a2b1ea1eeeb8db86a3"><code>238cced</code></a> Put event_loop first among the fixtures of asyncio tests, fixes <a href="https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/154">#154</a>.</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/e5e3dc7a90685a56747a2d4c3c770d74d573c497"><code>e5e3dc7</code></a> Added unittests for issue <a href="https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/154">#154</a>.</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/a7e5795335b823b4d58bb6b38b6e653f0e0d35b0"><code>a7e5795</code></a> 0.12.0 open for business!</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/1026c39495a963ff3e5fee7da2ae9f3a5d21fb83"><code>1026c39</code></a> 0.11.0</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/ab2b14048a691479fa9f8811aaa558018c6db6e3"><code>ab2b140</code></a> Test on Python 3.8, drop 3.3 and 3.4</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/6397a2255e3e9ef858439b164018438a8106f454"><code>6397a22</code></a> plugin: Use pytest 5.4.0 new Function API</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/21a0f9476be84ca0c84af60057f0f24c5fb2fd71"><code>21a0f94</code></a> Replace yield_fixture() by fixture()</li>
  <li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest-asyncio/compare/v0.10.0...v0.12.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/753/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTE5OTcxMzA=
title: Clean up aiofiles warnings on 3.8
state: closed
created_at: 2020-05-04T16:14:59Z
updated_at: 2020-05-04T16:22:30Z
closed_at: 2020-05-04T16:22:30Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://travis-ci.org/github/simonw/datasette/jobs/682624476

  Lots of warnings like this:
  ```
  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33

    /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/utils.py:33: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead

      def method(self, *args, **kwargs):

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/__init__.py:27

    /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/aiofiles/threadpool/__init__.py:27: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead

      def _open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None,
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/754/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTIwODI4NDI=
title: Fix "no such column: id" output in tests
state: closed
created_at: 2020-05-04T18:37:49Z
updated_at: 2020-05-04T18:42:14Z
closed_at: 2020-05-04T18:42:14Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  pytest
  ...
  tests/test_custom_pages.py ........                                                                                                                                            [ 33%]
  tests/test_database.py ......no such column: id
  ...                                                                                                                                               [ 35%]
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/755/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTIwODk5NDk=
title: Add pipx to installation documentation
state: closed
created_at: 2020-05-04T18:49:01Z
updated_at: 2020-05-04T19:19:06Z
closed_at: 2020-05-04T19:10:33Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Add to this page: https://datasette.readthedocs.io/en/stable/installation.html

  Here's how to install plugins: https://twitter.com/simonw/status/1257348687979778050
  ```
  $ datasette plugins
  []

  $ pipx inject datasette datasette-json-html            
    injected package datasette-json-html into venv datasette
  done! ✨ 🌟 ✨

  $ datasette plugins
  [
      {
          "name": "datasette-json-html",
          "static": false,
          "templates": false,
          "version": "0.6"
      }
  ]
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/756/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTIxNTE3Njc=
title: Expose scores from ZCOMPUTEDASSETATTRIBUTES
state: closed
created_at: 2020-05-04T20:36:07Z
updated_at: 2020-12-20T04:44:22Z
closed_at: 2020-05-05T00:11:45Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  The Apple Photos database has a `ZCOMPUTEDASSETATTRIBUTES` that looks absurdly interesting... it has calculated scores for every photo:

  <img width="273" alt="Photos__ZCOMPUTEDASSETATTRIBUTES" src="https://user-images.githubusercontent.com/9599/81011044-1f0e3d00-8e0c-11ea-84b6-f302b09e7bc9.png">

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTIyODcyMzQ=
title: Import machine-learning detected labels (dog, llama etc) from Apple Photos
state: open
created_at: 2020-05-05T02:45:43Z
updated_at: 2020-05-05T05:38:16Z
author_association: MEMBER
type: issue
body:
  Follow-on from #1. Apple Photos runs some very sophisticated machine learning on-device to figure out if photos are of dogs, llamas and so on. I really want to extract those labels out into my own database.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16/reactions", "total_count": 2, "+1": 0, "-1": 0, "laugh": 1, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTIzNzgyMDM=
title: Question: Any fixed date for the release with the uft8-encoding fix?
state: closed
created_at: 2020-05-05T06:51:20Z
updated_at: 2020-05-06T18:41:29Z
closed_at: 2020-05-06T18:41:29Z
author_association: NONE
body: Just a little impatient :)
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/757/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTIzODI2NDM=
title: Question: Access to immutable database-path
state: open
created_at: 2020-05-05T07:01:18Z
updated_at: 2020-05-28T08:23:27Z
author_association: NONE
type: issue
body:
  Hi Simon

  Is there anywhere in the app-context where one can access the hashed urlpath of the database? Currently it's included in the template-context (`databases[0]["path")` when rendering urls of the database (eg. `/db-44b06v9/cases`...), but where can I find the hashed url when rendering the index-page? I'm trying to avoid redirects. Thanks!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/758/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTI2NTg0NDQ=
title: table.create_index(..., ignore=True)
state: closed
created_at: 2020-05-05T14:44:21Z
updated_at: 2020-05-05T14:46:53Z
closed_at: 2020-05-05T14:46:53Z
author_association: OWNER
body: Option to silently do nothing if the index already exists.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/109/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTI2NzM5NDg=
title: fts search on a column doesn't work anymore due to escape_fts
state: closed
created_at: 2020-05-05T15:03:44Z
updated_at: 2021-07-16T02:11:54Z
closed_at: 2020-05-06T17:50:57Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi and first, thank you for this awesome work you make with this projet.

  On a db indexed in full text search, I can't query on indexed column anymore.

  This request "cauvin language:ita": is running smoothly on a old version of datasette but not on the current version.

  Compare the current version query
  `select uuid, title, authors, year, series, language, formats, publisher, tags, identifiers from summary where rowid in (select rowid from summary_fts where summary_fts match escape_fts(:search)) order by uuid limit 101`


  To an older version:

  `select title, authors, series, uuid, language, identifiers, tags, publisher, formats, year, links from summary where rowid in (select rowid from summary_fts where summary_fts match :search) order by uuid limit 101`

  _language_ is a searchable column but now the search string is known as "cauvin language:ita" literally as a search term. columns are not parsed.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/759/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTI4NjA1MzE=
title: Only install osxphotos if running on macOS
state: closed
created_at: 2020-05-05T20:03:26Z
updated_at: 2020-05-05T20:20:05Z
closed_at: 2020-05-05T20:11:23Z
author_association: MEMBER
body: The build is broken right now because you can't `pip install osxphotos` on Ubuntu.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTI4NjA3NTg=
title: Switch CI solution to GitHub Actions with a macOS runner
state: open
created_at: 2020-05-05T20:03:50Z
updated_at: 2020-05-05T23:49:18Z
author_association: MEMBER
body: Refs #17.
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTMwMDIyMjA=
title: apple-photos command should work even if upload has not run
state: closed
created_at: 2020-05-06T02:02:25Z
updated_at: 2020-05-19T20:59:59Z
closed_at: 2020-05-19T20:59:59Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I want people to be able to query their Apple Photos metadata without having to first run `upload` to upload all of their files to their own S3 bucket.

  To do this I can have `apple-photos` calculate SHA256 hashes of each photo if the `uploads` table does not yet exist (or does not contain that photo).

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/19/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTMwMDYzOTM=
title: Ability to serve thumbnailed Apple Photo from its place on disk
state: closed
created_at: 2020-05-06T02:17:50Z
updated_at: 2020-05-25T20:14:22Z
closed_at: 2020-05-25T20:09:41Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  A custom Datasette plugin that can be run locally on a Mac laptop which knows how to serve photos such that they can be seen in the browser.

  _Originally posted by @simonw in https://github.com/dogsheep/photos-to-sqlite/issues/19#issuecomment-624406285_

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTM0MjI2MzY=
title: Way of seeing full schema for a database
state: open
created_at: 2020-05-06T15:46:08Z
updated_at: 2020-05-06T23:49:06Z
author_association: OWNER
type: issue
body:
  I find myself wanting to quickly figure out all of the BLOB columns in a database.

  A `/-/schema` page showing the full schema (actually since it's per-database probably `/dbname/-/schema` or `/-/schema/dbname`) would be really handy.

  It would need to be carefully constructed from various queries against `sqlite_master` - just doing `select * from sqlite_master where type='table'` isn't quite enough because I also want to show indexes, triggers etc.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/760/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTM0NjczODI=
title: Allow-list pragma_table_info(tablename) and similar
state: closed
created_at: 2020-05-06T16:54:14Z
updated_at: 2020-05-07T03:09:05Z
closed_at: 2020-05-06T17:18:38Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It would be great if `pragma_table_info(tablename)` was allowed to be used in queries. See also https://github.com/simonw/til/blob/master/sqlite/list-all-columns-in-a-database.md
  > `select * from pragma_table_info(tablename);` is currently disallowed for user-provided queries via a regex restriction - but could help here too.
  > 
  > https://github.com/simonw/datasette/blob/d349d57cdf3d577afb62bdf784af342a4d5be660/datasette/utils/__init__.py#L174

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/760#issuecomment-624729459_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/761/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTM0OTEzNDI=
title: Experiment with PRAGMA hard_heap_limit
state: open
created_at: 2020-05-06T17:33:23Z
updated_at: 2020-05-07T03:08:44Z
author_association: OWNER
type: issue
body:
  This was added in SQLite 2020-01-22 (3.31.0): https://www.sqlite.org/changes.html#version_3_31_0

  > Add the [sqlite3_hard_heap_limit64()](https://www.sqlite.org/c3ref/hard_heap_limit64.html) interface and the corresponding [PRAGMA hard_heap_limit](https://www.sqlite.org/pragma.html#pragma_hard_heap_limit) command. 

  This sounds like it could be a nice extra safety measure.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/762/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTM3NTUwNDM=
title: Support decimal.Decimal type
state: closed
created_at: 2020-05-07T03:57:19Z
updated_at: 2020-05-11T01:58:20Z
closed_at: 2020-05-11T01:50:11Z
author_association: NONE
type: issue
state_reason: completed
body:
  Decimal types in Postgres cause a failure in db.py data type selection
  ---
  I have a Django app using a MoneyField, which uses a `numeric(14,0)` data type in Postgres (https://www.postgresql.org/docs/9.3/datatype-numeric.html). When attempting to export that table I get the following error:

  ```bash
  $ db-to-sqlite --table isaweb_proposal "postgres://connection" test.db
  ....
      column_type=COLUMN_TYPE_MAPPING[column_type],
  KeyError: <class 'decimal.Decimal'>
  ```

  Looking at `sql_utils.db.py` at 292-ish it's clear that there is no matching type for what I assume SQLAlchemy interprets as Python decimal.Decimal.

  From the [SQLite docs](https://www.sqlite.org/datatype3.html#affinity_name_examples) it looks like DECIMAL in other DBs are considered numeric.

  I'm not quite sure if it's as simple as adding a data type to that list or if there are repercussions beyond it.

  Thanks for a great tool!

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/110/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTM3NzcwNTY=
title: issues foreign key to repo isn't working
state: closed
created_at: 2020-05-07T05:11:48Z
updated_at: 2020-08-18T14:24:46Z
closed_at: 2020-08-18T14:23:56Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://github-to-sqlite.dogsheep.net/github/issues?_facet=repo

  <img width="725" alt="github__issues__2_303_rows_where_sorted_by_updated_at_descending" src="https://user-images.githubusercontent.com/9599/81256810-723be780-8fe6-11ea-9e1f-f37d40f4d3ae.png">

  If the foreign key was working those would be repository names.

  From the schema at the bottom of the page:
  ```
     [repo] TEXT,
  ```
  That's the wrong type and not a foreign key.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/39/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDE1Mjg2MTA1
title: Documentation + improvements for db.execute() and Results class
state: closed
created_at: 2020-05-08T15:16:02Z
updated_at: 2020-06-11T16:05:48Z
closed_at: 2020-05-08T16:05:46Z
author_association: OWNER
pull_request: simonw/datasette/pulls/763
type: pull
body:
  Refs #685

  Still TODO:

  - [x] Implement `results.first()`
  - [x] Implement `results.single_value()`
  - [x] Unit tests for the above

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/763/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTU0NzQ5OTA=
title: bpylist.archiver.CircularReference: archive has a cycle with uid(13)
state: closed
created_at: 2020-05-10T20:58:06Z
updated_at: 2020-12-19T07:44:49Z
closed_at: 2020-05-10T21:57:13Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  ```
  % python -i $(which photos-to-sqlite) apple-photos photos.db                     
  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py", line 611, in place
      return self._place  # pylint: disable=access-member-before-definition
  AttributeError: 'PhotoInfo' object has no attribute '_place'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/bin/photos-to-sqlite", line 11, in <module>
      load_entry_point('photos-to-sqlite', 'console_scripts', 'photos-to-sqlite')()
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/cli.py", line 249, in apple_photos
      photo_row = osxphoto_to_row(sha256, photo)
    File "/Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/utils.py", line 91, in osxphoto_to_row
      place = photo.place
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py", line 614, in place
      self._place = PlaceInfo5(self._info["reverse_geolocation"])
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/placeinfo.py", line 505, in __init__
      self._plrevgeoloc = archiver.unarchive(revgeoloc_bplist)
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 16, in unarchive
      return Unarchive(plist).top_object()
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 256, in top_object
      return self.decode_object(self.top_uid)
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 247, in decode_object
      obj = klass.decode_archive(ArchivedObject(raw_obj, self))
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/placeinfo.py", line 126, in decode_archive
      mapItem = archive.decode("mapItem")
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 140, in decode
      return self._unarchiver.decode_key(self._object, key)
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 216, in decode_key
      return self.decode_object(val)
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 247, in decode_object
      obj = klass.decode_archive(ArchivedObject(raw_obj, self))
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/placeinfo.py", line 180, in decode_archive
      sortedPlaceInfos = archive.decode("sortedPlaceInfos")
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 140, in decode
      return self._unarchiver.decode_key(self._object, key)
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 216, in decode_key
      return self.decode_object(val)
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 247, in decode_object
      obj = klass.decode_archive(ArchivedObject(raw_obj, self))
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 112, in decode_archive
      return [archive._decode_index(index) for index in uids]
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 112, in <listcomp>
      return [archive._decode_index(index) for index in uids]
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 137, in _decode_index
      return self._unarchiver.decode_object(index)
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 247, in decode_object
      obj = klass.decode_archive(ArchivedObject(raw_obj, self))
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/placeinfo.py", line 217, in decode_archive
      placeType = archive.decode("placeType")
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 140, in decode
      return self._unarchiver.decode_key(self._object, key)
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 216, in decode_key
      return self.decode_object(val)
    File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 227, in decode_object
      raise CircularReference(index)
  bpylist.archiver.CircularReference: archive has a cycle with uid(13)
  ```
  In the debugger I traced this back to:
  ```
  178  	    @staticmethod
  179  	    def decode_archive(archive):
  180  ->	        sortedPlaceInfos = archive.decode("sortedPlaceInfos")
  181  	        finalPlaceInfos = archive.decode("finalPlaceInfos")
  182  	        return PLRevGeoMapItem(sortedPlaceInfos, finalPlaceInfos)
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTU0NzcxMzE=
title: sqlite-utils drop-table and drop-view commands
state: closed
created_at: 2020-05-10T21:10:42Z
updated_at: 2020-05-11T01:58:36Z
closed_at: 2020-05-11T00:44:26Z
author_association: OWNER
body: Would be useful to be able to drop views and tables from the CLI.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/111/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTU2MjYxMTg=
title: Try out ExifReader
state: open
created_at: 2020-05-11T06:32:13Z
updated_at: 2020-05-14T05:59:53Z
author_association: MEMBER
type: issue
body:
  https://pypi.org/project/ExifReader/

  New fork that should be able to handle EXIF in HEIC files.

  Forked here: https://github.com/ianare/exif-py/issues/102#issuecomment-626376522

  Refs #3 

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTYwMTI0Mjc=
title: Add PyPI project urls to setup.py
state: closed
created_at: 2020-05-11T16:23:08Z
updated_at: 2020-05-27T20:21:36Z
closed_at: 2020-05-11T18:28:55Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Spotted this example here:
  ```python
      project_urls={
          "Issues": "https://gitlab.com/Cyb3r-Jak3/ExifReader/issues",
          "Source Code": "https://gitlab.com/Cyb3r-Jak3/ExifReader/-/tree/publish",
          "CI": "https://gitlab.com/Cyb3r-Jak3/ExifReader/pipelines",
          "Releases": "https://github.com/Cyb3r-Jak3/ExifReader"
      },
  ```
  Results in this on https://pypi.org/project/ExifReader/

  <img width="333" alt="ExifReader_·_PyPI" src="https://user-images.githubusercontent.com/9599/81585430-03a1b580-9369-11ea-9b58-e2f732cc13f8.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/764/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTYwODcxNDk=
title: publish heroku should default to currently tagged version
state: open
created_at: 2020-05-11T18:24:06Z
updated_at: 2020-05-11T18:25:43Z
author_association: OWNER
type: issue
body:
  Had a report that deploying to Heroku was using the previously installed version of Datasette, not the latest.

  Could be because of this:

  https://github.com/simonw/datasette/blob/af6c6c5d6f929f951c0e63bfd1c82e37a071b50f/datasette/publish/heroku.py#L172-L179

  Heroku documentation recommends pinning to specific versions https://devcenter.heroku.com/articles/python-pip

  So... we could ensure we default to an install value of `["datasette>=current_tag"]`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/765/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTYyNzEyMzY=
title: add_foreign_key(...., ignore=True)
state: closed
created_at: 2020-05-12T00:24:00Z
updated_at: 2020-09-20T22:17:34Z
closed_at: 2020-09-20T22:17:34Z
author_association: OWNER
type: issue
state_reason: completed
body:
  When using this library I often find myself wanting to "add this foreign key, but only if it doesn't exist yet". The `ignore=True` parameter is increasingly being used for this else where in the library (e.g. in `create_view()`).

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/112/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MTczMjM4NzM=
title: Enable wildcard-searches by default
state: open
created_at: 2020-05-13T10:14:48Z
updated_at: 2021-03-05T16:35:21Z
author_association: NONE
type: issue
body:
  Hi Simon.

  It seems that datasette currently has wildcard-searches disabled by default (along with the boolean search-options, NEAR-queries and more, and despite the docs). If I try out the search-url provided in the [docs](https://datasette.readthedocs.io/en/stable/full_text_search.html#the-table-page-and-table-view-api) (https://fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort), it does not handle wildcard-searches, and I'm unable to make it work on my datasette-instance.

  I would argue that wildcard-searches is such a standard query, that it should be enabled by default. Requiring "_searchmode=raw" when using prefix-searches seems unnecessary. Plus: What happens to non-ascii searches when using "_searchmode=raw"? Is the "escape_fts"-function from datasette.utils ignored?


  Thanks!

  /Claus

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/766/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjA5Njk0NjU=
title: Allow to specify a URL fragment for canned queries
state: closed
created_at: 2020-05-19T13:17:42Z
updated_at: 2020-05-27T21:52:25Z
closed_at: 2020-05-27T21:52:25Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Canned queries are very useful to direct users to prepared data and views. I like to use them with charts using datasette-vega a lot, because people get a direct impression at first glance.

  datasette-vega doesn't show up by default though, and users have to click through to it. Also, datasette-vega does not always guess the best way to render columns correctly though, so it would be nice if I could specify a URL fragment in my canned queries to make sure people see what I want them to see.

  My current workaround is to include a fragement link in ``description_html`` and ask people to reload the page, like [here](https://data.rixx.de/songs/show_by_bpm#g.mark=bar&g.x_column=bpm_floor&g.x_type=ordinal&g.y_column=bpm_count&g.y_type=quantitative), which is a bit hacky.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/767/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjEyODA1Mjk=
title: create-subset command for creating a publishable subset of a photos database
state: closed
created_at: 2020-05-19T20:58:20Z
updated_at: 2020-05-19T22:32:48Z
closed_at: 2020-05-19T22:32:37Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I want to share a subset of my photos, without sharing everything. Idea:

      $ photos-to-sqlite create-subset photos.db public.db "select sha256 from ... where ..."

  So the command takes a SQL query that returns sha256 hashes, then creates a new file called `public.db` containing just the data corresponding to those photos.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/23/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjEyODY4NzA=
title: Syntactic sugar for ATTACH DATABASE
state: closed
created_at: 2020-05-19T21:10:00Z
updated_at: 2021-02-19T05:09:12Z
closed_at: 2021-02-19T04:56:36Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://www.sqlite.org/lang_attach.html

  Maybe something like this:
  ```python
  db.attach("other_db", "other_db.db")
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/113/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjEzMjMzNDg=
title: Configurable URL for images
state: open
created_at: 2020-05-19T22:25:56Z
updated_at: 2020-05-20T06:00:29Z
author_association: MEMBER
type: issue
body:
  This is hard-coded at the moment, which is bad:
  https://github.com/dogsheep/photos-to-sqlite/blob/d5d69b9019703c47bc251444838578dd752801e2/photos_to_sqlite/cli.py#L269-L272

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/24/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjEzMzIyNDI=
title: Create a public demo
state: closed
created_at: 2020-05-19T22:47:20Z
updated_at: 2020-05-21T22:26:16Z
closed_at: 2020-05-20T05:54:18Z
author_association: MEMBER
body: So I can show people what this does, using some of my photos.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjE0NDQ3NjM=
title: Rename project to dogsheep-photos
state: closed
created_at: 2020-05-20T04:12:34Z
updated_at: 2020-05-20T04:31:02Z
closed_at: 2020-05-20T04:30:40Z
author_association: MEMBER
body: `photos-to-sqlite` doesn't really capture the full scope of this project anymore.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjE0ODYxMTU=
title: photos_with_apple_metadata view should include labels
state: open
created_at: 2020-05-20T06:06:17Z
updated_at: 2020-05-20T06:06:17Z
author_association: MEMBER
type: issue
body:
  https://dogsheep-photos.dogsheep.net/public/photos_with_apple_metadata?place_city=New+Orleans&_facet=place_city&_facet_array=albums&_facet_array=persons

  Here's one way to add that:
  ```sql
          select
            rowid,
            photo,
            (
              select
                json_group_array(
                  json_object(
                    'label',
                    normalized_string,
                    'href',
                    '/photos/labelled?_hide_sql=1&label=' || normalized_string
                  )
                )
              from
                labels
              where
                labels.uuid = photos_with_apple_metadata.uuid
            ) as labels,
            date,
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/27/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjE5ODk3NDA=
title: table.transform() method for advanced alter table
state: closed
created_at: 2020-05-20T18:20:46Z
updated_at: 2020-09-22T07:51:37Z
closed_at: 2020-09-22T04:20:02Z
author_association: OWNER
type: issue
state_reason: completed
body:
  SQLite's `ALTER TABLE` can only do the following:

  * Rename a table
  * Rename a column
  * Add a column

  Notably, it cannot drop columns - so tricks like "add a float version of this text column, populate it, then drop the old one and rename" won't work.

  The docs here https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes describe a way of implementing full alters safely within a transaction, but it's fiddly.

  1. Create new table
  2. Copy data
  3. Drop old table
  4. Rename new into old 

  It would be great if `sqlite-utils` provided an abstraction to help make these kinds of changes safely.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDIxNDkxODEw
title: Use dirs_exist_ok=True
state: closed
created_at: 2020-05-21T17:53:44Z
updated_at: 2020-05-27T20:21:56Z
closed_at: 2020-05-21T17:53:51Z
author_association: OWNER
pull_request: simonw/datasette/pulls/768
body: Refs #744
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/768/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjQ0OTA5Mjk=
title: Invalid SQL no such table: main.uploads
state: open
created_at: 2020-05-25T21:25:39Z
updated_at: 2020-12-24T22:26:22Z
author_association: NONE
type: issue
body:
  http://127.0.0.1:8001/photos/photos_with_apple_metadata gives "Invalid SQL no such table: main.uploads"

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/28/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDI0MDMyNDQ1
title: Backport of Python 3.8 shutil.copytree
state: closed
created_at: 2020-05-27T18:17:15Z
updated_at: 2020-05-27T20:21:56Z
closed_at: 2020-05-27T18:17:44Z
author_association: OWNER
pull_request: simonw/datasette/pulls/769
body: Closes #744
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/769/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjU5MzAyMDc=
title: register_output_renderer can_render mechanism
state: closed
created_at: 2020-05-27T18:29:14Z
updated_at: 2020-05-28T05:57:16Z
closed_at: 2020-05-28T05:57:16Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I would like is the ability for renderers to opt-in / opt-out of being displayed as options on the page.

  https://www.niche-museums.com/browse/museums for example shows a atom link because the datasette-atom plugin is installed... but clicking it will give you a 400 error because the correct columns are not present.

  <img width="1113" alt="browse__museums__102_rows" src="https://user-images.githubusercontent.com/9599/83058543-62278e80-a00d-11ea-9e04-298e02745118.png">

  Here's the code that passes a list of renderers to the template:

  https://github.com/simonw/datasette/blob/2d099ad9c657d2cab59de91cdb8bfed2da236ef6/datasette/views/base.py#L411-L423

  A renderer is currently defined as a two-key dictionary:
  ```python
  @hookimpl
  def register_output_renderer(datasette):
      return {
          'extension': 'test',
          'callback': render_test
      }
  ```
  I can add a third key, `"should_suggest"` which is a function that returns `True` or `False` for a given query. If that key is missing it is assumed to return `True`.

  One catch: what arguments should be passed to the `should_suggest(...)` function?

  UPDATE: now calling it `can_render` instead.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/581#issuecomment-634856748_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/770/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjU5ODAzMTc=
title: Unit test that checks that all plugin hooks have corresponding unit tests
state: closed
created_at: 2020-05-27T19:42:35Z
updated_at: 2020-05-27T20:21:36Z
closed_at: 2020-05-27T20:17:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Turns out some hooks are missing unit test coverage: https://github.com/simonw/datasette/issues/581#issuecomment-634893744_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/771/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDI0MDg1MjY0
title: Test that plugin hooks are unit tested
state: closed
created_at: 2020-05-27T20:01:32Z
updated_at: 2020-05-27T20:21:56Z
closed_at: 2020-05-27T20:16:03Z
author_association: OWNER
pull_request: simonw/datasette/pulls/772
body: Refs #771
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/772/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjYwMDE1MDE=
title: All plugin hooks should have unit tests
state: closed
created_at: 2020-05-27T20:17:41Z
updated_at: 2020-05-28T04:12:11Z
closed_at: 2020-05-28T04:09:25Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Four hooks currently missing tests:

  - [x] prepare_jinja2_environment
  - [x] publish_subcommand
  - [x] register_facet_classes
  - [x] register_output_renderer

  ```
  $ pytest -k test_plugin_hooks_have_tests -vv
  ====================================== test session starts ======================================
  platform darwin -- Python 3.7.7, pytest-5.2.4, py-1.8.1, pluggy-0.13.1 -- /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/python
  cachedir: .pytest_cache
  rootdir: /Users/simon/Dropbox/Development/datasette, inifile: pytest.ini
  plugins: asyncio-0.10.0
  collected 486 items / 475 deselected / 11 selected                                              

  tests/test_plugins.py::test_plugin_hooks_have_tests[asgi_wrapper] XPASS                   [  9%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[extra_body_script] XPASS              [ 18%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[extra_css_urls] XPASS                 [ 27%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[extra_js_urls] XPASS                  [ 36%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[extra_template_vars] XPASS            [ 45%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[prepare_connection] XPASS             [ 54%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[prepare_jinja2_environment] XFAIL     [ 63%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[publish_subcommand] XFAIL             [ 72%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[register_facet_classes] XFAIL         [ 81%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[register_output_renderer] XFAIL       [ 90%]
  tests/test_plugins.py::test_plugin_hooks_have_tests[render_cell] XPASS                    [100%]

  ========================= 475 deselected, 4 xfailed, 7 xpassed in 1.70s =========================

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/771#issuecomment-634915104_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/773/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjYwNzg1MjE=
title: Consolidate request.raw_args and request.args
state: closed
created_at: 2020-05-27T22:30:59Z
updated_at: 2020-05-29T23:27:35Z
closed_at: 2020-05-29T23:22:38Z
author_association: OWNER
type: issue
state_reason: completed
body:
  `request.raw_args` is not documented, and I'd like to remove it entirely.
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/706#issuecomment-634975252_

  I use it in a few places in other projects though, so I'll have to fix those first: https://github.com/search?q=user%3Asimonw+raw_args&type=Code

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/774/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjYxMzEzMDk=
title: Move test plugins into datasette/tests/plugins/ directory
state: closed
created_at: 2020-05-28T00:46:58Z
updated_at: 2020-05-28T00:57:31Z
closed_at: 2020-05-28T00:57:31Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Right now the plugins used during test runs are baked into strings. It would be nicer if they were actual files on disk.

  Will make #581 easier to write tests for.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/775/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjYxNjM5NzQ=
title: register_output_renderer render callback should be optionally awaitable
state: closed
created_at: 2020-05-28T02:26:29Z
updated_at: 2020-05-28T02:43:36Z
closed_at: 2020-05-28T02:43:36Z
author_association: OWNER
type: issue
state_reason: completed
body:
  In #581 I made a bunch of improvements to this, including making `datasette` available to it so it could execute queries.

  But... it needs to be able to `await` in order to do that. Which means it should be optionally-awaitable.

  Original idea here: https://github.com/simonw/datasette/issues/645#issuecomment-560036740

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/776/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjYxNzEyNDI=
title: Error pages not correctly loading CSS
state: closed
created_at: 2020-05-28T02:47:52Z
updated_at: 2020-06-09T00:35:29Z
closed_at: 2020-06-09T00:35:29Z
author_association: OWNER
type: issue
state_reason: completed
body:
  e.g. https://latest.datasette.io/fixtures/compound_three_primary_keys.tsv?_size=max

  <img width="856" alt="Error_404_and_All_plugin_hooks_should_have_unit_tests_·_Issue__773_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/83092786-e5b79e80-a052-11ea-9cd7-6a1c01eb08e7.png">

  The HTML starts like this:

  ```html
  <!DOCTYPE html>
  <html>
  <head>
      <title>Error 404</title>
      <link rel="stylesheet" href="-/static/app.css?">
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/777/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjYyMTE2NTg=
title: Ability to configure keyset pagination for views and queries
state: open
created_at: 2020-05-28T04:48:56Z
updated_at: 2020-10-02T02:26:25Z
author_association: OWNER
type: issue
body:
  Currently views offer pagination, but it uses offset/limit - e.g. https://latest.datasette.io/fixtures/paginated_view?_next=100

  This means pagination will perform poorly on deeper pages.

  If a view is based on a table that has a primary key it should be possible to configure efficient keyset pagination that works the same way that table pagination works.

  This may be as simple as configuring a column that can be treated as a "primary key" for the purpose of pagination using `metadata.json` - or with a `?_view_pk=colname` querystring argument.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/778/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjY1ODI2NTc=
title: Make human_description_en explicitly available to output renderers
state: open
created_at: 2020-05-28T14:59:54Z
updated_at: 2020-05-28T14:59:54Z
author_association: OWNER
type: issue
body:
  `datasette-atom` uses this:

  https://github.com/simonw/datasette-atom/blob/df98a6c43a443224b6cd232f84703ec297ef046b/datasette_atom/__init__.py#L36-L37
  ```python
      if data.get("human_description_en"):
          title += ": " + data["human_description_en"]
  ```
  It's a nice way to generate a useful title for a filtered table.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/779/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjY1OTM0MDI=
title: Internals documentation for datasette.metadata() method
state: open
created_at: 2020-05-28T15:14:22Z
updated_at: 2022-03-15T20:50:34Z
author_association: OWNER
type: issue
body:
  https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/datasette/app.py#L297-L328

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/780/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjY2NjMxMTk=
title: request.url and request.scheme should obey force_https_urls config setting
state: closed
created_at: 2020-05-28T16:54:47Z
updated_at: 2020-05-28T17:39:54Z
closed_at: 2020-05-28T17:10:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'm trying to get the https://www.niche-museums.com/browse/feed.atom feed to validate and I git this from https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fwww.niche-museums.com%2Fbrowse%2Ffeed.atom

  > This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.
  > 
  > [line 6](https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fwww.niche-museums.com%2Fbrowse%2Ffeed.atom#l6), column 73: Self reference doesn't match document location [[help](https://validator.w3.org/feed/docs/warning/SelfDoesntMatchLocation.html "more information about this error")]
  > 
  >     <link href="http://www.niche-museums.com/browse/feed.atom" rel="self"/>

  I tried to fix this using `force_https_urls` ([commit](https://github.com/simonw/museums/commit/5dc8e2c717c59f9e949b65e47a59878e01f929e4)) but it didn't work - because that setting isn't respected by the Request class:

  https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/datasette/utils/asgi.py#L15-L32

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/781/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mjc3OTQ4Nzk=
title: Redesign default .json format
state: closed
created_at: 2020-05-30T18:47:07Z
updated_at: 2023-08-10T00:07:17Z
closed_at: 2023-08-10T00:07:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The default JSON just isn't right. I find myself using `?_shape=array` for almost everything I build against the API.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/782/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDI1NTMxMjA1
title: Authentication: plugin hooks plus default --root auth mechanism
state: closed
created_at: 2020-05-30T22:25:47Z
updated_at: 2020-06-01T01:16:44Z
closed_at: 2020-06-01T01:16:43Z
author_association: OWNER
pull_request: simonw/datasette/pulls/783
body: See #699
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/783/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjgwMDM3MDc=
title: Ability to sign in to Datasette as a root account
state: closed
created_at: 2020-05-31T17:10:15Z
updated_at: 2020-07-06T19:31:53Z
closed_at: 2020-06-01T01:18:20Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I'm going to draw the line here: default Datasette supports authentication but only for a single user account ("admin"). Plugins can then add support for multiple user accounts, social auth, SSO etc.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/699#issuecomment-636498770_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/784/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjgwMjUxMDA=
title: Datasette secret mechanism - initially for signed cookies
state: closed
created_at: 2020-05-31T19:14:52Z
updated_at: 2020-06-06T00:43:40Z
closed_at: 2020-06-01T00:18:40Z
author_association: OWNER
type: issue
state_reason: completed
body:
  See comment in https://github.com/simonw/datasette/issues/784#issuecomment-636514974

  Datasette needs to be able to set signed cookies - which means it needs a mechanism for safely handling a signing secret.

  Since Datasette is a long-running process the default behaviour here can be to create a random secret on startup. This means that if the server restarts any signed cookies will be invalidated.

  If the user wants a persistent secret they'll have to generate it themselves - maybe by setting an environment variable?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/785/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjgwODc5NzE=
title: Documentation page describing Datasette's authentication system
state: closed
created_at: 2020-06-01T01:10:06Z
updated_at: 2020-06-06T19:40:20Z
closed_at: 2020-06-06T19:40:20Z
author_association: OWNER
type: issue
state_reason: completed
body:
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/699#issuecomment-636562999_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/786/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjgwODkzMTg=
title: "datasette publish" should bake in a random --secret
state: closed
created_at: 2020-06-01T01:15:26Z
updated_at: 2020-06-11T16:02:05Z
closed_at: 2020-06-11T16:02:05Z
author_association: OWNER
type: issue
state_reason: completed
body:
  To allow signed cookies etc to work reliably (see #785) all of the `datasette publish` commands should generate a random secret on publish and bake it into the configuration - probably by setting the `DATASETTE_SECRET` environment variable.

  - [ ] Cloud Run
  - [ ] Heroku
  - [ ] https://github.com/simonw/datasette-publish-now
  - [ ] https://github.com/simonw/datasette-publish-fly

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/787/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjgxMjEyMzQ=
title: /-/permissions debugging tool
state: closed
created_at: 2020-06-01T03:13:47Z
updated_at: 2020-06-06T00:43:40Z
closed_at: 2020-06-01T05:01:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Debugging tool idea: `/-/permissions` page which shows you the actor and lets you type in the strings for `action`, `resource_type` and `resource_identifier` - then shows you EVERY plugin hook that would have executed and what it would have said, plus when the chain would have terminated.
  >
  > Bonus: if you're logged in as the `root` user (or a user that matches some kind of permission check, maybe a check for `permissions_debug`) you get to see a rolling log of the last 30 permission checks and what the results were across the whole of Datasette. This should make figuring out permissions policies a whole lot easier.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/699#issuecomment-636576603_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/788/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MjgxNTY1Mjc=
title: Mechanism for enabling pluggy tracing
state: open
created_at: 2020-06-01T05:10:14Z
updated_at: 2020-06-01T05:11:03Z
author_association: OWNER
type: issue
body:
  Could be useful for debugging plugins: https://pluggy.readthedocs.io/en/latest/#call-tracing

  I tried this out by adding these two lines in `plugins.py`:
  ```python
  pm = pluggy.PluginManager("datasette")
  pm.add_hookspecs(hookspecs)
  # Added these:
  pm.trace.root.setwriter(print)
  pm.enable_tracing()
  ```
  Output looked something like this:
  ```
  INFO:     127.0.0.1:52724 - "GET /-/-/static/app.css HTTP/1.1" 404 Not Found
    actor_from_request [hook]
        datasette: <datasette.app.Datasette object at 0x106277ad0>
        request: <datasette.utils.asgi.Request object at 0x106550a50>

    finish actor_from_request --> [] [hook]

    extra_body_script [hook]
        template: show_json.html
        database: None
        table: None
        view_name: json_data
        datasette: <datasette.app.Datasette object at 0x106277ad0>

    finish extra_body_script --> [] [hook]

    extra_template_vars [hook]
        template: show_json.html
        database: None
        table: None
        view_name: json_data
        request: <datasette.utils.asgi.Request object at 0x1065504d0>
        datasette: <datasette.app.Datasette object at 0x106277ad0>

    finish extra_template_vars --> [] [hook]

    extra_css_urls [hook]
        template: show_json.html
        database: None
        table: None
        datasette: <datasette.app.Datasette object at 0x106277ad0>

    finish extra_css_urls --> [] [hook]

    extra_js_urls [hook]
        template: show_json.html
        database: None
        table: None
        datasette: <datasette.app.Datasette object at 0x106277ad0>

    finish extra_js_urls --> [] [hook]

  INFO:     127.0.0.1:52724 - "GET /-/actor HTTP/1.1" 200 OK
    actor_from_request [hook]
        datasette: <datasette.app.Datasette object at 0x106277ad0>
        request: <datasette.utils.asgi.Request object at 0x1065500d0>

    finish actor_from_request --> [] [hook]
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/789/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mjg0OTkwODY=
title: "flash messages" mechanism
state: closed
created_at: 2020-06-01T14:55:44Z
updated_at: 2020-06-08T19:33:59Z
closed_at: 2020-06-02T21:14:03Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Passing `?_success` like this isn't necessarily the best approach. Potential improvements include:
  > 
  > - Signing this message so it can't be tampered with (I could generate a signing secret on startup)
  > - Using a cookie with a temporary flash message in it instead
  > - Using HTML5 history API to remove the `?_success=` from the URL bar when the user lands on the page
  > 
  > If I add an option to redirect the user to another page after success I may need a mechanism to show a flash message on that page as well, in which case I'll need a general flash message solution that works for any page.

  _Originally posted by @simonw in https://github.com/simonw/datasette/pull/703_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/790/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mjg1NzI3MTY=
title: Tutorial: building a something-interesting with writable canned queries
state: open
created_at: 2020-06-01T16:32:05Z
updated_at: 2020-10-10T23:34:42Z
author_association: OWNER
body: Initial idea: TODO list, as a tutorial for #698 writable canned queries.
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/791/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mjk0NTk2Mzc=
title: Replace response.body.decode("utf8") with response.text in tests
state: closed
created_at: 2020-06-02T19:32:24Z
updated_at: 2020-06-02T21:29:58Z
closed_at: 2020-06-02T21:29:58Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Make use of the `response.text` property to clean up the tests a tiny bit:

  https://github.com/simonw/datasette/blob/57cf5139c552cb7feab9947daa949ca434cc0a66/tests/fixtures.py#L26-L38

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/792/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mjk0NzM4Mjc=
title: Set up a demo
state: open
created_at: 2020-06-02T19:56:49Z
updated_at: 2020-09-01T06:18:43Z
author_association: NONE
type: issue
body:
  First off, thanks for open sourcing this application! This is a suggestion to increase the amount of people that would make use of it: an example in the readme file would help.

  Currently, users have to clone the app, install it, authorize through pocket, run a command, an then find out if this application does what they hope it does.

  Another possibility is to add a file `example-output.db`, containing one (mock) Pocket article.

  Keep up the good work!

reactions:
  {"url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mjk1MjQyMDU=
title: CSRF protection for /-/messages tool and writable canned queries
state: closed
created_at: 2020-06-02T21:22:21Z
updated_at: 2020-06-06T00:43:41Z
closed_at: 2020-06-05T19:05:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > The `/-/messages` debug tool will need CSRF protection or people will be able to add messages using a hidden form on another website.
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/790#issuecomment-637790860_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/793/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mjk1MzU2Njk=
title: Show hooks implemented by each plugin on /-/plugins
state: closed
created_at: 2020-06-02T21:44:38Z
updated_at: 2020-06-02T22:30:17Z
closed_at: 2020-06-02T21:50:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  e.g.
  ```json
      {
          "name": "qs_actor.py",
          "static": false,
          "templates": false,
          "version": null,
          "hooks": [
              "actor_from_request"
          ]
      }
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/794/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mjk1NDEzOTU=
title: response.set_cookie() method
state: closed
created_at: 2020-06-02T21:57:05Z
updated_at: 2020-06-09T22:33:33Z
closed_at: 2020-06-09T22:19:48Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Mainly to clean up this code:

  https://github.com/simonw/datasette/blob/4fa7cf68536628344356d3ef8c92c25c249067a0/datasette/app.py#L439-L454

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/795/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDI2ODkxNDcx
title: New WIP writable canned queries
state: closed
created_at: 2020-06-03T00:08:00Z
updated_at: 2020-06-03T15:16:52Z
closed_at: 2020-06-03T15:16:50Z
author_association: OWNER
pull_request: simonw/datasette/pulls/796
type: pull
body:
  Refs #698. Replaces #703

  Still todo:

  - [x] Unit tests
  - ~~Figure out `.json` mode~~
  - [x] Flash message solution
  - ~~CSRF protection~~
  - [x] Better error message display on errors
  - [x] Documentation
  - ~~Maybe widgets?~~ I'll do these later

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/796/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzAxMjAyMzU=
title: Documentation for new "params" setting for canned queries
state: closed
created_at: 2020-06-03T15:55:11Z
updated_at: 2020-06-09T04:00:40Z
closed_at: 2020-06-03T21:04:51Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Added here: https://github.com/simonw/datasette/commit/aa82d0370463580f2cb10d9617f1bcbe45cc994a#diff-5e0ffd62fced7d46339b9b2cd167c2f9R236

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/797/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDI4MjEyNDIx
title: CSRF protection
state: closed
created_at: 2020-06-05T04:22:35Z
updated_at: 2020-06-06T00:43:41Z
closed_at: 2020-06-05T19:05:58Z
author_association: OWNER
pull_request: simonw/datasette/pulls/798
body: Refs #793
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/798/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzE3ODk0MjI=
title: TestResponse needs to handle multiple set-cookie headers
state: closed
created_at: 2020-06-05T17:39:52Z
updated_at: 2020-06-05T18:34:10Z
closed_at: 2020-06-05T18:34:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Seeing this test failure on #798:
  ```
  _______________________ test_auth_token _______________________
  app_client = <tests.fixtures.TestClient object at 0x11285c910>
      def test_auth_token(app_client):
          "The /-/auth-token endpoint sets the correct cookie"
          assert app_client.ds._root_token is not None
          path = "/-/auth-token?token={}".format(app_client.ds._root_token)
          response = app_client.get(path, allow_redirects=False,)
          assert 302 == response.status
          assert "/" == response.headers["Location"]
  >       assert {"id": "root"} == app_client.ds.unsign(response.cookies["ds_actor"], "actor")
  E       KeyError: 'ds_actor'
  datasette/tests/test_auth.py:12: KeyError
  ```
  It looks like that's happening because the ASGI middleware is adding another set-cookie header - but those two set-cookie headers are combined into one when the TestResponse is constructed:

  https://github.com/simonw/datasette/blob/0c064c5fe220b7b3d8dcf85b02b4e60452c47232/tests/fixtures.py#L113-L127

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/799/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzE5MzE0MDg=
title: Canned query permissions mechanism
state: closed
created_at: 2020-06-05T20:28:21Z
updated_at: 2020-06-07T16:22:53Z
closed_at: 2020-06-07T16:22:53Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Idea: default is anyone can execute a query.
  > 
  > Or you can specify the following:
  > 
  > ```json
  > 
  > {
  >     "databases": {
  >        "my-database": {
  >            "queries": {
  >                "add_twitter_handle": {
  >                    "sql": "insert into twitter_handles (username) values (:username)",
  >                    "write": true,
  >                    "allow": {
  >                        "id": ["simon"],
  >                        "role": ["staff"]
  >                    }
  >                }
  >            }
  >        }
  >     }
  > }
  > ```
  > These get matched against the actor JSON. If any of the fields in any of the keys of `"allow"` match a key on the actor, the query is allowed.
  > 
  > `"id": "*"` matches any actor with an `id` key.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/698#issuecomment-639784651_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/800/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzE5MzI5MjY=
title: allow_by_query setting for configuring permissions with a SQL statement
state: closed
created_at: 2020-06-05T20:30:19Z
updated_at: 2020-06-11T18:58:56Z
closed_at: 2020-06-11T18:58:49Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Idea: an `"allow_sql"` key with a SQL query that gets passed the actor JSON as `:actor` and can extract the relevant keys from it and return 1 or 0.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/698#issuecomment-639787304_

  See also #800

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/801/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzIwNTY4MjU=
title: "datasette plugins" command is broken
state: closed
created_at: 2020-06-05T23:33:01Z
updated_at: 2020-06-05T23:46:43Z
closed_at: 2020-06-05T23:46:43Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I broke it in https://github.com/simonw/datasette/commit/a7137dfe069e5fceca56f78631baebd4a6a19967 - and it turns out there was no test coverage so I didn't realize it was broken.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/802/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDI5MzY2NjQx
title: Canned query permissions
state: closed
created_at: 2020-06-06T18:20:00Z
updated_at: 2020-06-06T19:40:21Z
closed_at: 2020-06-06T19:40:20Z
author_association: OWNER
pull_request: simonw/datasette/pulls/803
body: Refs #800. Closes #786
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/803/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzI2NzM5NzI=
title: python tests/fixtures.py command has a bug
state: closed
created_at: 2020-06-06T19:17:36Z
updated_at: 2020-06-09T20:01:30Z
closed_at: 2020-06-09T19:58:34Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This command is meant to write out `fixtures.db`, `metadata.json` and a plugins directory:
  ```
  $ python tests/fixtures.py /tmp/fixtures.db /tmp/metadata.json /tmp/plugins/
  Test tables written to /tmp/fixtures.db
  - metadata written to /tmp/metadata.json
  Traceback (most recent call last):
    File "tests/fixtures.py", line 833, in <module>
      ("my_plugin.py", PLUGIN1),
  NameError: name 'PLUGIN1' is not defined
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/804/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzI3MjQxNTQ=
title: Writable canned queries live demo on Glitch
state: open
created_at: 2020-06-06T20:52:13Z
updated_at: 2020-07-01T22:44:01Z
author_association: OWNER
type: issue
body:
  Needs to run somewhere with a mutable disk drive, so not Cloud Run or Heroku or Vercel.

  I think I'll put it on Glitch.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/805/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzI3NTM4NTE=
title: Release Datasette 0.44
state: closed
created_at: 2020-06-06T21:49:52Z
updated_at: 2020-06-12T01:20:03Z
closed_at: 2020-06-12T01:20:03Z
author_association: OWNER
type: issue
state_reason: completed
body:
  See also [milestone](https://github.com/simonw/datasette/milestone/14). This is a pretty big release: flash messaging, writable canned queries, authentication and permissions!

  I'll want to ship some plugin releases in conjunction with this - `datasette-auth-github` for example.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/806/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzI4NDMwMzA=
title: Ability to ship alpha and beta releases
state: closed
created_at: 2020-06-07T00:12:55Z
updated_at: 2020-06-18T21:41:16Z
closed_at: 2020-06-18T21:41:16Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'd like to be able to ship alphas and betas to PyPI so in-development plugins can depend on them and help test unreleased plugin hooks.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/807/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzI5MTg3OTk=
title: Permission check for every view in Datasette (plus docs)
state: closed
created_at: 2020-06-07T01:59:23Z
updated_at: 2020-06-07T05:30:49Z
closed_at: 2020-06-07T05:30:49Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Every view in Datasette should perform a permission check to see if the current user/actor is allowed to view that page.

  This permission check will default to allowed, but having this check will allow plugins to lock down access selectively or even to everything in a Datasette instance.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/808/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDI5NjEzODkz
title: Publish secrets
state: closed
created_at: 2020-06-07T02:00:31Z
updated_at: 2020-06-11T16:02:13Z
closed_at: 2020-06-11T16:02:03Z
author_association: OWNER
pull_request: simonw/datasette/pulls/809
type: pull
body:
  Refs #787. Will need quite a bit of manual testing since this involves code which runs against Heroku and Cloud Run.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/809/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzMwNjYxMTQ=
title: Refactor permission check for canned query
state: closed
created_at: 2020-06-07T05:33:05Z
updated_at: 2020-06-07T17:03:15Z
closed_at: 2020-06-07T17:03:15Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This code here (TODO is follow-on from #808).

  https://github.com/simonw/datasette/blob/86dec9e8fffd6c4efec928ae9b5713748dec7e74/datasette/views/database.py#L133-L142

  I can improve this with extra code in https://github.com/simonw/datasette/blob/86dec9e8fffd6c4efec928ae9b5713748dec7e74/datasette/default_permissions.py

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/810/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzM1Nzg3Njk=
title: Support "allow" block on root, databases and tables, not just queries
state: closed
created_at: 2020-06-07T17:01:09Z
updated_at: 2020-06-08T19:34:00Z
closed_at: 2020-06-08T19:32:36Z
author_association: OWNER
type: issue
state_reason: completed
body:
  No reason not to expand the "allow" mechanism [described here](https://github.com/simonw/datasette/blob/86dec9e8fffd6c4efec928ae9b5713748dec7e74/docs/authentication.rst#permissions-for-canned-queries) to the root of `metadata.json` plus to databases and tables.

  Refs #810 and #800.

  ```json
  {
      "databases": {
          "mydatabase": {
              "allow": {
                  "id": ["root"]
              }
          }
      }
  }
  ```

  TODO:

  - [x] Instance level
  - [x] Database level
  - [x] Table level
  - [x] Query level
  - [x] Affects list of queries
  - [x] Affects list of tables on database page
  - [x] Affects truncated list of tables on index page
  - [x] Affects list of SQL views on database page
  - [x] Affects list of databases on index page
  - [x] Show 🔒 in header on index page for private instances
  - [x] Show 🔒 in header on private database page
  - [x] Show 🔒 in header on private table page
  - [x] Show 🔒 in header on private query page
  - [x] Move `assert_permissions_checked()` calls from `test_html.py` to `test_permissions.py`
  - [x] Update documentation

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/811/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzQxMTI2MDc=
title: Ability to customize what happens when a view permission fails
state: closed
created_at: 2020-06-08T04:26:14Z
updated_at: 2020-07-01T04:17:46Z
closed_at: 2020-07-01T04:17:45Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Currently view permission failures raise a `Forbidden` error which is transformed into a 403.

  It would be good if this page could offer a way forward - maybe just by linking to (or redirecting to) a login screen. This behaviour will vary based on authentication plugins, so a new plugin hook is probably the best way to do this.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/812/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzQxMzk4NDg=
title: Mechanism for specifying allow_sql permission in metadata.json
state: closed
created_at: 2020-06-08T04:57:19Z
updated_at: 2020-06-09T00:09:57Z
closed_at: 2020-06-09T00:07:19Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Split from #811. It would be useful if finely-grained permissions configured in `metadata.json` could be used to specify if a user is allowed to execute arbitrary SQL queries.

  We have a permission check call for this already: https://github.com/simonw/datasette/blob/9397d718345c4b35d2a5c55bfcbd1468876b5ab9/datasette/views/database.py#L159

  But there's currently no way to implement this check without writing a plugin.

  I think a `"allow_sql": {...}` block at the database level in `metadata.json` (sibling to the current `"allow"` block for that database implemented in #811) would be a good option for this.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/813/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzQ2NTEwNzk=
title: Remove --debug option from datasette serve
state: closed
created_at: 2020-06-08T14:10:14Z
updated_at: 2020-10-23T19:44:04Z
closed_at: 2020-10-10T23:39:43Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It doesn't appear to do anything useful at all:

  https://github.com/simonw/datasette/blob/f786033a5f0098371cb1df1ce83959b27c588115/datasette/cli.py#L251-L253

  https://github.com/simonw/datasette/blob/f786033a5f0098371cb1df1ce83959b27c588115/datasette/cli.py#L365-L367

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/814/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzQ2NjM1MDU=
title: Group permission checks by request on /-/permissions debug page
state: open
created_at: 2020-06-08T14:25:23Z
updated_at: 2020-12-17T22:06:48Z
author_association: OWNER
type: issue
body:
  Now that we're making a LOT more permission checks (on the DB index page we do a check for every listed table for example) the `/-/permissions` page gets filled up pretty quickly.

  Can make this more readable by grouping permission checks by request. Have most recent request at the top of the page but the permission requests within that page sorted chronologically by most recent last.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/815/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzQ3ODM1NzM=
title: Come up with a new example for extra_template_vars plugin
state: closed
created_at: 2020-06-08T16:57:59Z
updated_at: 2020-06-08T19:06:44Z
closed_at: 2020-06-08T19:06:11Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This example is obsolete, it's from a time before `request.actor` and authentication as a built-in concept (#699):
  https://github.com/simonw/datasette/blob/0c064c5fe220b7b3d8dcf85b02b4e60452c47232/docs/plugins.rst#L696-L700

  https://github.com/simonw/datasette/blob/0c064c5fe220b7b3d8dcf85b02b4e60452c47232/docs/plugins.rst#extra_template_varstemplate-database-table-view_name-request-datasette

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/816/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzQ4NDQ2MzQ=
title: Drop resource_type from permission_allowed system
state: closed
created_at: 2020-06-08T18:41:37Z
updated_at: 2020-06-08T19:00:12Z
closed_at: 2020-06-08T19:00:12Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Current signature:

      permission_allowed(datasette, actor, action, resource_type, resource_identifier)

  It turns out the `resource_type` is always the same thing for any given action, so it's not actually useful. I'm going to drop it.

  New signature will be:

      permission_allowed(datasette, actor, action, resource)

  Refs #811.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/817/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzQ5MTcwODg=
title: Example permissions plugin
state: closed
created_at: 2020-06-08T20:35:56Z
updated_at: 2020-06-11T05:40:07Z
closed_at: 2020-06-11T05:40:07Z
author_association: OWNER
body: To show how they work. Also useful to confirm how they interact with the default permissions.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/818/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDMxNDc4NzI0
title: register_routes() plugin hook
state: closed
created_at: 2020-06-09T01:20:44Z
updated_at: 2020-06-09T03:12:08Z
closed_at: 2020-06-09T03:12:07Z
author_association: OWNER
pull_request: simonw/datasette/pulls/819
body: Refs #215
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/819/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzUwNDkyOTY=
title: Idea: Plugin hook for registering canned queries
state: closed
created_at: 2020-06-09T01:58:21Z
updated_at: 2020-06-18T17:58:02Z
closed_at: 2020-06-18T17:58:02Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Thought of this while thinking about possible permissions plugins (#818).

  Imagine an API key plugin which allows access for API keys. It could let users register new API keys by providing a writable canned query for writing to the `api_keys` table.

  To do this the plugin needs to register the query. At the moment queries have to be registered in `metadata.json` - a plugin hook for registering additional queries could help solve this.

  One challenge: how does the plugin know which named database the query should be registered for?

  It could default to the first attached database and allow users to optionally tell the plugin "actually use this named database instead" in plugin configuration.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/820/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzUwNzYwNjY=
title: Add Response class to internals documentation
state: closed
created_at: 2020-06-09T03:11:06Z
updated_at: 2020-06-09T03:32:16Z
closed_at: 2020-06-09T03:32:16Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I'll need to add documentation of the `Response` object (and `Response.html()` and `Response.text()` class methods - I should add `Response.json()` too) to the internals page https://datasette.readthedocs.io/en/stable/internals.html

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/215#issuecomment-640971470_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/821/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzUwNzc2NTY=
title: request.url_vars helper property
state: closed
created_at: 2020-06-09T03:15:53Z
updated_at: 2020-06-09T03:40:07Z
closed_at: 2020-06-09T03:40:06Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This example: https://github.com/simonw/datasette/blob/f5e79adf26d0daa3831e3fba022f1b749a9efdee/docs/plugins.rst#register_routes

  ```python
  from datasette.utils.asgi import Response
  import html


  async def hello_from(scope):
      name = scope["url_route"]["kwargs"]["name"]
      return Response.html("Hello from {}".format(
          html.escape(name)
      ))


  @hookimpl
  def register_routes():
      return [
          (r"^/hello-from/(?P<name>.*)$"), hello_from)
      ]
  ```
  Would be nicer if you could easily get `scope["url_route"]["kwargs"]["name"]` directly from the request object, without looking at the `scope`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/822/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzUxMDczOTM=
title: Documentation is inconsistent about "id" as required field on actor
state: closed
created_at: 2020-06-09T04:47:58Z
updated_at: 2020-06-09T14:58:36Z
closed_at: 2020-06-09T14:58:19Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Docs at https://github.com/simonw/datasette/blob/5a6a73e3190cac103906b479d56129413e5ef190/docs/authentication.rst#actors say:

  > The only required field in an actor is `"id"`, which must be a string.

  But the example here returns `{"token": token}`:

  ```python
  @hookimpl
  def actor_from_request(datasette, request):
      async def inner():
          token = request.args.get("_token")
          if not token:
              return None
          # Look up ?_token=xxx in sessions table
          result = await datasette.get_database().execute(
              "select count(*) from sessions where token = ?", [token]
          )
          if result.first()[0]:
              return {"token": token}
          else:
              return None

      return inner
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/823/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzUxMDgwNzQ=
title: Example authentication plugin
state: closed
created_at: 2020-06-09T04:49:53Z
updated_at: 2020-06-12T00:11:51Z
closed_at: 2020-06-12T00:11:50Z
author_association: OWNER
body: https://github.com/simonw/datasette-auth-github/issues/62 will work for this.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/824/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzUxNDc3MTY=
title: Way to enable a default=False permission for anonymous users
state: closed
created_at: 2020-06-09T06:26:27Z
updated_at: 2020-06-09T17:19:19Z
closed_at: 2020-06-09T17:01:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'd like plugins to be able to ship with a default that says "anonymous users cannot do this", but allow site administrators to over-ride that such that anonymous users can use the feature after all.

  This is tricky because right now the anonymous user doesn't have an actor dictionary at all, so there's no key to match to an allow block.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/825/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzU1MTkzNTg=
title: Document the ds_actor signed cookie
state: closed
created_at: 2020-06-09T15:06:52Z
updated_at: 2020-06-09T22:33:12Z
closed_at: 2020-06-09T22:32:31Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Most authentication plugins (https://github.com/simonw/datasette-auth-github for example) are likely to work by setting the `ds_actor` signed cookie, which is already magically decoded and supported by default Datasette here:

  https://github.com/simonw/datasette/blob/4fa7cf68536628344356d3ef8c92c25c249067a0/datasette/actor_auth_cookie.py#L1-L13

  I should document this.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/826/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzU2OTY0MDA=
title: Document CSRF protection (for plugins)
state: closed
created_at: 2020-06-09T19:19:10Z
updated_at: 2020-06-09T19:38:30Z
closed_at: 2020-06-09T19:35:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Plugin authors need to know that if they want to POST a form they should include this:
  ```html+jinja
  <input type="hidden" name="csrftoken" value="{{ csrftoken() }}">
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/827/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzU5MTQ4MjI=
title: Horizontal scrollbar on changelog page on mobile
state: closed
created_at: 2020-06-10T04:18:54Z
updated_at: 2020-06-10T04:28:17Z
closed_at: 2020-06-10T04:28:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  You can scroll sideways on that page and it looks bad:

  <img width="419" alt="Changelog_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/84226483-b8c5ab80-aa96-11ea-8fa7-db0c37ab9441.png">

  The cause is these long links:

  <img width="452" alt="Changelog_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/84226511-d09d2f80-aa96-11ea-8bd9-891c07098516.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/828/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzY0MjY1MzA=
title: Ability to set ds_actor cookie such that it expires
state: closed
created_at: 2020-06-10T17:31:40Z
updated_at: 2020-06-10T19:41:35Z
closed_at: 2020-06-10T19:40:05Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I need this for `datasette-auth-github`: https://github.com/simonw/datasette-auth-github/issues/62#issuecomment-642152076

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/829/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzY1MTE2ODM=
title: Redesign register_facet_classes plugin hook
state: open
created_at: 2020-06-10T20:03:27Z
updated_at: 2021-12-16T19:58:22Z
author_association: OWNER
type: issue
body:
  Nothing uses this plugin hook yet, so the design is not yet proven.

  I'm going to build a real plugin against it and use that process to inform any design changes that may need to be made.

  I'll add a warning about this to the documentation.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/830/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzY2MTQ4Njg=
title: It would be more intuitive if "allow": none meant "no-one can do this"
state: closed
created_at: 2020-06-10T23:43:56Z
updated_at: 2020-06-10T23:57:25Z
closed_at: 2020-06-10T23:50:55Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Now that I'm starting to write alternative plugins to control permissions - see #818 - I think I need an easy way to tell Datasette "no-one has permission to do X unless a plugin says otherwise".

  One relatively intuitive way to do that could be like this:

  ```json
  {
    "databases": {
      "fixtures": {
        "allow": null
      }
    }
  }
  ```

  Right now I think that opens up permissions to everyone, which isn't as obvious.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/831/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzY3MjI1MDE=
title: Having view-table permission but NOT view-database should still grant access to /db/table
state: closed
created_at: 2020-06-11T05:12:59Z
updated_at: 2020-06-30T23:42:11Z
closed_at: 2020-06-30T23:42:11Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Stumbled into this while working on `datasette-permissions-sql`. I had granted table permissions, but the permission check wasn't even executed because the user failed the previous `view-database` check.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/832/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzcyNTM3ODk=
title: /-/metadata and so on should respect view-instance permission
state: closed
created_at: 2020-06-11T19:07:21Z
updated_at: 2020-06-11T22:15:32Z
closed_at: 2020-06-11T22:14:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The only URLs that should be available without authentication at all times are the `/-/static/` prefix, to allow for HTTP caching.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/833/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzczNDI1NTE=
title: startup() plugin hook
state: closed
created_at: 2020-06-11T21:48:14Z
updated_at: 2020-06-28T19:38:50Z
closed_at: 2020-06-13T17:56:12Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It might be useful to have an `startup` hook which gets passed the `datasette` object as soon as Datasette has finished initializing.

  My initial use-case for this is configuration verification - checking that the `"plugins"` configuration block for this plugin contains valid details.

  I imagine there are plenty of other potential uses for this as well.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/834/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzczNjM2ODY=
title: Mechanism for skipping CSRF checks on API posts
state: closed
created_at: 2020-06-11T22:41:10Z
updated_at: 2020-07-01T03:08:07Z
closed_at: 2020-07-01T03:08:07Z
author_association: OWNER
type: issue
state_reason: completed
body:
  While experimenting with https://github.com/simonw/datasette-auth-tokens I realized it's not currently possible to build API client programs that POST to Datasette because there's no mechanism for them to skip the CSRF checks added in #798.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/835/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzczNjU4MDE=
title: actor_matches_allow fails to consider all keys
state: closed
created_at: 2020-06-11T22:46:34Z
updated_at: 2020-06-11T22:47:25Z
closed_at: 2020-06-11T22:47:25Z
author_association: OWNER
type: issue
state_reason: completed
body:
  actor: `{"id": "root"}`

  allow block: `{"bot_id": "my-bot", "id": ["root"]}`

  This should pass, because the `id` matches - but it fails.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/836/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzczNzA2NTI=
title: Plugin $env secrets mechanism doesn't work inside lists
state: closed
created_at: 2020-06-11T22:59:54Z
updated_at: 2020-06-12T00:25:20Z
closed_at: 2020-06-12T00:25:19Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This didn't work:
  ```json
  {
      "plugins": {
          "datasette-auth-tokens": [
              {
                  "token": {
                      "$env": "BOT_TOKEN"
                  },
                  "actor": {
                      "bot_id": "my-bot"
                  }
              }
          ]
      }
  }
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/837/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzczOTUwOTc=
title: Incorrect URLs when served behind a proxy with base_url set
state: closed
created_at: 2020-06-11T23:58:55Z
updated_at: 2021-11-20T19:35:48Z
closed_at: 2021-11-20T19:35:48Z
author_association: NONE
type: issue
state_reason: completed
body:
  I'm running `datasette serve --config base_url:/foo/ …`, proxying to it with this Apache config:

          ProxyPass /foo/ http://localhost:8001/ 
          ProxyPassReverse /foo/ http://localhost:8001/ 

  and then accessing it via `https://example.com/foo/`.

  Although many of the URLs in the pages are correct (presumably because they either use absolute paths which include `base_url` or relative paths), the faceting and pagination links still use fully-qualified URLs pointing at `http://localhost:8001`.

  I looked into this a little in the source code, and it seems to be an issue anywhere `request.url` or `request.path` is used, as these contain the values for the request between the frontend (Apache) and backend (Datasette) server. Those properties are primarily used via the `path_with_…` family of utility functions and the `Datasette.absolute_url` method.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/838/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mzc0MDkxNDQ=
title: {"$file": ...} mechanism is broken
state: closed
created_at: 2020-06-12T00:46:24Z
updated_at: 2020-06-12T00:48:26Z
closed_at: 2020-06-12T00:48:26Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://travis-ci.org/github/simonw/datasette/jobs/697445318
  ```
      def test_plugin_config_file(app_client):
          open(TEMP_PLUGIN_SECRET_FILE, "w").write("FROM_FILE")
  >       assert {"foo": "FROM_FILE"} == app_client.ds.plugin_config("file-plugin")
  E       AssertionError: assert {'foo': 'FROM_FILE'} == {'foo': {'$fi...ugin-secret'}}
  E         Differing items:
  E         {'foo': 'FROM_FILE'} != {'foo': {'$file': '/tmp/plugin-secret'}}
  E         Use -v to get the full diff
  ```
  Broken in https://github.com/simonw/datasette/commit/fba8ff6e76253af2b03749ed8dd6e28985a7fb8f as part of #837

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/839/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mzc4ODk5NjQ=
title: Ability to execute insert/update statements with the CLI
state: closed
created_at: 2020-06-12T17:01:17Z
updated_at: 2020-06-12T17:51:11Z
closed_at: 2020-06-12T17:41:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  $ sqlite-utils github.db "update stars set starred_at = ''"
  Traceback (most recent call last):
    File "/Users/simon/.local/bin/sqlite-utils", line 8, in <module>
      sys.exit(cli())
    File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/cli.py", line 673, in query
      headers = [c[0] for c in cursor.description]
  TypeError: 'NoneType' object is not iterable
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/115/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mzc4OTk1Mzk=
title: Demo deploy is broken
state: closed
created_at: 2020-06-12T17:20:17Z
updated_at: 2020-06-12T18:06:48Z
closed_at: 2020-06-12T18:06:48Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://github.com/dogsheep/github-to-sqlite/runs/766180404?check_suite_focus=true

  ```
  The following NEW packages will be installed:
    sqlite3
  0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.
  Need to get 752 kB of archives.
  After this operation, 2482 kB of additional disk space will be used.
  Ign:1 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3
  Err:1 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3
    404  Not Found [IP: 52.177.174.250 80]
  E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/s/sqlite3/sqlite3_3.22.0-1ubuntu0.3_amd64.deb  404  Not Found [IP: 52.177.174.250 80]
  E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
  ##[error]Process completed with exit code 100.
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mzc5NjY4MzM=
title: Log out mechanism for clearing ds_actor cookie
state: closed
created_at: 2020-06-12T19:41:51Z
updated_at: 2020-06-29T04:31:43Z
closed_at: 2020-06-29T04:31:43Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Need a cookie clearing mechanism and a way to show that you are logged in.

  `datasette-auth-github` had a solution for this that can be pulled into core.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/840/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzgxMDQ1MjA=
title: Research feasibility of 100% test coverage
state: closed
created_at: 2020-06-13T06:07:01Z
updated_at: 2020-06-13T21:38:46Z
closed_at: 2020-06-13T21:38:46Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Inspired by https://twitter.com/mikeal/status/1271473021593636866
  > Almost every library I’ve written in the last 2 years has had 100% coverage and that’s probably not going to change in the future. It’s not that hard to start at 100% and hold onto it and the workflow it enables is so much nicer.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/841/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzgyMTIwODU=
title: Magic parameters for canned queries
state: closed
created_at: 2020-06-13T18:50:08Z
updated_at: 2020-06-28T03:30:31Z
closed_at: 2020-06-28T02:58:18Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Now that writable canned queries (#698) have landed, it would be neat if they supported "magic" parameters - parameters that are automatically populated with:

  - the current actor ID / other actor properties
  - the current date and time
  - the user's IP or user-agent

  And maybe other things potentially added by plugins.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/842/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzgyMjk0NDg=
title: Configure codecov.io
state: closed
created_at: 2020-06-13T20:45:00Z
updated_at: 2020-06-13T21:36:52Z
closed_at: 2020-06-13T21:36:52Z
author_association: OWNER
type: issue
state_reason: completed
body:
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/841#issuecomment-643660757_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/843/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDM0MDU1NzUy
title: Action to run tests and upload coverage report
state: closed
created_at: 2020-06-13T20:52:47Z
updated_at: 2020-06-13T21:36:52Z
closed_at: 2020-06-13T21:36:50Z
author_association: OWNER
pull_request: simonw/datasette/pulls/844
body: Refs #843
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/844/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzgyMzg1NDg=
title: Code coverage should ignore files in .coveragerc
state: open
created_at: 2020-06-13T21:45:42Z
updated_at: 2020-06-13T21:46:03Z
author_association: OWNER
type: issue
body:
  I'm not sure why this is, but the code coverage I have running in a GitHub Action doesn't take my `.coveragerc` file into account. It should:

  https://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/.github/workflows/test-coverage.yml#L31-L35

  Here's the bit that's ignored:

  https://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/.coveragerc#L1-L2

  As a result my coverage score is 84%, when it should be 92%:
  ```
  2020-06-13T21:41:18.4404252Z ----------- coverage: platform linux, python 3.8.3-final-0 -----------
  2020-06-13T21:41:18.4404570Z Name                                 Stmts   Miss  Cover
  2020-06-13T21:41:18.4404971Z --------------------------------------------------------
  2020-06-13T21:41:18.4405227Z datasette/__init__.py                    3      0   100%
  2020-06-13T21:41:18.4405441Z datasette/__main__.py                    3      3     0%
  2020-06-13T21:41:18.4405668Z datasette/_version.py                  279    279     0%
  2020-06-13T21:41:18.4405921Z datasette/actor_auth_cookie.py          20      0   100%
  2020-06-13T21:41:18.4406135Z datasette/app.py                       499     27    95%
  2020-06-13T21:41:18.4406343Z datasette/cli.py                       162     45    72%
  2020-06-13T21:41:18.4406553Z datasette/database.py                  236     17    93%
  2020-06-13T21:41:18.4406761Z datasette/default_permissions.py        40      0   100%
  2020-06-13T21:41:18.4406975Z datasette/facets.py                    210     24    89%
  2020-06-13T21:41:18.4407186Z datasette/filters.py                   122      7    94%
  2020-06-13T21:41:18.4407394Z datasette/hookspecs.py                  34      0   100%
  2020-06-13T21:41:18.4407600Z datasette/inspect.py                    36     23    36%
  2020-06-13T21:41:18.4407807Z datasette/plugins.py                    34      6    82%
  2020-06-13T21:41:18.4408014Z datasette/publish/__init__.py            0      0   100%
  2020-06-13T21:41:18.4408240Z datasette/publish/cloudrun.py           57      2    96%
  2020-06-13T21:41:18.4408786Z datasette/publish/common.py             19      1    95%
  2020-06-13T21:41:18.4409029Z datasette/publish/heroku.py             97     13    87%
  2020-06-13T21:41:18.4409243Z datasette/renderer.py                   63      4    94%
  2020-06-13T21:41:18.4409450Z datasette/sql_functions.py               5      0   100%
  2020-06-13T21:41:18.4410480Z datasette/tracer.py                     87     16    82%
  2020-06-13T21:41:18.4410972Z datasette/utils/__init__.py            504     31    94%
  2020-06-13T21:41:18.4411755Z datasette/utils/asgi.py                264     24    91%
  2020-06-13T21:41:18.4412173Z datasette/utils/shutil_backport.py      44     44     0%
  2020-06-13T21:41:18.4412822Z datasette/version.py                     4      0   100%
  2020-06-13T21:41:18.4413562Z datasette/views/__init__.py              0      0   100%
  2020-06-13T21:41:18.4414276Z datasette/views/base.py                288     19    93%
  2020-06-13T21:41:18.4414579Z datasette/views/database.py            120      2    98%
  2020-06-13T21:41:18.4414860Z datasette/views/index.py                57      2    96%
  2020-06-13T21:41:18.4415379Z datasette/views/special.py              72     16    78%
  2020-06-13T21:41:18.4418994Z datasette/views/table.py               418     18    96%
  2020-06-13T21:41:18.4428811Z --------------------------------------------------------
  2020-06-13T21:41:18.4430394Z TOTAL                                 3777    623    84%
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/845/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzgyNDE3Nzk=
title: "Too many open files" error running tests
state: closed
created_at: 2020-06-13T22:11:40Z
updated_at: 2020-06-14T00:26:31Z
closed_at: 2020-06-14T00:26:31Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I got this on my laptop:
  ```pytest
  ...
  /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.7/site-packages/jinja2/loaders.py:171: in get_source
      f = open_if_exists(filename)
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

  filename = '/Users/simon/Dropbox/Development/datasette/datasette/templates/400.html', mode = 'rb'

      def open_if_exists(filename, mode='rb'):
          """Returns a file descriptor for the filename if that file exists,
          otherwise `None`.
          """
          try:
  >           return open(filename, mode)
  E           OSError: [Errno 24] Too many open files: '/Users/simon/Dropbox/Development/datasette/datasette/templates/400.html'

  /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.7/site-packages/jinja2/utils.py:154: OSError
  ```
  Based on the conversation in https://github.com/pytest-dev/pytest/issues/2970 I'm worried that my tests are opening too many files without closing them.

  In particular... I call `sqlite3.connect(filepath)` a LOT - and I don't ever call `conn.close()` on those opened connections:

  https://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/datasette/database.py#L58-L60

  Could this be resulting in my tests eventually opening too many unclosed file handles? How could I confirm this?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/846/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzgyNTk2NDM=
title: Take advantage of .coverage being a SQLite database
state: closed
created_at: 2020-06-14T00:41:25Z
updated_at: 2020-06-28T20:50:21Z
closed_at: 2020-06-28T20:50:21Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The `.coverage` file generated by running `pytest-cov` is now a SQLite database!

  I could do something interesting with this. Maybe after each test run for a new commit I could store that database file somewhere?

  Lots of interesting challenges here.

  I got a change into `coveragepy` last year which helps make the custom SQL functions available for doing fun things in Datasette: https://github.com/nedbat/coveragepy/issues/868

  Bigger challenge: if I have a DB file for every commit, that's hundreds (potentially thousands) of DB files. Datasette isn't designed to handle thousands of files like that.

  So, do I figure out how to have Datasette open a file on-command for just a single request? Or, an easier option, do I copy data from those files into a single database with a modified schema to include the commit hash in each table row?

  (Following on from #841 and #844)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/847/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDM0MDg1MjM1
title: Reload support for config_dir mode.
state: closed
created_at: 2020-06-14T02:34:46Z
updated_at: 2020-07-03T02:44:54Z
closed_at: 2020-07-03T02:44:53Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/848
type: pull
body:
  A reference implementation for adding support to reload when datasette is in the config_dir mode. 

  This implementation is flawed since it is watching the entire directory and any changes to the database will reload the server and adding unrelated files to the directory will also reload the server. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/848/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDM0MTYyMzE2
title: Fixed bug in SQL query for photo scores
state: closed
created_at: 2020-06-14T15:39:22Z
updated_at: 2020-12-04T22:32:36Z
closed_at: 2020-12-04T22:32:27Z
author_association: CONTRIBUTOR
pull_request: dogsheep/dogsheep-photos/pulls/29
type: pull
body:
  The join on ZCOMPUTEDASSETATTRIBUTES used the wrong columns.  In most of the Photos database tables, table.ZASSET joins with ZGENERICASSET.Z_PK

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/29/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2MzkwNzI4MTE=
title: Rename master branch to main
state: closed
created_at: 2020-06-15T19:05:54Z
updated_at: 2022-10-27T13:57:08Z
closed_at: 2020-09-15T20:37:14Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I was waiting for consensus to form around this (and kind-of hoping for `trunk` since I like the tree metaphor) and it looks like `main` is it.

  I've seen convincing arguments against `trunk` too - it indicates that the branch has some special significance like in Subversion (where all branches come from trunk) when it doesn't. So `main` is better anyway.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/849/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mzk1NDI5NzQ=
title: Fall back to FTS4 if FTS5 is not available
state: open
created_at: 2020-06-16T10:11:23Z
updated_at: 2020-06-17T20:13:48Z
author_association: NONE
type: issue
body:
  got this with version 0.21.1 from pypi.  twitter-to-sqlite auth worked but then "twitter-to-sqlite user-timeline USER.db" produced a tracekback ending in "no such module: FTS5". 

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Mzk5OTM0Njc=
title: Proof of concept for Datasette on AWS Lambda with EFS
state: open
created_at: 2020-06-16T21:48:31Z
updated_at: 2020-06-16T23:52:16Z
author_association: OWNER
type: issue
body:
  https://aws.amazon.com/about-aws/whats-new/2020/06/aws-lambda-support-for-amazon-elastic-file-system-now-generally-/

  If Datasette can run on Lambda with access to EFS it could both read AND write large databases there.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/850/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDAzMzAyNzg=
title: Having trouble getting writable canned queries to work
state: closed
created_at: 2020-06-17T10:30:28Z
updated_at: 2020-06-17T10:33:25Z
closed_at: 2020-06-17T10:32:33Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Hey,

  I'm trying to get canned inserts to work. I have an DB with following metadata:

  ```text
  sqlite> .mode line

  sqlite> select name, sql from sqlite_master where name like '%search%';
   name = search
    sql = CREATE TABLE "search" ("id" INTEGER NOT NULL PRIMARY KEY, "name" VARCHAR(255) NOT NULL, "url" VARCHAR(255) NOT NULL)
  ```

  ```yaml
  # ...
  queries:
      add_search:
          sql: insert into search(name, url) VALUES (:name, :url),
          write: true
  ```
  which renders a form as expected, but when I submit the form I get `incomplete input` error.

  ![image](https://user-images.githubusercontent.com/3243482/84885285-7f46fe80-b09b-11ea-8a05-92c8986bbf7a.png)

  but when submit post the form

  I've attached a debugger to see where the error comes from, because `incomplete input` string doesn't appear in datasette codebase.

  Inside `datasette.database.Database.execute_write_fn` 

  https://github.com/simonw/datasette/blob/4fa7cf68536628344356d3ef8c92c25c249067a0/datasette/database.py#L69

  ```py
  result = await reply_queue.async_q.get()
  ```

  this line raises an exception. 

  That led me to believe I had something wrong with my SQL. But running the command in `sqlite3` inserts the record just fine.

  ```text
  sqlite> insert into search (name, url) values ('my name', 'my url');
  sqlite> SELECT last_insert_rowid();
  last_insert_rowid() = 3
  ```

  So I'm a bit lost here.

  ---
  - datasette, version 0.44
  - Python 3.8.1

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/851/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDA5MTczMjY=
title: canned_queries() plugin hook
state: closed
created_at: 2020-06-18T05:24:35Z
updated_at: 2020-06-20T03:08:40Z
closed_at: 2020-06-20T03:08:40Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Canned queries are currently baked into `metadata.json` which is read once on startup.

  Allowing users to interactively create new canned queries - even if just through a plugin - would make a lot of sense.

  Is this a new plugin hook or some other mechanism? Lots to think about here.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/852/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDA5NDM0NDE=
title: Ensure register_routes() works for POST
state: closed
created_at: 2020-06-18T06:24:55Z
updated_at: 2020-06-24T04:30:30Z
closed_at: 2020-06-18T16:22:02Z
author_association: OWNER
body: https://twitter.com/amjithr/status/1273496759684050944
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/853/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDE0NjAxNzk=
title: Respect default scope["actor"] if one exists
state: closed
created_at: 2020-06-18T18:25:08Z
updated_at: 2020-06-18T18:39:22Z
closed_at: 2020-06-18T18:39:22Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ASGI wrapper plugins that themselves set the `actor` scope variable should be respected (though `actor_from_request` plugins should still execute and get the chance to replace that initial `actor` value).

  Relevant code: https://github.com/simonw/datasette/blob/09a3479a5402df96489ed6cab6cc9fd674bf3433/datasette/app.py#L910-L921

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/854/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDIxMjczMDc=
title: Add instructions for using cookiecutter plugin template to plugin docs
state: closed
created_at: 2020-06-19T17:33:25Z
updated_at: 2020-06-22T02:51:38Z
closed_at: 2020-06-22T02:51:38Z
author_association: OWNER
body: Once I ship the `datasette-plugin` template: https://github.com/simonw/datasette-plugin/issues/1
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/855/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDIyOTY5ODk=
title: Consider pagination of canned queries
state: open
created_at: 2020-06-20T03:15:59Z
updated_at: 2021-05-21T14:22:41Z
author_association: OWNER
type: issue
body:
  The new `canned_queries()` plugin hook from #852 combined with plugins like https://github.com/simonw/datasette-saved-queries could mean that some installations end up with hundreds or even thousands of canned queries. I should consider pagination or some other way of ensuring that this doesn't cause performance problems for Datasette.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/856/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDIyOTc1MDU=
title: Comprehensive documentation for variables made available to templates
state: closed
created_at: 2020-06-20T03:19:43Z
updated_at: 2022-10-26T02:58:17Z
closed_at: 2022-10-26T02:58:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Needed for the Datasette 1.0 release, so template authors can trust that Datasette is unlikely to break their templates.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/857/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDIzODg1NjQ=
title: publish heroku does not work on Windows 10
state: open
created_at: 2020-06-20T14:40:28Z
updated_at: 2021-06-10T17:44:09Z
author_association: NONE
type: issue
body:
  When executing "datasette publish heroku schools.db" on Windows 10, I get the following error

  ```shell
    File "c:\users\dell\.virtualenvs\sec-schools-jn-cwk8z\lib\site-packages\datasette\publish\heroku.py", line 54, in heroku
      line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines()
    File "c:\python38\lib\subprocess.py", line 411, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "c:\python38\lib\subprocess.py", line 489, in run
      with Popen(*popenargs, **kwargs) as process:
    File "c:\python38\lib\subprocess.py", line 854, in __init__
      self._execute_child(args, executable, preexec_fn, close_fds,
    File "c:\python38\lib\subprocess.py", line 1307, in _execute_child
      hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
  FileNotFoundError: [WinError 2] The system cannot find the file specified
  ```

  Changing https://github.com/simonw/datasette/blob/55a6ffb93c57680e71a070416baae1129a0243b8/datasette/publish/heroku.py#L54

  to 

  ```python
  line.split()[0] for line in check_output(["heroku", "plugins"], shell=True).splitlines()
  ```

  as well as the other `check_output()` and `call()` within the same file leads me to another recursive error about temp files

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/858/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDI1NzI4NDE=
title: Database page loads too slowly with many large tables (due to table counts)
state: open
created_at: 2020-06-21T14:23:17Z
updated_at: 2021-08-25T21:59:55Z
author_association: CONTRIBUTOR
type: issue
body:
  Hey,
  I have a database that I save in HTML from couple of web scrapers. There are around 200k+, 50+ rows in a couple of tables, with sqlite file weighing around 600MB.

  The app runs on a VPS with 2 core CPU, 4GB RAM and refreshing database page regularly takes more than 10 seconds. I was suspecting that counting tables was the culprit, but manually running `select count(*) from table_name` for the largest table finishes under a second.

  I've looked at the source code. There's a check for index page for mutable databases larger than 100MB
  https://github.com/simonw/datasette/blob/799c5d53570d773203527f19530cf772dc2eeb24/datasette/views/index.py#L15

  but this check is not performed for database page. 
  I've manually crippled `Database::table_counts` method
  ```py
  async def table_counts(self, limit=10):
      if not self.is_mutable and self.cached_table_counts is not None:
          return self.cached_table_counts
      # Try to get counts for each table, $limit timeout for each count
      counts = {}
      for table in await self.table_names():
          try:
              # table_count = (
              #     await self.execute(
              #         "select count(*) from [{}]".format(table),
              #         custom_time_limit=limit,
              #     )
              # ).rows[0][0]
              counts[table] = 10 # table_count
          # In some cases I saw "SQL Logic Error" here in addition to
          # QueryInterrupted - so we catch that too:
          except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError):
              counts[table] = None
      if not self.is_mutable:
          self.cached_table_counts = counts
      return counts
  ```

  now the page loads in <100ms.

  Is it possible to apply size check on database page too?

  <details>
  <summary>
  /-/versions output
  </summary>
  <pre>
  {
      "python": {
          "version": "3.8.0",
          "full": "3.8.0 (default, Oct 28 2019, 16:14:01) \n[GCC 8.3.0]"
      },
      "datasette": {
          "version": "0.44"
      },
      "asgi": "3.0",
      "uvicorn": "0.11.5",
      "sqlite": {
          "version": "3.22.0",
          "fts_versions": [
              "FTS5",
              "FTS4",
              "FTS3"
          ],
          "extensions": {
              "json1": null
          },
          "compile_options": [
              "COMPILER=gcc-7.4.0",
              "ENABLE_COLUMN_METADATA",
              "ENABLE_DBSTAT_VTAB",
              "ENABLE_FTS3",
              "ENABLE_FTS3_PARENTHESIS",
              "ENABLE_FTS3_TOKENIZER",
              "ENABLE_FTS4",
              "ENABLE_FTS5",
              "ENABLE_JSON1",
              "ENABLE_LOAD_EXTENSION",
              "ENABLE_PREUPDATE_HOOK",
              "ENABLE_RTREE",
              "ENABLE_SESSION",
              "ENABLE_STMTVTAB",
              "ENABLE_UNLOCK_NOTIFY",
              "ENABLE_UPDATE_DELETE_LIMIT",
              "HAVE_ISNAN",
              "LIKE_DOESNT_MATCH_BLOBS",
              "MAX_SCHEMA_RETRY=25",
              "MAX_VARIABLE_NUMBER=250000",
              "OMIT_LOOKASIDE",
              "SECURE_DELETE",
              "SOUNDEX",
              "TEMP_STORE=1",
              "THREADSAFE=1"
          ]
      }
  }
  </pre>
  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/859/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDI2NTE1NzI=
title: Plugin hook for instance/database/table metadata
state: closed
created_at: 2020-06-21T22:20:25Z
updated_at: 2022-01-13T22:21:42Z
closed_at: 2021-06-26T22:56:28Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'm not happy with how `metadata.(json|yaml)` keeps growing new features. Rather than having a single plugin hook for all of `metadata.json` I'm going to split out the feature that shows actual real metadata for tables and databases - `source`, `license` etc - into its own plugin-powered mechanism.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/357#issuecomment-647189045_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/860/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDI2NTI4MDg=
title: Script to generate larger SQLite test files
state: closed
created_at: 2020-06-21T22:30:58Z
updated_at: 2020-06-23T03:44:18Z
closed_at: 2020-06-23T03:44:18Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I'll write a little script which generates a 300MB SQLite file with a bunch of tables with lots of randomly generated rows in to help test this.
  > 
  > Having a tool like that which can generate larger databases with different gnarly performance characteristics will be useful for other performance work too.
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/859#issuecomment-647189948_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/861/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDM1MTA4MjE=
title: Set an upper limit on total facet suggestion time for a page
state: open
created_at: 2020-06-23T03:57:55Z
updated_at: 2020-06-23T03:58:48Z
author_association: OWNER
type: issue
body:
  If a table has 100 columns the facet suggestion code will currently run 100 times, taking a max of `facet_suggest_time_limit_ms` which defaults to 50ms per column:

  https://github.com/simonw/datasette/blob/000528192eaf891118932250141dabe7a1561ece/datasette/facets.py#L142-L162

  So for 100 columns, that's 100 * 50ms = 5s total time that might be spent attempting to calculate facets on a large table!

  I should implement a hard upper limit on the total amount of time taken suggesting facets - probably of around 500ms. If it takes longer than that the remaining columns will not be considered.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/862/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDQxMjI2NjE=
title: Documentation for table.pks introspection property
state: closed
created_at: 2020-06-23T20:27:24Z
updated_at: 2020-06-23T21:21:33Z
closed_at: 2020-06-23T21:03:14Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L535-L540

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/116/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDQxNjEyMjE=
title: Support for compound (composite) foreign keys
state: open
created_at: 2020-06-23T21:33:42Z
updated_at: 2020-06-23T21:40:31Z
author_association: OWNER
type: issue
body:
  It turns out SQLite supports composite foreign keys: https://www.sqlite.org/foreignkeys.html#fk_composite

  Their example looks like this:
  ```sql
  CREATE TABLE album(
    albumartist TEXT,
    albumname TEXT,
    albumcover BINARY,
    PRIMARY KEY(albumartist, albumname)
  );

  CREATE TABLE song(
    songid     INTEGER,
    songartist TEXT,
    songalbum TEXT,
    songname   TEXT,
    FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname)
  );
  ```

  Here's what that looks like in sqlite-utils:

  ```
  In [1]: import sqlite_utils                                                                                                                

  In [2]: import sqlite3                                                                                                                     

  In [3]: conn = sqlite3.connect(":memory:")                                                                                                 

  In [4]: conn                                                                                                                               
  Out[4]: <sqlite3.Connection at 0x1087186c0>

  In [5]: conn.executescript(""" 
     ...: CREATE TABLE album( 
     ...:   albumartist TEXT, 
     ...:   albumname TEXT, 
     ...:   albumcover BINARY, 
     ...:   PRIMARY KEY(albumartist, albumname) 
     ...: ); 
     ...:  
     ...: CREATE TABLE song( 
     ...:   songid     INTEGER, 
     ...:   songartist TEXT, 
     ...:   songalbum TEXT, 
     ...:   songname   TEXT, 
     ...:   FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) 
     ...: ); 
     ...: """)                                                                                                                               
  Out[5]: <sqlite3.Cursor at 0x1088def10>

  In [6]: db = sqlite_utils.Database(conn)                                                                                                   

  In [7]: db.tables                                                                                                                          
  Out[7]: 
  [<Table album (albumartist, albumname, albumcover)>,
   <Table song (songid, songartist, songalbum, songname)>]

  In [8]: db.tables[0].foreign_keys                                                                                                          
  Out[8]: []

  In [9]: db.tables[1].foreign_keys                                                                                                          
  Out[9]: 
  [ForeignKey(table='song', column='songartist', other_table='album', other_column='albumartist'),
   ForeignKey(table='song', column='songalbum', other_table='album', other_column='albumname')]
  ```
  The table appears to have two separate foreign keys, when actually it has a single compound composite foreign key.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/117/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDQyODMyMTE=
title: {{ csrftoken() }} doesn't work with datasette.render_template()
state: closed
created_at: 2020-06-24T03:11:49Z
updated_at: 2020-06-24T04:30:30Z
closed_at: 2020-06-24T03:24:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The documentation here suggests that it will work:

  https://github.com/simonw/datasette/blob/eed116ac0599c7d21b7129af94d58ce03a923e4e/docs/internals.rst#L540-L546

  But right now the `csrftoken` variable is set in BaseView.render, which means it's not visible to plugins that try to render templates using `datasette.render_template`:

  https://github.com/simonw/datasette/blob/799c5d53570d773203527f19530cf772dc2eeb24/datasette/views/base.py#L99-L106

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/863/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDQzMDkwMTc=
title: datasette.add_message() doesn't work inside plugins
state: closed
created_at: 2020-06-24T04:30:06Z
updated_at: 2020-06-29T00:51:01Z
closed_at: 2020-06-29T00:51:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Similar problem to #863 - calling `datasette.add_message()` in a view registered using the `register_routes()` plugin hook doesn't work, because the code that writes accumulated messages to the `ds_messages` signed cookie lives in the `BaseView` class here:

  https://github.com/simonw/datasette/blob/28bb1c51897f3956861755e345e18b8e0b1423ac/datasette/views/base.py#L94-L97

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/864/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDQ1ODI5MjE=
title: base_url doesn't seem to work when adding criteria and clicking "apply"
state: closed
created_at: 2020-06-24T12:39:57Z
updated_at: 2020-11-12T23:49:24Z
closed_at: 2020-10-20T05:22:59Z
author_association: NONE
type: issue
state_reason: completed
body:
  Over on Apache Tika, we're using datasette to allow users to make sense of the metadata for our file regression testing corpus.

  This could be user error in how I've set up the reverse proxy!

  I started datasette like so:
  `docker run -d -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/corpora-metadata.db --config sql_time_limit_ms:60000 --config base_url:/datasette/`

  I then reverse proxied like so:

    ProxyPreserveHost On
    ProxyPass /datasette http://x.y.z.q:xxxx
    ProxyPassReverse /datasette http://x.y.z.q:xxx

  Regular sql works perfectly:
  https://corpora.tika.apache.org/datasette/corpora-metadata?sql=select+mime_string%2C+count%281%29+as+cnt%0D%0Afrom+profiles+p%0D%0Ajoin+mimes+m+on+p.mime_id%3Dm.mime_id%0D%0Agroup+by+mime_string%0D%0Aorder+by+cnt+desc


  However, adding criteria and clicking 'Apply' 
  https://corpora.tika.apache.org/datasette/corpora-metadata/tika_1_24_1_mimes?_sort=file&mime__exact=text%2Fplain

  bounces back to:
  https://corpora.tika.apache.org/corpora-metadata/tika_1_24_1_mimes?_sort=file&file__contains=bug&mime__exact=text%2Fplain

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/865/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDM5MjAzODA4
title: Update pytest-asyncio requirement from <0.13,>=0.10 to >=0.10,<0.15
state: closed
created_at: 2020-06-24T13:21:47Z
updated_at: 2020-06-24T18:50:57Z
closed_at: 2020-06-24T18:50:56Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/866
type: pull
body:
  Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version.
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/53f3da7aefc719e62cbaa89e57ab68a7e12cc3c3"><code>53f3da7</code></a> Prepare for release</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/e99569de645c37fc048964ab4c5073529080fd86"><code>e99569d</code></a> A line is added to the changelog.</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/4099b6351793611acabd8b26d93aabd44ce200c5"><code>4099b63</code></a> One import is not needed</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/68513b33616ff2ef93972acf1b992e7b1a18c4d1"><code>68513b3</code></a> Clarify names and comments, according to yanlend comments 26 May</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/907e8f24b9444f50cb30f17265665706882bc01f"><code>907e8f2</code></a> FIX new test_cases on python 3.5 &amp; 3.6</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/51d986cec83fdbc14fa08015424c79397afc7ad9"><code>51d986c</code></a> To solve test cases that fail:</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/f97e900f1fcb51a572a1b861c95ac49e69bbfdf9"><code>f97e900</code></a> 1) Test case (test_async_fixtures_with_finalizer) refactoring to pass on pyth...</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/c1131f8b5313189508dc81d7ef1937ccb136658b"><code>c1131f8</code></a> 1) A new test case that fails with 0.12.0, and pass with this commit.</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/7a255bc4cf82aba7aa4b213c1a97c81d532c1e85"><code>7a255bc</code></a> 0.13.0 open for business</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/b8e2a45e152a196cef7fdb6ddcf3e2d67a0f01ca"><code>b8e2a45</code></a> 0.12.0</li>
  <li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest-asyncio/compare/v0.10.0...v0.14.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/866/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDU5NzU2NDk=
title: register_routes() should support non-async view functions too
state: closed
created_at: 2020-06-26T03:11:25Z
updated_at: 2020-06-27T18:30:41Z
closed_at: 2020-06-27T18:30:40Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I was looking at this: https://github.com/simonw/datasette-block-robots/blob/main/datasette_block_robots/__init__.py
  ```python
  from datasette import hookimpl
  from datasette.utils.asgi import Response


  async def robots_txt():
      return Response.text("User-agent: *\nDisallow: /")


  @hookimpl
  def register_routes():
      return [
          (r"^/robots\.txt$", robots_txt),
      ]
  ```
  And I realized that if `register_routes()` could support non-async view functions it could be reduced to this:
  ```python
  @hookimpl
  def register_routes():
      return [
          (r"^/robots\.txt$", lambda: Response.text("User-agent: *\nDisallow: /")),
      ]
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/867/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDQwNzM1ODE0
title: initial windows ci setup
state: open
created_at: 2020-06-26T18:49:13Z
updated_at: 2021-07-10T23:41:43Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: simonw/datasette/pulls/868
body: Picking up the work done on #557 with a new PR. Seeing if I can get this working.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/868/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDQwOTQ2ODE3
title: Magic parameters for canned queries
state: closed
created_at: 2020-06-27T18:37:21Z
updated_at: 2020-06-28T02:58:18Z
closed_at: 2020-06-28T02:58:17Z
author_association: OWNER
pull_request: simonw/datasette/pulls/869
type: pull
body:
  Implementation for #842

  TODO:

  - [x] Add tests for built-in magic parameters
  - [x] Magic parameters should not show up as blank form fields on the query page
  - [x] Update documentation for new `_request_X` (now called `_header_X`) implementation where X is a key from the ASGI scope
  - [x] Make sure these only work for canned queries, not for arbitrary SQL queries (security issue)
  - [x] Add test for the `register_magic_parameters` plugin hook
  - [x] Add documentation for the `register_magic_parameters` plugin hook

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/869/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDY3Mzc1NTg=
title: Refactor default views to use register_routes
state: open
created_at: 2020-06-27T18:53:12Z
updated_at: 2022-03-15T20:07:18Z
author_association: OWNER
type: issue
body:
  It would be much cleaner if Datasette's default views were all registered using the new `register_routes()` plugin hook. Could dramatically reduce the code in `datasette/app.py`.

  > The ideal fix here would be to rework my `BaseView` subclass mechanism to work with `register_routes()` so that those views don't have any special privileges above plugin-provided views.
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/864#issuecomment-648580556_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/870/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDY4NDAyNzM=
title: Rename the _timestamp magic parameters to _now
state: closed
created_at: 2020-06-28T04:49:08Z
updated_at: 2020-06-28T19:49:49Z
closed_at: 2020-06-28T19:49:49Z
author_association: OWNER
body: I like the shorter name better. Follows on from #842.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/871/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDY5OTIwOTY=
title: Release non-alpha plugins when 0.45 is out
state: closed
created_at: 2020-06-28T19:42:01Z
updated_at: 2020-07-01T23:48:51Z
closed_at: 2020-07-01T23:48:51Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I have several plugins currently marked as alphas because they depend on `0.45a3`. When 0.45 is released I can ship new versions of these plugins that are full releases, not alphas - and switch them to depending on 0.45 (as opposed to the alpha):

  - [x] https://github.com/simonw/datasette-init
  - [x] https://github.com/simonw/datasette-glitch
  - [x] https://github.com/simonw/datasette-saved-queries
  - [x] https://github.com/simonw/datasette-write

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/872/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDcwOTU0ODc=
title: "datasette -p 0 --root" gives the wrong URL
state: open
created_at: 2020-06-29T04:03:06Z
updated_at: 2020-08-18T17:26:10Z
author_association: OWNER
type: issue
body:
  ```
  $ datasette -p 0 --root
  http://127.0.0.1:0/-/auth-token?token=2d498c...
  ```
  The port is incorrect.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/873/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDcwOTU4MDg=
title: /favicon.ico 500 error
state: closed
created_at: 2020-06-29T04:04:22Z
updated_at: 2020-06-29T04:27:18Z
closed_at: 2020-06-29T04:27:18Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  Traceback (most recent call last):
    File "...datasette/datasette/app.py", line 969, in route_path
      response = await view(request, send)
  TypeError: favicon() missing 1 required positional argument: 'send'
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/874/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDcxMDM3MzU=
title: "Logged in as: XXX - logout" navigation item
state: closed
created_at: 2020-06-29T04:31:14Z
updated_at: 2020-07-02T00:13:24Z
closed_at: 2020-06-29T18:43:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/840#issuecomment-650895874_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/875/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDc4Nzk3ODM=
title: Add log out link to the pattern portfolio
state: closed
created_at: 2020-06-30T05:42:15Z
updated_at: 2020-06-30T23:50:04Z
closed_at: 2020-06-30T23:47:31Z
author_association: OWNER
body: Follows #875
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/876/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDgyNDUwNzE=
title: Error thrown: table photos has no column named hasSticker
state: closed
created_at: 2020-06-30T14:54:37Z
updated_at: 2020-10-12T20:35:06Z
closed_at: 2020-10-12T20:25:24Z
author_association: NONE
type: issue
state_reason: completed
body:
  While running `swarm-to-sqlite` it throws an error:

      harper@:~/dogsheep/swarm$ swarm-to-sqlite checkins.db --save=checkins.json
      Please provide your Foursquare OAuth token:
      Importing 8127 checkins  [#################-------------------]   49%  00:01:52
      Traceback (most recent call last):
      File "/home/harper/.local/bin/swarm-to-sqlite", line 11, in <module>
          sys.exit(cli())
      File "/home/harper/.local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
          return self.main(*args, **kwargs)
      File "/home/harper/.local/lib/python3.6/site-packages/click/core.py", line 782, in main
          rv = self.invoke(ctx)
      File "/home/harper/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
          return ctx.invoke(self.callback, **ctx.params)
      File "/home/harper/.local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
          return callback(*args, **kwargs)
      File "/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/cli.py", line 73, in cli
          save_checkin(checkin, db)
      File "/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/utils.py", line 94, in save_checkin
          photos_table.insert(photo, replace=True)
      File "/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/db.py", line 963, in insert
          alter = self.value_or_default("alter", alter)
      File "/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/db.py", line 1142, in insert_all
          def upsert_all(
      sqlite3.OperationalError: table photos has no column named hasSticker

  Where should i dig in?

reactions:
  {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDg0MjExMDU=
title: Consider dropping explicit CSRF protection entirely?
state: closed
created_at: 2020-06-30T19:00:55Z
updated_at: 2020-09-15T20:42:05Z
closed_at: 2020-09-15T20:42:04Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://scotthelme.co.uk/csrf-is-dead/ from Feb 2017 has background here. The `SameSite=lax` cookie property effectively eliminates CSRF in modern browsers. https://caniuse.com/#search=SameSite shows 92.13% global support for it.

  Datasette already uses `SameSite=lax` when it sets cookies by default: https://github.com/simonw/datasette/blob/af350ba4571b8e3f9708c40f2ddb48fea7ac1084/datasette/utils/asgi.py#L327-L341

  A few options then. I could ditch CSRF protection entirely. I could make it optional - turn it off by default, but let users who care about that remaining 7.87% of global users opt back into it.

  One catch: login CSRF: I don't see how `SameSite=lax` protects against that attack.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/877/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDg0MzU4ODU=
title: New pattern for views that return either JSON or HTML, available for plugins
state: open
created_at: 2020-06-30T19:26:13Z
updated_at: 2022-03-19T16:19:30Z
author_association: OWNER
type: issue
body:
  Can be part of #870 - refactoring existing views to use `register_routes()`.

  > I'm going to put the new `check_permissions()` method on `BaseView` as well. If I want that method to be available to plugins I can do so by turning that `BaseView` class into a documented API that plugins are encouraged to use themselves.
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/832#issuecomment-651995453_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/878/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDg1NjkyMjc=
title: Database page documentation still talks about hashes in URLs
state: closed
created_at: 2020-06-30T23:43:17Z
updated_at: 2020-06-30T23:48:06Z
closed_at: 2020-06-30T23:45:42Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://datasette.readthedocs.io/en/0.44/pages.html

  > Note that these URLs end in a 7 character hash. This hash is derived from the contents of the database, and ensures that each URL is immutable: the data returned from a URL containing the hash will always be the same, since if the contents of the database file changes by even a single byte a new hash will be generated.

  This isn't accurate any more - that's not default behaviour, and it may be removed entirely in #647.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/879/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDg2Mzc2NjY=
title: POST to /db/canned-query that returns JSON should be supported (for API clients)
state: closed
created_at: 2020-07-01T03:14:43Z
updated_at: 2020-09-14T21:28:21Z
closed_at: 2020-09-14T21:25:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Now that CSRF is solved for API requests (#835) it would be good to support API requests to the `.json` extension.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/880/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDg2NTk1MzY=
title: Figure out why restore_working_directory is needed in some places
state: open
created_at: 2020-07-01T04:19:25Z
updated_at: 2020-07-01T04:19:25Z
author_association: OWNER
type: issue
body:
  This is a frustrating workaround. I have a `restore_working_directory` fixture that I wrote to solve errors that look like this:
  ```
  /Users/simon/Dropbox/Development/datasette/tests/test_publish_cloudrun.py:148: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:112: in __enter__
      return next(self.gen)
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  self = <click.testing.CliRunner object at 0x1135ad110>

      @contextlib.contextmanager
      def isolated_filesystem(self):
          """A context manager that creates a temporary folder and changes
          the current working directory to it for isolated filesystem tests.
          """
  >       cwd = os.getcwd()
  E       FileNotFoundError: [Errno 2] No such file or directory
  ```
  Here's an example of it in use: removing the `restore_working_directory` argument from this function causes the failure. https://github.com/simonw/datasette/blob/549b1c2063db48c4622ee5c7b478a1e3cbc1ac07/tests/test_plugins.py#L689-L690

  I'd like to not have to do this.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/881/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDg2NzM1NTY=
title: Release notes for 0.45
state: closed
created_at: 2020-07-01T05:00:17Z
updated_at: 2020-07-01T21:48:08Z
closed_at: 2020-07-01T21:48:08Z
author_association: OWNER
type: issue
state_reason: completed
body:
  These are mostly done thanks to the alphas, but I went to have more paragraphs of prose and less bullet points.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/882/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDQyNTA1MDg4
title: Skip counting hidden tables
state: open
created_at: 2020-07-01T07:38:08Z
updated_at: 2020-07-02T00:25:44Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/883
type: pull
body:
  Potential fix for https://github.com/simonw/datasette/issues/859.

  Disabling table counts for hidden tables speeds up database page quite a bit. In my setup it reduced load time by 2/3 (~300 -> ~90ms)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/883/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDkzMjkwMTM=
title: Only show "log out" button if user is authenticated using a ds_actor cookie
state: closed
created_at: 2020-07-01T21:21:28Z
updated_at: 2020-07-01T21:26:07Z
closed_at: 2020-07-01T21:26:06Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Right now the "Log out" button in the navigation will show up even if the user was authenticated by a plugin using a mechanism other than the `ds_actor` cookie. It should only show if the logged-in user has that cookie.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/884/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDkzNzM0NTE=
title: Blog entry about the release
state: closed
created_at: 2020-07-01T22:44:37Z
updated_at: 2020-07-01T22:44:48Z
closed_at: 2020-07-01T22:44:47Z
author_association: OWNER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/885/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDk0Mjk3NzI=
title: Reconsider how _actor_X magic parameter deals with missing values
state: open
created_at: 2020-07-02T00:00:38Z
updated_at: 2020-09-11T21:35:26Z
author_association: OWNER
type: issue
body:
  I had to build a custom `_actorornull` prefix for [datasette-saved-queries](https://github.com/simonw/datasette-saved-queries/blob/37c00e56ac398e1f9aa342d30357de013a9b37b4/datasette_saved_queries/__init__.py):
  ```python
  def actorornull(key, request):
      if request.actor is None:
          return None
      return request.actor.get(key)


  @hookimpl
  def register_magic_parameters():
      return [
          ("actorornull", actorornull),
      ]
  ```
  Maybe the `actor` magic in Datasette core should do that out of the box?

  https://github.com/simonw/datasette/blob/f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb/datasette/default_magic_parameters.py#L14-L17

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/886/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDk0Mzc1MzA=
title: Canned query page should show the name of the canned query
state: closed
created_at: 2020-07-02T00:10:39Z
updated_at: 2020-07-02T00:31:33Z
closed_at: 2020-07-02T00:23:45Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This page here - the URL is http://127.0.0.1:8001/data/all_tables but "all_tables" is not shown in the UI:

  <img width="784" alt="data__select_sqlite_master_name_as_table_name__table_info___from_sqlite_master_join_pragma_table_info_sqlite_master_name__as_table_info_order_by_sqlite_master_name__table_info_cid_and_data__insert_into_saved_queries__name__sql__author_id__v" src="https://user-images.githubusercontent.com/9599/86302528-d978b100-bbbd-11ea-9a07-7b5d7d3de321.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/887/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDk3MDI4MDE=
title: URLs in release notes point to 127.0.0.1
state: closed
created_at: 2020-07-02T07:28:04Z
updated_at: 2020-09-15T20:39:50Z
closed_at: 2020-09-15T20:39:49Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Just a quick heads up:

  Release notes for 0.45 include urls that point to localhost. 

  https://github.com/simonw/datasette/releases/tag/0.45

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/888/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NDk5MDc2NzY=
title: asgi_wrapper plugin hook is crashing at startup
state: closed
created_at: 2020-07-02T12:53:13Z
updated_at: 2020-09-15T20:40:52Z
closed_at: 2020-09-15T20:40:52Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Steps to reproduce: 

  1. Install datasette-media plugin 
  `pip install datasette-media`
  2. Launch datasette
  `datasette databasename.db`
  3. Error

  ```
  INFO:     Started server process [927704]
  INFO:     Waiting for application startup.
  ERROR:    Exception in 'lifespan' protocol
  Traceback (most recent call last):
    File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/uvicorn/lifespan/on.py", line 48, in main
      await app(scope, self.receive, self.send)
    File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
      return await self.app(scope, receive, send)
    File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/datasette_media/__init__.py", line 9, in wrapped_app
      path = scope["path"]
  KeyError: 'path'
  ERROR:    Application startup failed. Exiting.
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/889/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDQzODIzMDQw
title: Load only python files from plugins-dir.
state: closed
created_at: 2020-07-03T02:47:32Z
updated_at: 2020-07-03T03:08:33Z
closed_at: 2020-07-03T03:08:33Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/890
type: pull
body:
  The current behavior for `--plugins-dir` is to load every file in that folder as a python module. This can result in errors if there are non-python files in the plugins dir (such as .mypy_cache). 

  This PR restricts the module loading to only python files. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/890/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NTExNTk3Mjc=
title: Demo is failing to deploy
state: closed
created_at: 2020-07-05T22:40:33Z
updated_at: 2020-07-06T01:07:03Z
closed_at: 2020-07-06T01:07:02Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://github.com/dogsheep/github-to-sqlite/runs/837714622?check_suite_focus=true
  ```
  Creating Revision.........................................................................................................................................failed
  Deployment failed
  ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.
  Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette", line 8, in <module>
      sys.exit(cli())
    File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/cloudrun.py", line 138, in cloudrun
      check_call(
    File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/subprocess.py", line 364, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command 'gcloud run deploy --allow-unauthenticated --platform=managed --image gcr.io/datasette-222320/datasette github-to-sqlite' returned non-zero exit status 1.
  ##[error]Process completed with exit code 1.
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDQ1MDIzMzI2
title: Add insert --truncate option
state: closed
created_at: 2020-07-06T21:58:40Z
updated_at: 2020-07-08T17:26:21Z
closed_at: 2020-07-08T17:26:21Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/118
type: pull
body:
  Deletes all rows in the table (if it exists) before inserting new rows.
  SQLite doesn't implement a TRUNCATE TABLE statement but does optimize an
  unqualified DELETE FROM.

  This can be handy if you want to refresh the entire contents of a table
  but a) don't have a PK (so can't use --replace), b) don't want the table
  to disappear (even briefly) for other connections, and c) have to handle
  records that used to exist being deleted.

  Ideally the replacement of rows would appear instantaneous to other
  connections by putting the DELETE + INSERT in a transaction, but this is
  very difficult without breaking other code as the current transaction
  handling is inconsistent and non-systematic.  There exists the
  possibility for the DELETE to succeed but the INSERT to fail, leaving an
  empty table.  This is not much worse, however, than the current
  possibility of one chunked INSERT succeeding and being committed while
  the next chunked INSERT fails, leaving a partially complete operation.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NTI3MDA3NzA=
title: Ability to remove a foreign key
state: closed
created_at: 2020-07-07T22:31:37Z
updated_at: 2020-09-24T20:36:59Z
closed_at: 2020-09-24T20:36:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Useful if you add one but make a mistake and need to undo it without recreating the database from scratch.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/119/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDQ1ODMzOTA4
title: Fix query command's support for DML
state: closed
created_at: 2020-07-08T01:36:34Z
updated_at: 2020-07-08T05:14:04Z
closed_at: 2020-07-08T05:14:04Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/120
body: See commit messages for details. I ran into this while investigating another feature/issue.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/120/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NTI5NjE5MDc=
title: Improved (and better documented) support for transactions
state: open
created_at: 2020-07-08T04:56:51Z
updated_at: 2020-09-24T20:36:46Z
author_association: OWNER
type: issue
body:
  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393_

  We should put some thought into how this library supports and encourages smart use of transactions.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/121/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NTM1MjkwODg=
title: Consider using enable_callback_tracebacks(True)
state: closed
created_at: 2020-07-08T19:07:16Z
updated_at: 2020-10-08T23:54:23Z
closed_at: 2020-09-15T21:59:27Z
author_association: OWNER
type: issue
state_reason: completed
body:
  From https://docs.python.org/3/library/sqlite3.html#sqlite3.enable_callback_tracebacks

  > `sqlite3.``enable_callback_tracebacks`(*flag*)[¶](https://docs.python.org/3/library/sqlite3.html#sqlite3.enable_callback_tracebacks "Permalink to this definition")
  > 
  > By default you will not get any tracebacks in user-defined functions, aggregates, converters, authorizer callbacks etc. If you want to debug them, you can call this function with *flag* set to `True`. Afterwards, you will get tracebacks from callbacks on `sys.stderr`. Use [`False`](https://docs.python.org/3/library/constants.html#False "False") to disable the feature again.

  Maybe turn this on for all of Datasette? Are there any disadvantages to doing that?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/891/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NTQ0MDUzMDI=
title: Option for importing just specific repos
state: closed
created_at: 2020-07-09T23:20:15Z
updated_at: 2020-07-09T23:25:35Z
closed_at: 2020-07-09T23:25:35Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  For if you know which specific repos you care about, as opposed to loading everything owned by the authenticated user.

      github-to-sqlite repos specific.db -r simonw/datasette -r simonw/github-contents

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/42/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NTU0NjU4NjM=
title: "latest" in new documentation navbar is invisible
state: closed
created_at: 2020-07-12T19:57:21Z
updated_at: 2020-07-12T20:02:35Z
closed_at: 2020-07-12T20:02:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  On https://datasette.readthedocs.io/en/latest/

  <img width="572" alt="Datasette_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/87255376-1b67e980-c43f-11ea-8470-781ee8842336.png">

  Compare with https://datasette.readthedocs.io/en/0.45/

  <img width="584" alt="Datasette_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/87255387-2fabe680-c43f-11ea-875e-c474364691ca.png">

  Some custom CSS should fix it.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/892/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDQ4MzU1Njgw
state: open
created_at: 2020-07-13T16:15:26Z
updated_at: 2020-07-13T16:15:26Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: dogsheep/dogsheep-photos/pulls/30
type: pull
title:
  Handle empty bucket on first upload. Allow specifying the endpoint_url for services other than S3 (like b2 and digitalocean spaces)

body:
  Finally got around to trying dogsheep-photos but I want to use backblaze's b2 service instead of AWS S3.
  Had to add a way to optionally specify the endpoint_url to connect to. Then with the bucket being empty the initial key retrieval would fail. Probably a better way to see that the bucket is empty than doing a test inside the paginator loop.

  Also probably a better way to specify the endpoint_url as we get and test for it twice using the same code in two different places but did not want to spend too much time worrying about it.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/30/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NTY5NTk1ODQ=
title: pip3 install datasette not serving static on linuxbrew.
state: closed
created_at: 2020-07-14T23:33:38Z
updated_at: 2021-06-02T04:29:56Z
closed_at: 2021-06-02T04:29:56Z
author_association: NONE
type: issue
state_reason: completed
body:
  *This error wasn't thrown*

  ```
  Traceback (most recent call last):
    File "/home/linuxbrew/.linuxbrew/opt/python@3.8/lib/python3.8/site-packages/datasette/utils/asgi.py", line 289, in inner_static
      full_path.relative_to(root_path)
    File "/home/linuxbrew/.linuxbrew/opt/python@3.8/lib/python3.8/pathlib.py", line 904, in relative_to
      raise ValueError("{!r} does not start with {!r}"
  ValueError: '/home/linuxbrew/.linuxbrew/lib/python3.8/site-packages/datasette/static/app.css' does not start with '/home/linuxbrew/.linuxbrew/opt/python@3.8/lib/python3.8/site-packages/datasette/static'
  ```

  Linuxbrew install python@3.8 with symbolic links when  You call the full_path.relative_to(root_path) throw ValueError. This happened when you install from pip3

  when you install with python3 setup.py develop , works good.

  Well at the end the static wasn't serving.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/893/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NTc1NzI3NTM=
title: ?sort=colname~numeric to sort by by column cast to real
state: open
created_at: 2020-07-15T18:47:48Z
updated_at: 2021-08-20T02:07:53Z
author_association: OWNER
type: issue
body:
  If a text column actually contains numbers, being able to "sort by column, treated as numeric" would be really useful.

  Probably depends on column actions enabled by #690

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/894/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NTc3NDc5NTk=
title: SQL query output should show numeric values in a different colour
state: closed
created_at: 2020-07-16T00:28:03Z
updated_at: 2020-09-15T20:40:08Z
closed_at: 2020-09-15T20:40:08Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Compare https://latest.datasette.io/fixtures/sortable with https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+order+by+pk1%2C+pk2+limit+101

  <img width="1574" alt="fixtures__select_pk1__pk2__content__sortable__sortable_with_nulls__sortable_with_nulls_2__text_from_sortable_order_by_pk1__pk2_limit_101_and_fixtures__sortable__201_rows" src="https://user-images.githubusercontent.com/9599/87612845-82e09c00-c6c0-11ea-806e-93764ca468c4.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/895/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NTg0NzYwNTU=
title: Use white-space: pre-wrap on ALL table cell contents
state: closed
created_at: 2020-07-16T19:05:21Z
updated_at: 2020-07-17T01:26:08Z
closed_at: 2020-07-17T01:26:08Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Is there any reason NOT to apply `white-space: pre-wrap` to the contents of all table cells in Datasette?

  The default display mechanism of HTML (stripping leading/trailing slashes and collapsing all other whitespace) doesn't really make sense for displaying the kind of data that Datasette works with.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/896/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NTk1ODA0ODc=
title: Request method for retrieving the unparsed request body
state: closed
created_at: 2020-07-17T19:51:40Z
updated_at: 2020-07-17T20:16:02Z
closed_at: 2020-07-17T20:12:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'm writing a plugin (https://github.com/simonw/datasette-update-api/issues/2) that implements an API for inserting JSON data. As such, I'd like to `POST` a JSON blob rather than using `key=value` form encoded data.

  Right now there's a `request.post_vars()` method but no `request.post_body()` one:

  https://github.com/simonw/datasette/blob/c5f06bc356fb5917ef7fbb6fe4693f30d711cdb3/datasette/utils/asgi.py#L93-L103

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/897/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NTk4NzM2NjI=
title: datasette.utils.testing module
state: open
created_at: 2020-07-18T03:53:24Z
updated_at: 2020-07-18T03:57:46Z
author_association: OWNER
type: issue
body:
  The unit tests for plugins could benefit from reusing code from Datasette's own testing fixtures, e.g.:
  > I may need to borrow this function from Datasette for the tests:
  > https://github.com/simonw/datasette/blob/1f6a134369e6a7efaae9db469f15b1dd2b7f3709/tests/fixtures.py#L836-L851
  > 
  > It's not importable (it lives in `fixtures.py` and not in the `datasette` package that gets packaged for PyPI) - maybe I should fix that in Datasette by adding a `from datasette.utils.testing` module.
  _Originally posted by @simonw in https://github.com/simonw/datasette-update-api/issues/4#issuecomment-660419182_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/898/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjAzNTU5MDQ=
title: github-to-sqlite tags command for fetching tags
state: closed
created_at: 2020-07-18T20:14:12Z
updated_at: 2020-07-18T23:05:56Z
closed_at: 2020-07-18T21:52:15Z
author_association: MEMBER
body: Fetches paginated data from https://api.github.com/repos/simonw/datasette/tags
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjA0MTMyODE=
title: Rename tags.repo_id column to tags.repo
state: closed
created_at: 2020-07-18T22:13:46Z
updated_at: 2020-07-18T22:15:12Z
closed_at: 2020-07-18T22:15:12Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  For improved consistency with other tables. https://observablehq.com/@simonw/datasette-table-diagram

  ![datasette-table-diagram(1)](https://user-images.githubusercontent.com/9599/87862843-3cca4900-c909-11ea-9c76-58b3f4aca43f.png)

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/44/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjA0Mjk2MDE=
title: Fix the demo - it breaks because of the tags table change
state: closed
created_at: 2020-07-18T22:49:32Z
updated_at: 2020-07-18T23:03:14Z
closed_at: 2020-07-18T23:03:13Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://github.com/dogsheep/github-to-sqlite/runs/885773677
  ```
    File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 476, in save_tags
      db["tags"].insert_all(
    File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/sqlite_utils/db.py", line 1145, in insert_all
      result = self.db.conn.execute(query, params)
  sqlite3.OperationalError: table tags has no column named repo
  ```
  That's because I changed the name in #44. I thought this would be safe since no-one else could possibly be using this yet (it hadn't shipped in a release) but turns out I broke my demo!

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjA4Mjc1NDY=
title: How to setup a request limit per user
state: closed
created_at: 2020-07-19T13:08:25Z
updated_at: 2020-07-31T23:54:42Z
closed_at: 2020-07-31T23:54:42Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hello,

  Until now I'm using  datasette without any authentication system but I would like to setup a configuration  or limiting the number of requests per user (eventually by IP or with a cookie mechanism) and eventually allowing me to ban specific users/IPs.

  Is there a plugin available for this use case ? 
  If not what are your insights regarding this UC ?

  Should I write a plugin ? Should I deploy datasette behind a reverse proxy to manage this ?
   

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/899/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjE2MDU0ODk=
title: Some links don't honor base_url
state: closed
created_at: 2020-07-20T09:40:50Z
updated_at: 2020-10-23T19:44:04Z
closed_at: 2020-10-15T22:57:55Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi,

  I've been playing with Datasette behind Nginx (awesome tool, thanks !). It seems some URLs are OK but some aren't. For instance in https://github.com/simonw/datasette/blob/master/datasette/templates/query.html#L61 it seems that `url_csv` includes a `/` prefix, resulting in the `base_url` not beeing honored.

  Actually here, it seems that dropping the prefix `/` to make the link relative is enough (so it may not be strictly related to `base_url`).

  Additional information:

  ```
  datasette, version 0.45+0.gf1f581b.dirty
  ```

  Relevant Nginx configuration (note that all the trailing slashes have some effect):

  ```
    location /datasette/ {
      proxy_pass http://127.0.0.1:9001/;
      proxy_set_header Host $host;
    }
  ```

  Relelvant Datasette configuration (slashes matter too):

  ```
    --config base_url:/datasette/
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/900/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDUzODkwMjky
title: Use None as a default arg
state: closed
created_at: 2020-07-20T22:18:38Z
updated_at: 2020-07-31T18:42:39Z
closed_at: 2020-07-31T18:42:39Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/901
type: pull
body:
  When passing a mutable value as a default argument in a function, the default argument is mutated anytime that value is mutated. This poses a bug risk. Instead, use None as a default and assign the mutable value inside the function.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/901/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDUzOTk1MTc5
title: Don't install tests package
state: closed
created_at: 2020-07-21T01:08:50Z
updated_at: 2020-07-24T20:39:54Z
closed_at: 2020-07-24T20:39:54Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/902
body:
  The `exclude` argument to `find_packages` needs an iterable of package
  names.

  Fixes: #456

type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/902/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjMxNDUxMjI=
title: Add temporary plugin testing pattern to the testing docs
state: closed
created_at: 2020-07-21T16:22:34Z
updated_at: 2022-07-18T21:34:33Z
closed_at: 2022-07-18T21:31:22Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://til.simonwillison.net/pytest/registering-plugins-in-tests

  Would be useful to include this pattern on https://datasette.readthedocs.io/en/stable/testing_plugins.html

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/903/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjMyMjg5ODU=
state: closed
created_at: 2020-07-21T18:42:52Z
updated_at: 2020-10-23T19:44:05Z
closed_at: 2020-10-20T00:51:51Z
author_association: OWNER
type: issue
state_reason: completed
title:
  datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates

body:
  I tried using this block of template in a plugin and got an error:
  ```html
  {% block nav %}
      <p class="crumbs">
          <a href="{{ base_url }}">home</a> /
          <a href="{{ database_url(database) }}">{{ database }}</a> /
          <a href="{{ database_url(database) }}/{{ table|quote_plus }}">{{ table }}</a>
      </p>
      {{ super() }}
  {% endblock %}
  ```
  Error: `'database_url' is undefined`

  That's because `database_url` is only made available by the BaseView template here:

  https://github.com/simonw/datasette/blob/d6e03b04302a0852e7133dc030eab50177c37be7/datasette/views/base.py#L110-L125

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/904/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjMzMTc4NzU=
title: /database.db download should include content-length header
state: closed
created_at: 2020-07-21T21:23:48Z
updated_at: 2020-07-22T04:59:46Z
closed_at: 2020-07-22T04:52:45Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I can do this by modifying this function: https://github.com/simonw/datasette/blob/02dc6298bdbfb1d63e0d2a39ff597b5fcc60e06b/datasette/utils/asgi.py#L248-L270

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/905/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjM5NzY5NzY=
title: Add a table of contents to the README
state: closed
created_at: 2020-07-22T18:54:33Z
updated_at: 2020-07-23T17:46:07Z
closed_at: 2020-07-22T19:03:02Z
author_association: MEMBER
body: Using https://github.com/jonschlinkert/markdown-toc
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjQ0ODUwMjI=
title: Feature: pull request reviews and comments
state: open
created_at: 2020-07-23T13:43:45Z
updated_at: 2022-12-20T14:40:15Z
author_association: NONE
type: issue
body:
  Hi there! I saw your [presentation at Boston Python](https://www.meetup.com/bostonpython/events/271887195). I'm already a light user of Datasette (thank you!), but wasn't aware of this project.

  I've been working on a "pull request dashboard" to get a comprehensive view of the state of open PR's, esp. related to reviews (i.e., pending, approved, changes requested). Currently it's a CLI command, but I thought a Datasette UI might be fun.

  I see that PR's are available from the `issues` command, but I don't see reviews anywhere. From the [API docs](https://docs.github.com/en/rest/reference/pulls#reviews), it looks like there are separate endpoints for those (as well as pull requests in general). What do you think about adding that? Would you accept a PR? Any sense of the level of effort?

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjQ3OTMyNjA=
title: Yak shave
state: open
created_at: 2020-07-23T22:04:18Z
updated_at: 2020-07-23T22:04:18Z
author_association: NONE
type: issue
body:
  Just a quick note... The 23andme data is not exactly your genome, but a SNP chip of your genome. It's "some of your genotypes." Or about 0.1% of your genome. Nice work in any case! It deserves to be liberated!!!!!

reactions:
  {"url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjU0MDAyMjQ=
title: "allow": true for anyone, "allow": false for nobody
state: closed
created_at: 2020-07-24T20:28:10Z
updated_at: 2020-07-25T00:07:10Z
closed_at: 2020-07-25T00:05:04Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The "allow" syntax described at https://datasette.readthedocs.io/en/0.45/authentication.html#defining-permissions-with-allow-blocks currently says this:

  > An allow block can specify "no-one is allowed to do this" using an empty `{}`:
  > 
  > ```
  > {
  >     "allow": {}
  > }
  > ```

  `"allow": null` allows all access, though this isn't documented (it should be though).

  These are not very intuitive. How about also supporting `"allow": true` for "allow anyone" and `"allow": false` for "allow nobody"?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/906/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjU0MDM0MDM=
title: Allow documentation doesn't explain what happens with multiple allow keys
state: closed
created_at: 2020-07-24T20:34:40Z
updated_at: 2020-07-24T22:53:07Z
closed_at: 2020-07-24T22:53:07Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Documentation here: https://datasette.readthedocs.io/en/0.45/authentication.html#defining-permissions-with-allow-blocks

  Doesn't explain that with the following "allow" block:
  ```json
  {
    "allow": {
      "id": "simonw",
      "role": "staff"
    }
  }
  ```
  The rule will match if EITHER the id is simonw OR the role includes staff.

  The tests are missing this case too: https://github.com/simonw/datasette/blob/028f193dd6233fa116262ab4b07b13df7dcec9be/tests/test_utils.py#L504

  Related to #906

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/907/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjU0MDc2NjM=
title: Interactive debugging tool for "allow" blocks
state: closed
created_at: 2020-07-24T20:43:44Z
updated_at: 2020-07-25T00:06:15Z
closed_at: 2020-07-24T22:56:52Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > It might be good to have a little interactive tool which helps debug these things, since there are quite a few edge-cases and the damage caused if people use them incorrectly is substantial.
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/907#issuecomment-663726146_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/908/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjU3MDA0OTU=
title: CLI utility for inserting binary files into SQLite
state: closed
created_at: 2020-07-26T03:27:39Z
updated_at: 2020-07-27T07:10:41Z
closed_at: 2020-07-27T07:09:03Z
author_association: OWNER
type: issue
state_reason: completed
body:
  SQLite BLOB columns can store entire binary files. The challenge is inserting them, since they don't neatly fit into JSON objects.

  It would be great if the `sqlite-utils` CLI had a trick for helping with this.

  Inspired by https://github.com/simonw/datasette-media/issues/14

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/122/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjU3MDEyMTY=
title: --raw option for outputting binary content
state: closed
created_at: 2020-07-26T03:35:39Z
updated_at: 2020-07-26T16:44:11Z
closed_at: 2020-07-26T16:44:11Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Related to the `insert-files` work in #122 - it should be easy to get binary data back out of the database again.

  One way to do that could be:

      sqlite-utils files.db "select content from files where key = 'foo.jpg'" --raw

  The `--raw` option would cause just the contents of the first column to be output directly to stdout.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/123/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjU4MDI0MDU=
title: sqlite-utils query should support named parameters
state: closed
created_at: 2020-07-26T15:25:10Z
updated_at: 2020-07-30T22:57:51Z
closed_at: 2020-07-27T03:53:58Z
author_association: OWNER
type: issue
state_reason: completed
body:
  To help out with escaping - so you can run this:

      sqlite-utils query "insert into foo (blah) values (:blah)" --param blah `something here`

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/124/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjU4MTc1NzA=
title: Output binary columns in "sqlite-utils query" JSON
state: closed
created_at: 2020-07-26T16:47:02Z
updated_at: 2020-07-27T00:49:41Z
closed_at: 2020-07-27T00:48:45Z
author_association: OWNER
body: You get an error if you try to run a query that returns data from a BLOB.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/125/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjU4MTkwNDg=
title: Ability to insert binary data on the CLI using JSON
state: closed
created_at: 2020-07-26T16:54:14Z
updated_at: 2020-07-27T04:00:33Z
closed_at: 2020-07-27T03:59:45Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says "these columns are base64 encoded, store the decoded data in a BLOB".
  >
  > That would solve inserting binary data using JSON too.
  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/126/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjYwNDAzOTA=
title: Ability to insert files piped to insert-files stdin
state: closed
created_at: 2020-07-27T07:09:33Z
updated_at: 2020-07-30T03:08:52Z
closed_at: 2020-07-30T03:08:18Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Inserting files by piping them in should work - but since a filename cannot be derived this will need a `--name blah.gif` option.
  >
  >    cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif
  >
  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/127/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjY2MzkwNTE=
title: Support UUID and memoryview types
state: closed
created_at: 2020-07-27T23:08:34Z
updated_at: 2020-07-30T01:10:43Z
closed_at: 2020-07-30T01:10:43Z
author_association: OWNER
type: issue
state_reason: completed
body:
  `psycopg2` can return data from PostgreSQL as `uuid.UUID` or `memoryview` objects. These should to be supported by `sqlite-utils` - mainly for https://github.com/simonw/db-to-sqlite

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/128/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Njc0NjcxMjg=
title: AsgiFileDownload: filename not correctly passed
state: closed
created_at: 2020-07-29T00:41:43Z
updated_at: 2020-07-30T00:56:17Z
closed_at: 2020-07-29T21:34:48Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://github.com/simonw/datasette/blob/3c33b421320c0be81a625ca7307b2e4416a9ed5b/datasette/utils/asgi.py#L396-L405
  `self.filename` should be passed to `asgi_send_file()`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/909/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDU4NDM1NTky
title: Update pytest requirement from <5.5.0,>=5.2.2 to >=5.2.2,<6.1.0
state: closed
created_at: 2020-07-29T13:21:17Z
updated_at: 2020-07-29T21:26:05Z
closed_at: 2020-07-29T21:26:04Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/910
type: pull
body:
  Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/releases">pytest's releases</a>.</em></p>
  <blockquote>
  <h2>6.0.0</h2>
  <h1>pytest 6.0.0 (2020-07-28)</h1>
  <p>(<strong>Please see the full set of changes for this release also in the 6.0.0rc1 notes below</strong>)</p>
  <h2>Breaking Changes</h2>
  <ul>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/5584">#5584</a>: <strong>PytestDeprecationWarning are now errors by default.</strong></p>
  <p>Following our plan to remove deprecated features with as little disruption as
  possible, all warnings of type <code>PytestDeprecationWarning</code> now generate errors
  instead of warning messages.</p>
  <p><strong>The affected features will be effectively removed in pytest 6.1</strong>, so please consult the
  <a href="https://docs.pytest.org/en/latest/deprecations.html">Deprecations and Removals</a>
  section in the docs for directions on how to update existing code.</p>
  <p>In the pytest <code>6.0.X</code> series, it is possible to change the errors back into warnings as a
  stopgap measure by adding this to your <code>pytest.ini</code> file:</p>
  <pre lang="{.sourceCode" data-meta=".ini}"><code>[pytest]
  filterwarnings =
      ignore::pytest.PytestDeprecationWarning
  </code></pre>
  <p>But this will stop working when pytest <code>6.1</code> is released.</p>
  <p><strong>If you have concerns</strong> about the removal of a specific feature, please add a
  comment to <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/5584">#5584</a>.</p>
  </li>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7472">#7472</a>: The <code>exec_()</code> and <code>is_true()</code> methods of <code>_pytest._code.Frame</code> have been removed.</p>
  </li>
  </ul>
  <h2>Features</h2>
  <ul>
  <li><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7464">#7464</a>: Added support for NO_COLOR and FORCE_COLOR environment variables to control colored output.</li>
  </ul>
  <h2>Improvements</h2>
  <ul>
  <li><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7467">#7467</a>: <code>--log-file</code> CLI option and <code>log_file</code> ini marker now create subdirectories if needed.</li>
  <li><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7489">#7489</a>: The pytest.raises function has a clearer error message when <code>match</code> equals the obtained string but is not a regex match. In this case it is suggested to escape the regex.</li>
  </ul>
  <h2>Bug Fixes</h2>
  <ul>
  <li><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7392">#7392</a>: Fix the reported location of tests skipped with <code>@pytest.mark.skip</code> when <code>--runxfail</code> is used.</li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst">pytest's changelog</a>.</em></p>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pytest-dev/pytest/commit/41a453959441d9b03cba3e47730efca27fa2f252"><code>41a4539</code></a> Add link to 6.0.0rc1 changelog</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/45ced1dc056d586fe3714823fa033cab27055c9f"><code>45ced1d</code></a> Update doc/en/announce/release-6.0.0.rst</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/1e4b8d447cfaaf4ee7c4636d2a03cf484d06f1cd"><code>1e4b8d4</code></a> Prepare release version 6.0.0</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/38029828d1d6bdc15b63a873142b1e91265e1a1c"><code>3802982</code></a> Support generating major releases using issue comments (<a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7548">#7548</a>)</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/c2c0b7a5426ed74ba0df1bbc4997ffe45e302ccf"><code>c2c0b7a</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7545">#7545</a> from asottile/pylib_in_docs</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/9818899df4eacb65a4ec1397a93b5ebcf43e825c"><code>9818899</code></a> remove usage of pylib in docs</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/3a060b77e81ebf990159e59cc8f8de26ad998e12"><code>3a060b7</code></a> Revert change to traceback repr (<a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7535">#7535</a>)</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/7ec6401ffabf79d52938ece5b8ff566a8b9c260e"><code>7ec6401</code></a> Change pytest deprecation warnings into errors for 6.0 release (<a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7362">#7362</a>)</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/a9799f0b35fe8fcc88ebd19b007cd2c6ac6b2a4f"><code>a9799f0</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7531">#7531</a> from bluetech/changelog-mypy-version</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/102360b49051e7d2407003445157b0115ae18234"><code>102360b</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7519">#7519</a> from hroncok/pytest_warning_captured_deprecated</li>
  <li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest/compare/5.2.2...6.0.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/910/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjgwNjQwMjY=
title: Rethink the --name option to "datasette publish"
state: open
created_at: 2020-07-29T18:49:49Z
updated_at: 2020-07-29T18:49:49Z
author_association: OWNER
type: issue
body:
  `--name` works inconsistently across the different publish providers - on Cloud Run you should use `--service` instead for example. Need to review it across all of them and either remove it or clarify what it does.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/911/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjgwNjQ3Nzg=
title: Add "publishing to Vercel" to the publish docs
state: closed
created_at: 2020-07-29T18:50:58Z
updated_at: 2020-07-31T17:06:35Z
closed_at: 2020-07-31T17:06:35Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://datasette.readthedocs.io/en/0.45/publish.html#datasette-publish currently only lists Cloud Run, Heroku and Fly. It should list Vercel too.

  (I should probably rename `datasette-publish-now` to `datasette-publish-vercel`)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/912/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NjgzMDg3Nzc=
title: "insert-files --sqlar" for creating SQLite archives
state: closed
created_at: 2020-07-30T02:28:29Z
updated_at: 2020-07-30T22:41:01Z
closed_at: 2020-07-30T22:40:55Z
author_association: OWNER
type: issue
state_reason: completed
body:
  A `--sqlar` option could cause `insert-files` to behave in the same way as SQLite's own sqlar mechanism.

  https://www.sqlite.org/sqlar.html and https://sqlite.org/sqlar/doc/trunk/README.md

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/129/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzAyMDkzMzE=
title: Mechanism for passing additional options to `datasette my.db` that affect plugins
state: open
created_at: 2020-07-31T20:38:26Z
updated_at: 2021-01-04T20:04:11Z
author_association: OWNER
type: issue
body:
  > It's a shame there's no obvious mechanism for passing additional options to `datasette my.db` that affect how plugins work.
  >
  >The only way I can think of at the moment is via environment variables:
  >
  >     DATASETTE_INSERT_UNSAFE=1 datasette my.db
  >
  >This will have to do for the moment - it's ugly enough that people will at least know they are doing something unsafe, which is the goal here.
  _Originally posted by @simonw in https://github.com/simonw/datasette-insert/issues/15#issuecomment-667346438_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/913/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzEwNTY3ODg=
title: "Object of type bytes is not JSON serializable" for _nl=on
state: closed
created_at: 2020-08-01T17:43:10Z
updated_at: 2020-08-16T21:10:27Z
closed_at: 2020-08-16T18:26:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://latest.datasette.io/fixtures/binary_data.json?_sort_desc=data&_shape=array returns this:
  ```json
  [
    {
      "rowid": 1,
      "data": "this is binary data"
    }
  ]
  ```
  But adding `&_nl=on` returns this: https://latest.datasette.io/fixtures/binary_data.json?_sort_desc=data&_shape=array&_nl=on
  ```json
  {
    "ok": false,
    "error": "Object of type bytes is not JSON serializable",
    "status": 500,
    "title": null
  }
  ```
  I found this error by running `wget -r 127.0.0.1:8001` against my local `fixtures.db`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/914/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzExMzAzNzE=
title: Support tokenize option for FTS
state: closed
created_at: 2020-08-01T19:27:22Z
updated_at: 2020-08-01T20:51:28Z
closed_at: 2020-08-01T20:51:14Z
author_association: OWNER
type: issue
state_reason: completed
body:
  FTS5 supports things like porter stemming using a `tokenize=` option:

  https://www.sqlite.org/fts5.html#tokenizers

  Something like this in code:
  ```
              CREATE VIRTUAL TABLE [{table}_fts] USING {fts_version} (
                  {columns},
                  tokenize='porter',
                  content=[{table}]
              );
  ```
  I tried this out just now and it worked exactly as expected.

  So... `db[table].enable_fts(...) should accept a 'tokenize=` argument, and `sqlite-utils enable-fts ...` should support a `--tokenize` option.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/130/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzE3NjMxNjQ=
title: Refactor TableView class so things like datasette-graphql can reuse the logic
state: closed
created_at: 2020-08-03T03:13:33Z
updated_at: 2020-08-18T22:28:37Z
closed_at: 2020-08-18T22:28:37Z
author_association: OWNER
type: issue
state_reason: completed
body:
  _Originally posted by @simonw in https://github.com/simonw/datasette-graphql/issues/2#issuecomment-667780040_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/915/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzI0MjE0MTE=
title: Support reverse pagination (previous page, has-previous-items)
state: open
created_at: 2020-08-04T00:32:06Z
updated_at: 2021-04-03T23:43:11Z
author_association: OWNER
type: issue
body:
  I need this for `datasette-graphql` for full compatibility with the way Relay likes to paginate - using cursors for paginating backwards as well as for paginating forwards.

  > This may be the kick I need to get Datasette pagination to work in reverse too.
  _Originally posted by @simonw in https://github.com/simonw/datasette-graphql/issues/2#issuecomment-668305853_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/916/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzM2MDI4NTc=
title: Define a view that displays photos correctly
state: open
created_at: 2020-08-05T14:53:39Z
updated_at: 2020-08-05T14:53:39Z
author_association: MEMBER
type: issue
body:
  The `photos` table stores data like this:

  id | createdAt | source | prefix | suffix | width | height | visibility | created ▲ | user
  -- | -- | -- | -- | -- | -- | -- | -- | -- | --
  5e12c9708506bc000840262a | January 06, 2020 - 05:45:20 UTC | Swarm for iOS 1 | https://fastly.4sqi.net/img/general/ | /15889193_AXxGk4I1nbzUZuyYqObgbXdJNyEHiwj6AUDq0tPZWtw.jpg | 1920 | 1440 | public | 2020-01-06T05:45:20 | 15889193

  The photo URL can be derived from those pieces - define a SQL view which does that (using `datasette-json-html` to display the pictures)

reactions:
  {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzU1OTQzMjU=
title: Idea: "datasette publish" option for "only if the data has changed
state: open
created_at: 2020-08-08T21:58:27Z
updated_at: 2020-08-08T21:58:27Z
author_association: OWNER
type: issue
body:
  This is a pattern I often find myself needing. I usually implement this in GitHub Actions like this:

  https://github.com/simonw/covid-19-datasette/blob/efa01c39abc832b8641fc2a92840cc3acae2fb08/.github/workflows/scheduled.yml#L52-L63

  ```yaml
      - name: Set variables to decide if we should deploy
        id: decide_variables
        run: |-
          echo "##[set-output name=latest;]$(datasette inspect covid.db | jq '.covid.hash' -r)"
          echo "##[set-output name=deployed;]$(curl -s https://covid-19.datasettes.com/-/databases.json | jq '.[0].hash' -r)"
      - name: Set up Cloud Run
        if: github.event_name == 'workflow_dispatch' || steps.decide_variables.outputs.latest != steps.decide_variables.outputs.deployed
        uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
  ```
  This is pretty fiddly. It might be good for `datasette publish` to grow a helper option that does effectively this - hashes the databases (and the `metadata.json`) and compares them to the deployed version.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/917/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzU3MjQ5NTE=
title: Security issue: read-only canned queries leak CSRF token in URL
state: closed
created_at: 2020-08-09T16:03:01Z
updated_at: 2020-08-09T16:56:48Z
closed_at: 2020-08-09T16:11:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The HTML form for a read-only canned query includes the hidden CSRF token field added in #798 for writable canned queries (#698).

  This means that submitting those read-only forms exposes the CSRF token in the URL - for example on https://latest.datasette.io/fixtures/neighborhood_search submitting the form took me to:

  https://latest.datasette.io/fixtures/neighborhood_search?text=down&csrftoken=IlFubnoxVVpLU1NGT3NMVUoi.HbOPd2YH_epQmp8f_aAt0s-MxtU

  This token could potentially leak to an attacker if the resulting page has a link to an external site on it and the user clicks the link, since the token would be exposed in the referral logs.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/918/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzU3MjczNjY=
title: Travis should not build the master branch, only the main branch
state: closed
created_at: 2020-08-09T16:18:25Z
updated_at: 2020-08-09T16:26:18Z
closed_at: 2020-08-09T16:19:37Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Caused by #849 - since we are mirroring the two branches (to ensure old links to `master` keep working) Travis is building both.

  The following in `.travis.yml` should fix that:
  ```
  branches:
    except:
    - master
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/919/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzU3NTMwNDI=
title: sqlite-utils insert: options for column types
state: open
created_at: 2020-08-09T18:59:11Z
updated_at: 2022-03-15T13:21:42Z
author_association: OWNER
type: issue
body:
  The `insert` command currently results in string types for every column - at least when used against CSV or TSV inputs.

  It would be useful if you could do the following:

  - automatically detects the column types based on eg the first 1000 records
  - explicitly state the rule for specific columns

  `--detect-types` could work for the former - or it could do that by default and allow opt-out using `--no-detect-types`

  For specific columns maybe this:

      sqlite-utils insert db.db images images.tsv \
        --tsv \
        -c id int \
        -c score float

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/131/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzU4Mzk1MTI=
title: Features for enabling and disabling WAL mode
state: closed
created_at: 2020-08-10T03:25:44Z
updated_at: 2020-08-10T18:59:35Z
closed_at: 2020-08-10T18:59:35Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I finally figured out how to enable WAL - turns out it's a property of the database file itself: https://github.com/simonw/til/blob/master/sqlite/enabling-wal-mode.md

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/132/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzcwMzcwNDM=
title: Add homebrew installation to documentation
state: closed
created_at: 2020-08-11T16:54:31Z
updated_at: 2020-08-11T22:53:07Z
closed_at: 2020-08-11T22:52:46Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > ```
  > $ brew tap simonw/datasette
  > $ brew install simonw/datasette/datasette
  > $ datasette --version
  > datasette, version 0.46
  > ```
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/335#issuecomment-672088880_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/923/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzcyMjc5MTI=
title: "datasette install" and "datasette uninstall" commands
state: closed
created_at: 2020-08-11T22:04:32Z
updated_at: 2020-08-11T22:34:37Z
closed_at: 2020-08-11T22:32:12Z
author_association: OWNER
type: issue
state_reason: completed
body:
  When installing Datasette plugins it's crucial that they end up in the same virtual environment as Datasette itself.

  It's not necessarily obvious how to do this, especially if you install Datasette via pipx or homebrew.

  Solution: `datasette install datasette-vega` and `datasette uninstall datasette-vega` commands that know how to install to the correct place - a very thin wrapper around `pip install`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/925/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzcyNTA4MzQ=
title: datasette fixtures.db --get "/fixtures.json"
state: closed
created_at: 2020-08-11T22:55:36Z
updated_at: 2020-08-12T00:26:17Z
closed_at: 2020-08-12T00:24:42Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I can expose ALL of Datasette's functionality on the command-line (without even running a web server) by adding `--get` and `--post` options to `datasette serve`.

      datasette fixtures.db --get "/fixtures.json"

  This would instantiate the Datasette ASGI app, run a fake request for `/fixtures.json` through it, dump the results out to standard output and quit.

  A `--post` option could do the same for a POST request. Treating that as a stretch goal for the moment.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/926/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDY2NDEwNzU1
title: 'datasette --get' option, refs #926
state: closed
created_at: 2020-08-11T23:31:52Z
updated_at: 2020-08-12T00:24:42Z
closed_at: 2020-08-12T00:24:41Z
author_association: OWNER
pull_request: simonw/datasette/pulls/927
body: Refs #926, #898
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/927/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzcyNzI2MTg=
title: Test failures caused by failed attempts to mock pip
state: closed
created_at: 2020-08-11T23:53:18Z
updated_at: 2022-02-23T16:19:47Z
closed_at: 2020-08-12T00:07:49Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Errors like this one:

  https://github.com/simonw/datasette/pull/927/checks?check_run_id=973559696

  ```
  2020-08-11T23:36:39.8801334Z =================================== FAILURES ===================================
  2020-08-11T23:36:39.8802411Z _________________________________ test_install _________________________________
  2020-08-11T23:36:39.8803242Z 
  2020-08-11T23:36:39.8804935Z thing = <module 'pip._internal.cli' from '/opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/pip/_internal/cli/__init__.py'>
  2020-08-11T23:36:39.8806663Z comp = 'main', import_path = 'pip._internal.cli.main'
  2020-08-11T23:36:39.8807696Z 
  2020-08-11T23:36:39.8808728Z     def _dot_lookup(thing, comp, import_path):
  2020-08-11T23:36:39.8810573Z         try:
  2020-08-11T23:36:39.8812262Z >           return getattr(thing, comp)
  2020-08-11T23:36:39.8817136Z E           AttributeError: module 'pip._internal.cli' has no attribute 'main'
  2020-08-11T23:36:39.8843043Z 
  2020-08-11T23:36:39.8855951Z /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/unittest/mock.py:1215: AttributeError
  2020-08-11T23:36:39.8873372Z 
  2020-08-11T23:36:39.8877803Z During handling of the above exception, another exception occurred:
  2020-08-11T23:36:39.8906532Z 
  2020-08-11T23:36:39.8925767Z     def get_src_prefix():
  2020-08-11T23:36:39.8928277Z         # type: () -> str
  2020-08-11T23:36:39.8930068Z         if running_under_virtualenv():
  2020-08-11T23:36:39.8949721Z             src_prefix = os.path.join(sys.prefix, 'src')
  2020-08-11T23:36:39.8951813Z         else:
  2020-08-11T23:36:39.8969014Z             # FIXME: keep src in cwd for now (it is not a temporary folder)
  2020-08-11T23:36:39.9012110Z             try:
  2020-08-11T23:36:39.9013489Z >               src_prefix = os.path.join(os.getcwd(), 'src')
  2020-08-11T23:36:39.9014538Z E               FileNotFoundError: [Errno 2] No such file or directory
  2020-08-11T23:36:39.9016122Z 
  2020-08-11T23:36:39.9017617Z /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/pip/_internal/locations.py:50: FileNotFoundError
  2020-08-11T23:36:39.9018802Z 
  2020-08-11T23:36:39.9020070Z During handling of the above exception, another exception occurred:
  2020-08-11T23:36:39.9020930Z 
  2020-08-11T23:36:39.9022275Z args = (), keywargs = {}
  2020-08-11T23:36:39.9023183Z 
  2020-08-11T23:36:39.9024077Z     @wraps(func)
  2020-08-11T23:36:39.9024984Z     def patched(*args, **keywargs):
  2020-08-11T23:36:39.9028770Z >       with self.decoration_helper(patched,
  2020-08-11T23:36:39.9031861Z                                     args,
  2020-08-11T23:36:39.9038358Z                                     keywargs) as (newargs, newkeywargs):
  2020-08-11T23:36:39.9039654Z 
  2020-08-11T23:36:39.9040566Z /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/unittest/mock.py:1322: 
  2020-08-11T23:36:39.9041492Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/928/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2NzczMjYxNTU=
title: Datasette sdist is missing templates (hence broken when installing from Homebrew)
state: closed
created_at: 2020-08-12T02:20:16Z
updated_at: 2020-08-12T03:30:59Z
closed_at: 2020-08-12T03:30:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Pretty nasty bug this: I'm getting 500 errors for all pages that try to render a template after installing the newly released Datasette 0.47 - both from `pip install` and via Homebrew.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/930/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Nzc4Mzk5Nzk=
title: Release a sdist to PyPI
state: closed
created_at: 2020-08-12T16:55:09Z
updated_at: 2020-08-12T17:05:06Z
closed_at: 2020-08-12T17:05:06Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://pypi.org/project/sqlite-utils/#files currently just has a wheel. I need this to package for homebrew: https://github.com/simonw/homebrew-datasette/issues/10

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/133/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Nzc5MjY2MTM=
title: Docker container is no longer being pushed (it's stuck on 0.45)
state: closed
created_at: 2020-08-12T19:33:03Z
updated_at: 2020-08-12T21:36:20Z
closed_at: 2020-08-12T21:36:20Z
author_association: OWNER
type: issue
state_reason: completed
body:
  e.g. https://travis-ci.org/github/simonw/datasette/jobs/717123725

  Here's how it broke:
  ```
  --2020-08-12 03:08:17--  https://www.gaia-gis.it/gaia-sins/freexl-1.0.5.tar.gz
  Resolving www.gaia-gis.it (www.gaia-gis.it)... 212.83.162.51
  Connecting to www.gaia-gis.it (www.gaia-gis.it)|212.83.162.51|:443... connected.
  HTTP request sent, awaiting response... 404 Not Found
  2020-08-12 03:08:18 ERROR 404: Not Found.
  The command '/bin/sh -c wget "https://www.gaia-gis.it/gaia-sins/freexl-1.0.5.tar.gz" && tar zxf freexl-1.0.5.tar.gz     && cd freexl-1.0.5 && ./configure && make && make install' returned a non-zero code: 8
  The command "docker build -f Dockerfile -t $REPO:$TRAVIS_TAG ." exited with 8.
  0.07s$ docker tag $REPO:$TRAVIS_TAG $REPO:latest
  Error response from daemon: No such image: [secure]/datasette:0.47.1
  The command "docker tag $REPO:$TRAVIS_TAG $REPO:latest" exited with 1.
  0.08s$ docker push $REPO
  The push refers to repository [docker.io/[secure]/datasette]
  An image does not exist locally with the tag: [secure]/datasette
  The command "docker push $REPO" exited with 1.
  cache.2
  store build cache
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/931/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Nzg3NjA5ODg=
title: End-user documentation
state: open
created_at: 2020-08-13T22:04:39Z
updated_at: 2022-03-08T15:20:48Z
author_association: OWNER
type: issue
body:
  Datasette's documentation is aimed at people who install and configure it.

  What about end users of preconfigured and deployed Datasette instances?

  Something that can be linked to from the Datasette UI would be really useful.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/932/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Nzk2Mzc1MDE=
title: --get doesn't fully invoke the startup routine
state: closed
created_at: 2020-08-15T20:30:25Z
updated_at: 2020-08-15T20:53:49Z
closed_at: 2020-08-15T20:53:49Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://github.com/simonw/datasette/blob/7702ea602188899ee9b0446a874a6a9b546b564d/datasette/cli.py#L417-L433

  Spotted this working on https://github.com/simonw/latest-datasette-with-all-plugins/issues/3 - I'd like to be able to use `datasette --get /` as a sanity checking test, but that doesn't work if the init hooks aren't fully executed.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/934/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Nzk2NDY3MTA=
title: db.execute_write_fn(create_tables, block=True) hangs a thread if connection fails
state: closed
created_at: 2020-08-15T21:49:17Z
updated_at: 2020-08-15T22:35:33Z
closed_at: 2020-08-15T22:35:33Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Discovered in https://github.com/simonw/latest-datasette-with-all-plugins/issues/3#issuecomment-674449757

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/935/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDY4MzcwNjU4
title: Don't hang in db.execute_write_fn() if connection fails
state: closed
created_at: 2020-08-15T22:20:12Z
updated_at: 2020-08-15T22:35:33Z
closed_at: 2020-08-15T22:35:32Z
author_association: OWNER
pull_request: simonw/datasette/pulls/936
body: Refs #935
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/936/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDY4Mzc3MjEy
title: Docs now live at docs.datasette.io
state: closed
created_at: 2020-08-15T23:53:52Z
updated_at: 2020-08-15T23:57:06Z
closed_at: 2020-08-15T23:57:05Z
author_association: OWNER
pull_request: simonw/datasette/pulls/937
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/937/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Nzk3MDAyNjk=
title: Pass columns to extra CSS/JS/etc plugin hooks
state: closed
created_at: 2020-08-16T06:37:47Z
updated_at: 2020-09-30T20:36:12Z
closed_at: 2020-08-16T18:09:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'd like `datasette-cluster-map` to only add links to JavaScript on pages that have tables with latitude and longitude columns.

  Passing the names of the columns to the plugin hook can support this and will be backwards compatible thanks to pluggy.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/938/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Nzk3Nzk3OTc=
title: extra_ plugin hooks should take the same arguments
state: closed
created_at: 2020-08-16T16:04:54Z
updated_at: 2020-08-16T18:25:05Z
closed_at: 2020-08-16T16:50:29Z
author_association: OWNER
type: issue
state_reason: completed
body:
  - [x] `extra_css_urls(template, database, table, datasette)`
  - [x] `extra_js_urls(template, database, table, datasette)`
  - [x] `extra_body_script(template, database, table, view_name, datasette)`
  - [x] `extra_template_vars(template, database, table, view_name, request, datasette)`

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/938#issuecomment-674544691_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/939/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2Nzk4MDgxMjQ=
title: Move CI to GitHub Issues
state: closed
created_at: 2020-08-16T19:06:08Z
updated_at: 2020-09-14T22:09:35Z
closed_at: 2020-09-14T22:09:35Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It looks like the tests take 3m33s to run in GitHub Actions, but they're taking more than 8 minutes in Travis

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/940/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDY4NDg0MDMx
title: Run CI on GitHub Actions, not Travis
state: closed
created_at: 2020-08-16T19:13:39Z
updated_at: 2020-08-18T05:09:36Z
closed_at: 2020-08-18T05:09:35Z
author_association: OWNER
pull_request: simonw/datasette/pulls/941
body: Refs #940
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/941/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODEwODY2NTk=
title: emojis command
state: closed
created_at: 2020-08-18T14:26:26Z
updated_at: 2020-08-18T14:52:13Z
closed_at: 2020-08-18T14:52:13Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  For fun - it can import https://api.github.com/emojis - maybe with an option to fetch the binary representations in addition to the URLs.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/47/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDY5NjUxNzMy
title: Add pull requests
state: closed
created_at: 2020-08-18T17:58:44Z
updated_at: 2020-11-29T23:51:09Z
closed_at: 2020-11-29T23:51:09Z
author_association: CONTRIBUTOR
pull_request: dogsheep/github-to-sqlite/pulls/48
type: pull
body:
  ref #46 

  Issues don't have merge information on them, which means that PRs need to be pulled separately.

  Did my best to mimic the API of issues.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODEzMzQ5MTI=
title: Support column descriptions in metadata.json
state: closed
created_at: 2020-08-18T20:52:00Z
updated_at: 2022-01-13T22:21:42Z
closed_at: 2021-08-12T23:53:24Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Could look something like this:
  ```json
  {
      "title": "Five Thirty Eight",
      "license": "CC Attribution 4.0 License",
      "license_url": "https://creativecommons.org/licenses/by/4.0/",
      "source": "fivethirtyeight/data on GitHub",
      "source_url": "https://github.com/fivethirtyeight/data",
      "databases": {
          "fivethirtyeight": {
              "tables": {
                  "mueller-polls/mueller-approval-polls": {
                      "description_html": "<p>....</p>",
                      "columns": {
                          "name_of_column": "column_description goes here"
  }
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/942/reactions", "total_count": 4, "+1": 4, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODEzNzU0NjY=
title: await datasette.client.get(path) mechanism for executing internal requests
state: closed
created_at: 2020-08-18T22:17:42Z
updated_at: 2020-10-09T17:22:55Z
closed_at: 2020-10-09T16:11:26Z
author_association: OWNER
type: issue
state_reason: completed
body:
  `datasette-graphql` works by making internal requests to the TableView class (in order to take advantage of existing pagination logic, plus options like `?_search=` and `?_where=`) - see #915 

  I want to support a `mod_rewrite` style mechanism for putting nicer URLs on top of Datasette pages - I botched that together for a project here using an internal ASGI proxying trick: https://github.com/natbat/tidepools_near_me/commit/ec102c6da5a5d86f17628740d90b6365b671b5e1

  If the `datasette` object provided a documented method for executing internal requests (in a way that makes sense with logging etc - i.e. doesn't get logged as a separate request) both of these use-cases would be much neater.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/943/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODE1MTY5NzY=
title: Path parameters for custom pages
state: closed
created_at: 2020-08-19T03:25:17Z
updated_at: 2020-09-14T03:21:45Z
closed_at: 2020-09-14T02:34:58Z
author_association: OWNER
type: issue
state_reason: completed
body:
  [Custom pages](https://docs.datasette.io/en/stable/custom_templates.html#custom-pages) let you e.g. create a `templates/pages/about.html` page and have it automatically served at `/about`.

  It would be useful if these pages could capture path patterns. I like the Python format string syntax for this (also used by Starlette): `/foo/bar/{slug}`.

  So... how about embedding those patterns in the filenames themselves?

      templates/pages/museums/{slug}.html

  Would capture any hits to `/museums/something` and use that page to serve them.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/944/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDY5OTQ0OTk5
title: Document the use of --stop_after with favorites, refs #20
state: closed
created_at: 2020-08-19T06:10:52Z
updated_at: 2021-08-20T00:02:11Z
closed_at: 2021-08-20T00:02:11Z
author_association: CONTRIBUTOR
pull_request: dogsheep/twitter-to-sqlite/pulls/49
body: (I discovered this trawling the issues for how to use --since with favorites)
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODIwMDU1MzU=
title: datasette install -U for upgrading packages
state: closed
created_at: 2020-08-19T17:12:04Z
updated_at: 2020-08-28T04:53:14Z
closed_at: 2020-08-19T17:20:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This will also give Homebrew a way to upgrade Datasette itself without having to wait for the latest packaged version to land in Homebrew core.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/945/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODIxODQwNTA=
title: Exception in tracing code
state: closed
created_at: 2020-08-19T21:12:27Z
updated_at: 2020-09-15T20:16:50Z
closed_at: 2020-09-15T20:16:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  When using `?_trace=1`:
  ```
  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/rockybeaches-09H592sC/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 390, in run_asgi
      result = await app(self.scope, self.receive, self.send)
    File "/Users/simon/.local/share/virtualenvs/rockybeaches-09H592sC/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
      return await self.app(scope, receive, send)
    File "/Users/simon/.local/share/virtualenvs/rockybeaches-09H592sC/lib/python3.8/site-packages/datasette/utils/asgi.py", line 150, in __call__
      await self.app(scope, receive, send)
    File "/Users/simon/.local/share/virtualenvs/rockybeaches-09H592sC/lib/python3.8/site-packages/datasette/tracer.py", line 137, in __call__
      await self.app(scope, receive, wrapped_send)
    File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 120, in __exit__
      next(self.gen)
    File "/Users/simon/.local/share/virtualenvs/rockybeaches-09H592sC/lib/python3.8/site-packages/datasette/tracer.py", line 63, in capture_traces
      del tracers[task_id]
  KeyError: 4575365856
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/946/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODM4MDQxNzI=
title: --load-extension option for sqlite-utils query
state: closed
created_at: 2020-08-21T20:12:42Z
updated_at: 2020-08-21T21:06:26Z
closed_at: 2020-08-21T20:54:19Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I got this error:
  ```
  % sqlite-utils calands.db 'create table superunits_with_maps_view_concrete as select * from superunits_with_maps_view'
  Traceback (most recent call last):
  ...
      cursor = db.conn.execute(sql, dict(param))
  sqlite3.OperationalError: no such function: AsGeoJSON
  ```
  A `--load-extension=/usr/local/lib/mod_spatialite.dylib` option (imitating the same option for Datasette) would help.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/134/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODM4MDU0MzQ=
title: Code for finding SpatiaLite in the usual locations
state: closed
created_at: 2020-08-21T20:15:34Z
updated_at: 2022-02-05T00:04:26Z
closed_at: 2020-08-21T20:30:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I built this for `shapefile-to-sqlite` but it would be useful in `sqlite-utils` too:

  https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L16-L19

  ```python
  SPATIALITE_PATHS = (
      "/usr/lib/x86_64-linux-gnu/mod_spatialite.so",
      "/usr/local/lib/mod_spatialite.dylib",
  )
  ```

  https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L105-L109

  ```python
  def find_spatialite():
      for path in SPATIALITE_PATHS:
          if os.path.exists(path):
              return path
      return None
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/135/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODM4MTI2NDI=
title: --load-extension=spatialite shortcut option
state: closed
created_at: 2020-08-21T20:31:25Z
updated_at: 2022-02-05T00:04:26Z
closed_at: 2020-10-16T19:14:32Z
author_association: OWNER
type: issue
state_reason: completed
body:
  In conjunction with #135 - this would do the same thing as `--load-extension=path-to-spatialite` (see #134)

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/136/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODM4MzA0MTY=
title: --load-extension for other sqlite-utils commands
state: closed
created_at: 2020-08-21T21:12:56Z
updated_at: 2020-10-16T19:14:32Z
closed_at: 2020-10-16T19:14:32Z
author_association: OWNER
type: issue
state_reason: completed
body:
  e.g. for this:
  ```
  calands-datasette % sqlite-utils tables calands.db --counts
  [{"table": "spatial_ref_sys", "count": 4924},
   {"table": "spatialite_history", "count": 14},
   {"table": "sqlite_sequence", "count": 1},
   {"table": "geometry_columns", "count": 2},
   {"table": "spatial_ref_sys_aux", "count": 4873},
   {"table": "views_geometry_columns", "count": 0},
   {"table": "virts_geometry_columns", "count": 0},
   {"table": "geometry_columns_statistics", "count": 2},
   {"table": "views_geometry_columns_statistics", "count": 0},
   {"table": "virts_geometry_columns_statistics", "count": 0},
   {"table": "geometry_columns_field_infos", "count": 0},
   {"table": "views_geometry_columns_field_infos", "count": 0},
   {"table": "virts_geometry_columns_field_infos", "count": 0},
   {"table": "geometry_columns_time", "count": 2},
   {"table": "geometry_columns_auth", "count": 2},
   {"table": "views_geometry_columns_auth", "count": 0},
   {"table": "virts_geometry_columns_auth", "count": 0},
  Traceback (most recent call last):
    File "/usr/local/bin/sqlite-utils", line 8, in <module>
      sys.exit(cli())
    File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py", line 143, in tables
      for line in output_rows(_iter(), headers, nl, arrays, json_cols):
    File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py", line 922, in output_rows
      for row, next_row in itertools.zip_longest(current_iter, next_iter):
    File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py", line 123, in _iter
      row.append(db[name].count)
    File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/db.py", line 458, in count
      return self.db.conn.execute(
  sqlite3.OperationalError: no such module: VirtualSpatialIndex
  ```
  The `tables` command could take `--load-extension` too - as could `rows` and other similar commands.

  Follow-on from #134 

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/137/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODQxMTE5NTM=
title: datasette --get exit code should reflect HTTP errors
state: closed
created_at: 2020-08-23T04:17:08Z
updated_at: 2020-09-11T21:33:15Z
closed_at: 2020-09-11T21:33:15Z
author_association: OWNER
type: issue
state_reason: completed
body:
  If you run `datasette . --get /` and the result is a 500 or 404 error (anything that's not a 200 or a 30x) the exit code from the command should not be 0.

  It should still output the returned content to stdout.

  This will help with writing soundness checks, as seen in https://til.simonwillison.net/til/til/github-actions_grep-tests.md

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/947/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODQxMTg5NTA=
title: extracts= doesn't configure foreign keys
state: closed
created_at: 2020-08-23T05:21:15Z
updated_at: 2020-09-24T22:47:01Z
closed_at: 2020-09-24T22:46:52Z
author_association: OWNER
type: issue
state_reason: completed
body:
  In using `extracts=` for `shapefiles-to-sqlite` in https://github.com/simonw/shapefile-to-sqlite/issues/9 I've run into a couple of pretty serious flaws:

  - The columns in the original table are still `TEXT` even when the foreign key they are supposed to reference is an `INTEGER` - which means Datasette foreign key features don't actually work
  - Those foreign key relationships aren't setup automatically - creating them is left as an exercise for the developer

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/138/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODQ5MjU5MDc=
title: Upgrade CodeMirror
state: closed
created_at: 2020-08-24T19:55:33Z
updated_at: 2020-09-30T20:36:12Z
closed_at: 2020-08-30T18:03:07Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Datasette currently bundles 5.31.0 (from October 2017) - latest version is 5.57.0 (August 2020). https://codemirror.net/doc/releases.html

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/948/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODQ5NjE0NDk=
title: Try out CodeMirror SQL hints
state: closed
created_at: 2020-08-24T20:58:21Z
updated_at: 2023-11-03T05:28:58Z
closed_at: 2020-11-01T03:29:48Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > It would also be interesting to try out the SQL hint mode, which can autocomplete against tables and columns. This demo shows how to configure that: https://codemirror.net/mode/sql/
  > 
  > Some missing documentation: https://stackoverflow.com/questions/20023381/codemirror-how-add-tables-to-sql-hint
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/948#issuecomment-679355426_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/949/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODU4MDY1MTE=
title: Private/secret databases: database files that are only visible to plugins
state: closed
created_at: 2020-08-25T20:46:17Z
updated_at: 2023-08-24T22:26:09Z
closed_at: 2023-08-24T22:26:08Z
author_association: OWNER
type: issue
state_reason: completed
body:
  In thinking about the best way to implement https://github.com/simonw/datasette-auth-passwords/issues/6 (SQL-backed user accounts for `datasette-auth-passwords`) I realized that there are a few different use-cases where a plugin might want to store data that isn't visible to regular Datasette users:

  - Storing password hashes
  - Storing API tokens
  - Storing secrets that are used for data import integrations (secrets for talking to the Twitter API for example)

  Idea: allow one or more private database files to be attached to Datasette, something like this:

      datasette github.db linkedin.db -s secrets.db -m metadata.yml

  The `secrets.db` file would not be visible using any of the Datasette's usual interface or API routes - but plugins would be able to run queries against it.

  So `datasette-auth-passwords` might then be configured like this:

  ```yaml
  plugins:
    datasette-auth-passwords:
      database: secrets
      sql: "select password_hash from passwords where username = :username"
  ```
  The plugin could even refuse to operate against a database that hadn't been loaded as a secret database.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/950/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODY5NzgxMzE=
title: insert_all(..., alter=True) should work for new columns introduced after the first 100 records
state: closed
created_at: 2020-08-27T06:25:25Z
updated_at: 2020-08-28T22:48:51Z
closed_at: 2020-08-28T22:30:14Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Is there a way to make `.insert_all()` work properly when new columns are introduced outside the first 100 records (with or without the `alter=True` argument)?

  I'm using `.insert_all()` to bulk insert ~3-4k records at a time and it is common for records to need to introduce new columns.  However, if new columns are introduced after the first 100 records, `sqlite_utils` doesn't even raise the `OperationalError: table ... has no column named ...` exception; it just silently drops the extra data and moves on.

  It took me a while to find this little snippet in the [documentation for `.insert_all()`](https://sqlite-utils.readthedocs.io/en/stable/python-api.html#bulk-inserts) (it's not mentioned under [Adding columns automatically on insert/update](https://sqlite-utils.readthedocs.io/en/stable/python-api.html#bulk-inserts)):

  > The column types used in the CREATE TABLE statement are automatically derived from the types of data in that first batch of rows. **_Any additional or missing columns in subsequent batches will be ignored._**

  I tried changing the `batch_size` argument to the total number of records, but it seems only to effect the number of rows that are committed at a time, and has no influence on this problem.

  Is there a way around this that you would suggest?  It seems like it should raise an exception at least.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDc0NzAzMDA3
title: Update black requirement from ~=19.10b0 to >=19.10,<21.0
state: closed
created_at: 2020-08-27T13:31:36Z
updated_at: 2020-09-02T22:26:17Z
closed_at: 2020-09-02T22:26:16Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/952
type: pull
body:
  Updates the requirements on [black](https://github.com/psf/black) to permit the latest version.
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/blob/master/CHANGES.md">black's changelog</a>.</em></p>
  <blockquote>
  <h3>20.8b1</h3>
  <h4><em>Packaging</em></h4>
  <ul>
  <li>explicitly depend on Click 7.1.2 or newer as <code>Black</code> no longer works with versions
  older than 7.0</li>
  </ul>
  <h3>20.8b0</h3>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>re-implemented support for explicit trailing commas: now it works consistently within
  any bracket pair, including nested structures (<a href="https://github-redirect.dependabot.com/psf/black/issues/1288">#1288</a> and duplicates)</p>
  </li>
  <li>
  <p><code>Black</code> now reindents docstrings when reindenting code around it (<a href="https://github-redirect.dependabot.com/psf/black/issues/1053">#1053</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now shows colored diffs (<a href="https://github-redirect.dependabot.com/psf/black/issues/1266">#1266</a>)</p>
  </li>
  <li>
  <p><code>Black</code> is now packaged using 'py3' tagged wheels (<a href="https://github-redirect.dependabot.com/psf/black/issues/1388">#1388</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now supports Python 3.8 code, e.g. star expressions in return statements
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/1121">#1121</a>)</p>
  </li>
  <li>
  <p><code>Black</code> no longer normalizes capital R-string prefixes as those have a
  community-accepted meaning (<a href="https://github-redirect.dependabot.com/psf/black/issues/1244">#1244</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now uses exit code 2 when specified configuration file doesn't exit (<a href="https://github-redirect.dependabot.com/psf/black/issues/1361">#1361</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now works on AWS Lambda (<a href="https://github-redirect.dependabot.com/psf/black/issues/1141">#1141</a>)</p>
  </li>
  <li>
  <p>added <code>--force-exclude</code> argument (<a href="https://github-redirect.dependabot.com/psf/black/issues/1032">#1032</a>)</p>
  </li>
  <li>
  <p>removed deprecated <code>--py36</code> option (<a href="https://github-redirect.dependabot.com/psf/black/issues/1236">#1236</a>)</p>
  </li>
  <li>
  <p>fixed <code>--diff</code> output when EOF is encountered (<a href="https://github-redirect.dependabot.com/psf/black/issues/526">#526</a>)</p>
  </li>
  <li>
  <p>fixed <code># fmt: off</code> handling around decorators (<a href="https://github-redirect.dependabot.com/psf/black/issues/560">#560</a>)</p>
  </li>
  <li>
  <p>fixed unstable formatting with some <code># type: ignore</code> comments (<a href="https://github-redirect.dependabot.com/psf/black/issues/1113">#1113</a>)</p>
  </li>
  <li>
  <p>fixed invalid removal on organizing brackets followed by indexing (<a href="https://github-redirect.dependabot.com/psf/black/issues/1575">#1575</a>)</p>
  </li>
  <li>
  <p>introduced <code>black-primer</code>, a CI tool that allows us to run regression tests against
  existing open source users of Black (<a href="https://github-redirect.dependabot.com/psf/black/issues/1402">#1402</a>)</p>
  </li>
  <li>
  <p>introduced property-based fuzzing to our test suite based on Hypothesis and
  Hypothersmith (<a href="https://github-redirect.dependabot.com/psf/black/issues/1566">#1566</a>)</p>
  </li>
  <li>
  <p>implemented experimental and disabled by default long string rewrapping (<a href="https://github-redirect.dependabot.com/psf/black/issues/1132">#1132</a>),
  hidden under a <code>--experimental-string-processing</code> flag while it's being worked on;</p>
  </li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li>See full diff in <a href="https://github.com/psf/black/commits">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/952/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODc2ODEwMTg=
title: register_output_renderer render function should be able to return a Response
state: closed
created_at: 2020-08-28T03:21:21Z
updated_at: 2020-08-28T04:53:03Z
closed_at: 2020-08-28T04:03:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  That plugin hook was designed before Datasette had a documented Response class. It should optionally be allowed to return a Response in addition to the current custom dictionary.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/953/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODc2OTQ5NDc=
title: Remove old register_output_renderer dict mechanism in Datasette 1.0
state: open
created_at: 2020-08-28T04:04:23Z
updated_at: 2020-08-28T04:56:31Z
author_association: OWNER
type: issue
body:
  > Documentation says that the old dictionary mechanism will be deprecated by 1.0:
  > 
  > https://github.com/simonw/datasette/blob/799ecae94824640bdff21f86997f69844048d5c3/docs/plugin_hooks.rst#L460
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/953#issuecomment-682312494_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/954/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODc3MTE3MTM=
title: Release updated datasette-atom and datasette-ics
state: closed
created_at: 2020-08-28T04:55:21Z
updated_at: 2020-09-14T22:19:46Z
closed_at: 2020-09-14T22:19:46Z
author_association: OWNER
body: These should release straight after Datasette 0.49 with the change from #953.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/955/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODgzNTEwNTQ=
title: Idea: insert-files mechanism for adding extra columns with fixed values
state: open
created_at: 2020-08-28T20:57:36Z
updated_at: 2022-03-20T19:45:45Z
author_association: OWNER
type: issue
body:
  Say for example you want to populate a `file_type` column with the value `gif`. That could work like this:

  ```
  sqlite-utils insert-files gifs.db images *.gif \
      -c path -c md5 -c last_modified:mtime \
      -c file_type:text:gif --pk=path
  ```
  So a column defined as a `text` column with a value that follows a second colon.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/140/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODgzNTIxNDU=
title: insert-files support for compressed values
state: open
created_at: 2020-08-28T20:59:46Z
updated_at: 2020-09-24T20:36:08Z
author_association: OWNER
type: issue
body:
  The `sqlar` format supports this, it would be useful if `insert-files` could support this too.

  https://www.sqlite.org/sqlar.html

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/141/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDc1NjY1OTg0
title: insert_all(..., alter=True) should work for new columns introduced after the first 100 records
state: closed
created_at: 2020-08-28T22:22:57Z
updated_at: 2020-08-30T07:28:23Z
closed_at: 2020-08-28T22:30:14Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/142
body: Closes #139.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/142/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODgzODk5MzM=
title: Move to GitHub Actions CI
state: closed
created_at: 2020-08-28T22:34:11Z
updated_at: 2020-08-28T22:41:35Z
closed_at: 2020-08-28T22:41:35Z
author_association: OWNER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/143/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODgzOTUyNzU=
title: Run some tests against numpy
state: closed
created_at: 2020-08-28T22:53:00Z
updated_at: 2020-08-28T22:57:05Z
closed_at: 2020-08-28T22:57:04Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Accidentally removed in #143:

  https://github.com/simonw/sqlite-utils/blob/d7d3f962861ef32c5ead8f514c8756f5b6f7c4a0/.travis.yml#L18-L19

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/144/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODg0Mjc3NTE=
title: Push to Docker Hub failed - but it shouldn't run for alpha releases anyway
state: closed
created_at: 2020-08-29T01:09:12Z
updated_at: 2020-09-15T20:46:41Z
closed_at: 2020-09-15T20:36:34Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://github.com/simonw/datasette/runs/1043709494?check_suite_focus=true

  <img width="740" alt="Release_notes_for_0_49a0_·_simonw_datasette_7178126" src="https://user-images.githubusercontent.com/9599/91625110-80c55a80-e959-11ea-8fea-70508c53fcfb.png">

  - [x] This step should not run if a release is an alpha or beta
  - [x] When it DOES run it should work
  - [x] See it work for both an alpha and a non-alpha release, then close this ticket

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/956/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODg2MjIxNDg=
title: Simplify imports of common classes
state: closed
created_at: 2020-08-29T23:44:04Z
updated_at: 2022-02-06T06:36:41Z
closed_at: 2022-02-06T06:34:37Z
author_association: OWNER
type: issue
state_reason: completed
body:
  There are only a few classes that plugins need to import. It would be nice if these imports were as short and memorable as possible.

  For example:
  ```python
  from datasette.app import Datasette
  from datasette.utils.asgi import Response
  ```
  Could both become:
  ```python
  from datasette import Datasette
  from datasette import Response
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/957/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODg2NTkxODI=
title: Bug when first record contains fewer columns than subsequent records
state: closed
created_at: 2020-08-30T05:44:44Z
updated_at: 2020-09-08T23:21:23Z
closed_at: 2020-09-08T23:21:23Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  `insert_all()` selects the maximum batch size based on the number of fields in the first record.  If the first record has fewer fields than subsequent records (and `alter=True` is passed), this can result in SQL statements with more than the maximum permitted number of host parameters.  This situation is perhaps unlikely to occur, but could happen if the first record had, say, 10 columns, such that `batch_size` (based on  `SQLITE_MAX_VARIABLE_NUMBER = 999`) would be 99.  If the next 98 rows had 11 columns, the resulting SQL statement for the first batch would have `10 * 1 + 11 * 98 = 1088` host parameters (and subsequent batches, if the data were consistent from thereon out, would have `99 * 11 = 1089`).

  I suspect that this bug is masked somewhat by the fact that while:
  > [`SQLITE_MAX_VARIABLE_NUMBER`](https://www.sqlite.org/limits.html#max_variable_number) ... defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0.

  it is common that it is increased at compile time.  Debian-based systems, for example, seem to ship with a version of sqlite compiled with `SQLITE_MAX_VARIABLE_NUMBER` set to 250,000, and I believe this is the case for homebrew installations too.

  A test for this issue might look like this:
  ```python
  def test_columns_not_in_first_record_should_not_cause_batch_to_be_too_large(fresh_db):
      # sqlite on homebrew and Debian/Ubuntu etc. is typically compiled with
      #  SQLITE_MAX_VARIABLE_NUMBER set to 250,000, so we need to exceed this value to
      #  trigger the error on these systems.
      THRESHOLD = 250000
      extra_columns = 1 + (THRESHOLD - 1) // 99
      records = [
          {"c0": "first record"},  # one column in first record -> batch_size = 100
          # fill out the batch with 99 records with enough columns to exceed THRESHOLD
          *[
              dict([("c{}".format(i), j) for i in range(extra_columns)])
              for j in range(99)
          ]
      ]
      try:
          fresh_db["too_many_columns"].insert_all(records, alter=True)
      except sqlite3.OperationalError:
          raise
  ```

  The best solution, I think, is simply to process all the records when determining columns, column types, and the batch size.  In my tests this doesn't seem to be particularly costly at all, and cuts out a lot of complications (including obviating my implementation of #139 at #142).  I'll raise a PR for your consideration.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/145/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDc1ODc0NDkz
title: Handle case where subsequent records (after first batch) include extra columns
state: closed
created_at: 2020-08-30T07:13:58Z
updated_at: 2020-09-08T23:20:37Z
closed_at: 2020-09-08T23:20:37Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/146
type: pull
body:
  Addresses #145.

  I think this should do the job.  If it meets with your approval I'll update this PR to include an update to the documentation -- I came across this bug while preparing a PR to update the documentation around `batch_size` in any event.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/146/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODg2NzAxNTg=
title: SQLITE_MAX_VARS maybe hard-coded too low
state: open
created_at: 2020-08-30T07:26:45Z
updated_at: 2021-02-15T21:27:55Z
author_association: CONTRIBUTOR
type: issue
body:
  I came across this while about to open an issue and PR against the documentation for `batch_size`, which is a bit incomplete.

  As mentioned in #145, while:

  > [`SQLITE_MAX_VARIABLE_NUMBER`](https://www.sqlite.org/limits.html#max_variable_number) ... defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0.

  it is common that it is increased at compile time. Debian-based systems, for example, seem to ship with a version of sqlite compiled with SQLITE_MAX_VARIABLE_NUMBER set to 250,000, and I believe this is the case for homebrew installations too.

  In working to understand what `batch_size` was actually doing and why, I realized that by setting `SQLITE_MAX_VARS` in `db.py` to match the value my sqlite was compiled with (I'm on Debian), I was able to decrease the time to `insert_all()` my test data set (~128k records across 7 tables) from ~26.5s to ~3.5s.  Given that this about .05% of my total dataset, this is time I am keen to save...

  Unfortunately, it seems that `sqlite3` in the python standard library doesn't expose the `get_limit()` C API (even though `pysqlite` used to), so it's hard to know what value sqlite has been compiled with (note that this could mean, I suppose, that it's less than 999, and even hardcoding `SQLITE_MAX_VARS` to the conservative default might not be adequate.  It can also be lowered -- but not raised -- at runtime).  The best I could come up with is `echo "" | sqlite3 -cmd ".limits variable_number"` (only available in `sqlite >= 2015-05-07 (3.8.10)`).

  Obviously this couldn't be relied upon in `sqlite_utils`, but I wonder what your opinion would be about exposing `SQLITE_MAX_VARS` as a user-configurable parameter (with suitable "here be dragons" warnings)?  I'm going to go ahead and monkey-patch it for my purposes in any event, but it seems like it might be worth considering.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/147/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODk4MDAzMDc=
title: Add an index on the timestamp column
state: closed
created_at: 2020-09-01T04:33:37Z
updated_at: 2020-09-01T04:49:23Z
closed_at: 2020-09-01T04:49:23Z
author_association: MEMBER
body: Since default view will likely be ordered by timestamp descending.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODk4MDkyMjU=
title: Apply porter stemming
state: closed
created_at: 2020-09-01T04:57:55Z
updated_at: 2020-09-01T20:42:00Z
closed_at: 2020-09-01T20:40:24Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  This can be on by default. You can turn it off for a table in the config file using `stemming: none` - or maybe `tokenize: none` to match the terminology used by SQLite and `sqlite-utils`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#enabling-full-text-search

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODk4MTAzNDA=
title: Datasette plugin to provide custom page for running faceted, ranked searches
state: closed
created_at: 2020-09-01T05:00:22Z
updated_at: 2020-09-03T21:01:41Z
closed_at: 2020-09-03T21:01:41Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  This will be a page at `/-/beta` which renders using a custom template.

  It will offer a default timeline view plus search and facet by type/date.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODk4MzkzOTk=
title: Optimize the FTS table
state: closed
created_at: 2020-09-01T05:58:17Z
updated_at: 2020-09-01T06:10:08Z
closed_at: 2020-09-01T06:10:08Z
author_association: MEMBER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODk4NDczNjE=
title: Add a context column that's not searchable
state: closed
created_at: 2020-09-01T06:13:42Z
updated_at: 2020-09-03T18:43:50Z
closed_at: 2020-09-03T18:43:50Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I sometimes like to configure titles that are things like "Comment on issue X" or "Photo in Golden Gate Park" - these shouldn't be included in the search index but should be stored so they can be displayed to provide context.

  Add a column for this - probably called `context` - and make it so it can be populated.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODk4NDg4Mjc=
title: ISO timestamps
state: open
created_at: 2020-09-01T06:16:42Z
updated_at: 2020-09-01T06:16:42Z
author_association: MEMBER
type: issue
body:
  The `time_added`, `time_updated` and `time_read` columns currently store data like this:

      September 19, 2019 - 00:30:30 UTC

  Should use ISO instead, e.g. `2020-07-26T01:05:24+00:00`

reactions:
  {"url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2ODk4NTA4MTA=
title: Set up a demo instance
state: open
created_at: 2020-09-01T06:20:24Z
updated_at: 2020-09-01T06:20:24Z
author_association: MEMBER
type: issue
body:
  Once I've got the Datasette plugin to a state where it's worth building a demo: #3

  I can use data from my public https://github-to-sqlite.dogsheep.net/ demo plus the Pocket data subset I use for the demo in https://github.com/dogsheep/pocket-to-sqlite/issues/5 - I could pull in the https://dogsheep-photos.dogsheep.net/ photos data too.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTEyNjUxOTg=
title: Mechanism for differentiating between "by me" and "liked by me"
state: closed
created_at: 2020-09-02T17:44:37Z
updated_at: 2020-09-02T21:06:28Z
closed_at: 2020-09-02T21:06:28Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Some of the content I'm indexing is by me - photos I've taken, tweets I wrote, commits, comments I posted.

  Some of it is stuff that I've "liked" or "bookmarked" in some way - favourited tweets, Pocket articles, starred GitHub repos.

  It woud be useful to be able to differentiate between the two.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTEzNjk2OTE=
title: Create a view for running faceted searches
state: closed
created_at: 2020-09-02T19:44:07Z
updated_at: 2020-09-02T19:50:47Z
closed_at: 2020-09-02T19:50:47Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  ```sql
  select
    search_index_fts.rank,
    search_index.rowid,
    search_index.[table],
    search_index.key,
    search_index.title,
    search_index.timestamp,
    search_index.search_1
  from
    search_index join search_index_fts on search_index.rowid = search_index_fts.rowid
  order by
    search_index_fts.rank, search_index.timestamp desc
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTE0NzU0MDA=
title: Upgrade to latest Black (20.8b1)
state: closed
created_at: 2020-09-02T22:24:19Z
updated_at: 2020-09-11T21:34:24Z
closed_at: 2020-09-02T22:25:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Black has some changes: https://black.readthedocs.io/en/stable/change_log.html#b0 - in particular:

  > - re-implemented support for explicit trailing commas: now it works consistently within any bracket pair, including nested structures (#1288 and duplicates)
  > - Black now reindents docstrings when reindenting code around it (#1053)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/958/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTE1MjE5NjU=
title: Mechanism for defining custom display of results
state: closed
created_at: 2020-09-03T00:14:07Z
updated_at: 2020-09-03T21:12:14Z
closed_at: 2020-09-03T21:09:55Z
author_association: MEMBER
body: Part of #3 - in particular I want to make sure my photos are displayed with a thumbnail.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTE1Mzc0MjY=
title: Internals API idea: results.dicts in addition to results.rows
state: open
created_at: 2020-09-03T00:50:17Z
updated_at: 2020-09-03T00:50:17Z
author_association: OWNER
type: issue
body:
  I just wrote this code:
  ```python
      results = await database.execute(SEARCH_SQL, {"query": query})
      return [dict(r) for r in results.rows]
  ```
  How about having `results.dicts` as a utility property that does that?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/959/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


