# issue_comments

html_url: https://github.com/simonw/datasette/issues/1#issuecomment-338523957
issue_url: https://api.github.com/repos/simonw/datasette/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDMzODUyMzk1Nw==
created_at: 2017-10-23T01:09:05Z
updated_at: 2017-10-24T02:42:12Z
author_association: OWNER
body:
  I also need to solve for weird primary keys. If it’s a single integer or a single char field that’s easy. But what if it is a compound key with more than one chat field? What delimiter can I use that will definitely be safe?

  Let’s say I use hyphen. Now I need to find a durable encoding for any hyphens that might exist in the key fields themselves.

  How about I use URLencoding for every non-alpha-numeric character? That will turn hyphens into (I think) %2D. It should also solve for unicode characters, but it means the vast majority of keys (integers) will display neatly, including a compound key of eg 5678-345

reactions:
  {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/1#issuecomment-338524454
issue_url: https://api.github.com/repos/simonw/datasette/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDMzODUyNDQ1NA==
created_at: 2017-10-23T01:15:24Z
updated_at: 2017-10-23T01:15:24Z
author_association: OWNER
body:
  Table rendering logic needs to detect the primary key field and turn it into a hyperlink. If there is a compound primary key it should add an extra column at the start of the table which displays the compound key as a link

reactions:
  {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/5#issuecomment-338524857
issue_url: https://api.github.com/repos/simonw/datasette/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDMzODUyNDg1Nw==
created_at: 2017-10-23T01:20:30Z
updated_at: 2017-10-23T01:20:30Z
author_association: OWNER
body:
  https://stackoverflow.com/a/14468878/6083

  Looks like I should order by compound primary key and implement cursor-based pagination.

reactions:
  {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/4#issuecomment-338530389
issue_url: https://api.github.com/repos/simonw/datasette/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDMzODUzMDM4OQ==
created_at: 2017-10-23T02:15:41Z
updated_at: 2017-10-23T02:15:41Z
author_association: OWNER
body:
  This means I need a good solution for these compile time options while running in development mode  
   

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/4#issuecomment-338530480
issue_url: https://api.github.com/repos/simonw/datasette/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDMzODUzMDQ4MA==
created_at: 2017-10-23T02:16:33Z
updated_at: 2017-10-23T02:16:33Z
author_association: OWNER
body:
   How about when the service starts up it checks for a compile.json file and, if it is missing, creates it using the same code we run at compile time normally 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/4#issuecomment-338531827
issue_url: https://api.github.com/repos/simonw/datasette/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDMzODUzMTgyNw==
created_at: 2017-10-23T02:28:31Z
updated_at: 2017-10-23T02:29:05Z
author_association: OWNER
body:
  Many of the applications I want to implement with this would benefit from having permanent real URLs.

  So let’s have both. The sha1 urls will serve far future cache headers (and an etag derived from their path). The non sha1 URLs will serve 302 uncached redirects to the sha1 locations.

  We will have a setting that lets people opt out of this behavior.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/8#issuecomment-338697223
issue_url: https://api.github.com/repos/simonw/datasette/issues/8
node_id: MDEyOklzc3VlQ29tbWVudDMzODY5NzIyMw==
created_at: 2017-10-23T15:28:11Z
updated_at: 2017-10-23T15:28:11Z
author_association: OWNER
body:
  Now returning this:

      {
          "error": "attempt to write a readonly database",
          "ok": false
      }

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/16#issuecomment-338768860
issue_url: https://api.github.com/repos/simonw/datasette/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDMzODc2ODg2MA==
created_at: 2017-10-23T19:23:29Z
updated_at: 2017-10-23T19:23:29Z
author_association: OWNER
body:
  I could use the table-reflow mechanism demonstrated here: http://demos.jquerymobile.com/1.4.3/table-reflow/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/20#issuecomment-338769538
issue_url: https://api.github.com/repos/simonw/datasette/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDMzODc2OTUzOA==
created_at: 2017-10-23T19:25:55Z
updated_at: 2017-10-23T19:25:55Z
author_association: OWNER
body:
  Maybe this should be handled by views instead?

  https://stateless-datasets-wreplxalgu.now.sh/ lists some views

  https://stateless-datasets-wreplxalgu.now.sh/?sql=select%20*%20from%20%22Order%20Subtotals%22 is an example showing the content of a view.

  What would the URL to views be? I don't think a view can share a name with a table, so the same URL scheme could work for both.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/4#issuecomment-338789734
issue_url: https://api.github.com/repos/simonw/datasette/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDMzODc4OTczNA==
created_at: 2017-10-23T20:40:25Z
updated_at: 2017-10-23T21:10:19Z
author_association: OWNER
body:
  URL design:

       /database/table.json - redirects to /database-6753f4a/table.json

  So we always redirect to the version with the truncated hash in the URL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/4#issuecomment-338797522
issue_url: https://api.github.com/repos/simonw/datasette/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDMzODc5NzUyMg==
created_at: 2017-10-23T21:09:33Z
updated_at: 2017-10-23T21:09:33Z
author_association: OWNER
body:
  https://stackoverflow.com/a/18134919/6083 is a good answer about how many characters of the hash are needed to be unique. I say we default to 7 characters, like git does - but allow extras to be configured.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/4#issuecomment-338799438
issue_url: https://api.github.com/repos/simonw/datasette/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDMzODc5OTQzOA==
created_at: 2017-10-23T21:17:25Z
updated_at: 2017-10-23T21:17:25Z
author_association: OWNER
body:
  Can I take  advantage of HTTP/2 so even if you get redirected I start serving you the correct resource straight away?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/4#issuecomment-338804173
issue_url: https://api.github.com/repos/simonw/datasette/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDMzODgwNDE3Mw==
created_at: 2017-10-23T21:36:37Z
updated_at: 2017-10-23T21:36:37Z
author_association: OWNER
body:
  Looks like the easiest way to implement HTTP/2 server push today is to run behind Cloudflare and use this:

      Link: </asset/to/push.js>; rel=preload; as=script

  https://blog.cloudflare.com/announcing-support-for-http-2-server-push-2/

  Here's the W3C draft: https://w3c.github.io/preload/

  From https://w3c.github.io/preload/#as-attribute it looks like I should use `as=fetch` if the content is intended for consumption by fetch() or XMLHTTPRequest.

  Unclear if I should throw `as=fetch crossorigin` in there. Need to experiment on that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/4#issuecomment-338806718
issue_url: https://api.github.com/repos/simonw/datasette/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDMzODgwNjcxOA==
created_at: 2017-10-23T21:47:53Z
updated_at: 2017-10-23T21:47:53Z
author_association: OWNER
body:
  Here's what the homepage of cloudflare.com does (with newlines added within the link header for clarity):

      $ curl -i 'https://www.cloudflare.com/' 
      HTTP/1.1 200 OK
      Date: Mon, 23 Oct 2017 21:45:58 GMT
      Content-Type: text/html; charset=utf-8
      Transfer-Encoding: chunked
      Connection: keep-alive
      link:
          </vendor/bitdashplayer.min.css>; rel=preload; as=style,
          </vendor/bitdash-controls.min.css>; rel=preload; as=style,
          </video/marketing-video/cloudflare-marketing-video.mpd>; rel=preload,
          </video/marketing-video/cloudflare-marketing-video.m3u8>; rel=preload,
          </video/marketing-video/video_0_800000/dash/init.mp4>; rel=preload; as=video,
          </video/marketing-video/audio_0_128000/dash/init.mp4>; rel=preload; as=video,
          </video/marketing-video/video_0_800000/dash/segment_0.m4s>; rel=preload; as=video,
          </video/marketing-video/audio_0_128000/dash/segment_0.m4s>; rel=preload; as=video,
          </video/new-short-optimized-22.mp4>; rel=preload; as=video,
          </video/marketing-video/poster.jpg>; rel=preload; as=image

  The original header looked like this:

      link: </vendor/bitdashplayer.min.css>; rel=preload; as=style, </vendor/bitdash-controls.min.css>; rel=preload; as=style, </video/marketing-video/cloudflare-marketing-video.mpd>; rel=preload, </video/marketing-video/cloudflare-marketing-video.m3u8>; rel=preload, </video/marketing-video/video_0_800000/dash/init.mp4>; rel=preload; as=video, </video/marketing-video/audio_0_128000/dash/init.mp4>; rel=preload; as=video, </video/marketing-video/video_0_800000/dash/segment_0.m4s>; rel=preload; as=video, </video/marketing-video/audio_0_128000/dash/segment_0.m4s>; rel=preload; as=video, </video/new-short-optimized-22.mp4>; rel=preload; as=video, </video/marketing-video/poster.jpg>; rel=preload; as=image

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/23#issuecomment-338854988
issue_url: https://api.github.com/repos/simonw/datasette/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDMzODg1NDk4OA==
created_at: 2017-10-24T02:40:12Z
updated_at: 2017-10-25T00:05:46Z
author_association: OWNER
body:
      /database-name/table-name?name__contains=simon&sort=id+desc

  Note that if there's a column called "sort" you can still do sort__exact=blah

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/23#issuecomment-338859620
issue_url: https://api.github.com/repos/simonw/datasette/issues/23
node_id: MDEyOklzc3VlQ29tbWVudDMzODg1OTYyMA==
created_at: 2017-10-24T03:11:42Z
updated_at: 2017-10-24T03:11:42Z
author_association: OWNER
body:
  I’m going to implement everything in https://docs.djangoproject.com/en/1.11/ref/models/querysets/#field-lookups with the exception of range and the various date ones.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1#issuecomment-338872286
issue_url: https://api.github.com/repos/simonw/datasette/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDMzODg3MjI4Ng==
created_at: 2017-10-24T04:46:06Z
updated_at: 2017-10-24T04:46:06Z
author_association: OWNER
body:
  I'm going to use `,` as the separator between elements of a compound primary key. If those elements themselves include a comma I will use `%2C` in its place.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/29#issuecomment-339019873
issue_url: https://api.github.com/repos/simonw/datasette/issues/29
node_id: MDEyOklzc3VlQ29tbWVudDMzOTAxOTg3Mw==
created_at: 2017-10-24T14:58:33Z
updated_at: 2017-10-24T14:58:33Z
author_association: OWNER
body:
  Here's what I've got now:

  <img width="693" alt="localhost_8006_northwind-40d049b_categories_json" src="https://user-images.githubusercontent.com/9599/31950748-1bc85f3e-b891-11e7-8205-d89da6129f4b.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/5#issuecomment-339027711
issue_url: https://api.github.com/repos/simonw/datasette/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDMzOTAyNzcxMQ==
created_at: 2017-10-24T15:21:30Z
updated_at: 2017-10-24T15:21:30Z
author_association: OWNER
body:
  I have code to detect primary keys on tables... but what should I do for tables that lack primary keys? How should I even sort them?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/5#issuecomment-339028979
issue_url: https://api.github.com/repos/simonw/datasette/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDMzOTAyODk3OQ==
created_at: 2017-10-24T15:25:08Z
updated_at: 2017-10-24T15:25:08Z
author_association: OWNER
body:
  Looks like I can use the SQLite specific “rowid” in that case. It isn’t guaranteed to stay consistent across a VACUUM but that’s ok because we are immutable anyway.

   https://www.sqlite.org/lang_createtable.html#rowid

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/19#issuecomment-339366612
issue_url: https://api.github.com/repos/simonw/datasette/issues/19
node_id: MDEyOklzc3VlQ29tbWVudDMzOTM2NjYxMg==
created_at: 2017-10-25T15:21:16Z
updated_at: 2017-10-25T15:21:16Z
author_association: OWNER
body:
  I had to manually set the content disposition header:

      return await response.file_stream(
          filepath, headers={
              'Content-Disposition': 'attachment; filename="{}"'.format(ilepath)
          }
      )

  In the next release of Sanic I can just use the filename= argument instead:

  https://github.com/channelcat/sanic/commit/07e95dba4f5983afc1e673df14bdd278817288aa

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/37#issuecomment-339382054
issue_url: https://api.github.com/repos/simonw/datasette/issues/37
node_id: MDEyOklzc3VlQ29tbWVudDMzOTM4MjA1NA==
created_at: 2017-10-25T16:05:56Z
updated_at: 2017-10-25T16:05:56Z
author_association: OWNER
body:
  Could this be as simple as using the iterative JSON encoder and adding a yield statement in between each chunk?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/38#issuecomment-339388215
issue_url: https://api.github.com/repos/simonw/datasette/issues/38
node_id: MDEyOklzc3VlQ29tbWVudDMzOTM4ODIxNQ==
created_at: 2017-10-25T16:25:45Z
updated_at: 2017-10-25T16:25:45Z
author_association: OWNER
body:
  First experiment: hook up an iterative CSV dump (just because that’s a tiny bit easier to get started with than iterative a JSON). Have it execute a big select statement and then iterate through the result set 100 rows at a time using sqite fetchmany() - also have it async sleep for a second in between each batch of 100.

  Can this work without needing python threads? 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/38#issuecomment-339388771
issue_url: https://api.github.com/repos/simonw/datasette/issues/38
node_id: MDEyOklzc3VlQ29tbWVudDMzOTM4ODc3MQ==
created_at: 2017-10-25T16:27:29Z
updated_at: 2017-10-25T16:27:29Z
author_association: OWNER
body:
  If this does work, I need to figure it what to do about the HTML view. ASsuming I can  iteratively produce JSON and CSV, what to do about HTML? One option: render the first 500 rows as HTML, then hand off to an infinite scroll experience that iteratively loads more rows as JSON.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/38#issuecomment-339389105
issue_url: https://api.github.com/repos/simonw/datasette/issues/38
node_id: MDEyOklzc3VlQ29tbWVudDMzOTM4OTEwNQ==
created_at: 2017-10-25T16:28:39Z
updated_at: 2017-10-25T16:28:39Z
author_association: OWNER
body:
  The gold standard here is to be able to serve up increasingly large datasets without blocking the event loop and while using a sustainable amount of RAM

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/38#issuecomment-339389328
issue_url: https://api.github.com/repos/simonw/datasette/issues/38
node_id: MDEyOklzc3VlQ29tbWVudDMzOTM4OTMyOA==
created_at: 2017-10-25T16:29:23Z
updated_at: 2017-10-25T16:29:23Z
author_association: OWNER
body:
  Ideally we can get some serious gains from the fact that our database file is opened with the immutable option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/40#issuecomment-339395551
issue_url: https://api.github.com/repos/simonw/datasette/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDMzOTM5NTU1MQ==
created_at: 2017-10-25T16:49:32Z
updated_at: 2017-10-25T16:49:32Z
author_association: OWNER
body:
  Simplest implementation will be to create a temporary directory somewhere, copy in a Dockerfile and the databases and run “now” in it.

  Ideally I can use symlinks rather than copying potentially large database files around.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/39#issuecomment-339406634
issue_url: https://api.github.com/repos/simonw/datasette/issues/39
node_id: MDEyOklzc3VlQ29tbWVudDMzOTQwNjYzNA==
created_at: 2017-10-25T17:27:10Z
updated_at: 2017-10-25T17:27:10Z
author_association: OWNER
body:
  It certainly looks like some of the stuff in https://sqlite.org/pragma.html could be used to screw around with things. Example: `PRAGMA case_sensitive_like = 1` - would that affect future queries?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/39#issuecomment-339413825
issue_url: https://api.github.com/repos/simonw/datasette/issues/39
node_id: MDEyOklzc3VlQ29tbWVudDMzOTQxMzgyNQ==
created_at: 2017-10-25T17:48:48Z
updated_at: 2017-10-25T17:48:48Z
author_association: OWNER
body:
  Could I use https://sqlparse.readthedocs.io/en/latest/ to parse incoming statements and ensure they are pure SELECTs? Would that prevent people from using a compound SELECT statement to trigger an evil PRAGMA of some sort?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/16#issuecomment-339420462
issue_url: https://api.github.com/repos/simonw/datasette/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDMzOTQyMDQ2Mg==
created_at: 2017-10-25T18:10:51Z
updated_at: 2017-10-25T18:10:51Z
author_association: OWNER
body:
  https://sitesforprofit.com/responsive-table-plugins-and-patterns has some useful links.

  I really like the pattern from https://css-tricks.com/responsive-data-tables/

      /* 
      Max width before this PARTICULAR table gets nasty
      This query will take effect for any screen smaller than 760px
      and also iPads specifically.
      */
      @media 
      only screen and (max-width: 760px),
      (min-device-width: 768px) and (max-device-width: 1024px)  {

          /* Force table to not be like tables anymore */
          table, thead, tbody, th, td, tr { 
              display: block; 
          }
          
          /* Hide table headers (but not display: none;, for accessibility) */
          thead tr { 
              position: absolute;
              top: -9999px;
              left: -9999px;
          }
          
          tr { border: 1px solid #ccc; }
          
          td { 
              /* Behave  like a "row" */
              border: none;
              border-bottom: 1px solid #eee; 
              position: relative;
              padding-left: 50%; 
          }
          
          td:before { 
              /* Now like a table header */
              position: absolute;
              /* Top/left values mimic padding */
              top: 6px;
              left: 6px;
              width: 45%; 
              padding-right: 10px; 
              white-space: nowrap;
          }
          
          /*
          Label the data
          */
          td:nth-of-type(1):before { content: "First Name"; }
          td:nth-of-type(2):before { content: "Last Name"; }
          td:nth-of-type(3):before { content: "Job Title"; }
          td:nth-of-type(4):before { content: "Favorite Color"; }
          td:nth-of-type(5):before { content: "Wars of Trek?"; }
          td:nth-of-type(6):before { content: "Porn Name"; }
          td:nth-of-type(7):before { content: "Date of Birth"; }
          td:nth-of-type(8):before { content: "Dream Vacation City"; }
          td:nth-of-type(9):before { content: "GPA"; }
          td:nth-of-type(10):before { content: "Arbitrary Data"; }
      }

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/39#issuecomment-339510770
issue_url: https://api.github.com/repos/simonw/datasette/issues/39
node_id: MDEyOklzc3VlQ29tbWVudDMzOTUxMDc3MA==
created_at: 2017-10-26T00:07:40Z
updated_at: 2017-10-26T00:07:40Z
author_association: OWNER
body:
  It looks like I should double quote my columns and ensure they are correctly escaped https://blog.christosoft.de/2012/10/sqlite-escaping-table-acolumn-names/ - hopefully using ? placeholders for column names will work. I should use ? for tables too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/40#issuecomment-339514819
issue_url: https://api.github.com/repos/simonw/datasette/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDMzOTUxNDgxOQ==
created_at: 2017-10-26T00:35:46Z
updated_at: 2017-10-26T00:35:46Z
author_association: OWNER
body:
  I’m going to have a single command-line app that does everything.

  Name to be decided - options include dataset, stateless, datasite (I quite like that - it reflects SQLite and the fact that you create a website)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/40#issuecomment-339515822
issue_url: https://api.github.com/repos/simonw/datasette/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDMzOTUxNTgyMg==
created_at: 2017-10-26T00:43:34Z
updated_at: 2017-10-26T00:43:34Z
author_association: OWNER
body:
  datasite .

  - starts web app in current directory, serving all DB files

  datasite . -p 8001

  - serves on custom port

  datasite blah.db blah2.db

  - serves specified files

  You can’t specify more than one directory. You can specify as many files as you like. If you specify two files with different oaths but the same name then they must be accessed by hash.

  datasite publish .

  - publishes current directory to the internet! Uses now by default, if it detects it on your path. Other publishers will be eventually added as plugins.

  datasite publish http://path-to-db.db

  - publishes a DB available at a URL. Works by constructing the Dockerfile with wget  calls in it.

  datasite blah.db -m metadata.json

  If you specify a directory it looks for metadata.json in that directory. Otherwise you can pass an explicit metadata file oath with -m or —metadata

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/40#issuecomment-339517846
issue_url: https://api.github.com/repos/simonw/datasette/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDMzOTUxNzg0Ng==
created_at: 2017-10-26T00:58:39Z
updated_at: 2017-10-26T00:58:39Z
author_association: OWNER
body:
  I’m going to use Click for this

  http://nvie.com/posts/writing-a-cli-in-python-in-under-60-seconds/
  https://kushaldas.in/posts/building-command-line-tools-in-python-with-click.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/40#issuecomment-339724700
issue_url: https://api.github.com/repos/simonw/datasette/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDMzOTcyNDcwMA==
created_at: 2017-10-26T16:35:20Z
updated_at: 2017-10-26T16:35:20Z
author_association: OWNER
body:
  Here’s how to make the “serve” subcommand the default if it is called with no arguments:

      @click.group(invoke_without_command=True)
      def serve():
          # ...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/41#issuecomment-339866724
issue_url: https://api.github.com/repos/simonw/datasette/issues/41
node_id: MDEyOklzc3VlQ29tbWVudDMzOTg2NjcyNA==
created_at: 2017-10-27T04:04:52Z
updated_at: 2017-10-27T04:04:52Z
author_association: OWNER
body:
  <img width="624" alt="databases" src="https://user-images.githubusercontent.com/9599/32087550-4e10b8a4-ba91-11e7-992b-e628d77114d0.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/40#issuecomment-339891755
issue_url: https://api.github.com/repos/simonw/datasette/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDMzOTg5MTc1NQ==
created_at: 2017-10-27T07:10:53Z
updated_at: 2017-10-27T07:10:53Z
author_association: OWNER
body:
  Deploys to Now aren't working at the moment - they aren't showing the uploaded databases, because I've broken the path handling somehow. I need to do a bit more work here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/40#issuecomment-340561577
issue_url: https://api.github.com/repos/simonw/datasette/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDM0MDU2MTU3Nw==
created_at: 2017-10-30T19:43:40Z
updated_at: 2017-10-30T19:43:40Z
author_association: OWNER
body:
  http://the-hitchhikers-guide-to-packaging.readthedocs.io/en/latest/quickstart.html describes how to package this for PyPI

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/39#issuecomment-340787868
issue_url: https://api.github.com/repos/simonw/datasette/issues/39
node_id: MDEyOklzc3VlQ29tbWVudDM0MDc4Nzg2OA==
created_at: 2017-10-31T14:54:14Z
updated_at: 2017-10-31T14:54:14Z
author_association: OWNER
body:
  Here’s how I can (I think) provide safe execution of arbitrary SQL while blocking PRAGMA calls: let people use names parameters in their SQL and apply strict filtering to the SQL query but not to the parameter values.

      cur.execute(
          "select * from people where name_last=:who and age=:age", {
              "who": who,
              "age": age
      })

  In URL form:

      ?sql=select...&who=Terry&age=34

  Now we can apply strict, dumb validation rules to the SQL part while allowing anything in the named queries - so people can execute a search for PRAGMA without being able to execute a PRAGMA statement.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/40#issuecomment-341945420
issue_url: https://api.github.com/repos/simonw/datasette/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDM0MTk0NTQyMA==
created_at: 2017-11-05T02:55:07Z
updated_at: 2017-11-05T02:55:07Z
author_association: OWNER
body:
  To simplify things a bit, I'm going to require that every database is explicitly listed in the command line. I won't support "serve everything in this directory" for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/40#issuecomment-342030075
issue_url: https://api.github.com/repos/simonw/datasette/issues/40
node_id: MDEyOklzc3VlQ29tbWVudDM0MjAzMDA3NQ==
created_at: 2017-11-06T02:25:48Z
updated_at: 2017-11-06T02:25:48Z
author_association: OWNER
body:
  ... I tried that, I don't like it. I'm going to bring back "directory serving" by allowing you to pass a directory as an argument to `datasite` (including `datasite .`). I may even make `.` the default if you don't provide anything at all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/16#issuecomment-342032943
issue_url: https://api.github.com/repos/simonw/datasette/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDM0MjAzMjk0Mw==
created_at: 2017-11-06T02:50:07Z
updated_at: 2017-11-06T02:50:07Z
author_association: OWNER
body:
  Default look with Bootstrap 4 looks like this:
  <img width="871" alt="flights" src="https://user-images.githubusercontent.com/9599/32423242-1b4458ce-c25a-11e7-910f-2dc1de909b8f.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/48#issuecomment-343168796
issue_url: https://api.github.com/repos/simonw/datasette/issues/48
node_id: MDEyOklzc3VlQ29tbWVudDM0MzE2ODc5Ng==
created_at: 2017-11-09T14:22:21Z
updated_at: 2017-11-09T14:22:21Z
author_association: OWNER
body:
  Won't fix: ujson is not compatible with the custom JSON encoder I'm using here: https://github.com/simonw/immutabase/blob/b2dee11fcd989d9e2a7bf4de1e23dbc320c05013/immutabase/app.py#L401-L416

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/49#issuecomment-343237982
issue_url: https://api.github.com/repos/simonw/datasette/issues/49
node_id: MDEyOklzc3VlQ29tbWVudDM0MzIzNzk4Mg==
created_at: 2017-11-09T17:58:01Z
updated_at: 2017-11-09T17:58:01Z
author_association: OWNER
body:
  More terms:

  * publish
  * share
  * docker
  * host
  * stateless

  I want to capture the idea of publishing an immutable database in a stateless container.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/49#issuecomment-343238262
issue_url: https://api.github.com/repos/simonw/datasette/issues/49
node_id: MDEyOklzc3VlQ29tbWVudDM0MzIzODI2Mg==
created_at: 2017-11-09T17:58:59Z
updated_at: 2017-11-09T17:58:59Z
author_association: OWNER
body:
  The name should ideally be available on PyPI and should make sense as both a command line application and a library.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/52#issuecomment-343557070
issue_url: https://api.github.com/repos/simonw/datasette/issues/52
node_id: MDEyOklzc3VlQ29tbWVudDM0MzU1NzA3MA==
created_at: 2017-11-10T18:57:47Z
updated_at: 2017-11-10T18:57:47Z
author_association: OWNER
body:
  https://file.io/ looks like it could be good for this. It's been around since 2015, and lets you upload a temporary file which can be downloaded once.

      $ curl -s -F "file=@database.db" "https://file.io/?expires=1d"
      {"success":true,"key":"ySrl1j","link":"https://file.io/ySrl1j","expiry":"1 day"}

  Downloading from that URL serves up the data with a `Content-disposition` header containing the filename:

      simonw$ curl -vv https://file.io/ySrl1j | more
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
        0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 34.232.1.167...
      * Connected to file.io (34.232.1.167) port 443 (#0)
      * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      * Server certificate: file.io
      * Server certificate: Amazon
      * Server certificate: Amazon Root CA 1
      * Server certificate: Starfield Services Root Certificate Authority - G2
      > GET /ySrl1j HTTP/1.1
      > Host: file.io
      > User-Agent: curl/7.43.0
      > Accept: */*
      > 
      < HTTP/1.1 200 OK
      < Date: Fri, 10 Nov 2017 18:14:38 GMT
      < Content-Type: undefined
      < Transfer-Encoding: chunked
      < Connection: keep-alive
      < X-Powered-By: Express
      < X-RateLimit-Limit: 5
      < X-RateLimit-Remaining: 4
      < Access-Control-Allow-Origin: *
      < Access-Control-Allow-Headers: Cache-Control,X-reqed-With,x-requested-with
      < Content-disposition: attachment; filename=database.db
      ...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/20#issuecomment-343581130
issue_url: https://api.github.com/repos/simonw/datasette/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDM0MzU4MTEzMA==
created_at: 2017-11-10T20:44:38Z
updated_at: 2017-11-10T20:44:38Z
author_association: OWNER
body:
  I'm going to handle this a different way. I'm going to support a local history of your own queries stored in localStorage, but if you want to share a query you have to do it with a URL.

  If people really want canned query support, they can do that using custom templates - see #12 - or by adding views to their database before they publish it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/16#issuecomment-343643332
issue_url: https://api.github.com/repos/simonw/datasette/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY0MzMzMg==
created_at: 2017-11-11T06:00:04Z
updated_at: 2017-11-11T06:00:04Z
author_association: OWNER
body:
  Here's what a table looks like now at a smaller screen size:

  <img width="496" alt="parlgov-development__info_data_source" src="https://user-images.githubusercontent.com/9599/32686818-7ffa691e-c662-11e7-964d-201fce0694e4.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/54#issuecomment-343644891
issue_url: https://api.github.com/repos/simonw/datasette/issues/54
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY0NDg5MQ==
created_at: 2017-11-11T06:39:54Z
updated_at: 2017-11-11T06:39:54Z
author_association: OWNER
body:
  I can detect something is a view like this:

      SELECT name from sqlite_master WHERE type ='view';

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/26#issuecomment-343644976
issue_url: https://api.github.com/repos/simonw/datasette/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY0NDk3Ng==
created_at: 2017-11-11T06:42:23Z
updated_at: 2017-11-11T06:42:23Z
author_association: OWNER
body:
  Simplest version of this:

  1. Create a temporary directory
  2. Write a Dockerfile into it that pulls an image and pip installs datasette
  3. Add symlinks to the DBs they listed (so we don't have to copy them)
  4. Shell out to "now"
  5. Done!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/26#issuecomment-343645249
issue_url: https://api.github.com/repos/simonw/datasette/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY0NTI0OQ==
created_at: 2017-11-11T06:48:59Z
updated_at: 2017-11-11T06:48:59Z
author_association: OWNER
body:
  Doing this works:

      import os
      os.link('/tmp/databases/northwind.db', '/tmp/tmp-blah/northwind.db')

  That creates a link in tmp-blah - and then when I delete that entire directory like so:

      import shutil
      shutil.rmtree('/tmp/tmp-blah')

  The original database is not deleted, just the link.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/26#issuecomment-343645327
issue_url: https://api.github.com/repos/simonw/datasette/issues/26
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY0NTMyNw==
created_at: 2017-11-11T06:51:16Z
updated_at: 2017-11-11T06:51:16Z
author_association: OWNER
body:
  I can create the temporary directory like so:

      import tempfile
      t = tempfile.TemporaryDirectory()
      t
      <TemporaryDirectory '/var/folders/w9/0xm39tk94ng9h52g06z4b54c0000gp/T/tmpkym70wlp'>
      t.name
      '/var/folders/w9/0xm39tk94ng9h52g06z4b54c0000gp/T/tmpkym70wlp'

  And then to delete it all:

      t.cleanup()

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/47#issuecomment-343647102
issue_url: https://api.github.com/repos/simonw/datasette/issues/47
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY0NzEwMg==
created_at: 2017-11-11T07:36:00Z
updated_at: 2017-11-11T07:36:00Z
author_association: OWNER
body:
  http://2016.padjo.org/tutorials/data-primer-census-acs1-demographics/ has a sqlite database: http://2016.padjo.org/files/data/starterpack/census-acs-1year/acs-1-year-2015.sqlite

  I tested this by deploying it here: https://datasette-fewuggrvwr.now.sh/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/16#issuecomment-343647300
issue_url: https://api.github.com/repos/simonw/datasette/issues/16
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY0NzMwMA==
created_at: 2017-11-11T07:41:19Z
updated_at: 2017-11-11T07:53:09Z
author_association: OWNER
body:
  Still needed:

  - [ ] A link to the homepage from some kind of navigation bar in the header
  - [ ] link to github.com/simonw/datasette in the footer
  - [ ] Slightly better titles (maybe ditch the visited link colours for titles only? should keep those for primary key links)
  - [ ] Links to the .json and .jsono versions of every view

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-343675165
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY3NTE2NQ==
created_at: 2017-11-11T16:07:10Z
updated_at: 2017-11-11T16:07:10Z
author_association: OWNER
body:
  The plugin system can also allow alternative providers for the `publish` command - e.g. maybe hook up hyper.sh as an option for publishing containers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/60#issuecomment-343683566
issue_url: https://api.github.com/repos/simonw/datasette/issues/60
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY4MzU2Ng==
created_at: 2017-11-11T18:12:24Z
updated_at: 2017-11-11T18:12:24Z
author_association: OWNER
body:
  I’m going to solve this by making it an optional argument you can pass to the serve command. Then the Dockerfile can still build and use it but it won’t interfere with tests or dev.

  If argument is not passed, we will calculate hashes on startup and calculate table row counts on demand.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/47#issuecomment-343690060
issue_url: https://api.github.com/repos/simonw/datasette/issues/47
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY5MDA2MA==
created_at: 2017-11-11T19:56:08Z
updated_at: 2017-11-11T19:56:08Z
author_association: OWNER
body:
      "parlgov-development.db": {
          "url": "http://www.parlgov.org/"
      },
      "nhsadmin.sqlite": {
          "url": "https://github.com/psychemedia/openHealthDataDoodles"
      }

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/63#issuecomment-343697291
issue_url: https://api.github.com/repos/simonw/datasette/issues/63
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY5NzI5MQ==
created_at: 2017-11-11T22:05:06Z
updated_at: 2017-11-11T22:11:49Z
author_association: OWNER
body:
  I'm going to bundle sql and sql_params together into a query nested object like this:

      {
          "query": {
              "sql": "select ...",
              "params": {
                  "p0": "blah"
              }
          }
      }

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/50#issuecomment-343698214
issue_url: https://api.github.com/repos/simonw/datasette/issues/50
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY5ODIxNA==
created_at: 2017-11-11T22:23:21Z
updated_at: 2017-11-11T22:23:21Z
author_association: OWNER
body:
  I'm closing #50 - more tests will be added in the future, but the framework is neatly in place for them now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/53#issuecomment-343699115
issue_url: https://api.github.com/repos/simonw/datasette/issues/53
node_id: MDEyOklzc3VlQ29tbWVudDM0MzY5OTExNQ==
created_at: 2017-11-11T22:41:38Z
updated_at: 2017-11-11T22:41:38Z
author_association: OWNER
body:
  This needs to incorporate a sensible way of presenting custom SQL query results too. And let's get a textarea in there for executing SQL while we're at it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/53#issuecomment-343707676
issue_url: https://api.github.com/repos/simonw/datasette/issues/53
node_id: MDEyOklzc3VlQ29tbWVudDM0MzcwNzY3Ng==
created_at: 2017-11-12T01:49:07Z
updated_at: 2017-11-12T01:49:07Z
author_association: OWNER
body:
  Here's the new design:

  <img width="691" alt="parlgov-development" src="https://user-images.githubusercontent.com/9599/32695161-82821226-c708-11e7-835c-b3d91850b2e0.png">

  Also lists views at the bottom (refs #54):

  <img width="345" alt="parlgov-development" src="https://user-images.githubusercontent.com/9599/32695164-99efa7de-c708-11e7-8272-bc5f5b870b84.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/65#issuecomment-343709217
issue_url: https://api.github.com/repos/simonw/datasette/issues/65
node_id: MDEyOklzc3VlQ29tbWVudDM0MzcwOTIxNw==
created_at: 2017-11-12T02:36:37Z
updated_at: 2017-11-12T02:36:37Z
author_association: OWNER
body:
  <img width="982" alt="nhsadmin" src="https://user-images.githubusercontent.com/9599/32695392-3ea12612-c70f-11e7-873b-9e6ad2c869e7.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/25#issuecomment-343715915
issue_url: https://api.github.com/repos/simonw/datasette/issues/25
node_id: MDEyOklzc3VlQ29tbWVudDM0MzcxNTkxNQ==
created_at: 2017-11-12T06:08:28Z
updated_at: 2017-11-12T06:08:28Z
author_association: OWNER
body:
      con = sqlite3.connect('existing_db.db')
      with open('dump.sql', 'w') as f:
          for line in con.iterdump():
              f.write('%s\n' % line)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/69#issuecomment-343752579
issue_url: https://api.github.com/repos/simonw/datasette/issues/69
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc1MjU3OQ==
created_at: 2017-11-12T17:22:39Z
updated_at: 2017-11-12T17:22:39Z
author_association: OWNER
body:
  By default I'll allow LIMIT and OFFSET up to a maximum of X (where X is let's say 50,000 to start with, but can be custom configured to a larger number or set to None for no limit).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/66#issuecomment-343752683
issue_url: https://api.github.com/repos/simonw/datasette/issues/66
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc1MjY4Mw==
created_at: 2017-11-12T17:24:05Z
updated_at: 2017-11-12T17:24:21Z
author_association: OWNER
body:
  Maybe SQL views should have their own Sanic view class (`ViewView` is kinda funny), subclassed from `TableView`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/68#issuecomment-343753999
issue_url: https://api.github.com/repos/simonw/datasette/issues/68
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc1Mzk5OQ==
created_at: 2017-11-12T17:45:21Z
updated_at: 2017-11-12T19:38:33Z
author_association: OWNER
body:
  For initial launch, I could just support this as some optional command line arguments you pass to the publish command:

      datasette publish data.db --title="Title" --source="url"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/68#issuecomment-343754058
issue_url: https://api.github.com/repos/simonw/datasette/issues/68
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc1NDA1OA==
created_at: 2017-11-12T17:46:13Z
updated_at: 2017-11-12T17:46:13Z
author_association: OWNER
body:
  I’m going to store this stuff in a file called metadata.json and move the existing automatically generated metadata to a file called build.json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/57#issuecomment-343769692
issue_url: https://api.github.com/repos/simonw/datasette/issues/57
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc2OTY5Mg==
created_at: 2017-11-12T21:32:36Z
updated_at: 2017-11-12T21:32:36Z
author_association: OWNER
body:
  I have created a Docker Hub public repository for this: https://hub.docker.com/r/simonwillison/datasette/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/69#issuecomment-343780039
issue_url: https://api.github.com/repos/simonw/datasette/issues/69
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc4MDAzOQ==
created_at: 2017-11-13T00:05:27Z
updated_at: 2017-11-13T00:05:27Z
author_association: OWNER
body:
  I think the only safe way to do this is using SQLite `.fetchmany(1000)` - I can't guarantee that the user has not entered SQL that will outfox a limit in some way. So instead of attempting to edit their SQL, I'll always return 1001 records and let them know if they went over 1000 or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/71#issuecomment-343780141
issue_url: https://api.github.com/repos/simonw/datasette/issues/71
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc4MDE0MQ==
created_at: 2017-11-13T00:06:52Z
updated_at: 2017-11-13T00:06:52Z
author_association: OWNER
body:
  I've registered datasettes.com as a domain name for doing this. Now setting it up so Cloudflare and Now can serve content from it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/71#issuecomment-343780539
issue_url: https://api.github.com/repos/simonw/datasette/issues/71
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc4MDUzOQ==
created_at: 2017-11-13T00:13:29Z
updated_at: 2017-11-13T00:19:46Z
author_association: OWNER
body:
  https://zeit.co/docs/features/dns is docs

      now domain add -e datasettes.com

  I had to set up a custom TXT record on `_now.datasettes.com` to get this to work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/71#issuecomment-343780671
issue_url: https://api.github.com/repos/simonw/datasette/issues/71
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc4MDY3MQ==
created_at: 2017-11-13T00:15:21Z
updated_at: 2017-11-13T00:17:37Z
author_association: OWNER
body:
  - [x] Redirect https://datasettes.com/ and https://www.datasettes.com/ to https://github.com/simonw/datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/71#issuecomment-343780814
issue_url: https://api.github.com/repos/simonw/datasette/issues/71
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc4MDgxNA==
created_at: 2017-11-13T00:17:50Z
updated_at: 2017-11-13T00:18:19Z
author_association: OWNER
body:
  Achieved those redirects using Cloudflare "page rules": https://www.cloudflare.com/a/page-rules/datasettes.com

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/71#issuecomment-343781030
issue_url: https://api.github.com/repos/simonw/datasette/issues/71
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc4MTAzMA==
created_at: 2017-11-13T00:21:05Z
updated_at: 2017-11-13T02:09:32Z
author_association: OWNER
body:
  - [x] Have `now domain add -e datasettes.com` run without errors (hopefully just a matter of waiting for the DNS to update)
  - [x] Alias an example dataset hosted on Now on a datasettes.com subdomain
  - [x] Confirm that HTTP caching and HTTP/2 redirect pushing works as expected - this may require another page rule

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/71#issuecomment-343788581
issue_url: https://api.github.com/repos/simonw/datasette/issues/71
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc4ODU4MQ==
created_at: 2017-11-13T01:48:17Z
updated_at: 2017-11-13T01:48:17Z
author_association: OWNER
body:
  I had to add a rule like this to get letsencrypt certificates on now.sh working: https://github.com/zeit/now-cli/issues/188#issuecomment-270105052

  <img width="975" alt="page_rules__datasettes_com___cloudflare_-_web_performance___security" src="https://user-images.githubusercontent.com/9599/32706131-c3d88742-c7cf-11e7-8d39-07d3554ce3cf.png">

  I also have to flip this switch off every time I want to add a new alias:

  <img width="993" alt="crypto__datasettes_com___cloudflare_-_web_performance___security" src="https://user-images.githubusercontent.com/9599/32706326-a8ba1320-c7d1-11e7-8846-eb1e62efa3ed.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/71#issuecomment-343788780
issue_url: https://api.github.com/repos/simonw/datasette/issues/71
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc4ODc4MA==
created_at: 2017-11-13T01:50:01Z
updated_at: 2017-11-13T01:50:01Z
author_association: OWNER
body:
  Added another page rule in order to get Cloudflare to always obey cache headers sent by the server:

  <img width="978" alt="page_rules__datasettes_com___cloudflare_-_web_performance___security" src="https://user-images.githubusercontent.com/9599/32706355-ded7c60a-c7d1-11e7-93da-20989f40d527.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/71#issuecomment-343789162
issue_url: https://api.github.com/repos/simonw/datasette/issues/71
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc4OTE2Mg==
created_at: 2017-11-13T01:53:29Z
updated_at: 2017-11-13T01:53:29Z
author_association: OWNER
body:
  ```
  $ curl -i 'https://fivethirtyeight.datasettes.com/fivethirtyeight-75d605c/obama-commutations%2Fobama_commutations.csv.jsono'
  HTTP/1.1 200 OK
  Date: Mon, 13 Nov 2017 01:50:57 GMT
  Content-Type: application/json
  Transfer-Encoding: chunked
  Connection: keep-alive
  Set-Cookie: __cfduid=de836090f3e12a60579cc7a1696cf0d9e1510537857; expires=Tue, 13-Nov-18 01:50:57 GMT; path=/; domain=.datasettes.com; HttpOnly; Secure
  Access-Control-Allow-Origin: *
  Cache-Control: public, max-age=31536000
  X-Now-Region: now-sfo
  CF-Cache-Status: HIT
  Expires: Tue, 13 Nov 2018 01:50:57 GMT
  Server: cloudflare-nginx
  CF-RAY: 3bce154a6d9293b4-SJC

  {"database": "fivethirtyeight", "table": "obama-commutations/obama_commutations.csv"...```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/71#issuecomment-343790984
issue_url: https://api.github.com/repos/simonw/datasette/issues/71
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzc5MDk4NA==
created_at: 2017-11-13T02:09:34Z
updated_at: 2017-11-13T02:09:34Z
author_association: OWNER
body:
  HTTP/2 push totally worked on the redirect!

      fetch('https://fivethirtyeight.datasettes.com/fivethirtyeight/riddler-pick-lowest%2Flow_numbers.csv.jsono').then(r => r.json()).then(console.log)

  <img width="735" alt="eventbrite_api___v3_destination_search_" src="https://user-images.githubusercontent.com/9599/32706785-6c8dc178-c7d4-11e7-9130-8b6ec84a7430.png">

  Meanwhile, in the network pane...

  <img width="770" alt="eventbrite_api___v3_destination_search_" src="https://user-images.githubusercontent.com/9599/32706803-7f92aac2-c7d4-11e7-9b01-c87e3d2c3891.png">

  <img width="771" alt="eventbrite_api___v3_destination_search_" src="https://user-images.githubusercontent.com/9599/32706821-95106bb4-c7d4-11e7-9ad5-31107faa5795.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/73#issuecomment-343801392
issue_url: https://api.github.com/repos/simonw/datasette/issues/73
node_id: MDEyOklzc3VlQ29tbWVudDM0MzgwMTM5Mg==
created_at: 2017-11-13T03:36:47Z
updated_at: 2017-11-13T03:36:47Z
author_association: OWNER
body:
  While I’m at it, let’s allow people to opt out of HTTP/2 push with a ?_nopush=1 argument too - in case they decide they don’t want to receive large 302 responses.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/67#issuecomment-343961784
issue_url: https://api.github.com/repos/simonw/datasette/issues/67
node_id: MDEyOklzc3VlQ29tbWVudDM0Mzk2MTc4NA==
created_at: 2017-11-13T15:50:50Z
updated_at: 2017-11-13T15:50:50Z
author_association: OWNER
body:
  `datasette package ...` - same arguments as `datasette publish`. Creates Docker container in your local repo, optionally tagged with `--tag`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/75#issuecomment-344000982
issue_url: https://api.github.com/repos/simonw/datasette/issues/75
node_id: MDEyOklzc3VlQ29tbWVudDM0NDAwMDk4Mg==
created_at: 2017-11-13T17:50:27Z
updated_at: 2017-11-13T17:50:27Z
author_association: OWNER
body:
  This is necessary because one of the fun things to do with this tool is run it locally, e.g.:

       datasette ~/Library/Application\ Support/Google/Chrome/Default/History -p 8003

  BUT... if we enable CORS by default, an evil site could try sniffing for localhost:8003 and attempt to steal data.

  So we'll enable the CORS headers only if `--cors` is provided to the command, and then use that command in the default Dockerfile.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/74#issuecomment-344018680
issue_url: https://api.github.com/repos/simonw/datasette/issues/74
node_id: MDEyOklzc3VlQ29tbWVudDM0NDAxODY4MA==
created_at: 2017-11-13T18:49:58Z
updated_at: 2017-11-13T18:49:58Z
author_association: OWNER
body:
  Turns out it does this already: https://github.com/simonw/datasette/blob/6b3b05b6db0d2a7b7cec8b8dbb4ddc5e12a376b2/datasette/app.py#L96-L107

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/69#issuecomment-344048656
issue_url: https://api.github.com/repos/simonw/datasette/issues/69
node_id: MDEyOklzc3VlQ29tbWVudDM0NDA0ODY1Ng==
created_at: 2017-11-13T20:32:47Z
updated_at: 2017-11-13T20:32:47Z
author_association: OWNER
body:
  <img width="908" alt="ak" src="https://user-images.githubusercontent.com/9599/32745247-071aa764-c867-11e7-9748-88e22f5eee57.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/55#issuecomment-344060070
issue_url: https://api.github.com/repos/simonw/datasette/issues/55
node_id: MDEyOklzc3VlQ29tbWVudDM0NDA2MDA3MA==
created_at: 2017-11-13T21:14:13Z
updated_at: 2017-11-13T21:14:13Z
author_association: OWNER
body:
  I'm going to add some extra metadata to setup.py and then tag this as version 0.8:

      git tag 0.8
      git push --tags

  Then to ship to PyPI:

      python setup.py bdist_wheel
      twine register dist/datasette-0.8-py3-none-any.whl
      twine upload dist/datasette-0.8-py3-none-any.whl

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/80#issuecomment-344074443
issue_url: https://api.github.com/repos/simonw/datasette/issues/80
node_id: MDEyOklzc3VlQ29tbWVudDM0NDA3NDQ0Mw==
created_at: 2017-11-13T22:04:54Z
updated_at: 2017-11-13T22:05:02Z
author_association: OWNER
body:
  The fivethirtyeight dataset:

      datasette publish now --name fivethirtyeight --metadata metadata.json fivethirtyeight.db
      now alias https://fivethirtyeight-jyqfudvjli.now.sh fivethirtyeight.datasettes.com

  And parlgov:

      datasette publish now parlgov.db --name=parlgov --metadata=parlgov.json 
      now alias https://parlgov-hqvxuhmbyh.now.sh parlgov.datasettes.com

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/80#issuecomment-344075696
issue_url: https://api.github.com/repos/simonw/datasette/issues/80
node_id: MDEyOklzc3VlQ29tbWVudDM0NDA3NTY5Ng==
created_at: 2017-11-13T22:09:46Z
updated_at: 2017-11-13T22:09:46Z
author_association: OWNER
body:
  Parlgov was throwing errors on one of the views, which takes longer than 1000ms to execute - so I added the ability to customize the time limit in https://github.com/simonw/datasette/commit/1e698787a4dd6df0432021a6814c446c8b69bba2

      datasette publish now parlgov.db --metadata parlgov.json --name parlgov --extra-options="--sql_time_limit_ms=3500"
      now alias https://parlgov-nvkcowlixq.now.sh parlgov.datasettes.com

  https://parlgov.datasettes.com/parlgov-25f9855/view_cabinet now returns in just over 2.5s

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/81#issuecomment-344076554
issue_url: https://api.github.com/repos/simonw/datasette/issues/81
node_id: MDEyOklzc3VlQ29tbWVudDM0NDA3NjU1NA==
created_at: 2017-11-13T22:12:57Z
updated_at: 2017-11-13T22:12:57Z
author_association: OWNER
body:
  Hah, I haven't even announced this yet :) Travis is upset because I'm using SQL in the tests which isn't compatible with their version of Python 3.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/59#issuecomment-344081876
issue_url: https://api.github.com/repos/simonw/datasette/issues/59
node_id: MDEyOklzc3VlQ29tbWVudDM0NDA4MTg3Ng==
created_at: 2017-11-13T22:33:43Z
updated_at: 2017-11-13T22:33:43Z
author_association: OWNER
body:
  The `datasette package` command introduced in 4143e3b45c16cbae5e3e3419ef479a71810e7df3 is relevant here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/81#issuecomment-344125441
issue_url: https://api.github.com/repos/simonw/datasette/issues/81
node_id: MDEyOklzc3VlQ29tbWVudDM0NDEyNTQ0MQ==
created_at: 2017-11-14T02:24:54Z
updated_at: 2017-11-14T02:24:54Z
author_association: CONTRIBUTOR
body:
  Oops, if I jumped the gun. I saw the project in my github activity feed and saw some low hanging fruit :) 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/47#issuecomment-344132481
issue_url: https://api.github.com/repos/simonw/datasette/issues/47
node_id: MDEyOklzc3VlQ29tbWVudDM0NDEzMjQ4MQ==
created_at: 2017-11-14T03:08:13Z
updated_at: 2017-11-14T03:08:13Z
author_association: OWNER
body:
  I ended up shipping with https://fivethirtyeight.datasettes.com/ and https://parlgov.datasettes.com/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/59#issuecomment-344141199
issue_url: https://api.github.com/repos/simonw/datasette/issues/59
node_id: MDEyOklzc3VlQ29tbWVudDM0NDE0MTE5OQ==
created_at: 2017-11-14T04:13:11Z
updated_at: 2017-11-14T04:13:11Z
author_association: OWNER
body:
  I managed to do this manually:

      datasette package ~/parlgov-db/parlgov.db --metadata=parlgov.json
      # Output 8758ec31dda3 as the new image ID
      docker save 8758ec31dda3 > /tmp/my-image
      # I could have just piped this straight to hyper
      cat /tmp/my-image | hyper load
      # Now start the container running in hyper
      hyper run -d -p 80:8001 --name parlgov 8758ec31dda3
      # We need to assign an IP address so we can see it
      hyper fip allocate 1
      # Outputs 199.245.58.78
      hyper fip attach 199.245.58.78 parlgov

  At this point, visiting the IP address in a browser showed the parlgov UI.

  To clean up...

      hyper hyper fip detach parlgov
      hyper fip release 199.245.58.78
      hyper stop parlgov
      hyper rm parlgov

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/57#issuecomment-344145265
issue_url: https://api.github.com/repos/simonw/datasette/issues/57
node_id: MDEyOklzc3VlQ29tbWVudDM0NDE0NTI2NQ==
created_at: 2017-11-14T04:45:38Z
updated_at: 2017-11-14T04:45:38Z
author_association: CONTRIBUTOR
body:
  I'm happy to contribute this. Just let me know if you want a Dockerfile for development or production purposes, or both. 

  If it's prod then we can just pip install the source from pypi, otherwise for dev we'll need a `requirements.txt` to speed up rebuilds.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/57#issuecomment-344147583
issue_url: https://api.github.com/repos/simonw/datasette/issues/57
node_id: MDEyOklzc3VlQ29tbWVudDM0NDE0NzU4Mw==
created_at: 2017-11-14T05:03:47Z
updated_at: 2017-11-14T05:03:47Z
author_association: CONTRIBUTOR
body:
  Let me know if you'd like a PR. The image is usable as 
  `docker run --rm -t -i -p 9000:8001 -v $(pwd)/db:/db datasette datasette serve /db/chinook.db`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/57#issuecomment-344149165
issue_url: https://api.github.com/repos/simonw/datasette/issues/57
node_id: MDEyOklzc3VlQ29tbWVudDM0NDE0OTE2NQ==
created_at: 2017-11-14T05:16:34Z
updated_at: 2017-11-14T05:17:14Z
author_association: OWNER
body:
  I’m intrigued by this pattern: 

  https://github.com/macropin/datasette/blob/147195c2fdfa2b984d8f9fc1c6cab6634970a056/Dockerfile#L8

  What’s the benefit of doing that? Does it result in a smaller image size?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/57#issuecomment-344151223
issue_url: https://api.github.com/repos/simonw/datasette/issues/57
node_id: MDEyOklzc3VlQ29tbWVudDM0NDE1MTIyMw==
created_at: 2017-11-14T05:32:28Z
updated_at: 2017-11-14T05:33:03Z
author_association: CONTRIBUTOR
body:
  The pattern is called "multi-stage builds". And the result is a svelte 226MB image (201MB for 3.6-slim) vs 700MB+ for the full image. It's possible to get it even smaller, but that takes a lot more work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/46#issuecomment-344161226
issue_url: https://api.github.com/repos/simonw/datasette/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDM0NDE2MTIyNg==
created_at: 2017-11-14T06:41:21Z
updated_at: 2017-11-14T06:41:21Z
author_association: OWNER
body:
  Spatial extensions would be really useful too. https://www.gaia-gis.it/spatialite-2.1/SpatiaLite-manual.html

reactions:
  {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/43#issuecomment-344180866
issue_url: https://api.github.com/repos/simonw/datasette/issues/43
node_id: MDEyOklzc3VlQ29tbWVudDM0NDE4MDg2Ng==
created_at: 2017-11-14T08:25:37Z
updated_at: 2017-11-14T08:25:37Z
author_association: OWNER
body:
  This isn’t necessary - restarting the server is fast and easy, and I’ve not found myself needing this at all during development.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-344409906
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQwOTkwNg==
created_at: 2017-11-14T21:47:02Z
updated_at: 2017-11-14T21:47:02Z
author_association: OWNER
body:
  Even without bundling in the database file itself, I'd love to have a standalone binary version of the core `datasette` CLI utility.

  I think Sanic may have some complex dependencies, but I've never tried pyinstaller so I don't know how easy or hard it would be to get this working.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-344415756
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQxNTc1Ng==
created_at: 2017-11-14T22:09:13Z
updated_at: 2017-11-14T22:09:13Z
author_association: OWNER
body:
  Looks like we'd need to use this recipe: https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Setuptools-Entry-Point

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-344424382
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQyNDM4Mg==
created_at: 2017-11-14T22:42:16Z
updated_at: 2017-11-14T22:42:16Z
author_association: NONE
body:
  tried quickly, this seems working:

  ```
  ~ pip3 install pyinstaller
  ~ pyinstaller -F --add-data /usr/local/lib/python3.6/site-packages/datasette/templates:datasette/templates --add-data /usr/local/lib/python3.6/site-packages/datasette/static:datasette/static  /usr/local/bin/datasette

  ~ du -h dist/datasette
  6.8M	dist/datasette
  ~ file dist/datasette
  dist/datasette: Mach-O 64-bit executable x86_64
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-344426887
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQyNjg4Nw==
created_at: 2017-11-14T22:51:46Z
updated_at: 2017-11-14T22:51:46Z
author_association: OWNER
body:
  That didn't quite work for me. It built me a `dist/datasette` executable but when I try to run it I get an error:

      $ pwd
      /Users/simonw/Dropbox/Development/datasette
      $ source venv/bin/activate
      $ pyinstaller -F --add-data datasette/templates:datasette/templates --add-data datasette/static:datasette/static /Users/simonw/Dropbox/Development/datasette/venv/bin/datasette
      $ dist/datasette --help
      Traceback (most recent call last):
        File "datasette", line 11, in <module>
        File "site-packages/pkg_resources/__init__.py", line 572, in load_entry_point
        File "site-packages/pkg_resources/__init__.py", line 564, in get_distribution
        File "site-packages/pkg_resources/__init__.py", line 436, in get_provider
        File "site-packages/pkg_resources/__init__.py", line 984, in require
        File "site-packages/pkg_resources/__init__.py", line 870, in resolve
      pkg_resources.DistributionNotFound: The 'datasette' distribution was not found and is required by the application
      [99117] Failed to execute script datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/88#issuecomment-344427448
issue_url: https://api.github.com/repos/simonw/datasette/issues/88
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQyNzQ0OA==
created_at: 2017-11-14T22:54:06Z
updated_at: 2017-11-14T22:54:06Z
author_association: OWNER
body:
  Hooray! First dataset that wasn't deployed by me :) https://github.com/simonw/datasette/wiki/Datasettes

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/88#issuecomment-344427560
issue_url: https://api.github.com/repos/simonw/datasette/issues/88
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQyNzU2MA==
created_at: 2017-11-14T22:54:33Z
updated_at: 2017-11-14T22:54:33Z
author_association: OWNER
body:
  I'm getting an internal server error on http://run.plnkr.co/preview/cj9zlf1qc0003414y90ajkwpk/ at the moment

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-344430299
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQzMDI5OQ==
created_at: 2017-11-14T23:06:33Z
updated_at: 2017-11-14T23:06:33Z
author_association: NONE
body:
  i will look better tomorrow, it's late i surely made some mistake
  https://asciinema.org/a/ZyAWbetrlriDadwWyVPUWB94H

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/88#issuecomment-344430689
issue_url: https://api.github.com/repos/simonw/datasette/issues/88
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQzMDY4OQ==
created_at: 2017-11-14T23:08:22Z
updated_at: 2017-11-14T23:08:22Z
author_association: CONTRIBUTOR
body:
  > I'm getting an internal server error on http://run.plnkr.co/preview/cj9zlf1qc0003414y90ajkwpk/ at the moment

  Sorry about that - here's a working version on Netlify:

  https://nhs-england-map.netlify.com

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-344438724
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQzODcyNA==
created_at: 2017-11-14T23:47:54Z
updated_at: 2017-11-14T23:47:54Z
author_association: OWNER
body:
  Plugins should be able to interact with the build step. This would give plugins an opportunity to modify the SQL databases and help prepare them for serving - for example, a full-text search plugin might create additional FTS tables, or a mapping plugin might pre-calculate a bunch of geohashes for tables that have latitude/longitude values. Plugins could really take advantage of the immutable nature of the dataset here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-344440377
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQ0MDM3Nw==
created_at: 2017-11-14T23:56:35Z
updated_at: 2017-11-14T23:56:35Z
author_association: OWNER
body:
  It worked!

      $ pyinstaller -F \
          --add-data /usr/local/lib/python3.5/site-packages/datasette/templates:datasette/templates \
          --add-data /usr/local/lib/python3.5/site-packages/datasette/static:datasette/static \
          /usr/local/bin/datasette

      $ file dist/datasette 
      dist/datasette: Mach-O 64-bit executable x86_64
      $ dist/datasette --help
      Usage: datasette [OPTIONS] COMMAND [ARGS]...

        Datasette!

      Options:
        --help  Show this message and exit.

      Commands:
        serve*   Serve up specified SQLite database files with...
        build
        package  Package specified SQLite files into a new...
        publish  Publish specified SQLite database files to...

reactions:
  {"total_count": 3, "+1": 0, "-1": 0, "laugh": 0, "hooray": 3, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-344440658
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQ0MDY1OA==
created_at: 2017-11-14T23:58:07Z
updated_at: 2017-11-14T23:58:07Z
author_association: OWNER
body:
  It's a shame pyinstaller can't act as a cross-compiler - so I don't think I can get Travis CI to build packages. But it's fantastic that it's possible to turn the tool into a standalone executable!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/85#issuecomment-344452063
issue_url: https://api.github.com/repos/simonw/datasette/issues/85
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQ1MjA2Mw==
created_at: 2017-11-15T01:03:03Z
updated_at: 2017-11-15T01:03:03Z
author_association: OWNER
body:
  This can work in reverse too. If you view the row page for something that has foreign keys against it, we can show you “53 items in TABLE link to this” and provide a link to view them all.

  That count worry could be prohibitively expensive. To counter that, we could run the count query via Ajax and set a strict time limit on it. See #95

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/13#issuecomment-344462608
issue_url: https://api.github.com/repos/simonw/datasette/issues/13
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQ2MjYwOA==
created_at: 2017-11-15T02:04:51Z
updated_at: 2017-11-15T02:04:51Z
author_association: OWNER
body:
  Fixed in https://github.com/simonw/datasette/commit/8252daa4c14d73b4b69e3f2db4576bb39d73c070 - thanks, @tomdyson!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/95#issuecomment-344463436
issue_url: https://api.github.com/repos/simonw/datasette/issues/95
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQ2MzQzNg==
created_at: 2017-11-15T02:10:10Z
updated_at: 2017-11-15T02:10:10Z
author_association: OWNER
body:
  This means clients can ask questions but say "don't bother if it takes longer than X" - which is really handy when you're working against unknown databases that might be small or might be enormous.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/94#issuecomment-344472313
issue_url: https://api.github.com/repos/simonw/datasette/issues/94
node_id: MDEyOklzc3VlQ29tbWVudDM0NDQ3MjMxMw==
created_at: 2017-11-15T03:08:00Z
updated_at: 2017-11-15T03:08:00Z
author_association: OWNER
body:
  Works for me. I'm going to land this.

  Just one thing:

      simonw$ docker run --rm -t -i -p 9001:8001 c408e8cfbe40 datasette publish now
       The publish command requires "now" to be installed and configured 
      Follow the instructions at https://zeit.co/now#whats-now

  Maybe we should have the Docker container install the "now" client? Not sure how much size that would add though. I think it's OK without for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-344516406
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDM0NDUxNjQwNg==
created_at: 2017-11-15T08:09:41Z
updated_at: 2017-11-15T08:09:41Z
author_association: NONE
body:
  actually you can use travis to build for linux/macos and [appveyor](https://www.appveyor.com/) to build for windows.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/85#issuecomment-344657040
issue_url: https://api.github.com/repos/simonw/datasette/issues/85
node_id: MDEyOklzc3VlQ29tbWVudDM0NDY1NzA0MA==
created_at: 2017-11-15T16:56:48Z
updated_at: 2017-11-15T16:56:48Z
author_association: OWNER
body:
  Since detecting foreign keys that point to a specific table is a bit expensive (you have to call a PRAGMA on every other table) I’m going to add this to the build/inspect stage.

  Idea: if we detect that the foreign key table only has one other column in it (id, name) AND we know that the id is the primary key, we can add an efficient lookup on the table list view and prefetch a dictionary mapping IDs to their value. Then we can feed that dictionary in as extra tenplate context and use it to render labeled hyperlinks in the corresponding column.

  This means our build step should also cache which columns are indexed, and add a “label_column” property for tables with an obvious lane column.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/90#issuecomment-344667202
issue_url: https://api.github.com/repos/simonw/datasette/issues/90
node_id: MDEyOklzc3VlQ29tbWVudDM0NDY2NzIwMg==
created_at: 2017-11-15T17:29:38Z
updated_at: 2017-11-15T17:29:38Z
author_association: OWNER
body:
  @jacobian points out that a buildpack may be a better fit than a Docker container for implementing this: https://twitter.com/jacobian/status/930849058465255424

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/90#issuecomment-344680385
issue_url: https://api.github.com/repos/simonw/datasette/issues/90
node_id: MDEyOklzc3VlQ29tbWVudDM0NDY4MDM4NQ==
created_at: 2017-11-15T18:14:11Z
updated_at: 2017-11-15T18:14:11Z
author_association: OWNER
body:
  Maybe we don’t even need a buildpack... we could create a temporary directory, set up a classic heroku app with the datasette serve command in the Procfile and then git push to deploy.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/90#issuecomment-344686483
issue_url: https://api.github.com/repos/simonw/datasette/issues/90
node_id: MDEyOklzc3VlQ29tbWVudDM0NDY4NjQ4Mw==
created_at: 2017-11-15T18:36:23Z
updated_at: 2017-11-15T18:36:23Z
author_association: OWNER
body:
  The “datasette build” command would need to run in a bin/post_compile script eg https://github.com/simonw/simonwillisonblog/blob/cloudflare-ips/bin/post_compile

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/90#issuecomment-344687328
issue_url: https://api.github.com/repos/simonw/datasette/issues/90
node_id: MDEyOklzc3VlQ29tbWVudDM0NDY4NzMyOA==
created_at: 2017-11-15T18:39:14Z
updated_at: 2017-11-15T18:39:49Z
author_association: OWNER
body:
  By default the command could use a temporary directory that gets cleaned up after the deploy, but we could allow users to opt in to keeping the generated directory like so:

      datasette publish heroku mydb.py -d ~/dev/my-heroku-app

  This would create the my-heroku-app folder so you can later execute further git deploys from there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/104#issuecomment-344710204
issue_url: https://api.github.com/repos/simonw/datasette/issues/104
node_id: MDEyOklzc3VlQ29tbWVudDM0NDcxMDIwNA==
created_at: 2017-11-15T19:57:50Z
updated_at: 2017-11-15T19:57:50Z
author_association: CONTRIBUTOR
body:
  A first basic stab at making this work, just to prove the approach. Right now this requires [a Heroku CLI plugin](https://github.com/heroku/heroku-builds), which seems pretty unreasonable. I think this can be replaced with direct API calls, which could clean up a lot of things. But I wanted to prove it worked first, and it does.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/107#issuecomment-344770170
issue_url: https://api.github.com/repos/simonw/datasette/issues/107
node_id: MDEyOklzc3VlQ29tbWVudDM0NDc3MDE3MA==
created_at: 2017-11-16T00:01:00Z
updated_at: 2017-11-16T00:01:22Z
author_association: OWNER
body:
  It is - but I think this will break on this line since it expects two format string parameters:

  https://github.com/simonw/datasette/blob/f45ca30f91b92ac68adaba893bf034f13ec61ced/datasette/utils.py#L61

  Needs unit tests too, which live here: https://github.com/simonw/datasette/blob/f45ca30f91b92ac68adaba893bf034f13ec61ced/tests/test_utils.py#L49

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/100#issuecomment-344771130
issue_url: https://api.github.com/repos/simonw/datasette/issues/100
node_id: MDEyOklzc3VlQ29tbWVudDM0NDc3MTEzMA==
created_at: 2017-11-16T00:06:00Z
updated_at: 2017-11-16T00:06:00Z
author_association: OWNER
body:
  Aha... it looks like this is a Jinja version problem: https://github.com/ansible/ansible/issues/25381#issuecomment-306492389

  Datasette depends on sanic-jinja2 - and that doesn't depend on a particular jinja2 version: https://github.com/lixxu/sanic-jinja2/blob/7e9520850d8c6bb66faf43b7f252593d7efe3452/setup.py#L22

  So if you have an older version of Jinja installed, stuff breaks.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/96#issuecomment-344786528
issue_url: https://api.github.com/repos/simonw/datasette/issues/96
node_id: MDEyOklzc3VlQ29tbWVudDM0NDc4NjUyOA==
created_at: 2017-11-16T01:32:41Z
updated_at: 2017-11-16T01:32:41Z
author_association: OWNER
body:
  <img width="733" alt="australian-dogs" src="https://user-images.githubusercontent.com/9599/32869280-f82fa176-ca2a-11e7-8ad1-ac2a13c85089.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/96#issuecomment-344788435
issue_url: https://api.github.com/repos/simonw/datasette/issues/96
node_id: MDEyOklzc3VlQ29tbWVudDM0NDc4ODQzNQ==
created_at: 2017-11-16T01:43:52Z
updated_at: 2017-11-16T01:43:52Z
author_association: OWNER
body:
  Demo: https://australian-dogs.now.sh/australian-dogs-3ba9628?sql=select+name%2C+count%28*%29+as+n+from+%28%0D%0A%0D%0Aselect+upper%28%22Animal+name%22%29+as+name+from+%5BAdelaide-City-Council-dog-registrations-2013%5D+where+Breed+like+%3Abreed%0D%0A%0D%0Aunion+all%0D%0A%0D%0Aselect+upper%28Animal_Name%29+as+name+from+%5BAdelaide-City-Council-dog-registrations-2014%5D+where+Breed_Description+like+%3Abreed%0D%0A%0D%0Aunion+all+%0D%0A%0D%0Aselect+upper%28Animal_Name%29+as+name+from+%5BAdelaide-City-Council-dog-registrations-2015%5D+where+Breed_Description+like+%3Abreed%0D%0A%0D%0Aunion+all%0D%0A%0D%0Aselect+upper%28%22AnimalName%22%29+as+name+from+%5BCity-of-Port-Adelaide-Enfield-Dog_Registrations_2016%5D+where+AnimalBreed+like+%3Abreed%0D%0A%0D%0Aunion+all%0D%0A%0D%0Aselect+upper%28%22Animal+Name%22%29+as+name+from+%5BMitcham-dog-registrations-2015%5D+where+Breed+like+%3Abreed%0D%0A%0D%0Aunion+all%0D%0A%0D%0Aselect+upper%28%22DOG_NAME%22%29+as+name+from+%5Bburnside-dog-registrations-2015%5D+where+DOG_BREED+like+%3Abreed%0D%0A%0D%0Aunion+all+%0D%0A%0D%0Aselect+upper%28%22Animal_Name%22%29+as+name+from+%5Bcity-of-playford-2015-dog-registration%5D+where+Breed_Description+like+%3Abreed%0D%0A%0D%0Aunion+all%0D%0A%0D%0Aselect+upper%28%22Animal+Name%22%29+as+name+from+%5Bcity-of-prospect-dog-registration-details-2016%5D+where%22Breed+Description%22+like+%3Abreed%0D%0A%0D%0A%29+group+by+name+order+by+n+desc%3B&breed=chihuahua

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/96#issuecomment-344788763
issue_url: https://api.github.com/repos/simonw/datasette/issues/96
node_id: MDEyOklzc3VlQ29tbWVudDM0NDc4ODc2Mw==
created_at: 2017-11-16T01:45:51Z
updated_at: 2017-11-16T01:45:51Z
author_association: OWNER
body:
  Another demo - this time it lets you search by name and see the most popular breeds with that name: https://australian-dogs.now.sh/australian-dogs-3ba9628?sql=select+breed%2C+count%28*%29+as+n+from+%28%0D%0A%0D%0Aselect+upper%28%22Breed%22%29+as+breed+from+%5BAdelaide-City-Council-dog-registrations-2013%5D+where+%22Animal+name%22+like+%3Aname%0D%0A%0D%0Aunion+all%0D%0A%0D%0Aselect+upper%28%22Breed_Description%22%29+as+breed+from+%5BAdelaide-City-Council-dog-registrations-2014%5D+where+%22Animal_Name%22+like+%3Aname%0D%0A%0D%0Aunion+all+%0D%0A%0D%0Aselect+upper%28%22Breed_Description%22%29+as+breed+from+%5BAdelaide-City-Council-dog-registrations-2015%5D+where+%22Animal_Name%22+like+%3Aname%0D%0A%0D%0Aunion+all%0D%0A%0D%0Aselect+upper%28%22AnimalBreed%22%29+as+breed+from+%5BCity-of-Port-Adelaide-Enfield-Dog_Registrations_2016%5D+where+%22AnimalName%22+like+%3Aname%0D%0A%0D%0Aunion+all%0D%0A%0D%0Aselect+upper%28%22Breed%22%29+as+breed+from+%5BMitcham-dog-registrations-2015%5D+where+%22Animal+Name%22+like+%3Aname%0D%0A%0D%0Aunion+all%0D%0A%0D%0Aselect+upper%28%22DOG_BREED%22%29+as+breed+from+%5Bburnside-dog-registrations-2015%5D+where+%22DOG_NAME%22+like+%3Aname%0D%0A%0D%0Aunion+all+%0D%0A%0D%0Aselect+upper%28%22Breed_Description%22%29+as+breed+from+%5Bcity-of-playford-2015-dog-registration%5D+where+%22Animal_Name%22+like+%3Aname%0D%0A%0D%0Aunion+all%0D%0A%0D%0Aselect+upper%28%22Breed+Description%22%29+as+breed+from+%5Bcity-of-prospect-dog-registration-details-2016%5D+where+%22Animal+Name%22+like+%3Aname%0D%0A%0D%0A%29+group+by+breed+order+by+n+desc%3B&name=rex

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/46#issuecomment-344810525
issue_url: https://api.github.com/repos/simonw/datasette/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDM0NDgxMDUyNQ==
created_at: 2017-11-16T04:11:25Z
updated_at: 2017-11-16T04:11:25Z
author_association: CONTRIBUTOR
body:
  @simonw On the spatialite support, here is some info to make it work and a screenshot:

  <img width="1230" alt="screen shot 2017-11-15 at 11 08 14 pm" src="https://user-images.githubusercontent.com/54999/32873420-f8a6d5a0-ca59-11e7-8a73-7d58d467e413.png">

  I used the following Dockerfile:
  ```
  FROM prolocutor/python3-sqlite-ext:3.5.1-spatialite as build

  RUN mkdir /code
  ADD . /code/

  RUN pip install /code/

  EXPOSE 8001
  CMD ["datasette", "serve", "/code/ne.sqlite", "--host", "0.0.0.0"]
  ```

  and added this to `prepare_connection`:
  ```
    conn.enable_load_extension(True)
    conn.execute("SELECT load_extension('/usr/local/lib/mod_spatialite.so')")
  ```

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/pull/107#issuecomment-344811268
issue_url: https://api.github.com/repos/simonw/datasette/issues/107
node_id: MDEyOklzc3VlQ29tbWVudDM0NDgxMTI2OA==
created_at: 2017-11-16T04:17:45Z
updated_at: 2017-11-16T04:17:45Z
author_association: CONTRIBUTOR
body:
  Thanks for the guidance.  I added a unit test and made a slight change to utils.py.

  I didn't realize this, but evidently string.format only complains if you supply less arguments than there are format placeholders, so the original commit worked, but was adding a superfluous named param.

  I added a conditional that prevents the named param from being created and ensures the correct number of args are passed to sting.format. It has the side effect of hiding the SQL query in /templates/table.html when there are no other where clauses--not sure if that's the desired outcome here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/46#issuecomment-344975156
issue_url: https://api.github.com/repos/simonw/datasette/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDM0NDk3NTE1Ng==
created_at: 2017-11-16T16:19:44Z
updated_at: 2017-11-16T16:19:44Z
author_association: OWNER
body:
  That's fantastic! Thank you very much for that. 

  Do you know if it's possible to view the Dockerfile used by https://hub.docker.com/r/prolocutor/python3-sqlite-ext/ ?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/46#issuecomment-344976104
issue_url: https://api.github.com/repos/simonw/datasette/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDM0NDk3NjEwNA==
created_at: 2017-11-16T16:22:45Z
updated_at: 2017-11-16T16:22:45Z
author_association: OWNER
body:
  Found a relevant Dockerfile on Reddit: https://www.reddit.com/r/Python/comments/5unkb3/install_sqlite3_on_python_3/ddzdz2b/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/46#issuecomment-344976882
issue_url: https://api.github.com/repos/simonw/datasette/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDM0NDk3Njg4Mg==
created_at: 2017-11-16T16:25:07Z
updated_at: 2017-11-16T16:25:07Z
author_association: OWNER
body:
  Maybe part of the solution here is to add a `--load-extension` argument to `datasette` - so when you run the command you can specify SQLite extensions that should be loaded. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/110#issuecomment-344988263
issue_url: https://api.github.com/repos/simonw/datasette/issues/110
node_id: MDEyOklzc3VlQ29tbWVudDM0NDk4ODI2Mw==
created_at: 2017-11-16T16:58:48Z
updated_at: 2017-11-16T16:58:48Z
author_association: OWNER
body:
  Here's how I tested this.

  First I downloaded and started a docker container using https://hub.docker.com/r/prolocutor/python3-sqlite-ext - which includes the compiled spatialite extension. This downloads it, then starts a shell in that container.

      docker run -it -p 8018:8018 prolocutor/python3-sqlite-ext:3.5.1-spatialite /bin/sh

  Installed a pre-release build of datasette which includes the new `--load-extension` option.

      pip install https://static.simonwillison.net/static/2017/datasette-0.13-py3-none-any.whl

  Now grab a sample database from https://www.gaia-gis.it/spatialite-2.3.1/resources.html - and unzip and rename it (datasette doesn't yet like databases with dots in their filename):

      wget http://www.gaia-gis.it/spatialite-2.3.1/test-2.3.sqlite.gz
      gunzip test-2.3.sqlite.gz
      mv test-2.3.sqlite test23.sqlite

  Now start datasette on port 8018 (the port I exposed earlier) with the extension loaded:

      datasette test23.sqlite  -p 8018 -h 0.0.0.0 --load-extension /usr/local/lib/mod_spatialite.so

  Now I can confirm that it worked:

  http://localhost:8018/test23-c88bc35?sql=select+ST_AsText%28Geometry%29+from+HighWays+limit+1

  <img width="743" alt="test23" src="https://user-images.githubusercontent.com/9599/32904449-39789a3a-caac-11e7-9531-b49f06051e34.png">

  If I run datasette without `--load-extension` I get this:

      datasette test23.sqlite  -p 8018 -h 0.0.0.0

  <img width="747" alt="test23_and_turn_on_auto-escaping_in_jinja_ _simonw_datasette_82261a6" src="https://user-images.githubusercontent.com/9599/32904508-54e53d78-caac-11e7-9f80-4d96e9f9fb5f.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/46#issuecomment-344988591
issue_url: https://api.github.com/repos/simonw/datasette/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDM0NDk4ODU5MQ==
created_at: 2017-11-16T16:59:51Z
updated_at: 2017-11-16T16:59:51Z
author_association: OWNER
body:
  OK, `--load-extension` is now a supported command line option - see #110 which includes my notes on how I manually tested it using the `prolocutor/python3-sqlite-ext` Docker image.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/46#issuecomment-344989340
issue_url: https://api.github.com/repos/simonw/datasette/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDM0NDk4OTM0MA==
created_at: 2017-11-16T17:02:07Z
updated_at: 2017-11-16T17:02:07Z
author_association: OWNER
body:
  The fact that `prolocutor/python3-sqlite-ext` doesn't provide a visible Dockerfile and hasn't been updated in two years makes me hesitant to bake it into datasette itself. I'd rather put together a Dockerfile that enables the necessary extensions and can live in the datasette repository itself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/46#issuecomment-345002908
issue_url: https://api.github.com/repos/simonw/datasette/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDM0NTAwMjkwOA==
created_at: 2017-11-16T17:47:49Z
updated_at: 2017-11-16T17:47:49Z
author_association: CONTRIBUTOR
body:
  I'll try to find alternatives to the Dockerfile option - I also think we should not use that old one without sources or license.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/111#issuecomment-345013127
issue_url: https://api.github.com/repos/simonw/datasette/issues/111
node_id: MDEyOklzc3VlQ29tbWVudDM0NTAxMzEyNw==
created_at: 2017-11-16T18:23:56Z
updated_at: 2017-11-16T18:23:56Z
author_association: OWNER
body:
  Having this as a global option may not make sense when  publishing multiple databases. We can revisit that when we implement per-database and per-table metadata.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/110#issuecomment-345017256
issue_url: https://api.github.com/repos/simonw/datasette/issues/110
node_id: MDEyOklzc3VlQ29tbWVudDM0NTAxNzI1Ng==
created_at: 2017-11-16T18:38:30Z
updated_at: 2017-11-16T18:38:30Z
author_association: OWNER
body:
  To finish up, I committed the image I created in the above so I can run it again in the future:

      docker commit $(docker ps -lq) datasette-sqlite

  Now I can run it like this:

      docker run -it -p 8018:8018 datasette-sqlite datasette /tmp/test23.sqlite  -p 8018 -h 0.0.0.0 --load-extension /usr/local/lib/mod_spatialite.so

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-345067498
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM0NTA2NzQ5OA==
created_at: 2017-11-16T21:25:32Z
updated_at: 2017-11-16T21:26:22Z
author_association: OWNER
body:
  For visualizations, Google Maps should be made available as a plugin. The default visualizations can use Leaflet and Open Street Map, but there's no reason to not make Google Maps available as a plugin, especially if the plugin can provide a mechanism for configuring the necessary API key.

  I'm particularly excited in the Google Maps heatmap visualization https://developers.google.com/maps/documentation/javascript/heatmaplayer as seen on http://mochimachine.org/wasteland/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/107#issuecomment-345108644
issue_url: https://api.github.com/repos/simonw/datasette/issues/107
node_id: MDEyOklzc3VlQ29tbWVudDM0NTEwODY0NA==
created_at: 2017-11-17T00:34:46Z
updated_at: 2017-11-17T00:34:46Z
author_association: OWNER
body:
  Looks like your tests are failing because of a bug which I fixed in https://github.com/simonw/datasette/commit/9199945a1bcec4852e1cb866eb3642614dd32a48 - if you rebase to master the tests should pass.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/107#issuecomment-345117690
issue_url: https://api.github.com/repos/simonw/datasette/issues/107
node_id: MDEyOklzc3VlQ29tbWVudDM0NTExNzY5MA==
created_at: 2017-11-17T01:29:41Z
updated_at: 2017-11-17T01:29:41Z
author_association: CONTRIBUTOR
body:
  Thanks for bearing with me. I was getting a message about my branch diverging when I tried to push after rebasing, so I merged master into isnull, seems like that did the trick. Let me know if I should make any corrections.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/46#issuecomment-345138347
issue_url: https://api.github.com/repos/simonw/datasette/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDM0NTEzODM0Nw==
created_at: 2017-11-17T03:52:25Z
updated_at: 2017-11-17T03:52:25Z
author_association: OWNER
body:
  We now have a Dockerfile that compiles spatialite! https://github.com/simonw/datasette/pull/114/commits/6c6b63d890529eeefcefb7ab126ea3bd7b2315c1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/85#issuecomment-345150048
issue_url: https://api.github.com/repos/simonw/datasette/issues/85
node_id: MDEyOklzc3VlQ29tbWVudDM0NTE1MDA0OA==
created_at: 2017-11-17T05:35:25Z
updated_at: 2017-11-17T05:35:25Z
author_association: OWNER
body:
  `csvs-to-sqlite` is now capable of generating databases with foreign key lookup tables: https://github.com/simonw/csvs-to-sqlite/releases/tag/0.3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/85#issuecomment-345242447
issue_url: https://api.github.com/repos/simonw/datasette/issues/85
node_id: MDEyOklzc3VlQ29tbWVudDM0NTI0MjQ0Nw==
created_at: 2017-11-17T13:22:33Z
updated_at: 2017-11-17T13:23:14Z
author_association: OWNER
body:
  I could support explicit label columns using additional arguments to `datasette serve`:

      datasette serve mydb.py --label-column mydb:table1:name --label-column mydb:table2:title

  This would mean "in mydb, set the label column for table1 to name, and the label column for table2 to title"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/112#issuecomment-345255655
issue_url: https://api.github.com/repos/simonw/datasette/issues/112
node_id: MDEyOklzc3VlQ29tbWVudDM0NTI1NTY1NQ==
created_at: 2017-11-17T14:19:23Z
updated_at: 2017-11-17T14:19:23Z
author_association: OWNER
body:
  I tesed this by first building and running a container using the new
  Dockerfile from #114:

      docker build .
      docker run -it -p 8001:8001 6c9ca7e29181 /bin/sh

  Then I ran this inside the container itself:

      apt update && apt-get install wget -y \
          && wget http://www.gaia-gis.it/spatialite-2.3.1/test-2.3.sqlite.gz \
          && gunzip test-2.3.sqlite.gz \
          && mv test-2.3.sqlite test23.sqlite \
          && datasette -h 0.0.0.0 test23.sqlite

  I visited this URL to confirm I got an error due to spatialite not being
  loaded:

  http://localhost:8001/test23-c88bc35?sql=select+ST_AsText%28Geometry%29+from+HighWays+limit+1

  Then I checked that loading it with `--load-extension` worked correctly:

      datasette -h 0.0.0.0 test23.sqlite \
          --load-extension=/usr/lib/x86_64-linux-gnu/mod_spatialite.so

  Then, finally, I tested it with the new environment variable option:

      SQLITE_EXTENSIONS=/usr/lib/x86_64-linux-gnu/mod_spatialite.so \
          datasette -h 0.0.0.0 test23.sqlite

  Running it with an invalid environment variable option shows an error:

      $ SQLITE_EXTENSIONS=/usr/lib/x86_64-linux-gnu/blah.so datasette \
          -h 0.0.0.0 test23.sqlite
      Usage: datasette -h [OPTIONS] [FILES]...

      Error: Invalid value for "--load-extension": Path "/usr/lib/x86_64-linux-gnu/blah.so" does not exist.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/115#issuecomment-345256576
issue_url: https://api.github.com/repos/simonw/datasette/issues/115
node_id: MDEyOklzc3VlQ29tbWVudDM0NTI1NjU3Ng==
created_at: 2017-11-17T14:22:51Z
updated_at: 2017-11-17T14:22:51Z
author_association: OWNER
body:
  This is great - I've been frustrated by how CodeMirror prevents me from hitting tab-enter to activate the "Run SQL" button. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/46#issuecomment-345259115
issue_url: https://api.github.com/repos/simonw/datasette/issues/46
node_id: MDEyOklzc3VlQ29tbWVudDM0NTI1OTExNQ==
created_at: 2017-11-17T14:32:12Z
updated_at: 2017-11-17T14:32:12Z
author_association: OWNER
body:
  OK, I can confirm that the version in the new docker container supports FTS5, JSON *and* spatialite!

  Notes on how I built the container and tested the spatialite extension are here: https://github.com/simonw/datasette/issues/112#issuecomment-345255655

  To confirm that JSON and FTS5 are working, I ran the following:

      $ docker run -it -p 8001:8001 6c9ca7e29181 python
      Python 3.6.3 (default, Nov  4 2017, 14:24:48) 
      [GCC 6.3.0 20170516] on linux
      Type "help", "copyright", "credits" or "license" for more information.
      >>> import sqlite3
      >>> sqlite3.connect(':memory:').execute('CREATE VIRTUAL TABLE email USING fts5(sender, title, body);')
      <sqlite3.Cursor object at 0x7f2d90839960>
      >>> list(sqlite3.connect(':memory:').execute('''SELECT json(' { "this" : "is", "a": [ "test" ] } ') '''))
      [('{"this":"is","a":["test"]}',)]

  If I do the same thing in python3 on my OS X laptop directly, I get this:

      $ python3
      Python 3.5.1 (default, Apr 18 2016, 11:46:32) 
      [GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.29)] on darwin
      Type "help", "copyright", "credits" or "license" for more information.
      >>> import sqlite3
      >>> sqlite3.connect(':memory:').execute('CREATE VIRTUAL TABLE email USING fts5(sender, title, body);')
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      sqlite3.OperationalError: no such module: fts5
      >>> list(sqlite3.connect(':memory:').execute('''SELECT json(' { "this" : "is", "a": [ "test" ] } ') '''))
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      sqlite3.OperationalError: no such function: json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/36#issuecomment-345262738
issue_url: https://api.github.com/repos/simonw/datasette/issues/36
node_id: MDEyOklzc3VlQ29tbWVudDM0NTI2MjczOA==
created_at: 2017-11-17T14:45:37Z
updated_at: 2017-11-17T14:45:37Z
author_association: OWNER
body:
  Consider for example https://fivethirtyeight.datasettes.com/fivethirtyeight/inconvenient-sequel%2Fratings

  <img width="719" alt="fivethirtyeight__inconvenient-sequel_ratings" src="https://user-images.githubusercontent.com/9599/32952559-82b81ea8-cb62-11e7-817a-45c8bba7e9a2.png">

  The idea here is to be able to support querystring parameters like this:

  * `?timestamp___date=2017-07-17` - return every item where the timestamp falls on that date
  * `?timestamp___year=2017` - return every item where the timestamp falls within 2017
  * `?timestamp___month=1` - return every item where the month component is January
  * `?timestamp___day=10` - return every item where the day-of-the-month component is 10

  This is similar to #64 but a fair bit more complicated. 

  SQLite date functions are documented here: https://sqlite.org/lang_datefunc.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/36#issuecomment-345448756
issue_url: https://api.github.com/repos/simonw/datasette/issues/36
node_id: MDEyOklzc3VlQ29tbWVudDM0NTQ0ODc1Ng==
created_at: 2017-11-18T15:17:43Z
updated_at: 2017-11-18T15:17:43Z
author_association: OWNER
body:
  This may be useful:

  https://github.com/coleifer/peewee/blob/db85167d93861451a1fe7cde8c4f05748b222634/peewee.py#L162-L185

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/104#issuecomment-345452669
issue_url: https://api.github.com/repos/simonw/datasette/issues/104
node_id: MDEyOklzc3VlQ29tbWVudDM0NTQ1MjY2OQ==
created_at: 2017-11-18T16:18:45Z
updated_at: 2017-11-18T16:18:45Z
author_association: CONTRIBUTOR
body:
  I'd like to do a bit of cleanup, and some error checking in case heroku/heroku-builds isn't installed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/105#issuecomment-345493344
issue_url: https://api.github.com/repos/simonw/datasette/issues/105
node_id: MDEyOklzc3VlQ29tbWVudDM0NTQ5MzM0NA==
created_at: 2017-11-19T05:28:49Z
updated_at: 2017-11-19T05:28:49Z
author_association: OWNER
body:
  Looks like there are a ton of interesting datasets packaged in this way at http://datahub.io/docs/core-data - see also https://github.com/datasets

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/85#issuecomment-345494724
issue_url: https://api.github.com/repos/simonw/datasette/issues/85
node_id: MDEyOklzc3VlQ29tbWVudDM0NTQ5NDcyNA==
created_at: 2017-11-19T06:08:19Z
updated_at: 2017-11-19T06:08:19Z
author_association: OWNER
body:
  This is working really nicely now:

  <img width="1082" alt="sf-trees__street_tree_list" src="https://user-images.githubusercontent.com/9599/32987909-ef2c6008-ccac-11e7-81ce-4451f5372ab9.png">

  <img width="461" alt="sf-trees__qspecies" src="https://user-images.githubusercontent.com/9599/32987911-fa5ce538-ccac-11e7-83b9-2e97dea82d86.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/86#issuecomment-345494775
issue_url: https://api.github.com/repos/simonw/datasette/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDM0NTQ5NDc3NQ==
created_at: 2017-11-19T06:09:43Z
updated_at: 2017-11-19T06:09:43Z
author_association: OWNER
body:
  Now that we have foreign key support (#85) this is even more important, since foreign key support actively encourages linking to filtered table views.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/86#issuecomment-345494918
issue_url: https://api.github.com/repos/simonw/datasette/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDM0NTQ5NDkxOA==
created_at: 2017-11-19T06:14:17Z
updated_at: 2017-11-19T06:14:17Z
author_association: OWNER
body:
  If the selected relationship is a foreign key reference, we should resolve that foreign key and display it on the page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/44#issuecomment-345494971
issue_url: https://api.github.com/repos/simonw/datasette/issues/44
node_id: MDEyOklzc3VlQ29tbWVudDM0NTQ5NDk3MQ==
created_at: 2017-11-19T06:15:39Z
updated_at: 2017-11-19T06:15:39Z
author_association: OWNER
body:
  It would be great if this could support foreign key references and automatically resolve and hyperlink them if they are detected.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/127#issuecomment-345495046
issue_url: https://api.github.com/repos/simonw/datasette/issues/127
node_id: MDEyOklzc3VlQ29tbWVudDM0NTQ5NTA0Ng==
created_at: 2017-11-19T06:17:42Z
updated_at: 2017-11-19T06:17:42Z
author_association: OWNER
body:
  Maybe I should support `&_count=1` to handle this - that would be easy to Ajax-in in conjenction with the other filters.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/86#issuecomment-345496540
issue_url: https://api.github.com/repos/simonw/datasette/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDM0NTQ5NjU0MA==
created_at: 2017-11-19T06:59:40Z
updated_at: 2017-11-19T06:59:40Z
author_association: OWNER
body:
  OK,I've figured out how to do an initial version of this without JavaScript. I'll provide three form fields labell d "add filter":

  * a select box of all of the columns
  * a select box of the available operations
  * a value box

  Submit those and the site will redirect you to a correctly populated querystring for that filter.

  If you have filters applied, those will display as prepopulated form field triples.

  For foreign key reference filters, I will display the resolved value next to the text box containing the numeric ID. In the future this can get a select2 style treatment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/86#issuecomment-345497453
issue_url: https://api.github.com/repos/simonw/datasette/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDM0NTQ5NzQ1Mw==
created_at: 2017-11-19T07:21:22Z
updated_at: 2017-11-19T07:21:22Z
author_association: OWNER
body:
  I'm going to be a bit classier about this and auto generate a title for the page that describes the currently applied filters.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/86#issuecomment-345497689
issue_url: https://api.github.com/repos/simonw/datasette/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDM0NTQ5NzY4OQ==
created_at: 2017-11-19T07:27:40Z
updated_at: 2017-11-19T07:27:40Z
author_association: OWNER
body:
  I'll have to refactor the foreign key annotating code to be usable in other contexts - at the moment it only works for annotating displays of rows, but I need to use it to resolve selected filters as well. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/105#issuecomment-345503897
issue_url: https://api.github.com/repos/simonw/datasette/issues/105
node_id: MDEyOklzc3VlQ29tbWVudDM0NTUwMzg5Nw==
created_at: 2017-11-19T09:38:08Z
updated_at: 2017-11-19T09:38:08Z
author_association: CONTRIBUTOR
body:
  Thanks, I wrote this very simple reader because the default approach as described on the Datahub pages seemed to complicated. I had metadata from the `datapackage.json` attached to the returned DataFrames but removed this due to some attribute handling change in the latest Pandas version.

  This could also be useful for getting from Data Package to SQL db: https://github.com/frictionlessdata/tableschema-sql-py

  I maintain a few climate science related dataset at https://github.com/openclimatedata/

  The Data Retriever (mainly ecological data) by @ethanwhite et al. is also using the Data Package format for metadata and has some tooling for different dbs: 

  https://frictionlessdata.io/articles/the-data-retriever/
  https://github.com/weecology/retriever

  The Open Power System Data project also has a couple of datasets that show nicely how CSV is great for assembling and then already make SQLite files available. It's one of the first data sets I tried with Datasette, perfect for the use case of getting an API for putting power stations on a map ...

  https://data.open-power-system-data.org/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/97#issuecomment-345509500
issue_url: https://api.github.com/repos/simonw/datasette/issues/97
node_id: MDEyOklzc3VlQ29tbWVudDM0NTUwOTUwMA==
created_at: 2017-11-19T11:26:58Z
updated_at: 2017-11-19T11:26:58Z
author_association: NONE
body:
  Specifically docs should make it clearer this file exists

  https://parlgov.datasettes.com/.json

  And from that you can build https://parlgov.datasettes.com/parlgov-25f9855.json

  Then https://parlgov.datasettes.com/parlgov-25f9855/cabinet.json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/131#issuecomment-345526517
issue_url: https://api.github.com/repos/simonw/datasette/issues/131
node_id: MDEyOklzc3VlQ29tbWVudDM0NTUyNjUxNw==
created_at: 2017-11-19T15:48:28Z
updated_at: 2017-11-19T15:48:28Z
author_association: OWNER
body:
  Since SQLite supports column specifications in the MATCH body itself, there's no need to provide a separate mechanism for specifying columns in the query string: https://sqlite.org/fts5.html#fts5_column_filters

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/131#issuecomment-345533274
issue_url: https://api.github.com/repos/simonw/datasette/issues/131
node_id: MDEyOklzc3VlQ29tbWVudDM0NTUzMzI3NA==
created_at: 2017-11-19T17:17:37Z
updated_at: 2017-11-19T17:18:05Z
author_association: OWNER
body:
  Demo: https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List?_search=grove+st

  <img width="906" alt="sf-trees__street_tree_list_and_https___datasette-publish-dev_now_sh_auth__code_jigy80kxgsfddalfdwqwldzm" src="https://user-images.githubusercontent.com/9599/32993276-8bd62cce-cd0a-11e7-9d91-dc718d3eb4ab.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/122#issuecomment-345552358
issue_url: https://api.github.com/repos/simonw/datasette/issues/122
node_id: MDEyOklzc3VlQ29tbWVudDM0NTU1MjM1OA==
created_at: 2017-11-19T21:45:38Z
updated_at: 2017-12-05T19:09:52Z
author_association: OWNER
body:
  For the overall shape of the rows:

  `?_shape=lists` (default), `?_shape=objects`, `?_shape=object` (primary key as object keys)

  For getting back extra keys:

  `?_extras=schema,query,timing`

  For expanding columns:

  `?_expand_all=1`

  Or `?_expand=qSpecies&_expand=qCaretaker`

  The template view will only be allowed to work with data it can request using extra options.

  That leaves one sighted nasty edge-case: the default view will expand all columns, but the `.json` view of it won't?

  I think that's OK. The default view won't include the extras used by the template to render the page either.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/122#issuecomment-345552440
issue_url: https://api.github.com/repos/simonw/datasette/issues/122
node_id: MDEyOklzc3VlQ29tbWVudDM0NTU1MjQ0MA==
created_at: 2017-11-19T21:46:43Z
updated_at: 2017-11-19T21:46:43Z
author_association: OWNER
body:
  This calls for refactoring the code so the table view, the row view and the custom SQL view share as much logic as possible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/86#issuecomment-345559864
issue_url: https://api.github.com/repos/simonw/datasette/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDM0NTU1OTg2NA==
created_at: 2017-11-19T23:35:48Z
updated_at: 2017-11-19T23:35:48Z
author_association: OWNER
body:
  I need a nicer abstraction around the concept of filters. It needs to be able to:

  - convert querystring parameters into filters
  - convert filters into a querystring
  - iterate through currently applied filters
  - convert selected filters into a human description (e.g. for a title)
  - expand filters that involve a foreign key
  - add filters
  - remove filters
  - define different types of filters

  It should replace my current `build_where_clauses` implementation, in particular this bit:

  https://github.com/simonw/datasette/blob/a5881e105a02830d26f07e98177248d5910893da/datasette/utils.py#L38-L56

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/44#issuecomment-345601103
issue_url: https://api.github.com/repos/simonw/datasette/issues/44
node_id: MDEyOklzc3VlQ29tbWVudDM0NTYwMTEwMw==
created_at: 2017-11-20T06:13:35Z
updated_at: 2017-11-20T06:13:35Z
author_association: OWNER
body:
  Some demos:

  Single column: https://sf-trees-flat.now.sh/sf-trees-flat-ba738ce/Street_Tree_List?_group_count=qSpecies

  Multi column: https://sf-trees-flat.now.sh/sf-trees-flat-ba738ce/Street_Tree_List?_group_count=qLegalStatus&_group_count=qSpecies

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/27#issuecomment-345652450
issue_url: https://api.github.com/repos/simonw/datasette/issues/27
node_id: MDEyOklzc3VlQ29tbWVudDM0NTY1MjQ1MA==
created_at: 2017-11-20T10:19:39Z
updated_at: 2017-11-20T10:19:39Z
author_association: CONTRIBUTOR
body:
  If Data Package metadata gets adopted (#105) the views spec work might also be worth a look:

  http://frictionlessdata.io/specs/views/

  http://datahub.io/docs/features/views

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/137#issuecomment-345750135
issue_url: https://api.github.com/repos/simonw/datasette/issues/137
node_id: MDEyOklzc3VlQ29tbWVudDM0NTc1MDEzNQ==
created_at: 2017-11-20T16:30:56Z
updated_at: 2018-07-10T17:53:13Z
author_association: OWNER
body:
  One possible route: introduce prefixes eg `?a.Trees.age__gt=5&a.Trees._group_count=qSpecies&b.Trees.age__gt=10&b.Trees._group_count=qSpecies`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/129#issuecomment-345793887
issue_url: https://api.github.com/repos/simonw/datasette/issues/129
node_id: MDEyOklzc3VlQ29tbWVudDM0NTc5Mzg4Nw==
created_at: 2017-11-20T19:00:30Z
updated_at: 2017-11-20T19:00:30Z
author_association: OWNER
body:
  Need to hide these from the index summary page as well:

  <img width="1237" alt="datasette__archive-fires-indonesia__conventional_power_plants_eu__fatal-police-shootings-data__fivethirtyeight__flights__google-trends__nhsadmin__northwind__openelections__polar-bears__sf-trees-fks__venue_profile" src="https://user-images.githubusercontent.com/9599/33036011-0471beaa-cde2-11e7-8d01-86eb45476943.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/105#issuecomment-345809808
issue_url: https://api.github.com/repos/simonw/datasette/issues/105
node_id: MDEyOklzc3VlQ29tbWVudDM0NTgwOTgwOA==
created_at: 2017-11-20T19:50:53Z
updated_at: 2017-11-20T19:50:53Z
author_association: OWNER
body:
  OK, https://github.com/openclimatedata/global-carbon-budget/blob/master/datapackage.json really does look like it covers all of the bases I need for #138. Closing this ticket in favour of that new one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-345893877
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM0NTg5Mzg3Nw==
created_at: 2017-11-21T02:11:27Z
updated_at: 2017-11-21T02:11:27Z
author_association: OWNER
body:
  http://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins Is pretty good 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/104#issuecomment-346116745
issue_url: https://api.github.com/repos/simonw/datasette/issues/104
node_id: MDEyOklzc3VlQ29tbWVudDM0NjExNjc0NQ==
created_at: 2017-11-21T18:23:25Z
updated_at: 2017-11-21T18:23:25Z
author_association: CONTRIBUTOR
body:
  @simonw ready for a review and merge if you want.

  There's still some nasty duplicated code in cli.py and utils.py, which is just going to get worse if/when we start adding any other deploy targets (and I want to do one for cloud.gov, at least). I think there's an opportunity for some refactoring here. I'm happy to do that now as part of this PR, or if you merge this first I'll do it in a different one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/104#issuecomment-346124073
issue_url: https://api.github.com/repos/simonw/datasette/issues/104
node_id: MDEyOklzc3VlQ29tbWVudDM0NjEyNDA3Mw==
created_at: 2017-11-21T18:49:55Z
updated_at: 2017-11-21T18:49:55Z
author_association: CONTRIBUTOR
body:
  Actually hang on, don't merge - there are some bugs that #141 masked when I tested this out elsewhere.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/141#issuecomment-346157542
issue_url: https://api.github.com/repos/simonw/datasette/issues/141
node_id: MDEyOklzc3VlQ29tbWVudDM0NjE1NzU0Mg==
created_at: 2017-11-21T20:53:47Z
updated_at: 2017-11-21T20:53:47Z
author_association: OWNER
body:
  I think a copy is the right thing to do here - it will be cleaned up when the temp directory is removed. The hard link thing was always intended to save space, but if we can't do a hard link I don't see any harm in a temporary file copy.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/90#issuecomment-346161985
issue_url: https://api.github.com/repos/simonw/datasette/issues/90
node_id: MDEyOklzc3VlQ29tbWVudDM0NjE2MTk4NQ==
created_at: 2017-11-21T21:10:22Z
updated_at: 2017-11-21T21:10:22Z
author_association: OWNER
body:
  Woohoo!

  I've found one tiny issue: right now, the following doesn't work:

      datasette publish heroku ../demo-databses/google-trends.db

  It results in this error in the Heroku logs:

      2017-11-21T21:03:29.210511+00:00 app[web.1]: Usage: datasette serve [OPTIONS] [FILES]...
      2017-11-21T21:03:29.210524+00:00 app[web.1]: 
      2017-11-21T21:03:29.210555+00:00 app[web.1]: Error: Invalid value for "files": Path "../demo-databses/google-trends.db" does not exist.

  The command works fine if you run it in the same directory as the database file you are publishing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/90#issuecomment-346163513
issue_url: https://api.github.com/repos/simonw/datasette/issues/90
node_id: MDEyOklzc3VlQ29tbWVudDM0NjE2MzUxMw==
created_at: 2017-11-21T21:16:16Z
updated_at: 2017-11-21T21:16:16Z
author_association: OWNER
body:
  The reason relative paths work for `publish now` is that the `make_dockerfile()` function is called by passing the file names, not the full file paths:

  https://github.com/simonw/datasette/blob/e47117ce1d15f11246a3120aa49de70205713d05/datasette/utils.py#L166

  Clearly the correct thing to do here is for us to refactor the shared code between heroku/package/now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-346244871
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM0NjI0NDg3MQ==
created_at: 2017-11-22T05:06:30Z
updated_at: 2017-11-22T05:06:30Z
author_association: CONTRIBUTOR
body:
  I'd also suggest taking a look at [stevedore](https://docs.openstack.org/stevedore/latest/), which has a ton of tools for doing plugin stuff. I've had good luck with it in the past.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/144#issuecomment-346405660
issue_url: https://api.github.com/repos/simonw/datasette/issues/144
node_id: MDEyOklzc3VlQ29tbWVudDM0NjQwNTY2MA==
created_at: 2017-11-22T16:38:05Z
updated_at: 2017-11-22T16:38:05Z
author_association: OWNER
body:
  I have a solution for FTS already, but I'm interested in apsw as a mechanism for allowing custom virtual tables to be written in Python (pysqlite only lets you write custom functions)

  Not having PyPI support is pretty tough though. I'm planning a plugin/extension system which would be ideal for things like an optional apsw mode, but that's a lot harder if apsw isn't in PyPI.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/144#issuecomment-346427794
issue_url: https://api.github.com/repos/simonw/datasette/issues/144
node_id: MDEyOklzc3VlQ29tbWVudDM0NjQyNzc5NA==
created_at: 2017-11-22T17:55:45Z
updated_at: 2017-11-22T17:55:45Z
author_association: NONE
body:
  Thanks. There is a way to use pip to grab apsw, which also let's you configure it (flags to build extensions, use an internal sqlite, etc). Don't know how that works as a dependency for another package, though.

  On November 22, 2017 11:38:06 AM EST, Simon Willison <notifications@github.com> wrote:
  >I have a solution for FTS already, but I'm interested in apsw as a
  >mechanism for allowing custom virtual tables to be written in Python
  >(pysqlite only lets you write custom functions)
  >
  >Not having PyPI support is pretty tough though. I'm planning a
  >plugin/extension system which would be ideal for things like an
  >optional apsw mode, but that's a lot harder if apsw isn't in PyPI.
  >
  >-- 
  >You are receiving this because you authored the thread.
  >Reply to this email directly or view it on GitHub:
  >https://github.com/simonw/datasette/issues/144#issuecomment-346405660

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/129#issuecomment-346463342
issue_url: https://api.github.com/repos/simonw/datasette/issues/129
node_id: MDEyOklzc3VlQ29tbWVudDM0NjQ2MzM0Mg==
created_at: 2017-11-22T20:22:02Z
updated_at: 2017-11-22T20:22:02Z
author_association: OWNER
body:
  On the index page:

  <img width="590" alt="datasette__archive-fires-indonesia__conventional_power_plants_eu__fatal-police-shootings-data__fivethirtyeight__flights__google-trends__nhsadmin__northwind__openelections__polar-bears__sf-trees__sf-trees-fks__venue_profile_and_hide_fts-crea" src="https://user-images.githubusercontent.com/9599/33148136-7b8964ea-cf7f-11e7-87ff-654b4d152645.png">

  On the database index page:

  <img width="321" alt="sf-trees" src="https://user-images.githubusercontent.com/9599/33148156-87df1aa0-cf7f-11e7-8480-64cc29201984.png">

  After clicking that link:

  <img width="744" alt="sf-trees" src="https://user-images.githubusercontent.com/9599/33148178-9f7d510e-cf7f-11e7-8959-5ad64f8637a8.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/86#issuecomment-346530498
issue_url: https://api.github.com/repos/simonw/datasette/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDM0NjUzMDQ5OA==
created_at: 2017-11-23T04:35:07Z
updated_at: 2017-11-23T04:35:07Z
author_association: OWNER
body:
  Here's where I am now. Needs a bit of UI tidy up and it will be good to release:

  <img width="1126" alt="conventional_power_plants_eu__conventional_power_plants_eu" src="https://user-images.githubusercontent.com/9599/33159075-9e985bb8-cfc4-11e7-888d-6828ebdeb1ca.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/146#issuecomment-346682905
issue_url: https://api.github.com/repos/simonw/datasette/issues/146
node_id: MDEyOklzc3VlQ29tbWVudDM0NjY4MjkwNQ==
created_at: 2017-11-23T18:55:08Z
updated_at: 2017-11-23T18:55:08Z
author_association: OWNER
body:
  <img width="478" alt="compute_engine_-_simonwillisonblog" src="https://user-images.githubusercontent.com/9599/33186224-c3e32d44-d03c-11e7-9cd8-6acc6c1d1d01.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/86#issuecomment-346691243
issue_url: https://api.github.com/repos/simonw/datasette/issues/86
node_id: MDEyOklzc3VlQ29tbWVudDM0NjY5MTI0Mw==
created_at: 2017-11-23T20:07:15Z
updated_at: 2017-11-23T20:07:15Z
author_association: OWNER
body:
  <img width="566" alt="fivethirtyeight__bob-ross_elements-by-episode_csv" src="https://user-images.githubusercontent.com/9599/33187492-d6fb6068-d046-11e7-80fc-470c04a46386.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/132#issuecomment-346701751
issue_url: https://api.github.com/repos/simonw/datasette/issues/132
node_id: MDEyOklzc3VlQ29tbWVudDM0NjcwMTc1MQ==
created_at: 2017-11-23T21:51:51Z
updated_at: 2017-11-23T21:51:51Z
author_association: OWNER
body:
  <img width="846" alt="fatal-police-shootings-data__fatal-police-shootings-data" src="https://user-images.githubusercontent.com/9599/33189202-73891af2-d055-11e7-9e79-6be7d8df36aa.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/133#issuecomment-346705879
issue_url: https://api.github.com/repos/simonw/datasette/issues/133
node_id: MDEyOklzc3VlQ29tbWVudDM0NjcwNTg3OQ==
created_at: 2017-11-23T22:43:42Z
updated_at: 2017-11-24T22:07:46Z
author_association: OWNER
body:
  Easiest way to do this will be to move it into the same `<form>` as the filters.

  Would be nice to detect `?_search=<blank>` and redirect to URL without the `_search` parameter, just for aesthetics.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/147#issuecomment-346900554
issue_url: https://api.github.com/repos/simonw/datasette/issues/147
node_id: MDEyOklzc3VlQ29tbWVudDM0NjkwMDU1NA==
created_at: 2017-11-24T22:02:22Z
updated_at: 2017-11-24T22:02:22Z
author_association: OWNER
body:
  <img width="798" alt="conventional_power_plants_eu__conventional_power_plants_eu__14_rows_where_company____nuon_" src="https://user-images.githubusercontent.com/9599/33224603-15e7a406-d120-11e7-89c1-32d3d0aff021.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/133#issuecomment-346902583
issue_url: https://api.github.com/repos/simonw/datasette/issues/133
node_id: MDEyOklzc3VlQ29tbWVudDM0NjkwMjU4Mw==
created_at: 2017-11-24T22:30:32Z
updated_at: 2017-11-24T22:30:32Z
author_association: OWNER
body:
  <img width="662" alt="sf-trees__street_tree_list__1_row_where_search_matches__ocean___qcareassistant____1__qcareassistant_is_not_blank_and_qlegalstatus___1" src="https://user-images.githubusercontent.com/9599/33224944-03bf054a-d124-11e7-8a97-58c8dee892a1.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/149#issuecomment-346903317
issue_url: https://api.github.com/repos/simonw/datasette/issues/149
node_id: MDEyOklzc3VlQ29tbWVudDM0NjkwMzMxNw==
created_at: 2017-11-24T22:41:58Z
updated_at: 2017-11-24T22:41:58Z
author_association: OWNER
body:
  Custom SQL results now look like this:

  <img width="769" alt="banners_and_alerts_and_conventional_power_plants_eu" src="https://user-images.githubusercontent.com/9599/33225056-9c09c58c-d125-11e7-9981-a209a52ac7fb.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/124#issuecomment-346987395
issue_url: https://api.github.com/repos/simonw/datasette/issues/124
node_id: MDEyOklzc3VlQ29tbWVudDM0Njk4NzM5NQ==
created_at: 2017-11-26T06:24:08Z
updated_at: 2017-11-26T06:24:08Z
author_association: NONE
body:
  Are there performance gains when using immutable as opposed to read-only? From what I see other processes can still modify the DB when immutable, but there are no change notifications.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/124#issuecomment-347049888
issue_url: https://api.github.com/repos/simonw/datasette/issues/124
node_id: MDEyOklzc3VlQ29tbWVudDM0NzA0OTg4OA==
created_at: 2017-11-27T00:01:08Z
updated_at: 2017-11-27T00:01:08Z
author_association: OWNER
body:
  https://sqlite.org/c3ref/open.html Is the only documentation I've been able to find of the immutable option:

  > **immutable**: The immutable parameter is a boolean query parameter that indicates that the database file is stored on read-only media. When immutable is set, SQLite assumes that the database file cannot be changed, even by a process with higher privilege, and so the database is opened read-only and all locking and change detection is disabled. Caution: Setting the immutable property on a database file that does in fact change can result in incorrect query results and/or SQLITE_CORRUPT errors. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/153#issuecomment-347050235
issue_url: https://api.github.com/repos/simonw/datasette/issues/153
node_id: MDEyOklzc3VlQ29tbWVudDM0NzA1MDIzNQ==
created_at: 2017-11-27T00:06:24Z
updated_at: 2017-11-27T00:06:24Z
author_association: OWNER
body:
  I've been thinking about 1. a bit - I actually think it would be fine to have a rule that says "if the contents of the cell starts with `http://` or `https://` and doesn't contain any whitespace, turn that into a link". If you need the non-linked version that will always be available in the JSON.

  For the other two... I think #12 may be the way to go here: if you can easily over-ride the `row.html` and `table.html` templates for specific databases you can easily set pre-formatted text or similar for certain values - maybe even with CSS that targets a specific table column.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/153#issuecomment-347051331
issue_url: https://api.github.com/repos/simonw/datasette/issues/153
node_id: MDEyOklzc3VlQ29tbWVudDM0NzA1MTMzMQ==
created_at: 2017-11-27T00:23:40Z
updated_at: 2017-11-27T03:58:49Z
author_association: OWNER
body:
  One quick fix could be to add a `extra_css_url` key to the `metadata.json` format (which currently hosts `title`, `license_url` etc) - if populated, we can inject a link to that stylesheet on every page. We could add a few classes in strategic places that include the database and table names to give people styling hooks.

  While we're at it, an `extra_js_url` key would let people go really nuts!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/124#issuecomment-347123991
issue_url: https://api.github.com/repos/simonw/datasette/issues/124
node_id: MDEyOklzc3VlQ29tbWVudDM0NzEyMzk5MQ==
created_at: 2017-11-27T09:25:15Z
updated_at: 2017-11-27T09:25:15Z
author_association: NONE
body:
  That's the only reference to immutable I saw as well, making me think that there may be no perceivable advantages over simply using mode=ro. Since the database is never or seldom updated the change notifications should not impact performance.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/124#issuecomment-347236102
issue_url: https://api.github.com/repos/simonw/datasette/issues/124
node_id: MDEyOklzc3VlQ29tbWVudDM0NzIzNjEwMg==
created_at: 2017-11-27T16:24:15Z
updated_at: 2017-11-27T16:24:15Z
author_association: OWNER
body:
  I'd really like to get some benchmarks working so I can see the actual impact of this kind of thing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/155#issuecomment-347713453
issue_url: https://api.github.com/repos/simonw/datasette/issues/155
node_id: MDEyOklzc3VlQ29tbWVudDM0NzcxMzQ1Mw==
created_at: 2017-11-29T00:41:30Z
updated_at: 2017-11-29T00:41:30Z
author_association: OWNER
body:
  Could you provide the SQL to create a reproducible test case (both CREATE TABLE and INSERT statements)?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/155#issuecomment-347714314
issue_url: https://api.github.com/repos/simonw/datasette/issues/155
node_id: MDEyOklzc3VlQ29tbWVudDM0NzcxNDMxNA==
created_at: 2017-11-29T00:46:25Z
updated_at: 2017-11-29T00:46:25Z
author_association: NONE
body:
  ```
  CREATE TABLE rhs (
    id INTEGER PRIMARY KEY,
    name TEXT
  );

  CREATE TABLE lhs (
    symbol INTEGER PRIMARY KEY,
    FOREIGN KEY (symbol) REFERENCES rhs(id)
  );

  INSERT INTO rhs VALUES (1, "foo");
  INSERT INTO rhs VALUES (2, "bar");
  INSERT INTO lhs VALUES (1);
  INSERT INTO lhs VALUES (2);
  ```

  It's expected that in lhs's view, foo / bar should be displayed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/155#issuecomment-347715452
issue_url: https://api.github.com/repos/simonw/datasette/issues/155
node_id: MDEyOklzc3VlQ29tbWVudDM0NzcxNTQ1Mg==
created_at: 2017-11-29T00:52:30Z
updated_at: 2017-11-29T00:52:30Z
author_association: OWNER
body:
  Interestingly, it almost does the right thing on the individual row page: https://bug-155-dkcqckhgki.now.sh/bug-155-9a7bb68/lhs/1

  <img width="303" alt="cursor_and_bug-155__lhs" src="https://user-images.githubusercontent.com/9599/33352129-511419e4-d45c-11e7-98f3-9c1dfefa6350.png">

  The symbol has been expanded, but there's a rogue '1' that shouldn't be there at all - I think that's bug #152

  The table view itself is definitely doing the wrong thing: https://bug-155-dkcqckhgki.now.sh/bug-155-9a7bb68/lhs

  <img width="546" alt="cursor_and_bug-155__lhs__2_rows_and_paradise_papers_tie_kushner_startup_to_russian_money_ _the_forward" src="https://user-images.githubusercontent.com/9599/33352178-8491b9a2-d45c-11e7-8d89-fedd455cfcba.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/153#issuecomment-347735334
issue_url: https://api.github.com/repos/simonw/datasette/issues/153
node_id: MDEyOklzc3VlQ29tbWVudDM0NzczNTMzNA==
created_at: 2017-11-29T02:45:03Z
updated_at: 2017-11-29T02:45:03Z
author_association: OWNER
body:
  @ftrain OK I've shipped the first version of this. Here's the initial documentation:

  Create a `metadata.json` file that looks like this:

      {
          "extra_css_urls": [
              "https://simonwillison.net/static/css/all.bf8cd891642c.css"
          ],
          "extra_js_urls": [
              "https://code.jquery.com/jquery-3.2.1.slim.min.js"
          ]
      }

  Then start datasette like this:

      datasette mydb.db --metadata=metadata.json

  The CSS and JavaScript files will be linked in the `<head>` of every page.

  You can also specify a SRI (subresource integrity hash) for these assets:

      {
          "extra_css_urls": [
              {
                  "url": "https://simonwillison.net/static/css/all.bf8cd891642c.css",
                  "sri": "sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI"
              }
          ],
          "extra_js_urls": [
              {
                  "url": "https://code.jquery.com/jquery-3.2.1.slim.min.js",
                  "sri": "sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g="
              }
          ]
      }

  Modern browsers will only execute the stylsheet or JavaScript if the SRI hash
  matches the content served. You can generate hashes using www.srihash.org

  This isn't shipped in a release yet, but you can still access these features in `datasette publish` like so:

      datasette publish now mydb.db --metadata=metadata.json --branch=master

  The `--branch=master` option will pull the latest master build of Datasette from GitHub.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/153#issuecomment-347735598
issue_url: https://api.github.com/repos/simonw/datasette/issues/153
node_id: MDEyOklzc3VlQ29tbWVudDM0NzczNTU5OA==
created_at: 2017-11-29T02:46:31Z
updated_at: 2017-11-29T02:47:27Z
author_association: OWNER
body:
  To style individual columns you'll currently need to use the `nth-of-type` selector, e.g.:

      td:nth-of-type(5):before { white-space: pre }

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/153#issuecomment-348103270
issue_url: https://api.github.com/repos/simonw/datasette/issues/153
node_id: MDEyOklzc3VlQ29tbWVudDM0ODEwMzI3MA==
created_at: 2017-11-30T07:16:40Z
updated_at: 2017-11-30T07:16:40Z
author_association: OWNER
body:
  Every template now gets CSS classes in the body designed to support custom
  styling.

  The index template (the top level page at /) gets this:

      <body class="index">

  The database template (/dbname/) gets this:

      <body class="db db-dbname">

  The table template (/dbname/tablename) gets:

      <body class="table db-dbname table-tablename">

  The row template (/dbname/tablename/rowid) gets:

      <body class="row db-dbname table-tablename">

  The db-x and table-x classes use the database or table names themselves IF
  they are valid CSS identifiers. If they aren't, we strip any invalid
  characters out and append a 6 character md5 digest of the original name, in
  order to ensure that multiple tables which resolve to the same stripped
  character version still have different CSS classes.

  Some examples (extracted from the unit tests):

      "simple" => "simple"
      "MixedCase" => "MixedCase"
      "-no-leading-hyphens" => "no-leading-hyphens-65bea6"
      "_no-leading-underscores" => "no-leading-underscores-b921bc"
      "no spaces" => "no-spaces-7088d7"
      "-" => "336d5e"
      "no $ characters" => "no--characters-59e024"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/12#issuecomment-348245757
issue_url: https://api.github.com/repos/simonw/datasette/issues/12
node_id: MDEyOklzc3VlQ29tbWVudDM0ODI0NTc1Nw==
created_at: 2017-11-30T16:39:45Z
updated_at: 2017-11-30T16:39:45Z
author_association: OWNER
body:
  It is now possible to over-ride templates on a per-database / per-row or per-
  table basis.

  When you access e.g. `/mydatabase/mytable` Datasette will look for the following:

      - table-mydatabase-mytable.html
      - table.html

  If you provided a `--template-dir` argument to datasette serve it will look in
  that directory first.

  The lookup rules are as follows:

      Index page (/):
          index.html

      Database page (/mydatabase):
          database-mydatabase.html
          database.html

      Table page (/mydatabase/mytable):
          table-mydatabase-mytable.html
          table.html

      Row page (/mydatabase/mytable/id):
          row-mydatabase-mytable.html
          row.html

  If a table name has spaces or other unexpected characters in it, the template
  filename will follow the same rules as our custom `<body>` CSS classes
  introduced in 8ab3a16 - for example, a table called "Food Trucks"
  will attempt to load the following templates:

      table-mydatabase-Food-Trucks-399138.html
      table.html

  It is possible to extend the default templates using Jinja template
  inheritance. If you want to customize EVERY row template with some additional
  content you can do so by creating a `row.html` template like this:

      {% extends "default:row.html" %}

      {% block content %}
      <h1>EXTRA HTML AT THE TOP OF THE CONTENT BLOCK</h1>
      <p>This line renders the original block:</p>
      {{ super() }}
      {% endblock %}

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/153#issuecomment-348252037
issue_url: https://api.github.com/repos/simonw/datasette/issues/153
node_id: MDEyOklzc3VlQ29tbWVudDM0ODI1MjAzNw==
created_at: 2017-11-30T16:59:00Z
updated_at: 2017-11-30T16:59:00Z
author_association: NONE
body:
  WOW!

  --
  Paul Ford // (646) 369-7128 // @ftrain

  On Thu, Nov 30, 2017 at 11:47 AM, Simon Willison <notifications@github.com>
  wrote:

  > Remaining work on this now lives in a milestone:
  > https://github.com/simonw/datasette/milestone/6
  >
  > —
  > You are receiving this because you were mentioned.
  > Reply to this email directly, view it on GitHub
  > <https://github.com/simonw/datasette/issues/153#issuecomment-348248406>,
  > or mute the thread
  > <https://github.com/notifications/unsubscribe-auth/AABPKHzaVPKwTOoHouK2aMUnM-mPnPk6ks5s7twzgaJpZM4Qq2zW>
  > .
  >

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/160#issuecomment-348404864
issue_url: https://api.github.com/repos/simonw/datasette/issues/160
node_id: MDEyOklzc3VlQ29tbWVudDM0ODQwNDg2NA==
created_at: 2017-12-01T05:26:57Z
updated_at: 2017-12-01T05:26:57Z
author_association: OWNER
body:
  Question is... what should happen to the default static stuff? At the moment that's just https://fivethirtyeight.datasettes.com/-/static/app.css - though I want to improve that to include a content hash, see #154 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/154#issuecomment-348404988
issue_url: https://api.github.com/repos/simonw/datasette/issues/154
node_id: MDEyOklzc3VlQ29tbWVudDM0ODQwNDk4OA==
created_at: 2017-12-01T05:27:40Z
updated_at: 2017-12-01T05:27:40Z
author_association: OWNER
body:
  If I do add additional static file bundling should that automatically get content hashes as well? #160 - problem with that is then I might have to parse the CSS files and rewrite their internal background-url references etc.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/20#issuecomment-348420129
issue_url: https://api.github.com/repos/simonw/datasette/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDM0ODQyMDEyOQ==
created_at: 2017-12-01T07:16:25Z
updated_at: 2017-12-01T07:16:25Z
author_association: OWNER
body:
  I've found some examples of canned queries I want to support that can't be represented as views, so I'm going to reopen this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/20#issuecomment-348420955
issue_url: https://api.github.com/repos/simonw/datasette/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDM0ODQyMDk1NQ==
created_at: 2017-12-01T07:21:08Z
updated_at: 2017-12-01T07:21:08Z
author_association: OWNER
body:
  I'll use the existing metadata.json file:

      {
          "databases": {
              "mydb": {
                  "queries": {
                       "custom_thingy": {...

  The query definition can either be just a string of SQL, or it can be an object with a sql key and optional title and description keys.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/160#issuecomment-348719680
issue_url: https://api.github.com/repos/simonw/datasette/issues/160
node_id: MDEyOklzc3VlQ29tbWVudDM0ODcxOTY4MA==
created_at: 2017-12-02T20:59:27Z
updated_at: 2017-12-02T20:59:27Z
author_association: OWNER
body:
  This is about more than just CSS and JavaScript - there are plenty of reasons someone might want to bundle HTML as well, e.g. for building something like https://sf-tree-search.now.sh/

  So, instead of thinking about this in terms of /static/, I'm going to think about this in terms of allowing people to mount one or more document roots (or docroots).

      datasette serve mydb.db -d my-doc-root/

  This will cause the root of the server to show content from the `my-doc-root/` directory (assuming it has an index.html file in it).

  A more common option will be to mount specific folders to specific directories, like this:

      datasette serve mydb.db -d static:my-static/

  Now any hits to `/static/foo.css` will serve content from `my-static/foo.css`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/160#issuecomment-348793054
issue_url: https://api.github.com/repos/simonw/datasette/issues/160
node_id: MDEyOklzc3VlQ29tbWVudDM0ODc5MzA1NA==
created_at: 2017-12-03T16:35:22Z
updated_at: 2017-12-03T16:35:22Z
author_association: OWNER
body:
  You can now tell Datasette to serve static files from a specific location at a
  specific mountpoint.

  For example:

  	datasette serve mydb.db --static extra-css:/tmp/static/css

  Now if you visit this URL:

  	http://localhost:8001/extra-css/blah.css

  The following file will be served:

  	/tmp/static/css/blah.css

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/20#issuecomment-349027974
issue_url: https://api.github.com/repos/simonw/datasette/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDM0OTAyNzk3NA==
created_at: 2017-12-04T17:01:19Z
updated_at: 2017-12-04T17:01:19Z
author_association: OWNER
body:
   This is also a good opportunity to re-factor out a separate query.html template - right now the database.html template is doing two jobs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/135#issuecomment-349047335
issue_url: https://api.github.com/repos/simonw/datasette/issues/135
node_id: MDEyOklzc3VlQ29tbWVudDM0OTA0NzMzNQ==
created_at: 2017-12-04T17:57:08Z
updated_at: 2017-12-04T17:57:08Z
author_association: OWNER
body:
  Turns out there's a bug in this: https://timezones-now-hrjgkinozh.now.sh/timezones-0d61a90/ElementaryGeometries should not be showing the search box.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/20#issuecomment-349359498
issue_url: https://api.github.com/repos/simonw/datasette/issues/20
node_id: MDEyOklzc3VlQ29tbWVudDM0OTM1OTQ5OA==
created_at: 2017-12-05T16:30:06Z
updated_at: 2017-12-05T16:30:06Z
author_association: OWNER
body:
  Named canned queries can now be defined in metadata.json like this:
      
          {
              "databases": {
                  "timezones": {
                      "queries": {
                          "timezone_for_point": "select tzid from timezones ..."
                      }
                  }
              }
          }
      
  These will be shown in a new "Queries" section beneath "Views" on the database page.

  <img width="528" alt="timezones" src="https://user-images.githubusercontent.com/9599/33618097-762ae188-d996-11e7-8f85-29f998ef7a18.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/122#issuecomment-349408214
issue_url: https://api.github.com/repos/simonw/datasette/issues/122
node_id: MDEyOklzc3VlQ29tbWVudDM0OTQwODIxNA==
created_at: 2017-12-05T19:08:04Z
updated_at: 2017-12-05T19:08:04Z
author_association: OWNER
body:
  I think `.json` should continue to return rows as list-of-lists - it's a nice default because it produces a smaller overall JSON file. Encouraging people to specify an alternative shape to get the current `.jsono` format feels appropriate.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/135#issuecomment-349860851
issue_url: https://api.github.com/repos/simonw/datasette/issues/135
node_id: MDEyOklzc3VlQ29tbWVudDM0OTg2MDg1MQ==
created_at: 2017-12-07T04:37:59Z
updated_at: 2017-12-07T04:37:59Z
author_association: OWNER
body:
  I'm testing this like so:

       datasette ~/Dropbox/Development/timezones-api/timezones.db --reload --load-extension /usr/local/lib/mod_spatialite.dylib 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/135#issuecomment-349861461
issue_url: https://api.github.com/repos/simonw/datasette/issues/135
node_id: MDEyOklzc3VlQ29tbWVudDM0OTg2MTQ2MQ==
created_at: 2017-12-07T04:43:12Z
updated_at: 2017-12-07T04:43:12Z
author_association: OWNER
body:
  This query looks like it does the right thing:

          select * from sqlite_master
              where rootpage = 0
              and (
                  sql like '%VIRTUAL TABLE%USING FTS%content="ElementaryGeometries"%'
                  or (
                    tbl_name = "ElementaryGeometries"
                    and sql like '%VIRTUAL TABLE%USING FTS%'
                  )
              ) 

  Against a table that should not be shown as FTS: https://timezones-now-hrjgkinozh.now.sh/timezones-0d61a90?sql=++++++++select+*+from+sqlite_master%0D%0A++++++++++++where+rootpage+%3D+0%0D%0A++++++++++++and+%28%0D%0A++++++++++++++++sql+like+%27%25VIRTUAL+TABLE%25USING+FTS%25content%3D%22ElementaryGeometries%22%25%27%0D%0A++++++++++++++++or+%28%0D%0A++++++++++++++++++tbl_name+%3D+%22ElementaryGeometries%22%0D%0A++++++++++++++++++and+sql+like+%27%25VIRTUAL+TABLE%25USING+FTS%25%27%0D%0A++++++++++++++++%29%0D%0A++++++++++++%29+

  Against a table that SHOULD match: https://sf-trees.now.sh/sf-trees-ebc2ad9?sql=++++++++select+*+from+sqlite_master%0D%0A++++++++++++where+rootpage+%3D+0%0D%0A++++++++++++and+%28%0D%0A++++++++++++++++sql+like+%27%25VIRTUAL+TABLE%25USING+FTS%25content%3D%22Street_Tree_List_fts%22%25%27%0D%0A++++++++++++++++or+%28%0D%0A++++++++++++++++++tbl_name+%3D+%22Street_Tree_List_fts%22%0D%0A++++++++++++++++++and+sql+like+%27%25VIRTUAL+TABLE%25USING+FTS%25%27%0D%0A++++++++++++++++%29%0D%0A++++++++++++%29+

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/158#issuecomment-349868849
issue_url: https://api.github.com/repos/simonw/datasette/issues/158
node_id: MDEyOklzc3VlQ29tbWVudDM0OTg2ODg0OQ==
created_at: 2017-12-07T05:41:08Z
updated_at: 2017-12-07T05:41:08Z
author_association: OWNER
body:
  I'm happy with this - we have extra_head, content, body_class and title blocks which should provide enough hooks for most reasonable customizations.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/153#issuecomment-349874052
issue_url: https://api.github.com/repos/simonw/datasette/issues/153
node_id: MDEyOklzc3VlQ29tbWVudDM0OTg3NDA1Mg==
created_at: 2017-12-07T06:17:33Z
updated_at: 2017-12-07T06:17:33Z
author_association: OWNER
body:
  In #159 I added a mechanism for easily customizing per-column displays, and I've added documentation showing an example of using this mechanism to set certain columns to display as unescaped HTML: http://datasette.readthedocs.io/en/latest/custom_templates.html#custom-templates

  This fixes item 3, so I'm closing this ticket!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/164#issuecomment-349874709
issue_url: https://api.github.com/repos/simonw/datasette/issues/164
node_id: MDEyOklzc3VlQ29tbWVudDM0OTg3NDcwOQ==
created_at: 2017-12-07T06:22:10Z
updated_at: 2017-12-07T06:22:10Z
author_association: OWNER
body:
  Example usage:

      datasette skeleton parlgov.db -m parlgov.json

  Generates a `parlgov.json` file containing this:

      {
          "title": null,
          "description": null,
          "description_html": null,
          "license": null,
          "license_url": null,
          "source": null,
          "source_url": null,
          "databases": {
              "parlgov": {
                  "title": null,
                  "description": null,
                  "description_html": null,
                  "license": null,
                  "license_url": null,
                  "source": null,
                  "source_url": null,
                  "queries": {},
                  "tables": {
                      "info_data_source": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "external_party_castles_mair": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "external_party_chess": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "external_party_huber_inglehart": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "info_table": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "external_party_euprofiler": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "party_family": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "info_id": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "sqlite_stat1": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "external_party_benoit_laver": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "external_country_iso": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "viewcalc_party_position": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "viewcalc_election_parameter": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "viewcalc_parliament_composition": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "viewcalc_country_year_share": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "election": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "politician_president": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "party_name_change": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "external_commissioner_doering": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "external_party_ray": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "party_change": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "cabinet_party": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "external_party_ees": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "party": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "external_party_cmp": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "country": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "cabinet": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "info_variable": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      },
                      "election_result": {
                          "title": null,
                          "description": null,
                          "description_html": null,
                          "license": null,
                          "license_url": null,
                          "source": null,
                          "source_url": null
                      }
                  }
              }
          }
      }

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/165#issuecomment-350026183
issue_url: https://api.github.com/repos/simonw/datasette/issues/165
node_id: MDEyOklzc3VlQ29tbWVudDM1MDAyNjE4Mw==
created_at: 2017-12-07T16:47:46Z
updated_at: 2017-12-07T16:47:46Z
author_association: OWNER
body:
  Here's an example metadata.json file illustrating custom per-database and per-
  table metadata:

      {
          "title": "Overall datasette title",
          "description_html": "This is a <em>description with HTML</em>.",
          "databases": {
              "db1": {
                  "title": "First database",
                  "description": "This is a string description & has no HTML",
                  "license_url": "http://example.com/",
          		"license": "The example license",
                  "queries": {
                  	"canned_query": "select * from table1 limit 3;"
                  },
                  "tables": {
                      "table1": {
                          "title": "Custom title for table1",
                          "description": "Tables can have descriptions too",
                          "source": "This has a custom source",
                          "source_url": "http://example.com/"
                      }
                  }
              }
          }
      }

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/161#issuecomment-350108113
issue_url: https://api.github.com/repos/simonw/datasette/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDM1MDEwODExMw==
created_at: 2017-12-07T22:02:24Z
updated_at: 2017-12-07T22:02:24Z
author_association: NONE
body:
  It's not throwing the validation error anymore, but i still cannot run following with query:
  ```
  WITH RECURSIVE cnt(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM cnt LIMIT 10) SELECT x FROM cnt;
  ```

  I got `near "WITH": syntax error`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/167#issuecomment-350125953
issue_url: https://api.github.com/repos/simonw/datasette/issues/167
node_id: MDEyOklzc3VlQ29tbWVudDM1MDEyNTk1Mw==
created_at: 2017-12-07T23:25:28Z
updated_at: 2017-12-07T23:25:28Z
author_association: OWNER
body:
  My column/row HTML display logic has got way too convoluted. This is a sign I need to add proper unit tests for it and clean it up.

  The complexity comes from:

  * Displaying a rowid for tables that do not have a primary key
  * Showing an additional Link column for rows with a primary key
  * Not displaying that Link column on the individual row pages
  * Trying to get foreign keys working correctly in all cases, e.g. #152 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/161#issuecomment-350158037
issue_url: https://api.github.com/repos/simonw/datasette/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDM1MDE1ODAzNw==
created_at: 2017-12-08T02:52:34Z
updated_at: 2017-12-08T02:52:34Z
author_association: OWNER
body:
  That might mean your version of SQLite doesn't support that syntax. Unfortunately the version bundled with Python is a bit old - the one built by the Dockerfile in this repo should handle it though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/141#issuecomment-350292364
issue_url: https://api.github.com/repos/simonw/datasette/issues/141
node_id: MDEyOklzc3VlQ29tbWVudDM1MDI5MjM2NA==
created_at: 2017-12-08T15:33:18Z
updated_at: 2017-12-08T15:33:18Z
author_association: OWNER
body:
  I can emulate this on OS X using a disk image (Disk Utility -> File -> New Image -> Blank Image...) - once mounted, I get the following:

      >>> os.link('/tmp/hello', '/Volumes/Untitled/hello')
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      OSError: [Errno 18] Cross-device link: '/tmp/hello' -> '/Volumes/Untitled/hello'

  I can simulate that in a mock like this:

  	>>> from unittest.mock import patch
  	>>> @patch('os.link')
  	... def test_link(mock_link):
  	...     mock_link.side_effect = OSError
  	...     mock_link()
  	... 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/141#issuecomment-350301248
issue_url: https://api.github.com/repos/simonw/datasette/issues/141
node_id: MDEyOklzc3VlQ29tbWVudDM1MDMwMTI0OA==
created_at: 2017-12-08T16:07:04Z
updated_at: 2017-12-08T16:07:04Z
author_association: OWNER
body:
  This fix should work, please have a go with latest master and let me know if you run into any problems.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/154#issuecomment-350302417
issue_url: https://api.github.com/repos/simonw/datasette/issues/154
node_id: MDEyOklzc3VlQ29tbWVudDM1MDMwMjQxNw==
created_at: 2017-12-08T16:11:24Z
updated_at: 2017-12-08T16:11:24Z
author_association: OWNER
body:
  I think I'll do this as a custom Jinja template filter. That way template authors can re-use it for their own static files if they want.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/167#issuecomment-350421661
issue_url: https://api.github.com/repos/simonw/datasette/issues/167
node_id: MDEyOklzc3VlQ29tbWVudDM1MDQyMTY2MQ==
created_at: 2017-12-09T03:52:46Z
updated_at: 2017-12-09T03:52:46Z
author_association: OWNER
body:
  Input: results from the database, foreign key definitions, primary key definitions, type of page

  Output: display_columns and display_rows

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/167#issuecomment-350424595
issue_url: https://api.github.com/repos/simonw/datasette/issues/167
node_id: MDEyOklzc3VlQ29tbWVudDM1MDQyNDU5NQ==
created_at: 2017-12-09T05:08:27Z
updated_at: 2017-12-09T05:08:27Z
author_association: OWNER
body:
  Perhaps the row.html and table.html templates should be passed the same data but should themselves decide if they will display the Link column 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/160#issuecomment-350496258
issue_url: https://api.github.com/repos/simonw/datasette/issues/160
node_id: MDEyOklzc3VlQ29tbWVudDM1MDQ5NjI1OA==
created_at: 2017-12-09T18:29:28Z
updated_at: 2017-12-09T18:29:28Z
author_association: OWNER
body:
  Example usage:

      datasette package --static css:extra-css/ --static js:extra-js/ \
      	sf-trees.db --template-dir templates/ --tag sf-trees --branch master

  This creates a local Docker image that includes copies of the templates/,
  extra-css/ and extra-js/ directories. You can then run it like this:

  	docker run -p 8001:8001 sf-trees

  For publishing to Zeit now:

  	datasette publish now --static css:extra-css/ --static js:extra-js/ \
  		sf-trees.db --template-dir templates/ --name sf-trees --branch master

  Example: https://sf-trees-wbihszoazc.now.sh/sf-trees-02c8ef1/Street_Tree_List

  For publishing to Heroku:

  	datasette publish heroku --static css:extra-css/ --static js:extra-js/ \
  		sf-trees.db --template-dir templates/ --branch master

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/170#issuecomment-350506593
issue_url: https://api.github.com/repos/simonw/datasette/issues/170
node_id: MDEyOklzc3VlQ29tbWVudDM1MDUwNjU5Mw==
created_at: 2017-12-09T21:25:50Z
updated_at: 2017-12-09T21:25:50Z
author_association: OWNER
body:
  Turns out this is already supported: https://github.com/simonw/datasette/blob/6bdfcf60760c27e29ff34692d06e62b36aeecc56/datasette/app.py#L307

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/170#issuecomment-350506751
issue_url: https://api.github.com/repos/simonw/datasette/issues/170
node_id: MDEyOklzc3VlQ29tbWVudDM1MDUwNjc1MQ==
created_at: 2017-12-09T21:28:32Z
updated_at: 2017-12-09T21:28:32Z
author_association: OWNER
body:
  My mistake, that's using the database name - there isn't a way of customizing for a specific named query yet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/170#issuecomment-350507155
issue_url: https://api.github.com/repos/simonw/datasette/issues/170
node_id: MDEyOklzc3VlQ29tbWVudDM1MDUwNzE1NQ==
created_at: 2017-12-09T21:35:30Z
updated_at: 2017-12-09T21:35:30Z
author_association: OWNER
body:
      Canned query page (/mydatabase/canned-query):
          query-mydatabase-canned-query.html
          query-mydatabase.html
          query.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/171#issuecomment-350508049
issue_url: https://api.github.com/repos/simonw/datasette/issues/171
node_id: MDEyOklzc3VlQ29tbWVudDM1MDUwODA0OQ==
created_at: 2017-12-09T21:50:50Z
updated_at: 2017-12-09T21:50:50Z
author_association: OWNER
body:
  Quoting the new documentation:

  You can find out which templates were considered for a specific page by viewing
  source on that page and looking for an HTML comment at the bottom. The comment
  will look something like this:

      <!-- Templates considered: *query-mydb-tz.html, query-mydb.html, query.html -->

  This example is from the canned query page for a query called "tz" in the
  database called "mydb". The asterisk shows which template was selected - so in
  this case, Datasette found a template file called `query-mydb-tz.html` and
  used that - but if that template had not been found, it would have tried for
  `query-mydb.html` or the default `query.html`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/167#issuecomment-350515616
issue_url: https://api.github.com/repos/simonw/datasette/issues/167
node_id: MDEyOklzc3VlQ29tbWVudDM1MDUxNTYxNg==
created_at: 2017-12-10T00:21:58Z
updated_at: 2017-12-10T00:21:58Z
author_association: OWNER
body:
  This function signature is pretty gross: https://github.com/simonw/datasette/blob/7a7e4b2ed8c76c6d002a9d707dbc840f6a2abf7f/datasette/app.py#L418

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/167#issuecomment-350515985
issue_url: https://api.github.com/repos/simonw/datasette/issues/167
node_id: MDEyOklzc3VlQ29tbWVudDM1MDUxNTk4NQ==
created_at: 2017-12-10T00:28:39Z
updated_at: 2017-12-10T00:28:39Z
author_association: OWNER
body:
  A better alternative:

  ```async def display_columns_and_rows(self, database, table, rows, link_column=False):```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/167#issuecomment-350516782
issue_url: https://api.github.com/repos/simonw/datasette/issues/167
node_id: MDEyOklzc3VlQ29tbWVudDM1MDUxNjc4Mg==
created_at: 2017-12-10T00:48:54Z
updated_at: 2017-12-10T00:48:54Z
author_association: OWNER
body:
  I can simplify this all by dropping the nicety where if a table is using a rowid the Link column is titled rowid instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/153#issuecomment-350519736
issue_url: https://api.github.com/repos/simonw/datasette/issues/153
node_id: MDEyOklzc3VlQ29tbWVudDM1MDUxOTczNg==
created_at: 2017-12-10T02:06:01Z
updated_at: 2017-12-10T02:06:01Z
author_association: OWNER
body:
  @ftrain Datasette 0.14 is now released with all of the above: https://github.com/simonw/datasette/releases/tag/0.14

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/153#issuecomment-350519821
issue_url: https://api.github.com/repos/simonw/datasette/issues/153
node_id: MDEyOklzc3VlQ29tbWVudDM1MDUxOTgyMQ==
created_at: 2017-12-10T02:08:45Z
updated_at: 2017-12-10T02:08:45Z
author_association: OWNER
body:
  Also worth mentioning: as of #160 and #157 the `datasette publish now`, `datasette publish heroku` and `datasette package` commands all know how to bundle up any `--static` or `--template-dir` content and include it in the Docker image / Heroku/Now deployment that gets generated.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/91#issuecomment-350521780
issue_url: https://api.github.com/repos/simonw/datasette/issues/91
node_id: MDEyOklzc3VlQ29tbWVudDM1MDUyMTc4MA==
created_at: 2017-12-10T03:07:53Z
updated_at: 2017-12-10T03:07:53Z
author_association: OWNER
body:
  Won't fix - I think the custom templates and static stuff in https://github.com/simonw/datasette/releases/tag/0.14 renders this obsolete.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/138#issuecomment-350521806
issue_url: https://api.github.com/repos/simonw/datasette/issues/138
node_id: MDEyOklzc3VlQ29tbWVudDM1MDUyMTgwNg==
created_at: 2017-12-10T03:08:26Z
updated_at: 2017-12-10T03:08:36Z
author_association: OWNER
body:
  Implemented this in 80bf3afa43e3cb396c7a7c9b168eedbc6fe0fa15 and #165. Didn't use data package though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/121#issuecomment-350527283
issue_url: https://api.github.com/repos/simonw/datasette/issues/121
node_id: MDEyOklzc3VlQ29tbWVudDM1MDUyNzI4Mw==
created_at: 2017-12-10T06:00:47Z
updated_at: 2017-12-10T06:00:47Z
author_association: OWNER
body:
  This is also really interesting when combined with the spatialite AsGeoJSON function: http://www.gaia-gis.it/gaia-sins/spatialite-sql-4.2.0.html#p3misc

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/120#issuecomment-355487646
issue_url: https://api.github.com/repos/simonw/datasette/issues/120
node_id: MDEyOklzc3VlQ29tbWVudDM1NTQ4NzY0Ng==
created_at: 2018-01-05T07:10:12Z
updated_at: 2018-01-05T07:10:12Z
author_association: NONE
body:
  Ah, glad I found this issue. I have private data that I'd like to share to a few different people. Personally, a shared username and password would be sufficient for me, more-or-less Basic Auth. Do you have more complex requirements in mind?

  I'm not sure if "plugin" means "build a plugin" or "find a plugin" or something else entirely. FWIW, I stumbled upon [sanic-auth](https://github.com/pyx/sanic-auth) which looks like a new project to bring some interfaces around auth to sanic, similar to Flask.

  Alternatively, it shouldn't be too bad to add in Basic Auth. If we went down that route, that would probably be best built as a separate package for sanic that `datasette` brings in.

  What are your thoughts around this?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/176#issuecomment-356115657
issue_url: https://api.github.com/repos/simonw/datasette/issues/176
node_id: MDEyOklzc3VlQ29tbWVudDM1NjExNTY1Nw==
created_at: 2018-01-08T22:22:32Z
updated_at: 2018-01-08T22:22:32Z
author_association: NONE
body:
  This project probably would not be the place for that. This is a layer for sqllite specifically. It solves a similar problem as graphql, so adding that here wouldn't make sense.

  Here's an example i found from google that uses micro to run a graphql microservice. you'd just then need to connect your db.
  https://github.com/timneutkens/micro-graphql

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/176#issuecomment-356161672
issue_url: https://api.github.com/repos/simonw/datasette/issues/176
node_id: MDEyOklzc3VlQ29tbWVudDM1NjE2MTY3Mg==
created_at: 2018-01-09T02:35:35Z
updated_at: 2018-01-09T02:35:35Z
author_association: NONE
body:
  @wulfmann I think I disagree, except I'm not entirely sure what you mean by that first paragraph. The JSON API that Datasette currently exposes is quite different to GraphQL.

  Furthermore, there's no "just" about connecting micro-graphql to a DB; at least, no more "just" than adding any other API. You still need to configure the schema, which is exactly the kind of thing that Datasette does for JSON API. This is why I think that GraphQL's a good fit here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/176#issuecomment-356175667
issue_url: https://api.github.com/repos/simonw/datasette/issues/176
node_id: MDEyOklzc3VlQ29tbWVudDM1NjE3NTY2Nw==
created_at: 2018-01-09T04:19:03Z
updated_at: 2018-01-09T04:19:03Z
author_association: NONE
body:
  @yozlet Yes I think that I was confused when I posted my original comment. I see your main point now and am in agreement.

reactions:
  {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 2, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/179#issuecomment-360535979
issue_url: https://api.github.com/repos/simonw/datasette/issues/179
node_id: MDEyOklzc3VlQ29tbWVudDM2MDUzNTk3OQ==
created_at: 2018-01-25T17:18:24Z
updated_at: 2018-01-25T17:18:24Z
author_association: CONTRIBUTOR
body:
  To summarise that thread:

  -  expose full `metadata.json` object to the index page template, eg to allow tables to be referred to by name;
  - ability to import multiple `metadata.json` files, eg to allow metadata files created for a specific SQLite db to be reused in a datasette referring to several database files;

  It could also be useful to allow users to import a python file containing custom functions that can that be loaded into scope and made available to custom templates.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/185#issuecomment-370273359
issue_url: https://api.github.com/repos/simonw/datasette/issues/185
node_id: MDEyOklzc3VlQ29tbWVudDM3MDI3MzM1OQ==
created_at: 2018-03-04T23:10:56Z
updated_at: 2018-03-04T23:10:56Z
author_association: OWNER
body:
  Are you talking specifically about accessing metadata from HTML templates? That makes a lot of sense, I'll think about how this could work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/185#issuecomment-370461231
issue_url: https://api.github.com/repos/simonw/datasette/issues/185
node_id: MDEyOklzc3VlQ29tbWVudDM3MDQ2MTIzMQ==
created_at: 2018-03-05T15:43:56Z
updated_at: 2018-03-05T15:44:27Z
author_association: NONE
body:
  Yes. I think the simplest implementation is to change lines like

  ```python
          metadata = self.ds.metadata.get('databases', {}).get(name, {})
  ```

  to

  ```python
  metadata = {
      **self.ds.metadata,
      **self.ds.metadata.get('databases', {}).get(name, {}),
  }
  ```

  so that specified inner values overwrite outer values, but only if they exist.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/186#issuecomment-374810115
issue_url: https://api.github.com/repos/simonw/datasette/issues/186
node_id: MDEyOklzc3VlQ29tbWVudDM3NDgxMDExNQ==
created_at: 2018-03-21T01:21:13Z
updated_at: 2018-03-21T01:21:13Z
author_association: OWNER
body:
  Hah, this is exactly the opposite of datasette's default approach to caching, which is to cache everything for as long as possible.

  I don't think we'll need to add `Cache-Control: no-cache` headers provided we instead set it up so you can turn off Datasette's caching.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/186#issuecomment-374811114
issue_url: https://api.github.com/repos/simonw/datasette/issues/186
node_id: MDEyOklzc3VlQ29tbWVudDM3NDgxMTExNA==
created_at: 2018-03-21T01:28:30Z
updated_at: 2018-03-21T01:28:30Z
author_association: OWNER
body:
  We actually have this already:

  https://github.com/simonw/datasette/blob/012fc7c5cd3e9160c9a4c19cc964253e97fb054a/datasette/cli.py#L253-L255

  You can disable the cache headers using the `datasette --debug` option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/185#issuecomment-376585911
issue_url: https://api.github.com/repos/simonw/datasette/issues/185
node_id: MDEyOklzc3VlQ29tbWVudDM3NjU4NTkxMQ==
created_at: 2018-03-27T16:19:43Z
updated_at: 2018-03-27T16:19:43Z
author_association: OWNER
body:
  OK, I have an implementation of this. I realised that not ALL metadata should be inherited: it makes sense for source/source_url/license/license_url to be inherited, but it doesn't make sense for the title and description to be inherited down to the individual databases and tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/185#issuecomment-376587017
issue_url: https://api.github.com/repos/simonw/datasette/issues/185
node_id: MDEyOklzc3VlQ29tbWVudDM3NjU4NzAxNw==
created_at: 2018-03-27T16:22:59Z
updated_at: 2018-03-27T16:22:59Z
author_association: OWNER
body:
  One thing that's missing from this: if you set source/license data at the individual database level they should be inherited by tables within that database.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/185#issuecomment-376589591
issue_url: https://api.github.com/repos/simonw/datasette/issues/185
node_id: MDEyOklzc3VlQ29tbWVudDM3NjU4OTU5MQ==
created_at: 2018-03-27T16:30:51Z
updated_at: 2018-03-27T16:30:51Z
author_association: OWNER
body:
  Also needed: the ability to unset metadata. If the root metadata specifies a license_url it should be possible to set "license_url": null on a child database or table. The current implementation will ignore null (or empty string) values and default to the top level value.

  I think the templates themselves should be able to indicate if they want the inherited values or not. That way we could support arbitrary key/values and avoid the application code having special knowledge of license_url etc.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/185#issuecomment-376590265
issue_url: https://api.github.com/repos/simonw/datasette/issues/185
node_id: MDEyOklzc3VlQ29tbWVudDM3NjU5MDI2NQ==
created_at: 2018-03-27T16:32:51Z
updated_at: 2018-03-27T16:32:51Z
author_association: NONE
body:
  >I think the templates themselves should be able to indicate if they want the inherited values or not. That way we could support arbitrary key/values and avoid the application code having special knowledge of license_url etc.

  Yes, you could have `metadata` that works like `metadata` does currently and `inherited_metadata` that works with inheritance.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/185#issuecomment-376592044
issue_url: https://api.github.com/repos/simonw/datasette/issues/185
node_id: MDEyOklzc3VlQ29tbWVudDM3NjU5MjA0NA==
created_at: 2018-03-27T16:38:23Z
updated_at: 2018-03-27T16:38:23Z
author_association: NONE
body:
  It would be nice to also allow arbitrary keys (maybe under a parent key called params or something to prevent conflicts). For our datasette project, we just have a bunch of dictionaries defined in the base template for things like site URL and column humanized names:  https://github.com/baltimore-sun-data/salaries-datasette/blob/master/templates/base.html It would be cleaner if this were in the metadata.json.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/188#issuecomment-376594727
issue_url: https://api.github.com/repos/simonw/datasette/issues/188
node_id: MDEyOklzc3VlQ29tbWVudDM3NjU5NDcyNw==
created_at: 2018-03-27T16:46:49Z
updated_at: 2018-05-28T21:34:34Z
author_association: OWNER
body:
  One point of complexity: datasette can be used to bundle multiple .db files into a single "app".

  I think that's OK. We could require that the `datasette_files` table is present in the first database file passed on the command-line. Or we could even construct a search path and consult multiple versions of the table spread across multiple files.

  That said... any configuration that corresponds to a specific table should live in the same database file as that table.

  Ditto for general metadata: if we have license/source information for a specific table or database that information should be able to live in the same .db file as the data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/185#issuecomment-376604558
issue_url: https://api.github.com/repos/simonw/datasette/issues/185
node_id: MDEyOklzc3VlQ29tbWVudDM3NjYwNDU1OA==
created_at: 2018-03-27T17:16:27Z
updated_at: 2018-03-27T17:16:27Z
author_association: OWNER
body:
  I am SO inspired by what you've done with https://salaries.news.baltimoresun.com/ - that's pretty much my ideal use-case for Datasette, and it's by far the most elaborate customization I've seen so far. I'd love to hear other ideas that came up while building that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/185#issuecomment-376614973
issue_url: https://api.github.com/repos/simonw/datasette/issues/185
node_id: MDEyOklzc3VlQ29tbWVudDM3NjYxNDk3Mw==
created_at: 2018-03-27T17:49:00Z
updated_at: 2018-03-27T17:49:00Z
author_association: NONE
body:
  @simonw Other than metadata, the biggest item on wishlist for the salaries project was the ability to reorder by column. Of course, that could be done with a custom SQL query, but we didn't want to have to reimplement all the nav/pagination stuff from scratch. 

  @carolinp, feel free to add your thoughts.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-376981291
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3Njk4MTI5MQ==
created_at: 2018-03-28T18:06:08Z
updated_at: 2018-03-28T18:06:08Z
author_association: OWNER
body:
  Given how unlikely it is that this will pose a real problem I think I like option 1: enable sort-by-column by default for all tables, then allow power users to instead switch to explicit enabling of the functionality in their `metadata.json` if they know their data is too big.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-376983741
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3Njk4Mzc0MQ==
created_at: 2018-03-28T18:12:35Z
updated_at: 2018-03-28T18:12:35Z
author_association: OWNER
body:
  I think this can work with a `?_sort=xxx` parameter - and `?_sort=-xxx` to sort in the opposite direction.

  I'd like to support "sort by X descending, then by Y ascending if there are dupes for X" as well. Two ways that could work:

  `?_sort=-xxx,yyy`

  Or...

  `?_sort=-xxx&_sort=yyy`

  The second option is probably better in that it makes it easier for columns to have a comma in their name.

  Is it possible for a SQLite column to start with a `-` character?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-376986668
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3Njk4NjY2OA==
created_at: 2018-03-28T18:21:53Z
updated_at: 2018-03-28T18:21:53Z
author_association: OWNER
body:
  Might have to do something special to get sort-by-nulls-last: https://stackoverflow.com/questions/12503120/how-to-do-nulls-last-in-sqlite

      order by ifnull(column_name, -999999)

  Would need to figure out a smart way to get the default value - maybe by running a min() or max() against the column first?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-377049625
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3NzA0OTYyNQ==
created_at: 2018-03-28T21:52:05Z
updated_at: 2018-03-28T21:52:05Z
author_association: OWNER
body:
  This is a better pattern as you don't have to pick a minimum value:

      ORDER BY CASE WHEN SOMECOL IS NULL THEN 1 ELSE 0 END, SOMECOL

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-377050461
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3NzA1MDQ2MQ==
created_at: 2018-03-28T21:55:14Z
updated_at: 2018-03-28T22:06:30Z
author_association: OWNER
body:
  I think there are actually four kinds of sort order we need to support;

  * ascending
  * descending
  * ascending, nulls last
  * descending, nulls last

  It looks like [-blah] is a valid SQLite table name, so mark I descending with a hyphen prefix isn't good.

  Instead, maybe this:

      ?_sort_asc=col1&_sort_desc_nulls_last=col2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-377051018
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3NzA1MTAxOA==
created_at: 2018-03-28T21:57:20Z
updated_at: 2018-03-28T22:00:17Z
author_association: OWNER
body:
  I'd like to continue to support _next=token pagination even for custom sort orders.

  To do that I should include rowid (or general primary key) as the tie breaker on all sorts so I can incorporate that it into the _next= token.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-377052634
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3NzA1MjYzNA==
created_at: 2018-03-28T22:03:16Z
updated_at: 2018-03-28T22:03:16Z
author_association: OWNER
body:
  In terms of user interface: the obvious place to put this is as a drop down menu on the column headers.

  This also means the UI can support combined sort orders. Assuming you are already sorted by county descending and you select the candidate column header, the options could be:

  * sort all by candidate
  * sort all by candidate, descending
  * sort by county descending, then by candidate
  * sort by county descending, then by candidate descending

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-377054358
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3NzA1NDM1OA==
created_at: 2018-03-28T22:09:25Z
updated_at: 2018-03-28T22:09:25Z
author_association: OWNER
body:
  I'm tempted to put these verbose sorting options inline in the page HTML but have them in the table footer so they don't clog up the top half of the page with uninteresting links - then use JavaScript to hoik them out into a dropdown menu attached to each column header.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-377055663
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3NzA1NTY2Mw==
created_at: 2018-03-28T22:14:53Z
updated_at: 2018-03-28T22:14:53Z
author_association: OWNER
body:
  There is one other interesting option for auto-enabling/disabling sort: the inspect command could include data about column index presence and whether or not a column has any null values in it.

  This would allow us to dynamically include a "nulls last" option but only for columns that contain at least one null.

  It's quite a lot of additional engineering for a very minor feature though, so I think I'll punt on that for the moment.

  We may find that the _group_count feature can benefit from column value statistics later on though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/190#issuecomment-377065541
issue_url: https://api.github.com/repos/simonw/datasette/issues/190
node_id: MDEyOklzc3VlQ29tbWVudDM3NzA2NTU0MQ==
created_at: 2018-03-28T22:58:52Z
updated_at: 2018-03-28T22:58:52Z
author_association: OWNER
body:
  This is because the SQL we are using here is:

      select * from compound_primary_key where "pk1" > "d" and "pk2" > "v" order by pk1, pk2 limit 101

  This is incorrect. The correct SQL syntax (according to the example on https://www.sqlite.org/rowvalue.html#scrolling_window_queries ) is:

      select * from compound_primary_key where ("pk1", "pk2") > ("d", "v") order by pk1, pk2 limit 101

  BUT... this uses "row values" syntax which was only added to SQLite in version 3.15.0 in October 2016: https://sqlite.org/changes.html#version_3_15_0

  The version on https://datasette-issue-190-compound-pks.now.sh/compound-pks-9aafe8f?sql=select+sqlite_version%28%29%3B is 3.8.7.1 from October 2014.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/190#issuecomment-377066466
issue_url: https://api.github.com/repos/simonw/datasette/issues/190
node_id: MDEyOklzc3VlQ29tbWVudDM3NzA2NjQ2Ng==
created_at: 2018-03-28T23:03:45Z
updated_at: 2018-03-28T23:03:57Z
author_association: OWNER
body:
  Without row values syntax, the necessary SQL to retrieve the next page after `d, v` gets a bit gnarly:

      select * from compound_primary_key
      where pk1 >= "d" and not (pk1 = "d" and pk2 <= "v")
      order by pk1, pk2

  See https://datasette-issue-190-compound-pks.now.sh/compound-pks-9aafe8f?sql=select+*+from+compound_primary_key+where+pk1+%3E%3D+%22d%22+and+not+%28pk1+%3D+%22d%22+and+pk2+%3C%3D+%22v%22%29+order+by+pk1%2C+pk2

  This article was useful for figuring this out: https://use-the-index-luke.com/sql/partial-results/fetch-next-page

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/190#issuecomment-377067541
issue_url: https://api.github.com/repos/simonw/datasette/issues/190
node_id: MDEyOklzc3VlQ29tbWVudDM3NzA2NzU0MQ==
created_at: 2018-03-28T23:09:18Z
updated_at: 2018-03-28T23:09:51Z
author_association: OWNER
body:
  Here's how I generated the table for testing this with 3 compound primary keys:

      CREATE_SQL = '''
      CREATE TABLE compound_three_primary_keys (
          pk1 varchar(30),
          pk2 varchar(30),
          pk3 varchar(30),
          content text,
          PRIMARY KEY (pk1, pk2, pk3)
      );'''
      alphabet = 'abcdefghijklmnopqrstuvwxyz'
      for a in alphabet:
          for b in alphabet:
              for c in alphabet:
                  print('''
      INSERT INTO compound_three_primary_keys VALUES ('{}', '{}', '{}', '{}');
              '''.strip().format(a, b, c, '{}-{}-{}-{}-{}-{}'.format(a,b,c,a,b,c)))

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/190#issuecomment-377072022
issue_url: https://api.github.com/repos/simonw/datasette/issues/190
node_id: MDEyOklzc3VlQ29tbWVudDM3NzA3MjAyMg==
created_at: 2018-03-28T23:32:24Z
updated_at: 2018-03-28T23:32:24Z
author_association: OWNER
body:
  Here's the SQL for a next page with three compound primary keys:

  https://datasette-issue-190-compound-pks.now.sh/compound-pks-8e99805?sql=select+*+from+compound_three_primary_keys%0D%0Awhere%0D%0A++%28pk1+%3E+%3Apk1%29%0D%0A++++or%0D%0A++%28pk1+%3D+%3Apk1+and+pk2+%3E+%3Apk2%29%0D%0A++++or%0D%0A++%28pk1+%3D+%3Apk1+and+pk2+%3D+%3Apk2+and+pk3+%3E+%3Apk3%29%0D%0Aorder+by+pk1%2C+pk2%2C+pk3%3B%0D%0A%0D%0A%0D%0A&pk1=a&pk2=d&pk3=v

  ```
  select * from compound_three_primary_keys
  where
    (pk1 > :pk1)
      or
    (pk1 = :pk1 and pk2 > :pk2)
      or
    (pk1 = :pk1 and pk2 = :pk2 and pk3 > :pk3)
  order by pk1, pk2, pk3;
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-377362466
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3NzM2MjQ2Ng==
created_at: 2018-03-29T20:29:14Z
updated_at: 2018-03-29T20:29:14Z
author_association: OWNER
body:
  Alternative idea: by default enable all sorting in the UI.

  If a table has more than 100,000 rows disable sorting UI except for columns that have an index.

  Allow this to be overridden in metadata.json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/190#issuecomment-377454591
issue_url: https://api.github.com/repos/simonw/datasette/issues/190
node_id: MDEyOklzc3VlQ29tbWVudDM3NzQ1NDU5MQ==
created_at: 2018-03-30T06:11:59Z
updated_at: 2018-03-30T06:11:59Z
author_association: OWNER
body:
  Re-opening this issue: my fix doesn't play nicely with extra filter arguments.

  Consider this page: https://datasette-issue-190-compound-pks-not-quite-fixed.now.sh/compound-pks-8e99805/compound_three_primary_keys?content__contains=d

  The next link is to `?_next=f%2Cz%2Ct&content__contains=z` (that's next of `f,z,t`) but that gives us https://datasette-issue-190-compound-pks-not-quite-fixed.now.sh/compound-pks-8e99805/compound_three_primary_keys?_next=b%2Cx%2Cd&content__contains=d which shows `a,a,d` at the top.

  Sure enough, the generated SQL looks like this: https://datasette-issue-190-compound-pks-not-quite-fixed.now.sh/compound-pks-8e99805?sql=select+%2A+from+compound_three_primary_keys+where+%22content%22+like+%3Ap0+and+%28%5Bpk1%5D+%3E+%3Ap0%29%0A++or%0A%28%5Bpk1%5D+%3D+%3Ap0+and+%5Bpk2%5D+%3E+%3Ap1%29%0A++or%0A%28%5Bpk1%5D+%3D+%3Ap0+and+%5Bpk2%5D+%3D+%3Ap1+and+%5Bpk3%5D+%3E+%3Ap2%29+order+by+pk1%2C+pk2%2C+pk3+limit+101&p0=%25d%25&p1=b&p2=x&p3=d

  	select * from compound_three_primary_keys where "content" like :p0 and ([pk1] > :p0)
  	  or
  	([pk1] = :p0 and [pk2] > :p1)
  	  or
  	([pk1] = :p0 and [pk2] = :p1 and [pk3] > :p2) order by pk1, pk2, pk3 limit 101

  The parameters here are confused. The :p0 should be reserved just for the like clause - the other parameters should be p1, p2 and p3 (not p0, p1 and p2).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/190#issuecomment-377457087
issue_url: https://api.github.com/repos/simonw/datasette/issues/190
node_id: MDEyOklzc3VlQ29tbWVudDM3NzQ1NzA4Nw==
created_at: 2018-03-30T06:30:23Z
updated_at: 2018-03-30T06:30:23Z
author_association: OWNER
body:
  Interestingly, in deploying a copy of the database to demonstrate this final bug fix I had to use the `--force` argument like so:

      datasette publish now --branch=master compound-pks.db --force

  This is because `now` had already deployed a Dockerfile referencing `--branch=master` once already, so it thought nothing had changed and it could re-use that last deployment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/190#issuecomment-377457214
issue_url: https://api.github.com/repos/simonw/datasette/issues/190
node_id: MDEyOklzc3VlQ29tbWVudDM3NzQ1NzIxNA==
created_at: 2018-03-30T06:31:15Z
updated_at: 2018-03-30T06:31:15Z
author_association: OWNER
body:
  Fixed! https://datasette-issue-190-compound-pks-second-fix.now.sh/compound-pks-8e99805/compound_three_primary_keys?_next=b%2Cx%2Cd&content__contains=d now correctly shows `b,y,d` as the first row on the page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-377459579
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3NzQ1OTU3OQ==
created_at: 2018-03-30T06:47:52Z
updated_at: 2018-03-30T06:47:52Z
author_association: OWNER
body:
  I'm not entirely sure how to get `_next=` pagination working against sorted collections when a tie-breaker is needed.

  Consider this data:

  https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9?sql=select+rowid%2C+*+from+%5Bnfl-wide-receivers%2Fadvanced-historical%5D%0D%0Aorder+by+case+when+career_ranypa+is+null+then+1+else+0+end%2C+career_ranypa%2C+rowid+limit+11

  ![2018-03-29 at 11 46 pm](https://user-images.githubusercontent.com/9599/38127549-790c8bd0-33ab-11e8-8d32-66f5d3847c8a.png)

  If the page size was set to 9 rather than 11, the page divide would be between those two rows with the same value in the `career_ranypa` column. What would the `?_next=` token look like such that the correct row would be returned?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-377460127
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3NzQ2MDEyNw==
created_at: 2018-03-30T06:51:29Z
updated_at: 2018-03-30T06:51:52Z
author_association: OWNER
body:
  The problem is that our `_next=` pagination currently works based on a `>` - but for this case a `>=` for the value is needed combined with a `>` on the tie-breaker (which would be the `rowid` column).

  So I think this is the right SQL:

  ```
  select rowid, * from [nfl-wide-receivers/advanced-historical]
  where career_ranypa >= -6.331167749 and rowid > 2736
  order by case when career_ranypa is null then 1 else 0 end, career_ranypa, rowid limit 11
  ```

  https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9?sql=select+rowid%2C+*+from+%5Bnfl-wide-receivers%2Fadvanced-historical%5D%0D%0Awhere+career_ranypa+%3E%3D+-6.331167749+and+rowid+%3E+2736%0D%0Aorder+by+case+when+career_ranypa+is+null+then+1+else+0+end%2C+career_ranypa%2C+rowid+limit+11

  But how do I encode a `_next` token that means ">= X and > Y"?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-377462334
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3NzQ2MjMzNA==
created_at: 2018-03-30T07:06:21Z
updated_at: 2018-03-30T07:06:21Z
author_association: OWNER
body:
  Maybe the answer here is that anything that's encoded in the next token is treated as >= with the exception of columns known to be primary keys, which are treated as >

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-377546510
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3NzU0NjUxMA==
created_at: 2018-03-30T15:13:11Z
updated_at: 2018-03-30T15:13:11Z
author_association: OWNER
body:
  Pushed some work-in-progress with failing unit tests here: https://github.com/simonw/datasette/commit/2f8359c6f25768805431c80c74e5ec4213c2b2a6

  Here's a demo: https://datasette-column-sort-wip.now.sh/sortable-4bbaa6f/sortable?_sort=sortable - note that the `_sort_desc` and `_sort_nulls_last` options aren't done yet, plus it doesn't correctly paginate (the `_next` tokens do not yet take sorting into account).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-377547265
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3NzU0NzI2NQ==
created_at: 2018-03-30T15:16:43Z
updated_at: 2018-03-30T15:16:43Z
author_association: OWNER
body:
  I think this is the right incantation for a "next" link: https://datasette-column-sort-wip.now.sh/sortable-4bbaa6f?sql=select+*+from+sortable%0D%0Awhere+sortable+%3C%3D+94%0D%0Aand+%28%0D%0A++%28pk1+%3E+%27d%27%29%0D%0A++or%0D%0A++%28pk1+%3D+%27d%27+and+pk2+%3E+%27w%27%29%0D%0A%29%0D%0Aorder+by+sortable+desc%2C+pk1%2C+pk2%0D%0Alimit+7

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/122#issuecomment-378279612
issue_url: https://api.github.com/repos/simonw/datasette/issues/122
node_id: MDEyOklzc3VlQ29tbWVudDM3ODI3OTYxMg==
created_at: 2018-04-03T14:55:54Z
updated_at: 2018-04-03T14:55:54Z
author_association: OWNER
body:
  The new documentation for the `_shape=` parameter is now live at http://datasette.readthedocs.io/en/latest/json_api.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/183#issuecomment-378281740
issue_url: https://api.github.com/repos/simonw/datasette/issues/183
node_id: MDEyOklzc3VlQ29tbWVudDM3ODI4MTc0MA==
created_at: 2018-04-03T15:01:43Z
updated_at: 2018-04-03T15:01:43Z
author_association: OWNER
body:
  I'm having trouble replicating this bug. In particular, I don't understand what you mean by "these are then rendered in the datasette query box using single quotes" - since canned queries aren't displayed in a textarea. Do you have an example database / metadata.json I can use to investigate this further?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/181#issuecomment-378293484
issue_url: https://api.github.com/repos/simonw/datasette/issues/181
node_id: MDEyOklzc3VlQ29tbWVudDM3ODI5MzQ4NA==
created_at: 2018-04-03T15:34:29Z
updated_at: 2018-04-03T15:34:29Z
author_association: OWNER
body:
  Here's what this looks like:

  ![2018-04-03 at 8 32 am](https://user-images.githubusercontent.com/9599/38259345-9e1c75ea-3719-11e8-83c9-2160c6fa079c.png)

  I need to figure out the right way to handle licensing of bundled software like this - it's MIT licensed which is compatible with Datasette's Apache 2 license, but I feel like bundled licensed software (including codemirror) needs to be recognized in the README or docs somehow.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/181#issuecomment-378293599
issue_url: https://api.github.com/repos/simonw/datasette/issues/181
node_id: MDEyOklzc3VlQ29tbWVudDM3ODI5MzU5OQ==
created_at: 2018-04-03T15:34:50Z
updated_at: 2018-04-03T15:36:58Z
author_association: OWNER
body:
  Let's only show the "Format SQL" button if the user has JavaScript enabled. We can do that in this code here:

  https://github.com/bsmithgall/datasette/blob/4a7151a58d6ab7c8404a91beef7083e8a5807cf8/datasette/templates/_codemirror_foot.html#L14-L21

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/181#issuecomment-378295376
issue_url: https://api.github.com/repos/simonw/datasette/issues/181
node_id: MDEyOklzc3VlQ29tbWVudDM3ODI5NTM3Ng==
created_at: 2018-04-03T15:39:57Z
updated_at: 2018-04-03T15:39:57Z
author_association: OWNER
body:
  On the licensing front: it looks like the way Django handles this is to keep the licensing header in the files intact, e.g. https://github.com/django/django/blob/6deaddcca367d0143c815aaa42342021baa3b41e/django/contrib/admin/static/admin/js/vendor/jquery/jquery.js

  So for this change, adding a comment at the top of `sql-formatter.min.js` which references the MIT license would do the trick.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/181#issuecomment-378297842
issue_url: https://api.github.com/repos/simonw/datasette/issues/181
node_id: MDEyOklzc3VlQ29tbWVudDM3ODI5Nzg0Mg==
created_at: 2018-04-03T15:47:13Z
updated_at: 2018-04-03T15:47:13Z
author_association: NONE
body:
  I can work on that -- would you prefer to inline a `display: hidden` and then have the javascript flip the visibility or include it as css?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/193#issuecomment-379142500
issue_url: https://api.github.com/repos/simonw/datasette/issues/193
node_id: MDEyOklzc3VlQ29tbWVudDM3OTE0MjUwMA==
created_at: 2018-04-06T04:05:58Z
updated_at: 2018-04-06T04:05:58Z
author_association: NONE
body:
  You could try pulling out a validate query strings method. If it fails validation build the error object from the message. If it passes, you only need to go down a happy path. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-379555484
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU1NTQ4NA==
created_at: 2018-04-08T14:39:57Z
updated_at: 2018-04-08T14:39:57Z
author_association: OWNER
body:
  I'm going to combine the code for explicit sorting with the existing code for _next= pagination - so even tables without an explicit sort order will run through the same code since they are ordered and paginated by primary key.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/48#issuecomment-379556637
issue_url: https://api.github.com/repos/simonw/datasette/issues/48
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU1NjYzNw==
created_at: 2018-04-08T14:56:52Z
updated_at: 2018-04-08T14:56:52Z
author_association: OWNER
body:
  It would be useful to have a microbenchmark in place to help understand how much of a performance benefit this would actually provide.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-379556774
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU1Njc3NA==
created_at: 2018-04-08T14:59:05Z
updated_at: 2018-04-08T14:59:05Z
author_association: OWNER
body:
  A common problem with keyset pagination is that it can distort the "total number of rows" logic - every time you navigate to a further page the total rows count can decrease due to the extra arguments in the `where` clause. The `filtered_table_rows` value (see #194) calculated using `count_sql` currently has this problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/194#issuecomment-379556981
issue_url: https://api.github.com/repos/simonw/datasette/issues/194
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU1Njk4MQ==
created_at: 2018-04-08T15:02:23Z
updated_at: 2018-04-08T15:02:23Z
author_association: OWNER
body:
  Maybe `table_rows_filtered_count` would be more aesthetically pleasing than `filtered_table_rows_count`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/195#issuecomment-379557743
issue_url: https://api.github.com/repos/simonw/datasette/issues/195
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU1Nzc0Mw==
created_at: 2018-04-08T15:13:18Z
updated_at: 2018-04-08T15:13:18Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/446d47fdb005b3776bc06ad8d1f44b01fc2e938b/datasette/app.py#L93-L102

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-379557982
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU1Nzk4Mg==
created_at: 2018-04-08T15:16:49Z
updated_at: 2018-04-08T15:16:49Z
author_association: OWNER
body:
  A note about views: a view cannot be paginated using keyset pagination because records returned from a view don't have a primary key - so there's no way to reliably distinguish between _next= records when the sorted column has duplicates with the same value.

  Datasette already takes this into account: views are paginated using offset/limit instead. We can continue to do that even for views that have been sorted using a `_sort` parameter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/195#issuecomment-379559074
issue_url: https://api.github.com/repos/simonw/datasette/issues/195
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU1OTA3NA==
created_at: 2018-04-08T15:31:49Z
updated_at: 2018-04-08T15:31:49Z
author_association: OWNER
body:
  While I'm at it, doing the same thing for fts_table detection is worth considering:

  https://github.com/simonw/datasette/blob/446d47fdb005b3776bc06ad8d1f44b01fc2e938b/datasette/app.py#L598-L603

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/150#issuecomment-379559214
issue_url: https://api.github.com/repos/simonw/datasette/issues/150
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU1OTIxNA==
created_at: 2018-04-08T15:33:58Z
updated_at: 2018-04-08T15:33:58Z
author_association: OWNER
body:
  The single biggest challenge here is expanding foreign key references. This is the blocker that prevents `_group_count` from being useful at the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/150#issuecomment-379559319
issue_url: https://api.github.com/repos/simonw/datasette/issues/150
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU1OTMxOQ==
created_at: 2018-04-08T15:35:43Z
updated_at: 2018-04-08T15:35:43Z
author_association: OWNER
body:
  From a code point of view, the current mechanism for `_group_count` makes the `TableView` even **more** complicated:

  https://github.com/simonw/datasette/blob/446d47fdb005b3776bc06ad8d1f44b01fc2e938b/datasette/app.py#L644-L653

  Instead, I think if `_group_count` is detected we should generate the SQL and then defer to `self.custom_sql`, like we do for canned queries:

  https://github.com/simonw/datasette/blob/446d47fdb005b3776bc06ad8d1f44b01fc2e938b/datasette/app.py#L539-L541

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/195#issuecomment-379588602
issue_url: https://api.github.com/repos/simonw/datasette/issues/195
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU4ODYwMg==
created_at: 2018-04-08T22:40:16Z
updated_at: 2018-04-08T22:40:16Z
author_association: OWNER
body:
  Could also identify all views for that database, which would save on these queries:

  https://github.com/simonw/datasette/blob/b2188f044265c95f7e54860e28107c17d2a6ed2e/datasette/app.py#L543-L545

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-379591062
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU5MTA2Mg==
created_at: 2018-04-08T23:23:12Z
updated_at: 2018-04-08T23:23:12Z
author_association: OWNER
body:
  To break this up into smaller units, the first implementation of this will only support a single `_sort` or `_sort_desc` querystring parameter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-379592393
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU5MjM5Mw==
created_at: 2018-04-08T23:45:42Z
updated_at: 2018-04-08T23:46:31Z
author_association: OWNER
body:
  Actually next page SQL when sorting looks more like this:

  ```
  select rowid, * from [alcohol-consumption/drinks]
  where "country" like :p0
  and (
      beer_servings > 111
      or (beer_servings = 111 and rowid > 190)
  )
  order by beer_servings, rowid limit 101
  ```

  The next page after row 190 with sortable value 111 should show either records that are greater than 111 or records that match 111 but have a greater primary key than the last one seen.

  https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9?sql=select+rowid%2C+*+from+%5Balcohol-consumption%2Fdrinks%5D%0D%0Awhere+%22country%22+like+%3Ap0%0D%0Aand+%28%0D%0A++++beer_servings+%3E+111%0D%0A++++or+%28beer_servings+%3D+111+and+rowid+%3E+190%29%0D%0A%29%0D%0Aorder+by+beer_servings%2C+rowid+limit+101&p0=%25a%25

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-379594529
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU5NDUyOQ==
created_at: 2018-04-09T00:15:03Z
updated_at: 2018-04-09T00:15:03Z
author_association: OWNER
body:
  Demo: senator tweets ordered by number of replies:

  https://datasette-issue-189-demo.now.sh/fivethirtyeight-2628db9/twitter-ratio%2Fsenators?_sort_desc=replies

  Page 2 (note that since Senators retweet things there are tweets with the same text/number-of-replies but retweeted by different senators that span the page break): https://datasette-issue-189-demo.now.sh/fivethirtyeight-2628db9/twitter-ratio%2Fsenators?_next=8556%2C121799&_sort_desc=replies

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/185#issuecomment-379595253
issue_url: https://api.github.com/repos/simonw/datasette/issues/185
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU5NTI1Mw==
created_at: 2018-04-09T00:24:10Z
updated_at: 2018-04-09T00:24:10Z
author_association: OWNER
body:
  @carlmjohnson in case you aren't following along with #189 I've shipped the first working prototype of sort-by-column - you can try it out here: https://datasette-issue-189-demo-2.now.sh/salaries-7859114-7859114/2017+Maryland+state+salaries?_search=university&_sort_desc=annual_salary

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-379595274
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3OTU5NTI3NA==
created_at: 2018-04-09T00:24:37Z
updated_at: 2018-04-09T00:29:46Z
author_association: OWNER
body:
  Another demo: https://datasette-issue-189-demo-2.now.sh/salaries-7859114-7859114/2017+Maryland+state+salaries?_search=university&_sort_desc=annual_salary

  https://datasette-issue-189-demo-2.now.sh/salaries-7859114-7859114/2017+Maryland+state+salaries?_search=university&last_name__exact=JOHNSON&_sort_desc=annual_salary

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-379602690
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3OTYwMjY5MA==
created_at: 2018-04-09T01:37:03Z
updated_at: 2018-04-09T01:37:03Z
author_association: OWNER
body:
  I'm going to split the following out into separate tickets:

  * Ability to sort by multiple columns e.g. `?_sort=name&sort_desc=age&_sort=height`
  * Ability to specify nulls last e.g. `?_sort_desc_nulls_last=age`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-379608977
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3OTYwODk3Nw==
created_at: 2018-04-09T02:22:59Z
updated_at: 2018-04-09T02:22:59Z
author_association: OWNER
body:
  Here's a demo of the new clickable column headers: https://datasette-issue-189-demo-3.now.sh/salaries-7859114-7859114/2017+Maryland+state+salaries?_search=university&_sort_desc=last_name

  ![2018-04-08 at 7 22 pm](https://user-images.githubusercontent.com/9599/38476370-3e62a60e-3b62-11e8-9d30-8dc6608133dd.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/193#issuecomment-379624163
issue_url: https://api.github.com/repos/simonw/datasette/issues/193
node_id: MDEyOklzc3VlQ29tbWVudDM3OTYyNDE2Mw==
created_at: 2018-04-09T04:03:49Z
updated_at: 2018-04-09T04:03:49Z
author_association: OWNER
body:
  This is harder than I thought, because the `_shape=` logic actually runs AFTER the main block of code which is set up to catch exceptions - this code here:

  https://github.com/simonw/datasette/blob/0abd3abacb309a2bd5913a7a2df4e9256585b1bb/datasette/app.py#L200-L216

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/184#issuecomment-379636068
issue_url: https://api.github.com/repos/simonw/datasette/issues/184
node_id: MDEyOklzc3VlQ29tbWVudDM3OTYzNjA2OA==
created_at: 2018-04-09T05:26:21Z
updated_at: 2018-04-09T05:26:21Z
author_association: OWNER
body:
  Do you have steps to reproduce here - ideally a small example SQLite database that exhibits the error?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/181#issuecomment-379636695
issue_url: https://api.github.com/repos/simonw/datasette/issues/181
node_id: MDEyOklzc3VlQ29tbWVudDM3OTYzNjY5NQ==
created_at: 2018-04-09T05:30:16Z
updated_at: 2018-04-09T05:30:16Z
author_association: OWNER
body:
  I'd prefer to have the JavaScript actually manipulate the DOM to add the button - something like this:

      var button = document.createElement('button');
      button.value = 'Format SQL';
      button.addEventListener(
          'click', format, false
      );
      document.getElementById('run-sql').parentNode.appendChild(button);

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/181#issuecomment-379759875
issue_url: https://api.github.com/repos/simonw/datasette/issues/181
node_id: MDEyOklzc3VlQ29tbWVudDM3OTc1OTg3NQ==
created_at: 2018-04-09T13:53:14Z
updated_at: 2018-04-09T13:53:14Z
author_association: NONE
body:
  I've implemented that approach in 86ac746. It does cause the button to pop in only after Codemirror is finished rendering which is a bit awkward.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-379830529
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM3OTgzMDUyOQ==
created_at: 2018-04-09T17:28:47Z
updated_at: 2018-04-09T17:28:47Z
author_association: OWNER
body:
  Another demo: https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9/congress-age%2Fcongress-terms

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/199#issuecomment-379833216
issue_url: https://api.github.com/repos/simonw/datasette/issues/199
node_id: MDEyOklzc3VlQ29tbWVudDM3OTgzMzIxNg==
created_at: 2018-04-09T17:37:47Z
updated_at: 2018-04-09T17:37:47Z
author_association: OWNER
body:
  I may do this by adding select boxes for _sort and _sort_desc to the filters UI. This would allow sorting in mobile portrait mode but would also ensure that the existing sort order is persisted if the user edits the current filters (right now sort resets when filters are applied).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/199#issuecomment-379833481
issue_url: https://api.github.com/repos/simonw/datasette/issues/199
node_id: MDEyOklzc3VlQ29tbWVudDM3OTgzMzQ4MQ==
created_at: 2018-04-09T17:38:39Z
updated_at: 2018-04-09T17:38:39Z
author_association: OWNER
body:
  Since you can't apply `_sort` and `_sort_desc` at the same time, maybe just one select box for picking the column to sort by and a boolean checkbox for "sort descending" - which then redirects to the `_sort_desc=` URL variant.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/199#issuecomment-379936068
issue_url: https://api.github.com/repos/simonw/datasette/issues/199
node_id: MDEyOklzc3VlQ29tbWVudDM3OTkzNjA2OA==
created_at: 2018-04-10T00:32:37Z
updated_at: 2018-04-10T00:32:37Z
author_association: OWNER
body:
  ![2018-04-09 at 5 32 pm](https://user-images.githubusercontent.com/9599/38529802-fd2a7e68-3c1b-11e8-974a-bf5438fec701.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/199#issuecomment-379936832
issue_url: https://api.github.com/repos/simonw/datasette/issues/199
node_id: MDEyOklzc3VlQ29tbWVudDM3OTkzNjgzMg==
created_at: 2018-04-10T00:37:52Z
updated_at: 2018-04-10T00:37:52Z
author_association: OWNER
body:
  Demo: https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9/twitter-ratio%2Fsenators?_sort_desc=replies&text__contains=bipartisan

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/200#issuecomment-380606998
issue_url: https://api.github.com/repos/simonw/datasette/issues/200
node_id: MDEyOklzc3VlQ29tbWVudDM4MDYwNjk5OA==
created_at: 2018-04-11T21:50:14Z
updated_at: 2018-04-11T21:50:14Z
author_association: OWNER
body:
  We should only do this if we're certain the spatialite module has been loaded. I could imagine someone having a `sql_statements_log` table of their own without using spatialite for example.

  I think the most reliable way to detect spatialite is to run `SELECT AddGeometryColumn(1, 2, 3, 4, 5);` against a `:memory:` database and see if it throws an exception - similar to how we detect FTS. We could add this as a `detect_spatialite()` function in `utils.py` and call it once on startup.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/184#issuecomment-380608340
issue_url: https://api.github.com/repos/simonw/datasette/issues/184
node_id: MDEyOklzc3VlQ29tbWVudDM4MDYwODM0MA==
created_at: 2018-04-11T21:55:41Z
updated_at: 2018-04-11T21:55:41Z
author_association: OWNER
body:
  Yuck, nasty - OK I get it, this happens with ANY non-existent table name. Let's fix that - these should clearly return an HTTP 404.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/200#issuecomment-380608372
issue_url: https://api.github.com/repos/simonw/datasette/issues/200
node_id: MDEyOklzc3VlQ29tbWVudDM4MDYwODM3Mg==
created_at: 2018-04-11T21:55:46Z
updated_at: 2018-04-11T21:55:46Z
author_association: CONTRIBUTOR
body:
  > I think the most reliable way to detect spatialite is to run `SELECT AddGeometryColumn(1, 2, 3, 4, 5);` against a `:memory:` database and see if it throws an exception

  Or just see if there's a `geometry_columns` table? I think that's quite unlikely to be added by accident (and it's an OGC standard). It also tells you if Spatialite is installed in the database rather than just loaded.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/203#issuecomment-380951815
issue_url: https://api.github.com/repos/simonw/datasette/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDM4MDk1MTgxNQ==
created_at: 2018-04-12T21:36:10Z
updated_at: 2018-04-12T21:36:10Z
author_association: OWNER
body:
  I like this. I'd like to be able to attach a full description to a column as well. We could support these in `metadata.json`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/203#issuecomment-380951920
issue_url: https://api.github.com/repos/simonw/datasette/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDM4MDk1MTkyMA==
created_at: 2018-04-12T21:36:38Z
updated_at: 2018-04-12T21:36:38Z
author_association: OWNER
body:
  This also feeds into the visualization features I want to add - we could use this kind of metadata to automatically apply meaningful labels to graphs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/203#issuecomment-380966565
issue_url: https://api.github.com/repos/simonw/datasette/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDM4MDk2NjU2NQ==
created_at: 2018-04-12T22:43:08Z
updated_at: 2018-04-12T22:43:08Z
author_association: CONTRIBUTOR
body:
  Looks like [pint](https://pint.readthedocs.io/en/latest/tutorial.html) is pretty good at this.

  ```python
  In [1]: import pint

  In [2]: ureg = pint.UnitRegistry()

  In [3]: q = 3e6 * ureg('Hz')

  In [4]: '{:~P}'.format(q.to_compact())
  Out[4]: '3.0 MHz'

  In [5]: q = 0.3 * ureg('m')

  In [5]: '{:~P}'.format(q.to_compact())
  Out[5]: '300.0 mm'

  In [6]: q = 5 * ureg('')

  In [7]: '{:~P}'.format(q.to_compact())
  Out[7]: '5'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/201#issuecomment-381262824
issue_url: https://api.github.com/repos/simonw/datasette/issues/201
node_id: MDEyOklzc3VlQ29tbWVudDM4MTI2MjgyNA==
created_at: 2018-04-13T21:17:14Z
updated_at: 2018-04-13T21:17:14Z
author_association: OWNER
body:
  Demo: https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9?sql=explain+query+plan+select+*+from+%5Bmost-common-name%2Fsurnames%5D+order+by+rank+desc

  https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9?sql=explain+select+*+from+%5Bmost-common-name%2Fsurnames%5D+order+by+rank+desc

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/203#issuecomment-381300336
issue_url: https://api.github.com/repos/simonw/datasette/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDM4MTMwMDMzNg==
created_at: 2018-04-14T03:35:02Z
updated_at: 2018-04-14T03:35:02Z
author_association: OWNER
body:
  This is really cool - I'm very impressed by pint.

  I'd like to figure out a sensible opt-in way to expose this in the JSON output as well. Maybe with a `&_units=true` parameter?

  We should definitely expose the units section from the table metadata in the output of https://wtr-api.herokuapp.com/wtr-663ea99/license_frequency.json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/203#issuecomment-381300386
issue_url: https://api.github.com/repos/simonw/datasette/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDM4MTMwMDM4Ng==
created_at: 2018-04-14T03:35:56Z
updated_at: 2018-04-14T03:35:56Z
author_association: OWNER
body:
  In #204 you said "I'd like to add support for using units when querying but this is PR is pretty usable as-is." - I'm fascinated to hear more about how this could work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/203#issuecomment-381315675
issue_url: https://api.github.com/repos/simonw/datasette/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDM4MTMxNTY3NQ==
created_at: 2018-04-14T09:14:45Z
updated_at: 2018-04-14T09:27:30Z
author_association: CONTRIBUTOR
body:
  > I'd like to figure out a sensible opt-in way to expose this in the JSON output as well. Maybe with a &_units=true parameter?

  <s>From a machine-readable perspective I'm not sure why it would be useful to decorate the values with units</s>. Edit: Should have had some coffee first. It's clearly useful for stuff like map rendering!

  I agree that the unit metadata should definitely be exposed in the JSON.

  > In #204 you said "I'd like to add support for using units when querying but this is PR is pretty usable as-is." - I'm fascinated to hear more about how this could work.

  I'm thinking about a couple of approaches here. I think the simplest one is: if the column has a unit attached, optionally accept units in query fields:

  ```python
  column_units = ureg("Hz") #  Create a unit object for the column's unit
  query_variable = ureg("4 GHz") # Supplied query variable

  # Now we can convert the query units into column units before querying
  supplied_value.to(column_units).magnitude
  > 4000000000.0

  # If the user doesn't supply units, pint just returns the plain
  # number and we can query as usual assuming it's the base unit
  query_variable = ureg("50")
  query_variable
  > 50

  isinstance(query_variable, numbers.Number)
  > True
  ```

  This also lets us do some nice unit conversion on querying:

  ```python
  column_units = ureg("m")
  query_variable = ureg("50 ft")

  supplied_value.to(column_units)
  > <Quantity(15.239999999999998, 'meter')>
  ```

  The alternative would be to provide a dropdown of units next to the query field (so a "Hz" field would give you "kHz", "MHz", "GHz"). Although this would be clearer to the user, it isn't so easy - we'd need to know more about the context of the field to give you sensible SI prefixes (I'm not so interested in nanoHertz, for example).

  You also lose the bonus of being able to convert - although pint will happily show you all the compatible units, it again suffers from a lack of context:

  ```python
  ureg("m").compatible_units()
  > frozenset({<Unit('angstrom')>,
             <Unit('thou')>,
             <Unit('inch')>,
             <Unit('link')>,
             <Unit('foot')>,
             <Unit('survey_foot')>,
             <Unit('yard')>,
             <Unit('rod')>,
             <Unit('mile')>,
             <Unit('survey_mile')>,
             <Unit('league')>,
             <Unit('light_year')>})
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/203#issuecomment-381330075
issue_url: https://api.github.com/repos/simonw/datasette/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDM4MTMzMDA3NQ==
created_at: 2018-04-14T13:41:53Z
updated_at: 2018-04-14T13:41:53Z
author_association: OWNER
body:
  Presumably units only work for numeric fields? If that's the case then automatically processing them if the incoming query string argument has a unit suffix makes total sense to me.

  Here's a pretty crazy idea: what if we exposed unit conversion to SQL as a custom SQLite function? That way it would be possible to optionally use units in actual custom SQL queries. 

  I'd have to think quite carefully about performance implications here - wouldn't want a poorly considered unit calculation over a 500,000 row table to lock up the server. But I think the 1s query time limit might still prevent that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/205#issuecomment-381330220
issue_url: https://api.github.com/repos/simonw/datasette/issues/205
node_id: MDEyOklzc3VlQ29tbWVudDM4MTMzMDIyMA==
created_at: 2018-04-14T13:44:15Z
updated_at: 2018-04-14T13:44:15Z
author_association: OWNER
body:
  This looks great so far - love the new documentation. Let's throw in a unit test or two for the basic unit filters (mainly as a protection against accidental regressions in the future).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/203#issuecomment-381348849
issue_url: https://api.github.com/repos/simonw/datasette/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDM4MTM0ODg0OQ==
created_at: 2018-04-14T18:12:52Z
updated_at: 2018-04-14T18:12:52Z
author_association: OWNER
body:
  I think I'm going to hold on to the custom sql function idea for the moment and implement it as an example plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/125#issuecomment-381361734
issue_url: https://api.github.com/repos/simonw/datasette/issues/125
node_id: MDEyOklzc3VlQ29tbWVudDM4MTM2MTczNA==
created_at: 2018-04-14T21:26:30Z
updated_at: 2018-04-14T21:26:30Z
author_association: CONTRIBUTOR
body:
  FWIW I am now doing this on my WTR app (instead of silently limiting maps to 1000).

  [Telefonica](https://wtr-api.herokuapp.com/wtr-663ea99/licensee/18325) now has about 4000 markers and good old [BT](https://wtr-api.herokuapp.com/wtr-663ea99/licensee/8412) has 22,000 or so.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/189#issuecomment-381429213
issue_url: https://api.github.com/repos/simonw/datasette/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQyOTIxMw==
created_at: 2018-04-15T18:54:22Z
updated_at: 2018-04-15T18:54:22Z
author_association: NONE
body:
  I think I found a bug. I tried to sort by middle initial in my salaries set, and many middle initials are null. The next_url gets set by Datasette to:

  http://localhost:8001/salaries-d3a5631/2017+Maryland+state+salaries?_next=None%2C391&_sort=middle_initial

  But then `None` is interpreted literally and it tries to find a name with the middle initial "None" and ends up skipping ahead to O on page 2.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/209#issuecomment-381441392
issue_url: https://api.github.com/repos/simonw/datasette/issues/209
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ0MTM5Mg==
created_at: 2018-04-15T21:59:15Z
updated_at: 2018-04-15T21:59:15Z
author_association: CONTRIBUTOR
body:
  I suspected this would cause some test failures, but I'll wait for opinions before attempting to fix them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-381442233
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ0MjIzMw==
created_at: 2018-04-15T22:13:06Z
updated_at: 2018-04-15T22:13:06Z
author_association: OWNER
body:
  I started a thread on Twitter asking people for good examples of Python projects with a strong plugin ecosystem: https://twitter.com/simonw/status/985377670388105216

  The most impressive example that came back was pytest - which now has nearly 400 plugins: https://plugincompat.herokuapp.com/

  The pytest plugin infrastructure is available as an independent package called pluggy - which appears to offer everything I need for Datasette. I'm going to give that a go and see how well it works: https://pluggy.readthedocs.io/en/latest/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-381442494
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ0MjQ5NA==
created_at: 2018-04-15T22:17:59Z
updated_at: 2018-04-15T22:17:59Z
author_association: OWNER
body:
  Datasette 1.0 will be the release of Datasette that attempts to provide a stable plugin API: https://github.com/simonw/datasette/milestone/7

  There's a lot of work to be done before then, but as a starting point I'm going to support two very simple extension mechanisms:

  * Template system plugins - where the hook gets passed the Jinja environment and can freely register new template tags and filters
  * SQLite connection plugins - where the hook gets passed a new SQLite connection and can register custom SQLite functions

  The template system hook will go near here:

  https://github.com/simonw/datasette/blob/efbb4e83374a2c795e436c72fa79f70da72309b8/datasette/app.py#L1225-L1228

  The SQLite connection hook will go near here:

  https://github.com/simonw/datasette/blob/efbb4e83374a2c795e436c72fa79f70da72309b8/datasette/app.py#L1094-L1098

  These two feel simple enough that I'm not worried that I might design an API that I later regret.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-381443728
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ0MzcyOA==
created_at: 2018-04-15T22:39:00Z
updated_at: 2018-04-15T22:39:00Z
author_association: OWNER
body:
  Tox is a good example of a project that uses pluggy in the way I want to use it (function hooks rather than classes): https://github.com/tox-dev/tox/blob/master/tox/hookspecs.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-381446392
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ0NjM5Mg==
created_at: 2018-04-15T23:22:40Z
updated_at: 2018-04-16T05:25:57Z
author_association: OWNER
body:
  OK, from that prototype in f2720b0c6b7172ebe8820 it looks like pluggy provides a solid path forward.

  Next steps:

  - [x] Build a demo plugin that uses setuptools entrypoints to register with the `datasette` plugin manager via pluggy
  - [x] Figure out a mechanism for registering plugins without first needing to publish them to PyPI. Can I load plugins from a special `plugins/` directory similar to the `--template-dir=templates/` option already supported by Datasette? #211

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-381446511
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ0NjUxMQ==
created_at: 2018-04-15T23:25:04Z
updated_at: 2018-04-15T23:25:04Z
author_association: OWNER
body:
  Here's a demo of the `convert_units()` SQL function I prototyped in f2720b0c6b7172ebe88

  ![2018-04-15 at 4 23 pm](https://user-images.githubusercontent.com/9599/38784633-8c43821e-40c9-11e8-97dd-697755a0f858.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/203#issuecomment-381446554
issue_url: https://api.github.com/repos/simonw/datasette/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ0NjU1NA==
created_at: 2018-04-15T23:25:54Z
updated_at: 2018-04-15T23:26:03Z
author_association: OWNER
body:
  I built a prototype of the `convert_units()` custom SQL function as a plugin over in https://github.com/simonw/datasette/issues/14#issuecomment-381446511

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-381446906
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ0NjkwNg==
created_at: 2018-04-15T23:31:58Z
updated_at: 2018-04-15T23:34:10Z
author_association: OWNER
body:
  Once I've got the plugins mechanism stable and people start releasing plugins it would be useful to have a dedicated Trove classifier on PyPI for Datasette plugins - `Framework :: Datasette` for example.

  This would help me build a Datasette equivalent of the http://plugincompat.herokuapp.com/ site, which works by scanning PyPI for items with the ``Framework :: Pytest`` classifier:

  https://github.com/pytest-dev/plugincompat/blob/8bdf1a6fb82807091ece0c68c196103ee8270194/update_index.py#L52-L53

  It looks like the mechanism for requesting new PyPI classifiers is to file a ticket against warehouse, like these ones: https://github.com/pypa/warehouse/issues/3570 and https://github.com/pypa/warehouse/issues/2881

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-381450394
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ1MDM5NA==
created_at: 2018-04-16T00:27:23Z
updated_at: 2018-04-16T00:27:23Z
author_association: OWNER
body:
  I created https://github.com/simonw/datasette-plugin-demos which is now published to PyPI and can be installed with `pip install datasette-plugin-demos` - I've confirmed that if you DO install it my Datasette `plugins` branch picks up the plugins, and `select random_integer(1, 4)` works as it should.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-381450591
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ1MDU5MQ==
created_at: 2018-04-16T00:30:22Z
updated_at: 2018-04-16T00:34:42Z
author_association: OWNER
body:
  Slight code design problem... when I tried installing my branch in a fresh virtual environment I got this error, because `setup.py` now depends on `pluggy` (from importing `__version__`):

  ```
        File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/setup.py", line 2, in <module>
          from datasette import __version__
        File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/datasette/__init__.py", line 2, in <module>
          from .hookspecs import hookimpl # noqa
        File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/datasette/hookspecs.py", line 1, in <module>
          from pluggy import HookimplMarker
      ModuleNotFoundError: No module named 'pluggy'
  ```

  Looks like I've run into point 6 on https://packaging.python.org/guides/single-sourcing-package-version/ :

  ![2018-04-15 at 5 34 pm](https://user-images.githubusercontent.com/9599/38785314-403ce86a-40d3-11e8-8542-ba426eddf4ac.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/139#issuecomment-381455054
issue_url: https://api.github.com/repos/simonw/datasette/issues/139
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ1NTA1NA==
created_at: 2018-04-16T01:24:13Z
updated_at: 2018-04-16T01:24:13Z
author_association: OWNER
body:
  I think Vega-Lite is the way to go here: https://vega.github.io/vega-lite/

  I've been playing around with it and Datasette with some really positive initial results: 

  https://vega.github.io/editor/#/gist/vega-lite/simonw/89100ce80573d062d70f780d10e5e609/decada131575825875c0a076e418c661c2adb014/vice-shootings-gender-race-by-department.vl.json

  https://vega.github.io/editor/#/gist/vega-lite/simonw/5f69fbe29380b0d5d95f31a385f49ee4/7087b64df03cf9dba44a5258a606f29182cb8619/trees-san-francisco.vl.json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/211#issuecomment-381456434
issue_url: https://api.github.com/repos/simonw/datasette/issues/211
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ1NjQzNA==
created_at: 2018-04-16T01:36:16Z
updated_at: 2018-04-16T01:37:44Z
author_association: OWNER
body:
  The easiest way to implement this in Python 2 would be `execfile(...)` - but that was removed in Python 3. According to https://stackoverflow.com/a/437857/6083 `2to3` replaces that with this, which ensures the filename is associated with the code for debugging purposes:

  ```
  with open("somefile.py") as f:
      code = compile(f.read(), "somefile.py", 'exec')
      exec(code, global_vars, local_vars)
  ```

  Implementing it this way would force this kind of plugin to be self-contained in a single file. I think that's OK: if you want a more complex plugin you can use the standard pluggy-powered setuptools mechanism to build it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/211#issuecomment-381462005
issue_url: https://api.github.com/repos/simonw/datasette/issues/211
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ2MjAwNQ==
created_at: 2018-04-16T02:23:07Z
updated_at: 2018-04-16T02:23:07Z
author_association: OWNER
body:
  This needs unit tests. I also need to manually test the `datasette package` and `datesette publish` commands.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/211#issuecomment-381478217
issue_url: https://api.github.com/repos/simonw/datasette/issues/211
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ3ODIxNw==
created_at: 2018-04-16T04:41:38Z
updated_at: 2018-04-16T04:41:38Z
author_association: OWNER
body:
  Here's the result of running:

      datasette publish now fivethirtyeight.db \
          --plugins-dir=plugins/ --title="FiveThirtyEight" --branch=plugins-dir

  https://datasette-phjtvzwwzl.now.sh/fivethirtyeight-2628db9?sql=select+convert_units%28100%2C+%27m%27%2C+%27ft%27%29

  Where `plugins/pint_plugin.py` contains the following:
  ```
  from datasette import hookimpl
  import pint

  ureg = pint.UnitRegistry()

  @hookimpl
  def prepare_connection(conn):
      def convert_units(amount, from_, to_):
          "select convert_units(100, 'm', 'ft');"
          return (amount * ureg(from_)).to(to_).to_tuple()[0]
      conn.create_function('convert_units', 3, convert_units)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/211#issuecomment-381478253
issue_url: https://api.github.com/repos/simonw/datasette/issues/211
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ3ODI1Mw==
created_at: 2018-04-16T04:42:02Z
updated_at: 2018-04-16T04:42:02Z
author_association: OWNER
body:
  This worked as well:

      datasette package fivethirtyeight.db \
          --plugins-dir=plugins/ --title="FiveThirtyEight" --branch=plugins-dir

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/211#issuecomment-381482407
issue_url: https://api.github.com/repos/simonw/datasette/issues/211
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ4MjQwNw==
created_at: 2018-04-16T05:18:29Z
updated_at: 2018-04-16T05:18:29Z
author_association: OWNER
body:
  Here's the result of running this:

      datasette publish heroku fivethirtyeight.db \
          --plugins-dir=plugins/ --title="FiveThirtyEight" --branch=plugins-dir

  https://intense-river-24599.herokuapp.com/fivethirtyeight-2628db9?sql=select+convert_units%28100%2C+%27m%27%2C+%27ft%27%29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/191#issuecomment-381488049
issue_url: https://api.github.com/repos/simonw/datasette/issues/191
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ4ODA0OQ==
created_at: 2018-04-16T05:58:15Z
updated_at: 2018-04-16T05:58:15Z
author_association: OWNER
body:
  I think this is pretty hard. @coleifer has done some work in this direction, including https://github.com/coleifer/pysqlite3 which ports the standalone pysqlite module to Python 3. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/214#issuecomment-381490361
issue_url: https://api.github.com/repos/simonw/datasette/issues/214
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ5MDM2MQ==
created_at: 2018-04-16T06:13:02Z
updated_at: 2018-04-16T06:13:02Z
author_association: OWNER
body:
  Packaging JS and CSS in a pip installable wheel is fiddly but possible. http://peak.telecommunity.com/DevCenter/PythonEggs#accessing-package-resources

      from pkg_resources import resource_string
      foo_config = resource_string(__name__, 'foo.conf')

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/214#issuecomment-381491707
issue_url: https://api.github.com/repos/simonw/datasette/issues/214
node_id: MDEyOklzc3VlQ29tbWVudDM4MTQ5MTcwNw==
created_at: 2018-04-16T06:21:23Z
updated_at: 2018-04-16T06:21:23Z
author_association: OWNER
body:
  This looks like a good example: https://github.com/funkey/nyroglancer/commit/d4438ab42171360b2b8e9020f672846dd70c8d80

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/191#issuecomment-381602005
issue_url: https://api.github.com/repos/simonw/datasette/issues/191
node_id: MDEyOklzc3VlQ29tbWVudDM4MTYwMjAwNQ==
created_at: 2018-04-16T13:37:32Z
updated_at: 2018-04-16T13:37:32Z
author_association: NONE
body:
  I don't think it should be too difficult... you can look at what @ghaering did with pysqlite (and similarly what I copied for pysqlite3). You would theoretically take an amalgamation build of Sqlite (all code in a single .c and .h file). The `AmalgamationLibSqliteBuilder` class detects the presence of this amalgamated source file and builds a statically-linked pysqlite.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-381611738
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MTYxMTczOA==
created_at: 2018-04-16T14:07:30Z
updated_at: 2018-04-16T14:07:30Z
author_association: OWNER
body:
  I should check if it's possible to have two template registration function plugins in a single plugin module. If it isn't maybe I should use class plugins instead of module plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-381621338
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MTYyMTMzOA==
created_at: 2018-04-16T14:36:27Z
updated_at: 2018-04-16T14:36:27Z
author_association: OWNER
body:
  Annoyingly, the following only results in the last of the two `prepare_connection` hooks being registered:

  ```
  from datasette import hookimpl
  import pint
  import random

  ureg = pint.UnitRegistry()


  @hookimpl
  def prepare_connection(conn):
      def convert_units(amount, from_, to_):
          "select convert_units(100, 'm', 'ft');"
          return (amount * ureg(from_)).to(to_).to_tuple()[0]
      conn.create_function('convert_units', 3, convert_units)


  @hookimpl
  def prepare_connection(conn):
      conn.create_function('random_integer', 2, random.randint)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-381622793
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MTYyMjc5Mw==
created_at: 2018-04-16T14:40:39Z
updated_at: 2018-04-17T01:47:15Z
author_association: OWNER
body:
  I think that's OK. The two plugins I've implemented so far (`prepare_connection` and `prepare_jinja2_environment`) both make sense if they can only be defined once-per-plugin. For the moment I'll assume I can define future hooks to work well with the same limitation.

  The syntactic sugar idea in #220 can help here too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/216#issuecomment-381644355
issue_url: https://api.github.com/repos/simonw/datasette/issues/216
node_id: MDEyOklzc3VlQ29tbWVudDM4MTY0NDM1NQ==
created_at: 2018-04-16T15:24:38Z
updated_at: 2018-04-16T15:24:38Z
author_association: OWNER
body:
  So there are two tricky problems to solve here:

  * I need a way of encoding `null` into that `_next=` that is unambiguous from the string `None` or `null`. This means introducing some kind of escaping mechanism in those strings. I already use URL encoding as part of the construction of those components here, maybe that can help here?
  * I need to figure out what the SQL should be for the "next" set of results if the previous value was null. Thankfully we use the primary key as a tie-breaker so this shouldn't be impossible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/216#issuecomment-381645274
issue_url: https://api.github.com/repos/simonw/datasette/issues/216
node_id: MDEyOklzc3VlQ29tbWVudDM4MTY0NTI3NA==
created_at: 2018-04-16T15:27:16Z
updated_at: 2018-04-16T15:27:16Z
author_association: OWNER
body:
  Relevant code:

  https://github.com/simonw/datasette/blob/904f1c75a3c17671d25c53b91e177c249d14ab3b/datasette/app.py#L828-L832

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/216#issuecomment-381645973
issue_url: https://api.github.com/repos/simonw/datasette/issues/216
node_id: MDEyOklzc3VlQ29tbWVudDM4MTY0NTk3Mw==
created_at: 2018-04-16T15:29:11Z
updated_at: 2018-04-16T15:29:11Z
author_association: OWNER
body:
  I could use `$null` as a magic value that means None. Since I'm applying `quote_plus()` to actual values, any legit strings that look like this will be encoded as `%24null`:

  ```
  >>> urllib.parse.quote_plus('$null')
  '%24null'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/216#issuecomment-381648053
issue_url: https://api.github.com/repos/simonw/datasette/issues/216
node_id: MDEyOklzc3VlQ29tbWVudDM4MTY0ODA1Mw==
created_at: 2018-04-16T15:35:17Z
updated_at: 2018-04-16T15:35:17Z
author_association: OWNER
body:
  I think the correct SQL is this: https://datasette-issue-189-demo-3.now.sh/salaries-7859114-7859114?sql=select+rowid%2C+*+from+%5B2017+Maryland+state+salaries%5D%0D%0Awhere+%28middle_initial+is+not+null+or+%28middle_initial+is+null+and+rowid+%3E+%3Ap0%29%29%0D%0Aorder+by+middle_initial+limit+101&p0=391

  ```
  select rowid, * from [2017 Maryland state salaries]
  where (middle_initial is not null or (middle_initial is null and rowid > :p0))
  order by middle_initial limit 101
  ```

  Though this will also need to be taken into account for #198 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/216#issuecomment-381649437
issue_url: https://api.github.com/repos/simonw/datasette/issues/216
node_id: MDEyOklzc3VlQ29tbWVudDM4MTY0OTQzNw==
created_at: 2018-04-16T15:39:21Z
updated_at: 2018-04-16T15:39:21Z
author_association: OWNER
body:
  Here's where that SQL gets constructed at the moment:

  https://github.com/simonw/datasette/blob/10a34f995c70daa37a8a2aa02c3135a4b023a24c/datasette/app.py#L761-L771

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/209#issuecomment-381738137
issue_url: https://api.github.com/repos/simonw/datasette/issues/209
node_id: MDEyOklzc3VlQ29tbWVudDM4MTczODEzNw==
created_at: 2018-04-16T20:27:43Z
updated_at: 2018-04-16T20:27:43Z
author_association: CONTRIBUTOR
body:
  Tests now fixed, honest. The failing test on Travis looks like an intermittent sqlite failure which should resolve itself on a retry...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/203#issuecomment-381763651
issue_url: https://api.github.com/repos/simonw/datasette/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDM4MTc2MzY1MQ==
created_at: 2018-04-16T21:59:17Z
updated_at: 2018-04-16T21:59:17Z
author_association: CONTRIBUTOR
body:
  Ah, I had no idea you could bind python functions into sqlite!

  I think the primary purpose of this issue has been served now - I'm going to close this and create a new issue for the only bit of this that hasn't been touched yet, which is (optionally) exposing units in the JSON API.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/220#issuecomment-381777108
issue_url: https://api.github.com/repos/simonw/datasette/issues/220
node_id: MDEyOklzc3VlQ29tbWVudDM4MTc3NzEwOA==
created_at: 2018-04-16T23:04:04Z
updated_at: 2018-04-16T23:04:04Z
author_association: OWNER
body:
  This could also help workaround the current predicament that a single plugin can only define one prepare_connection hook.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/216#issuecomment-381786522
issue_url: https://api.github.com/repos/simonw/datasette/issues/216
node_id: MDEyOklzc3VlQ29tbWVudDM4MTc4NjUyMg==
created_at: 2018-04-16T23:58:45Z
updated_at: 2018-04-16T23:59:13Z
author_association: OWNER
body:
  Weird... tests are failing in Travis, despite passing on my local machine. https://travis-ci.org/simonw/datasette/builds/367423706

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/216#issuecomment-381798786
issue_url: https://api.github.com/repos/simonw/datasette/issues/216
node_id: MDEyOklzc3VlQ29tbWVudDM4MTc5ODc4Ng==
created_at: 2018-04-17T01:18:25Z
updated_at: 2018-04-17T01:18:25Z
author_association: OWNER
body:
  Here's the test that's failing:

  https://github.com/simonw/datasette/blob/59a3aa859c0e782aeda9a515b1b52c358e8458a2/tests/test_api.py#L437-L470

  I got Travis to spit out the `fetched` and `expected` variables.

  `expected` has 201 items in it and is identical to what I get on my local laptop.

  `fetched` has 250 items in it, so it's clearly different from my local environment.

  I've managed to replicate the bug in production! I created a test database like this:

      python tests/fixtures.py sortable.db

  Then deployed that database like so:

      datasette publish now sortable.db \
          --extra-options="--page_size=50" --branch=debug-travis-issue-216

  And... if you click "next" on this page https://datasette-issue-216-pagination.now.sh/sortable-5679797/sortable?_sort_desc=sortable_with_nulls five times you get back 250 results, when you should only get back 201.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/216#issuecomment-381799267
issue_url: https://api.github.com/repos/simonw/datasette/issues/216
node_id: MDEyOklzc3VlQ29tbWVudDM4MTc5OTI2Nw==
created_at: 2018-04-17T01:21:35Z
updated_at: 2018-04-17T01:21:35Z
author_association: OWNER
body:
  The version that I deployed which exhibits the bug is running SQLite `3.8.7.1` - https://datasette-issue-216-pagination.now.sh/sortable-5679797?sql=select+sqlite_version%28%29

  The version that I have running locally which does NOT exhibit the bug is running SQLite `3.23.0`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/216#issuecomment-381799408
issue_url: https://api.github.com/repos/simonw/datasette/issues/216
node_id: MDEyOklzc3VlQ29tbWVudDM4MTc5OTQwOA==
created_at: 2018-04-17T01:22:30Z
updated_at: 2018-04-17T01:22:30Z
author_association: OWNER
body:
  ... which is VERY surprising, because `3.23.0` only came out on 2nd April this year: https://www.sqlite.org/changes.html - I have no idea how I came to be running that version on my laptop.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/216#issuecomment-381801302
issue_url: https://api.github.com/repos/simonw/datasette/issues/216
node_id: MDEyOklzc3VlQ29tbWVudDM4MTgwMTMwMg==
created_at: 2018-04-17T01:33:43Z
updated_at: 2018-04-17T01:33:43Z
author_association: OWNER
body:
  This is the SQL that returns differing results in production and on my laptop: https://datasette-issue-216-pagination.now.sh/sortable-5679797?sql=select+%2A+from+sortable+where+%28sortable_with_nulls+is+null+and+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29+order+by+sortable_with_nulls+desc+limit+51&p0=b&p1=t

  ```
  select * from sortable where (sortable_with_nulls is null and ((pk1 > :p0)
    or
  (pk1 = :p0 and pk2 > :p1))) order by sortable_with_nulls desc limit 51
  ```
  I think that `order by sortable_with_nulls desc` bit is at fault - the primary keys should be included in that order by as well.

  Sure enough, changing the query to this one returns the same results across both environments:

  ```
  select * from sortable where (sortable_with_nulls is null and ((pk1 > :p0)
    or
  (pk1 = :p0 and pk2 > :p1))) order by sortable_with_nulls desc, pk1, pk2 limit 51
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-381809998
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MTgwOTk5OA==
created_at: 2018-04-17T02:23:39Z
updated_at: 2018-04-17T02:23:39Z
author_association: OWNER
body:
  I just shipped Datasette 0.19 with where I'm at so far: https://github.com/simonw/datasette/releases/tag/0.19

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/209#issuecomment-381905593
issue_url: https://api.github.com/repos/simonw/datasette/issues/209
node_id: MDEyOklzc3VlQ29tbWVudDM4MTkwNTU5Mw==
created_at: 2018-04-17T08:50:28Z
updated_at: 2018-04-17T08:50:28Z
author_association: CONTRIBUTOR
body:
  I've added another commit which puts classes a class on each `<td>` by default with its column name, and I've also made the PK column bold.

  Unfortunately the tests are still failing on 3.6, which is weird. I can't reproduce locally...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/214#issuecomment-382038613
issue_url: https://api.github.com/repos/simonw/datasette/issues/214
node_id: MDEyOklzc3VlQ29tbWVudDM4MjAzODYxMw==
created_at: 2018-04-17T15:38:23Z
updated_at: 2018-04-17T15:38:23Z
author_association: OWNER
body:
  I figured out the recipe for bundling static assets in a plugin: https://github.com/simonw/datasette-plugin-demos/commit/26c5548f4ab7c6cc6d398df17767950be50d0edf (and then `python3 setup.py bdist_wheel`)

  Having done that, I ran `pip install ../datasette-plugin-demos/dist/datasette_plugin_demos-0.2-py3-none-any.whl` from my Datasette virtual environment and then did the following:

  ```
  >>> import pkg_resources
  >>> pkg_resources.resource_stream(
  ...    'datasette_plugin_demos', 'static/plugin.js'
  ... ).read()
  b"alert('hello');\n"
  >>> pkg_resources.resource_filename(
  ...    'datasette_plugin_demos', 'static/plugin.js'
  ... )
  '..../venv/lib/python3.6/site-packages/datasette_plugin_demos/static/plugin.js'
  >>> pkg_resources.resource_string(
  ...    'datasette_plugin_demos', 'static/plugin.js'
  ... )
  b"alert('hello');\n"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/214#issuecomment-382048582
issue_url: https://api.github.com/repos/simonw/datasette/issues/214
node_id: MDEyOklzc3VlQ29tbWVudDM4MjA0ODU4Mg==
created_at: 2018-04-17T16:04:42Z
updated_at: 2018-04-18T02:24:46Z
author_association: OWNER
body:
  One possible option: let plugins bundle their own `static/` directory and then register themselves with Datasette, then have `/-/static-plugins/name-of-plugin/...` serve files from that directory.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/214#issuecomment-382069980
issue_url: https://api.github.com/repos/simonw/datasette/issues/214
node_id: MDEyOklzc3VlQ29tbWVudDM4MjA2OTk4MA==
created_at: 2018-04-17T17:08:28Z
updated_at: 2018-04-17T17:08:28Z
author_association: OWNER
body:
  Even if we automatically serve ALL `static/` content from installed plugins, we'll still need them to register which files need to be linked to from `extra_css_urls` and `extra_js_urls`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/209#issuecomment-382205189
issue_url: https://api.github.com/repos/simonw/datasette/issues/209
node_id: MDEyOklzc3VlQ29tbWVudDM4MjIwNTE4OQ==
created_at: 2018-04-18T00:42:44Z
updated_at: 2018-04-18T00:43:02Z
author_association: OWNER
body:
  I managed to get a better error message out of that test. The server is returning this (but only on Python 3.6, not on Python 3.5 - and only in Travis, not in my local environment):

  ```{'error': 'interrupted', 'ok': False, 'status': 400, 'title': 'Invalid SQL'}```

  https://travis-ci.org/simonw/datasette/jobs/367929134

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-382256729
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MjI1NjcyOQ==
created_at: 2018-04-18T04:29:29Z
updated_at: 2018-04-18T04:30:14Z
author_association: OWNER
body:
  I added a mechanism for plugins to serve static files and define custom CSS and JS URLs in #214 - see new documentation on http://datasette.readthedocs.io/en/latest/plugins.html#static-assets and http://datasette.readthedocs.io/en/latest/plugins.html#extra-css-urls

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/223#issuecomment-382408128
issue_url: https://api.github.com/repos/simonw/datasette/issues/223
node_id: MDEyOklzc3VlQ29tbWVudDM4MjQwODEyOA==
created_at: 2018-04-18T14:33:09Z
updated_at: 2018-04-18T14:33:09Z
author_association: OWNER
body:
  Demo:

      datasette publish now sortable.db --install datasette-plugin-demos --branch=master

  Produced this deployment, with both the `random_integer()` function and the static file from https://github.com/simonw/datasette-plugin-demos/tree/0.2

  https://datasette-issue-223.now.sh/-/static-plugins/datasette_plugin_demos/plugin.js

  https://datasette-issue-223.now.sh/sortable-4bbaa6f?sql=select+random_integer%280%2C+10%29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/223#issuecomment-382409989
issue_url: https://api.github.com/repos/simonw/datasette/issues/223
node_id: MDEyOklzc3VlQ29tbWVudDM4MjQwOTk4OQ==
created_at: 2018-04-18T14:38:08Z
updated_at: 2018-04-18T14:38:08Z
author_association: OWNER
body:
  Tested on Heroku as well.

      datasette publish heroku sortable.db --install datasette-plugin-demos --branch=master

  https://morning-tor-45944.herokuapp.com/-/static-plugins/datasette_plugin_demos/plugin.js

  https://morning-tor-45944.herokuapp.com/sortable-4bbaa6f?sql=select+random_integer%280%2C+10%29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/223#issuecomment-382413121
issue_url: https://api.github.com/repos/simonw/datasette/issues/223
node_id: MDEyOklzc3VlQ29tbWVudDM4MjQxMzEyMQ==
created_at: 2018-04-18T14:47:18Z
updated_at: 2018-04-18T14:47:18Z
author_association: OWNER
body:
  And tested `datasette package` - this time exercising the ability to pass more than one `--install` option:

  ```
  $ datasette package sortable.db --branch=master --install requests --install datasette-plugin-demos
  Sending build context to Docker daemon  125.4kB
  Step 1/7 : FROM python:3
   ---> 79e1dc9af1c1
  Step 2/7 : COPY . /app
   ---> 6e8e40bce378
  Step 3/7 : WORKDIR /app
  Removing intermediate container 7cdc9ab20d09
   ---> f42258c2211f
  Step 4/7 : RUN pip install https://github.com/simonw/datasette/archive/master.zip requests datasette-plugin-demos
   ---> Running in a0f17cec08a4
  Collecting ...
  Removing intermediate container a0f17cec08a4
   ---> beea84e73271
  Step 5/7 : RUN datasette inspect sortable.db --inspect-file inspect-data.json
   ---> Running in 4daa28792348
  Removing intermediate container 4daa28792348
   ---> c60312d21b99
  Step 6/7 : EXPOSE 8001
   ---> Running in fa728468482d
  Removing intermediate container fa728468482d
   ---> 8f219a61fddc
  Step 7/7 : CMD ["datasette", "serve", "--host", "0.0.0.0", "sortable.db", "--cors", "--port", "8001", "--inspect-file", "inspect-data.json"]
   ---> Running in cd4eaeb2ce9e
  Removing intermediate container cd4eaeb2ce9e
   ---> 066e257c7c44
  Successfully built 066e257c7c44
  (venv) datasette $ docker run -p 8081:8001 066e257c7c44
  Serve! files=('sortable.db',) on port 8001
  [2018-04-18 14:40:18 +0000] [1] [INFO] Goin' Fast @ http://0.0.0.0:8001
  [2018-04-18 14:40:18 +0000] [1] [INFO] Starting worker [1]
  [2018-04-18 14:46:01 +0000] - (sanic.access)[INFO][1:7]: GET http://localhost:8081/-/static-plugins/datasette_plugin_demos/plugin.js  200 16
  ```    

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/224#issuecomment-382616527
issue_url: https://api.github.com/repos/simonw/datasette/issues/224
node_id: MDEyOklzc3VlQ29tbWVudDM4MjYxNjUyNw==
created_at: 2018-04-19T05:40:28Z
updated_at: 2018-04-19T05:40:28Z
author_association: OWNER
body:
  No need to use `PackageLoader` after all, we can use the same mechanism we used for the static path:

  https://github.com/simonw/datasette/blob/b55809a1e20986bb2e638b698815a77902e8708d/datasette/utils.py#L694-L695

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/227#issuecomment-382808266
issue_url: https://api.github.com/repos/simonw/datasette/issues/227
node_id: MDEyOklzc3VlQ29tbWVudDM4MjgwODI2Ng==
created_at: 2018-04-19T16:59:23Z
updated_at: 2018-04-19T16:59:23Z
author_association: OWNER
body:
  Maybe this should have a second argument indicating which codepath was being handled. That way plugins could say "only inject this extra context variable on the row page".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/228#issuecomment-382924910
issue_url: https://api.github.com/repos/simonw/datasette/issues/228
node_id: MDEyOklzc3VlQ29tbWVudDM4MjkyNDkxMA==
created_at: 2018-04-20T00:35:48Z
updated_at: 2018-04-20T00:35:48Z
author_association: OWNER
body:
  Hiding tables with the `idx_` prefix should be good enough here, since false positives aren't very harmful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/227#issuecomment-382959857
issue_url: https://api.github.com/repos/simonw/datasette/issues/227
node_id: MDEyOklzc3VlQ29tbWVudDM4Mjk1OTg1Nw==
created_at: 2018-04-20T03:21:43Z
updated_at: 2018-04-20T03:21:43Z
author_association: OWNER
body:
  Plus a generic prepare_context() hook called in the common render method.

  prepare_context_table(), prepare_context_row() etc

  Arguments are context, request, self (hence can access self.ds)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/227#issuecomment-382964794
issue_url: https://api.github.com/repos/simonw/datasette/issues/227
node_id: MDEyOklzc3VlQ29tbWVudDM4Mjk2NDc5NA==
created_at: 2018-04-20T03:45:18Z
updated_at: 2018-04-20T03:45:18Z
author_association: OWNER
body:
  What if the context needs to make await calls?

  One possible option: plugins can either manipulate the context in place OR they can return an awaitable. If they do that, the caller will await it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/227#issuecomment-382966604
issue_url: https://api.github.com/repos/simonw/datasette/issues/227
node_id: MDEyOklzc3VlQ29tbWVudDM4Mjk2NjYwNA==
created_at: 2018-04-20T03:54:56Z
updated_at: 2018-04-20T03:54:56Z
author_association: OWNER
body:
  Should this differentiate between preparing the data to be sent back as JSON and preparing the context for the template?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-383139889
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MzEzOTg4OQ==
created_at: 2018-04-20T15:51:47Z
updated_at: 2018-04-20T15:51:47Z
author_association: OWNER
body:
  I released everything we have so far in [Datasette 0.20](https://github.com/simonw/datasette/releases/tag/0.20) and built and released an example plugin, [datasette-cluster-map](https://pypi.org/project/datasette-cluster-map/). Here's my blog entry about it: https://simonwillison.net/2018/Apr/20/datasette-plugins/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/14#issuecomment-383140111
issue_url: https://api.github.com/repos/simonw/datasette/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDM4MzE0MDExMQ==
created_at: 2018-04-20T15:52:33Z
updated_at: 2018-04-20T15:52:33Z
author_association: OWNER
body:
  Here's a link demonstrating my new plugin: https://datasette-cluster-map-demo.now.sh/polar-bears-455fe3a/USGS_WC_eartags_output_files_2009-2011-Status

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/231#issuecomment-383315348
issue_url: https://api.github.com/repos/simonw/datasette/issues/231
node_id: MDEyOklzc3VlQ29tbWVudDM4MzMxNTM0OA==
created_at: 2018-04-21T17:37:50Z
updated_at: 2018-04-22T23:06:04Z
author_association: OWNER
body:
  I could also have an `"autodetect": false` option for that plugin to turn off autodetecting entirely.

  Would be useful if the plugin didn't append its JavaScript in pages that it wasn't used for - that might require making the `extra_js_urls()` hook optionally aware of the columns and table and metadata.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/234#issuecomment-383398182
issue_url: https://api.github.com/repos/simonw/datasette/issues/234
node_id: MDEyOklzc3VlQ29tbWVudDM4MzM5ODE4Mg==
created_at: 2018-04-22T17:31:12Z
updated_at: 2018-04-22T17:31:12Z
author_association: OWNER
body:
  ```{
      "databases": {
          "database1": {
              "tables": {
                  "example_table": {
                      "label_column": "name"
                  }
              }
          }
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/234#issuecomment-383399762
issue_url: https://api.github.com/repos/simonw/datasette/issues/234
node_id: MDEyOklzc3VlQ29tbWVudDM4MzM5OTc2Mg==
created_at: 2018-04-22T17:54:39Z
updated_at: 2018-04-22T17:54:39Z
author_association: OWNER
body:
  Docs here: http://datasette.readthedocs.io/en/latest/metadata.html#specifying-the-label-column-for-a-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/234#issuecomment-383410146
issue_url: https://api.github.com/repos/simonw/datasette/issues/234
node_id: MDEyOklzc3VlQ29tbWVudDM4MzQxMDE0Ng==
created_at: 2018-04-22T20:32:30Z
updated_at: 2018-04-22T20:47:02Z
author_association: OWNER
body:
  I built this wrong: my implementation is looking for the `label_column` on the table-being-displayed, but it should be looking for it on the table-the-foreign-key-links-to.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/235#issuecomment-383727973
issue_url: https://api.github.com/repos/simonw/datasette/issues/235
node_id: MDEyOklzc3VlQ29tbWVudDM4MzcyNzk3Mw==
created_at: 2018-04-23T21:23:59Z
updated_at: 2018-04-23T21:23:59Z
author_association: OWNER
body:
  There might also be something clever we can do here with PRAGMA statements: https://stackoverflow.com/questions/14146881/limit-the-maximum-amount-of-memory-sqlite3-uses

  And https://www.sqlite.org/pragma.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/235#issuecomment-383764533
issue_url: https://api.github.com/repos/simonw/datasette/issues/235
node_id: MDEyOklzc3VlQ29tbWVudDM4Mzc2NDUzMw==
created_at: 2018-04-24T00:30:02Z
updated_at: 2018-04-24T00:30:02Z
author_association: OWNER
body:
  The `resource` module in he standard library has the ability to set limits on memory usage for the current process: https://pymotw.com/2/resource/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/238#issuecomment-384362028
issue_url: https://api.github.com/repos/simonw/datasette/issues/238
node_id: MDEyOklzc3VlQ29tbWVudDM4NDM2MjAyOA==
created_at: 2018-04-25T17:07:11Z
updated_at: 2018-04-25T17:07:11Z
author_association: OWNER
body:
  On further thought: this is actually only an issue for immutable deployments to platforms like Zeit Now and Heroku.

  As such, adding it to `datasette serve` feels clumsy. Maybe `datasette publish` should instead gain the ability to optionally install an extra mechanism that periodically pulls a fresh copy of `metadata.json` from a URL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/239#issuecomment-384500327
issue_url: https://api.github.com/repos/simonw/datasette/issues/239
node_id: MDEyOklzc3VlQ29tbWVudDM4NDUwMDMyNw==
created_at: 2018-04-26T03:18:12Z
updated_at: 2018-04-26T03:18:20Z
author_association: OWNER
body:
  ```
  {
      "databases": {
          "database1": {
              "tables": {
                  "example_table": {
                      "hidden": true
                  }
              }
          }
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/79#issuecomment-384675792
issue_url: https://api.github.com/repos/simonw/datasette/issues/79
node_id: MDEyOklzc3VlQ29tbWVudDM4NDY3NTc5Mg==
created_at: 2018-04-26T15:08:13Z
updated_at: 2018-04-26T15:08:13Z
author_association: OWNER
body:
  Docs now live at http://datasette.readthedocs.io/

  I still need to document a few more parts of the API before closing this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/125#issuecomment-384678319
issue_url: https://api.github.com/repos/simonw/datasette/issues/125
node_id: MDEyOklzc3VlQ29tbWVudDM4NDY3ODMxOQ==
created_at: 2018-04-26T15:14:31Z
updated_at: 2018-04-26T15:14:31Z
author_association: OWNER
body:
  I shipped this last week as the first plugin: https://simonwillison.net/2018/Apr/20/datasette-plugins/

  Demo: https://datasette-cluster-map-demo.datasettes.com/polar-bears-455fe3a/USGS_WC_eartags_output_files_2009-2011-Status

  Plugin: https://github.com/simonw/datasette-cluster-map

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/245#issuecomment-386310149
issue_url: https://api.github.com/repos/simonw/datasette/issues/245
node_id: MDEyOklzc3VlQ29tbWVudDM4NjMxMDE0OQ==
created_at: 2018-05-03T14:14:33Z
updated_at: 2018-05-03T14:14:33Z
author_association: OWNER
body:
  Demos:

  * https://datasette-versions-and-shape-demo.now.sh/sf-trees-02c8ef1/qSpecies.json?_shape=array
  * https://datasette-versions-and-shape-demo.now.sh/sf-trees-02c8ef1/qSpecies.json?_shape=object
  * https://datasette-versions-and-shape-demo.now.sh/sf-trees-02c8ef1/qSpecies.json?_shape=arrays
  * https://datasette-versions-and-shape-demo.now.sh/sf-trees-02c8ef1/qSpecies.json?_shape=objects

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/248#issuecomment-386357645
issue_url: https://api.github.com/repos/simonw/datasette/issues/248
node_id: MDEyOklzc3VlQ29tbWVudDM4NjM1NzY0NQ==
created_at: 2018-05-03T16:36:59Z
updated_at: 2018-05-03T16:36:59Z
author_association: OWNER
body:
  Even better: use `plugin_manager.list_plugin_distinfo()` from pluggy to get back a list of tuples, the second item in each tuple is a `pkg_resources.DistInfoDistribution` with a `.version` attribute.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/237#issuecomment-386840806
issue_url: https://api.github.com/repos/simonw/datasette/issues/237
node_id: MDEyOklzc3VlQ29tbWVudDM4Njg0MDgwNg==
created_at: 2018-05-05T22:56:42Z
updated_at: 2018-05-05T22:56:42Z
author_association: OWNER
body:
  Demo:

      datasette publish now ../datasettes/san-francisco/sf-film-locations.db --branch=master --name datasette-column-search-demo

  https://datasette-column-search-demo.now.sh/sf-film-locations/Film_Locations_in_San_Francisco?_search_Locations=justin

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/251#issuecomment-386879509
issue_url: https://api.github.com/repos/simonw/datasette/issues/251
node_id: MDEyOklzc3VlQ29tbWVudDM4Njg3OTUwOQ==
created_at: 2018-05-06T13:29:26Z
updated_at: 2018-05-06T13:29:26Z
author_association: OWNER
body:
  We can solve this using the `sqlite_timelimit(conn, 20)` helper, which can tell SQLite to give up after 20ms. We can wrap that around the following SQL:

      select distinct COLUMN from TABLE limit 21;

  Then we look at the number of rows returned. If it's 21 or more we know that this table had more than 21 distinct values, so we'll treat it as "unlimited". Likewise, if the SQL times out before 20ms is up we will skip this introspection.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/251#issuecomment-386879840
issue_url: https://api.github.com/repos/simonw/datasette/issues/251
node_id: MDEyOklzc3VlQ29tbWVudDM4Njg3OTg0MA==
created_at: 2018-05-06T13:34:24Z
updated_at: 2018-05-06T13:34:24Z
author_association: OWNER
body:
  Here's a quick demo of that exploration: https://datasette-distinct-column-values.now.sh/-/inspect

  Example output:

  ```
  {
      "antiquities-act/actions_under_antiquities_act": {
          "columns": [
              "current_name",
              "states",
              "original_name",
              "current_agency",
              "action",
              "date",
              "year",
              "pres_or_congress",
              "acres_affected"
          ],
          "count": 344,
          "distinct_values_by_column": {
              "acres_affected": null,
              "action": null,
              "current_agency": [
                  "NPS",
                  "State of Montana",
                  "BLM",
                  "State of Arizona",
                  "USFS",
                  "State of North Dakota",
                  "NPS, BLM",
                  "State of South Carolina",
                  "State of New York",
                  "FWS",
                  "FWS, NOAA",
                  "NPS, FWS",
                  "NOAA",
                  "BLM, USFS",
                  "NOAA, FWS"
              ],
              "current_name": null,
              "date": null,
              "original_name": null,
              "pres_or_congress": null,
              "states": null,
              "year": null
          },
          "foreign_keys": {
              "incoming": [],
              "outgoing": []
          },
          "fts_table": null,
          "hidden": false,
          "label_column": null,
          "name": "antiquities-act/actions_under_antiquities_act",
          "primary_keys": []
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/251#issuecomment-386879878
issue_url: https://api.github.com/repos/simonw/datasette/issues/251
node_id: MDEyOklzc3VlQ29tbWVudDM4Njg3OTg3OA==
created_at: 2018-05-06T13:34:57Z
updated_at: 2018-05-06T13:34:57Z
author_association: OWNER
body:
  If I'm going to expand column introspection in this way it would be useful to also capture column type information.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/254#issuecomment-388360255
issue_url: https://api.github.com/repos/simonw/datasette/issues/254
node_id: MDEyOklzc3VlQ29tbWVudDM4ODM2MDI1NQ==
created_at: 2018-05-11T13:16:09Z
updated_at: 2018-05-11T22:45:31Z
author_association: OWNER
body:
  Do you have an example I can look at?

  I think I have a possible route for fixing this, but it's pretty tricky (it involves adding a full SQL statement parser, but that's needed for some other potential improvements as well).

  In the meantime, is this causing actual errors for you or is it more of an inconvenience (form fields being displayed that don't actually do anything)?

  Another potential solution here could be to allow canned queries to optionally declare their parameters in metadata.json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/254#issuecomment-388367027
issue_url: https://api.github.com/repos/simonw/datasette/issues/254
node_id: MDEyOklzc3VlQ29tbWVudDM4ODM2NzAyNw==
created_at: 2018-05-11T13:41:46Z
updated_at: 2018-05-11T13:41:46Z
author_association: NONE
body:
  An example deployment @ https://datasette-zkcvlwdrhl.now.sh/simplestreams-270f20c/cloudimage?content_id__exact=com.ubuntu.cloud%3Areleased%3Adownload

  It is not causing errors, more of an inconvenience. I have worked around it using a `like` query instead. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/254#issuecomment-388497467
issue_url: https://api.github.com/repos/simonw/datasette/issues/254
node_id: MDEyOklzc3VlQ29tbWVudDM4ODQ5NzQ2Nw==
created_at: 2018-05-11T22:06:00Z
updated_at: 2018-05-11T22:06:34Z
author_association: OWNER
body:
  Got it, this seems to trigger the problem: https://datasette-zkcvlwdrhl.now.sh/simplestreams-270f20c?sql=select+*+from+cloudimage+where+%22content_id%22+%3D+%22com.ubuntu.cloud%3Areleased%3Adownload%22+order+by+id+limit+10

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/255#issuecomment-388588998
issue_url: https://api.github.com/repos/simonw/datasette/issues/255
node_id: MDEyOklzc3VlQ29tbWVudDM4ODU4ODk5OA==
created_at: 2018-05-12T22:57:30Z
updated_at: 2018-05-12T23:00:24Z
author_association: OWNER
body:
  A few demos:

  * https://datasette-facets-demo.now.sh/fivethirtyeight-2628db9/college-majors%2Fall-ages?_facet=Major_category
  * https://datasette-facets-demo.now.sh/fivethirtyeight-2628db9/congress-age%2Fcongress-terms?_facet=chamber&_facet=state&_facet=party&_facet=incumbent
  * https://datasette-facets-demo.now.sh/fivethirtyeight-2628db9/bechdel%2Fmovies?_facet=binary&_facet=test

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/255#issuecomment-388589072
issue_url: https://api.github.com/repos/simonw/datasette/issues/255
node_id: MDEyOklzc3VlQ29tbWVudDM4ODU4OTA3Mg==
created_at: 2018-05-12T22:59:07Z
updated_at: 2018-05-12T22:59:07Z
author_association: OWNER
body:
  I need to decide how to display these. They currently look like this:

  https://datasette-facets-demo.now.sh/fivethirtyeight-2628db9/congress-age%2Fcongress-terms?_facet=chamber&_facet=state&_facet=party&_facet=incumbent&state=MO

  ![2018-05-12 at 7 58 pm](https://user-images.githubusercontent.com/9599/39962230-e7bf9e10-561e-11e8-80a7-0941b8991318.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/257#issuecomment-388625703
issue_url: https://api.github.com/repos/simonw/datasette/issues/257
node_id: MDEyOklzc3VlQ29tbWVudDM4ODYyNTcwMw==
created_at: 2018-05-13T13:10:09Z
updated_at: 2018-05-13T13:10:09Z
author_association: OWNER
body:
  I'm still seeing intermittent Python 3.5 failures due to dictionary ordering differences.

  https://travis-ci.org/simonw/datasette/jobs/378356802

  ```
  >       assert expected_facet_results == facet_results
  E       AssertionError: assert {'city': [{'c...alue': 'MI'}]} == {'city': [{'co...alue': 'MI'}]}
  E         Omitting 1 identical items, use -vv to show
  E         Differing items:
  E         {'city': [{'count': 4, 'toggle_url': '_facet=state&_facet=city&state=MI&city=Detroit', 'value': 'Detroit'}]} != {'city': [{'count': 4, 'toggle_url': 'state=MI&_facet=state&_facet=city&city=Detroit', 'value': 'Detroit'}]}
  E         Use -v to get the full diff
  ```

  To solve these cleanly I need to be able to run Python 3.5 on my local laptop rather than relying on Travis every time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/257#issuecomment-388626721
issue_url: https://api.github.com/repos/simonw/datasette/issues/257
node_id: MDEyOklzc3VlQ29tbWVudDM4ODYyNjcyMQ==
created_at: 2018-05-13T13:27:04Z
updated_at: 2018-05-13T13:27:04Z
author_association: OWNER
body:
  I managed to get Python 3.5.0 running on my laptop using [pyenv](https://github.com/pyenv/pyenv). Here's the incantation I used:

  ```
  # Install pyenv using homebrew (turns out I already had it)
  brew install pyenv
  # Check which versions of Python I have installed
  pyenv versions
  # Install Python 3.5.0
  pyenv install 3.5.0
  # Figure out where pyenv has been installing things
  pyenv root
  # Check I can run my newly installed Python 3.5.0
  /Users/simonw/.pyenv/versions/3.5.0/bin/python
  # Use it to create a new virtualenv
  /Users/simonw/.pyenv/versions/3.5.0/bin/python -mvenv venv35
  source venv35/bin/activate
  # Install datasette into that virtualenv
  python setup.py install
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/257#issuecomment-388626804
issue_url: https://api.github.com/repos/simonw/datasette/issues/257
node_id: MDEyOklzc3VlQ29tbWVudDM4ODYyNjgwNA==
created_at: 2018-05-13T13:28:20Z
updated_at: 2018-05-13T13:28:20Z
author_association: OWNER
body:
  Unfortunately, running `python setup.py test` on my laptop using Python 3.5.0 in that virtualenv results in a flow of weird Sanic-related errors:

  ```
    File "/Users/simonw/Dropbox/Development/datasette/venv35/lib/python3.5/site-packages/sanic-0.7.0-py3.5.egg/sanic/testing.py", line 16, in _local_request
      import aiohttp
    File "/Users/simonw/Dropbox/Development/datasette/.eggs/aiohttp-2.3.2-py3.5-macosx-10.13-x86_64.egg/aiohttp/__init__.py", line 6, in <module>
      from .client import *  # noqa
    File "/Users/simonw/Dropbox/Development/datasette/.eggs/aiohttp-2.3.2-py3.5-macosx-10.13-x86_64.egg/aiohttp/client.py", line 13, in <module>
      from yarl import URL
    File "/Users/simonw/Dropbox/Development/datasette/.eggs/yarl-1.2.4-py3.5-macosx-10.13-x86_64.egg/yarl/__init__.py", line 11, in <module>
      from .quoting import _Quoter, _Unquoter
    File "/Users/simonw/Dropbox/Development/datasette/.eggs/yarl-1.2.4-py3.5-macosx-10.13-x86_64.egg/yarl/quoting.py", line 3, in <module>
      from typing import Optional, TYPE_CHECKING, cast
  ImportError: cannot import name 'TYPE_CHECKING'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/257#issuecomment-388627281
issue_url: https://api.github.com/repos/simonw/datasette/issues/257
node_id: MDEyOklzc3VlQ29tbWVudDM4ODYyNzI4MQ==
created_at: 2018-05-13T13:36:21Z
updated_at: 2018-05-13T13:36:21Z
author_association: OWNER
body:
  https://github.com/rtfd/readthedocs.org/issues/3812#issuecomment-373780860 suggests Python 3.5.2 may have the fix.

  Yup, that worked:

  ```
  pyenv install 3.5.2
  rm -rf venv35
  /Users/simonw/.pyenv/versions/3.5.2/bin/python -mvenv venv35
  source venv35/bin/activate
  # Not sure why I need this in my local environment but I do:
  pip install datasette_plugin_demos
  python setup.py test
  ```

  This is now giving me the same test failure locally that I am seeing in Travis.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/257#issuecomment-388628966
issue_url: https://api.github.com/repos/simonw/datasette/issues/257
node_id: MDEyOklzc3VlQ29tbWVudDM4ODYyODk2Ng==
created_at: 2018-05-13T14:00:47Z
updated_at: 2018-05-13T14:06:35Z
author_association: OWNER
body:
  Running specific tests:
  ```
  venv35/bin/pip install pytest beautifulsoup4 aiohttp
  venv35/bin/pytest tests/test_utils.py
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/255#issuecomment-388645828
issue_url: https://api.github.com/repos/simonw/datasette/issues/255
node_id: MDEyOklzc3VlQ29tbWVudDM4ODY0NTgyOA==
created_at: 2018-05-13T18:18:56Z
updated_at: 2018-05-13T18:20:02Z
author_association: OWNER
body:
  I may be able to run the SQL for all of the facet counts in one go using a WITH CTE query - will have to microbenchmark this to make sure it is worthwhile: https://datasette-facets-demo.now.sh/fivethirtyeight-2628db9?sql=with+blah+as+%28select+*+from+%5Bcollege-majors%2Fall-ages%5D%29%0D%0Aselect+*+from+%28select+%22Major_category%22%2C+Major_category%2C+count%28*%29+as+n+from%0D%0Ablah+group+by+Major_category+order+by+n+desc+limit+10%29%0D%0Aunion+all%0D%0Aselect+*+from+%28select+%22Major_category2%22%2C+Major_category%2C+count%28*%29+as+n+from%0D%0Ablah+group+by+Major_category+order+by+n+desc+limit+10%29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/256#issuecomment-388684356
issue_url: https://api.github.com/repos/simonw/datasette/issues/256
node_id: MDEyOklzc3VlQ29tbWVudDM4ODY4NDM1Ng==
created_at: 2018-05-14T03:05:37Z
updated_at: 2018-05-14T03:05:37Z
author_association: OWNER
body:
  I just landed pull request #257 - I haven't refactored the tests, I may do that later if it looks worthwhile.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/255#issuecomment-388686463
issue_url: https://api.github.com/repos/simonw/datasette/issues/255
node_id: MDEyOklzc3VlQ29tbWVudDM4ODY4NjQ2Mw==
created_at: 2018-05-14T03:23:44Z
updated_at: 2018-05-14T03:25:22Z
author_association: OWNER
body:
  It would be neat if there was a mechanism for calculating aggregates per facet - e.g. calculating the sum() of specific columns against each facet result on  https://datasette-facets-demo.now.sh/fivethirtyeight-2628db9/nba-elo%2Fnbaallelo?_facet=lg_id&_facet=fran_id&lg_id=ABA&_facet=team_id

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/255#issuecomment-388784063
issue_url: https://api.github.com/repos/simonw/datasette/issues/255
node_id: MDEyOklzc3VlQ29tbWVudDM4ODc4NDA2Mw==
created_at: 2018-05-14T11:25:00Z
updated_at: 2018-05-14T11:25:15Z
author_association: OWNER
body:
  Can I get facets working across many2many relationships?

  This would be fiendishly useful, but the querystring and `metadata.json` syntax is non-obvious.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/255#issuecomment-388784787
issue_url: https://api.github.com/repos/simonw/datasette/issues/255
node_id: MDEyOklzc3VlQ29tbWVudDM4ODc4NDc4Nw==
created_at: 2018-05-14T11:28:05Z
updated_at: 2018-05-14T11:28:05Z
author_association: OWNER
body:
  To decide which facets to suggest: for each column, is the unique value count less than the number of rows matching the current query or is it less than 20 (if we are showing more than 20 rows)?

  Maybe only do this if there are less than ten non-float columns. Or always try for foreign keys and booleans, then if there are none of those try indexed text and integer fields, then finally try non-indexed text and integer fields but only if there are less than ten.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/259#issuecomment-388797919
issue_url: https://api.github.com/repos/simonw/datasette/issues/259
node_id: MDEyOklzc3VlQ29tbWVudDM4ODc5NzkxOQ==
created_at: 2018-05-14T12:23:11Z
updated_at: 2018-05-14T12:23:11Z
author_association: OWNER
body:
  For M2M to work we will need a mechanism for applying IN queries to the table view, so you can select multiple M2M filters. Maybe this would work:

      ?_m2m_category=123&_m2m_category=865

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/255#issuecomment-389147608
issue_url: https://api.github.com/repos/simonw/datasette/issues/255
node_id: MDEyOklzc3VlQ29tbWVudDM4OTE0NzYwOA==
created_at: 2018-05-15T12:24:46Z
updated_at: 2018-05-15T12:24:46Z
author_association: OWNER
body:
  New demo (published with `datasette publish now --branch=suggested-facets fivethirtyeight.db sf-trees.db --name=datastte-suggested-facets-demo`): https://datasette-suggested-facets-demo.now.sh/fivethirtyeight-2628db9/comic-characters%2Fmarvel-wikia-data

  After turning on a couple of suggested facets... https://datasette-suggested-facets-demo.now.sh/fivethirtyeight-2628db9/comic-characters%2Fmarvel-wikia-data?_facet=SEX&_facet=ID

  ![2018-05-15 at 7 24 am](https://user-images.githubusercontent.com/9599/40056411-fa265d16-5810-11e8-89ec-e38fe29ffb2c.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/258#issuecomment-389386142
issue_url: https://api.github.com/repos/simonw/datasette/issues/258
node_id: MDEyOklzc3VlQ29tbWVudDM4OTM4NjE0Mg==
created_at: 2018-05-16T03:51:13Z
updated_at: 2018-05-16T03:51:13Z
author_association: OWNER
body:
  The URL does persist across deployments already, in that you can use the URL without the hash and it will redirect to the current location. Here's an example of that: https://san-francisco.datasettes.com/sf-trees/Street_Tree_List.json

  This also works if you attempt to hit the incorrect hash, e.g. if you have deployed a new version of the database with an updated hash. The old hash will redirect, e.g. https://san-francisco.datasettes.com/sf-trees-c4b972c/Street_Tree_List.json

  If you serve Datasette from a HTTP/2 proxy (I've been using Cloudflare for this) you won't even have to pay the cost of the redirect - Datasette sends a `Link: <URL>; rel=preload` header with those redirects, which causes Cloudflare to push out the redirected source as part of that HTTP/2 request. You can fire up the Chrome DevTools to watch this happen.

  https://github.com/simonw/datasette/blob/2b79f2bdeb1efa86e0756e741292d625f91cb93d/datasette/views/base.py#L91

  All of that said... I'm not at all opposed to this feature. For consistency with other Datasette options (e.g. `--cors`) I'd prefer to do this as an optional argument to the `datasette serve` command - something like this:

      datasette serve mydb.db --no-url-hash

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/255#issuecomment-389386919
issue_url: https://api.github.com/repos/simonw/datasette/issues/255
node_id: MDEyOklzc3VlQ29tbWVudDM4OTM4NjkxOQ==
created_at: 2018-05-16T03:57:47Z
updated_at: 2018-05-16T03:58:30Z
author_association: OWNER
body:
  I updated that demo to demonstrate the new foreign key label expansions: https://datasette-suggested-facets-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List?_facet=qLegalStatus

  ![2018-05-15 at 8 58 pm](https://user-images.githubusercontent.com/9599/40095806-b645026a-5882-11e8-8100-76136df50212.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/258#issuecomment-389536870
issue_url: https://api.github.com/repos/simonw/datasette/issues/258
node_id: MDEyOklzc3VlQ29tbWVudDM4OTUzNjg3MA==
created_at: 2018-05-16T14:22:31Z
updated_at: 2018-05-16T14:22:31Z
author_association: OWNER
body:
  The principle benefit provided by the hash URLs is that Datasette can set a far-future cache expiry header on every response. This is particularly useful for JavaScript API work as it makes fantastic use of the browser's cache. It also means that if you are serving your API from behind a caching proxy like Cloudflare you get a fantastic cache hit rate.

  An option to serve without persistent hashes would also need to turn off the cache headers.

  Maybe the option should support both? If you hit a page with the hash in the URL you still get the cache headers, but hits to the URL without the hash serve uncashed content directly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/255#issuecomment-389546040
issue_url: https://api.github.com/repos/simonw/datasette/issues/255
node_id: MDEyOklzc3VlQ29tbWVudDM4OTU0NjA0MA==
created_at: 2018-05-16T14:47:34Z
updated_at: 2018-05-16T14:47:34Z
author_association: OWNER
body:
  Latest demo - now with multiple columns: https://datasette-suggested-facets-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List?_facet=qCaretaker&_facet=qCareAssistant&_facet=qLegalStatus

  ![2018-05-16 at 7 47 am](https://user-images.githubusercontent.com/9599/40124418-63e680ba-58dd-11e8-8063-9686826abb8e.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/265#issuecomment-389566147
issue_url: https://api.github.com/repos/simonw/datasette/issues/265
node_id: MDEyOklzc3VlQ29tbWVudDM4OTU2NjE0Nw==
created_at: 2018-05-16T15:41:42Z
updated_at: 2018-05-16T15:41:42Z
author_association: OWNER
body:
  An official demo instance of Datasette dedicated to this use-case would be useful, especially if it was automatically deployed by Travis for every commit to master that passes the tests.

  Maybe there should be a permanent version of it deployed for each released version too?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-389570841
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM4OTU3MDg0MQ==
created_at: 2018-05-16T15:54:49Z
updated_at: 2018-06-15T07:41:09Z
author_association: OWNER
body:
  At the most basic level, this will work based on an extension. Most places you currently put a `.json` extension should also allow a `.csv` extension.

  By default this will return the exact results you see on the current page (default max will remain 1000).

  ## Streaming all records

  Where things get interested is *streaming mode*. This will be an option which returns ALL matching records as a streaming CSV file, even if that ends up being millions of records.

  I think the best way to build this will be on top of the existing mechanism used to efficiently implement keyset pagination via `_next=` tokens.

  ## Expanding foreign keys

  For tables with foreign key references it would be useful if the CSV format could expand those references to include the labels from `label_column` - maybe via an additional `?_expand=1` option.

  When expanding each foreign key column will be shown twice:

      rowid,city_id,city_id_label,state

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-389572201
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM4OTU3MjIwMQ==
created_at: 2018-05-16T15:58:43Z
updated_at: 2018-05-16T16:00:47Z
author_association: OWNER
body:
  This will likely be implemented in the `BaseView` class, which needs to know how to spot the `.csv` extension, call the underlying JSON generating function and then return the `columns` and `rows` as correctly formatted CSV.

  https://github.com/simonw/datasette/blob/9959a9e4deec8e3e178f919e8b494214d5faa7fd/datasette/views/base.py#L201-L207

  This means it will take ALL arguments that are available to the `.json` view. It may ignore some (e.g. `_facet=` makes no sense since CSV tables don't have space to show the facet results).

  In streaming mode, things will behave a little bit differently - in particular, if `_stream=1` then `_next=` will be forbidden.

  It can't include a length header because we don't know how many bytes it will be

  CSV output will throw an error if the endpoint doesn't have rows and columns keys eg `/-/inspect.json`

  So the implementation...

  - looks for the `.csv` extension
  - internally fetches the `.json` data instead
  - If no `_stream` it just transposes that JSON to CSV with the correct content type header
  - If `_stream=1` - checks for `_next=` and throws an error if it was provided
  - Otherwise... fetch first page and emit CSV header and first set of rows
  - Then start async looping, emitting more CSV rows and following the `_next=` internal reference until done

  I like that this takes advantage of efficient pagination. It may not work so well for views which use offset/limit though.

  It won't work at all for custom SQL because custom SQL doesn't support _next= pagination. That's fine.

  For views... easiest fix is to cut off after first X000 records. That seems OK. View JSON would need to include a property that the mechanism can identify.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-389579363
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM4OTU3OTM2Mw==
created_at: 2018-05-16T16:20:06Z
updated_at: 2018-05-16T16:20:06Z
author_association: OWNER
body:
  I started a thread on Twitter discussing various CSV output dialects: https://twitter.com/simonw/status/996783395504979968 - I want to pick defaults which will work as well as possible for whatever tools people might be using to consume the data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-389579762
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM4OTU3OTc2Mg==
created_at: 2018-05-16T16:21:12Z
updated_at: 2018-05-16T16:21:12Z
author_association: OWNER
body:
  > I basically want someone to tell me which arguments I can pass to Python's csv.writer() function that will result in the least complaints from people who try to parse the results :)
  https://twitter.com/simonw/status/996786815938977792

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-389592566
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM4OTU5MjU2Ng==
created_at: 2018-05-16T17:01:29Z
updated_at: 2018-05-16T17:02:21Z
author_association: OWNER
body:
  Let's provide a CSV Dialect definition too: https://frictionlessdata.io/specs/csv-dialect/ - via https://twitter.com/drewdaraabrams/status/996794915680997382

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-389608473
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM4OTYwODQ3Mw==
created_at: 2018-05-16T17:52:35Z
updated_at: 2018-05-16T17:54:11Z
author_association: OWNER
body:
  There are some code examples in this issue which should help with the streaming part: https://github.com/channelcat/sanic/issues/1067

  Also https://github.com/channelcat/sanic/blob/master/docs/sanic/streaming.md#response-streaming

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-389626715
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM4OTYyNjcxNQ==
created_at: 2018-05-16T18:50:46Z
updated_at: 2018-05-16T18:50:46Z
author_association: OWNER
body:
  > I’d recommend using the Windows-1252 encoding for maximum compatibility, unless you have any characters not in that set, in which case use UTF8 with a byte order mark. Bit of a pain, but some progams (eg various versions of Excel) don’t read UTF8.
  **frankieroberto** https://twitter.com/frankieroberto/status/996823071947460616

  > There is software that consumes CSV and doesn't speak UTF8!? Huh. Well I can't just use Windows-1252 because I need to support the full UTF8 range of potential data - maybe I should support an optional ?_encoding=windows-1252 argument
  **simonw** https://twitter.com/simonw/status/996824677245857793

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-389702480
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: MDEyOklzc3VlQ29tbWVudDM4OTcwMjQ4MA==
created_at: 2018-05-17T00:00:39Z
updated_at: 2020-09-12T18:19:30Z
author_association: OWNER
body:
  Idea: `?_extra=sqllog` could output a lot of every individual SQL statement that was executed in order to generate the page - useful for seeing how foreign key expansion and faceting actually works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-389893810
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM4OTg5MzgxMA==
created_at: 2018-05-17T14:49:35Z
updated_at: 2018-05-17T14:49:35Z
author_association: OWNER
body:
  Idea: add a `supports_csv = False` property to `BaseView` and over-ride it to `True` just on the view classes that should support CSV (Table and Row). Slight subtlety: the `DatabaseView` class only supports CSV in the `custom_sql()` path. Maybe that needs to be refactored a bit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-389894382
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM4OTg5NDM4Mg==
created_at: 2018-05-17T14:51:13Z
updated_at: 2018-05-17T14:53:23Z
author_association: OWNER
body:
  I should definitely sanity check if the `_next=` route really is the most efficient way to build this. It may turn out that iterating over a SQLite cursor with a million rows in it is super-efficient and would provide much more reliable performance (plus solve the problem for retrieving full custom SQL queries where we can't do keyset pagination).

  Problem here is that we run SQL queries in a thread pool. A query that returns millions of rows would presumably tie up a SQL thread until it has finished, which could block the server. This may be a reason to stick with `_next=` keyset pagination - since it ensures each SQL thread yields back again after each 1,000 rows.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/271#issuecomment-389989015
issue_url: https://api.github.com/repos/simonw/datasette/issues/271
node_id: MDEyOklzc3VlQ29tbWVudDM4OTk4OTAxNQ==
created_at: 2018-05-17T19:54:10Z
updated_at: 2018-05-17T19:54:10Z
author_association: OWNER
body:
  This is a departure from how Datasette has been designed so far, and it may turn out that it's not feasible or it requires too many philosophical changes to be worthwhile.

  If we CAN do it though it would mean Datasette could stay running pointed at a directory on disk and new SQLite databases could be dropped into that directory by another process and served directly as they become available.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/271#issuecomment-389989615
issue_url: https://api.github.com/repos/simonw/datasette/issues/271
node_id: MDEyOklzc3VlQ29tbWVudDM4OTk4OTYxNQ==
created_at: 2018-05-17T19:56:13Z
updated_at: 2018-05-17T19:56:13Z
author_association: OWNER
body:
  From https://www.sqlite.org/c3ref/open.html

  > **immutable**: The immutable parameter is a boolean query parameter that indicates that the database file is stored on read-only media. When immutable is set, SQLite assumes that the database file cannot be changed, even by a process with higher privilege, and so the database is opened read-only and all locking and change detection is disabled. Caution: Setting the immutable property on a database file that does in fact change can result in incorrect query results and/or SQLITE_CORRUPT errors. See also: SQLITE_IOCAP_IMMUTABLE.

  So this would probably have to be a new mode, `datasette serve --detect-db-changes`, which no longer opens in immutable mode. Or maybe current behavior becomes not-the-default and you opt into it with `datasette serve --immutable`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/273#issuecomment-390250253
issue_url: https://api.github.com/repos/simonw/datasette/issues/273
node_id: MDEyOklzc3VlQ29tbWVudDM5MDI1MDI1Mw==
created_at: 2018-05-18T15:49:52Z
updated_at: 2018-05-18T15:49:52Z
author_association: CONTRIBUTOR
body:
  Shouldn't [versioneer](https://github.com/warner/python-versioneer) do that?

  E.g. 0.21+2.g1076c97

  You'd need to install via `pip install git+https://github.com/simow/datasette.git` though, this does a temp git clone.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/258#issuecomment-390577711
issue_url: https://api.github.com/repos/simonw/datasette/issues/258
node_id: MDEyOklzc3VlQ29tbWVudDM5MDU3NzcxMQ==
created_at: 2018-05-21T07:38:15Z
updated_at: 2018-05-21T07:38:15Z
author_association: NONE
body:
  Excellent, I was not aware of the auto redirect to the new hash. My bad

  This solves my use case.

  I do agree that your suggested --no-url-hash approach is much neater. I will investigate 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/247#issuecomment-390689406
issue_url: https://api.github.com/repos/simonw/datasette/issues/247
node_id: MDEyOklzc3VlQ29tbWVudDM5MDY4OTQwNg==
created_at: 2018-05-21T15:29:31Z
updated_at: 2018-05-21T15:29:31Z
author_association: NONE
body:
  I've changed my mind about the way to support external connectors aside of SQLite and I'm working in a more simple style that respects the original Datasette, i.e. less refactoring. I present you [a version of Datasette wich supports other database connectors](https://github.com/jsancho-gpl/datasette/tree/external-connectors) and [a Datasette connector for HDF5/PyTables files](https://github.com/jsancho-gpl/datasette-pytables).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/277#issuecomment-390707183
issue_url: https://api.github.com/repos/simonw/datasette/issues/277
node_id: MDEyOklzc3VlQ29tbWVudDM5MDcwNzE4Mw==
created_at: 2018-05-21T16:28:39Z
updated_at: 2018-05-21T16:28:39Z
author_association: OWNER
body:
  This is definitely a big improvement.

  I'd like to refactor the unit tests that cover .inspect() too - currently they are a huge ugly blob at the top of test_api.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-390707760
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MDcwNzc2MA==
created_at: 2018-05-21T16:30:35Z
updated_at: 2018-05-21T16:30:35Z
author_association: OWNER
body:
  This probably needs to be in a plugin simply because getting Spatialite compiled and installed is a bit of a pain.

  It's a great opportunity to expand the plugin hooks in useful ways though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-390795067
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MDc5NTA2Nw==
created_at: 2018-05-21T21:55:57Z
updated_at: 2018-05-21T21:55:57Z
author_association: CONTRIBUTOR
body:
  Well, we do have the capability to detect spatialite so my intention certainly wasn't to require it. 

  I can see the advantage of having it as a plugin but it does touch a number of points in the code. I think I'm going to attack this by refactoring the necessary bits and seeing where that leads (which was my plan anyway).

  I think my main concern is - if I add certain plugin hooks for this, is anything else ever going to use them? I'm not sure I have an answer to that question yet, either way.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/277#issuecomment-390804333
issue_url: https://api.github.com/repos/simonw/datasette/issues/277
node_id: MDEyOklzc3VlQ29tbWVudDM5MDgwNDMzMw==
created_at: 2018-05-21T22:40:16Z
updated_at: 2018-05-21T22:43:50Z
author_association: OWNER
body:
  We should merge this before refactoring the tests though, because that way we don't couple the new tests to the verification of this change.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/278#issuecomment-390991640
issue_url: https://api.github.com/repos/simonw/datasette/issues/278
node_id: MDEyOklzc3VlQ29tbWVudDM5MDk5MTY0MA==
created_at: 2018-05-22T13:33:46Z
updated_at: 2018-05-22T13:33:46Z
author_association: OWNER
body:
  For SpatiaLite this example may be useful - though it's building 4.3.0 and not 4.4.0: https://github.com/terranodo/spatialite-docker/blob/master/Dockerfile

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/278#issuecomment-390993397
issue_url: https://api.github.com/repos/simonw/datasette/issues/278
node_id: MDEyOklzc3VlQ29tbWVudDM5MDk5MzM5Nw==
created_at: 2018-05-22T13:38:57Z
updated_at: 2018-05-22T13:38:57Z
author_association: OWNER
body:
  Useful GitHub code search: https://github.com/search?utf8=✓&q=%22libspatialite-4.4.0%22+%22RC0%22&type=Code

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/278#issuecomment-390993861
issue_url: https://api.github.com/repos/simonw/datasette/issues/278
node_id: MDEyOklzc3VlQ29tbWVudDM5MDk5Mzg2MQ==
created_at: 2018-05-22T13:40:14Z
updated_at: 2018-05-22T14:38:05Z
author_association: OWNER
body:
  If we can't get `import sqlite3` to load the latest version but we can get `import pysqlite3` to work that's fine too - I can teach Datasette to import the best available version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/255#issuecomment-390999055
issue_url: https://api.github.com/repos/simonw/datasette/issues/255
node_id: MDEyOklzc3VlQ29tbWVudDM5MDk5OTA1NQ==
created_at: 2018-05-22T13:54:55Z
updated_at: 2018-05-22T13:54:55Z
author_association: OWNER
body:
  This shipped in Datasette 0.22. Here's my blog post about it: https://simonwillison.net/2018/May/20/datasette-facets/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-391000659
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MTAwMDY1OQ==
created_at: 2018-05-22T13:59:27Z
updated_at: 2018-05-22T13:59:27Z
author_association: OWNER
body:
  Right now the plugin stuff is early enough that I'd like to get as many potential plugin hooks as possible crafted out A much easier to judge if they should be added as actual hooks if we have a working branch prototype of them.

  Some kind of mechanism for custom column display is already needed - eg there are columns where I want to say "render this as markdown" or "URLify any links in this text" - or even "use this date format" or "add commas to this integer".

  You can do it with a custom template but a lower-level mechanism would be nicer. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/273#issuecomment-391003285
issue_url: https://api.github.com/repos/simonw/datasette/issues/273
node_id: MDEyOklzc3VlQ29tbWVudDM5MTAwMzI4NQ==
created_at: 2018-05-22T14:06:40Z
updated_at: 2018-05-22T14:06:40Z
author_association: OWNER
body:
  That looks great. I don't think it's possible to derive the current commit version from the .zip downloaded directly from GitHub, so needing to pip install via git+https feels reasonable to me.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-391011268
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDM5MTAxMTI2OA==
created_at: 2018-05-22T14:28:12Z
updated_at: 2018-05-22T14:28:12Z
author_association: OWNER
body:
  I think I can do this almost entirely within my existing BaseView class structure.

  First, decouple the async data() methods by teaching them to take a querystring object as an argument instead of a Sanic request object. The get() method can then send that new object instead of a request.

  Next teach the base class how to obey the ASGI protocol.

  I should be able to get support for both Sanic and uvicorn/daphne working in the same codebase, which will make it easy to compare their performance. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-391025841
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MTAyNTg0MQ==
created_at: 2018-05-22T15:06:36Z
updated_at: 2018-05-22T15:06:36Z
author_association: OWNER
body:
  The other reason I mention plugins is that I have an idea to outlaw JavaScript entirely from Datasette core and instead encourage ALL JavaScript functionality to move into plugins.right now that just means CodeMirror. I may set up some of those plugins (like CodeMirror) as default dependencies so you get them from "pip install datasette".

  I like the neatness of saying that core Datasette is a very simple JSON + HTML application, then encouraging people to go completely wild with JavaScript in the plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-391050113
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MTA1MDExMw==
created_at: 2018-05-22T16:13:00Z
updated_at: 2018-05-22T16:13:00Z
author_association: CONTRIBUTOR
body:
  Yup, I'll have a think about it. My current thoughts are for spatialite we'll need to hook into the following places:

  * Inspection, so we can detect which columns are geometry columns. (We also currently ignore spatialite tables during inspection, it may be worth moving that to the plugin as well.)
  * After data load, so we can convert WKB into the correct intermediate format for display. The alternative here is to alter the select SQL itself and get spatialite to do this conversion, but that strikes me as a bit more complex and possibly not as useful.
  * HTML rendering.
  * Querying?

  The rendering and querying hooks could also potentially be used to move the units support into a plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/279#issuecomment-391055490
issue_url: https://api.github.com/repos/simonw/datasette/issues/279
node_id: MDEyOklzc3VlQ29tbWVudDM5MTA1NTQ5MA==
created_at: 2018-05-22T16:29:30Z
updated_at: 2018-05-22T16:29:30Z
author_association: OWNER
body:
  This is fantastic!

  I think I prefer the aesthetics of just "0.22" for the version string if it's a tagged release with no additional changes - does that work?

  I'd like to continue to provide a tuple that can be imported from the version.py module as well, as seen here:

  https://github.com/simonw/datasette/blob/558d9d7bfef3dd633eb16389281b67d42c9bdeef/datasette/version.py#L1

  Presumably we can generate that from the versioneer string?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/280#issuecomment-391059008
issue_url: https://api.github.com/repos/simonw/datasette/issues/280
node_id: MDEyOklzc3VlQ29tbWVudDM5MTA1OTAwOA==
created_at: 2018-05-22T16:40:27Z
updated_at: 2018-05-22T16:40:27Z
author_association: CONTRIBUTOR
body:
  ```python
  >>> import sqlite3
  >>> sqlite3.sqlite_version
  '3.23.1'
  >>> 
  ```
  running the above in the container seems to show 3.23.1 too so maybe we don't need pysqlite3 at all?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/279#issuecomment-391073009
issue_url: https://api.github.com/repos/simonw/datasette/issues/279
node_id: MDEyOklzc3VlQ29tbWVudDM5MTA3MzAwOQ==
created_at: 2018-05-22T17:23:26Z
updated_at: 2018-05-22T17:23:26Z
author_association: CONTRIBUTOR
body:
  > I think I prefer the aesthetics of just "0.22" for the version string if it's a tagged release with no additional changes - does that work?

  Yes! That's the default versioneer behaviour.

  > I'd like to continue to provide a tuple that can be imported from the version.py module as well, as seen here:

  Should work now, it can be a two (for a tagged version), three or four items tuple.

  ```
  In [2]: datasette.__version__
  Out[2]: '0.12+292.ga70c2a8.dirty'

  In [3]: datasette.__version_info__
  Out[3]: ('0', '12+292', 'ga70c2a8', 'dirty')
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/280#issuecomment-391141391
issue_url: https://api.github.com/repos/simonw/datasette/issues/280
node_id: MDEyOklzc3VlQ29tbWVudDM5MTE0MTM5MQ==
created_at: 2018-05-22T21:08:39Z
updated_at: 2018-05-22T21:08:39Z
author_association: CONTRIBUTOR
body:
  I'm going to clean this up for consistency tomorrow morning so hold off
  merging until then please

  On Tue, May 22, 2018 at 6:34 PM, Simon Willison <notifications@github.com>
  wrote:

  > Yeah let's try this without pysqlite3 and see if we still get the correct
  > version.
  >
  > —
  > You are receiving this because you authored the thread.
  > Reply to this email directly, view it on GitHub
  > <https://github.com/simonw/datasette/pull/280#issuecomment-391076458>, or mute
  > the thread
  > <https://github.com/notifications/unsubscribe-auth/AAihfMI-H6CBt-Py0xdBbH2xDK0KsjT2ks5t1EwYgaJpZM4UI_2m>
  > .
  >

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/280#issuecomment-391190497
issue_url: https://api.github.com/repos/simonw/datasette/issues/280
node_id: MDEyOklzc3VlQ29tbWVudDM5MTE5MDQ5Nw==
created_at: 2018-05-23T01:22:53Z
updated_at: 2018-05-23T01:22:53Z
author_association: OWNER
body:
  I grabbed just your Dockerfile and built it like this:

      docker build . -t datasette

  Once it had built, I ran it like this:

      docker run -p 8001:8001 -v `pwd`:/mnt datasette \
          datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db \
              --load-extension=/usr/local/lib/mod_spatialite.so

  (The fixtures.db file is created by running `python tests/fixtures.py fixtures.db`)

  Then I visited http://localhost:8001/-/versions and I got this:

      {
          "datasette": {
              "version": "0+unknown"
          },
          "python": {
              "full": "3.6.3 (default, Dec 12 2017, 06:37:05) \n[GCC 6.3.0 20170516]",
              "version": "3.6.3"
          },
          "sqlite": {
              "extensions": {
                  "json1": null,
                  "spatialite": "4.4.0-RC0"
              },
              "fts_versions": [
                  "FTS4",
                  "FTS3"
              ],
              "version": "3.23.1"
          }
      }

  Fantastic! I'm getting SQLite `3.23.1` and SpatiaLite `4.4.0-RC0`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/280#issuecomment-391290271
issue_url: https://api.github.com/repos/simonw/datasette/issues/280
node_id: MDEyOklzc3VlQ29tbWVudDM5MTI5MDI3MQ==
created_at: 2018-05-23T09:53:38Z
updated_at: 2018-05-23T09:53:38Z
author_association: CONTRIBUTOR
body:
  Running:
  ```bash
  docker run -p 8001:8001 -v `pwd`:/mnt datasette \
      datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db \
          --load-extension=/usr/local/lib/mod_spatialite.so
  ```

  is now returning FTS5 enabled in the versions output:

  ```json
  {
      "datasette": {
          "version": "0.22"
      },
      "python": {
          "full": "3.6.5 (default, May  5 2018, 03:07:21) \n[GCC 6.3.0 20170516]",
          "version": "3.6.5"
      },
      "sqlite": {
          "extensions": {
              "json1": null,
              "spatialite": "4.4.0-RC0"
          },
          "fts_versions": [
              "FTS5",
              "FTS4",
              "FTS3"
          ],
          "version": "3.23.1"
      }
  }
  ```
  The old query didn't work because specifying `(t TEXT)` caused an error

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/280#issuecomment-391355030
issue_url: https://api.github.com/repos/simonw/datasette/issues/280
node_id: MDEyOklzc3VlQ29tbWVudDM5MTM1NTAzMA==
created_at: 2018-05-23T13:53:27Z
updated_at: 2018-05-23T15:22:45Z
author_association: CONTRIBUTOR
body:
  No objections;
  It's good to go @simonw

  On Wed, 23 May 2018, 14:51 Simon Willison, <notifications@github.com> wrote:

  > @r4vi <https://github.com/r4vi> any objections to me merging this?
  >
  > —
  > You are receiving this because you were mentioned.
  > Reply to this email directly, view it on GitHub
  > <https://github.com/simonw/datasette/pull/280#issuecomment-391354237>, or mute
  > the thread
  > <https://github.com/notifications/unsubscribe-auth/AAihfM_2DN5WR2mkO-VK6ozDmkUQ4IMjks5t1WlcgaJpZM4UI_2m>
  > .
  >

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/282#issuecomment-391355099
issue_url: https://api.github.com/repos/simonw/datasette/issues/282
node_id: MDEyOklzc3VlQ29tbWVudDM5MTM1NTA5OQ==
created_at: 2018-05-23T13:53:39Z
updated_at: 2018-05-23T13:53:39Z
author_association: OWNER
body:
  Confirmed fixed: https://fivethirtyeight-datasette-mipwbeadvr.now.sh/fivethirtyeight-5de27e3/nba-elo%2Fnbaallelo?_facet=lg_id&_next=100 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/281#issuecomment-391437462
issue_url: https://api.github.com/repos/simonw/datasette/issues/281
node_id: MDEyOklzc3VlQ29tbWVudDM5MTQzNzQ2Mg==
created_at: 2018-05-23T17:45:07Z
updated_at: 2018-05-23T17:45:07Z
author_association: OWNER
body:
  I'm afraid I just merged #280 which means this no longer applies. You're very welcome to see if you can further optimize the new Dockerfile though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-391504199
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MTUwNDE5OQ==
created_at: 2018-05-23T21:35:17Z
updated_at: 2018-05-23T21:35:17Z
author_association: OWNER
body:
  I'm not keen on anything that modifies the SQLite file itself on startup - part of the datasette contract is that it should work with any SQLite file you throw at it without having any side-effects.

  A neat thing about SQLite is that because everything happens in the same process there's very little additional overhead involved in executing extra SQL queries - even if we ran a query-per-row to transform data in one specific column it shouldn't add more than a few ms to the total page load time (whereas with MySQL all of the extra query overhead would kill us).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-391504757
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MTUwNDc1Nw==
created_at: 2018-05-23T21:37:07Z
updated_at: 2018-05-23T21:37:18Z
author_association: OWNER
body:
  That said, it looks like we may be able to use a library like https://github.com/geomet/geomet to run the conversion from WKB entirely in Python space.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-391505930
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MTUwNTkzMA==
created_at: 2018-05-23T21:41:37Z
updated_at: 2018-05-23T21:41:37Z
author_association: CONTRIBUTOR
body:
  > I'm not keen on anything that modifies the SQLite file itself on startup

  Ah I didn't mean that - I meant altering the SELECT query to fetch the data so that it ran a spatialite function to transform that specific column.

  I think that's less useful as a general-purpose plugin hook though, and it's not that hard to parse the WKB in Python (my default approach would be to use [shapely](https://github.com/Toblerity/Shapely), which is great, but geomet looks like an interesting pure-python alternative).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-391583528
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDM5MTU4MzUyOA==
created_at: 2018-05-24T04:21:49Z
updated_at: 2018-05-24T04:21:49Z
author_association: OWNER
body:
  The challenge here is which database should be the "default" database. The first database attached to SQLite is treated as the default - if no database is specified in a query, that's the database that queries will be executed against.

  Currently, each database URL in Datasette (e.g. https://san-francisco.datasettes.com/sf-film-locations-84594a7 v.s. https://san-francisco.datasettes.com/sf-trees-ebc2ad9 ) gets its own independent connection, and all queries within that base URL run against that database.

  If we're going to attach multiple databases to the same connection, how do we set which database gets to be the default?

  The easiest thing to do here will be to have a special database (maybe which is turned off by default and can be enabled using `datasette serve --enable-cross-database-joins` or similar) which attaches to ALL the databases. Perhaps it starts as an in-memory database, maybe at `/memory`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-391584112
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDM5MTU4NDExMg==
created_at: 2018-05-24T04:26:29Z
updated_at: 2018-05-24T04:30:50Z
author_association: OWNER
body:
  I built a very rough prototype of this to prove it could work. It's deployed here - and here's an example of a query that joins across two different databases:

  https://datasette-cross-database-joins-prototype.now.sh/memory?sql=select+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D.rowid%2C%0D%0Afivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D.actors%2C%0D%0A%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D.city%0D%0Afrom+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D%0D%0Ajoin+%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D%0D%0A++on+%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D.rowid+%3D+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D.rowid

  ```
  select fivethirtyeight.[love-actually/love_actually_adjacencies].rowid,
  fivethirtyeight.[love-actually/love_actually_adjacencies].actors,
  [google-trends].[20150430_UKDebate].city
  from fivethirtyeight.[love-actually/love_actually_adjacencies]
  join [google-trends].[20150430_UKDebate]
    on [google-trends].[20150430_UKDebate].rowid = fivethirtyeight.[love-actually/love_actually_adjacencies].rowid
  ```
  I deployed it like this:

      datasette publish now --branch=cross-database-joins fivethirtyeight.db google-trends.db --name=datasette-cross-database-joins-prototype

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-391584366
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDM5MTU4NDM2Ng==
created_at: 2018-05-24T04:28:20Z
updated_at: 2018-05-24T04:28:20Z
author_association: OWNER
body:
  I used some pretty ugly hacks, like faking an entire `.inspect()` block for the `:memory:` database just to get past the errors I was seeing. To ship this as a feature it will need quite a bit of code refactoring to make those hacks unnecessary.

  https://github.com/simonw/datasette/blob/7a3040f5782375373b2b66e5969bc2c49b3a6f0e/datasette/views/database.py#L18-L26

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-391584527
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDM5MTU4NDUyNw==
created_at: 2018-05-24T04:29:40Z
updated_at: 2018-05-24T04:29:40Z
author_association: OWNER
body:
  Rather than stealing the `/memory` namespace for this it would be nicer if these cross-database joins could be executed at the very top-level URL of the Datasette instance - `https://example.com/?sql=...`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-391752218
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDM5MTc1MjIxOA==
created_at: 2018-05-24T15:15:19Z
updated_at: 2018-05-24T15:15:19Z
author_association: OWNER
body:
  Most of the time Datasette is used with just a single database file. So maybe it makes sense for this option to be turned on by default and to ALWAYS be available on the Datasette instance homepage unless the user has explicitly disabled it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-391752425
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDM5MTc1MjQyNQ==
created_at: 2018-05-24T15:15:51Z
updated_at: 2018-05-24T15:15:51Z
author_association: OWNER
body:
  This would make Datasett's SQL features a lot more instantly obvious to people who land on a homepage, which is probably a good thing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-391752629
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDM5MTc1MjYyOQ==
created_at: 2018-05-24T15:16:25Z
updated_at: 2018-05-24T15:16:25Z
author_association: OWNER
body:
  Should this support canned queries too? I think it should, though that raises interesting questions regarding their URL structure.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-391754506
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDM5MTc1NDUwNg==
created_at: 2018-05-24T15:21:37Z
updated_at: 2018-05-24T15:21:53Z
author_association: OWNER
body:
  Giving it `/all/` would be easier since that way the existing URL routes (including canned queries) would all work... but I would have to teach it NOT to expect a database content hash on that URL.

  Or maybe it should still have a content hash (to enable far-future cache expiry headers on query results) but the hash should be constructed out of all of the other database hashes concatenated together.

  That way the URLs would be `/all-5de27e3` and `/all-5de27e3/canned-query-name`

  Only downside: this would make it impossible to have a database file with the name `all.db`. I think that's probably an OK trade-off. You could turn the feature off with a config flag if you really want to use that filename (for whatever reason).

  How about `/-all-5de27e3/` instead to avoid collisions?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-391755300
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDM5MTc1NTMwMA==
created_at: 2018-05-24T15:23:37Z
updated_at: 2018-05-24T15:23:37Z
author_association: OWNER
body:
  On the `/-all-5de27e3` page we can show the regular https://fivethirtyeight.datasettes.com/fivethirtyeight-5de27e3 interface but instead of the list of tables we can show a list of attached databases plus some help text showing how to construct a cross-database join.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-391756841
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDM5MTc1Njg0MQ==
created_at: 2018-05-24T15:27:42Z
updated_at: 2018-05-24T15:27:42Z
author_association: OWNER
body:
  For an example query that pre-populates that textarea... maybe a UNION that pulls the first 10 rows from the first table of each of the first two databases?

  ```
  select * from (select rowid, actors from fivethirtyeight.[love-actually/love_actually_adjacencies] limit 10)
     union all
  select * from (select rowid, city from [google-trends].[20150430_UKDebate] limit 10)
  ```

  https://datasette-cross-database-joins-prototype.now.sh/memory?sql=select+*+from+%28select+rowid%2C+actors+from+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D+limit+10%29%0D%0A+++union+all%0D%0Aselect+*+from+%28select+rowid%2C+city+from+%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D+limit+10%29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/284#issuecomment-391765973
issue_url: https://api.github.com/repos/simonw/datasette/issues/284
node_id: MDEyOklzc3VlQ29tbWVudDM5MTc2NTk3Mw==
created_at: 2018-05-24T15:53:08Z
updated_at: 2018-05-24T15:53:08Z
author_association: OWNER
body:
  This will also give us a mechanism for turning on and off the cross-database joins feature from #283

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/275#issuecomment-391771202
issue_url: https://api.github.com/repos/simonw/datasette/issues/275
node_id: MDEyOklzc3VlQ29tbWVudDM5MTc3MTIwMg==
created_at: 2018-05-24T16:08:41Z
updated_at: 2018-05-24T16:08:41Z
author_association: OWNER
body:
  So the lookup priority order should be:

  * table level in metadata
  * database level in metadata
  * root level in metadata
  * `--config` options passed to `datasette serve`
  * `DATASETTE_X` environment variables

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/275#issuecomment-391771658
issue_url: https://api.github.com/repos/simonw/datasette/issues/275
node_id: MDEyOklzc3VlQ29tbWVudDM5MTc3MTY1OA==
created_at: 2018-05-24T16:09:55Z
updated_at: 2018-05-24T16:09:55Z
author_association: OWNER
body:
  It feels slightly weird continuing to call it `metadata.json` as it starts to grow support for config options (which already started with the `units` and `facets` keys) but I can live with that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/284#issuecomment-391912392
issue_url: https://api.github.com/repos/simonw/datasette/issues/284
node_id: MDEyOklzc3VlQ29tbWVudDM5MTkxMjM5Mg==
created_at: 2018-05-25T01:16:56Z
updated_at: 2018-05-25T01:17:13Z
author_association: OWNER
body:
  `allow_sql` should only affect the `?sql=` parameter and whether or not the form is displayed. You should still be able to use and execute canned queries even if this option is turned off.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/284#issuecomment-391950691
issue_url: https://api.github.com/repos/simonw/datasette/issues/284
node_id: MDEyOklzc3VlQ29tbWVudDM5MTk1MDY5MQ==
created_at: 2018-05-25T06:01:23Z
updated_at: 2018-05-25T06:05:02Z
author_association: OWNER
body:
  Demo:

      datasette publish now --branch=master fixtures.db \
          --source="#284 Demo" \
          --source_url="https://github.com/simonw/datasette/issues/284" \
          --extra-options "--config allow_sql:off --config allow_facet:off --config allow_download:off" \
          --name=datasette-demo-284

      now alias https://datasette-demo-284-jogjwngegj.now.sh datasette-demo-284.now.sh

  https://datasette-demo-284.now.sh/

  Note the following:

  * https://datasette-demo-284.now.sh/fixtures-fda0fea has no SQL input textarea
  * https://datasette-demo-284.now.sh/fixtures-fda0fea has no database download link
  * https://datasette-demo-284.now.sh/fixtures-fda0fea.db returns 403 forbidden
  * https://datasette-demo-284.now.sh/fixtures-fda0fea?sql=select%20*%20from%20sqlite_master throws error 400
  * https://datasette-demo-284.now.sh/fixtures-fda0fea/facetable shows no suggested facets
  * https://datasette-demo-284.now.sh/fixtures-fda0fea/facetable?_facet=city_id throws error 400

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/272#issuecomment-392118755
issue_url: https://api.github.com/repos/simonw/datasette/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDM5MjExODc1NQ==
created_at: 2018-05-25T16:56:40Z
updated_at: 2018-06-05T16:01:13Z
author_association: OWNER
body:
  Thinking about this further, maybe I should embrace ASGI turtles-all-the-way-down and teach each datasette view class to take a scope to the constructor and act entirely as an ASGI component. Would be a nice way of diving deep into ASGI and I can add utility helpers for things like querystring evaluation as I need them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/286#issuecomment-392121500
issue_url: https://api.github.com/repos/simonw/datasette/issues/286
node_id: MDEyOklzc3VlQ29tbWVudDM5MjEyMTUwMA==
created_at: 2018-05-25T17:06:46Z
updated_at: 2018-05-25T17:06:46Z
author_association: OWNER
body:
  A few extra thoughts:

  * Some users may want to opt out of this. We could have `--config version_in_hash:false`
  * should this affect the filename for the downloadable copy of the SQLite database? Maybe that should stay as just the hash of the contents, but that's a fair bit more complex
  * What about users who stick with the same version of datasette but deploy changes to their custom templates - how can we help them cache bust? Maybe with `--config cache_version:2`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/286#issuecomment-392121743
issue_url: https://api.github.com/repos/simonw/datasette/issues/286
node_id: MDEyOklzc3VlQ29tbWVudDM5MjEyMTc0Mw==
created_at: 2018-05-25T17:07:36Z
updated_at: 2018-05-25T17:07:36Z
author_association: OWNER
body:
  This is also a great excuse to finally write up some detailed documentation on Datasette's caching strategy

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/259#issuecomment-392212119
issue_url: https://api.github.com/repos/simonw/datasette/issues/259
node_id: MDEyOklzc3VlQ29tbWVudDM5MjIxMjExOQ==
created_at: 2018-05-25T23:22:26Z
updated_at: 2018-05-25T23:22:26Z
author_association: OWNER
body:
  This should detect any table which can be linked to the current table via some other table, based on the other table having a foreign key to them both.

  These join tables could be arbitrarily complicated. They might have foreign keys to more than two other tables, maybe even multiple foreign keys to the same column.

  Ideally M2M defection would catch all of these cases. Maybe the resulting inspect data looks something like this:

  ```
  "artists": {
      ...
      "m2m": [{
           "other_table": "festivals",
           "through": "performances",
           "our_fk": "artist_id",
           "other_fk": "performance_id"
      }]
  ```

  Let's ignore compound primary keys: we k it detect m2m relationships where the join table has foreign keys to a single primary key on the other two tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/259#issuecomment-392214791
issue_url: https://api.github.com/repos/simonw/datasette/issues/259
node_id: MDEyOklzc3VlQ29tbWVudDM5MjIxNDc5MQ==
created_at: 2018-05-25T23:43:15Z
updated_at: 2018-07-29T00:56:03Z
author_association: OWNER
body:
  We may need to derive a usable name for each of these relationships that can be used in eg querystring parameters.

  The name of the join table is a reasonable choice here. Say the join table is called `event_tags` - the querystring for returning all events that are tagged `badger` could be `/db/events?_m2m_event_tags__tag=badger` perhaps?

  But what if `event_tags` has more than one foreign key back to `events`? Might need to specify the column in `events` that is referred back to by `event_tags` somehow in that case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-392279508
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MjI3OTUwOA==
created_at: 2018-05-26T18:32:07Z
updated_at: 2018-05-26T18:32:07Z
author_association: OWNER
body:
  Related: I started the documentation for using SpatiaLite with Datasette here: https://datasette.readthedocs.io/en/latest/spatialite.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-392279644
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MjI3OTY0NA==
created_at: 2018-05-26T18:34:21Z
updated_at: 2018-05-26T18:34:21Z
author_association: OWNER
body:
  I've been thinking a bit about modifying the SQL select statement used for the table view recently. I've run into a few examples of SQLite database that slow to a crawl when viewed with datasette because the rows are too big, so there's definitely scope for supporting custom select clauses (avoiding some columns, showing length(colname) for others).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/288#issuecomment-392288531
issue_url: https://api.github.com/repos/simonw/datasette/issues/288
node_id: MDEyOklzc3VlQ29tbWVudDM5MjI4ODUzMQ==
created_at: 2018-05-26T21:14:37Z
updated_at: 2019-04-15T23:01:17Z
author_association: OWNER
body:
  This might also be an opportunity to support an __in= operator - though that's an odd one as it acts equivalent to an OR whereas every other parameter is combined with an AND

  UPDATE 15th April 2019: I implemented `?column__in=` in a different way, see #433 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/289#issuecomment-392288990
issue_url: https://api.github.com/repos/simonw/datasette/issues/289
node_id: MDEyOklzc3VlQ29tbWVudDM5MjI4ODk5MA==
created_at: 2018-05-26T21:24:10Z
updated_at: 2018-05-26T21:24:10Z
author_association: OWNER
body:
  An example of a query where you might want to use this option: https://fivethirtyeight.datasettes.com/fivethirtyeight-5de27e3?sql=select+rowid%2C+*+from+%5Balcohol-consumption%2Fdrinks%5D+order+by+random%28%29+limit+1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/289#issuecomment-392291605
issue_url: https://api.github.com/repos/simonw/datasette/issues/289
node_id: MDEyOklzc3VlQ29tbWVudDM5MjI5MTYwNQ==
created_at: 2018-05-26T22:20:02Z
updated_at: 2018-05-26T22:20:02Z
author_association: OWNER
body:
  Documented here https://datasette.readthedocs.io/en/latest/json_api.html#special-table-arguments and here: https://datasette.readthedocs.io/en/latest/config.html#default-cache-ttl

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/289#issuecomment-392291716
issue_url: https://api.github.com/repos/simonw/datasette/issues/289
node_id: MDEyOklzc3VlQ29tbWVudDM5MjI5MTcxNg==
created_at: 2018-05-26T22:22:47Z
updated_at: 2018-05-26T22:22:47Z
author_association: OWNER
body:
  Demo: hit refresh on https://fivethirtyeight.datasettes.com/fivethirtyeight-5de27e3?sql=select+rowid%2C+*+from+%5Balcohol-consumption%2Fdrinks%5D+order+by+random%28%29+limit+1&_ttl=0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/285#issuecomment-392297392
issue_url: https://api.github.com/repos/simonw/datasette/issues/285
node_id: MDEyOklzc3VlQ29tbWVudDM5MjI5NzM5Mg==
created_at: 2018-05-27T00:50:27Z
updated_at: 2018-05-27T00:50:27Z
author_association: OWNER
body:
  I ran a very rough micro-benchmark on the new `num_sql_threads` config option.

      datasette --config num_sql_threads:1 fivethirtyeight.db

  Then

      ab -n 100 -c 10 'http://127.0.0.1:8011/fivethirtyeight-2628db9/twitter-ratio%2Fsenators'

  | Number of threads | Requests/second |
  |---|---|
  | 1 | 4.57 |
  | 3 | 9.77 |
  | 10 | 13.53 |
  | 20 | 15.24 
  | 50 |  8.21 | 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/291#issuecomment-392302406
issue_url: https://api.github.com/repos/simonw/datasette/issues/291
node_id: MDEyOklzc3VlQ29tbWVudDM5MjMwMjQwNg==
created_at: 2018-05-27T03:18:06Z
updated_at: 2018-05-27T03:18:06Z
author_association: OWNER
body:
  My first attempt at this was to have plugins depend on each other - so there would be a `datasette-leaflet` plugin which adds Leaflet to the page, and the `datasette-cluster-map` and `datasette-leaflet-geojson` plugins would depend on that plugin.

  I tried this and it didn't work, because it turns out the order in which plugins are loaded isn't predictable. `datasette-cluster-map` ended up adding it's script link before Leaflet had been loaded by `datasette-leaflet`, resulting in JavaScript errors.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/291#issuecomment-392302416
issue_url: https://api.github.com/repos/simonw/datasette/issues/291
node_id: MDEyOklzc3VlQ29tbWVudDM5MjMwMjQxNg==
created_at: 2018-05-27T03:18:16Z
updated_at: 2018-05-27T03:18:16Z
author_association: OWNER
body:
  For the moment then I'm going with a really simple solution: when iterating through `extra_css_urls` and `extra_js_urls` de-dupe by URL and avoid outputting the same link twice.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/291#issuecomment-392302456
issue_url: https://api.github.com/repos/simonw/datasette/issues/291
node_id: MDEyOklzc3VlQ29tbWVudDM5MjMwMjQ1Ng==
created_at: 2018-05-27T03:19:24Z
updated_at: 2018-05-27T03:19:24Z
author_association: OWNER
body:
  The big gap in this solution is conflicting versions: I don't yet have a story for what happens if two plugins attempt to load different versions of Leaflet. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-392316250
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MjMxNjI1MA==
created_at: 2018-05-27T08:59:46Z
updated_at: 2018-05-27T08:59:46Z
author_association: OWNER
body:
  It looks like we can use the `geometry_columns` table to introspect which columns are SpatiaLite geometries. It includes a `geometry_type` integer which is documented here: https://www.gaia-gis.it/fossil/libspatialite/wiki?name=switching-to-4.0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-392316306
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MjMxNjMwNg==
created_at: 2018-05-27T09:00:46Z
updated_at: 2018-05-27T09:00:46Z
author_association: OWNER
body:
  Relevant to this ticket: I've been playing with a plugin that automatically renders any GeoJSON cells as leaflet maps: https://github.com/simonw/datasette-leaflet-geojson

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/292#issuecomment-392316673
issue_url: https://api.github.com/repos/simonw/datasette/issues/292
node_id: MDEyOklzc3VlQ29tbWVudDM5MjMxNjY3Mw==
created_at: 2018-05-27T09:08:06Z
updated_at: 2018-05-27T09:08:06Z
author_association: OWNER
body:
  Open question: how should this affect the row page? Just because columns were hidden on the table page doesn't necessarily mean they should be hidden on the row page as well. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/292#issuecomment-392316701
issue_url: https://api.github.com/repos/simonw/datasette/issues/292
node_id: MDEyOklzc3VlQ29tbWVudDM5MjMxNjcwMQ==
created_at: 2018-05-27T09:08:49Z
updated_at: 2018-05-27T09:08:49Z
author_association: OWNER
body:
  I could certainly see people wanting different custom column selects for the row page compared to the table page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/292#issuecomment-392338130
issue_url: https://api.github.com/repos/simonw/datasette/issues/292
node_id: MDEyOklzc3VlQ29tbWVudDM5MjMzODEzMA==
created_at: 2018-05-27T15:09:18Z
updated_at: 2018-05-27T15:09:28Z
author_association: OWNER
body:
  Here's my first sketch at a metadata format for this:

  * `columns`: optional list of columns to include - if missing, shows all
  * `column_selects`: dictionary mapping column names to alternative select clauses

  `column_selects` can also invent new keys and use them to create derived columns. These new keys will be selected at the end of the list of columns UNLESS they are mentioned in `columns`, in which case that sequence will define the order.

  Can you facet by things that are customized using `column_selects`? Yes, and let's try running suggested facets against those columns as well.
  ```
  {
      "databases": {
          "databasename": {
              "tables": {
                  "tablename": {
                      "columns": [
                          "id", "name", "size"
                      ],
                      "column_selects": {
                          "name": "upper(name)",
                          "geo_json": "AsGeoJSON(Geometry)"
                      }
                      "row_columns": [...]
                      "row_column_selects": {...}
                  }
  ```
  The `row_columns` and `row_column_selects` properties work the same as the `column*` ones, except they are applied on the row page instead.

  If omitted, the `column*` ones will be used on the row page as well.

  If you want the row page to switch back to Datasette's default behaviour you can set `"row_columns": [], "row_column_selects": {}`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/292#issuecomment-392342269
issue_url: https://api.github.com/repos/simonw/datasette/issues/292
node_id: MDEyOklzc3VlQ29tbWVudDM5MjM0MjI2OQ==
created_at: 2018-05-27T15:55:40Z
updated_at: 2018-05-27T16:01:26Z
author_association: OWNER
body:
  Here's the metadata I tried against that first working prototype:

  ```
  {
      "databases": {
          "timezones": {
              "tables": {
                  "timezones": {
                      "columns": ["PK_UID"],
                      "column_selects": {
                          "upper_tzid": "upper(tzid)",
                          "Geometry": "AsGeoJSON(Geometry)"
                      }
                  }
              }
          },
          "wtr": {
              "tables": {
                  "license_frequency": {
                      "columns": ["id", "license", "tx_rx", "frequency"],
                      "column_selects": {
                          "latitude": "Y(Geometry)",
                          "longitude": "X(Geometry)"
                      }
                  }
              }
          }
      }
  }
  ```

  Run using this:

      datasette timezones.db wtr.db \
          --reload --debug --load-extension=/usr/local/lib/mod_spatialite.dylib \
          -m column-metadata.json --config sql_time_limit_ms:10000

  Usefully, the `--reload` flag detects changes to the `metadata.json` file as well as Datasette's own Python code.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/292#issuecomment-392343690
issue_url: https://api.github.com/repos/simonw/datasette/issues/292
node_id: MDEyOklzc3VlQ29tbWVudDM5MjM0MzY5MA==
created_at: 2018-05-27T16:08:25Z
updated_at: 2018-05-27T16:08:40Z
author_association: OWNER
body:
  Turns out it's actually possible to pull data from other tables using the mechanism in the prototype:

  ```
  {
      "databases": {
          "wtr": {
              "tables": {
                  "license": {
                      "column_selects": {
                          "count": "(select count(*) from license_frequency where license_frequency.license = license.id)"
                      }
                  }
              }
          }
      }
  }
  ```
  Performance using this technique is pretty terrible though:

  ![2018-05-27 at 9 07 am](https://user-images.githubusercontent.com/9599/40588124-8169d7fa-618d-11e8-9880-ccc1904b05d9.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/292#issuecomment-392343839
issue_url: https://api.github.com/repos/simonw/datasette/issues/292
node_id: MDEyOklzc3VlQ29tbWVudDM5MjM0MzgzOQ==
created_at: 2018-05-27T16:10:09Z
updated_at: 2018-06-04T17:38:04Z
author_association: OWNER
body:
  The more efficient way of doing this kind of count would be to provide a mechanism which can also add extra fragments to a `GROUP BY` clause used for the `SELECT`.

  Or... how about a mechanism similar to Django's `prefetch_related` which lets you define extra queries that will be called with a list of primary keys (or values from other columns) and used to populate a new column? A little unconventional but could be extremely useful and efficient.

  Related to that: since the per-query overhead in SQLite is tiny, could even define an extra query to be run once-per-row before returning results.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/292#issuecomment-392345062
issue_url: https://api.github.com/repos/simonw/datasette/issues/292
node_id: MDEyOklzc3VlQ29tbWVudDM5MjM0NTA2Mg==
created_at: 2018-05-27T16:26:53Z
updated_at: 2018-05-27T16:26:53Z
author_association: OWNER
body:
  There needs to be a way to turn this off and return to Datasette default bahviour. Maybe a `?_raw=1` querystring parameter for the table view.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/292#issuecomment-392350495
issue_url: https://api.github.com/repos/simonw/datasette/issues/292
node_id: MDEyOklzc3VlQ29tbWVudDM5MjM1MDQ5NQ==
created_at: 2018-05-27T17:47:31Z
updated_at: 2018-05-27T17:47:31Z
author_association: OWNER
body:
  Querystring design:

  * `?_column=a&_column=b` - equivalent of `"columns": ["a", "b"]` in `metadata.json`
  * `?_select_nameupper=upper(name)` - equivalent of `"column_selects": {"nameupper": "upper(name)"}`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/292#issuecomment-392350980
issue_url: https://api.github.com/repos/simonw/datasette/issues/292
node_id: MDEyOklzc3VlQ29tbWVudDM5MjM1MDk4MA==
created_at: 2018-05-27T17:56:30Z
updated_at: 2018-05-27T17:56:50Z
author_association: OWNER
body:
  Should `?_raw=1` also turn off foreign key expansions? No, we will eventually provide a separate mechanism for that (or leave it to nerds who care to figure out using JSON or CSV export).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/116#issuecomment-392574208
issue_url: https://api.github.com/repos/simonw/datasette/issues/116
node_id: MDEyOklzc3VlQ29tbWVudDM5MjU3NDIwOA==
created_at: 2018-05-28T17:23:41Z
updated_at: 2018-05-28T17:23:41Z
author_association: OWNER
body:
  I'm handling this as separate documentation sections instead, e.g. http://datasette.readthedocs.io/en/latest/spatialite.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/79#issuecomment-392574358
issue_url: https://api.github.com/repos/simonw/datasette/issues/79
node_id: MDEyOklzc3VlQ29tbWVudDM5MjU3NDM1OA==
created_at: 2018-05-28T17:24:48Z
updated_at: 2018-05-28T17:24:48Z
author_association: OWNER
body:
  Closing this as obsolete in favor of other issues [tagged documentation](https://github.com/simonw/datasette/labels/documentation).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/36#issuecomment-392575160
issue_url: https://api.github.com/repos/simonw/datasette/issues/36
node_id: MDEyOklzc3VlQ29tbWVudDM5MjU3NTE2MA==
created_at: 2018-05-28T17:30:52Z
updated_at: 2018-05-28T17:30:52Z
author_association: OWNER
body:
  I've changed my mind about this.

  "Select every record on the 3rd day of the month" doesn't strike me as an actually useful feature.

  "Select every record in 2018 / in May 2018 / on 1st May 2018", if you are using the SQLite-preferred datestring format, are already supported using LIKE queries (or the startswith filter):

  * https://fivethirtyeight.datasettes.com/fivethirtyeight/inconvenient-sequel%2Fratings?timestamp__startswith=2017
  * https://fivethirtyeight.datasettes.com/fivethirtyeight/inconvenient-sequel%2Fratings?timestamp__startswith=2017-08
  * https://fivethirtyeight.datasettes.com/fivethirtyeight/inconvenient-sequel%2Fratings?timestamp__startswith=2017-08-29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/121#issuecomment-392575448
issue_url: https://api.github.com/repos/simonw/datasette/issues/121
node_id: MDEyOklzc3VlQ29tbWVudDM5MjU3NTQ0OA==
created_at: 2018-05-28T17:33:07Z
updated_at: 2018-05-28T17:33:07Z
author_association: OWNER
body:
  This shouldn't be a comma-separated list, it should be an argument you can pass multiple times to better match #255 and #292

  Maybe `?_json=foo&_json=bar`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/121#issuecomment-392580902
issue_url: https://api.github.com/repos/simonw/datasette/issues/121
node_id: MDEyOklzc3VlQ29tbWVudDM5MjU4MDkwMg==
created_at: 2018-05-28T18:11:51Z
updated_at: 2018-05-28T18:11:51Z
author_association: OWNER
body:
  Implemented in 76d11eb768e2f05f593c4d37a25280c0fcdf8fd6

  Documented here: http://datasette.readthedocs.io/en/latest/json_api.html#special-json-arguments

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/38#issuecomment-392601114
issue_url: https://api.github.com/repos/simonw/datasette/issues/38
node_id: MDEyOklzc3VlQ29tbWVudDM5MjYwMTExNA==
created_at: 2018-05-28T20:47:31Z
updated_at: 2018-05-28T20:47:31Z
author_association: OWNER
body:
  I think the way Datasette executes SQL queries in a thread pool introduced in #45 is a good solution for this ticket.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/56#issuecomment-392601478
issue_url: https://api.github.com/repos/simonw/datasette/issues/56
node_id: MDEyOklzc3VlQ29tbWVudDM5MjYwMTQ3OA==
created_at: 2018-05-28T20:50:24Z
updated_at: 2018-05-28T20:50:24Z
author_association: OWNER
body:
  I'm going to close this as WONTFIX for the moment. Once Plugins #14 grows the ability to add extra URL paths and views someone who needs this could build it as a plugin instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/97#issuecomment-392602334
issue_url: https://api.github.com/repos/simonw/datasette/issues/97
node_id: MDEyOklzc3VlQ29tbWVudDM5MjYwMjMzNA==
created_at: 2018-05-28T20:57:21Z
updated_at: 2018-05-28T20:57:21Z
author_association: OWNER
body:
  The `/.json` endpoint is more of an implementation detail of the homepage at this point. A better, documented ( http://datasette.readthedocs.io/en/stable/introspection.html#inspect ) endpoint for finding all of the databases and tables is https://parlgov.datasettes.com/-/inspect.json

reactions:
  {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/142#issuecomment-392605574
issue_url: https://api.github.com/repos/simonw/datasette/issues/142
node_id: MDEyOklzc3VlQ29tbWVudDM5MjYwNTU3NA==
created_at: 2018-05-28T21:25:05Z
updated_at: 2018-05-28T21:25:05Z
author_association: OWNER
body:
  ![2018-05-28 at 2 24 pm](https://user-images.githubusercontent.com/9599/40629887-e991c61c-6282-11e8-9d66-6387f90e87ca.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/144#issuecomment-392606044
issue_url: https://api.github.com/repos/simonw/datasette/issues/144
node_id: MDEyOklzc3VlQ29tbWVudDM5MjYwNjA0NA==
created_at: 2018-05-28T21:29:42Z
updated_at: 2018-05-28T21:29:42Z
author_association: OWNER
body:
  The other major limitation of APSW is its treatment of unicode: https://rogerbinns.github.io/apsw/types.html - it tells you that it is your responsibility to ensure that TEXT columns in your SQLite database are correctly encoded.

  Since Datasette is designed to work against ANY SQLite database that someone may have already created, I see that as a show-stopping limitation.

  Thanks to https://github.com/coleifer/sqlite-vtfunc I now have a working mechanism for virtual tables (I've even built a demo plugin with them - https://github.com/simonw/datasette-sql-scraper ) which was the main thing that interested me about APSW.

  I'm going to close this as WONTFIX - I think Python's built-in `sqlite3` is good enough, and is now so firmly embedded in the project that making it pluggable would be more trouble than it's worth.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/179#issuecomment-392606418
issue_url: https://api.github.com/repos/simonw/datasette/issues/179
node_id: MDEyOklzc3VlQ29tbWVudDM5MjYwNjQxOA==
created_at: 2018-05-28T21:32:37Z
updated_at: 2018-05-28T21:32:37Z
author_association: OWNER
body:
  > It could also be useful to allow users to import a python file containing custom functions that can that be loaded into scope and made available to custom templates.

  That's now covered by the plugins mechanism - you can create plugins that define custom template functions: http://datasette.readthedocs.io/en/stable/plugins.html#prepare-jinja2-environment-env

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-392815673
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MjgxNTY3Mw==
created_at: 2018-05-29T15:17:04Z
updated_at: 2018-05-29T15:17:04Z
author_association: OWNER
body:
  I'm coming round to the idea that this should be baked into Datasette core - see above referenced issues for some of the explorations I've been doing around this area.

  Datasette should absolutely work without SpatiaLite, but it's such a huge bonus part of the SQLite ecosystem that I'm happy to ship features that take advantage of it without being relegated to plugins.

  I'm also becoming aware that there aren't really that many other interesting loadable extensions for SQLite. If SpatiaLite was one of dozens I'd feel that a rule that "anything dependent on an extension lives in a plugin" would make sense, but as it stands I think 99% of the time the only loadable extensions people will be using will be SpatiaLite and json1 (and json1 is available in the amalgamation anyway).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/191#issuecomment-392822050
issue_url: https://api.github.com/repos/simonw/datasette/issues/191
node_id: MDEyOklzc3VlQ29tbWVudDM5MjgyMjA1MA==
created_at: 2018-05-29T15:33:25Z
updated_at: 2018-05-29T15:33:25Z
author_association: OWNER
body:
  I don't know how it happened, but I've somehow got myself into a state where my local SQLite for Python 3 on OS X is `3.23.1`:

  ```
  ~ $ python3
  Python 3.6.5 (default, Mar 30 2018, 06:41:53) 
  [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import sqlite3
  >>> sqlite3.connect(':memory:').execute('select sqlite_version()').fetchall()
  [('3.23.1',)]
  >>> 
  ```
  Maybe I did something in homebrew that changed this? I'd love to understand what exactly I did to get to this state.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-392825746
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MjgyNTc0Ng==
created_at: 2018-05-29T15:42:53Z
updated_at: 2018-05-29T15:42:53Z
author_association: CONTRIBUTOR
body:
  I haven't had time to look further into this, but if doing this as a plugin results in useful hooks then I think we should do it that way. We could always require the plugin as a standard dependency. 

  I think this is going to result in quite a bit of refactoring anyway so it's a good time to add hooks regardless. 

  On the other hand, if we have to add lots of specialist hooks for it then maybe it's worth integrating into the core.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/191#issuecomment-392828475
issue_url: https://api.github.com/repos/simonw/datasette/issues/191
node_id: MDEyOklzc3VlQ29tbWVudDM5MjgyODQ3NQ==
created_at: 2018-05-29T15:50:18Z
updated_at: 2018-05-29T15:50:18Z
author_association: NONE
body:
  Python standard-library SQLite dynamically links against the system sqlite3. So presumably you installed a more up-to-date sqlite3 somewhere on your `LD_LIBRARY_PATH`.

  To compile a statically-linked pysqlite you need to include an amalgamation in the project root when building the extension. Read the relevant setup.py.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/191#issuecomment-392831543
issue_url: https://api.github.com/repos/simonw/datasette/issues/191
node_id: MDEyOklzc3VlQ29tbWVudDM5MjgzMTU0Mw==
created_at: 2018-05-29T15:58:33Z
updated_at: 2018-05-29T15:58:33Z
author_association: OWNER
body:
  I ran an informal survey on twitter and most people were on 3.21 - https://twitter.com/simonw/status/1001487546289815553

  Maybe this is from upgrading to the latest OS X release.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/296#issuecomment-392840811
issue_url: https://api.github.com/repos/simonw/datasette/issues/296
node_id: MDEyOklzc3VlQ29tbWVudDM5Mjg0MDgxMQ==
created_at: 2018-05-29T16:26:27Z
updated_at: 2018-05-29T19:43:23Z
author_association: OWNER
body:
  Since #275 will allow configs to be overridden at the table and database level it also makes sense to expose a completely evaluated list of configs at:

  * `/dbname/-/config`
  * `/dbname/tablename/-/config`

  Similar to https://fivethirtyeight.datasettes.com/-/config

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/265#issuecomment-392890045
issue_url: https://api.github.com/repos/simonw/datasette/issues/265
node_id: MDEyOklzc3VlQ29tbWVudDM5Mjg5MDA0NQ==
created_at: 2018-05-29T18:37:49Z
updated_at: 2018-05-29T18:37:49Z
author_association: NONE
body:
  Just about to ask for this!  Move this page https://github.com/simonw/datasette/wiki/Datasettes

  into a datasette, with some concept of versioning as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/298#issuecomment-392917380
issue_url: https://api.github.com/repos/simonw/datasette/issues/298
node_id: MDEyOklzc3VlQ29tbWVudDM5MjkxNzM4MA==
created_at: 2018-05-29T19:41:59Z
updated_at: 2018-05-29T19:41:59Z
author_association: OWNER
body:
  Creating URLs using concatenation as seen in `('https://twitter.com/' || user) as user_url` is likely to have all sorts of useful applications for ad-hoc analysis.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-392969173
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5Mjk2OTE3Mw==
created_at: 2018-05-29T22:32:08Z
updated_at: 2018-05-29T22:32:08Z
author_association: OWNER
body:
  The more time I spend with SpatiaLite the more convinced I am that this should be default behavior. There's nothing useful about the binary Geometry representation - it's not even valid WKB. I'm on board with WKT as the default display in HTML and GeoJSON as the default for `.json`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/265#issuecomment-393003340
issue_url: https://api.github.com/repos/simonw/datasette/issues/265
node_id: MDEyOklzc3VlQ29tbWVudDM5MzAwMzM0MA==
created_at: 2018-05-30T01:44:22Z
updated_at: 2018-05-30T01:44:22Z
author_association: OWNER
body:
  Funny you should mention that... I'm planning on doing that as part of the official Datasette website at some point soon. A Datasette instance that lists other Datasette instances feels pleasingly appropriate.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-393014943
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MzAxNDk0Mw==
created_at: 2018-05-30T02:59:53Z
updated_at: 2018-05-30T02:59:53Z
author_association: OWNER
body:
  I just realised a problem with GeoJSON is that it assumes that the underlying geometry is WGS 84 latitude/longitude points - but it's very possible for a SpatiaLite geometry to contain geometric data that's nothing to do with geospatial projections.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-393020749
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5MzAyMDc0OQ==
created_at: 2018-05-30T03:42:54Z
updated_at: 2018-05-30T03:42:54Z
author_association: OWNER
body:
  Challenge: how to deal with tables where the name ends in `.csv`? I actually have one of these in the test suite at the moment:

  https://github.com/simonw/datasette/blob/d69ebce53385b7c6fafb85fdab3b136dbf3f332c/tests/fixtures.py#L234-L237

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/265#issuecomment-393064224
issue_url: https://api.github.com/repos/simonw/datasette/issues/265
node_id: MDEyOklzc3VlQ29tbWVudDM5MzA2NDIyNA==
created_at: 2018-05-30T07:48:37Z
updated_at: 2018-05-30T07:48:37Z
author_association: OWNER
body:
  https://datasette-registry.now.sh Is now live, powered by https://github.com/simonw/datasette-registry - still needs plenty of work but it's an interesting start.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/276#issuecomment-393106520
issue_url: https://api.github.com/repos/simonw/datasette/issues/276
node_id: MDEyOklzc3VlQ29tbWVudDM5MzEwNjUyMA==
created_at: 2018-05-30T10:09:25Z
updated_at: 2018-05-30T10:09:25Z
author_association: CONTRIBUTOR
body:
  I don't think it's unreasonable to only support spatialite geometries in a coordinate reference system which is at least transformable to WGS84. It would be nice to support different CRSes in the database so conversion to spatialite from the source data is lossless.

  I think the working CRS for datasette should be WGS84 though (leaflet requires it, for example) - it's just a case of calling `ST_Transform(geom, 4326)` on the column while we're loading the data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/243#issuecomment-393544357
issue_url: https://api.github.com/repos/simonw/datasette/issues/243
node_id: MDEyOklzc3VlQ29tbWVudDM5MzU0NDM1Nw==
created_at: 2018-05-31T14:14:49Z
updated_at: 2018-05-31T14:14:49Z
author_association: OWNER
body:
  Demo: https://datasette-publish-spatialite-demo.now.sh/spatialite-test-c88bc35?sql=select+AsText(Geometry)+from+HighWays+limit+1%3B

  Published using `datasette publish now --spatialite /tmp/spatialite-test.db`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/294#issuecomment-393547960
issue_url: https://api.github.com/repos/simonw/datasette/issues/294
node_id: MDEyOklzc3VlQ29tbWVudDM5MzU0Nzk2MA==
created_at: 2018-05-31T14:25:43Z
updated_at: 2018-05-31T14:25:43Z
author_association: OWNER
body:
  SpatialLite columns are actually quite a bit more interesting than this - they also have a `geometry_type` (point, polygon, linestring etc), a `coord_dimension` (usually 2 but can be higher) and an `srid`.

  For example:

  https://datasette-publish-spatialite-demo.now.sh/spatialite-test-c88bc35/geometry_columns

  ![2018-05-31 at 7 22 am](https://user-images.githubusercontent.com/9599/40787843-6f9600ee-64a3-11e8-84e5-64d7cc69603a.png)

  The SRID here is particularly interesting, because it helps hint at the fact that the results from these queries won't be latitude/longitude co-ordinates - which means that `AsGeoJSON()` won't return results that can be easily rendered by Leaflet:

  https://datasette-publish-spatialite-demo.now.sh/spatialite-test-c88bc35?sql=select+AsGeoJSON(Geometry)+from+HighWays%20limit1

  Compare with https://timezones-api.now.sh/timezones-a99b2e3/geometry_columns:

  ![2018-05-31 at 7 25 am](https://user-images.githubusercontent.com/9599/40787991-d2650756-64a3-11e8-936e-2dcce7dd1515.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/294#issuecomment-393548602
issue_url: https://api.github.com/repos/simonw/datasette/issues/294
node_id: MDEyOklzc3VlQ29tbWVudDM5MzU0ODYwMg==
created_at: 2018-05-31T14:27:41Z
updated_at: 2018-05-31T14:27:56Z
author_association: OWNER
body:
  Presumably the difference in primary key structure between those two is caused by the fact that the `spatialite-test` database (actually https://www.gaia-gis.it/spatialite-2.3.1/test-2.3.sqlite.gz downloaded from https://www.gaia-gis.it/spatialite-2.3.1/resources.html ) was created by a much older version of SpatialLite - presumably v2.3.1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/294#issuecomment-393549215
issue_url: https://api.github.com/repos/simonw/datasette/issues/294
node_id: MDEyOklzc3VlQ29tbWVudDM5MzU0OTIxNQ==
created_at: 2018-05-31T14:29:37Z
updated_at: 2018-05-31T14:29:37Z
author_association: OWNER
body:
  Also of note: `spatialite-test` uses readable strings in the `type` column, while `timezones` has a `geometry_type` column with integers in it.

  Those integers are documented here: https://www.gaia-gis.it/fossil/libspatialite/wiki?name=switching-to-4.0

  ![2018-05-31 at 7 29 am](https://user-images.githubusercontent.com/9599/40788210-5d0f0dd4-64a4-11e8-8141-0386b5c7b384.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/303#issuecomment-393557406
issue_url: https://api.github.com/repos/simonw/datasette/issues/303
node_id: MDEyOklzc3VlQ29tbWVudDM5MzU1NzQwNg==
created_at: 2018-05-31T14:54:03Z
updated_at: 2018-05-31T14:54:03Z
author_association: OWNER
body:
  Our test fixtures currently have a table with a name ending in `.csv`:

  https://github.com/simonw/datasette/blob/d69ebce53385b7c6fafb85fdab3b136dbf3f332c/tests/fixtures.py#L234-L237

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/294#issuecomment-393557968
issue_url: https://api.github.com/repos/simonw/datasette/issues/294
node_id: MDEyOklzc3VlQ29tbWVudDM5MzU1Nzk2OA==
created_at: 2018-05-31T14:55:46Z
updated_at: 2018-05-31T14:55:46Z
author_association: OWNER
body:
  I'm not sure what the best JSON shape for this would be considering the potential complexity of geospatial columns. I do think it's worth exposing these in the inspect JSON though, mainly so Datasette Registry can keep track of all of the openly available geodata out there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/303#issuecomment-393599840
issue_url: https://api.github.com/repos/simonw/datasette/issues/303
node_id: MDEyOklzc3VlQ29tbWVudDM5MzU5OTg0MA==
created_at: 2018-05-31T16:54:22Z
updated_at: 2018-05-31T16:54:32Z
author_association: OWNER
body:
  The interesting thing about this is that it requires URL routing to become aware of the names of all of the available tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/303#issuecomment-393600441
issue_url: https://api.github.com/repos/simonw/datasette/issues/303
node_id: MDEyOklzc3VlQ29tbWVudDM5MzYwMDQ0MQ==
created_at: 2018-05-31T16:56:25Z
updated_at: 2018-05-31T16:57:41Z
author_association: OWNER
body:
  Here's a nasty challenge: what happens if a database has the following two tables:

  * `blah`
  * `blah.json`

  What would the URL be for the JSON endpoint for the `blah` table?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/304#issuecomment-393610731
issue_url: https://api.github.com/repos/simonw/datasette/issues/304
node_id: MDEyOklzc3VlQ29tbWVudDM5MzYxMDczMQ==
created_at: 2018-05-31T17:29:31Z
updated_at: 2018-05-31T17:30:05Z
author_association: OWNER
body:
  I prototyped this a while ago here https://github.com/simonw/datasette/commit/04476ead53758044a5f272ae8696b63d6703115e before we had the ``--config`` mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/303#issuecomment-394037368
issue_url: https://api.github.com/repos/simonw/datasette/issues/303
node_id: MDEyOklzc3VlQ29tbWVudDM5NDAzNzM2OA==
created_at: 2018-06-01T23:50:17Z
updated_at: 2018-06-01T23:50:35Z
author_association: OWNER
body:
  Solution for he above: support an optional `?_format=json/csv` parameter on the regular table view. 

  Then if you have tables with the above colliding names you can use `/db/blah.json?_format=json` 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/266#issuecomment-394417567
issue_url: https://api.github.com/repos/simonw/datasette/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDM5NDQxNzU2Nw==
created_at: 2018-06-04T16:30:48Z
updated_at: 2018-06-04T16:32:55Z
author_association: OWNER
body:
  When serving streaming responses, I need to check that a large CSV file doesn't completely max out the CPU in a way that is harmful to the rest of the instance.

  If it does, one option may be to insert an async sleep call in between each chunk that is streamed back. This could be controlled by a `csv_pause_ms` config setting, defaulting to maybe 5 but can be disabled entirely by setting to 0.

  That's only if testing proves that this is a necessary mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


