5,578 rows sorted by node_id

View and edit SQL

Suggested facets: created_at (date), updated_at (date)

author_association

id html_url issue_url node_id ▼ user created_at updated_at author_association body reactions issue performed_via_github_app
340561577 https://github.com/simonw/datasette/issues/40#issuecomment-340561577 https://api.github.com/repos/simonw/datasette/issues/40 MDEyOklzc3VlQ29tbWVudDM0MDU2MTU3Nw== simonw 9599 2017-10-30T19:43:40Z 2017-10-30T19:43:40Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Implement command-line tool interface 268470572  
340787868 https://github.com/simonw/datasette/issues/39#issuecomment-340787868 https://api.github.com/repos/simonw/datasette/issues/39 MDEyOklzc3VlQ29tbWVudDM0MDc4Nzg2OA== simonw 9599 2017-10-31T14:54:14Z 2017-10-31T14:54:14Z OWNER

Here’s how I can (I think) provide safe execution of arbitrary SQL while blocking PRAGMA calls: let people use names parameters in their SQL and apply strict filtering to the SQL query but not to the parameter values.

cur.execute(
    "select * from people where name_last=:who and age=:age", {
        "who": who,
        "age": age
})

In URL form:

?sql=select...&who=Terry&age=34

Now we can apply strict, dumb validation rules to the SQL part while allowing anything in the named queries - so people can execute a search for PRAGMA without being able to execute a PRAGMA statement.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Protect against malicious SQL that causes damage even though our DB is immutable 268469569  
341945420 https://github.com/simonw/datasette/issues/40#issuecomment-341945420 https://api.github.com/repos/simonw/datasette/issues/40 MDEyOklzc3VlQ29tbWVudDM0MTk0NTQyMA== simonw 9599 2017-11-05T02:55:07Z 2017-11-05T02:55:07Z OWNER

To simplify things a bit, I'm going to require that every database is explicitly listed in the command line. I won't support "serve everything in this directory" for the moment.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Implement command-line tool interface 268470572  
341938424 https://github.com/simonw/datasette/issues/10#issuecomment-341938424 https://api.github.com/repos/simonw/datasette/issues/10 MDEyOklzc3VlQ29tbWVudDM0MTkzODQyNA== simonw 9599 2017-11-04T23:48:57Z 2017-11-04T23:48:57Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Set up Travis 267517381  
342030075 https://github.com/simonw/datasette/issues/40#issuecomment-342030075 https://api.github.com/repos/simonw/datasette/issues/40 MDEyOklzc3VlQ29tbWVudDM0MjAzMDA3NQ== simonw 9599 2017-11-06T02:25:48Z 2017-11-06T02:25:48Z OWNER

... I tried that, I don't like it. I'm going to bring back "directory serving" by allowing you to pass a directory as an argument to datasite (including datasite .). I may even make . the default if you don't provide anything at all.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Implement command-line tool interface 268470572  
342032943 https://github.com/simonw/datasette/issues/16#issuecomment-342032943 https://api.github.com/repos/simonw/datasette/issues/16 MDEyOklzc3VlQ29tbWVudDM0MjAzMjk0Mw== simonw 9599 2017-11-06T02:50:07Z 2017-11-06T02:50:07Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Default HTML/CSS needs to look reasonable and be responsive 267726219  
342484889 https://github.com/simonw/datasette/issues/44#issuecomment-342484889 https://api.github.com/repos/simonw/datasette/issues/44 MDEyOklzc3VlQ29tbWVudDM0MjQ4NDg4OQ== simonw 9599 2017-11-07T13:39:49Z 2017-11-07T13:39:49Z OWNER

I’m going to call this feature “count values”

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_group_count=country - return counts by specific column(s) 269731374  
342521344 https://github.com/simonw/datasette/issues/47#issuecomment-342521344 https://api.github.com/repos/simonw/datasette/issues/47 MDEyOklzc3VlQ29tbWVudDM0MjUyMTM0NA== simonw 9599 2017-11-07T15:37:45Z 2017-11-07T15:37:45Z OWNER

GDS Registries could be fun too: https://registers.cloudapps.digital/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create neat example database 271831408  
343164111 https://github.com/simonw/datasette/issues/32#issuecomment-343164111 https://api.github.com/repos/simonw/datasette/issues/32 MDEyOklzc3VlQ29tbWVudDM0MzE2NDExMQ== simonw 9599 2017-11-09T14:05:56Z 2017-11-09T14:05:56Z OWNER

Implemented in 31b21f5c5e15fc3acab7fabb170c1da71dc3c98c

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Try running SQLite queries in a separate thread 268106803  
343168796 https://github.com/simonw/datasette/issues/48#issuecomment-343168796 https://api.github.com/repos/simonw/datasette/issues/48 MDEyOklzc3VlQ29tbWVudDM0MzE2ODc5Ng== simonw 9599 2017-11-09T14:22:21Z 2017-11-09T14:22:21Z OWNER

Won't fix: ujson is not compatible with the custom JSON encoder I'm using here: https://github.com/simonw/immutabase/blob/b2dee11fcd989d9e2a7bf4de1e23dbc320c05013/immutabase/app.py#L401-L416

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Switch to ujson 272391665  
343266326 https://github.com/simonw/datasette/issues/50#issuecomment-343266326 https://api.github.com/repos/simonw/datasette/issues/50 MDEyOklzc3VlQ29tbWVudDM0MzI2NjMyNg== simonw 9599 2017-11-09T19:33:18Z 2017-11-09T19:33:18Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Unit tests against application itself 272694136  
343281876 https://github.com/simonw/datasette/issues/49#issuecomment-343281876 https://api.github.com/repos/simonw/datasette/issues/49 MDEyOklzc3VlQ29tbWVudDM0MzI4MTg3Ng== simonw 9599 2017-11-09T20:30:42Z 2017-11-09T20:30:42Z OWNER

How about datasette?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Pick a name 272661336  
343237982 https://github.com/simonw/datasette/issues/49#issuecomment-343237982 https://api.github.com/repos/simonw/datasette/issues/49 MDEyOklzc3VlQ29tbWVudDM0MzIzNzk4Mg== simonw 9599 2017-11-09T17:58:01Z 2017-11-09T17:58:01Z OWNER

More terms:

  • publish
  • share
  • docker
  • host
  • stateless

I want to capture the idea of publishing an immutable database in a stateless container.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Pick a name 272661336  
343238262 https://github.com/simonw/datasette/issues/49#issuecomment-343238262 https://api.github.com/repos/simonw/datasette/issues/49 MDEyOklzc3VlQ29tbWVudDM0MzIzODI2Mg== simonw 9599 2017-11-09T17:58:59Z 2017-11-09T17:58:59Z OWNER

The name should ideally be available on PyPI and should make sense as both a command line application and a library.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Pick a name 272661336  
343239062 https://github.com/simonw/datasette/issues/48#issuecomment-343239062 https://api.github.com/repos/simonw/datasette/issues/48 MDEyOklzc3VlQ29tbWVudDM0MzIzOTA2Mg== simonw 9599 2017-11-09T18:01:46Z 2017-11-09T18:01:46Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Switch to ujson 272391665  
343551356 https://github.com/simonw/datasette/issues/49#issuecomment-343551356 https://api.github.com/repos/simonw/datasette/issues/49 MDEyOklzc3VlQ29tbWVudDM0MzU1MTM1Ng== simonw 9599 2017-11-10T18:33:22Z 2017-11-10T18:33:22Z OWNER

I'm going with datasette.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Pick a name 272661336  
343557070 https://github.com/simonw/datasette/issues/52#issuecomment-343557070 https://api.github.com/repos/simonw/datasette/issues/52 MDEyOklzc3VlQ29tbWVudDM0MzU1NzA3MA== simonw 9599 2017-11-10T18:57:47Z 2017-11-10T18:57:47Z OWNER

https://file.io/ looks like it could be good for this. It's been around since 2015, and lets you upload a temporary file which can be downloaded once.

$ curl -s -F "file=@database.db" "https://file.io/?expires=1d"
{"success":true,"key":"ySrl1j","link":"https://file.io/ySrl1j","expiry":"1 day"}

Downloading from that URL serves up the data with a Content-disposition header containing the filename:

simonw$ curl -vv https://file.io/ySrl1j | more
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 34.232.1.167...
* Connected to file.io (34.232.1.167) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: file.io
* Server certificate: Amazon
* Server certificate: Amazon Root CA 1
* Server certificate: Starfield Services Root Certificate Authority - G2
> GET /ySrl1j HTTP/1.1
> Host: file.io
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Fri, 10 Nov 2017 18:14:38 GMT
< Content-Type: undefined
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: Express
< X-RateLimit-Limit: 5
< X-RateLimit-Remaining: 4
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: Cache-Control,X-reqed-With,x-requested-with
< Content-disposition: attachment; filename=database.db
...
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Solution for temporarily uploading DB so it can be built by docker 273026602  
343581130 https://github.com/simonw/datasette/issues/20#issuecomment-343581130 https://api.github.com/repos/simonw/datasette/issues/20 MDEyOklzc3VlQ29tbWVudDM0MzU4MTEzMA== simonw 9599 2017-11-10T20:44:38Z 2017-11-10T20:44:38Z OWNER

I'm going to handle this a different way. I'm going to support a local history of your own queries stored in localStorage, but if you want to share a query you have to do it with a URL.

If people really want canned query support, they can do that using custom templates - see #12 - or by adding views to their database before they publish it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Config file with support for defining canned queries 267759136  
343581332 https://github.com/simonw/datasette/issues/21#issuecomment-343581332 https://api.github.com/repos/simonw/datasette/issues/21 MDEyOklzc3VlQ29tbWVudDM0MzU4MTMzMg== simonw 9599 2017-11-10T20:45:42Z 2017-11-10T20:45:42Z OWNER

I'm not going to use Sanic's mechanism for this. I'll use arguments passed to my cli instead.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use Sanic configuration mechanism  267769034  
343643332 https://github.com/simonw/datasette/issues/16#issuecomment-343643332 https://api.github.com/repos/simonw/datasette/issues/16 MDEyOklzc3VlQ29tbWVudDM0MzY0MzMzMg== simonw 9599 2017-11-11T06:00:04Z 2017-11-11T06:00:04Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Default HTML/CSS needs to look reasonable and be responsive 267726219  
343644891 https://github.com/simonw/datasette/issues/54#issuecomment-343644891 https://api.github.com/repos/simonw/datasette/issues/54 MDEyOklzc3VlQ29tbWVudDM0MzY0NDg5MQ== simonw 9599 2017-11-11T06:39:54Z 2017-11-11T06:39:54Z OWNER

I can detect something is a view like this:

SELECT name from sqlite_master WHERE type ='view';
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Views should not attempt to link to records / use rowids 273121803  
343644976 https://github.com/simonw/datasette/issues/26#issuecomment-343644976 https://api.github.com/repos/simonw/datasette/issues/26 MDEyOklzc3VlQ29tbWVudDM0MzY0NDk3Ng== simonw 9599 2017-11-11T06:42:23Z 2017-11-11T06:42:23Z OWNER

Simplest version of this:

  1. Create a temporary directory
  2. Write a Dockerfile into it that pulls an image and pip installs datasette
  3. Add symlinks to the DBs they listed (so we don't have to copy them)
  4. Shell out to "now"
  5. Done!
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command line tool for uploading one or more DBs to Now 267861210  
343645249 https://github.com/simonw/datasette/issues/26#issuecomment-343645249 https://api.github.com/repos/simonw/datasette/issues/26 MDEyOklzc3VlQ29tbWVudDM0MzY0NTI0OQ== simonw 9599 2017-11-11T06:48:59Z 2017-11-11T06:48:59Z OWNER

Doing this works:

import os
os.link('/tmp/databases/northwind.db', '/tmp/tmp-blah/northwind.db')

That creates a link in tmp-blah - and then when I delete that entire directory like so:

import shutil
shutil.rmtree('/tmp/tmp-blah')

The original database is not deleted, just the link.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command line tool for uploading one or more DBs to Now 267861210  
343645327 https://github.com/simonw/datasette/issues/26#issuecomment-343645327 https://api.github.com/repos/simonw/datasette/issues/26 MDEyOklzc3VlQ29tbWVudDM0MzY0NTMyNw== simonw 9599 2017-11-11T06:51:16Z 2017-11-11T06:51:16Z OWNER

I can create the temporary directory like so:

import tempfile
t = tempfile.TemporaryDirectory()
t
<TemporaryDirectory '/var/folders/w9/0xm39tk94ng9h52g06z4b54c0000gp/T/tmpkym70wlp'>
t.name
'/var/folders/w9/0xm39tk94ng9h52g06z4b54c0000gp/T/tmpkym70wlp'

And then to delete it all:

t.cleanup()
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command line tool for uploading one or more DBs to Now 267861210  
343646740 https://github.com/simonw/datasette/issues/40#issuecomment-343646740 https://api.github.com/repos/simonw/datasette/issues/40 MDEyOklzc3VlQ29tbWVudDM0MzY0Njc0MA== simonw 9599 2017-11-11T07:27:33Z 2017-11-11T07:27:33Z OWNER

I'm happy with this now that I've implemented the publish command in #26

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Implement command-line tool interface 268470572  
343647102 https://github.com/simonw/datasette/issues/47#issuecomment-343647102 https://api.github.com/repos/simonw/datasette/issues/47 MDEyOklzc3VlQ29tbWVudDM0MzY0NzEwMg== simonw 9599 2017-11-11T07:36:00Z 2017-11-11T07:36:00Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create neat example database 271831408  
343647300 https://github.com/simonw/datasette/issues/16#issuecomment-343647300 https://api.github.com/repos/simonw/datasette/issues/16 MDEyOklzc3VlQ29tbWVudDM0MzY0NzMwMA== simonw 9599 2017-11-11T07:41:19Z 2017-11-11T07:53:09Z OWNER

Still needed:

  • A link to the homepage from some kind of navigation bar in the header
  • link to github.com/simonw/datasette in the footer
  • Slightly better titles (maybe ditch the visited link colours for titles only? should keep those for primary key links)
  • Links to the .json and .jsono versions of every view
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Default HTML/CSS needs to look reasonable and be responsive 267726219  
343675165 https://github.com/simonw/datasette/issues/14#issuecomment-343675165 https://api.github.com/repos/simonw/datasette/issues/14 MDEyOklzc3VlQ29tbWVudDM0MzY3NTE2NQ== simonw 9599 2017-11-11T16:07:10Z 2017-11-11T16:07:10Z OWNER

The plugin system can also allow alternative providers for the publish command - e.g. maybe hook up hyper.sh as an option for publishing containers.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette Plugins 267707940  
343676574 https://github.com/simonw/datasette/issues/59#issuecomment-343676574 https://api.github.com/repos/simonw/datasette/issues/59 MDEyOklzc3VlQ29tbWVudDM0MzY3NjU3NA== simonw 9599 2017-11-11T16:29:48Z 2017-11-11T16:29:48Z OWNER

See also #14

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish hyper 273157085  
343683566 https://github.com/simonw/datasette/issues/60#issuecomment-343683566 https://api.github.com/repos/simonw/datasette/issues/60 MDEyOklzc3VlQ29tbWVudDM0MzY4MzU2Ng== simonw 9599 2017-11-11T18:12:24Z 2017-11-11T18:12:24Z OWNER

I’m going to solve this by making it an optional argument you can pass to the serve command. Then the Dockerfile can still build and use it but it won’t interfere with tests or dev.

If argument is not passed, we will calculate hashes on startup and calculate table row counts on demand.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink how metadata is generated and stored 273163905  
343690060 https://github.com/simonw/datasette/issues/47#issuecomment-343690060 https://api.github.com/repos/simonw/datasette/issues/47 MDEyOklzc3VlQ29tbWVudDM0MzY5MDA2MA== simonw 9599 2017-11-11T19:56:08Z 2017-11-11T19:56:08Z OWNER
"parlgov-development.db": {
    "url": "http://www.parlgov.org/"
},
"nhsadmin.sqlite": {
    "url": "https://github.com/psychemedia/openHealthDataDoodles"
}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create neat example database 271831408  
343691342 https://github.com/simonw/datasette/issues/16#issuecomment-343691342 https://api.github.com/repos/simonw/datasette/issues/16 MDEyOklzc3VlQ29tbWVudDM0MzY5MTM0Mg== simonw 9599 2017-11-11T20:19:07Z 2017-11-11T20:19:07Z OWNER

Closing this, opening a fresh ticket for the navigation stuff.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Default HTML/CSS needs to look reasonable and be responsive 267726219  
343697291 https://github.com/simonw/datasette/issues/63#issuecomment-343697291 https://api.github.com/repos/simonw/datasette/issues/63 MDEyOklzc3VlQ29tbWVudDM0MzY5NzI5MQ== simonw 9599 2017-11-11T22:05:06Z 2017-11-11T22:11:49Z OWNER

I'm going to bundle sql and sql_params together into a query nested object like this:

{
    "query": {
        "sql": "select ...",
        "params": {
            "p0": "blah"
        }
    }
}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Review design of JSON output 273174447  
343698214 https://github.com/simonw/datasette/issues/50#issuecomment-343698214 https://api.github.com/repos/simonw/datasette/issues/50 MDEyOklzc3VlQ29tbWVudDM0MzY5ODIxNA== simonw 9599 2017-11-11T22:23:21Z 2017-11-11T22:23:21Z OWNER

I'm closing #50 - more tests will be added in the future, but the framework is neatly in place for them now.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Unit tests against application itself 272694136  
343699115 https://github.com/simonw/datasette/issues/53#issuecomment-343699115 https://api.github.com/repos/simonw/datasette/issues/53 MDEyOklzc3VlQ29tbWVudDM0MzY5OTExNQ== simonw 9599 2017-11-11T22:41:38Z 2017-11-11T22:41:38Z OWNER

This needs to incorporate a sensible way of presenting custom SQL query results too. And let's get a textarea in there for executing SQL while we're at it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Implement a better database index page 273054652  
343752404 https://github.com/simonw/datasette/issues/42#issuecomment-343752404 https://api.github.com/repos/simonw/datasette/issues/42 MDEyOklzc3VlQ29tbWVudDM0Mzc1MjQwNA== simonw 9599 2017-11-12T17:20:10Z 2017-11-12T17:20:10Z OWNER

Re-opening this - I've decided to bring back this concept, see #68

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Homepage UI for editing metadata file 268591332  
343752579 https://github.com/simonw/datasette/issues/69#issuecomment-343752579 https://api.github.com/repos/simonw/datasette/issues/69 MDEyOklzc3VlQ29tbWVudDM0Mzc1MjU3OQ== simonw 9599 2017-11-12T17:22:39Z 2017-11-12T17:22:39Z OWNER

By default I'll allow LIMIT and OFFSET up to a maximum of X (where X is let's say 50,000 to start with, but can be custom configured to a larger number or set to None for no limit).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Enforce pagination (or at least limits) for arbitrary custom SQL 273248366  
343752683 https://github.com/simonw/datasette/issues/66#issuecomment-343752683 https://api.github.com/repos/simonw/datasette/issues/66 MDEyOklzc3VlQ29tbWVudDM0Mzc1MjY4Mw== simonw 9599 2017-11-12T17:24:05Z 2017-11-12T17:24:21Z OWNER

Maybe SQL views should have their own Sanic view class (ViewView is kinda funny), subclassed from TableView?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Show table SQL on table page 273191806  
343753999 https://github.com/simonw/datasette/issues/68#issuecomment-343753999 https://api.github.com/repos/simonw/datasette/issues/68 MDEyOklzc3VlQ29tbWVudDM0Mzc1Mzk5OQ== simonw 9599 2017-11-12T17:45:21Z 2017-11-12T19:38:33Z OWNER

For initial launch, I could just support this as some optional command line arguments you pass to the publish command:

datasette publish data.db --title="Title" --source="url"
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support for title/source/license metadata 273247186  
343754058 https://github.com/simonw/datasette/issues/68#issuecomment-343754058 https://api.github.com/repos/simonw/datasette/issues/68 MDEyOklzc3VlQ29tbWVudDM0Mzc1NDA1OA== simonw 9599 2017-11-12T17:46:13Z 2017-11-12T17:46:13Z OWNER

I’m going to store this stuff in a file called metadata.json and move the existing automatically generated metadata to a file called build.json

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support for title/source/license metadata 273247186  
343769692 https://github.com/simonw/datasette/issues/57#issuecomment-343769692 https://api.github.com/repos/simonw/datasette/issues/57 MDEyOklzc3VlQ29tbWVudDM0Mzc2OTY5Mg== simonw 9599 2017-11-12T21:32:36Z 2017-11-12T21:32:36Z OWNER

I have created a Docker Hub public repository for this: https://hub.docker.com/r/simonwillison/datasette/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ship a Docker image of the whole thing 273127694  
343780039 https://github.com/simonw/datasette/issues/69#issuecomment-343780039 https://api.github.com/repos/simonw/datasette/issues/69 MDEyOklzc3VlQ29tbWVudDM0Mzc4MDAzOQ== simonw 9599 2017-11-13T00:05:27Z 2017-11-13T00:05:27Z OWNER

I think the only safe way to do this is using SQLite .fetchmany(1000) - I can't guarantee that the user has not entered SQL that will outfox a limit in some way. So instead of attempting to edit their SQL, I'll always return 1001 records and let them know if they went over 1000 or not.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Enforce pagination (or at least limits) for arbitrary custom SQL 273248366  
343780141 https://github.com/simonw/datasette/issues/71#issuecomment-343780141 https://api.github.com/repos/simonw/datasette/issues/71 MDEyOklzc3VlQ29tbWVudDM0Mzc4MDE0MQ== simonw 9599 2017-11-13T00:06:52Z 2017-11-13T00:06:52Z OWNER

I've registered datasettes.com as a domain name for doing this. Now setting it up so Cloudflare and Now can serve content from it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Set up some example datasets on a Cloudflare-backed domain 273278840  
343780539 https://github.com/simonw/datasette/issues/71#issuecomment-343780539 https://api.github.com/repos/simonw/datasette/issues/71 MDEyOklzc3VlQ29tbWVudDM0Mzc4MDUzOQ== simonw 9599 2017-11-13T00:13:29Z 2017-11-13T00:19:46Z OWNER

https://zeit.co/docs/features/dns is docs

now domain add -e datasettes.com

I had to set up a custom TXT record on _now.datasettes.com to get this to work.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Set up some example datasets on a Cloudflare-backed domain 273278840  
343780671 https://github.com/simonw/datasette/issues/71#issuecomment-343780671 https://api.github.com/repos/simonw/datasette/issues/71 MDEyOklzc3VlQ29tbWVudDM0Mzc4MDY3MQ== simonw 9599 2017-11-13T00:15:21Z 2017-11-13T00:17:37Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Set up some example datasets on a Cloudflare-backed domain 273278840  
343780814 https://github.com/simonw/datasette/issues/71#issuecomment-343780814 https://api.github.com/repos/simonw/datasette/issues/71 MDEyOklzc3VlQ29tbWVudDM0Mzc4MDgxNA== simonw 9599 2017-11-13T00:17:50Z 2017-11-13T00:18:19Z OWNER

Achieved those redirects using Cloudflare "page rules": https://www.cloudflare.com/a/page-rules/datasettes.com

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Set up some example datasets on a Cloudflare-backed domain 273278840  
343781030 https://github.com/simonw/datasette/issues/71#issuecomment-343781030 https://api.github.com/repos/simonw/datasette/issues/71 MDEyOklzc3VlQ29tbWVudDM0Mzc4MTAzMA== simonw 9599 2017-11-13T00:21:05Z 2017-11-13T02:09:32Z OWNER
  • Have now domain add -e datasettes.com run without errors (hopefully just a matter of waiting for the DNS to update)
  • Alias an example dataset hosted on Now on a datasettes.com subdomain
  • Confirm that HTTP caching and HTTP/2 redirect pushing works as expected - this may require another page rule
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Set up some example datasets on a Cloudflare-backed domain 273278840  
343788581 https://github.com/simonw/datasette/issues/71#issuecomment-343788581 https://api.github.com/repos/simonw/datasette/issues/71 MDEyOklzc3VlQ29tbWVudDM0Mzc4ODU4MQ== simonw 9599 2017-11-13T01:48:17Z 2017-11-13T01:48:17Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Set up some example datasets on a Cloudflare-backed domain 273278840  
343788780 https://github.com/simonw/datasette/issues/71#issuecomment-343788780 https://api.github.com/repos/simonw/datasette/issues/71 MDEyOklzc3VlQ29tbWVudDM0Mzc4ODc4MA== simonw 9599 2017-11-13T01:50:01Z 2017-11-13T01:50:01Z OWNER

Added another page rule in order to get Cloudflare to always obey cache headers sent by the server:

https://user-images.githubusercontent.com/9599/32706355-ded7c60a-c7d1-11e7-93da-20989f40d527.png">

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Set up some example datasets on a Cloudflare-backed domain 273278840  
343788817 https://github.com/simonw/datasette/issues/71#issuecomment-343788817 https://api.github.com/repos/simonw/datasette/issues/71 MDEyOklzc3VlQ29tbWVudDM0Mzc4ODgxNw== simonw 9599 2017-11-13T01:50:27Z 2017-11-13T01:50:27Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Set up some example datasets on a Cloudflare-backed domain 273278840  
343789162 https://github.com/simonw/datasette/issues/71#issuecomment-343789162 https://api.github.com/repos/simonw/datasette/issues/71 MDEyOklzc3VlQ29tbWVudDM0Mzc4OTE2Mg== simonw 9599 2017-11-13T01:53:29Z 2017-11-13T01:53:29Z OWNER

```
$ curl -i 'https://fivethirtyeight.datasettes.com/fivethirtyeight-75d605c/obama-commutations%2Fobama_commutations.csv.jsono'
HTTP/1.1 200 OK
Date: Mon, 13 Nov 2017 01:50:57 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=de836090f3e12a60579cc7a1696cf0d9e1510537857; expires=Tue, 13-Nov-18 01:50:57 GMT; path=/; domain=.datasettes.com; HttpOnly; Secure
Access-Control-Allow-Origin: * Cache-Control: public, max-age=31536000
X-Now-Region: now-sfo
CF-Cache-Status: HIT
Expires: Tue, 13 Nov 2018 01:50:57 GMT
Server: cloudflare-nginx
CF-RAY: 3bce154a6d9293b4-SJC

{"database": "fivethirtyeight", "table": "obama-commutations/obama_commutations.csv"...```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Set up some example datasets on a Cloudflare-backed domain 273278840  
343790984 https://github.com/simonw/datasette/issues/71#issuecomment-343790984 https://api.github.com/repos/simonw/datasette/issues/71 MDEyOklzc3VlQ29tbWVudDM0Mzc5MDk4NA== simonw 9599 2017-11-13T02:09:34Z 2017-11-13T02:09:34Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Set up some example datasets on a Cloudflare-backed domain 273278840  
343791348 https://github.com/simonw/datasette/issues/68#issuecomment-343791348 https://api.github.com/repos/simonw/datasette/issues/68 MDEyOklzc3VlQ29tbWVudDM0Mzc5MTM0OA== simonw 9599 2017-11-13T02:12:58Z 2017-11-13T02:12:58Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support for title/source/license metadata 273247186  
343705966 https://github.com/simonw/datasette/issues/47#issuecomment-343705966 https://api.github.com/repos/simonw/datasette/issues/47 MDEyOklzc3VlQ29tbWVudDM0MzcwNTk2Ng== simonw 9599 2017-11-12T01:00:20Z 2017-11-12T01:00:20Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create neat example database 271831408  
343707676 https://github.com/simonw/datasette/issues/53#issuecomment-343707676 https://api.github.com/repos/simonw/datasette/issues/53 MDEyOklzc3VlQ29tbWVudDM0MzcwNzY3Ng== simonw 9599 2017-11-12T01:49:07Z 2017-11-12T01:49:07Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Implement a better database index page 273054652  
343707624 https://github.com/simonw/datasette/issues/53#issuecomment-343707624 https://api.github.com/repos/simonw/datasette/issues/53 MDEyOklzc3VlQ29tbWVudDM0MzcwNzYyNA== simonw 9599 2017-11-12T01:47:45Z 2017-11-12T01:47:45Z OWNER

Split the SQL thing out into #65

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Implement a better database index page 273054652  
343708447 https://github.com/simonw/datasette/issues/42#issuecomment-343708447 https://api.github.com/repos/simonw/datasette/issues/42 MDEyOklzc3VlQ29tbWVudDM0MzcwODQ0Nw== simonw 9599 2017-11-12T02:12:15Z 2017-11-12T02:12:15Z OWNER

I ditched the metadata file concept.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Homepage UI for editing metadata file 268591332  
343709217 https://github.com/simonw/datasette/issues/65#issuecomment-343709217 https://api.github.com/repos/simonw/datasette/issues/65 MDEyOklzc3VlQ29tbWVudDM0MzcwOTIxNw== simonw 9599 2017-11-12T02:36:37Z 2017-11-12T02:36:37Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Re-implement ?sql= mode 273191608  
343715915 https://github.com/simonw/datasette/issues/25#issuecomment-343715915 https://api.github.com/repos/simonw/datasette/issues/25 MDEyOklzc3VlQ29tbWVudDM0MzcxNTkxNQ== simonw 9599 2017-11-12T06:08:28Z 2017-11-12T06:08:28Z OWNER
con = sqlite3.connect('existing_db.db')
with open('dump.sql', 'w') as f:
    for line in con.iterdump():
        f.write('%s\n' % line)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Endpoint that returns SQL ready to be piped into DB 267857622  
343801392 https://github.com/simonw/datasette/issues/73#issuecomment-343801392 https://api.github.com/repos/simonw/datasette/issues/73 MDEyOklzc3VlQ29tbWVudDM0MzgwMTM5Mg== simonw 9599 2017-11-13T03:36:47Z 2017-11-13T03:36:47Z OWNER

While I’m at it, let’s allow people to opt out of HTTP/2 push with a ?_nopush=1 argument too - in case they decide they don’t want to receive large 302 responses.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
_nocache=1 query string option for use with sort-by-random 273296178  
343951751 https://github.com/simonw/datasette/issues/68#issuecomment-343951751 https://api.github.com/repos/simonw/datasette/issues/68 MDEyOklzc3VlQ29tbWVudDM0Mzk1MTc1MQ== simonw 9599 2017-11-13T15:21:04Z 2017-11-13T15:21:04Z OWNER

For first version, I'm just supporting title, source and license information at the database level.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support for title/source/license metadata 273247186  
343961784 https://github.com/simonw/datasette/issues/67#issuecomment-343961784 https://api.github.com/repos/simonw/datasette/issues/67 MDEyOklzc3VlQ29tbWVudDM0Mzk2MTc4NA== simonw 9599 2017-11-13T15:50:50Z 2017-11-13T15:50:50Z OWNER

datasette package ... - same arguments as datasette publish. Creates Docker container in your local repo, optionally tagged with --tag

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command that builds a local docker container 273192789  
343967020 https://github.com/simonw/datasette/issues/67#issuecomment-343967020 https://api.github.com/repos/simonw/datasette/issues/67 MDEyOklzc3VlQ29tbWVudDM0Mzk2NzAyMA== simonw 9599 2017-11-13T16:06:10Z 2017-11-13T16:06:10Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command that builds a local docker container 273192789  
344048656 https://github.com/simonw/datasette/issues/69#issuecomment-344048656 https://api.github.com/repos/simonw/datasette/issues/69 MDEyOklzc3VlQ29tbWVudDM0NDA0ODY1Ng== simonw 9599 2017-11-13T20:32:47Z 2017-11-13T20:32:47Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Enforce pagination (or at least limits) for arbitrary custom SQL 273248366  
344060070 https://github.com/simonw/datasette/issues/55#issuecomment-344060070 https://api.github.com/repos/simonw/datasette/issues/55 MDEyOklzc3VlQ29tbWVudDM0NDA2MDA3MA== simonw 9599 2017-11-13T21:14:13Z 2017-11-13T21:14:13Z OWNER

I'm going to add some extra metadata to setup.py and then tag this as version 0.8:

git tag 0.8
git push --tags

Then to ship to PyPI:

python setup.py bdist_wheel
twine register dist/datasette-0.8-py3-none-any.whl
twine upload dist/datasette-0.8-py3-none-any.whl
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ship first version to PyPI 273127117  
344061762 https://github.com/simonw/datasette/issues/55#issuecomment-344061762 https://api.github.com/repos/simonw/datasette/issues/55 MDEyOklzc3VlQ29tbWVudDM0NDA2MTc2Mg== simonw 9599 2017-11-13T21:19:43Z 2017-11-13T21:19:43Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ship first version to PyPI 273127117  
344074443 https://github.com/simonw/datasette/issues/80#issuecomment-344074443 https://api.github.com/repos/simonw/datasette/issues/80 MDEyOklzc3VlQ29tbWVudDM0NDA3NDQ0Mw== simonw 9599 2017-11-13T22:04:54Z 2017-11-13T22:05:02Z OWNER

The fivethirtyeight dataset:

datasette publish now --name fivethirtyeight --metadata metadata.json fivethirtyeight.db
now alias https://fivethirtyeight-jyqfudvjli.now.sh fivethirtyeight.datasettes.com

And parlgov:

datasette publish now parlgov.db --name=parlgov --metadata=parlgov.json 
now alias https://parlgov-hqvxuhmbyh.now.sh parlgov.datasettes.com
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Deploy final versions of fivethirtyeight and parlgov datasets (with view pagination) 273569477  
344075696 https://github.com/simonw/datasette/issues/80#issuecomment-344075696 https://api.github.com/repos/simonw/datasette/issues/80 MDEyOklzc3VlQ29tbWVudDM0NDA3NTY5Ng== simonw 9599 2017-11-13T22:09:46Z 2017-11-13T22:09:46Z OWNER

Parlgov was throwing errors on one of the views, which takes longer than 1000ms to execute - so I added the ability to customize the time limit in https://github.com/simonw/datasette/commit/1e698787a4dd6df0432021a6814c446c8b69bba2

datasette publish now parlgov.db --metadata parlgov.json --name parlgov --extra-options="--sql_time_limit_ms=3500"
now alias https://parlgov-nvkcowlixq.now.sh parlgov.datasettes.com

https://parlgov.datasettes.com/parlgov-25f9855/view_cabinet now returns in just over 2.5s

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Deploy final versions of fivethirtyeight and parlgov datasets (with view pagination) 273569477  
344076554 https://github.com/simonw/datasette/pull/81#issuecomment-344076554 https://api.github.com/repos/simonw/datasette/issues/81 MDEyOklzc3VlQ29tbWVudDM0NDA3NjU1NA== simonw 9599 2017-11-13T22:12:57Z 2017-11-13T22:12:57Z OWNER

Hah, I haven't even announced this yet :) Travis is upset because I'm using SQL in the tests which isn't compatible with their version of Python 3.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
:fire: Removes DS_Store 273595473  
344081876 https://github.com/simonw/datasette/issues/59#issuecomment-344081876 https://api.github.com/repos/simonw/datasette/issues/59 MDEyOklzc3VlQ29tbWVudDM0NDA4MTg3Ng== simonw 9599 2017-11-13T22:33:43Z 2017-11-13T22:33:43Z OWNER

The datasette package command introduced in 4143e3b45c16cbae5e3e3419ef479a71810e7df3 is relevant here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish hyper 273157085  
344000982 https://github.com/simonw/datasette/issues/75#issuecomment-344000982 https://api.github.com/repos/simonw/datasette/issues/75 MDEyOklzc3VlQ29tbWVudDM0NDAwMDk4Mg== simonw 9599 2017-11-13T17:50:27Z 2017-11-13T17:50:27Z OWNER

This is necessary because one of the fun things to do with this tool is run it locally, e.g.:

 datasette ~/Library/Application\ Support/Google/Chrome/Default/History -p 8003

BUT... if we enable CORS by default, an evil site could try sniffing for localhost:8003 and attempt to steal data.

So we'll enable the CORS headers only if --cors is provided to the command, and then use that command in the default Dockerfile.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add --cors argument to serve 273509159  
344017088 https://github.com/simonw/datasette/issues/51#issuecomment-344017088 https://api.github.com/repos/simonw/datasette/issues/51 MDEyOklzc3VlQ29tbWVudDM0NDAxNzA4OA== simonw 9599 2017-11-13T18:44:23Z 2017-11-13T18:44:23Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make a proper README 272735257  
344018680 https://github.com/simonw/datasette/issues/74#issuecomment-344018680 https://api.github.com/repos/simonw/datasette/issues/74 MDEyOklzc3VlQ29tbWVudDM0NDAxODY4MA== simonw 9599 2017-11-13T18:49:58Z 2017-11-13T18:49:58Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Send a 302 redirect to the new hash for hits to old hashes 273296684  
344019631 https://github.com/simonw/datasette/issues/69#issuecomment-344019631 https://api.github.com/repos/simonw/datasette/issues/69 MDEyOklzc3VlQ29tbWVudDM0NDAxOTYzMQ== simonw 9599 2017-11-13T18:53:13Z 2017-11-13T18:53:13Z OWNER

I'm going with a page size of 100 and a max limit of 1000

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Enforce pagination (or at least limits) for arbitrary custom SQL 273248366  
344141199 https://github.com/simonw/datasette/issues/59#issuecomment-344141199 https://api.github.com/repos/simonw/datasette/issues/59 MDEyOklzc3VlQ29tbWVudDM0NDE0MTE5OQ== simonw 9599 2017-11-14T04:13:11Z 2017-11-14T04:13:11Z OWNER

I managed to do this manually:

datasette package ~/parlgov-db/parlgov.db --metadata=parlgov.json
# Output 8758ec31dda3 as the new image ID
docker save 8758ec31dda3 > /tmp/my-image
# I could have just piped this straight to hyper
cat /tmp/my-image | hyper load
# Now start the container running in hyper
hyper run -d -p 80:8001 --name parlgov 8758ec31dda3
# We need to assign an IP address so we can see it
hyper fip allocate 1
# Outputs 199.245.58.78
hyper fip attach 199.245.58.78 parlgov

At this point, visiting the IP address in a browser showed the parlgov UI.

To clean up...

hyper hyper fip detach parlgov
hyper fip release 199.245.58.78
hyper stop parlgov
hyper rm parlgov
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish hyper 273157085  
344141515 https://github.com/simonw/datasette/issues/79#issuecomment-344141515 https://api.github.com/repos/simonw/datasette/issues/79 MDEyOklzc3VlQ29tbWVudDM0NDE0MTUxNQ== simonw 9599 2017-11-14T04:16:01Z 2017-11-14T04:16:01Z OWNER

This is probably a bit too much for the README - I should get readthedocs working.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add more detailed API documentation to the README 273569068  
344145265 https://github.com/simonw/datasette/issues/57#issuecomment-344145265 https://api.github.com/repos/simonw/datasette/issues/57 MDEyOklzc3VlQ29tbWVudDM0NDE0NTI2NQ== macropin 247192 2017-11-14T04:45:38Z 2017-11-14T04:45:38Z CONTRIBUTOR

I'm happy to contribute this. Just let me know if you want a Dockerfile for development or production purposes, or both.

If it's prod then we can just pip install the source from pypi, otherwise for dev we'll need a requirements.txt to speed up rebuilds.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ship a Docker image of the whole thing 273127694  
344147583 https://github.com/simonw/datasette/issues/57#issuecomment-344147583 https://api.github.com/repos/simonw/datasette/issues/57 MDEyOklzc3VlQ29tbWVudDM0NDE0NzU4Mw== macropin 247192 2017-11-14T05:03:47Z 2017-11-14T05:03:47Z CONTRIBUTOR

Let me know if you'd like a PR. The image is usable as
docker run --rm -t -i -p 9000:8001 -v $(pwd)/db:/db datasette datasette serve /db/chinook.db

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ship a Docker image of the whole thing 273127694  
344149165 https://github.com/simonw/datasette/issues/57#issuecomment-344149165 https://api.github.com/repos/simonw/datasette/issues/57 MDEyOklzc3VlQ29tbWVudDM0NDE0OTE2NQ== simonw 9599 2017-11-14T05:16:34Z 2017-11-14T05:17:14Z OWNER

I’m intrigued by this pattern:

https://github.com/macropin/datasette/blob/147195c2fdfa2b984d8f9fc1c6cab6634970a056/Dockerfile#L8

What’s the benefit of doing that? Does it result in a smaller image size?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ship a Docker image of the whole thing 273127694  
344151223 https://github.com/simonw/datasette/issues/57#issuecomment-344151223 https://api.github.com/repos/simonw/datasette/issues/57 MDEyOklzc3VlQ29tbWVudDM0NDE1MTIyMw== macropin 247192 2017-11-14T05:32:28Z 2017-11-14T05:33:03Z CONTRIBUTOR

The pattern is called "multi-stage builds". And the result is a svelte 226MB image (201MB for 3.6-slim) vs 700MB+ for the full image. It's possible to get it even smaller, but that takes a lot more work.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ship a Docker image of the whole thing 273127694  
344161226 https://github.com/simonw/datasette/issues/46#issuecomment-344161226 https://api.github.com/repos/simonw/datasette/issues/46 MDEyOklzc3VlQ29tbWVudDM0NDE2MTIyNg== simonw 9599 2017-11-14T06:41:21Z 2017-11-14T06:41:21Z OWNER

Spatial extensions would be really useful too. https://www.gaia-gis.it/spatialite-2.1/SpatiaLite-manual.html

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Dockerfile should build more recent SQLite with FTS5 and spatialite support 271301468  
344161371 https://github.com/simonw/datasette/issues/46#issuecomment-344161371 https://api.github.com/repos/simonw/datasette/issues/46 MDEyOklzc3VlQ29tbWVudDM0NDE2MTM3MQ== simonw 9599 2017-11-14T06:42:15Z 2017-11-14T06:42:15Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Dockerfile should build more recent SQLite with FTS5 and spatialite support 271301468  
344161430 https://github.com/simonw/datasette/issues/46#issuecomment-344161430 https://api.github.com/repos/simonw/datasette/issues/46 MDEyOklzc3VlQ29tbWVudDM0NDE2MTQzMA== simonw 9599 2017-11-14T06:42:44Z 2017-11-14T06:42:44Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Dockerfile should build more recent SQLite with FTS5 and spatialite support 271301468  
344179878 https://github.com/simonw/datasette/issues/27#issuecomment-344179878 https://api.github.com/repos/simonw/datasette/issues/27 MDEyOklzc3VlQ29tbWVudDM0NDE3OTg3OA== simonw 9599 2017-11-14T08:21:22Z 2017-11-14T08:21:22Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to plot a simple graph 267886330  
344180866 https://github.com/simonw/datasette/issues/43#issuecomment-344180866 https://api.github.com/repos/simonw/datasette/issues/43 MDEyOklzc3VlQ29tbWVudDM0NDE4MDg2Ng== simonw 9599 2017-11-14T08:25:37Z 2017-11-14T08:25:37Z OWNER

This isn’t necessary - restarting the server is fast and easy, and I’ve not found myself needing this at all during development.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
While running, server should spot new db files added to its directory  268592894  
344185817 https://github.com/simonw/datasette/issues/57#issuecomment-344185817 https://api.github.com/repos/simonw/datasette/issues/57 MDEyOklzc3VlQ29tbWVudDM0NDE4NTgxNw== simonw 9599 2017-11-14T08:46:24Z 2017-11-14T08:46:24Z OWNER

Thanks for the explanation! Please do start a pull request.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ship a Docker image of the whole thing 273127694  
344118849 https://github.com/simonw/datasette/issues/82#issuecomment-344118849 https://api.github.com/repos/simonw/datasette/issues/82 MDEyOklzc3VlQ29tbWVudDM0NDExODg0OQ== simonw 9599 2017-11-14T01:46:10Z 2017-11-14T01:46:10Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Post a blog entry announcing it to the world 273596159  
344125441 https://github.com/simonw/datasette/pull/81#issuecomment-344125441 https://api.github.com/repos/simonw/datasette/issues/81 MDEyOklzc3VlQ29tbWVudDM0NDEyNTQ0MQ== jefftriplett 50527 2017-11-14T02:24:54Z 2017-11-14T02:24:54Z CONTRIBUTOR

Oops, if I jumped the gun. I saw the project in my github activity feed and saw some low hanging fruit :)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
:fire: Removes DS_Store 273595473  
344132481 https://github.com/simonw/datasette/issues/47#issuecomment-344132481 https://api.github.com/repos/simonw/datasette/issues/47 MDEyOklzc3VlQ29tbWVudDM0NDEzMjQ4MQ== simonw 9599 2017-11-14T03:08:13Z 2017-11-14T03:08:13Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create neat example database 271831408  
344352573 https://github.com/simonw/datasette/issues/30#issuecomment-344352573 https://api.github.com/repos/simonw/datasette/issues/30 MDEyOklzc3VlQ29tbWVudDM0NDM1MjU3Mw== simonw 9599 2017-11-14T18:29:01Z 2017-11-14T18:29:01Z OWNER

This is a dupe of #85

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Do something neat with foreign keys 268078453  
344440377 https://github.com/simonw/datasette/issues/93#issuecomment-344440377 https://api.github.com/repos/simonw/datasette/issues/93 MDEyOklzc3VlQ29tbWVudDM0NDQ0MDM3Nw== simonw 9599 2017-11-14T23:56:35Z 2017-11-14T23:56:35Z OWNER

It worked!

$ pyinstaller -F \
    --add-data /usr/local/lib/python3.5/site-packages/datasette/templates:datasette/templates \
    --add-data /usr/local/lib/python3.5/site-packages/datasette/static:datasette/static \
    /usr/local/bin/datasette

$ file dist/datasette 
dist/datasette: Mach-O 64-bit executable x86_64
$ dist/datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --help  Show this message and exit.

Commands:
  serve*   Serve up specified SQLite database files with...
  build
  package  Package specified SQLite files into a new...
  publish  Publish specified SQLite database files to...
{
    "total_count": 2,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 2,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Package as standalone binary 273944952  
344440658 https://github.com/simonw/datasette/issues/93#issuecomment-344440658 https://api.github.com/repos/simonw/datasette/issues/93 MDEyOklzc3VlQ29tbWVudDM0NDQ0MDY1OA== simonw 9599 2017-11-14T23:58:07Z 2017-11-14T23:58:07Z OWNER

It's a shame pyinstaller can't act as a cross-compiler - so I don't think I can get Travis CI to build packages. But it's fantastic that it's possible to turn the tool into a standalone executable!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Package as standalone binary 273944952  
344452063 https://github.com/simonw/datasette/issues/85#issuecomment-344452063 https://api.github.com/repos/simonw/datasette/issues/85 MDEyOklzc3VlQ29tbWVudDM0NDQ1MjA2Mw== simonw 9599 2017-11-15T01:03:03Z 2017-11-15T01:03:03Z OWNER

This can work in reverse too. If you view the row page for something that has foreign keys against it, we can show you “53 items in TABLE link to this” and provide a link to view them all.

That count worry could be prohibitively expensive. To counter that, we could run the count query via Ajax and set a strict time limit on it. See #95

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Detect foreign keys and use them to link HTML pages together 273678673  
344452326 https://github.com/simonw/datasette/issues/85#issuecomment-344452326 https://api.github.com/repos/simonw/datasette/issues/85 MDEyOklzc3VlQ29tbWVudDM0NDQ1MjMyNg== simonw 9599 2017-11-15T01:04:38Z 2017-11-15T01:04:38Z OWNER

This will work well in conjunction with https://github.com/simonw/csvs-to-sqlite/issues/2

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Detect foreign keys and use them to link HTML pages together 273678673  
344462277 https://github.com/simonw/datasette/pull/89#issuecomment-344462277 https://api.github.com/repos/simonw/datasette/issues/89 MDEyOklzc3VlQ29tbWVudDM0NDQ2MjI3Nw== simonw 9599 2017-11-15T02:02:52Z 2017-11-15T02:02:52Z OWNER

This is exactly what I was after, thanks!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
SQL syntax highlighting with CodeMirror 273816720  
344462608 https://github.com/simonw/datasette/issues/13#issuecomment-344462608 https://api.github.com/repos/simonw/datasette/issues/13 MDEyOklzc3VlQ29tbWVudDM0NDQ2MjYwOA== simonw 9599 2017-11-15T02:04:51Z 2017-11-15T02:04:51Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add a syntax highlighting SQL editor 267542338  
344463436 https://github.com/simonw/datasette/issues/95#issuecomment-344463436 https://api.github.com/repos/simonw/datasette/issues/95 MDEyOklzc3VlQ29tbWVudDM0NDQ2MzQzNg== simonw 9599 2017-11-15T02:10:10Z 2017-11-15T02:10:10Z OWNER

This means clients can ask questions but say "don't bother if it takes longer than X" - which is really handy when you're working against unknown databases that might be small or might be enormous.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow shorter time limits to be set using a ?_sql_time_limit_ms =20 query string limit 273998513  
344472313 https://github.com/simonw/datasette/pull/94#issuecomment-344472313 https://api.github.com/repos/simonw/datasette/issues/94 MDEyOklzc3VlQ29tbWVudDM0NDQ3MjMxMw== simonw 9599 2017-11-15T03:08:00Z 2017-11-15T03:08:00Z OWNER

Works for me. I'm going to land this.

Just one thing:

simonw$ docker run --rm -t -i -p 9001:8001 c408e8cfbe40 datasette publish now
 The publish command requires "now" to be installed and configured 
Follow the instructions at https://zeit.co/now#whats-now

Maybe we should have the Docker container install the "now" client? Not sure how much size that would add though. I think it's OK without for the moment.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Initial add simple prod ready Dockerfile refs #57 273961179  
344487639 https://github.com/simonw/datasette/issues/25#issuecomment-344487639 https://api.github.com/repos/simonw/datasette/issues/25 MDEyOklzc3VlQ29tbWVudDM0NDQ4NzYzOQ== simonw 9599 2017-11-15T05:11:11Z 2017-11-15T05:11:11Z OWNER

Since you can already download the database directly, I'm not going to bother with this one.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Endpoint that returns SQL ready to be piped into DB 267857622  
344409906 https://github.com/simonw/datasette/issues/93#issuecomment-344409906 https://api.github.com/repos/simonw/datasette/issues/93 MDEyOklzc3VlQ29tbWVudDM0NDQwOTkwNg== simonw 9599 2017-11-14T21:47:02Z 2017-11-14T21:47:02Z OWNER

Even without bundling in the database file itself, I'd love to have a standalone binary version of the core datasette CLI utility.

I think Sanic may have some complex dependencies, but I've never tried pyinstaller so I don't know how easy or hard it would be to get this working.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Package as standalone binary 273944952  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);