issues

Table actions

49 rows where milestone = 2857392 and repo = 107914493 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: comments, created_at (date), updated_at (date), closed_at (date)

type 1

state 1

repo 1

  • datasette · 49 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
268469569 MDU6SXNzdWUyNjg0Njk1Njk= 39 Protect against malicious SQL that causes damage even though our DB is immutable simonw 9599 closed 0   Ship first public release 2857392 4 2017-10-25T16:44:27Z 2021-08-17T23:52:07Z 2017-11-05T02:53:47Z OWNER  

I’m currently operating under the assumption that it’s safe to allow arbitrary SQL statements because we are dealing with an immutable database. But this might not be the case - there are some pretty weird SQLite language extensions (ATTACH, PRAGMA etc) and I’m not certain they cannot be used to break things in a way that would affect future requests to the API.

Solution: provide a “safe mode” option which disables the ?sql= mechanism. This still leaves the URL filter lookups, so I need to make sure that those are “safe”.

In the future I may also implement a whitelist option where datasets can be configured to only allow specific filters against specific columns.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/39/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267769431 MDU6SXNzdWUyNjc3Njk0MzE= 22 Refactor to use class based views simonw 9599 closed 0   Ship first public release 2857392 0 2017-10-23T18:26:22Z 2019-05-27T20:05:56Z 2017-10-24T02:25:53Z OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/22/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273247186 MDU6SXNzdWUyNzMyNDcxODY= 68 Support for title/source/license metadata simonw 9599 closed 0   Ship first public release 2857392 4 2017-11-12T17:04:21Z 2017-12-04T04:55:43Z 2017-11-13T15:26:11Z OWNER  

I've decided this is important for launch: I want to set a precedent for people citing, licensing and documenting their datasets.

Not sure how best to go about supporting this. I'd like to allow for the following data to be optionally attached to any given database:

  • Title
  • Description, potentially in markdown?
  • Original source URL
  • License

I'd also like the ability to attach descriptions to individual tables - and maybe even to table columns?

The question then becomes: how should this information be stored. A few options:

  • In the SQLite database itself, in a specially named table. Problem here is that this means having to modify SQLite databases before publishing them.
  • In a separate SQLite database that can be published alongside the databases we are publishing.
  • In a JSON file. This is neat, but JSON files are not a great editing experience once you start including multiple lines (e.g. a markdown description).
  • In a YAML file. This is a better format for multi-line descriptions, but still isn't a great editing experience.

Whatever the format, it can be made much more usable by offering a web-based editing UI for populating it (a special mode the server can be run in).

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/68/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273596159 MDU6SXNzdWUyNzM1OTYxNTk= 82 Post a blog entry announcing it to the world simonw 9599 closed 0   Ship first public release 2857392 1 2017-11-13T22:10:35Z 2017-11-14T01:46:10Z 2017-11-14T01:46:10Z OWNER  
datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/82/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273569477 MDU6SXNzdWUyNzM1Njk0Nzc= 80 Deploy final versions of fivethirtyeight and parlgov datasets (with view pagination) simonw 9599 closed 0   Ship first public release 2857392 2 2017-11-13T20:37:46Z 2017-11-13T22:09:46Z 2017-11-13T22:09:46Z OWNER  

Final versions should be deployed using the first released version of datasette.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/80/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273537940 MDU6SXNzdWUyNzM1Mzc5NDA= 77 Add Travis CI badge to README simonw 9599 closed 0   Ship first public release 2857392 0 2017-11-13T18:52:25Z 2017-11-13T21:24:15Z 2017-11-13T21:24:15Z OWNER  

Also fix this newline issue:

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/77/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273127117 MDU6SXNzdWUyNzMxMjcxMTc= 55 Ship first version to PyPI simonw 9599 closed 0   Ship first public release 2857392 2 2017-11-11T07:38:48Z 2017-11-13T21:19:43Z 2017-11-13T21:19:43Z OWNER  

Just before doing this, update the Dockerfile template to pip install datasette https://github.com/simonw/datasette/blob/65e350ca2a4845c25752a62c16ba58cfe2c14b9b/datasette/utils.py#L125

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/55/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273267081 MDU6SXNzdWUyNzMyNjcwODE= 70 Paginate views using OFFSET/LIMIT simonw 9599 closed 0   Ship first public release 2857392 0 2017-11-12T21:30:29Z 2017-11-13T21:11:01Z 2017-11-13T21:11:01Z OWNER  

As with #69 these should obey a maximum offset setting, which can be over-ridden.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/70/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273554949 MDU6SXNzdWUyNzM1NTQ5NDk= 78 Rename after to next and provide a next_url simonw 9599 closed 0   Ship first public release 2857392 0 2017-11-13T19:48:31Z 2017-11-13T20:35:03Z 2017-11-13T20:35:03Z OWNER  
datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/78/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273248366 MDU6SXNzdWUyNzMyNDgzNjY= 69 Enforce pagination (or at least limits) for arbitrary custom SQL simonw 9599 closed 0   Ship first public release 2857392 4 2017-11-12T17:21:33Z 2017-11-13T20:32:47Z 2017-11-13T19:35:47Z OWNER  

It's way too easy to accidentally trigger a page that returns 100,000 rows at the moment. I need to use the LIMIT clause on views and custom SQL - I can support pagination "next" links using offset as well.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/69/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273296684 MDU6SXNzdWUyNzMyOTY2ODQ= 74 Send a 302 redirect to the new hash for hits to old hashes simonw 9599 closed 0   Ship first public release 2857392 1 2017-11-13T03:00:59Z 2017-11-13T18:49:59Z 2017-11-13T18:49:59Z OWNER  
datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/74/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
272735257 MDU6SXNzdWUyNzI3MzUyNTc= 51 Make a proper README simonw 9599 closed 0   Ship first public release 2857392 1 2017-11-09T21:46:07Z 2017-11-13T18:44:23Z 2017-11-13T18:44:23Z OWNER  

Include instructions on building a local Docker container - currently detailed here: https://gist.github.com/simonw/0ea5c960608c2d876e4637a5e48aa95d (those instructions don't work now that we have removed the Dockerfile in favour of a template generated by datasette publish)

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/51/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273510781 MDU6SXNzdWUyNzM1MTA3ODE= 76 publish should have required argument specifying publisher simonw 9599 closed 0   Ship first public release 2857392 0 2017-11-13T17:21:26Z 2017-11-13T18:41:01Z 2017-11-13T18:41:01Z OWNER  

Initially the only argument will be “now” - but “hyper” can be added in the future

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/76/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273509159 MDU6SXNzdWUyNzM1MDkxNTk= 75 Add --cors argument to serve simonw 9599 closed 0   Ship first public release 2857392 1 2017-11-13T17:16:19Z 2017-11-13T18:17:52Z 2017-11-13T18:17:52Z OWNER  
datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/75/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273192789 MDU6SXNzdWUyNzMxOTI3ODk= 67 Command that builds a local docker container simonw 9599 closed 0   Ship first public release 2857392 2 2017-11-12T02:13:29Z 2017-11-13T16:17:52Z 2017-11-13T16:17:52Z OWNER  

Be nice to indicate that this isn't just for Now. Shouldn't be too hard either.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/67/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273283166 MDU6SXNzdWUyNzMyODMxNjY= 72 publish command should take an optional --name argument simonw 9599 closed 0   Ship first public release 2857392 0 2017-11-13T00:59:35Z 2017-11-13T02:12:27Z 2017-11-13T02:12:27Z OWNER  

To set the directory name so that now will inherit it as the name of the app.

Defaults to datasette

datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/72/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273278840 MDU6SXNzdWUyNzMyNzg4NDA= 71 Set up some example datasets on a Cloudflare-backed domain simonw 9599 closed 0   Ship first public release 2857392 10 2017-11-13T00:06:30Z 2017-11-13T02:09:34Z 2017-11-13T02:09:34Z OWNER  

To better demonstrate the caching and HTTP/2 features, I'd like to go live with some demos that are hosted behind Cloudflare.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/71/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273121803 MDU6SXNzdWUyNzMxMjE4MDM= 54 Views should not attempt to link to records / use rowids simonw 9599 closed 0   Ship first public release 2857392 1 2017-11-11T05:44:54Z 2017-11-12T21:29:42Z 2017-11-12T21:29:33Z OWNER  

http://localhost:8001/parlgov-development-25f9855/view_variable

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/54/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273054652 MDU6SXNzdWUyNzMwNTQ2NTI= 53 Implement a better database index page simonw 9599 closed 0   Ship first public release 2857392 3 2017-11-10T20:47:36Z 2017-11-12T21:19:33Z 2017-11-12T01:50:27Z OWNER  

This view isn't great. I should do a better job of separating out tables from views and indexes, showing the count of rows in each table, and maybe move the SQL to the individual table pages.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/53/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273191806 MDU6SXNzdWUyNzMxOTE4MDY= 66 Show table SQL on table page simonw 9599 closed 0   Ship first public release 2857392 1 2017-11-12T01:51:23Z 2017-11-12T21:17:29Z 2017-11-12T21:17:29Z OWNER  

Let's do the SQL for the table you are looking at, plus SQL for any indexes that mention that table.

The page for a view should show the SQL for that view.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/66/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273191608 MDU6SXNzdWUyNzMxOTE2MDg= 65 Re-implement ?sql= mode simonw 9599 closed 0   Ship first public release 2857392 1 2017-11-12T01:47:17Z 2017-11-12T02:36:37Z 2017-11-12T02:35:42Z OWNER  

Here's the code I removed:

async def data(self, request, name, hash):
    sql = 'select * from sqlite_master'
    custom_sql = False
    params = {}
    if request.args.get('sql'):
        params = request.raw_args
        sql = params.pop('sql')
        validate_sql_select(sql)
        custom_sql = True
    rows = await self.execute(name, sql, params)
    columns = [r[0] for r in rows.description]
    return {
        'database': name,
        'rows': rows,
        'columns': columns,
        'query': {
            'sql': sql,
            'params': params,
        }
    }, {
        'database_hash': hash,
        'custom_sql': custom_sql,
    }
datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/65/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273174397 MDU6SXNzdWUyNzMxNzQzOTc= 62 Link to .json and .jsono versions on various pages simonw 9599 closed 0   Ship first public release 2857392 0 2017-11-11T20:37:47Z 2017-11-11T22:41:06Z 2017-11-11T22:41:06Z OWNER  
datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/62/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
272694136 MDU6SXNzdWUyNzI2OTQxMzY= 50 Unit tests against application itself simonw 9599 closed 0   Ship first public release 2857392 2 2017-11-09T19:31:49Z 2017-11-11T22:23:22Z 2017-11-11T22:23:22Z OWNER  

Use Sanic’s testing mechanism. Test should create a temporary SQLite database file on disk by executing sql that is stored in the test themselves.

For the moment we can just test the JSON API more thoroughly and just sanity check that the HTML output doesn’t throw any errors.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/50/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273174447 MDU6SXNzdWUyNzMxNzQ0NDc= 63 Review design of JSON output simonw 9599 closed 0   Ship first public release 2857392 1 2017-11-11T20:38:33Z 2017-11-11T22:20:17Z 2017-11-11T22:20:17Z OWNER  
datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/63/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273173116 MDU6SXNzdWUyNzMxNzMxMTY= 61 Common header and footer simonw 9599 closed 0   Ship first public release 2857392 0 2017-11-11T20:20:08Z 2017-11-11T20:37:19Z 2017-11-11T20:37:19Z OWNER  

Split from #16

  • [x] A link to the homepage from some kind of navigation bar in the header
  • [x] link to github.com/simonw/datasette in the footer
  • [x] Slightly better titles (maybe ditch the visited link colours for titles only? should keep those for primary key links)
 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/61/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267726219 MDU6SXNzdWUyNjc3MjYyMTk= 16 Default HTML/CSS needs to look reasonable and be responsive simonw 9599 closed 0   Ship first public release 2857392 6 2017-10-23T16:05:22Z 2017-11-11T20:19:07Z 2017-11-11T20:19:07Z OWNER  

Version one should have the following characteristics:

  • Looks OK
  • Works great on mobile
  • Loads extremely fast
  • No JavaScript! At least not in v1.
 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/16/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273163905 MDU6SXNzdWUyNzMxNjM5MDU= 60 Rethink how metadata is generated and stored simonw 9599 closed 0   Ship first public release 2857392 1 2017-11-11T18:01:28Z 2017-11-11T20:12:17Z 2017-11-11T20:12:16Z OWNER  

I broke the existing mechanism in 407795b61217205625f2d4e084afbf69f1db781b In order to get unit tests for the sanic app working.

I think i should ditch the build-metadata.json cache file entirely and calculate the SHA hashes on startup.

Not sure what to do about the table row counts.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/60/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
273128608 MDU6SXNzdWUyNzMxMjg2MDg= 58 publish command should detect if "now" is installed simonw 9599 closed 0   Ship first public release 2857392 0 2017-11-11T08:10:17Z 2017-11-11T16:00:07Z 2017-11-11T16:00:07Z OWNER  

If now is not installed, it should tell you where to get it.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/58/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
268470572 MDU6SXNzdWUyNjg0NzA1NzI= 40 Implement command-line tool interface simonw 9599 closed 0   Ship first public release 2857392 11 2017-10-25T16:47:15Z 2017-11-11T07:27:33Z 2017-11-11T07:27:33Z OWNER  

The first version needs to take one or more file names or URLs, then generate and deploy an app to Now. It will assume you already have the now command installed and configured.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/40/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267861210 MDU6SXNzdWUyNjc4NjEyMTA= 26 Command line tool for uploading one or more DBs to Now simonw 9599 closed 0   Ship first public release 2857392 3 2017-10-24T00:43:10Z 2017-11-11T07:25:30Z 2017-11-11T07:25:30Z OWNER  

Uploading files appears to be undocumented, but I found it in their code here: https://github.com/zeit/now-cli/blob/0ca7d1fe44ebdf460b64fdc38ba543b8e295ac40/src/providers/sh/util/index.js#L291

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/26/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267516066 MDU6SXNzdWUyNjc1MTYwNjY= 5 Implement sensible query pagination simonw 9599 closed 0   Ship first public release 2857392 3 2017-10-23T01:16:00Z 2017-11-10T20:41:39Z 2017-11-10T20:41:39Z OWNER  
datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/5/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
272661336 MDU6SXNzdWUyNzI2NjEzMzY= 49 Pick a name simonw 9599 closed 0   Ship first public release 2857392 4 2017-11-09T17:56:17Z 2017-11-10T18:33:22Z 2017-11-10T18:33:22Z OWNER  

Options so far:

  • immutabase
  • datasite
  • sqlstatic
  • dbserve
  • sqlserve

Terms to play with:

  • immutable
  • sqlite
  • dataset
  • json
  • static
  • serve
 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/49/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
271242824 MDU6SXNzdWUyNzEyNDI4MjQ= 45 Run SQLite operations in a thread pool simonw 9599 closed 0   Ship first public release 2857392 0 2017-11-05T02:27:12Z 2017-11-05T02:27:34Z 2017-11-05T02:27:33Z OWNER  

Let's run SQLite operations in threads, so we don't end up blocking our core event loop.

These articles are helpful:

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/45/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
268590777 MDU6SXNzdWUyNjg1OTA3Nzc= 41 Homepage should show summary of databases simonw 9599 closed 0   Ship first public release 2857392 1 2017-10-26T00:18:11Z 2017-10-27T04:05:35Z 2017-10-27T04:05:35Z OWNER  

I sch database should have a name, optional description, download link and a summary of the tables

Flights.db Flights and suchlike blah. URL? License? 577373 rows across 14 tables airports, routes, airlines...

Title of the homepage is derived from the databases or can be manually overridden e. “Datasets of Flights, NHS, Blah...” - or if only one database just the title of that.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/41/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267741262 MDU6SXNzdWUyNjc3NDEyNjI= 19 Efficient url for downloading the raw database file simonw 9599 closed 0   Ship first public release 2857392 1 2017-10-23T16:52:17Z 2017-10-25T15:21:16Z 2017-10-25T15:19:37Z OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/19/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267788884 MDU6SXNzdWUyNjc3ODg4ODQ= 23 Support Django-style filters in querystring arguments simonw 9599 closed 0   Ship first public release 2857392 6 2017-10-23T19:29:42Z 2017-10-25T04:23:03Z 2017-10-25T04:23:02Z OWNER  

e.g

/database/table?name__contains=Simon&age__gte=4

Same format as Django: double underscore as the split.

If you need to match against a column that happens to contain a double underscore in its official name, do this:

/database/table?weird__column__exact=Simon

__exact is the default operation if none is supplied.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/23/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267886865 MDU6SXNzdWUyNjc4ODY4NjU= 28 /database?sql= should redirect correctly simonw 9599 closed 0   Ship first public release 2857392 0 2017-10-24T03:38:44Z 2017-10-24T23:54:30Z 2017-10-24T23:54:30Z OWNER  

Needs to redirect to the location with the hash while retaining the query string. This should also work with the .json extension.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/28/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267516329 MDU6SXNzdWUyNjc1MTYzMjk= 6 Better JSON response options simonw 9599 closed 0   Ship first public release 2857392 0 2017-10-23T01:18:47Z 2017-10-24T15:07:58Z 2017-10-24T15:07:58Z OWNER  

Default returns this:

{
    “Columns”: [“id”, “name”, “age”],
    “Rows”: [
         [45, “Simon”, 36]
    ]
}

.jsono instead returns a list of objects each duplicating the headers in its keys.

They both probably share the same pagination mechanism so it might not be a jsono flat list.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/6/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
268050821 MDU6SXNzdWUyNjgwNTA4MjE= 29 Handle bytestring records encoding to JSON simonw 9599 closed 0   Ship first public release 2857392 1 2017-10-24T14:18:45Z 2017-10-24T14:59:00Z 2017-10-24T14:58:47Z OWNER  

http://localhost:8006/northwind-40d049b/Categories.json 500s right now

The string representation of one of the values looks like this:

b"\x15\x1c/\x00\x02\x00

This is a bytestring from the database which cannot be naively converted to a unicode string.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/29/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267828746 MDU6SXNzdWUyNjc4Mjg3NDY= 24 Implement full URL design simonw 9599 closed 0   Ship first public release 2857392 2 2017-10-23T21:49:05Z 2017-10-24T14:12:00Z 2017-10-24T14:12:00Z OWNER  

Full URL design:

/database-name
/database-name.json
/database-name-7sha256
/database-name-7sha256.json
/database-name/table-name
/database-name/table-name.json
/database-name-7sha256/table-name
/database-name-7sha256/table-name.json
/database-name-7sha256/table-name/compound-pk
/database-name-7sha256/table-name/compound-pk.json
datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/24/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267513424 MDU6SXNzdWUyNjc1MTM0MjQ= 1 Addressable pages for every row in a table simonw 9599 closed 0   Ship first public release 2857392 6 2017-10-23T00:44:16Z 2017-10-24T14:11:04Z 2017-10-24T14:11:03Z OWNER   
/database-name-7sha256/table-name/compound-pk
/database-name-7sha256/table-name/compound-pk.json

Tricky part will be figuring out what the private key is - especially since it could be a compound primary key and it might involve different data types.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267517348 MDU6SXNzdWUyNjc1MTczNDg= 9 Initial test suite simonw 9599 closed 0   Ship first public release 2857392 2 2017-10-23T01:28:46Z 2017-10-24T05:55:33Z 2017-10-24T05:55:33Z OWNER  
datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/9/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267515836 MDU6SXNzdWUyNjc1MTU4MzY= 4 Make URLs immutable simonw 9599 closed 0   Ship first public release 2857392 8 2017-10-23T01:13:30Z 2017-10-24T02:38:24Z 2017-10-24T02:38:24Z OWNER  

Absolutely everything should have a far-future expires header

Part of the URL will be the truncated sha1 hash of the database file itself, calculated at build time

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/4/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267516650 MDU6SXNzdWUyNjc1MTY2NTA= 7 Framework where by every page is JSON plus a template simonw 9599 closed 0   Ship first public release 2857392 1 2017-10-23T01:22:03Z 2017-10-24T02:27:25Z 2017-10-24T02:27:25Z OWNER  

Every single page of my interface should be implemented as a function that returns JSON.

I can then build my jinja templates on top of the exact data that would be returned by the API version.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/7/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267732005 MDU6SXNzdWUyNjc3MzIwMDU= 17 In development mode, should still pick up new .db files simonw 9599 closed 0   Ship first public release 2857392 1 2017-10-23T16:22:40Z 2017-10-24T02:26:48Z 2017-10-24T02:26:47Z OWNER  

Follow on from #11

datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/17/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267713226 MDU6SXNzdWUyNjc3MTMyMjY= 15 Support multiple databases simonw 9599 closed 0   Ship first public release 2857392 0 2017-10-23T15:29:51Z 2017-10-24T02:01:38Z 2017-10-24T02:01:38Z OWNER  

I'm going to loop through every database file in the app root directory and bundle all of them.

Each one will be accessible at /databasename

Note this is without the file extension, and we will disallow multiple files with the same name but different extensions.

Supported extensions to start with will be .db and .sqlite and .sqlite3

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/15/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267522549 MDU6SXNzdWUyNjc1MjI1NDk= 11 Code that generates compile-time properties about the database simonw 9599 closed 0   Ship first public release 2857392 1 2017-10-23T02:18:24Z 2017-10-23T16:04:23Z 2017-10-23T16:04:23Z OWNER  

At a minimum this will include:

  • sha hash of each database file
  • list of tables with row counts for each database file
 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/11/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267517314 MDU6SXNzdWUyNjc1MTczMTQ= 8 Attempting an INSERT or UPDATE should return a sane error message simonw 9599 closed 0   Ship first public release 2857392 1 2017-10-23T01:28:25Z 2017-10-23T15:28:12Z 2017-10-23T15:28:08Z OWNER  
datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/8/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
267513523 MDU6SXNzdWUyNjc1MTM1MjM= 2 Initial proof-of-concept simonw 9599 closed 0   Ship first public release 2857392 0 2017-10-23T00:45:37Z 2017-10-23T01:26:39Z 2017-10-23T00:45:53Z OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);