5,578 rows sorted by author_association

View and edit SQL

Suggested facets: created_at (date), updated_at (date)

author_association

id html_url issue_url node_id user created_at updated_at author_association ▼ body reactions issue performed_via_github_app
344125441 https://github.com/simonw/datasette/pull/81#issuecomment-344125441 https://api.github.com/repos/simonw/datasette/issues/81 MDEyOklzc3VlQ29tbWVudDM0NDEyNTQ0MQ== jefftriplett 50527 2017-11-14T02:24:54Z 2017-11-14T02:24:54Z CONTRIBUTOR

Oops, if I jumped the gun. I saw the project in my github activity feed and saw some low hanging fruit :)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
:fire: Removes DS_Store 273595473  
344145265 https://github.com/simonw/datasette/issues/57#issuecomment-344145265 https://api.github.com/repos/simonw/datasette/issues/57 MDEyOklzc3VlQ29tbWVudDM0NDE0NTI2NQ== macropin 247192 2017-11-14T04:45:38Z 2017-11-14T04:45:38Z CONTRIBUTOR

I'm happy to contribute this. Just let me know if you want a Dockerfile for development or production purposes, or both.

If it's prod then we can just pip install the source from pypi, otherwise for dev we'll need a requirements.txt to speed up rebuilds.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ship a Docker image of the whole thing 273127694  
344147583 https://github.com/simonw/datasette/issues/57#issuecomment-344147583 https://api.github.com/repos/simonw/datasette/issues/57 MDEyOklzc3VlQ29tbWVudDM0NDE0NzU4Mw== macropin 247192 2017-11-14T05:03:47Z 2017-11-14T05:03:47Z CONTRIBUTOR

Let me know if you'd like a PR. The image is usable as
docker run --rm -t -i -p 9000:8001 -v $(pwd)/db:/db datasette datasette serve /db/chinook.db

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ship a Docker image of the whole thing 273127694  
344151223 https://github.com/simonw/datasette/issues/57#issuecomment-344151223 https://api.github.com/repos/simonw/datasette/issues/57 MDEyOklzc3VlQ29tbWVudDM0NDE1MTIyMw== macropin 247192 2017-11-14T05:32:28Z 2017-11-14T05:33:03Z CONTRIBUTOR

The pattern is called "multi-stage builds". And the result is a svelte 226MB image (201MB for 3.6-slim) vs 700MB+ for the full image. It's possible to get it even smaller, but that takes a lot more work.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ship a Docker image of the whole thing 273127694  
344430689 https://github.com/simonw/datasette/issues/88#issuecomment-344430689 https://api.github.com/repos/simonw/datasette/issues/88 MDEyOklzc3VlQ29tbWVudDM0NDQzMDY4OQ== tomdyson 15543 2017-11-14T23:08:22Z 2017-11-14T23:08:22Z CONTRIBUTOR

I'm getting an internal server error on http://run.plnkr.co/preview/cj9zlf1qc0003414y90ajkwpk/ at the moment

Sorry about that - here's a working version on Netlify:

https://nhs-england-map.netlify.com

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add NHS England Hospitals example to wiki 273775212  
344710204 https://github.com/simonw/datasette/pull/104#issuecomment-344710204 https://api.github.com/repos/simonw/datasette/issues/104 MDEyOklzc3VlQ29tbWVudDM0NDcxMDIwNA== jacobian 21148 2017-11-15T19:57:50Z 2017-11-15T19:57:50Z CONTRIBUTOR

A first basic stab at making this work, just to prove the approach. Right now this requires a Heroku CLI plugin, which seems pretty unreasonable. I think this can be replaced with direct API calls, which could clean up a lot of things. But I wanted to prove it worked first, and it does.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
[WIP] Add publish to heroku support 274284246  
344810525 https://github.com/simonw/datasette/issues/46#issuecomment-344810525 https://api.github.com/repos/simonw/datasette/issues/46 MDEyOklzc3VlQ29tbWVudDM0NDgxMDUyNQ== ingenieroariel 54999 2017-11-16T04:11:25Z 2017-11-16T04:11:25Z CONTRIBUTOR

@simonw On the spatialite support, here is some info to make it work and a screenshot:

https://user-images.githubusercontent.com/54999/32873420-f8a6d5a0-ca59-11e7-8a73-7d58d467e413.png">

I used the following Dockerfile:

FROM prolocutor/python3-sqlite-ext:3.5.1-spatialite as build

RUN mkdir /code
ADD . /code/

RUN pip install /code/

EXPOSE 8001
CMD ["datasette", "serve", "/code/ne.sqlite", "--host", "0.0.0.0"]

and added this to prepare_connection:

  conn.enable_load_extension(True)
  conn.execute("SELECT load_extension('/usr/local/lib/mod_spatialite.so')")
{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Dockerfile should build more recent SQLite with FTS5 and spatialite support 271301468  
344811268 https://github.com/simonw/datasette/pull/107#issuecomment-344811268 https://api.github.com/repos/simonw/datasette/issues/107 MDEyOklzc3VlQ29tbWVudDM0NDgxMTI2OA== raynae 3433657 2017-11-16T04:17:45Z 2017-11-16T04:17:45Z CONTRIBUTOR

Thanks for the guidance. I added a unit test and made a slight change to utils.py.

I didn't realize this, but evidently string.format only complains if you supply less arguments than there are format placeholders, so the original commit worked, but was adding a superfluous named param.

I added a conditional that prevents the named param from being created and ensures the correct number of args are passed to sting.format. It has the side effect of hiding the SQL query in /templates/table.html when there are no other where clauses--not sure if that's the desired outcome here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
add support for ?field__isnull=1 274343647  
345002908 https://github.com/simonw/datasette/issues/46#issuecomment-345002908 https://api.github.com/repos/simonw/datasette/issues/46 MDEyOklzc3VlQ29tbWVudDM0NTAwMjkwOA== ingenieroariel 54999 2017-11-16T17:47:49Z 2017-11-16T17:47:49Z CONTRIBUTOR

I'll try to find alternatives to the Dockerfile option - I also think we should not use that old one without sources or license.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Dockerfile should build more recent SQLite with FTS5 and spatialite support 271301468  
345117690 https://github.com/simonw/datasette/pull/107#issuecomment-345117690 https://api.github.com/repos/simonw/datasette/issues/107 MDEyOklzc3VlQ29tbWVudDM0NTExNzY5MA== raynae 3433657 2017-11-17T01:29:41Z 2017-11-17T01:29:41Z CONTRIBUTOR

Thanks for bearing with me. I was getting a message about my branch diverging when I tried to push after rebasing, so I merged master into isnull, seems like that did the trick. Let me know if I should make any corrections.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
add support for ?field__isnull=1 274343647  
345452669 https://github.com/simonw/datasette/pull/104#issuecomment-345452669 https://api.github.com/repos/simonw/datasette/issues/104 MDEyOklzc3VlQ29tbWVudDM0NTQ1MjY2OQ== jacobian 21148 2017-11-18T16:18:45Z 2017-11-18T16:18:45Z CONTRIBUTOR

I'd like to do a bit of cleanup, and some error checking in case heroku/heroku-builds isn't installed.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
[WIP] Add publish to heroku support 274284246  
345503897 https://github.com/simonw/datasette/issues/105#issuecomment-345503897 https://api.github.com/repos/simonw/datasette/issues/105 MDEyOklzc3VlQ29tbWVudDM0NTUwMzg5Nw== rgieseke 198537 2017-11-19T09:38:08Z 2017-11-19T09:38:08Z CONTRIBUTOR

Thanks, I wrote this very simple reader because the default approach as described on the Datahub pages seemed to complicated. I had metadata from the datapackage.json attached to the returned DataFrames but removed this due to some attribute handling change in the latest Pandas version.

This could also be useful for getting from Data Package to SQL db: https://github.com/frictionlessdata/tableschema-sql-py

I maintain a few climate science related dataset at https://github.com/openclimatedata/

The Data Retriever (mainly ecological data) by @ethanwhite et al. is also using the Data Package format for metadata and has some tooling for different dbs:

https://frictionlessdata.io/articles/the-data-retriever/
https://github.com/weecology/retriever

The Open Power System Data project also has a couple of datasets that show nicely how CSV is great for assembling and then already make SQLite files available. It's one of the first data sets I tried with Datasette, perfect for the use case of getting an API for putting power stations on a map ...

https://data.open-power-system-data.org/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider data-package as a format for metadata 274314940  
345652450 https://github.com/simonw/datasette/issues/27#issuecomment-345652450 https://api.github.com/repos/simonw/datasette/issues/27 MDEyOklzc3VlQ29tbWVudDM0NTY1MjQ1MA== rgieseke 198537 2017-11-20T10:19:39Z 2017-11-20T10:19:39Z CONTRIBUTOR

If Data Package metadata gets adopted (#105) the views spec work might also be worth a look:

http://frictionlessdata.io/specs/views/

http://datahub.io/docs/features/views

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to plot a simple graph 267886330  
346116745 https://github.com/simonw/datasette/pull/104#issuecomment-346116745 https://api.github.com/repos/simonw/datasette/issues/104 MDEyOklzc3VlQ29tbWVudDM0NjExNjc0NQ== jacobian 21148 2017-11-21T18:23:25Z 2017-11-21T18:23:25Z CONTRIBUTOR

@simonw ready for a review and merge if you want.

There's still some nasty duplicated code in cli.py and utils.py, which is just going to get worse if/when we start adding any other deploy targets (and I want to do one for cloud.gov, at least). I think there's an opportunity for some refactoring here. I'm happy to do that now as part of this PR, or if you merge this first I'll do it in a different one.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
[WIP] Add publish to heroku support 274284246  
346124073 https://github.com/simonw/datasette/pull/104#issuecomment-346124073 https://api.github.com/repos/simonw/datasette/issues/104 MDEyOklzc3VlQ29tbWVudDM0NjEyNDA3Mw== jacobian 21148 2017-11-21T18:49:55Z 2017-11-21T18:49:55Z CONTRIBUTOR

Actually hang on, don't merge - there are some bugs that #141 masked when I tested this out elsewhere.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
[WIP] Add publish to heroku support 274284246  
346124764 https://github.com/simonw/datasette/pull/104#issuecomment-346124764 https://api.github.com/repos/simonw/datasette/issues/104 MDEyOklzc3VlQ29tbWVudDM0NjEyNDc2NA== jacobian 21148 2017-11-21T18:52:14Z 2017-11-21T18:52:14Z CONTRIBUTOR

OK, now this should work.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
[WIP] Add publish to heroku support 274284246  
346244871 https://github.com/simonw/datasette/issues/14#issuecomment-346244871 https://api.github.com/repos/simonw/datasette/issues/14 MDEyOklzc3VlQ29tbWVudDM0NjI0NDg3MQ== jacobian 21148 2017-11-22T05:06:30Z 2017-11-22T05:06:30Z CONTRIBUTOR

I'd also suggest taking a look at stevedore, which has a ton of tools for doing plugin stuff. I've had good luck with it in the past.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette Plugins 267707940  
360535979 https://github.com/simonw/datasette/issues/179#issuecomment-360535979 https://api.github.com/repos/simonw/datasette/issues/179 MDEyOklzc3VlQ29tbWVudDM2MDUzNTk3OQ== psychemedia 82988 2018-01-25T17:18:24Z 2018-01-25T17:18:24Z CONTRIBUTOR

To summarise that thread:

  • expose full metadata.json object to the index page template, eg to allow tables to be referred to by name;
  • ability to import multiple metadata.json files, eg to allow metadata files created for a specific SQLite db to be reused in a datasette referring to several database files;

It could also be useful to allow users to import a python file containing custom functions that can that be loaded into scope and made available to custom templates.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
More metadata options for template authors  288438570  
380608372 https://github.com/simonw/datasette/pull/200#issuecomment-380608372 https://api.github.com/repos/simonw/datasette/issues/200 MDEyOklzc3VlQ29tbWVudDM4MDYwODM3Mg== russss 45057 2018-04-11T21:55:46Z 2018-04-11T21:55:46Z CONTRIBUTOR

I think the most reliable way to detect spatialite is to run SELECT AddGeometryColumn(1, 2, 3, 4, 5); against a :memory: database and see if it throws an exception

Or just see if there's a geometry_columns table? I think that's quite unlikely to be added by accident (and it's an OGC standard). It also tells you if Spatialite is installed in the database rather than just loaded.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Hide Spatialite system tables 313494458  
380966565 https://github.com/simonw/datasette/issues/203#issuecomment-380966565 https://api.github.com/repos/simonw/datasette/issues/203 MDEyOklzc3VlQ29tbWVudDM4MDk2NjU2NQ== russss 45057 2018-04-12T22:43:08Z 2018-04-12T22:43:08Z CONTRIBUTOR

Looks like pint is pretty good at this.

In [1]: import pint

In [2]: ureg = pint.UnitRegistry()

In [3]: q = 3e6 * ureg('Hz')

In [4]: '{:~P}'.format(q.to_compact())
Out[4]: '3.0 MHz'

In [5]: q = 0.3 * ureg('m')

In [5]: '{:~P}'.format(q.to_compact())
Out[5]: '300.0 mm'

In [6]: q = 5 * ureg('')

In [7]: '{:~P}'.format(q.to_compact())
Out[7]: '5'
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support for units 313837303  
381237440 https://github.com/simonw/datasette/pull/202#issuecomment-381237440 https://api.github.com/repos/simonw/datasette/issues/202 MDEyOklzc3VlQ29tbWVudDM4MTIzNzQ0MA== russss 45057 2018-04-13T19:22:53Z 2018-04-13T19:22:53Z CONTRIBUTOR

I spotted you'd mentioned that in #184 but only after I'd written the patch!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Raise 404 on nonexistent table URLs 313785206  
381315675 https://github.com/simonw/datasette/issues/203#issuecomment-381315675 https://api.github.com/repos/simonw/datasette/issues/203 MDEyOklzc3VlQ29tbWVudDM4MTMxNTY3NQ== russss 45057 2018-04-14T09:14:45Z 2018-04-14T09:27:30Z CONTRIBUTOR

I'd like to figure out a sensible opt-in way to expose this in the JSON output as well. Maybe with a &_units=true parameter?

<s>From a machine-readable perspective I'm not sure why it would be useful to decorate the values with units</s>. Edit: Should have had some coffee first. It's clearly useful for stuff like map rendering!

I agree that the unit metadata should definitely be exposed in the JSON.

In #204 you said "I'd like to add support for using units when querying but this is PR is pretty usable as-is." - I'm fascinated to hear more about how this could work.

I'm thinking about a couple of approaches here. I think the simplest one is: if the column has a unit attached, optionally accept units in query fields:

column_units = ureg("Hz") #  Create a unit object for the column's unit
query_variable = ureg("4 GHz") # Supplied query variable

# Now we can convert the query units into column units before querying
supplied_value.to(column_units).magnitude
> 4000000000.0

# If the user doesn't supply units, pint just returns the plain
# number and we can query as usual assuming it's the base unit
query_variable = ureg("50")
query_variable
> 50

isinstance(query_variable, numbers.Number)
> True

This also lets us do some nice unit conversion on querying:

column_units = ureg("m")
query_variable = ureg("50 ft")

supplied_value.to(column_units)
> <Quantity(15.239999999999998, 'meter')>

The alternative would be to provide a dropdown of units next to the query field (so a "Hz" field would give you "kHz", "MHz", "GHz"). Although this would be clearer to the user, it isn't so easy - we'd need to know more about the context of the field to give you sensible SI prefixes (I'm not so interested in nanoHertz, for example).

You also lose the bonus of being able to convert - although pint will happily show you all the compatible units, it again suffers from a lack of context:

ureg("m").compatible_units()
> frozenset({<Unit('angstrom')>,
           <Unit('thou')>,
           <Unit('inch')>,
           <Unit('link')>,
           <Unit('foot')>,
           <Unit('survey_foot')>,
           <Unit('yard')>,
           <Unit('rod')>,
           <Unit('mile')>,
           <Unit('survey_mile')>,
           <Unit('league')>,
           <Unit('light_year')>})
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support for units 313837303  
381332222 https://github.com/simonw/datasette/pull/205#issuecomment-381332222 https://api.github.com/repos/simonw/datasette/issues/205 MDEyOklzc3VlQ29tbWVudDM4MTMzMjIyMg== russss 45057 2018-04-14T14:16:35Z 2018-04-14T14:16:35Z CONTRIBUTOR

I've added some tests and that docs link.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support filtering with units and more 314319372  
381361734 https://github.com/simonw/datasette/issues/125#issuecomment-381361734 https://api.github.com/repos/simonw/datasette/issues/125 MDEyOklzc3VlQ29tbWVudDM4MTM2MTczNA== russss 45057 2018-04-14T21:26:30Z 2018-04-14T21:26:30Z CONTRIBUTOR

FWIW I am now doing this on my WTR app (instead of silently limiting maps to 1000).

Telefonica now has about 4000 markers and good old BT has 22,000 or so.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plot rows on a map with Leaflet and Leaflet.markercluster 275135393  
381441392 https://github.com/simonw/datasette/pull/209#issuecomment-381441392 https://api.github.com/repos/simonw/datasette/issues/209 MDEyOklzc3VlQ29tbWVudDM4MTQ0MTM5Mg== russss 45057 2018-04-15T21:59:15Z 2018-04-15T21:59:15Z CONTRIBUTOR

I suspected this would cause some test failures, but I'll wait for opinions before attempting to fix them.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't duplicate simple primary keys in the link column 314455877  
381738137 https://github.com/simonw/datasette/pull/209#issuecomment-381738137 https://api.github.com/repos/simonw/datasette/issues/209 MDEyOklzc3VlQ29tbWVudDM4MTczODEzNw== russss 45057 2018-04-16T20:27:43Z 2018-04-16T20:27:43Z CONTRIBUTOR

Tests now fixed, honest. The failing test on Travis looks like an intermittent sqlite failure which should resolve itself on a retry...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't duplicate simple primary keys in the link column 314455877  
381763651 https://github.com/simonw/datasette/issues/203#issuecomment-381763651 https://api.github.com/repos/simonw/datasette/issues/203 MDEyOklzc3VlQ29tbWVudDM4MTc2MzY1MQ== russss 45057 2018-04-16T21:59:17Z 2018-04-16T21:59:17Z CONTRIBUTOR

Ah, I had no idea you could bind python functions into sqlite!

I think the primary purpose of this issue has been served now - I'm going to close this and create a new issue for the only bit of this that hasn't been touched yet, which is (optionally) exposing units in the JSON API.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support for units 313837303  
381905593 https://github.com/simonw/datasette/pull/209#issuecomment-381905593 https://api.github.com/repos/simonw/datasette/issues/209 MDEyOklzc3VlQ29tbWVudDM4MTkwNTU5Mw== russss 45057 2018-04-17T08:50:28Z 2018-04-17T08:50:28Z CONTRIBUTOR

I've added another commit which puts classes a class on each <td> by default with its column name, and I've also made the PK column bold.

Unfortunately the tests are still failing on 3.6, which is weird. I can't reproduce locally...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't duplicate simple primary keys in the link column 314455877  
390250253 https://github.com/simonw/datasette/issues/273#issuecomment-390250253 https://api.github.com/repos/simonw/datasette/issues/273 MDEyOklzc3VlQ29tbWVudDM5MDI1MDI1Mw== rgieseke 198537 2018-05-18T15:49:52Z 2018-05-18T15:49:52Z CONTRIBUTOR

Shouldn't versioneer do that?

E.g. 0.21+2.g1076c97

You'd need to install via pip install git+https://github.com/simow/datasette.git though, this does a temp git clone.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out a way to have /-/version return current git commit hash 324451322  
390795067 https://github.com/simonw/datasette/issues/276#issuecomment-390795067 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MDc5NTA2Nw== russss 45057 2018-05-21T21:55:57Z 2018-05-21T21:55:57Z CONTRIBUTOR

Well, we do have the capability to detect spatialite so my intention certainly wasn't to require it.

I can see the advantage of having it as a plugin but it does touch a number of points in the code. I think I'm going to attack this by refactoring the necessary bits and seeing where that leads (which was my plan anyway).

I think my main concern is - if I add certain plugin hooks for this, is anything else ever going to use them? I'm not sure I have an answer to that question yet, either way.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838  
391050113 https://github.com/simonw/datasette/issues/276#issuecomment-391050113 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MTA1MDExMw== russss 45057 2018-05-22T16:13:00Z 2018-05-22T16:13:00Z CONTRIBUTOR

Yup, I'll have a think about it. My current thoughts are for spatialite we'll need to hook into the following places:

  • Inspection, so we can detect which columns are geometry columns. (We also currently ignore spatialite tables during inspection, it may be worth moving that to the plugin as well.)
  • After data load, so we can convert WKB into the correct intermediate format for display. The alternative here is to alter the select SQL itself and get spatialite to do this conversion, but that strikes me as a bit more complex and possibly not as useful.
  • HTML rendering.
  • Querying?

The rendering and querying hooks could also potentially be used to move the units support into a plugin.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838  
391059008 https://github.com/simonw/datasette/pull/280#issuecomment-391059008 https://api.github.com/repos/simonw/datasette/issues/280 MDEyOklzc3VlQ29tbWVudDM5MTA1OTAwOA== r4vi 565628 2018-05-22T16:40:27Z 2018-05-22T16:40:27Z CONTRIBUTOR
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.23.1'
>>> 

running the above in the container seems to show 3.23.1 too so maybe we don't need pysqlite3 at all?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Build Dockerfile with recent Sqlite + Spatialite 325373747  
391073009 https://github.com/simonw/datasette/pull/279#issuecomment-391073009 https://api.github.com/repos/simonw/datasette/issues/279 MDEyOklzc3VlQ29tbWVudDM5MTA3MzAwOQ== rgieseke 198537 2018-05-22T17:23:26Z 2018-05-22T17:23:26Z CONTRIBUTOR

I think I prefer the aesthetics of just "0.22" for the version string if it's a tagged release with no additional changes - does that work?

Yes! That's the default versioneer behaviour.

I'd like to continue to provide a tuple that can be imported from the version.py module as well, as seen here:

Should work now, it can be a two (for a tagged version), three or four items tuple.

In [2]: datasette.__version__
Out[2]: '0.12+292.ga70c2a8.dirty'

In [3]: datasette.__version_info__
Out[3]: ('0', '12+292', 'ga70c2a8', 'dirty')
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add version number support with Versioneer 325352370  
391073267 https://github.com/simonw/datasette/pull/279#issuecomment-391073267 https://api.github.com/repos/simonw/datasette/issues/279 MDEyOklzc3VlQ29tbWVudDM5MTA3MzI2Nw== rgieseke 198537 2018-05-22T17:24:16Z 2018-05-22T17:24:16Z CONTRIBUTOR

Sorry, just realised you rely on version being a module ...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add version number support with Versioneer 325352370  
391077700 https://github.com/simonw/datasette/pull/279#issuecomment-391077700 https://api.github.com/repos/simonw/datasette/issues/279 MDEyOklzc3VlQ29tbWVudDM5MTA3NzcwMA== rgieseke 198537 2018-05-22T17:38:17Z 2018-05-22T17:38:17Z CONTRIBUTOR

Alright, that should work now -- let me know if you would prefer any different behaviour.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add version number support with Versioneer 325352370  
391141391 https://github.com/simonw/datasette/pull/280#issuecomment-391141391 https://api.github.com/repos/simonw/datasette/issues/280 MDEyOklzc3VlQ29tbWVudDM5MTE0MTM5MQ== r4vi 565628 2018-05-22T21:08:39Z 2018-05-22T21:08:39Z CONTRIBUTOR

I'm going to clean this up for consistency tomorrow morning so hold off
merging until then please

On Tue, May 22, 2018 at 6:34 PM, Simon Willison notifications@github.com
wrote:

Yeah let's try this without pysqlite3 and see if we still get the correct
version.


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/simonw/datasette/pull/280#issuecomment-391076458, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAihfMI-H6CBt-Py0xdBbH2xDK0KsjT2ks5t1EwYgaJpZM4UI_2m
.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Build Dockerfile with recent Sqlite + Spatialite 325373747  
391290271 https://github.com/simonw/datasette/pull/280#issuecomment-391290271 https://api.github.com/repos/simonw/datasette/issues/280 MDEyOklzc3VlQ29tbWVudDM5MTI5MDI3MQ== r4vi 565628 2018-05-23T09:53:38Z 2018-05-23T09:53:38Z CONTRIBUTOR

Running:

docker run -p 8001:8001 -v `pwd`:/mnt datasette \
    datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db \
        --load-extension=/usr/local/lib/mod_spatialite.so

is now returning FTS5 enabled in the versions output:

{
    "datasette": {
        "version": "0.22"
    },
    "python": {
        "full": "3.6.5 (default, May  5 2018, 03:07:21) \n[GCC 6.3.0 20170516]",
        "version": "3.6.5"
    },
    "sqlite": {
        "extensions": {
            "json1": null,
            "spatialite": "4.4.0-RC0"
        },
        "fts_versions": [
            "FTS5",
            "FTS4",
            "FTS3"
        ],
        "version": "3.23.1"
    }
}

The old query didn't work because specifying (t TEXT) caused an error

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Build Dockerfile with recent Sqlite + Spatialite 325373747  
391355030 https://github.com/simonw/datasette/pull/280#issuecomment-391355030 https://api.github.com/repos/simonw/datasette/issues/280 MDEyOklzc3VlQ29tbWVudDM5MTM1NTAzMA== r4vi 565628 2018-05-23T13:53:27Z 2018-05-23T15:22:45Z CONTRIBUTOR

No objections;
It's good to go @simonw

On Wed, 23 May 2018, 14:51 Simon Willison, notifications@github.com wrote:

@r4vi https://github.com/r4vi any objections to me merging this?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/simonw/datasette/pull/280#issuecomment-391354237, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAihfM_2DN5WR2mkO-VK6ozDmkUQ4IMjks5t1WlcgaJpZM4UI_2m
.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Build Dockerfile with recent Sqlite + Spatialite 325373747  
391505930 https://github.com/simonw/datasette/issues/276#issuecomment-391505930 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MTUwNTkzMA== russss 45057 2018-05-23T21:41:37Z 2018-05-23T21:41:37Z CONTRIBUTOR

I'm not keen on anything that modifies the SQLite file itself on startup

Ah I didn't mean that - I meant altering the SELECT query to fetch the data so that it ran a spatialite function to transform that specific column.

I think that's less useful as a general-purpose plugin hook though, and it's not that hard to parse the WKB in Python (my default approach would be to use shapely, which is great, but geomet looks like an interesting pure-python alternative).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838  
392825746 https://github.com/simonw/datasette/issues/276#issuecomment-392825746 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MjgyNTc0Ng== russss 45057 2018-05-29T15:42:53Z 2018-05-29T15:42:53Z CONTRIBUTOR

I haven't had time to look further into this, but if doing this as a plugin results in useful hooks then I think we should do it that way. We could always require the plugin as a standard dependency.

I think this is going to result in quite a bit of refactoring anyway so it's a good time to add hooks regardless.

On the other hand, if we have to add lots of specialist hooks for it then maybe it's worth integrating into the core.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838  
393106520 https://github.com/simonw/datasette/issues/276#issuecomment-393106520 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MzEwNjUyMA== russss 45057 2018-05-30T10:09:25Z 2018-05-30T10:09:25Z CONTRIBUTOR

I don't think it's unreasonable to only support spatialite geometries in a coordinate reference system which is at least transformable to WGS84. It would be nice to support different CRSes in the database so conversion to spatialite from the source data is lossless.

I think the working CRS for datasette should be WGS84 though (leaflet requires it, for example) - it's just a case of calling ST_Transform(geom, 4326) on the column while we're loading the data.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838  
401310732 https://github.com/simonw/datasette/issues/276#issuecomment-401310732 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDQwMTMxMDczMg== psychemedia 82988 2018-06-29T10:05:04Z 2018-06-29T10:07:25Z CONTRIBUTOR

@russs Different map projections can presumably be handled on the client side using a leaflet plugin to transform the geometry (eg kartena/Proj4Leaflet) although the leaflet side would need to detect or be informed of the original projection?

Another possibility would be to provide an easy way/guidance for users to create an FK'd table containing the WGS84 projection of a non-WGS84 geometry in the original/principle table? This could then as a proxy for serving GeoJSON to the leaflet map?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838  
401312981 https://github.com/simonw/datasette/issues/276#issuecomment-401312981 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDQwMTMxMjk4MQ== russss 45057 2018-06-29T10:14:54Z 2018-06-29T10:14:54Z CONTRIBUTOR

@RusSs Different map projections can presumably be handled on the client side using a leaflet plugin to transform the geometry (eg kartena/Proj4Leaflet) although the leaflet side would need to detect or be informed of the original projection?

Well, as @simonw mentioned, GeoJSON only supports WGS84, and GeoJSON (and/or TopoJSON) is the standard we probably want to aim for. On-the-fly reprojection in spatialite is not an issue anyway, and in general I think you want to be serving stuff to web maps in WGS84 or Web Mercator.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838  
405022335 https://github.com/simonw/datasette/issues/344#issuecomment-405022335 https://api.github.com/repos/simonw/datasette/issues/344 MDEyOklzc3VlQ29tbWVudDQwNTAyMjMzNQ== russss 45057 2018-07-14T13:00:48Z 2018-07-14T13:00:48Z CONTRIBUTOR

Looks like this was a red herring actually, and heroku had a blip when I was testing it...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish heroku fails without name provided 341229113  
405026441 https://github.com/simonw/datasette/issues/343#issuecomment-405026441 https://api.github.com/repos/simonw/datasette/issues/343 MDEyOklzc3VlQ29tbWVudDQwNTAyNjQ0MQ== russss 45057 2018-07-14T14:17:14Z 2018-07-14T14:17:14Z CONTRIBUTOR

This probably depends on #294.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Render boolean fields better by default 341228846  
405026800 https://github.com/simonw/datasette/issues/294#issuecomment-405026800 https://api.github.com/repos/simonw/datasette/issues/294 MDEyOklzc3VlQ29tbWVudDQwNTAyNjgwMA== russss 45057 2018-07-14T14:24:31Z 2018-07-14T14:24:31Z CONTRIBUTOR

I had a quick look at this in relation to #343 and I feel like it might be worth modelling the inspected table metadata internally as an object rather than a dict. (We'd still have to serialise it back to JSON.)

There are a few places where we rely on the structure of this metadata dict for various reasons, including in templates (and potentially also in user templates). It would be nice to have a reasonably well defined API for accessing metadata internally so that it's clearer what we're breaking.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
inspect should record column types 327365110  
422821483 https://github.com/simonw/datasette/issues/329#issuecomment-422821483 https://api.github.com/repos/simonw/datasette/issues/329 MDEyOklzc3VlQ29tbWVudDQyMjgyMTQ4Mw== jaywgraves 418191 2018-09-19T14:17:42Z 2018-09-19T14:17:42Z CONTRIBUTOR

I'm using the docker image (0.23.2) and notice some differences/bugs between the docs and the published version with canned queries. (submitted a tiny doc fix also)

I was able to build the docker container locally using master and I'm using that for now.
Would it be possible to manually push 0.24 to DockerHub until the TravisCI stuff is fixed?

I would like to run this in our Kubernetes cluster but don't want to publish a version in our internal registry if I don't have to.
Thanks!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Travis should push tagged images to Docker Hub for each release 336465018  
422915450 https://github.com/simonw/datasette/issues/329#issuecomment-422915450 https://api.github.com/repos/simonw/datasette/issues/329 MDEyOklzc3VlQ29tbWVudDQyMjkxNTQ1MA== jaywgraves 418191 2018-09-19T18:45:02Z 2018-09-20T10:50:50Z CONTRIBUTOR

That works for me. Was able to pull the public image and no errors on my canned query. (~although a small rendering bug. I'll create an issue and if I have time today, a PR to fix~ this turned out to be my error.)
Thanks for the quick response!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Travis should push tagged images to Docker Hub for each release 336465018  
435862009 https://github.com/simonw/datasette/issues/371#issuecomment-435862009 https://api.github.com/repos/simonw/datasette/issues/371 MDEyOklzc3VlQ29tbWVudDQzNTg2MjAwOQ== psychemedia 82988 2018-11-05T12:48:35Z 2018-11-05T12:48:35Z CONTRIBUTOR

I think you need to register a domain name you own separately in order to get a non-IP address address? https://www.digitalocean.com/docs/networking/dns/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish digitalocean plugin 377156339  
436037692 https://github.com/simonw/datasette/issues/370#issuecomment-436037692 https://api.github.com/repos/simonw/datasette/issues/370 MDEyOklzc3VlQ29tbWVudDQzNjAzNzY5Mg== psychemedia 82988 2018-11-05T21:15:47Z 2018-11-05T21:18:37Z CONTRIBUTOR

In terms of integration with pandas, I was pondering two different ways datasette/csvs_to_sqlite integration may work:

  • like pandasql, to provide a SQL query layer either by a direct connection to the sqlite db or via datasette API;
  • as an improvement of pandas.to_sql(), which is a bit ropey (e.g. pandas.to_sql_from_csvs(), routing the dataframe to sqlite via csvs_tosqlite rather than the dodgy mapping that pandas supports).

The pandas.publish_* idea could be quite interesting though... Would it be useful/fruitful to think about publish_ as a complement to pandas.to_?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Integration with JupyterLab 377155320  
436042445 https://github.com/simonw/datasette/issues/370#issuecomment-436042445 https://api.github.com/repos/simonw/datasette/issues/370 MDEyOklzc3VlQ29tbWVudDQzNjA0MjQ0NQ== psychemedia 82988 2018-11-05T21:30:42Z 2018-11-05T21:31:48Z CONTRIBUTOR

Another route would be something like creating a datasette IPython magic for notebooks to take a dataframe and easily render it as a datasette. You'd need to run the app in the background rather than block execution in the notebook. Related to that, or to publishing a dataframe in notebook cell for use in other cells in a non-blocking way, there may be cribs in something like https://github.com/micahscopes/nbmultitask .

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Integration with JupyterLab 377155320  
459915995 https://github.com/simonw/datasette/issues/160#issuecomment-459915995 https://api.github.com/repos/simonw/datasette/issues/160 MDEyOklzc3VlQ29tbWVudDQ1OTkxNTk5NQ== psychemedia 82988 2019-02-02T00:43:16Z 2019-02-02T00:58:20Z CONTRIBUTOR

Do you have any simple working examples of how to use --static? Inspection of default served files suggests locations such as http://example.com/-/static/app.css?0e06ee.

If datasette is being proxied to http://example.com/foo/datasette, what form should arguments to --static take so that static files are correctly referenced?

Use case is here: https://github.com/psychemedia/jupyterserverproxy-datasette-demo Trying to do a really simple datasette demo in MyBinder using jupyter-server-proxy.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to bundle and serve additional static files 278208011  
474280581 https://github.com/simonw/datasette/issues/417#issuecomment-474280581 https://api.github.com/repos/simonw/datasette/issues/417 MDEyOklzc3VlQ29tbWVudDQ3NDI4MDU4MQ== psychemedia 82988 2019-03-19T10:06:42Z 2019-03-19T10:06:42Z CONTRIBUTOR

This would be really interesting but several possibilities in use arise, I think?

For example:

  • I put a new CSV file into the import dir and a new table is created therefrom
  • I put a CSV file into the import dir that replaces a previous file / table of the same name as a pre-existing table (eg files that contain monthly data in year to date). The data may also patch previous months, so a full replace / DROP on the original table may well be in order.
  • I put a CSV file into the import dir that updates a table of the same name as a pre-existing table (eg files that contain last month's data)

CSV files may also have messy names compared to the table you want. Or for an update CSV, may have the form MYTABLENAME-February2019.csv etc

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette Library 421546944  
474282321 https://github.com/simonw/datasette/issues/412#issuecomment-474282321 https://api.github.com/repos/simonw/datasette/issues/412 MDEyOklzc3VlQ29tbWVudDQ3NDI4MjMyMQ== psychemedia 82988 2019-03-19T10:09:46Z 2019-03-19T10:09:46Z CONTRIBUTOR

Does this also relate to https://github.com/simonw/datasette/issues/283 and the ability to ATTACH DATABASE?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Linked Data(sette) 411257981  
483017176 https://github.com/simonw/datasette/issues/431#issuecomment-483017176 https://api.github.com/repos/simonw/datasette/issues/431 MDEyOklzc3VlQ29tbWVudDQ4MzAxNzE3Ng== psychemedia 82988 2019-04-14T16:58:37Z 2019-04-14T16:58:37Z CONTRIBUTOR

Hmm... nope... I see an updated timestamp from ls -al on the db but no reload?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette doesn't reload when database file changes 432870248  
483202658 https://github.com/simonw/datasette/issues/429#issuecomment-483202658 https://api.github.com/repos/simonw/datasette/issues/429 MDEyOklzc3VlQ29tbWVudDQ4MzIwMjY1OA== psychemedia 82988 2019-04-15T10:48:01Z 2019-04-15T10:48:01Z CONTRIBUTOR

Minor UI observation:

_where= renders a [remove] link whereas _facet= gets a cross to remove it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_where=sql-fragment parameter for table views 432636432  
487537452 https://github.com/simonw/datasette/pull/437#issuecomment-487537452 https://api.github.com/repos/simonw/datasette/issues/437 MDEyOklzc3VlQ29tbWVudDQ4NzUzNzQ1Mg== russss 45057 2019-04-29T10:58:49Z 2019-04-29T10:58:49Z CONTRIBUTOR

I've just spotted that this implements #215.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add inspect and prepare_sanic hooks 438048318  
487542486 https://github.com/simonw/datasette/pull/439#issuecomment-487542486 https://api.github.com/repos/simonw/datasette/issues/439 MDEyOklzc3VlQ29tbWVudDQ4NzU0MjQ4Ng== russss 45057 2019-04-29T11:20:30Z 2019-04-29T11:20:30Z CONTRIBUTOR

Actually I think this is not the whole story because of the rowid issue. I'm going to think about this one a bit more.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
[WIP] Add primary key to the extra_body_script hook arguments 438240541  
487686655 https://github.com/simonw/datasette/pull/441#issuecomment-487686655 https://api.github.com/repos/simonw/datasette/issues/441 MDEyOklzc3VlQ29tbWVudDQ4NzY4NjY1NQ== russss 45057 2019-04-29T18:14:25Z 2019-04-29T18:14:25Z CONTRIBUTOR

Subsidiary note which I forgot in the commit message:

I've decided to give each view a short string name to aid in differentiating which view a hook is being called from. Since hooks are functions and not subclasses, and can get called from different places in the URL hierarchy, it's sometimes difficult to distinguish what data you're actually operating on. I think this will come in handy for other hooks as well.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add register_output_renderer hook 438437973  
487689477 https://github.com/simonw/datasette/pull/424#issuecomment-487689477 https://api.github.com/repos/simonw/datasette/issues/424 MDEyOklzc3VlQ29tbWVudDQ4NzY4OTQ3Nw== russss 45057 2019-04-29T18:22:40Z 2019-04-29T18:22:40Z CONTRIBUTOR

This is pretty conflicty because I forgot how to use git fetch. If you're interested in merging this I'll rewrite it against an actual modern checkout...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Column types in inspected metadata 427429265  
487692377 https://github.com/simonw/datasette/pull/424#issuecomment-487692377 https://api.github.com/repos/simonw/datasette/issues/424 MDEyOklzc3VlQ29tbWVudDQ4NzY5MjM3Nw== russss 45057 2019-04-29T18:30:46Z 2019-04-29T18:30:46Z CONTRIBUTOR

Actually no, I ended up not using the inspected column types in my plugin, and the binary column issue can be solved a lot more simply, so I'll close this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Column types in inspected metadata 427429265  
487723476 https://github.com/simonw/datasette/pull/441#issuecomment-487723476 https://api.github.com/repos/simonw/datasette/issues/441 MDEyOklzc3VlQ29tbWVudDQ4NzcyMzQ3Ng== russss 45057 2019-04-29T20:05:23Z 2019-04-29T20:05:23Z CONTRIBUTOR

This is the minimal example (I also included it in the docs):

from datasette import hookimpl

def render_test(args, data, view_name):
    return {
       'body': 'Hello World',
       'content_type': 'text/plain'
    }

@hookimpl
def register_output_renderer():
    return {
        'extension': 'test',
        'callback': render_test
    }

I'm working on the GeoJSON one now and it should be ready soon. (I forgot I was going to run into the same problem as before - that Spatialite's stupid binary format isn't WKB and I have no way of altering the query to change that - but I've just managed to write some code to rearrange the bytes from Spatialite blob-geometry into WKB...)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add register_output_renderer hook 438437973  
487724539 https://github.com/simonw/datasette/pull/441#issuecomment-487724539 https://api.github.com/repos/simonw/datasette/issues/441 MDEyOklzc3VlQ29tbWVudDQ4NzcyNDUzOQ== russss 45057 2019-04-29T20:08:32Z 2019-04-29T20:08:32Z CONTRIBUTOR

I also just realised that I should be passing the datasette object into the hook function...as I just found I need it. So hold off merging until I've fixed that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add register_output_renderer hook 438437973  
487735247 https://github.com/simonw/datasette/pull/441#issuecomment-487735247 https://api.github.com/repos/simonw/datasette/issues/441 MDEyOklzc3VlQ29tbWVudDQ4NzczNTI0Nw== russss 45057 2019-04-29T20:39:43Z 2019-04-29T20:39:43Z CONTRIBUTOR

I updated the hook to pass the datasette object through now.

You can see the working GeoJSON render function here - the hook function is here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add register_output_renderer hook 438437973  
487748271 https://github.com/simonw/datasette/pull/441#issuecomment-487748271 https://api.github.com/repos/simonw/datasette/issues/441 MDEyOklzc3VlQ29tbWVudDQ4Nzc0ODI3MQ== russss 45057 2019-04-29T21:20:17Z 2019-04-29T21:20:17Z CONTRIBUTOR

Also I just pushed a change to add registered output renderers to the templates:

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add register_output_renderer hook 438437973  
487859345 https://github.com/simonw/datasette/pull/439#issuecomment-487859345 https://api.github.com/repos/simonw/datasette/issues/439 MDEyOklzc3VlQ29tbWVudDQ4Nzg1OTM0NQ== russss 45057 2019-04-30T08:21:19Z 2019-04-30T08:21:19Z CONTRIBUTOR

I think the best approach to this is to pass through the view_name parameter I added in #441. It's then simple enough for me to add .geojson to the URL in JS - I don't need the pkey.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
[WIP] Add primary key to the extra_body_script hook arguments 438240541  
488247617 https://github.com/simonw/datasette/pull/441#issuecomment-488247617 https://api.github.com/repos/simonw/datasette/issues/441 MDEyOklzc3VlQ29tbWVudDQ4ODI0NzYxNw== russss 45057 2019-05-01T09:57:50Z 2019-05-01T09:57:50Z CONTRIBUTOR

Just for the record, this PR is now finished and ready to merge from my perspective.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add register_output_renderer hook 438437973  
488595724 https://github.com/simonw/datasette/pull/432#issuecomment-488595724 https://api.github.com/repos/simonw/datasette/issues/432 MDEyOklzc3VlQ29tbWVudDQ4ODU5NTcyNA== russss 45057 2019-05-02T08:50:53Z 2019-05-02T08:50:53Z CONTRIBUTOR

Can I pull those needs out of the Facet class somehow?

I was thinking that it might be handy for datasette to have a request object which wraps the Sanic Request. This could include the datasette-specific querystring decoding and the special_args parsing from TableView.data.

This would mean that we could expose the request object to plugin hooks without coupling them to Sanic.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor facets to a class and new plugin, refs #427 432893491  
489060765 https://github.com/simonw/datasette/issues/419#issuecomment-489060765 https://api.github.com/repos/simonw/datasette/issues/419 MDEyOklzc3VlQ29tbWVudDQ4OTA2MDc2NQ== russss 45057 2019-05-03T11:07:42Z 2019-05-03T11:07:42Z CONTRIBUTOR

Are you planning on removing inspect entirely?

I didn't spot this work before I started on datasette-geo, but ironically I think it has a use case which really needs the inspect functionality (or some replacement).

Datasette-geo uses it to store the bounding box of all the geographic features in the table. This is needed when rendering the map because it avoids having to send loads of tile requests for areas which are empty.

Even with relatively small datasets, calculating the bounding box seems to take around 5 seconds, so I don't think it's really feasible to do this on page load.

One possible fix would be to do this on startup, and then in a thread which watches the database for changes.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Default to opening files in mutable mode, special option for immutable files 421551434  
489163939 https://github.com/simonw/datasette/pull/434#issuecomment-489163939 https://api.github.com/repos/simonw/datasette/issues/434 MDEyOklzc3VlQ29tbWVudDQ4OTE2MzkzOQ== rprimet 10352819 2019-05-03T16:49:45Z 2019-05-03T16:50:03Z CONTRIBUTOR

The second time I ran the command I got an error:

ERROR: (gcloud.beta.run.deploy) Deployment endpoint was not found. Perhaps the
provided region was invalid. Set the run/region property to a valid region and
retry. Ex: gcloud config set run/region us-central1

Yes, I was able to reproduce this; I used to get prompted for a run region interactively by the gcloud tool before, but maybe this is changing? (the documentation now assumes run/region is set).

Not sure which course of action is best: making datasette ensure that run/region is set beforehand or wait a bit until the gcloud CLI stabilizes?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette publish cloudrun" command to publish to Google Cloud Run 434321685  
489221481 https://github.com/simonw/datasette/issues/446#issuecomment-489221481 https://api.github.com/repos/simonw/datasette/issues/446 MDEyOklzc3VlQ29tbWVudDQ4OTIyMTQ4MQ== russss 45057 2019-05-03T19:58:31Z 2019-05-03T19:58:31Z CONTRIBUTOR

In this particular case I don't think there's an issue making all those required. However, I suspect we might have to allow optional values at some point - my preferred solution to russss/datasette-geo#2 would need one.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Define mechanism for plugins to return structured data 440134714  
489222223 https://github.com/simonw/datasette/issues/446#issuecomment-489222223 https://api.github.com/repos/simonw/datasette/issues/446 MDEyOklzc3VlQ29tbWVudDQ4OTIyMjIyMw== russss 45057 2019-05-03T20:01:19Z 2019-05-03T20:01:29Z CONTRIBUTOR

Also I have a slight preference against (ab)using __slots__ to enforce fields, although I have done it myself in the past. It would be possible to do this with __setattr__ instead, although that's an implementation detail and I'm not too fussed about it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Define mechanism for plugins to return structured data 440134714  
489342728 https://github.com/simonw/datasette/pull/450#issuecomment-489342728 https://api.github.com/repos/simonw/datasette/issues/450 MDEyOklzc3VlQ29tbWVudDQ4OTM0MjcyOA== russss 45057 2019-05-04T16:37:35Z 2019-05-04T16:37:35Z CONTRIBUTOR

For a bit more context: this fixes a crash with unsupported operand type(s) for +: 'int' and 'NoneType' on the index page for me.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Coalesce hidden table count to 0 440304714  
499320973 https://github.com/simonw/datasette/issues/394#issuecomment-499320973 https://api.github.com/repos/simonw/datasette/issues/394 MDEyOklzc3VlQ29tbWVudDQ5OTMyMDk3Mw== kevindkeogh 13896256 2019-06-06T02:07:59Z 2019-06-06T02:07:59Z CONTRIBUTOR

Hey was this ever merged? Trying to run this behind nginx, and encountering this issue.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
base_url configuration setting 396212021  
499923145 https://github.com/simonw/datasette/issues/394#issuecomment-499923145 https://api.github.com/repos/simonw/datasette/issues/394 MDEyOklzc3VlQ29tbWVudDQ5OTkyMzE0NQ== kevindkeogh 13896256 2019-06-07T15:10:57Z 2019-06-07T15:11:07Z CONTRIBUTOR

Putting this here in case anyone else encounters the same issue with nginx, I was able to resolve it by passing the header in the nginx proxy config (i.e., proxy_set_header Host $host).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
base_url configuration setting 396212021  
504662904 https://github.com/simonw/datasette/issues/514#issuecomment-504662904 https://api.github.com/repos/simonw/datasette/issues/514 MDEyOklzc3VlQ29tbWVudDUwNDY2MjkwNA== russss 45057 2019-06-22T12:45:21Z 2019-06-22T12:45:39Z CONTRIBUTOR

On most modern Linux distros, systemd is the easiest answer.

Example systemd unit file (save to /etc/systemd/system/datasette.service):

[Unit]
Description=Datasette
After=network.target

[Service]
Type=simple
User=<username>
WorkingDirectory=/path/to/data
ExecStart=/path/to/datasette serve -h 0.0.0.0 ./my.db
Restart=on-failure

[Install]
WantedBy=multi-user.target

Activate it with:

$ sudo systemctl daemon-reload
$ sudo systemctl enable datasette
$ sudo systemctl start datasette

Logs are best viewed using journalctl -u datasette -f.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation with recommendations on running Datasette in production without using Docker 459397625  
504663766 https://github.com/simonw/datasette/issues/514#issuecomment-504663766 https://api.github.com/repos/simonw/datasette/issues/514 MDEyOklzc3VlQ29tbWVudDUwNDY2Mzc2Ng== russss 45057 2019-06-22T12:57:59Z 2019-06-22T12:57:59Z CONTRIBUTOR

This example is useful to - I like how it has a Makefile that knows how to set up systemd: https://github.com/pikesley/Queube

I wasn't even aware it was possible to add a systemd service at an arbitrary path, but it seems a little messy to me.

Maybe worth noting that systemd does support per-user services which don't require root access. Cool but probably overkill for most people (especially when you're going to need root to listen on port 80 anyway, directly or via a reverse proxy).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation with recommendations on running Datasette in production without using Docker 459397625  
504684831 https://github.com/simonw/datasette/issues/514#issuecomment-504684831 https://api.github.com/repos/simonw/datasette/issues/514 MDEyOklzc3VlQ29tbWVudDUwNDY4NDgzMQ== russss 45057 2019-06-22T17:38:23Z 2019-06-22T17:38:23Z CONTRIBUTOR

WorkingDirectory=/path/to/data

@russss, Which directory does this represent?

It's the working directory (cwd) of the spawned process. In this case if you set it to the directory your data is in, you can use relative paths to the db (and metadata/templates/etc) in the ExecStart command.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation with recommendations on running Datasette in production without using Docker 459397625  
504690927 https://github.com/simonw/datasette/issues/514#issuecomment-504690927 https://api.github.com/repos/simonw/datasette/issues/514 MDEyOklzc3VlQ29tbWVudDUwNDY5MDkyNw== russss 45057 2019-06-22T19:06:07Z 2019-06-22T19:06:07Z CONTRIBUTOR

I'd rather not turn this into a systemd support thread, but you're trying to execute the package directory there. Your datasette executable is probably at /home/chris/Env/datasette/bin/datasette.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation with recommendations on running Datasette in production without using Docker 459397625  
504809397 https://github.com/simonw/datasette/issues/523#issuecomment-504809397 https://api.github.com/repos/simonw/datasette/issues/523 MDEyOklzc3VlQ29tbWVudDUwNDgwOTM5Nw== rixx 2657547 2019-06-24T01:38:14Z 2019-06-24T01:38:14Z CONTRIBUTOR

Ah, apologies – I had found and read those issues, but I was under the impression that they refered only to the filtered row count, not the unfiltered total row count.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Show total/unfiltered row count when filtering 459627549  
509013413 https://github.com/simonw/datasette/issues/507#issuecomment-509013413 https://api.github.com/repos/simonw/datasette/issues/507 MDEyOklzc3VlQ29tbWVudDUwOTAxMzQxMw== psychemedia 82988 2019-07-07T16:31:57Z 2019-07-07T16:31:57Z CONTRIBUTOR

Chrome and Firefox both support headless screengrabs from command line, but I don't know how parameterised they can be?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Every datasette plugin on the ecosystem page should have a screenshot 455852801  
509618339 https://github.com/simonw/datasette/pull/554#issuecomment-509618339 https://api.github.com/repos/simonw/datasette/issues/554 MDEyOklzc3VlQ29tbWVudDUwOTYxODMzOQ== abdusco 3243482 2019-07-09T12:16:32Z 2019-07-09T12:16:32Z CONTRIBUTOR

I've also added another fix for using static mounts with absolute paths on Windows.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix static mounts using relative paths and prevent traversal exploits 465728430  
509629331 https://github.com/simonw/datasette/pull/554#issuecomment-509629331 https://api.github.com/repos/simonw/datasette/issues/554 MDEyOklzc3VlQ29tbWVudDUwOTYyOTMzMQ== abdusco 3243482 2019-07-09T12:51:35Z 2019-07-09T12:51:35Z CONTRIBUTOR

I wanted to add a test for it too, but I've realized it's impossible to test a server process as we cannot get its exit code.

# tests/test_cli.py
def test_static_mounts_on_windows():
    if sys.platform != "win32":
        return
    runner = CliRunner()
    result = runner.invoke(
        cli, ["serve", "--static", r"s:C:\\"]
    )
    assert result.exit_code == 0
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix static mounts using relative paths and prevent traversal exploits 465728430  
510730200 https://github.com/simonw/datasette/issues/511#issuecomment-510730200 https://api.github.com/repos/simonw/datasette/issues/511 MDEyOklzc3VlQ29tbWVudDUxMDczMDIwMA== abdusco 3243482 2019-07-12T03:23:22Z 2019-07-12T03:23:22Z CONTRIBUTOR

@simonw yes it works fine on Windows, but test suite doesn't run properly, for that I had to use WSL

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Get Datasette working on Windows, including CI 456578474  
527209840 https://github.com/simonw/sqlite-utils/pull/56#issuecomment-527209840 https://api.github.com/repos/simonw/sqlite-utils/issues/56 MDEyOklzc3VlQ29tbWVudDUyNzIwOTg0MA== amjith 49260 2019-09-02T17:23:21Z 2019-09-02T17:23:21Z CONTRIBUTOR

I have updated the other PR with the changes from this one and added tests. I have also changed the escaping from double quotes to brackets.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Escape the table name in populate_fts and search. 487847945  
527211047 https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527211047 https://api.github.com/repos/simonw/sqlite-utils/issues/57 MDEyOklzc3VlQ29tbWVudDUyNzIxMTA0Nw== amjith 49260 2019-09-02T17:30:43Z 2019-09-02T17:30:43Z CONTRIBUTOR

I have merged the other PR (#56) into this one.

I have incorporated your suggestions. Cheers!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add triggers while enabling FTS 487987958  
533818697 https://github.com/simonw/sqlite-utils/issues/61#issuecomment-533818697 https://api.github.com/repos/simonw/sqlite-utils/issues/61 MDEyOklzc3VlQ29tbWVudDUzMzgxODY5Nw== amjith 49260 2019-09-21T18:09:01Z 2019-09-21T18:09:28Z CONTRIBUTOR

@witeshadow The library version doesn't have helpers around CSV (at least not from what I can see in the code).

But here's a snippet that makes it easy to insert from CSV using the library.

import csv
from sqlite_utils import Database

# CSV Reader

csv_file = open("filename.csv")   # open the csv file.
reader = csv.reader(csv_file)  # Create a CSV reader
headers = next(reader)   # First line is the header
docs = (dict(zip(headers, row)) for row in reader)

# Now you can use the `sqlite_utils` library. 

db = Database("my_database.db")
db["table_name"].insert_all(docs)

This snippet is adapted from reading the CLI source code on how it implements the csv option.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
importing CSV to SQLite as library 491219910  
541052329 https://github.com/simonw/datasette/issues/585#issuecomment-541052329 https://api.github.com/repos/simonw/datasette/issues/585 MDEyOklzc3VlQ29tbWVudDU0MTA1MjMyOQ== rixx 2657547 2019-10-11T12:53:51Z 2019-10-11T12:53:51Z CONTRIBUTOR

I think this would be good, yeah – currently, databases are explicitly sorted by name in the IndexView, we could just remove that part (and use an OrderedDict for consistency, I suppose)?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Databases on index page should display in order they were passed to "datasette serve"? 503217375  
541118904 https://github.com/simonw/datasette/issues/507#issuecomment-541118904 https://api.github.com/repos/simonw/datasette/issues/507 MDEyOklzc3VlQ29tbWVudDU0MTExODkwNA== rixx 2657547 2019-10-11T15:48:49Z 2019-10-11T15:48:49Z CONTRIBUTOR

Headless Chrome and Firefox via Selenium are a solid choice in my experience. You may be interested in how pretix and pretalx solve this problem: They use pytest to create those screenshots on release to make sure they are up to date. See this writeup and this repo.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Every datasette plugin on the ecosystem page should have a screenshot 455852801  
541119038 https://github.com/simonw/datasette/issues/512#issuecomment-541119038 https://api.github.com/repos/simonw/datasette/issues/512 MDEyOklzc3VlQ29tbWVudDU0MTExOTAzOA== rixx 2657547 2019-10-11T15:49:13Z 2019-10-11T15:49:13Z CONTRIBUTOR

How open are you to changing the config variable names (with appropriate deprecation, of course)? "about_url_text", "license_url_text" etc might be better suited to convey that these are just meant as basically URL titles.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"about" parameter in metadata does not appear when alone 457147936  
541562581 https://github.com/simonw/datasette/pull/590#issuecomment-541562581 https://api.github.com/repos/simonw/datasette/issues/590 MDEyOklzc3VlQ29tbWVudDU0MTU2MjU4MQ== rixx 2657547 2019-10-14T08:57:46Z 2019-10-14T08:57:46Z CONTRIBUTOR

Ah, thank you – I saw the need for unit tests but wasn't sure what the best way to add one would be.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spaces in DB names 505818256  
541587823 https://github.com/simonw/datasette/pull/590#issuecomment-541587823 https://api.github.com/repos/simonw/datasette/issues/590 MDEyOklzc3VlQ29tbWVudDU0MTU4NzgyMw== rixx 2657547 2019-10-14T09:58:23Z 2019-10-14T09:58:23Z CONTRIBUTOR

Added tests.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spaces in DB names 505818256  
544008463 https://github.com/simonw/datasette/pull/601#issuecomment-544008463 https://api.github.com/repos/simonw/datasette/issues/601 MDEyOklzc3VlQ29tbWVudDU0NDAwODQ2Mw== rixx 2657547 2019-10-18T23:39:21Z 2019-10-18T23:39:21Z CONTRIBUTOR

That looks right, and I completely agree with the intent.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't auto-format SQL on page load 509340359  
544008944 https://github.com/simonw/datasette/pull/601#issuecomment-544008944 https://api.github.com/repos/simonw/datasette/issues/601 MDEyOklzc3VlQ29tbWVudDU0NDAwODk0NA== rixx 2657547 2019-10-18T23:40:48Z 2019-10-18T23:40:48Z CONTRIBUTOR

The only negative impact that comes to mind is that now you have no way to get the read-only query to be formatted nicely, I think, so maybe a second PR adding the formatting functionality even to the read-only page would be good?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't auto-format SQL on page load 509340359  
544214418 https://github.com/simonw/datasette/pull/601#issuecomment-544214418 https://api.github.com/repos/simonw/datasette/issues/601 MDEyOklzc3VlQ29tbWVudDU0NDIxNDQxOA== rixx 2657547 2019-10-20T02:29:49Z 2019-10-20T02:29:49Z CONTRIBUTOR

Submitted in #602!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't auto-format SQL on page load 509340359  
549246007 https://github.com/simonw/datasette/pull/602#issuecomment-549246007 https://api.github.com/repos/simonw/datasette/issues/602 MDEyOklzc3VlQ29tbWVudDU0OTI0NjAwNw== rixx 2657547 2019-11-04T07:29:33Z 2019-11-04T07:29:33Z CONTRIBUTOR

Not sure – I'm always a bit weirded out when elements that I clicked disappear on me.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Offer to format readonly SQL 509535510  
556749086 https://github.com/simonw/datasette/issues/394#issuecomment-556749086 https://api.github.com/repos/simonw/datasette/issues/394 MDEyOklzc3VlQ29tbWVudDU1Njc0OTA4Ng== jsfenfen 639012 2019-11-21T01:15:34Z 2019-11-21T01:21:45Z CONTRIBUTOR

Hey @simonw is the url_prefix config option available in another branch, it looks like you've written some tests for it above? In 0.32 I get "url_prefix is not a valid option". I think this would be really helpful!

This would be really handy for proxying datasette in another domain's subdirectory I believe this will allow folks to run upstream authentication, but the links break if the url_prefix doesn't match.

I'd prefer not to host a proxied version of datasette on a subdomain (e.g. datasette.myurl.com b/c then I gotta worry about sharing authorization cookies with the subdomain, which I just assume not do, but...)

Edit: I see the wip-url-prefix branch, I may try with that https://github.com/simonw/datasette/commit/8da2db4b71096b19e7a9ef1929369b8483d448bf

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
base_url configuration setting 396212021  
558687342 https://github.com/simonw/datasette/issues/639#issuecomment-558687342 https://api.github.com/repos/simonw/datasette/issues/639 MDEyOklzc3VlQ29tbWVudDU1ODY4NzM0Mg== jacobian 21148 2019-11-26T15:40:00Z 2019-11-26T15:40:00Z CONTRIBUTOR

A bit of background: the reason heroku git:clone brings down an empty directory is because datasette publish heroku uses the builds API, rather than a git push, to release the app. I originally did this because it seemed like a lower bar than having a working git, but the downside is, as you found out, that tweaking the created app is hard.

So there's one option -- change datasette publish heroku to use git push instead of heroku builds:create.

@pkoppstein - what you suggested seems like it ought to work (you don't need maintenance mode, though). I'm not sure why it doesn't.

You could also look into using the slugs API to download the slug, change metadata.json, re-pack and re-upload the slug.

Ultimately though I think I think @simonw's idea of reading metadata.json from an external source might be better (#357). Reading from an alternate URL would be fine, or you could also just stuff the whole metadata.json into a Heroku config var, and write a plugin to read it from there.

Hope this helps a bit!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
updating metadata.json without recreating the app 527670799  
559207224 https://github.com/simonw/datasette/issues/642#issuecomment-559207224 https://api.github.com/repos/simonw/datasette/issues/642 MDEyOklzc3VlQ29tbWVudDU1OTIwNzIyNA== psychemedia 82988 2019-11-27T18:40:57Z 2019-11-27T18:41:07Z CONTRIBUTOR

Would cookie cutter approaches also work for creating various flavours of customised templates?

I need to try to create a couple of sites for myself to get a feel for what sorts of thing are easily doable, and what cribbable cookie cutter items might be. I'm guessing https://simonwillison.net/2019/Nov/25/niche-museums/ is a good place to start from?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Provide a cookiecutter template for creating new plugins 529429214  
559632608 https://github.com/simonw/datasette/issues/573#issuecomment-559632608 https://api.github.com/repos/simonw/datasette/issues/573 MDEyOklzc3VlQ29tbWVudDU1OTYzMjYwOA== psychemedia 82988 2019-11-29T01:43:38Z 2019-11-29T01:43:38Z CONTRIBUTOR

In passing, it looks like a start was made on a datasette Jupyter server extension in https://github.com/lucasdurand/jupyter-datasette although the build fails in MyBinder.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Exposing Datasette via Jupyter-server-proxy 492153532  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);