https://latest.datasette.io/fixtures/facetable.json doesn't currently include the CREATE TABLE statement for the page, even though it's available on the HTML version at https://latest.datasette.io/fixtures/facetable

https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_parse%28%2210+october+2020+3pm%22%29%2C%0D%0A++dateutil_easter%28%222020%22%29%2C%0D%0A++dateutil_parse_fuzzy%28%22This+is+due+10+september%22%29%2C%0D%0A++dateutil_parse%28%221%2F2%2F2020%22%29%2C%0D%0A++dateutil_parse%28%222020-03-04%22%29%2C%0D%0A++dateutil_parse_dayfirst%28%222020-03-04%22%29%2C%0D%0A++dateutil_easter%282020%29

Updates the requirements on pytest to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

When executing "datasette publish heroku schools.db" on Windows 10, I get the following error

  File "c:\users\dell\.virtualenvs\sec-schools-jn-cwk8z\lib\site-packages\datasette\publish\heroku.py", line 54, in heroku
    line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines()
  File "c:\python38\lib\subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "c:\python38\lib\subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "c:\python38\lib\subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "c:\python38\lib\subprocess.py", line 1307, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

Changing https://github.com/simonw/datasette/blob/55a6ffb93c57680e71a070416baae1129a0243b8/datasette/publish/heroku.py#L54

to

line.split()[0] for line in check_output(["heroku", "plugins"], shell=True).splitlines()

as well as the other check_output() and call() within the same file leads me to another recursive error about temp files

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Idea: if a database only has a single table, this could open straight to /db/table. If it has multiple tables but a single database it could open straight to /db.
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/970#issuecomment-698434236_

This is a request for a "convenience" feature, and only a nice to have. It's based on seeing this feature in several little command line hypertext server apps.

If you run, for example:

datasette.exe serve --open "mydb.s3db"

I would like it if default browser is launched, at the URL that is being served.

The angular cli does this, for example

ng serve <project> --open #see https://angular.io/cli/serve

...as does my usual mini web server of choice when inspecting local static files....

npx http-server -o # see https://www.npmjs.com/package/http-server

Just a tiny thing. Love your work!

This would remove the csvs-to-sqlite step which I end up using for almost everything.

I'm hesitant to introduce pandas as a required dependency though since it require compiling numpy. Could build it so this option is only available if you have pandas installed.

Using datasette to solve some frustrating problems with our fulfillment provider today, I was surprised to see repeated requests for assets under /-/static and the favicon. While it won't likely be a huge performance bottleneck, I bet datasette would feel a bit zippier if you had Uvicorn serving up some caching-related headers telling the browser it was safe to cache static assets.

https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++%5Bfoo%5D

Would be useful if this page showed you the invalid SQL you entered so you can edit it and try again.

To give an indication as to when the data was last updated.

This should be a field in the metadata that is then shown on the index page and in the footer, if it is set.

Also support setting it using an option to “datasette publish” and “datasette package” - which can either be a string or can be the magic string “today” to set it to today’s date:

datasette publish file.db --last_updated=today

I'm getting this when latest is deployed to Cloud Run:

Traceback (most recent call last):
  File "/usr/local/bin/datasette", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/datasette/cli.py", line 406, in serve
    inspect_data = json.load(open(inspect_file))
TypeError: 'bool' object is not callable

I think I may have broken things in #970 - a980199e61fe7ccf02c2123849d86172d2ae54ff

This issue is about how best to pass additional options to tools used for publishing datasettes. A concrete example is wanting to pass the --tar flag to the heroku CLI tool. I think there are at least two options for doing this: documentation for each publishing tool to explain how to set flags via env variables (if possible) or building a mechanism that lets users pass additional flags through datasette.

When using datasette publish heroku binder-launches.db --extra-options="--config facet_time_limit_ms:35000 --config sql_time_limit_ms:35000" --name=binderlytics --install=datasette-vega to publish https://binderlytics.herokuapp.com/ the following error happens:

 ›   Warning: heroku update available from 7.42.1 to 7.43.0.
 ›   Warning: heroku update available from 7.42.1 to 7.43.0.
 ›   Warning: heroku update available from 7.42.1 to 7.43.0.
Setting WEB_CONCURRENCY and restarting ⬢ binderlytics... done, v13
WEB_CONCURRENCY: 1
 ›   Warning: heroku update available from 7.42.1 to 7.43.0.
 ▸    Couldn't detect GNU tar. Builds could fail due to decompression errors
 ▸    See https://devcenter.heroku.com/articles/platform-api-deploying-slugs#create-slug-archive
 ▸    Please install it, or specify the '--tar' option
 ▸    Falling back to node's built-in compressor
buffer.js:358
    throw new ERR_INVALID_OPT_VALUE.RangeError('size', size);
    ^

RangeError [ERR_INVALID_OPT_VALUE]: The value "3303763968" is invalid for option "size"
    at Function.alloc (buffer.js:367:3)
    at new Buffer (buffer.js:281:19)
    at Readable.<anonymous> (/Users/thead/.local/share/heroku/node_modules/archiver-utils/index.js:39:15)
    at Readable.emit (events.js:322:22)
    at endReadableNT (/Users/thead/.local/share/heroku/node_modules/readable-stream/lib/_stream_readable.js:1010:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ERR_INVALID_OPT_VALUE'
}

After installing GNU tar with brew install gnu-tar and modifying datasette/publish/heroku.py to include the --tar=/path/to/gnu-tar publishing works.

I think the problem occurs once your heroku slug reaches a certain size. At least when I add only a few 100 entries to the datasette then the error does not occcur.

datasette version 0.49.1
OSX 10.14.6 (18G103)

datasette-graphql works by making internal requests to the TableView class (in order to take advantage of existing pagination logic, plus options like ?_search= and ?_where=) - see #915

I want to support a mod_rewrite style mechanism for putting nicer URLs on top of Datasette pages - I botched that together for a project here using an internal ASGI proxying trick: https://github.com/natbat/tidepools_near_me/commit/ec102c6da5a5d86f17628740d90b6365b671b5e1

If the datasette object provided a documented method for executing internal requests (in a way that makes sense with logging etc - i.e. doesn't get logged as a separate request) both of these use-cases would be much neater.

... support for running facets against arbitrary custom SQL queries is half-done in that facets now execute against wrapped subqueries as-of ea66c45df96479ef66a89caa71fff1a97a862646

https://github.com/simonw/datasette/blob/ea66c45df96479ef66a89caa71fff1a97a862646/datasette/facets.py#L192-L200
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/971#issuecomment-696307922_

dbstat is a table that is usually available on SQLite giving statistics about the database. For example:

https://fivethirtyeight.datasettes.com/fivethirtyeight?sql=SELECT+*+FROM+%22dbstat%22+WHERE+name%3D%27bachelorette%2Fbachelorette%27%3B

Other than direct select * from dbsat queries it is completely invisible.

It would be cool if https://fivethirtyeight.datasettes.com/fivethirtyeight/dbstat didn't 404 (on databases for which that table was available).

I thought this would never happen, but now that I'm deep in the weeds of running SQLite in production for Datasette Cloud I'm starting to reconsider my policy of only supporting SQLite.

Some of the factors making me think PostgreSQL support could be worth the effort:
- Serverless. I'm getting increasingly excited about writable-database use-cases for Datasette. If it could talk to PostgreSQL then users could easily deploy it on Heroku or other serverless providers that can talk to a managed RDS-style PostgreSQL.
- Existing databases. Plenty of organizations have PostgreSQL databases. They can export to SQLite using db-to-sqlite but that's a pretty big barrier to getting started - being able to run datasette postgresql://connection-string and start trying it out would be a massively better experience.
- Data size. I keep running into use-cases where I want to run Datasette against many GBs of data. SQLite can do this but PostgreSQL is much more optimized for large data, especially given the existence of tools like Citus.
- Marketing. Convincing people to trust their data to SQLite is potentially a big barrier to adoption. Even if I've convinced myself it's trustworthy I still have to convince everyone else.
- It might not be that hard? If this required a ground-up rewrite it wouldn't be worth the effort, but I have a hunch that it may not be too hard - most of the SQL in Datasette should work on both databases since it's almost all portable SELECT statements. If Datasette did DML this would be a lot harder, but it doesn't.
- Plugins! This feels like a natural surface for a plugin - at which point people could add MySQL support and suchlike in the future.

The above reasons feel strong enough to justify a prototype.

https://github.com/simonw/datasette/blob/master/docs/ecosystem.rst

From https://docs.python.org/3/library/sqlite3.html#sqlite3.enable_callback_tracebacks

sqlite3.``enable_callback_tracebacks(flag)¶

By default you will not get any tracebacks in user-defined functions, aggregates, converters, authorizer callbacks etc. If you want to debug them, you can call this function with flag set to True. Afterwards, you will get tracebacks from callbacks on sys.stderr. Use False to disable the feature again.

Maybe turn this on for all of Datasette? Are there any disadvantages to doing that?

https://github.com/simonw/datasette/runs/1043709494?check_suite_focus=true

https://user-images.githubusercontent.com/9599/91625110-80c55a80-e959-11ea-8fea-70508c53fcfb.png">

• This step should not run if a release is an alpha or beta
• When it DOES run it should work
• See it work for both an alpha and a non-alpha release, then close this ticket

https://scotthelme.co.uk/csrf-is-dead/ from Feb 2017 has background here. The SameSite=lax cookie property effectively eliminates CSRF in modern browsers. https://caniuse.com/#search=SameSite shows 92.13% global support for it.

Datasette already uses SameSite=lax when it sets cookies by default: https://github.com/simonw/datasette/blob/af350ba4571b8e3f9708c40f2ddb48fea7ac1084/datasette/utils/asgi.py#L327-L341

A few options then. I could ditch CSRF protection entirely. I could make it optional - turn it off by default, but let users who care about that remaining 7.87% of global users opt back into it.

One catch: login CSRF: I don't see how SameSite=lax protects against that attack.

Steps to reproduce:

1. Install datasette-media plugin
   pip install datasette-media
2. Launch datasette
   datasette databasename.db
3. Error

INFO:     Started server process [927704]
INFO:     Waiting for application startup.
ERROR:    Exception in 'lifespan' protocol
Traceback (most recent call last):
  File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/uvicorn/lifespan/on.py", line 48, in main
    await app(scope, self.receive, self.send)
  File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/datasette_media/__init__.py", line 9, in wrapped_app
    path = scope["path"]
KeyError: 'path'
ERROR:    Application startup failed. Exiting.

Compare https://latest.datasette.io/fixtures/sortable with https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+order+by+pk1%2C+pk2+limit+101

https://user-images.githubusercontent.com/9599/87612845-82e09c00-c6c0-11ea-806e-93764ca468c4.png">

Just a quick heads up:

Release notes for 0.45 include urls that point to localhost.

https://github.com/simonw/datasette/releases/tag/0.45

Shipping a release currently runs the tests twice: https://travis-ci.org/simonw/datasette/builds/611463728

It does a regular test run on Python 3.6/7/8 - then the "Release tagged version" step runs the tests again before publishing to PyPI! This second run is not necessary.

I was waiting for consensus to form around this (and kind-of hoping for trunk since I like the tree metaphor) and it looks like main is it.

I've seen convincing arguments against trunk too - it indicates that the branch has some special significance like in Subversion (where all branches come from trunk) when it doesn't. So main is better anyway.

When using ?_trace=1:

Traceback (most recent call last):
  File "/Users/simon/.local/share/virtualenvs/rockybeaches-09H592sC/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 390, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/Users/simon/.local/share/virtualenvs/rockybeaches-09H592sC/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/Users/simon/.local/share/virtualenvs/rockybeaches-09H592sC/lib/python3.8/site-packages/datasette/utils/asgi.py", line 150, in __call__
    await self.app(scope, receive, send)
  File "/Users/simon/.local/share/virtualenvs/rockybeaches-09H592sC/lib/python3.8/site-packages/datasette/tracer.py", line 137, in __call__
    await self.app(scope, receive, wrapped_send)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/Users/simon/.local/share/virtualenvs/rockybeaches-09H592sC/lib/python3.8/site-packages/datasette/tracer.py", line 63, in capture_traces
    del tracers[task_id]
KeyError: 4575365856

When I try to use the new ?_json=1 feature from #880 with magic parameters from #842 I get this error:

Incorrect number of bindings supplied. The current statement uses 1, and there are 0 supplied

Steps to reproduce: visit https://2a4b892.datasette.io/fixtures/roadside_attractions?_facet_m2m=attraction_characteristic and click "Apply"

Result is a 500: no such column: attraction_characteristic

The error occurs because of this hidden HTML input:

<input type="hidden" name="_facet" value="attraction_characteristic">

This should be:

<input type="hidden" name="_facet_m2m" value="attraction_characteristic">

_request_ip isn't valid, so it shouldn't be in the example: https://github.com/simonw/datasette/blob/cb515a9d75430adaf5e545a840bbc111648e8bfd/docs/sql_queries.rst#L320-L322

There are only a few classes that plugins need to import. It would be nice if these imports were as short and memorable as possible.

For example:

from datasette.app import Datasette
from datasette.utils.asgi import Response

Could both become:

from datasette import Datasette
from datasette import Response

These should release straight after Datasette 0.49 with the change from #953.

It looks like the tests take 3m33s to run in GitHub Actions, but they're taking more than 8 minutes in Travis

Now that CSRF is solved for API requests (#835) it would be good to support API requests to the .json extension.

This mechanism is not documented: https://github.com/simonw/datasette/blob/30b98e4d2955073ca2bca92ca7b3d97fcd0191bf/datasette/app.py#L1119-L1129

Having tried this out I think it does need a raise_404() mechanism - which needs to be smart enough to trigger the default 404 handler without accidentally going into an infinite loop.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/944#issuecomment-691788478_

Custom pages let you e.g. create a templates/pages/about.html page and have it automatically served at /about.

It would be useful if these pages could capture path patterns. I like the Python format string syntax for this (also used by Starlette): /foo/bar/{slug}.

So... how about embedding those patterns in the filenames themselves?

templates/pages/museums/{slug}.html

Would capture any hits to /museums/something and use that page to serve them.

Datasette ASGI #272 is a big part of it... but 1.0 will generally be an indicator that Datasette is a stable platform for developers to write plugins and custom templates against. So lots to think about.

Faceted search uses JSON array elements as facets rather than the arrays. However, if a search is "Apply"ed (using the Apply button), the array itself rather than its elements used.

To reproduce:
https://latest.datasette.io/fixtures/facetable?_sort=pk&_facet=created&_facet=tags&_facet_array=tags

Press "Apply", which might be done when removing a filter. Notice that the "tags" facet values are now arrays, not array elements. It appears the "&_facet_array=tags" element of the query string is dropped.

The default JSON just isn't right. I find myself using ?_shape=array for almost everything I build against the API.

Datasette views currently work by creating a set of data that should be returned as JSON, then defining an additional, optional template_data() function which is called if the view is being rendered as HTML.

This template_data() function calculates extra template context variables which are necessary for the HTML view but should not be included in the JSON.

Example of how that is used today: https://github.com/simonw/datasette/blob/2b79f2bdeb1efa86e0756e741292d625f91cb93d/datasette/views/table.py#L672-L704

With features like Facets in #255 I'm beginning to want to move more items into the template_data() - in the case of facets it's the suggested_facets array. This saves that feature from being calculated (involving several SQL queries) for the JSON case where it is unlikely to be used.

But... as an API user, I want to still optionally be able to access that information.

Solution: Add a ?_extra=suggested_facets&_extra=table_metadata argument which can be used to optionally request additional blocks to be added to the JSON API.

Then redefine as many of the current template_data() features as extra arguments instead, and teach Datasette to return certain extras by default when rendering templates.

This could allow the JSON representation to be slimmed down further (removing e.g. the table_definition and view_definition keys) while still making that information available to API users who need it.

The most manual part of the release process right now is having to post a GitHub release that matches the updated changelog.

This is particularly annoying because the changelog is in .rst while the GitHub release needs markdown - so I currently manually translate between the two.

Having the release script automatically post a GitHub release at the end would be much more convenient.

I had to build a custom _actorornull prefix for datasette-saved-queries:

def actorornull(key, request):
    if request.actor is None:
        return None
    return request.actor.get(key)


@hookimpl
def register_magic_parameters():
    return [
        ("actorornull", actorornull),
    ]

Maybe the actor magic in Datasette core should do that out of the box?

https://github.com/simonw/datasette/blob/f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb/datasette/default_magic_parameters.py#L14-L17

Datasette currently bundles 5.31.0 (from October 2017) - latest version is 5.57.0 (August 2020). https://codemirror.net/doc/releases.html

Black has some changes: https://black.readthedocs.io/en/stable/change_log.html#b0 - in particular:

• re-implemented support for explicit trailing commas: now it works consistently within any bracket pair, including nested structures (#1288 and duplicates)
• Black now reindents docstrings when reindenting code around it (#1053)

I needed to debug an exception from deep inside a Jinja template the other day. I hacked this together and it helped.

Now that we have support for plugins that can write I'm seeing all sorts of places where a plugin might need to add UI to the table page.

Some examples:

• datasette-configure-fts needs to add a "configure search for this table" link
• a plugin that lets you render or delete tables needs to add a link or button somewhere
• existing plugins like datasette-vega and datasette-cluster-map already do this with JavaScript

The challenge here is that multiple plugins may want to do this, so simply overriding templates and populating names blocks doesn't entirely work as templates may override each other.

If you run datasette . --get / and the result is a 500 or 404 error (anything that's not a 200 or a 30x) the exit code from the command should not be 0.

It should still output the returned content to stdout.

This will help with writing soundness checks, as seen in https://til.simonwillison.net/til/til/github-actions_grep-tests.md

I lost a bunch of time yesterday trying to figure out why a Datasette instance wasn't starting up - it turned out it was because I had a facets: reference that mentioned a column that did not exist.

Catching these on startup would be good.

I just wrote this code:

    results = await database.execute(SEARCH_SQL, {"query": query})
    return [dict(r) for r in results.rows]

How about having results.dicts as a utility property that does that?

Updates the requirements on black to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I'm having a lot of trouble copying and pasting from the codemirror editor on my iPhone.

Documentation says that the old dictionary mechanism will be deprecated by 1.0:

https://github.com/simonw/datasette/blob/799ecae94824640bdff21f86997f69844048d5c3/docs/plugin_hooks.rst#L460
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/953#issuecomment-682312494_

This will also give Homebrew a way to upgrade Datasette itself without having to wait for the latest packaged version to land in Homebrew core.

That plugin hook was designed before Datasette had a documented Response class. It should optionally be allowed to return a Response in addition to the current custom dictionary.

In thinking about the best way to implement https://github.com/simonw/datasette-auth-passwords/issues/6 (SQL-backed user accounts for datasette-auth-passwords) I realized that there are a few different use-cases where a plugin might want to store data that isn't visible to regular Datasette users:

• Storing password hashes
• Storing API tokens
• Storing secrets that are used for data import integrations (secrets for talking to the Twitter API for example)

Idea: allow one or more private database files to be attached to Datasette, something like this:

datasette github.db linkedin.db -s secrets.db -m metadata.yml

The secrets.db file would not be visible using any of the Datasette's usual interface or API routes - but plugins would be able to run queries against it.

So datasette-auth-passwords might then be configured like this:

plugins:
  datasette-auth-passwords:
    database: secrets
    sql: "select password_hash from passwords where username = :username"

The plugin could even refuse to operate against a database that hadn't been loaded as a secret database.

It would also be interesting to try out the SQL hint mode, which can autocomplete against tables and columns. This demo shows how to configure that: https://codemirror.net/mode/sql/

Some missing documentation: https://stackoverflow.com/questions/20023381/codemirror-how-add-tables-to-sql-hint
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/948#issuecomment-679355426_

_Originally posted by @simonw in https://github.com/simonw/datasette-graphql/issues/2#issuecomment-667780040_

This isn't particularly straight-forward - all the more reason for Datasette to implement it for you. This article is helpful: http://charlesleifer.com/blog/using-sqlite-full-text-search-with-python/

Could look something like this:

{
    "title": "Five Thirty Eight",
    "license": "CC Attribution 4.0 License",
    "license_url": "https://creativecommons.org/licenses/by/4.0/",
    "source": "fivethirtyeight/data on GitHub",
    "source_url": "https://github.com/fivethirtyeight/data",
    "databases": {
        "fivethirtyeight": {
            "tables": {
                "mueller-polls/mueller-approval-polls": {
                    "description_html": "<p>....</p>",
                    "columns": {
                        "name_of_column": "column_description goes here"
}

$ datasette -p 0 --root
http://127.0.0.1:0/-/auth-token?token=2d498c...

The port is incorrect.

Refs #940

https://latest.datasette.io/fixtures/binary_data.json?_sort_desc=data&_shape=array returns this:

[
  {
    "rowid": 1,
    "data": "this is binary data"
  }
]

But adding &_nl=on returns this: https://latest.datasette.io/fixtures/binary_data.json?_sort_desc=data&_shape=array&_nl=on

{
  "ok": false,
  "error": "Object of type bytes is not JSON serializable",
  "status": 500,
  "title": null
}

I found this error by running wget -r 127.0.0.1:8001 against my local fixtures.db.

• extra_css_urls(template, database, table, datasette)
• extra_js_urls(template, database, table, datasette)
• extra_body_script(template, database, table, view_name, datasette)
• extra_template_vars(template, database, table, view_name, request, datasette)

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/938#issuecomment-674544691_

I'd like datasette-cluster-map to only add links to JavaScript on pages that have tables with latitude and longitude columns.

Passing the names of the columns to the plugin hook can support this and will be backwards compatible thanks to pluggy.

Discovered in https://github.com/simonw/latest-datasette-with-all-plugins/issues/3#issuecomment-674449757

Refs #935

https://github.com/simonw/datasette/blob/7702ea602188899ee9b0446a874a6a9b546b564d/datasette/cli.py#L417-L433

Spotted this working on https://github.com/simonw/latest-datasette-with-all-plugins/issues/3 - I'd like to be able to use datasette --get / as a sanity checking test, but that doesn't work if the init hooks aren't fully executed.

Datasette's documentation is aimed at people who install and configure it.

What about end users of preconfigured and deployed Datasette instances?

Something that can be linked to from the Datasette UI would be really useful.

e.g. https://travis-ci.org/github/simonw/datasette/jobs/717123725

Here's how it broke:

--2020-08-12 03:08:17--  https://www.gaia-gis.it/gaia-sins/freexl-1.0.5.tar.gz
Resolving www.gaia-gis.it (www.gaia-gis.it)... 212.83.162.51
Connecting to www.gaia-gis.it (www.gaia-gis.it)|212.83.162.51|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-08-12 03:08:18 ERROR 404: Not Found.
The command '/bin/sh -c wget "https://www.gaia-gis.it/gaia-sins/freexl-1.0.5.tar.gz" && tar zxf freexl-1.0.5.tar.gz     && cd freexl-1.0.5 && ./configure && make && make install' returned a non-zero code: 8
The command "docker build -f Dockerfile -t $REPO:$TRAVIS_TAG ." exited with 8.
0.07s$ docker tag $REPO:$TRAVIS_TAG $REPO:latest
Error response from daemon: No such image: [secure]/datasette:0.47.1
The command "docker tag $REPO:$TRAVIS_TAG $REPO:latest" exited with 1.
0.08s$ docker push $REPO
The push refers to repository [docker.io/[secure]/datasette]
An image does not exist locally with the tag: [secure]/datasette
The command "docker push $REPO" exited with 1.
cache.2
store build cache

Pretty nasty bug this: I'm getting 500 errors for all pages that try to render a template after installing the newly released Datasette 0.47 - both from pip install and via Homebrew.

I can expose ALL of Datasette's functionality on the command-line (without even running a web server) by adding --get and --post options to datasette serve.

datasette fixtures.db --get "/fixtures.json"

This would instantiate the Datasette ASGI app, run a fake request for /fixtures.json through it, dump the results out to standard output and quit.

A --post option could do the same for a POST request. Treating that as a stretch goal for the moment.

Refs #926, #898

Errors like this one:

https://github.com/simonw/datasette/pull/927/checks?check_run_id=973559696

2020-08-11T23:36:39.8801334Z =================================== FAILURES ===================================
2020-08-11T23:36:39.8802411Z _________________________________ test_install _________________________________
2020-08-11T23:36:39.8803242Z 
2020-08-11T23:36:39.8804935Z thing = <module 'pip._internal.cli' from '/opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/pip/_internal/cli/__init__.py'>
2020-08-11T23:36:39.8806663Z comp = 'main', import_path = 'pip._internal.cli.main'
2020-08-11T23:36:39.8807696Z 
2020-08-11T23:36:39.8808728Z     def _dot_lookup(thing, comp, import_path):
2020-08-11T23:36:39.8810573Z         try:
2020-08-11T23:36:39.8812262Z >           return getattr(thing, comp)
2020-08-11T23:36:39.8817136Z E           AttributeError: module 'pip._internal.cli' has no attribute 'main'
2020-08-11T23:36:39.8843043Z 
2020-08-11T23:36:39.8855951Z /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/unittest/mock.py:1215: AttributeError
2020-08-11T23:36:39.8873372Z 
2020-08-11T23:36:39.8877803Z During handling of the above exception, another exception occurred:
2020-08-11T23:36:39.8906532Z 
2020-08-11T23:36:39.8925767Z     def get_src_prefix():
2020-08-11T23:36:39.8928277Z         # type: () -> str
2020-08-11T23:36:39.8930068Z         if running_under_virtualenv():
2020-08-11T23:36:39.8949721Z             src_prefix = os.path.join(sys.prefix, 'src')
2020-08-11T23:36:39.8951813Z         else:
2020-08-11T23:36:39.8969014Z             # FIXME: keep src in cwd for now (it is not a temporary folder)
2020-08-11T23:36:39.9012110Z             try:
2020-08-11T23:36:39.9013489Z >               src_prefix = os.path.join(os.getcwd(), 'src')
2020-08-11T23:36:39.9014538Z E               FileNotFoundError: [Errno 2] No such file or directory
2020-08-11T23:36:39.9016122Z 
2020-08-11T23:36:39.9017617Z /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/pip/_internal/locations.py:50: FileNotFoundError
2020-08-11T23:36:39.9018802Z 
2020-08-11T23:36:39.9020070Z During handling of the above exception, another exception occurred:
2020-08-11T23:36:39.9020930Z 
2020-08-11T23:36:39.9022275Z args = (), keywargs = {}
2020-08-11T23:36:39.9023183Z 
2020-08-11T23:36:39.9024077Z     @wraps(func)
2020-08-11T23:36:39.9024984Z     def patched(*args, **keywargs):
2020-08-11T23:36:39.9028770Z >       with self.decoration_helper(patched,
2020-08-11T23:36:39.9031861Z                                     args,
2020-08-11T23:36:39.9038358Z                                     keywargs) as (newargs, newkeywargs):
2020-08-11T23:36:39.9039654Z 
2020-08-11T23:36:39.9040566Z /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/unittest/mock.py:1322: 
2020-08-11T23:36:39.9041492Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

$ brew tap simonw/datasette $ brew install simonw/datasette/datasette $ datasette --version datasette, version 0.46
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/335#issuecomment-672088880_

When installing Datasette plugins it's crucial that they end up in the same virtual environment as Datasette itself.

It's not necessarily obvious how to do this, especially if you install Datasette via pipx or homebrew.

Solution: datasette install datasette-vega and datasette uninstall datasette-vega commands that know how to install to the correct place - a very thin wrapper around pip install.

https://docs.brew.sh/Python-for-Formula-Authors describes how.

Applications should be installed into a Python virtualenv environment rooted in libexec. This prevents the app’s Python modules from contaminating the system site-packages and vice versa.

It recommends using https://github.com/tdsmith/homebrew-pypi-poet

The HTML form for a read-only canned query includes the hidden CSRF token field added in #798 for writable canned queries (#698).

This means that submitting those read-only forms exposes the CSRF token in the URL - for example on https://latest.datasette.io/fixtures/neighborhood_search submitting the form took me to:

https://latest.datasette.io/fixtures/neighborhood_search?text=down&csrftoken=IlFubnoxVVpLU1NGT3NMVUoi.HbOPd2YH_epQmp8f_aAt0s-MxtU

This token could potentially leak to an attacker if the resulting page has a link to an external site on it and the user clicks the link, since the token would be exposed in the referral logs.

Caused by #849 - since we are mirroring the two branches (to ensure old links to master keep working) Travis is building both.

The following in .travis.yml should fix that:

branches:
  except:
  - master

This is a pattern I often find myself needing. I usually implement this in GitHub Actions like this:

https://github.com/simonw/covid-19-datasette/blob/efa01c39abc832b8641fc2a92840cc3acae2fb08/.github/workflows/scheduled.yml#L52-L63

    - name: Set variables to decide if we should deploy
      id: decide_variables
      run: |-
        echo "##[set-output name=latest;]$(datasette inspect covid.db | jq '.covid.hash' -r)"
        echo "##[set-output name=deployed;]$(curl -s https://covid-19.datasettes.com/-/databases.json | jq '.[0].hash' -r)"
    - name: Set up Cloud Run
      if: github.event_name == 'workflow_dispatch' || steps.decide_variables.outputs.latest != steps.decide_variables.outputs.deployed
      uses: GoogleCloudPlatform/github-actions/setup-gcloud@master

This is pretty fiddly. It might be good for datasette publish to grow a helper option that does effectively this - hashes the databases (and the metadata.json) and compares them to the deployed version.

I need this for datasette-graphql for full compatibility with the way Relay likes to paginate - using cursors for paginating backwards as well as for paginating forwards.

This may be the kick I need to get Datasette pagination to work in reverse too.
_Originally posted by @simonw in https://github.com/simonw/datasette-graphql/issues/2#issuecomment-668305853_

Hi,

I've been playing with Datasette behind Nginx (awesome tool, thanks !). It seems some URLs are OK but some aren't. For instance in https://github.com/simonw/datasette/blob/master/datasette/templates/query.html#L61 it seems that url_csv includes a / prefix, resulting in the base_url not beeing honored.

Actually here, it seems that dropping the prefix / to make the link relative is enough (so it may not be strictly related to base_url).

Additional information:

datasette, version 0.45+0.gf1f581b.dirty

Relevant Nginx configuration (note that all the trailing slashes have some effect):

  location /datasette/ {
    proxy_pass http://127.0.0.1:9001/;
    proxy_set_header Host $host;
  }

Relelvant Datasette configuration (slashes matter too):

  --config base_url:/datasette/

Hello,

Until now I'm using datasette without any authentication system but I would like to setup a configuration or limiting the number of requests per user (eventually by IP or with a cookie mechanism) and eventually allowing me to ban specific users/IPs.

Is there a plugin available for this use case ?
If not what are your insights regarding this UC ?

Should I write a plugin ? Should I deploy datasette behind a reverse proxy to manage this ?

It's a shame there's no obvious mechanism for passing additional options to datasette my.db that affect how plugins work.

The only way I can think of at the moment is via environment variables:

DATASETTE_INSERT_UNSAFE=1 datasette my.db

This will have to do for the moment - it's ugly enough that people will at least know they are doing something unsafe, which is the goal here.
_Originally posted by @simonw in https://github.com/simonw/datasette-insert/issues/15#issuecomment-667346438_

When passing a mutable value as a default argument in a function, the default argument is mutated anytime that value is mutated. This poses a bug risk. Instead, use None as a default and assign the mutable value inside the function.

https://datasette.readthedocs.io/en/0.45/publish.html#datasette-publish currently only lists Cloud Run, Heroku and Fly. It should list Vercel too.

(I should probably rename datasette-publish-now to datasette-publish-vercel)

https://github.com/simonw/datasette/blob/3c33b421320c0be81a625ca7307b2e4416a9ed5b/datasette/utils/asgi.py#L396-L405
self.filename should be passed to asgi_send_file()

Updates the requirements on pytest to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

--name works inconsistently across the different publish providers - on Cloud Run you should use --service instead for example. Need to review it across all of them and either remove it or clarify what it does.

Picking up the work done on #557 with a new PR. Seeing if I can get this working.

The "allow" syntax described at https://datasette.readthedocs.io/en/0.45/authentication.html#defining-permissions-with-allow-blocks currently says this:

An allow block can specify "no-one is allowed to do this" using an empty {}:

{ "allow": {} }

"allow": null allows all access, though this isn't documented (it should be though).

These are not very intuitive. How about also supporting "allow": true for "allow anyone" and "allow": false for "allow nobody"?

It might be good to have a little interactive tool which helps debug these things, since there are quite a few edge-cases and the damage caused if people use them incorrectly is substantial.
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/907#issuecomment-663726146_

Documentation here: https://datasette.readthedocs.io/en/0.45/authentication.html#defining-permissions-with-allow-blocks

Doesn't explain that with the following "allow" block:

{
  "allow": {
    "id": "simonw",
    "role": "staff"
  }
}

The rule will match if EITHER the id is simonw OR the role includes staff.

The tests are missing this case too: https://github.com/simonw/datasette/blob/028f193dd6233fa116262ab4b07b13df7dcec9be/tests/test_utils.py#L504

Related to #906

Because setup.py uses find_packages and tests is on the top-level, pip install datasette will install a top-level package called tests, which is probably not desired behavior.

The offending line is here:
https://github.com/simonw/datasette/blob/bfa2ae0d16d39bb82dbe4da4f3fdc3c7f6257418/setup.py#L40

And only pip uninstall datasette with a conflicting package would warn you by default; apparently another package had the same problem, which is why I get this message when uninstalling:

$ pip uninstall datasette
Uninstalling datasette-0.27:
  Would remove:
    /usr/local/bin/datasette
    /usr/local/lib/python3.7/site-packages/datasette-0.27.dist-info/*
    /usr/local/lib/python3.7/site-packages/datasette/*
    /usr/local/lib/python3.7/site-packages/tests/*
  Would not remove (might be manually added):
    [ .. snip .. ]
Proceed (y/n)? 

This should be a relatively simple fix, and I could drop a PR if desired!

Cheers

The exclude argument to find_packages needs an iterable of package
names.

Fixes: #456

I can do this by modifying this function: https://github.com/simonw/datasette/blob/02dc6298bdbfb1d63e0d2a39ff597b5fcc60e06b/datasette/utils/asgi.py#L248-L270

I tried using this block of template in a plugin and got an error:

{% block nav %}
    <p class="crumbs">
        <a href="{{ base_url }}">home</a> /
        <a href="{{ database_url(database) }}">{{ database }}</a> /
        <a href="{{ database_url(database) }}/{{ table|quote_plus }}">{{ table }}</a>
    </p>
    {{ super() }}
{% endblock %}

Error: 'database_url' is undefined

That's because database_url is only made available by the BaseView template here:

https://github.com/simonw/datasette/blob/d6e03b04302a0852e7133dc030eab50177c37be7/datasette/views/base.py#L110-L125

https://github.com/simonw/til/blob/master/pytest/registering-plugins-in-tests.md

Would be useful to include this pattern on https://datasette.readthedocs.io/en/stable/testing_plugins.html

This PR fixes #456 by adding tests to the package exclusion list.

Cheers