issues

From https://docs.python.org/3/library/sqlite3.html#sqlite3.enable_callback_tracebacks

sqlite3.``enable_callback_tracebacks(flag)¶

By default you will not get any tracebacks in user-defined functions, aggregates, converters, authorizer callbacks etc. If you want to debug them, you can call this function with flag set to True. Afterwards, you will get tracebacks from callbacks on sys.stderr. Use False to disable the feature again.

Maybe turn this on for all of Datasette? Are there any disadvantages to doing that?

I'm going to draw the line here: default Datasette supports authentication but only for a single user account ("admin"). Plugins can then add support for multiple user accounts, social auth, SSO etc.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/699#issuecomment-636498770

The current behavior for --plugins-dir is to load every file in that folder as a python module. This can result in errors if there are non-python files in the plugins dir (such as .mypy_cache).

This PR restricts the module loading to only python files.

A reference implementation for adding support to reload when datasette is in the config_dir mode.

This implementation is flawed since it is watching the entire directory and any changes to the database will reload the server and adding unrelated files to the directory will also reload the server.

Now that we have support for plugins that can write I'm seeing all sorts of places where a plugin might need to add UI to the table page.

Some examples:

• datasette-configure-fts needs to add a "configure search for this table" link
• a plugin that lets you render or delete tables needs to add a link or button somewhere
• existing plugins like datasette-vega and datasette-cluster-map already do this with JavaScript

The challenge here is that multiple plugins may want to do this, so simply overriding templates and populating names blocks doesn't entirely work as templates may override each other.

The default JSON just isn't right. I find myself using ?_shape=array for almost everything I build against the API.

Steps to reproduce:

1. Install datasette-media plugin
   pip install datasette-media
2. Launch datasette
   datasette databasename.db
3. Error

INFO:     Started server process [927704]
INFO:     Waiting for application startup.
ERROR:    Exception in 'lifespan' protocol
Traceback (most recent call last):
  File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/uvicorn/lifespan/on.py", line 48, in main
    await app(scope, self.receive, self.send)
  File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/datasette_media/__init__.py", line 9, in wrapped_app
    path = scope["path"]
KeyError: 'path'
ERROR:    Application startup failed. Exiting.

Just a quick heads up:

Release notes for 0.45 include urls that point to localhost.

https://github.com/simonw/datasette/releases/tag/0.45

I had to build a custom _actorornull prefix for datasette-saved-queries:

def actorornull(key, request):
    if request.actor is None:
        return None
    return request.actor.get(key)


@hookimpl
def register_magic_parameters():
    return [
        ("actorornull", actorornull),
    ]

Maybe the actor magic in Datasette core should do that out of the box?

https://github.com/simonw/datasette/blob/f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb/datasette/default_magic_parameters.py#L14-L17

This page here - the URL is http://127.0.0.1:8001/data/all_tables but "all_tables" is not shown in the UI:

Potential fix for https://github.com/simonw/datasette/issues/859.

Disabling table counts for hidden tables speeds up database page quite a bit. In my setup it reduced load time by 2/3 (~300 -> ~90ms)

https://www.slideshare.net/nataliedowne/practical-maintainable-css/75

This will be a single page that demonstrates all of the different CSS styles and classes available to Datasette.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/840#issuecomment-650895874

I have several plugins currently marked as alphas because they depend on 0.45a3. When 0.45 is released I can ship new versions of these plugins that are full releases, not alphas - and switch them to depending on 0.45 (as opposed to the alpha):

• https://github.com/simonw/datasette-init
• https://github.com/simonw/datasette-glitch
• https://github.com/simonw/datasette-saved-queries
• https://github.com/simonw/datasette-write

Needs to run somewhere with a mutable disk drive, so not Cloud Run or Heroku or Vercel.

I think I'll put it on Glitch.

These are mostly done thanks to the alphas, but I went to have more paragraphs of prose and less bullet points.

Right now the "Log out" button in the navigation will show up even if the user was authenticated by a plugin using a mechanism other than the ds_actor cookie. It should only show if the logged-in user has that cookie.

Now that CSRF is solved for API requests (#835) it would be good to support API requests to the .json extension.

https://scotthelme.co.uk/csrf-is-dead/ from Feb 2017 has background here. The SameSite=lax cookie property effectively eliminates CSRF in modern browsers. https://caniuse.com/#search=SameSite shows 92.13% global support for it.

Datasette already uses SameSite=lax when it sets cookies by default: https://github.com/simonw/datasette/blob/af350ba4571b8e3f9708c40f2ddb48fea7ac1084/datasette/utils/asgi.py#L327-L341

A few options then. I could ditch CSRF protection entirely. I could make it optional - turn it off by default, but let users who care about that remaining 7.87% of global users opt back into it.

One catch: login CSRF: I don't see how SameSite=lax protects against that attack.

This is a frustrating workaround. I have a restore_working_directory fixture that I wrote to solve errors that look like this:

/Users/simon/Dropbox/Development/datasette/tests/test_publish_cloudrun.py:148: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:112: in __enter__
    return next(self.gen)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <click.testing.CliRunner object at 0x1135ad110>

    @contextlib.contextmanager
    def isolated_filesystem(self):
        """A context manager that creates a temporary folder and changes
        the current working directory to it for isolated filesystem tests.
        """
>       cwd = os.getcwd()
E       FileNotFoundError: [Errno 2] No such file or directory

Here's an example of it in use: removing the restore_working_directory argument from this function causes the failure. https://github.com/simonw/datasette/blob/549b1c2063db48c4622ee5c7b478a1e3cbc1ac07/tests/test_plugins.py#L689-L690

I'd like to not have to do this.

Currently view permission failures raise a Forbidden error which is transformed into a 403.

It would be good if this page could offer a way forward - maybe just by linking to (or redirecting to) a login screen. This behaviour will vary based on authentication plugins, so a new plugin hook is probably the best way to do this.

Hey,
I have a database that I save in HTML from couple of web scrapers. There are around 200k+, 50+ rows in a couple of tables, with sqlite file weighing around 600MB.

The app runs on a VPS with 2 core CPU, 4GB RAM and refreshing database page regularly takes more than 10 seconds. I was suspecting that counting tables was the culprit, but manually running select count(*) from table_name for the largest table finishes under a second.

I've looked at the source code. There's a check for index page for mutable databases larger than 100MB
https://github.com/simonw/datasette/blob/799c5d53570d773203527f19530cf772dc2eeb24/datasette/views/index.py#L15

but this check is not performed for database page.
I've manually crippled Database::table_counts method

async def table_counts(self, limit=10):
    if not self.is_mutable and self.cached_table_counts is not None:
        return self.cached_table_counts
    # Try to get counts for each table, $limit timeout for each count
    counts = {}
    for table in await self.table_names():
        try:
            # table_count = (
            #     await self.execute(
            #         "select count(*) from [{}]".format(table),
            #         custom_time_limit=limit,
            #     )
            # ).rows[0][0]
            counts[table] = 10 # table_count
        # In some cases I saw "SQL Logic Error" here in addition to
        # QueryInterrupted - so we catch that too:
        except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError):
            counts[table] = None
    if not self.is_mutable:
        self.cached_table_counts = counts
    return counts

now the page loads in <100ms.

Is it possible to apply size check on database page too?

{

    "python": {

        "version": "3.8.0",

        "full": "3.8.0 (default, Oct 28 2019, 16:14:01) \n[GCC 8.3.0]"

    },

    "datasette": {

        "version": "0.44"

    },

    "asgi": "3.0",

    "uvicorn": "0.11.5",

    "sqlite": {

        "version": "3.22.0",

        "fts_versions": [

            "FTS5",

            "FTS4",

            "FTS3"

        ],

        "extensions": {

            "json1": null

        },

        "compile_options": [

            "COMPILER=gcc-7.4.0",

            "ENABLE_COLUMN_METADATA",

            "ENABLE_DBSTAT_VTAB",

            "ENABLE_FTS3",

            "ENABLE_FTS3_PARENTHESIS",

            "ENABLE_FTS3_TOKENIZER",

            "ENABLE_FTS4",

            "ENABLE_FTS5",

            "ENABLE_JSON1",

            "ENABLE_LOAD_EXTENSION",

            "ENABLE_PREUPDATE_HOOK",

            "ENABLE_RTREE",

            "ENABLE_SESSION",

            "ENABLE_STMTVTAB",

            "ENABLE_UNLOCK_NOTIFY",

            "ENABLE_UPDATE_DELETE_LIMIT",

            "HAVE_ISNAN",

            "LIKE_DOESNT_MATCH_BLOBS",

            "MAX_SCHEMA_RETRY=25",

            "MAX_VARIABLE_NUMBER=250000",

            "OMIT_LOOKASIDE",

            "SECURE_DELETE",

            "SOUNDEX",

            "TEMP_STORE=1",

            "THREADSAFE=1"

        ]

    }

}

While experimenting with https://github.com/simonw/datasette-auth-tokens I realized it's not currently possible to build API client programs that POST to Datasette because there's no mechanism for them to skip the CSRF checks added in #798.

Follows #875

https://datasette.readthedocs.io/en/0.44/pages.html

Note that these URLs end in a 7 character hash. This hash is derived from the contents of the database, and ensures that each URL is immutable: the data returned from a URL containing the hash will always be the same, since if the contents of the database file changes by even a single byte a new hash will be generated.

This isn't accurate any more - that's not default behaviour, and it may be removed entirely in #647.

Stumbled into this while working on datasette-permissions-sql. I had granted table permissions, but the permission check wasn't even executed because the user failed the previous view-database check.

It would be much cleaner if Datasette's default views were all registered using the new register_routes() plugin hook. Could dramatically reduce the code in datasette/app.py.

The ideal fix here would be to rework my BaseView subclass mechanism to work with register_routes() so that those views don't have any special privileges above plugin-provided views.
Originally posted by @simonw in https://github.com/simonw/datasette/issues/864#issuecomment-648580556

Can be part of #870 - refactoring existing views to use register_routes().

I'm going to put the new check_permissions() method on BaseView as well. If I want that method to be available to plugins I can do so by turning that BaseView class into a documented API that plugins are encouraged to use themselves.
Originally posted by @simonw in https://github.com/simonw/datasette/issues/832#issuecomment-651995453

Picking up the work done on #557 with a new PR. Seeing if I can get this working.

$ datasette -p 0 --root
http://127.0.0.1:0/-/auth-token?token=2d498c...

The port is incorrect.

Need a cookie clearing mechanism and a way to show that you are logged in.

datasette-auth-github had a solution for this that can be pulled into core.

Traceback (most recent call last):
  File "...datasette/datasette/app.py", line 969, in route_path
    response = await view(request, send)
TypeError: favicon() missing 1 required positional argument: 'send'

Similar problem to #863 - calling datasette.add_message() in a view registered using the register_routes() plugin hook doesn't work, because the code that writes accumulated messages to the ds_messages signed cookie lives in the BaseView class here:

https://github.com/simonw/datasette/blob/28bb1c51897f3956861755e345e18b8e0b1423ac/datasette/views/base.py#L94-L97

The .coverage file generated by running pytest-cov is now a SQLite database!

I could do something interesting with this. Maybe after each test run for a new commit I could store that database file somewhere?

Lots of interesting challenges here.

I got a change into coveragepy last year which helps make the custom SQL functions available for doing fun things in Datasette: https://github.com/nedbat/coveragepy/issues/868

Bigger challenge: if I have a DB file for every commit, that's hundreds (potentially thousands) of DB files. Datasette isn't designed to handle thousands of files like that.

So, do I figure out how to have Datasette open a file on-command for just a single request? Or, an easier option, do I copy data from those files into a single database with a modified schema to include the commit hash in each table row?

(Following on from #841 and #844)

I like the shorter name better. Follows on from #842.

It might be useful to have an startup hook which gets passed the datasette object as soon as Datasette has finished initializing.

My initial use-case for this is configuration verification - checking that the "plugins" configuration block for this plugin contains valid details.

I imagine there are plenty of other potential uses for this as well.

Now that writable canned queries (#698) have landed, it would be neat if they supported "magic" parameters - parameters that are automatically populated with:

• the current actor ID / other actor properties
• the current date and time
• the user's IP or user-agent

And maybe other things potentially added by plugins.

Implementation for #842

TODO:

• Add tests for built-in magic parameters
• Magic parameters should not show up as blank form fields on the query page
• Update documentation for new _request_X (now called _header_X) implementation where X is a key from the ASGI scope
• Make sure these only work for canned queries, not for arbitrary SQL queries (security issue)
• Add test for the register_magic_parameters plugin hook
• Add documentation for the register_magic_parameters plugin hook

I was looking at this: https://github.com/simonw/datasette-block-robots/blob/main/datasette_block_robots/init.py

from datasette import hookimpl
from datasette.utils.asgi import Response


async def robots_txt():
    return Response.text("User-agent: *\nDisallow: /")


@hookimpl
def register_routes():
    return [
        (r"^/robots\.txt$", robots_txt),
    ]

And I realized that if register_routes() could support non-async view functions it could be reduced to this:

@hookimpl
def register_routes():
    return [
        (r"^/robots\.txt$", lambda: Response.text("User-agent: *\nDisallow: /")),
    ]

I was waiting for consensus to form around this (and kind-of hoping for trunk since I like the tree metaphor) and it looks like main is it.

I've seen convincing arguments against trunk too - it indicates that the branch has some special significance like in Subversion (where all branches come from trunk) when it doesn't. So main is better anyway.

Updates the requirements on pytest-asyncio to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Over on Apache Tika, we're using datasette to allow users to make sense of the metadata for our file regression testing corpus.

This could be user error in how I've set up the reverse proxy!

I started datasette like so:
docker run -d -p 8001:8001 -vpwd:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/corpora-metadata.db --config sql_time_limit_ms:60000 --config base_url:/datasette/

I then reverse proxied like so:

ProxyPreserveHost On
ProxyPass /datasette http://x.y.z.q:xxxx
ProxyPassReverse /datasette http://x.y.z.q:xxx

Regular sql works perfectly:
https://corpora.tika.apache.org/datasette/corpora-metadata?sql=select+mime_string%2C+count%281%29+as+cnt%0D%0Afrom+profiles+p%0D%0Ajoin+mimes+m+on+p.mime_id%3Dm.mime_id%0D%0Agroup+by+mime_string%0D%0Aorder+by+cnt+desc

However, adding criteria and clicking 'Apply'
https://corpora.tika.apache.org/datasette/corpora-metadata/tika_1_24_1_mimes?_sort=file&mime__exact=text%2Fplain

bounces back to:
https://corpora.tika.apache.org/corpora-metadata/tika_1_24_1_mimes?_sort=file&file__contains=bug&mime__exact=text%2Fplain

When executing "datasette publish heroku schools.db" on Windows 10, I get the following error

  File "c:\users\dell\.virtualenvs\sec-schools-jn-cwk8z\lib\site-packages\datasette\publish\heroku.py", line 54, in heroku
    line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines()
  File "c:\python38\lib\subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "c:\python38\lib\subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "c:\python38\lib\subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "c:\python38\lib\subprocess.py", line 1307, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

Changing https://github.com/simonw/datasette/blob/55a6ffb93c57680e71a070416baae1129a0243b8/datasette/publish/heroku.py#L54

to

line.split()[0] for line in check_output(["heroku", "plugins"], shell=True).splitlines()

as well as the other check_output() and call() within the same file leads me to another recursive error about temp files

I'm running datasette serve --config base_url:/foo/ …, proxying to it with this Apache config:

    ProxyPass /foo/ http://localhost:8001/ 
    ProxyPassReverse /foo/ http://localhost:8001/

and then accessing it via https://example.com/foo/.

Although many of the URLs in the pages are correct (presumably because they either use absolute paths which include base_url or relative paths), the faceting and pagination links still use fully-qualified URLs pointing at http://localhost:8001.

I looked into this a little in the source code, and it seems to be an issue anywhere request.url or request.path is used, as these contain the values for the request between the frontend (Apache) and backend (Datasette) server. Those properties are primarily used via the path_with_… family of utility functions and the Datasette.absolute_url method.

https://twitter.com/amjithr/status/1273496759684050944

The documentation here suggests that it will work:

https://github.com/simonw/datasette/blob/eed116ac0599c7d21b7129af94d58ce03a923e4e/docs/internals.rst#L540-L546

But right now the csrftoken variable is set in BaseView.render, which means it's not visible to plugins that try to render templates using datasette.render_template:

https://github.com/simonw/datasette/blob/799c5d53570d773203527f19530cf772dc2eeb24/datasette/views/base.py#L99-L106

Got this error deploying large (603MB) database with Cloud Run

X Deploying... Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revi
sion might contain more information.                                                                                                                               
  X Creating Revision... Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for
   this revision might contain more information.                                                                                                                   
  . Routing traffic...                                                                                                                                             
  ✓ Setting IAM Policy...                                                                                                                                          
Deployment failed                                                                                                                                                  
ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.

If a table has hundreds of columns the Datasette UI starts getting unwieldy.

Addressing this would be neat. One option would be to only select the first 30 columns by default and provide a UI for selecting more.

If a table has 100 columns the facet suggestion code will currently run 100 times, taking a max of facet_suggest_time_limit_ms which defaults to 50ms per column:

https://github.com/simonw/datasette/blob/000528192eaf891118932250141dabe7a1561ece/datasette/facets.py#L142-L162

So for 100 columns, that's 100 * 50ms = 5s total time that might be spent attempting to calculate facets on a large table!

I should implement a hard upper limit on the total amount of time taken suggesting facets - probably of around 500ms. If it takes longer than that the remaining columns will not be considered.

I'll write a little script which generates a 300MB SQLite file with a bunch of tables with lots of randomly generated rows in to help test this.

Having a tool like that which can generate larger databases with different gnarly performance characteristics will be useful for other performance work too.
Originally posted by @simonw in https://github.com/simonw/datasette/issues/859#issuecomment-647189948

I think the plugins docs need to extend beyond a single page now. I want to add a whole section on writing tests for plugins, showing how httpx can be used as seen in https://github.com/simonw/datasette-atom/issues/3 and suchlike.

Once I ship the datasette-plugin template: https://github.com/simonw/datasette-plugin/issues/1

I'm not happy with how metadata.(json|yaml) keeps growing new features. Rather than having a single plugin hook for all of metadata.json I'm going to split out the feature that shows actual real metadata for tables and databases - source, license etc - into its own plugin-powered mechanism.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/357#issuecomment-647189045

For https://github.com/simonw/russian-ira-facebook-ads-datasette/tree/af6d956995e14afd585c35a6a06bb01da32043ba I wrote a script to convert YAML to JSON because YAML is a better format for embedding multi-line HTML descriptions and canned SQL statements.

Example yaml metadata file: https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/af6d956995e14afd585c35a6a06bb01da32043ba/russian-ads-metadata.yaml

It would be useful if Datasette could be fed a YAML file directly:

datasette -m metadata.yaml

Question is... should this be a native feature (hence adding a YAML dependency) or should it be handled by a datasette-metadata-yaml plugin, using a new plugin hook for loading metadata? If so, what would other use-cases for that plugin hook be?

See this conversation: https://twitter.com/psychemedia/status/1199707352540368896

Needed for the Datasette 1.0 release, so template authors can trust that Datasette is unlikely to break their templates.

The new canned_queries() plugin hook from #852 combined with plugins like https://github.com/simonw/datasette-saved-queries could mean that some installations end up with hundreds or even thousands of canned queries. I should consider pagination or some other way of ensuring that this doesn't cause performance problems for Datasette.

Canned queries are currently baked into metadata.json which is read once on startup.

Allowing users to interactively create new canned queries - even if just through a plugin - would make a lot of sense.

Is this a new plugin hook or some other mechanism? Lots to think about here.

I'd like to be able to ship alphas and betas to PyPI so in-development plugins can depend on them and help test unreleased plugin hooks.

ASGI wrapper plugins that themselves set the actor scope variable should be respected (though actor_from_request plugins should still execute and get the chance to replace that initial actor value).

Relevant code: https://github.com/simonw/datasette/blob/09a3479a5402df96489ed6cab6cc9fd674bf3433/datasette/app.py#L910-L921

Thought of this while thinking about possible permissions plugins (#818).

Imagine an API key plugin which allows access for API keys. It could let users register new API keys by providing a writable canned query for writing to the api_keys table.

To do this the plugin needs to register the query. At the moment queries have to be registered in metadata.json - a plugin hook for registering additional queries could help solve this.

One challenge: how does the plugin know which named database the query should be registered for?

It could default to the first attached database and allow users to optionally tell the plugin "actually use this named database instead" in plugin configuration.

Hey,

I'm trying to get canned inserts to work. I have an DB with following metadata:

sqlite> .mode line

sqlite> select name, sql from sqlite_master where name like '%search%';
 name = search
  sql = CREATE TABLE "search" ("id" INTEGER NOT NULL PRIMARY KEY, "name" VARCHAR(255) NOT NULL, "url" VARCHAR(255) NOT NULL)

# ...
queries:
    add_search:
        sql: insert into search(name, url) VALUES (:name, :url),
        write: true

which renders a form as expected, but when I submit the form I get incomplete input error.

but when submit post the form

I've attached a debugger to see where the error comes from, because incomplete input string doesn't appear in datasette codebase.

Inside datasette.database.Database.execute_write_fn

https://github.com/simonw/datasette/blob/4fa7cf68536628344356d3ef8c92c25c249067a0/datasette/database.py#L69

result = await reply_queue.async_q.get()

this line raises an exception.

That led me to believe I had something wrong with my SQL. But running the command in sqlite3 inserts the record just fine.

sqlite> insert into search (name, url) values ('my name', 'my url');
sqlite> SELECT last_insert_rowid();
last_insert_rowid() = 3

So I'm a bit lost here.

• datasette, version 0.44
• Python 3.8.1

https://aws.amazon.com/about-aws/whats-new/2020/06/aws-lambda-support-for-amazon-elastic-file-system-now-generally-/

If Datasette can run on Lambda with access to EFS it could both read AND write large databases there.

Refs #217 - create a publish plugin that can deploy to AWS Lambda.

https://docs.aws.amazon.com/lambda/latest/dg/limits.html says lambda packages can be up to 50 MB, so this would only work with smaller databases (the command can check the filesize before attempting to package and deploy it).

Lambdas do get a 512 MB /tmp directory too, so for larger databases the function could start and then download up to 512MB from an S3 bucket - so the plugin could take an optional S3 bucket to write to and know how to upload the .db file there and then have the lambda download it on startup.

I got this on my laptop:

...
/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.7/site-packages/jinja2/loaders.py:171: in get_source
    f = open_if_exists(filename)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

filename = '/Users/simon/Dropbox/Development/datasette/datasette/templates/400.html', mode = 'rb'

    def open_if_exists(filename, mode='rb'):
        """Returns a file descriptor for the filename if that file exists,
        otherwise `None`.
        """
        try:
>           return open(filename, mode)
E           OSError: [Errno 24] Too many open files: '/Users/simon/Dropbox/Development/datasette/datasette/templates/400.html'

/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.7/site-packages/jinja2/utils.py:154: OSError

Based on the conversation in https://github.com/pytest-dev/pytest/issues/2970 I'm worried that my tests are opening too many files without closing them.

In particular... I call sqlite3.connect(filepath) a LOT - and I don't ever call conn.close() on those opened connections:

https://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/datasette/database.py#L58-L60

Could this be resulting in my tests eventually opening too many unclosed file handles? How could I confirm this?

I'm not sure why this is, but the code coverage I have running in a GitHub Action doesn't take my .coveragerc file into account. It should:

https://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/.github/workflows/test-coverage.yml#L31-L35

Here's the bit that's ignored:

https://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/.coveragerc#L1-L2

As a result my coverage score is 84%, when it should be 92%:

2020-06-13T21:41:18.4404252Z ----------- coverage: platform linux, python 3.8.3-final-0 -----------
2020-06-13T21:41:18.4404570Z Name                                 Stmts   Miss  Cover
2020-06-13T21:41:18.4404971Z --------------------------------------------------------
2020-06-13T21:41:18.4405227Z datasette/__init__.py                    3      0   100%
2020-06-13T21:41:18.4405441Z datasette/__main__.py                    3      3     0%
2020-06-13T21:41:18.4405668Z datasette/_version.py                  279    279     0%
2020-06-13T21:41:18.4405921Z datasette/actor_auth_cookie.py          20      0   100%
2020-06-13T21:41:18.4406135Z datasette/app.py                       499     27    95%
2020-06-13T21:41:18.4406343Z datasette/cli.py                       162     45    72%
2020-06-13T21:41:18.4406553Z datasette/database.py                  236     17    93%
2020-06-13T21:41:18.4406761Z datasette/default_permissions.py        40      0   100%
2020-06-13T21:41:18.4406975Z datasette/facets.py                    210     24    89%
2020-06-13T21:41:18.4407186Z datasette/filters.py                   122      7    94%
2020-06-13T21:41:18.4407394Z datasette/hookspecs.py                  34      0   100%
2020-06-13T21:41:18.4407600Z datasette/inspect.py                    36     23    36%
2020-06-13T21:41:18.4407807Z datasette/plugins.py                    34      6    82%
2020-06-13T21:41:18.4408014Z datasette/publish/__init__.py            0      0   100%
2020-06-13T21:41:18.4408240Z datasette/publish/cloudrun.py           57      2    96%
2020-06-13T21:41:18.4408786Z datasette/publish/common.py             19      1    95%
2020-06-13T21:41:18.4409029Z datasette/publish/heroku.py             97     13    87%
2020-06-13T21:41:18.4409243Z datasette/renderer.py                   63      4    94%
2020-06-13T21:41:18.4409450Z datasette/sql_functions.py               5      0   100%
2020-06-13T21:41:18.4410480Z datasette/tracer.py                     87     16    82%
2020-06-13T21:41:18.4410972Z datasette/utils/__init__.py            504     31    94%
2020-06-13T21:41:18.4411755Z datasette/utils/asgi.py                264     24    91%
2020-06-13T21:41:18.4412173Z datasette/utils/shutil_backport.py      44     44     0%
2020-06-13T21:41:18.4412822Z datasette/version.py                     4      0   100%
2020-06-13T21:41:18.4413562Z datasette/views/__init__.py              0      0   100%
2020-06-13T21:41:18.4414276Z datasette/views/base.py                288     19    93%
2020-06-13T21:41:18.4414579Z datasette/views/database.py            120      2    98%
2020-06-13T21:41:18.4414860Z datasette/views/index.py                57      2    96%
2020-06-13T21:41:18.4415379Z datasette/views/special.py              72     16    78%
2020-06-13T21:41:18.4418994Z datasette/views/table.py               418     18    96%
2020-06-13T21:41:18.4428811Z --------------------------------------------------------
2020-06-13T21:41:18.4430394Z TOTAL                                 3777    623    84%

Inspired by https://twitter.com/mikeal/status/1271473021593636866

Almost every library I’ve written in the last 2 years has had 100% coverage and that’s probably not going to change in the future. It’s not that hard to start at 100% and hold onto it and the workflow it enables is so much nicer.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/841#issuecomment-643660757

Refs #843

See also milestone. This is a pretty big release: flash messaging, writable canned queries, authentication and permissions!

I'll want to ship some plugin releases in conjunction with this - datasette-auth-github for example.

https://travis-ci.org/github/simonw/datasette/jobs/697445318

    def test_plugin_config_file(app_client):
        open(TEMP_PLUGIN_SECRET_FILE, "w").write("FROM_FILE")
>       assert {"foo": "FROM_FILE"} == app_client.ds.plugin_config("file-plugin")
E       AssertionError: assert {'foo': 'FROM_FILE'} == {'foo': {'$fi...ugin-secret'}}
E         Differing items:
E         {'foo': 'FROM_FILE'} != {'foo': {'$file': '/tmp/plugin-secret'}}
E         Use -v to get the full diff

Broken in https://github.com/simonw/datasette/commit/fba8ff6e76253af2b03749ed8dd6e28985a7fb8f as part of #837

This didn't work:

{
    "plugins": {
        "datasette-auth-tokens": [
            {
                "token": {
                    "$env": "BOT_TOKEN"
                },
                "actor": {
                    "bot_id": "my-bot"
                }
            }
        ]
    }
}

https://github.com/simonw/datasette-auth-github/issues/62 will work for this.

actor: {"id": "root"}

allow block: {"bot_id": "my-bot", "id": ["root"]}

This should pass, because the id matches - but it fails.

The only URLs that should be available without authentication at all times are the /-/static/ prefix, to allow for HTTP caching.

Suggested by @andrewhayward on Twitter: https://twitter.com/arhayward/status/986015118965268480?s=21

Have you considered a basic abstraction on top of that, for standard hook features?

@sql_function
random_integer(a,b):
    return random.randint(a,b)

@template_filter
uppercase(str):
    return str.upper()

Maybe from datasette.plugins import template_filter?

Would have to work out how to get this to play well with pluggy

Idea: an "allow_sql" key with a SQL query that gets passed the actor JSON as :actor and can extract the relevant keys from it and return 1 or 0.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/698#issuecomment-639787304

See also #800

Refs #685

Still TODO:

• Implement results.first()
• Implement results.single_value()
• Unit tests for the above

Refs #787. Will need quite a bit of manual testing since this involves code which runs against Heroku and Cloud Run.

To allow signed cookies etc to work reliably (see #785) all of the datasette publish commands should generate a random secret on publish and bake it into the configuration - probably by setting the DATASETTE_SECRET environment variable.

• Cloud Run
• Heroku
• https://github.com/simonw/datasette-publish-now
• https://github.com/simonw/datasette-publish-fly

I've identified a couple of use-cases for running Datasette in a way that over-rides the default way that internal URLs are generated.

1. Running behind a reverse proxy. I tried running Datasette behind a proxy and found that some of the generated internal links incorrectly referenced http://127.0.0.1:8001/fixtures/... - when they should have been referencing http://my-host.my-domain.com/fixtures/... - this is a problem both for links within the HTML interface but also for the toggle_url keys returned in the JSON as part of the facets datastructure.
2. I would like it to be possible to host a Datasette instance at e.g. https://www.mynewspaper.com/interactives/2018/election-results/ - either through careful HTTP proxying or, once Datasette has been ported to ASGI, by mounting a Datasette ASGI instance deep within an existing set of URL routes.

I'm going to add a url_prefix configuration option. This will default to "", which means Datasette will behave as it does at the moment - it will use / for most URL prefixes in the HTML version, and an absolute URL derived from the incoming Host header for URLs that are returned as part of the JSON output.

If url_prefix is set to another value (either a full URL or a path) then this path will be appended to all generated URLs.

To show how they work. Also useful to confirm how they interact with the default permissions.

Now that I'm starting to write alternative plugins to control permissions - see #818 - I think I need an easy way to tell Datasette "no-one has permission to do X unless a plugin says otherwise".

One relatively intuitive way to do that could be like this:

{
  "databases": {
    "fixtures": {
      "allow": null
    }
  }
}

Right now I think that opens up permissions to everyone, which isn't as obvious.

Nothing uses this plugin hook yet, so the design is not yet proven.

I'm going to build a real plugin against it and use that process to inform any design changes that may need to be made.

I'll add a warning about this to the documentation.

I need this for datasette-auth-github: https://github.com/simonw/datasette-auth-github/issues/62#issuecomment-642152076

You can scroll sideways on that page and it looks bad:

The cause is these long links:

Mainly to clean up this code:

https://github.com/simonw/datasette/blob/4fa7cf68536628344356d3ef8c92c25c249067a0/datasette/app.py#L439-L454

Most authentication plugins (https://github.com/simonw/datasette-auth-github for example) are likely to work by setting the ds_actor signed cookie, which is already magically decoded and supported by default Datasette here:

https://github.com/simonw/datasette/blob/4fa7cf68536628344356d3ef8c92c25c249067a0/datasette/actor_auth_cookie.py#L1-L13

I should document this.

This command is meant to write out fixtures.db, metadata.json and a plugins directory:

$ python tests/fixtures.py /tmp/fixtures.db /tmp/metadata.json /tmp/plugins/
Test tables written to /tmp/fixtures.db
- metadata written to /tmp/metadata.json
Traceback (most recent call last):
  File "tests/fixtures.py", line 833, in <module>
    ("my_plugin.py", PLUGIN1),
NameError: name 'PLUGIN1' is not defined

Plugin authors need to know that if they want to POST a form they should include this:

<input type="hidden" name="csrftoken" value="{{ csrftoken() }}">

I'd like plugins to be able to ship with a default that says "anonymous users cannot do this", but allow site administrators to over-ride that such that anonymous users can use the feature after all.

This is tricky because right now the anonymous user doesn't have an actor dictionary at all, so there's no key to match to an allow block.

Docs at https://github.com/simonw/datasette/blob/5a6a73e3190cac103906b479d56129413e5ef190/docs/authentication.rst#actors say:

The only required field in an actor is "id", which must be a string.

But the example here returns {"token": token}:

@hookimpl
def actor_from_request(datasette, request):
    async def inner():
        token = request.args.get("_token")
        if not token:
            return None
        # Look up ?_token=xxx in sessions table
        result = await datasette.get_database().execute(
            "select count(*) from sessions where token = ?", [token]
        )
        if result.first()[0]:
            return {"token": token}
        else:
            return None

    return inner

Added here: https://github.com/simonw/datasette/commit/aa82d0370463580f2cb10d9617f1bcbe45cc994a#diff-5e0ffd62fced7d46339b9b2cd167c2f9R236

This example: https://github.com/simonw/datasette/blob/f5e79adf26d0daa3831e3fba022f1b749a9efdee/docs/plugins.rst#register_routes

from datasette.utils.asgi import Response
import html


async def hello_from(scope):
    name = scope["url_route"]["kwargs"]["name"]
    return Response.html("Hello from {}".format(
        html.escape(name)
    ))


@hookimpl
def register_routes():
    return [
        (r"^/hello-from/(?P<name>.*)$"), hello_from)
    ]

Would be nicer if you could easily get scope["url_route"]["kwargs"]["name"] directly from the request object, without looking at the scope.

I'll need to add documentation of the Response object (and Response.html() and Response.text() class methods - I should add Response.json() too) to the internals page https://datasette.readthedocs.io/en/stable/internals.html

Originally posted by @simonw in https://github.com/simonw/datasette/issues/215#issuecomment-640971470

Might be as simple as having plugins get passed the app after the other routes have been defined:

https://github.com/simonw/datasette/blob/b2955d9065ea019500c7d072bcd9d49d1967f051/datasette/app.py#L1270-L1274

Refs #14

Refs #215

e.g. https://latest.datasette.io/fixtures/compound_three_primary_keys.tsv?_size=max

The HTML starts like this:

<!DOCTYPE html>
<html>
<head>
    <title>Error 404</title>
    <link rel="stylesheet" href="-/static/app.css?">