I need this for datasette-graphql for full compatibility with the way Relay likes to paginate - using cursors for paginating backwards as well as for paginating forwards.

This may be the kick I need to get Datasette pagination to work in reverse too.
_Originally posted by @simonw in https://github.com/simonw/datasette-graphql/issues/2#issuecomment-668305853_

_Originally posted by @simonw in https://github.com/simonw/datasette-graphql/issues/2#issuecomment-667780040_

FTS5 supports things like porter stemming using a tokenize= option:

https://www.sqlite.org/fts5.html#tokenizers

Something like this in code:

            CREATE VIRTUAL TABLE [{table}_fts] USING {fts_version} (
                {columns},
                tokenize='porter',
                content=[{table}]
            );

I tried this out just now and it worked exactly as expected.

So... db[table].enable_fts(...) should accept a 'tokenize= argument, and sqlite-utils enable-fts ... should support a --tokenize option.

https://latest.datasette.io/fixtures/binary_data.json?_sort_desc=data&_shape=array returns this:

[
  {
    "rowid": 1,
    "data": "this is binary data"
  }
]

But adding &_nl=on returns this: https://latest.datasette.io/fixtures/binary_data.json?_sort_desc=data&_shape=array&_nl=on

{
  "ok": false,
  "error": "Object of type bytes is not JSON serializable",
  "status": 500,
  "title": null
}

I found this error by running wget -r 127.0.0.1:8001 against my local fixtures.db.

Hi,

I've been playing with Datasette behind Nginx (awesome tool, thanks !). It seems some URLs are OK but some aren't. For instance in https://github.com/simonw/datasette/blob/master/datasette/templates/query.html#L61 it seems that url_csv includes a / prefix, resulting in the base_url not beeing honored.

Actually here, it seems that dropping the prefix / to make the link relative is enough (so it may not be strictly related to base_url).

Additional information:

datasette, version 0.45+0.gf1f581b.dirty

Relevant Nginx configuration (note that all the trailing slashes have some effect):

  location /datasette/ {
    proxy_pass http://127.0.0.1:9001/;
    proxy_set_header Host $host;
  }

Relelvant Datasette configuration (slashes matter too):

  --config base_url:/datasette/

Hello,

Until now I'm using datasette without any authentication system but I would like to setup a configuration or limiting the number of requests per user (eventually by IP or with a cookie mechanism) and eventually allowing me to ban specific users/IPs.

Is there a plugin available for this use case ?
If not what are your insights regarding this UC ?

Should I write a plugin ? Should I deploy datasette behind a reverse proxy to manage this ?

It's a shame there's no obvious mechanism for passing additional options to datasette my.db that affect how plugins work.

The only way I can think of at the moment is via environment variables:

DATASETTE_INSERT_UNSAFE=1 datasette my.db

This will have to do for the moment - it's ugly enough that people will at least know they are doing something unsafe, which is the goal here.
_Originally posted by @simonw in https://github.com/simonw/datasette-insert/issues/15#issuecomment-667346438_

I was waiting for consensus to form around this (and kind-of hoping for trunk since I like the tree metaphor) and it looks like main is it.

I've seen convincing arguments against trunk too - it indicates that the branch has some special significance like in Subversion (where all branches come from trunk) when it doesn't. So main is better anyway.

When passing a mutable value as a default argument in a function, the default argument is mutated anytime that value is mutated. This poses a bug risk. Instead, use None as a default and assign the mutable value inside the function.

https://datasette.readthedocs.io/en/0.45/publish.html#datasette-publish currently only lists Cloud Run, Heroku and Fly. It should list Vercel too.

(I should probably rename datasette-publish-now to datasette-publish-vercel)

To help out with escaping - so you can run this:

sqlite-utils query "insert into foo (blah) values (:blah)" --param blah `something here`

A --sqlar option could cause insert-files to behave in the same way as SQLite's own sqlar mechanism.

https://www.sqlite.org/sqlar.html and https://sqlite.org/sqlar/doc/trunk/README.md

Inserting files by piping them in should work - but since a filename cannot be derived this will need a --name blah.gif option.

cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071_

psycopg2 can return data from PostgreSQL as uuid.UUID or memoryview objects. These should to be supported by sqlite-utils - mainly for https://github.com/simonw/db-to-sqlite

https://github.com/simonw/datasette/blob/3c33b421320c0be81a625ca7307b2e4416a9ed5b/datasette/utils/asgi.py#L396-L405
self.filename should be passed to asgi_send_file()

Updates the requirements on pytest to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

--name works inconsistently across the different publish providers - on Cloud Run you should use --service instead for example. Need to review it across all of them and either remove it or clarify what it does.

SQLite BLOB columns can store entire binary files. The challenge is inserting them, since they don't neatly fit into JSON objects.

It would be great if the sqlite-utils CLI had a trick for helping with this.

Inspired by https://github.com/simonw/datasette-media/issues/14

SQLite's ALTER TABLE can only do the following:

• Rename a table
• Rename a column
• Add a column

Notably, it cannot drop columns - so tricks like "add a float version of this text column, populate it, then drop the old one and rename" won't work.

The docs here https://www.sqlite.org/lang_altertable.html describe a way of implementing full alters safely within a transaction, but it's fiddly.

1. Create new table
2. Copy data
3. Drop old table
4. Rename new into old

It would be great if sqlite-utils provided an abstraction to help make these kinds of changes safely.

I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says "these columns are base64 encoded, store the decoded data in a BLOB".

That would solve inserting binary data using JSON too.
_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247_

You get an error if you try to run a query that returns data from a BLOB.

Related to the insert-files work in #122 - it should be easy to get binary data back out of the database again.

One way to do that could be:

sqlite-utils files.db "select content from files where key = 'foo.jpg'" --raw

The --raw option would cause just the contents of the first column to be output directly to stdout.

Picking up the work done on #557 with a new PR. Seeing if I can get this working.

The "allow" syntax described at https://datasette.readthedocs.io/en/0.45/authentication.html#defining-permissions-with-allow-blocks currently says this:

An allow block can specify "no-one is allowed to do this" using an empty {}:

{ "allow": {} }

"allow": null allows all access, though this isn't documented (it should be though).

These are not very intuitive. How about also supporting "allow": true for "allow anyone" and "allow": false for "allow nobody"?

It might be good to have a little interactive tool which helps debug these things, since there are quite a few edge-cases and the damage caused if people use them incorrectly is substantial.
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/907#issuecomment-663726146_

Documentation here: https://datasette.readthedocs.io/en/0.45/authentication.html#defining-permissions-with-allow-blocks

Doesn't explain that with the following "allow" block:

{
  "allow": {
    "id": "simonw",
    "role": "staff"
  }
}

The rule will match if EITHER the id is simonw OR the role includes staff.

The tests are missing this case too: https://github.com/simonw/datasette/blob/028f193dd6233fa116262ab4b07b13df7dcec9be/tests/test_utils.py#L504

Related to #906

Because setup.py uses find_packages and tests is on the top-level, pip install datasette will install a top-level package called tests, which is probably not desired behavior.

The offending line is here:
https://github.com/simonw/datasette/blob/bfa2ae0d16d39bb82dbe4da4f3fdc3c7f6257418/setup.py#L40

And only pip uninstall datasette with a conflicting package would warn you by default; apparently another package had the same problem, which is why I get this message when uninstalling:

$ pip uninstall datasette
Uninstalling datasette-0.27:
  Would remove:
    /usr/local/bin/datasette
    /usr/local/lib/python3.7/site-packages/datasette-0.27.dist-info/*
    /usr/local/lib/python3.7/site-packages/datasette/*
    /usr/local/lib/python3.7/site-packages/tests/*
  Would not remove (might be manually added):
    [ .. snip .. ]
Proceed (y/n)? 

This should be a relatively simple fix, and I could drop a PR if desired!

Cheers

The exclude argument to find_packages needs an iterable of package
names.

Fixes: #456

Just a quick note... The 23andme data is not exactly your genome, but a SNP chip of your genome. It's "some of your genotypes." Or about 0.1% of your genome. Nice work in any case! It deserves to be liberated!!!!!

Using https://github.com/jonschlinkert/markdown-toc

Hi there! I saw your presentation at Boston Python. I'm already a light user of Datasette (thank you!), but wasn't aware of this project.

I've been working on a "pull request dashboard" to get a comprehensive view of the state of open PR's, esp. related to reviews (i.e., pending, approved, changes requested). Currently it's a CLI command, but I thought a Datasette UI might be fun.

I see that PR's are available from the issues command, but I don't see reviews anywhere. From the API docs, it looks like there are separate endpoints for those (as well as pull requests in general). What do you think about adding that? Would you accept a PR? Any sense of the level of effort?

I can do this by modifying this function: https://github.com/simonw/datasette/blob/02dc6298bdbfb1d63e0d2a39ff597b5fcc60e06b/datasette/utils/asgi.py#L248-L270

I tried using this block of template in a plugin and got an error:

{% block nav %}
    <p class="crumbs">
        <a href="{{ base_url }}">home</a> /
        <a href="{{ database_url(database) }}">{{ database }}</a> /
        <a href="{{ database_url(database) }}/{{ table|quote_plus }}">{{ table }}</a>
    </p>
    {{ super() }}
{% endblock %}

Error: 'database_url' is undefined

That's because database_url is only made available by the BaseView template here:

https://github.com/simonw/datasette/blob/d6e03b04302a0852e7133dc030eab50177c37be7/datasette/views/base.py#L110-L125

https://github.com/simonw/til/blob/master/pytest/registering-plugins-in-tests.md

Would be useful to include this pattern on https://datasette.readthedocs.io/en/stable/testing_plugins.html

This PR fixes #456 by adding tests to the package exclusion list.

Cheers

Fetches paginated data from https://api.github.com/repos/simonw/datasette/tags

https://github.com/dogsheep/github-to-sqlite/runs/885773677

  File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 476, in save_tags
    db["tags"].insert_all(
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/sqlite_utils/db.py", line 1145, in insert_all
    result = self.db.conn.execute(query, params)
sqlite3.OperationalError: table tags has no column named repo

That's because I changed the name in #44. I thought this would be safe since no-one else could possibly be using this yet (it hadn't shipped in a release) but turns out I broke my demo!

For improved consistency with other tables. https://observablehq.com/@simonw/datasette-table-diagram

The unit tests for plugins could benefit from reusing code from Datasette's own testing fixtures, e.g.:

I may need to borrow this function from Datasette for the tests:
https://github.com/simonw/datasette/blob/1f6a134369e6a7efaae9db469f15b1dd2b7f3709/tests/fixtures.py#L836-L851

It's not importable (it lives in fixtures.py and not in the datasette package that gets packaged for PyPI) - maybe I should fix that in Datasette by adding a from datasette.utils.testing module.
_Originally posted by @simonw in https://github.com/simonw/datasette-update-api/issues/4#issuecomment-660419182_

I'm writing a plugin (https://github.com/simonw/datasette-update-api/issues/2) that implements an API for inserting JSON data. As such, I'd like to POST a JSON blob rather than using key=value form encoded data.

Right now there's a request.post_vars() method but no request.post_body() one:

https://github.com/simonw/datasette/blob/c5f06bc356fb5917ef7fbb6fe4693f30d711cdb3/datasette/utils/asgi.py#L93-L103

Is there any reason NOT to apply white-space: pre-wrap to the contents of all table cells in Datasette?

The default display mechanism of HTML (stripping leading/trailing slashes and collapsing all other whitespace) doesn't really make sense for displaying the kind of data that Datasette works with.

Compare https://latest.datasette.io/fixtures/sortable with https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+order+by+pk1%2C+pk2+limit+101

https://user-images.githubusercontent.com/9599/87612845-82e09c00-c6c0-11ea-806e-93764ca468c4.png">

If a text column actually contains numbers, being able to "sort by column, treated as numeric" would be really useful.

Probably depends on column actions enabled by #690

This error wasn't thrown

Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/opt/python@3.8/lib/python3.8/site-packages/datasette/utils/asgi.py", line 289, in inner_static
    full_path.relative_to(root_path)
  File "/home/linuxbrew/.linuxbrew/opt/python@3.8/lib/python3.8/pathlib.py", line 904, in relative_to
    raise ValueError("{!r} does not start with {!r}"
ValueError: '/home/linuxbrew/.linuxbrew/lib/python3.8/site-packages/datasette/static/app.css' does not start with '/home/linuxbrew/.linuxbrew/opt/python@3.8/lib/python3.8/site-packages/datasette/static'

Linuxbrew install python@3.8 with symbolic links when You call the full_path.relative_to(root_path) throw ValueError. This happened when you install from pip3

when you install with python3 setup.py develop , works good.

Well at the end the static wasn't serving.

Finally got around to trying dogsheep-photos but I want to use backblaze's b2 service instead of AWS S3.
Had to add a way to optionally specify the endpoint_url to connect to. Then with the bucket being empty the initial key retrieval would fail. Probably a better way to see that the bucket is empty than doing a test inside the paginator loop.

Also probably a better way to specify the endpoint_url as we get and test for it twice using the same code in two different places but did not want to spend too much time worrying about it.

On https://datasette.readthedocs.io/en/latest/

https://user-images.githubusercontent.com/9599/87255376-1b67e980-c43f-11ea-8470-781ee8842336.png">

Compare with https://datasette.readthedocs.io/en/0.45/

https://user-images.githubusercontent.com/9599/87255387-2fabe680-c43f-11ea-875e-c474364691ca.png">

Some custom CSS should fix it.

For if you know which specific repos you care about, as opposed to loading everything owned by the authenticated user.

github-to-sqlite repos specific.db -r simonw/datasette -r simonw/github-contents

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393_

We should put some thought into how this library supports and encourages smart use of transactions.

From https://docs.python.org/3/library/sqlite3.html#sqlite3.enable_callback_tracebacks

sqlite3.``enable_callback_tracebacks(flag)¶

By default you will not get any tracebacks in user-defined functions, aggregates, converters, authorizer callbacks etc. If you want to debug them, you can call this function with flag set to True. Afterwards, you will get tracebacks from callbacks on sys.stderr. Use False to disable the feature again.

Maybe turn this on for all of Datasette? Are there any disadvantages to doing that?

Useful if you add one but make a mistake and need to undo it without recreating the database from scratch.

Deletes all rows in the table (if it exists) before inserting new rows.
SQLite doesn't implement a TRUNCATE TABLE statement but does optimize an
unqualified DELETE FROM.

This can be handy if you want to refresh the entire contents of a table
but a) don't have a PK (so can't use --replace), b) don't want the table
to disappear (even briefly) for other connections, and c) have to handle
records that used to exist being deleted.

Ideally the replacement of rows would appear instantaneous to other
connections by putting the DELETE + INSERT in a transaction, but this is
very difficult without breaking other code as the current transaction
handling is inconsistent and non-systematic. There exists the
possibility for the DELETE to succeed but the INSERT to fail, leaving an
empty table. This is not much worse, however, than the current
possibility of one chunked INSERT succeeding and being committed while
the next chunked INSERT fails, leaving a partially complete operation.

See commit messages for details. I ran into this while investigating another feature/issue.

I'm going to draw the line here: default Datasette supports authentication but only for a single user account ("admin"). Plugins can then add support for multiple user accounts, social auth, SSO etc.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/699#issuecomment-636498770_

https://github.com/dogsheep/github-to-sqlite/runs/837714622?check_suite_focus=true

Creating Revision.........................................................................................................................................failed
Deployment failed
ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette", line 8, in <module>
    sys.exit(cli())
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/cloudrun.py", line 138, in cloudrun
    check_call(
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'gcloud run deploy --allow-unauthenticated --platform=managed --image gcr.io/datasette-222320/datasette github-to-sqlite' returned non-zero exit status 1.
##[error]Process completed with exit code 1.

The current behavior for --plugins-dir is to load every file in that folder as a python module. This can result in errors if there are non-python files in the plugins dir (such as .mypy_cache).

This PR restricts the module loading to only python files.

A reference implementation for adding support to reload when datasette is in the config_dir mode.

This implementation is flawed since it is watching the entire directory and any changes to the database will reload the server and adding unrelated files to the directory will also reload the server.

Now that we have support for plugins that can write I'm seeing all sorts of places where a plugin might need to add UI to the table page.

Some examples:

• datasette-configure-fts needs to add a "configure search for this table" link
• a plugin that lets you render or delete tables needs to add a link or button somewhere
• existing plugins like datasette-vega and datasette-cluster-map already do this with JavaScript

The challenge here is that multiple plugins may want to do this, so simply overriding templates and populating names blocks doesn't entirely work as templates may override each other.

The default JSON just isn't right. I find myself using ?_shape=array for almost everything I build against the API.

Steps to reproduce:

1. Install datasette-media plugin
   pip install datasette-media
2. Launch datasette
   datasette databasename.db
3. Error

INFO:     Started server process [927704]
INFO:     Waiting for application startup.
ERROR:    Exception in 'lifespan' protocol
Traceback (most recent call last):
  File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/uvicorn/lifespan/on.py", line 48, in main
    await app(scope, self.receive, self.send)
  File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/datasette_media/__init__.py", line 9, in wrapped_app
    path = scope["path"]
KeyError: 'path'
ERROR:    Application startup failed. Exiting.

Just a quick heads up:

Release notes for 0.45 include urls that point to localhost.

https://github.com/simonw/datasette/releases/tag/0.45

I had to build a custom _actorornull prefix for datasette-saved-queries:

def actorornull(key, request):
    if request.actor is None:
        return None
    return request.actor.get(key)


@hookimpl
def register_magic_parameters():
    return [
        ("actorornull", actorornull),
    ]

Maybe the actor magic in Datasette core should do that out of the box?

https://github.com/simonw/datasette/blob/f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb/datasette/default_magic_parameters.py#L14-L17

This page here - the URL is http://127.0.0.1:8001/data/all_tables but "all_tables" is not shown in the UI:

https://user-images.githubusercontent.com/9599/86302528-d978b100-bbbd-11ea-9a07-7b5d7d3de321.png">

Potential fix for https://github.com/simonw/datasette/issues/859.

Disabling table counts for hidden tables speeds up database page quite a bit. In my setup it reduced load time by 2/3 (~300 -> ~90ms)

https://www.slideshare.net/nataliedowne/practical-maintainable-css/75

This will be a single page that demonstrates all of the different CSS styles and classes available to Datasette.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/840#issuecomment-650895874_

I have several plugins currently marked as alphas because they depend on 0.45a3. When 0.45 is released I can ship new versions of these plugins that are full releases, not alphas - and switch them to depending on 0.45 (as opposed to the alpha):

• https://github.com/simonw/datasette-init
• https://github.com/simonw/datasette-glitch
• https://github.com/simonw/datasette-saved-queries
• https://github.com/simonw/datasette-write

Needs to run somewhere with a mutable disk drive, so not Cloud Run or Heroku or Vercel.

I think I'll put it on Glitch.

These are mostly done thanks to the alphas, but I went to have more paragraphs of prose and less bullet points.

Right now the "Log out" button in the navigation will show up even if the user was authenticated by a plugin using a mechanism other than the ds_actor cookie. It should only show if the logged-in user has that cookie.

Now that CSRF is solved for API requests (#835) it would be good to support API requests to the .json extension.

https://scotthelme.co.uk/csrf-is-dead/ from Feb 2017 has background here. The SameSite=lax cookie property effectively eliminates CSRF in modern browsers. https://caniuse.com/#search=SameSite shows 92.13% global support for it.

Datasette already uses SameSite=lax when it sets cookies by default: https://github.com/simonw/datasette/blob/af350ba4571b8e3f9708c40f2ddb48fea7ac1084/datasette/utils/asgi.py#L327-L341

A few options then. I could ditch CSRF protection entirely. I could make it optional - turn it off by default, but let users who care about that remaining 7.87% of global users opt back into it.

One catch: login CSRF: I don't see how SameSite=lax protects against that attack.

This is a frustrating workaround. I have a restore_working_directory fixture that I wrote to solve errors that look like this:

/Users/simon/Dropbox/Development/datasette/tests/test_publish_cloudrun.py:148: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:112: in __enter__
    return next(self.gen)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <click.testing.CliRunner object at 0x1135ad110>

    @contextlib.contextmanager
    def isolated_filesystem(self):
        """A context manager that creates a temporary folder and changes
        the current working directory to it for isolated filesystem tests.
        """
>       cwd = os.getcwd()
E       FileNotFoundError: [Errno 2] No such file or directory

Here's an example of it in use: removing the restore_working_directory argument from this function causes the failure. https://github.com/simonw/datasette/blob/549b1c2063db48c4622ee5c7b478a1e3cbc1ac07/tests/test_plugins.py#L689-L690

I'd like to not have to do this.

Currently view permission failures raise a Forbidden error which is transformed into a 403.

It would be good if this page could offer a way forward - maybe just by linking to (or redirecting to) a login screen. This behaviour will vary based on authentication plugins, so a new plugin hook is probably the best way to do this.

Hey,
I have a database that I save in HTML from couple of web scrapers. There are around 200k+, 50+ rows in a couple of tables, with sqlite file weighing around 600MB.

The app runs on a VPS with 2 core CPU, 4GB RAM and refreshing database page regularly takes more than 10 seconds. I was suspecting that counting tables was the culprit, but manually running select count(*) from table_name for the largest table finishes under a second.

I've looked at the source code. There's a check for index page for mutable databases larger than 100MB
https://github.com/simonw/datasette/blob/799c5d53570d773203527f19530cf772dc2eeb24/datasette/views/index.py#L15

but this check is not performed for database page.
I've manually crippled Database::table_counts method

async def table_counts(self, limit=10):
    if not self.is_mutable and self.cached_table_counts is not None:
        return self.cached_table_counts
    # Try to get counts for each table, $limit timeout for each count
    counts = {}
    for table in await self.table_names():
        try:
            # table_count = (
            #     await self.execute(
            #         "select count(*) from [{}]".format(table),
            #         custom_time_limit=limit,
            #     )
            # ).rows[0][0]
            counts[table] = 10 # table_count
        # In some cases I saw "SQL Logic Error" here in addition to
        # QueryInterrupted - so we catch that too:
        except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError):
            counts[table] = None
    if not self.is_mutable:
        self.cached_table_counts = counts
    return counts

now the page loads in <100ms.

Is it possible to apply size check on database page too?

{
    "python": {
        "version": "3.8.0",
        "full": "3.8.0 (default, Oct 28 2019, 16:14:01) \n[GCC 8.3.0]"
    },
    "datasette": {
        "version": "0.44"
    },
    "asgi": "3.0",
    "uvicorn": "0.11.5",
    "sqlite": {
        "version": "3.22.0",
        "fts_versions": [
            "FTS5",
            "FTS4",
            "FTS3"
        ],
        "extensions": {
            "json1": null
        },
        "compile_options": [
            "COMPILER=gcc-7.4.0",
            "ENABLE_COLUMN_METADATA",
            "ENABLE_DBSTAT_VTAB",
            "ENABLE_FTS3",
            "ENABLE_FTS3_PARENTHESIS",
            "ENABLE_FTS3_TOKENIZER",
            "ENABLE_FTS4",
            "ENABLE_FTS5",
            "ENABLE_JSON1",
            "ENABLE_LOAD_EXTENSION",
            "ENABLE_PREUPDATE_HOOK",
            "ENABLE_RTREE",
            "ENABLE_SESSION",
            "ENABLE_STMTVTAB",
            "ENABLE_UNLOCK_NOTIFY",
            "ENABLE_UPDATE_DELETE_LIMIT",
            "HAVE_ISNAN",
            "LIKE_DOESNT_MATCH_BLOBS",
            "MAX_SCHEMA_RETRY=25",
            "MAX_VARIABLE_NUMBER=250000",
            "OMIT_LOOKASIDE",
            "SECURE_DELETE",
            "SOUNDEX",
            "TEMP_STORE=1",
            "THREADSAFE=1"
        ]
    }
}

While experimenting with https://github.com/simonw/datasette-auth-tokens I realized it's not currently possible to build API client programs that POST to Datasette because there's no mechanism for them to skip the CSRF checks added in #798.

Follows #875

https://datasette.readthedocs.io/en/0.44/pages.html

Note that these URLs end in a 7 character hash. This hash is derived from the contents of the database, and ensures that each URL is immutable: the data returned from a URL containing the hash will always be the same, since if the contents of the database file changes by even a single byte a new hash will be generated.

This isn't accurate any more - that's not default behaviour, and it may be removed entirely in #647.

Stumbled into this while working on datasette-permissions-sql. I had granted table permissions, but the permission check wasn't even executed because the user failed the previous view-database check.

It would be much cleaner if Datasette's default views were all registered using the new register_routes() plugin hook. Could dramatically reduce the code in datasette/app.py.

The ideal fix here would be to rework my BaseView subclass mechanism to work with register_routes() so that those views don't have any special privileges above plugin-provided views.
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/864#issuecomment-648580556_

Can be part of #870 - refactoring existing views to use register_routes().

I'm going to put the new check_permissions() method on BaseView as well. If I want that method to be available to plugins I can do so by turning that BaseView class into a documented API that plugins are encouraged to use themselves.
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/832#issuecomment-651995453_

While running swarm-to-sqlite it throws an error:

harper@:~/dogsheep/swarm$ swarm-to-sqlite checkins.db --save=checkins.json
Please provide your Foursquare OAuth token:
Importing 8127 checkins  [#################-------------------]   49%  00:01:52
Traceback (most recent call last):
File "/home/harper/.local/bin/swarm-to-sqlite", line 11, in <module>
    sys.exit(cli())
File "/home/harper/.local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
File "/home/harper/.local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
File "/home/harper/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/home/harper/.local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
File "/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/cli.py", line 73, in cli
    save_checkin(checkin, db)
File "/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/utils.py", line 94, in save_checkin
    photos_table.insert(photo, replace=True)
File "/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/db.py", line 963, in insert
    alter = self.value_or_default("alter", alter)
File "/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/db.py", line 1142, in insert_all
    def upsert_all(
sqlite3.OperationalError: table photos has no column named hasSticker

Where should i dig in?

$ datasette -p 0 --root
http://127.0.0.1:0/-/auth-token?token=2d498c...

The port is incorrect.

Need a cookie clearing mechanism and a way to show that you are logged in.

datasette-auth-github had a solution for this that can be pulled into core.

Traceback (most recent call last):
  File "...datasette/datasette/app.py", line 969, in route_path
    response = await view(request, send)
TypeError: favicon() missing 1 required positional argument: 'send'

Similar problem to #863 - calling datasette.add_message() in a view registered using the register_routes() plugin hook doesn't work, because the code that writes accumulated messages to the ds_messages signed cookie lives in the BaseView class here:

https://github.com/simonw/datasette/blob/28bb1c51897f3956861755e345e18b8e0b1423ac/datasette/views/base.py#L94-L97

The .coverage file generated by running pytest-cov is now a SQLite database!

I could do something interesting with this. Maybe after each test run for a new commit I could store that database file somewhere?

Lots of interesting challenges here.

I got a change into coveragepy last year which helps make the custom SQL functions available for doing fun things in Datasette: https://github.com/nedbat/coveragepy/issues/868

Bigger challenge: if I have a DB file for every commit, that's hundreds (potentially thousands) of DB files. Datasette isn't designed to handle thousands of files like that.

So, do I figure out how to have Datasette open a file on-command for just a single request? Or, an easier option, do I copy data from those files into a single database with a modified schema to include the commit hash in each table row?

(Following on from #841 and #844)

I like the shorter name better. Follows on from #842.

It might be useful to have an startup hook which gets passed the datasette object as soon as Datasette has finished initializing.

My initial use-case for this is configuration verification - checking that the "plugins" configuration block for this plugin contains valid details.

I imagine there are plenty of other potential uses for this as well.

Now that writable canned queries (#698) have landed, it would be neat if they supported "magic" parameters - parameters that are automatically populated with:

• the current actor ID / other actor properties
• the current date and time
• the user's IP or user-agent

And maybe other things potentially added by plugins.

Implementation for #842

TODO:

• Add tests for built-in magic parameters
• Magic parameters should not show up as blank form fields on the query page
• Update documentation for new _request_X (now called _header_X) implementation where X is a key from the ASGI scope
• Make sure these only work for canned queries, not for arbitrary SQL queries (security issue)
• Add test for the register_magic_parameters plugin hook
• Add documentation for the register_magic_parameters plugin hook

I was looking at this: https://github.com/simonw/datasette-block-robots/blob/main/datasette_block_robots/__init__.py

from datasette import hookimpl
from datasette.utils.asgi import Response


async def robots_txt():
    return Response.text("User-agent: *\nDisallow: /")


@hookimpl
def register_routes():
    return [
        (r"^/robots\.txt$", robots_txt),
    ]

And I realized that if register_routes() could support non-async view functions it could be reduced to this:

@hookimpl
def register_routes():
    return [
        (r"^/robots\.txt$", lambda: Response.text("User-agent: *\nDisallow: /")),
    ]

Updates the requirements on pytest-asyncio to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Over on Apache Tika, we're using datasette to allow users to make sense of the metadata for our file regression testing corpus.

This could be user error in how I've set up the reverse proxy!

I started datasette like so:
docker run -d -p 8001:8001 -vpwd:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/corpora-metadata.db --config sql_time_limit_ms:60000 --config base_url:/datasette/

I then reverse proxied like so:

ProxyPreserveHost On
ProxyPass /datasette http://x.y.z.q:xxxx
ProxyPassReverse /datasette http://x.y.z.q:xxx

Regular sql works perfectly:
https://corpora.tika.apache.org/datasette/corpora-metadata?sql=select+mime_string%2C+count%281%29+as+cnt%0D%0Afrom+profiles+p%0D%0Ajoin+mimes+m+on+p.mime_id%3Dm.mime_id%0D%0Agroup+by+mime_string%0D%0Aorder+by+cnt+desc

However, adding criteria and clicking 'Apply'
https://corpora.tika.apache.org/datasette/corpora-metadata/tika_1_24_1_mimes?_sort=file&mime__exact=text%2Fplain

bounces back to:
https://corpora.tika.apache.org/corpora-metadata/tika_1_24_1_mimes?_sort=file&file__contains=bug&mime__exact=text%2Fplain

When executing "datasette publish heroku schools.db" on Windows 10, I get the following error

  File "c:\users\dell\.virtualenvs\sec-schools-jn-cwk8z\lib\site-packages\datasette\publish\heroku.py", line 54, in heroku
    line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines()
  File "c:\python38\lib\subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "c:\python38\lib\subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "c:\python38\lib\subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "c:\python38\lib\subprocess.py", line 1307, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

Changing https://github.com/simonw/datasette/blob/55a6ffb93c57680e71a070416baae1129a0243b8/datasette/publish/heroku.py#L54

to

line.split()[0] for line in check_output(["heroku", "plugins"], shell=True).splitlines()

as well as the other check_output() and call() within the same file leads me to another recursive error about temp files

I'm running datasette serve --config base_url:/foo/ …, proxying to it with this Apache config:

    ProxyPass /foo/ http://localhost:8001/ 
    ProxyPassReverse /foo/ http://localhost:8001/

and then accessing it via https://example.com/foo/.

Although many of the URLs in the pages are correct (presumably because they either use absolute paths which include base_url or relative paths), the faceting and pagination links still use fully-qualified URLs pointing at http://localhost:8001.

I looked into this a little in the source code, and it seems to be an issue anywhere request.url or request.path is used, as these contain the values for the request between the frontend (Apache) and backend (Datasette) server. Those properties are primarily used via the path_with_… family of utility functions and the Datasette.absolute_url method.

https://twitter.com/amjithr/status/1273496759684050944

The documentation here suggests that it will work:

https://github.com/simonw/datasette/blob/eed116ac0599c7d21b7129af94d58ce03a923e4e/docs/internals.rst#L540-L546

But right now the csrftoken variable is set in BaseView.render, which means it's not visible to plugins that try to render templates using datasette.render_template:

https://github.com/simonw/datasette/blob/799c5d53570d773203527f19530cf772dc2eeb24/datasette/views/base.py#L99-L106

It turns out SQLite supports composite foreign keys: https://www.sqlite.org/foreignkeys.html#fk_composite

Their example looks like this:

CREATE TABLE album(
  albumartist TEXT,
  albumname TEXT,
  albumcover BINARY,
  PRIMARY KEY(albumartist, albumname)
);

CREATE TABLE song(
  songid     INTEGER,
  songartist TEXT,
  songalbum TEXT,
  songname   TEXT,
  FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname)
);

Here's what that looks like in sqlite-utils:

In [1]: import sqlite_utils                                                                                                                

In [2]: import sqlite3                                                                                                                     

In [3]: conn = sqlite3.connect(":memory:")                                                                                                 

In [4]: conn                                                                                                                               
Out[4]: <sqlite3.Connection at 0x1087186c0>

In [5]: conn.executescript(""" 
   ...: CREATE TABLE album( 
   ...:   albumartist TEXT, 
   ...:   albumname TEXT, 
   ...:   albumcover BINARY, 
   ...:   PRIMARY KEY(albumartist, albumname) 
   ...: ); 
   ...:  
   ...: CREATE TABLE song( 
   ...:   songid     INTEGER, 
   ...:   songartist TEXT, 
   ...:   songalbum TEXT, 
   ...:   songname   TEXT, 
   ...:   FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) 
   ...: ); 
   ...: """)                                                                                                                               
Out[5]: <sqlite3.Cursor at 0x1088def10>

In [6]: db = sqlite_utils.Database(conn)                                                                                                   

In [7]: db.tables                                                                                                                          
Out[7]: 
[<Table album (albumartist, albumname, albumcover)>,
 <Table song (songid, songartist, songalbum, songname)>]

In [8]: db.tables[0].foreign_keys                                                                                                          
Out[8]: []

In [9]: db.tables[1].foreign_keys                                                                                                          
Out[9]: 
[ForeignKey(table='song', column='songartist', other_table='album', other_column='albumartist'),
 ForeignKey(table='song', column='songalbum', other_table='album', other_column='albumname')]

The table appears to have two separate foreign keys, when actually it has a single compound composite foreign key.

https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L535-L540