issues

There's a long justification for why this might be needed here: - https://github.com/simonw/datasette-auth-tokens/issues/10#issuecomment-1701820001

Short version: it would be neat if it was possible to stash some data on the request object such that a later plugin/middleware-type-thing could use that to influence the final returned response - similar to the kinds of things you can do with Django middleware.

The asgi_wrapper() mechanism doesn't have access to the request or response objects - it gets scope and can mess around with receive and send, but those are pretty low-level primitives.

Since Datasette has well-defined request and response objects now it might be nice to have a middleware layer that can manipulate those directly.

I use datasette in an iframe inside another HTML file that contains other ways to represent my data (mostly leaflets maps built with R on summarized data), and the datasette iframe is a tab in that page.

This particular use prevents users to access the full URLs of their datasette views and queries, which is a shame because the way datasette handles URLs to make every view or query easy to share is awesome. I know how to get the URL from the context menu of my browser, but I don't think many visitors would do it or even notice that datasette uses permalinks for pretty much every action they do. Would it be possible to add a "Share link" button to the interface, either in datasette itself or in a plugin?

I'm interested in making a plugin that saves every query that gets run to a table in the database. (I know about datasette-query-history but thought it would be good to have a server-side option.)

As far as I can tell reading the docs, there isn't really a hook setup to allow this.

Maybe I could hack it with some of the hooks that are passed requests, but that doesn't seem good.

I'm a little surprised this isn't possible, so I thought I would open an issue and see if that's a deeply considered decision or just "haven't needed it yet." I'm potentially interested in implementing the hook if the latter.

Ruff supports over 500 lint rules and can be used to replace Flake8 (plus dozens of plugins), isort, pydocstyle, yesqa, eradicate, pyupgrade, and autoflake, all while executing (in Rust) tens or hundreds of times faster than any individual tool.

The ruff Action uses minimal steps to run in ~5 seconds, rapidly providing intuitive GitHub Annotations to contributors.

:books: Documentation preview :books:: https://datasette--2056.org.readthedocs.build/en/2056/

Reported as a bug in #1098 because they caused 500 errors - but it would be even better if Datasette could hyperlink to related rows via compound foreign keys.

Ran inspect on a spatialite database and got these warnings:

ERROR: conn=<sqlite3.Connection object at 0x119e46110>, sql = 'select count(*) from [SpatialIndex]', params = None: no such module: VirtualSpatialIndex ERROR: conn=<sqlite3.Connection object at 0x119e46110>, sql = 'select count(*) from [ElementaryGeometries]', params = None: no such module: VirtualElementary ERROR: conn=<sqlite3.Connection object at 0x119e46110>, sql = 'select count(*) from [KNN]', params = None: no such module: VirtualKNN

It still worked, but probably want to catch this.

I started sketching out a plugin to add a datasette query subcommand to export data from the command line. This is based on discussions in #1356 and #1605. Before I get too far down this rabbit hole, I figure it's worth getting some feedback here (unless this should happen in Discussions). Here's what I'm thinking:

At its most basic, it will write the results of a query to STDOUT.

sh datasette query -d data.db 'select * from data' > results.json

This isn't much improvement over using sqlite-utils. To make better use of datasette and its ecosystem, run datasette query using a canned query defined in a metadata.yml file.

For example, using the metadata file from alltheplaces-datasette:

sh cd alltheplaces-datasette datasette query -d alltheplaces.db -m metadata.yml count_by_spider

That query would be good to get as CSV, and we can auto-discover metadata and databases in the current directory:

sh cd alltheplaces-datasette datasette query count_by_spider -f csv

In this case, count_by_spider is a canned query defined on the alltheplaces database. If the same query is defined on multiple databases or its otherwise unclear which database query should use, pass the -d or --database option.

If a query takes parameters, I can pass them in at runtime, using the --param or -p option:

sh datasette query -d data.db -p value something 'select * from neighborhoods where some_column = :value'

I'm very interested in feedback on this, including whether it should be a plugin or in Datasette core. (I don't have a strong opinion about this, but I'm prototyping it as a plugin to start.)

Datasette's documentation is aimed at people who install and configure it.

What about end users of preconfigured and deployed Datasette instances?

Something that can be linked to from the Datasette UI would be really useful.

It would be nice if the --cors option accepted an origin regex to more securely allow secure local development.

As an example, Observable notebooks namespace every user's notebooks by their username and user content is served from username.observableusercontent.com, so you would set --cors-origin username.observableusercontent.com to restrict access to a local development Datasette instance to only your own notebooks, rather than exposing the data to any website that makes a request.

Thank you for all of your work on Datasette!

First of all, thanks for creating this cool project, and also supporting publishing to various hosting services out of the box.

While testing out, I noticed legitimate queries such as select * from books where title like 'Pragmatic%' or select * from books where title = 'The Pragmatic Programmer' are blocked, due to the regular expression check here: https://github.com/simonw/datasette/blob/main/datasette/utils/init.py#L185

Example as seen from a Datasette instance: https://fivethirtyeight.datasettes.com/polls?sql=select+*+from+books+where+title+like+%27Pragmatic%25%27%0D%0A

I'd propose a regular expression like re.compile(f"pragma_(?!({'|'.join(allowed_pragmas)}))"), instead of re.compile(f"pragma(?!_({'|'.join(allowed_pragmas)}))"),

I can create a pull request with this change, unless the maintainers think it would allow unwanted queries to be executed.

Hovering over that could work a little bit like GitHub issue links:

I followed the instructions here, and even after running python update-docs-help.py I get the following failed tests -- any thoughts?

FAILED tests/test_api.py::test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3] FAILED tests/test_api.py::test_searchmode[table_metadata1-_search=te*+AND+do*-expected_rows1] FAILED tests/test_api.py::test_searchmode[table_metadata2-_search=te*+AND+do*&_searchmode=raw-expected_rows2]

This is with python 3.9.7 and lots of other packages, as in attached environment listing from conda list. conda-installed.txt

Datasette currently only inflates foreign keys (into names hyperlinks) if it detects them as foreign key constraints in the underlying database.

It would be useful if you could specify additional "foreign keys" using both metadata.json and the querystring - similar time how you can pass ?_fts_table=x https://datasette.readthedocs.io/en/stable/full_text_search.html#configuring-full-text-search-for-a-table-or-view

I stumbled into this while working on Dogsheep Beta: the requests it re-dispatched through TableView did not carry authentication cookies, and since this was against a private instance they were thus denied.

https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/init.py#L223-L231

This made me think that datasette.client.get() (which Dogsheep Beta will start using shortly) could benefit from some kind of utility mechanism for passing through the cookies and general authenticated state from the current request.

For https://github.com/simonw/russian-ira-facebook-ads-datasette/tree/af6d956995e14afd585c35a6a06bb01da32043ba I wrote a script to convert YAML to JSON because YAML is a better format for embedding multi-line HTML descriptions and canned SQL statements.

Example yaml metadata file: https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/af6d956995e14afd585c35a6a06bb01da32043ba/russian-ads-metadata.yaml

It would be useful if Datasette could be fed a YAML file directly:

datasette -m metadata.yaml

Question is... should this be a native feature (hence adding a YAML dependency) or should it be handled by a datasette-metadata-yaml plugin, using a new plugin hook for loading metadata? If so, what would other use-cases for that plugin hook be?

Hi Simon

Is there anywhere in the app-context where one can access the hashed urlpath of the database? Currently it's included in the template-context (databases[0]["path") when rendering urls of the database (eg. /db-44b06v9/cases...), but where can I find the hashed url when rendering the index-page? I'm trying to avoid redirects. Thanks!

I've sucessfully "uploaded" an SQLite database (with a metadata.json file) to heroku using:

$ datasette publish heroku so-sales.db -m metadata.json -n so-sales

The question is: how can I modify the (small) metadata.json file without having to upload the (large) SQLite database.

The directions on heroku indicate I should run:

heroku git:clone -a so-sales

But this just results in an empty directory with a warning: warning: You appear to have cloned an empty repository.

I've been able to "clone" the heroku "app" using the command:

$ heroku slugs:download -a so-sales

but this is not a git repository....

Ideally, it seems to me, there'd be an option of the datasette CLI to allow a file to be updated, or there'd be some way to create a local git "clone" of the app so that the heroku instructions for "Deploying with git" would apply.

(p.s. I ran datasette publish heroku -m metadata.json -n so-sales in the hope that that would not cause the .db file to be wiped, but of course it was.)

(p.p.s. Thanks for Datasette!)