issues

Sometimes a plugin may want to add content to an existing default template - for example datasette-search-all adds a new search box at the top of index.html. I also want datasette-upload-csvs to add a CTA on the database.html page: https://github.com/simonw/datasette-upload-csvs/issues/18

Currently plugins can do this by providing a new version of the index.html template - but if multiple plugins try to do that only one of them will succeed.

It would be better if there were known areas of those templates which plugins could add additional content to, such that multiple plugins can use the same spot.

Refs: - #2049

TODO:

• [x] Read queries JSON
• Implement error display with "ok": false and an errors key
• Read queries HTML
• Read queries other formats (plugins)
• Canned read queries (dispatched to from table)
• Write queries (a canned query thing)
• Implement different shapes, refactoring to share code with table
• Implement a sensible subset of extras, also refactoring to share code with table
• Get all tests passing

:books: Documentation preview :books:: https://datasette--2053.org.readthedocs.build/en/2053/

Part of: - #262 - #1518

It really feels like this should be accompanied by a /db/-/create API for creating tables. I had to add that to sqlite-utils eventually (initially it only supported creating by passing in an example document):

https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1862#issuecomment-1299073433

Similar to https://github.com/simonw/datasette-insert/blob/0.8/README.md#inserting-data-and-creating-tables

I expect this to become by far the most common way that data gets into a Datasette instance - more so than the individual row API in: - #1851

It would be really cool if you could register additional custom SQL functions for use with the sqlite-utils query command - something like this:

``` sqlite-utils data.db 'update images set domain = extract_domain(url)' --functions ' from urllib.parse import urlparse

def extract_domain(url): return urlparse(url).netloc ' ``` Every function defined in that code block would be registered with the connection, unless the name began with an underscore.

For example on this page: https://docs.datasette.io/en/stable/writing_plugins.html#packaging-a-plugin

I wonder if there's an easy way for me to enforce this for Sphinx documentation?

Refs #1439

I tried running some parallel SQL queries using asyncio.gather() but the SQL that was executed didn't show up in the trace rendered by https://datasette.io/plugins/datasette-pretty-traces

I realized that was because traces are keyed against the current task ID, which changes when a sub-task is run using asyncio.gather or similar.

The faceting and suggest faceting queries are missing from this trace:

The reason they aren't showing up in the traces is that traces are stored just for the currently executing asyncio task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25

This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively "child tasks" of the current request, and hence should be tracked the same.

https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using asyncio.gather() to work around this kind of problem.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1518#issuecomment-999870993

Original title: Can I keep "pip install datasette" working on Python 3.6?

I dropped support for 3.6 in: - #1577

I'm getting reports that pip3 install datasette throws an error on that Python, even though I haven't made that new release yet - presumably due to lack of pinning of Uvicorn: https://twitter.com/ldodds/status/1484289475195080706

Is it possible to get pip on that version of Python to install the highest possible version of the packages that are still known to support Python 3.6?

If so, how?

This isn't particularly straight-forward - all the more reason for Datasette to implement it for you. This article is helpful: http://charlesleifer.com/blog/using-sqlite-full-text-search-with-python/

The homepage currently performs a massive flurry of permission checks - one for each, database, table and view: https://github.com/simonw/datasette/blob/0.53/datasette/views/index.py#L21-L75

A paginated version of this is a little daunting as the permission checks would have to be carried out in every single table just to calculate the count that will be paginated.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1150#issuecomment-747864831

Consider this page: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_search=plant&_facet=owner&_facet=country_long&_facet=primary_fuel

Datasette needs to run the main query for the rows on that page, a count query for the total query, then a separate query for each of those three specified facets.

This is a _search= query, so it needs to execute the FTS code once for the rows, again for the count, and then three more times for each of the facets.

Could running that query as a CTE and doing the other queries as part of the same large query produce significant speed improvements?

I created this view: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads-8dbda00/ads_with_targets

CREATE VIEW ads_with_targets as select ads.*, json_group_array(targets.name) as target_names from ads join ad_targets on ad_targets.ad_id = ads.id join targets on ad_targets.target_id = targets.id group by ad_targets.ad_id

When I try to apply faceting by array it appears to work at first: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads/ads_with_targets?_facet_array=target_names

But actually it's doing the wrong thing - the SQL for the facets uses rowid, but rowid is not present on views at all! These results are incorrect, and clicking to select a facet will fail to produce any rows: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads/ads_with_targets?_facet_array=target_names&target_names__arraycontains=people_who_match%3Ainterests%3AAfrican-American+Civil+Rights+Movement+%281954%E2%80%9468%29

Here's the SQL it should be using when you select a facet (note that it does not use a rowid):

https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select+*+from+ads_with_targets+where+id+in+%28%0D%0A++++++++++++select+ads_with_targets.id+from+ads_with_targets%2C+json_each%28ads_with_targets.target_names%29+j%0D%0A++++++++++++where+j.value+%3D+%3Ap0%0D%0A++++++++%29+limit+101&p0=people_who_match%3Ainterests%3ABlack+%28Color%29

So we need to do something a lot smarter here. I'm not sure what the fix will look like, or even if it's feasible given that views don't have a rowid to hook into so the JSON faceting SQL may have to be completely rewritten.

datasette publish cloudrun \ russian-ads.db \ --name json-view-facet-bug-demo \ --branch master \ --extra-options "--config sql_time_limit_ms:5000 --config facet_time_limit_ms:5000"

I think it's best to still keep [ and ] out of column names though. Transforming them into ( and ) seems reasonable - but should that happen here or in sqlite-utils? I think in sqlite-utils.

Originally posted by @simonw in https://github.com/simonw/datasette-app/issues/121#issuecomment-926200398

This is a rethinking of the solution to:

• https://github.com/simonw/sqlite-utils/issues/86

To give an indication as to when the data was last updated.

This should be a field in the metadata that is then shown on the index page and in the footer, if it is set.

Also support setting it using an option to “datasette publish” and “datasette package” - which can either be a string or can be the magic string “today” to set it to today’s date:

datasette publish file.db --updated=today

SQLite lets you create indexes in reverse order, which can have a surprisingly big impact on performance, see https://github.com/simonw/covid-19-datasette/issues/27

I tried doing this using sqlite-utils like so, but it's didn't work:

python db["ny_times_us_counties"].create_index(["date desc"])

The ability to run Datasette in a mode where it automatically picks up new (or modified) files in a directory tree without needing to restart the server.

Suggested command:

datasette library /path/to/mydbs/

https://docs.brew.sh/Python-for-Formula-Authors describes how.

Applications should be installed into a Python virtualenv environment rooted in libexec. This prevents the app’s Python modules from contaminating the system site-packages and vice versa.

It recommends using https://github.com/tdsmith/homebrew-pypi-poet

Stumbled into this while working on datasette-permissions-sql. I had granted table permissions, but the permission check wasn't even executed because the user failed the previous view-database check.

Thanks! I'm trying to launch Datasette from within a notebook using the jupyter-server-proxy and the new base_url parameter. While the assets load ok, and the breadcrumb navigation works, the facet links don't seem to use the base_url. Or have I missed something?

Originally posted by @wragge in https://github.com/simonw/datasette/issues/394#issuecomment-604166918

This is a trick used by Starlette so that other layers of ASGI middleware can see which route was selected.

They added it here: https://github.com/encode/starlette/commit/34d0097feb6f057bd050d5057df5a2f96b97384e

If Datasette supports it as well we can benefit from it if we integrate this sentry_asgi middleware (probably as a datasette-sentry plugin): https://github.com/encode/sentry-asgi/blob/c6a42d44d31f85885b79e4ee898683ecf8104971/sentry_asgi/middleware.py#L34-L35

Most of the code here was fleshed out in comments on #272 (Port Datasette to ASGI) - this pull request will track the final pieces:

• [x] Update test harness to more correctly simulate the raw_path issue
• [x] Use raw_path so table names containing / can work correctly
• [x] Bug: JSON not served with correct content-type
• [x] Get ?_trace=1 working again
• [x] Replacement for @app.listener("before_server_start")
• [x] Bug: /fixtures/table%2Fwith%2Fslashes.csv?_format=json downloads as CSV
• [x] Replace Sanic request and response objects with my own classes, so I can remove Sanic dependency
• [x] Final code tidy-up before merging to master

It turns out if you load this CSV using csvs-to-sqlite you get an Infinity value in SQLite:

name,num sasha,10 terry,Inf cathy,0.5

csvs-to-sqlite infinity-bug.csv infinity-bug.db

I deployed this using:

datasette publish now infinity-bug.db --name=datasette-infinity-bug --install=datasette-vega

Datasette outputs that as Infinity in the JSON format, which causes JavaScript errors.

Demo * https://datasette-infinity-bug.now.sh/infinity-bug-0d0224e/infinity-bug - HTML view works * https://datasette-infinity-bug.now.sh/infinity-bug-0d0224e/infinity-bug.json?_shape=array - this outputs the following:

[ { "rowid": 1, "name": "sasha", "num": 10.0 }, { "rowid": 2, "name": "terry", "num": Infinity }, { "rowid": 3, "name": "cathy", "num": 0.5 } ]

But... in Firefox that gets rendered like this:

And if you click the "Show charting options" button you get this error in the console:

SyntaxError: JSON.parse: unexpected character at line 1 column 83 of the JSON data