issue_comments

This is pretty annoying for FTS because sqlite throws an error instead of just doing something like returning all or no results. This makes users who are unfamiliar with SQL and Datasette think the canned query page is broken and is a frequent source of confusion.

To anyone dealing with this: My solution is to modify the canned query so that it returns no results which cues people to fill in the blank parameters.

So instead of emails_fts match escape_fts(:search))

My canned queries now look like this:

emails_fts match escape_fts(iif(:search=="", "*", :search))

There are no asterisks in my data so the result is always blank.

Ultimately it would be nice to be able to handle this in the metadata. Either making some named parameters required or setting some default values.

I've got a URL shortening plugin that I would like to embed on the query page but I'd like avoid capturing the entire query.html template. A feature like this would solve it. Where's this at and how can I help?

This feature is pretty important and would be nice if it would be all within Datasette (no separate CLI/deploy required). My workflow now is to basically just copy the result and paste into a Google Sheet, which works, but then it's not discoverable to other journalists browsing the Datasette instance. I started building a plugin similar to datasette-saved-queries but one that maintains its own DB (required if you're working with all immutable DBs), but got bogged down in details.

Ah, sorry, I didn't get what you were saying you the first time. Using _metadata_local in that way makes total sense -- I agree, refreshing metadata each cell was seeming quite excessive. Now I'm on the same page! :)

All good. Report back any issues you find with this stuff. Metadata/dynamic config hasn't been tested widely outside of what I've done AFAIK. If you find a strong use case for async meta, it's going to be better to know sooner rather than later!

For my reference, did you include a render_cell plugin calling get_metadata in those tests?

You shouldn't need to do this, as I mentioned previously. The code inside render_cell hook already has access to the most recently sync'd metadata via datasette._metadata_local. Refreshing the metadata for every cell seems ... excessive.

If I'm understanding your plugin code correctly, you query the db using the sync handle every time get_metdata is called, right? Won't this become a pretty big bottleneck if a hook into render_cell is trying to read metadata / plugin config?

Reading from sqlite DBs is pretty quick and I didn't notice significant performance issues when I was benchmarking. I tested on very large Datasette deployments (hundreds of DBs, millions of rows). See "Many small queries are efficient in sqlite" for more information on the rationale here. Also note that in the datasette-live-config reference plugin, the DB connection is cached, so that eliminated most of the performance worries we had.

If you need to ensure fresh metadata is being read inside of a render_cell hook specifically, you don't need to do anything further! get_metadata gets called before render_cell every request, so it already has access to the synced meta. There shouldn't be a need to call get_metadata(...) or metadata(...) inside render_cell, you can just use datasette._metadata_local if you're really worried about performance.

The plugin is close, but looks like it only grabs remote metadata, is that right? Instead what I'm wanting is to grab metadata embedded in the attached databases.

Yes correct, the datadette-remote-metadata plugin doesn't do that. But the datasette-live-config plugin does. It supports a __metadata table that, when it exists on an attached DB, gets pulled into the Datasette internal _metadata and is also accessible via get_metadata. Updating is instantaneous so there's no gotchas for users or security issues for users relying on the metadata-based permissions. Simon talked about eventually making something like this a standard feature of Datasette, but I'm not sure what the status is on that!

Good luck!

Hello! Just wanted to chime in and note that there's a plugin to have Datasette watch for updates to an external metadata.yaml/json and update the internal settings accordingly, so I think the cache/poll use case is already covered. @khusmann If you don't need truly dynamic metadata then what you've come up with or the plugin ought to work fine.

Making the get_metadata async won't improve the situation by itself as only some of the code paths accessing metadata use that hook. The other paths use the internal metadata dict. Trying to force all paths through a async hook would have performance ramifications and making everything use the internal meta will cause problems for users that need changes to take effect immediately. This is why I came to the non-async solution as it was the path of least change within Datasette. As always, open to new ideas, etc!

I did two tests: one with 1000 5-30mb DBs and a second with 20 multi gig DBs. For the second, I created them like so: for i in {1..20}; do sqlite-generate db$i.db --tables ${i}00 --rows 100,2000 --columns 5,100 --pks 0 --fks 0; done

This was for deciding whether to use lots of small DBs or to group things into a smaller number of bigger DBs. The second strategy wins.

By simply persisting the _internal DB to disk, I was able to avoid most of the performance issues I was experiencing previously. (To do this, I changed the datasette/internal_db.py:init_internal_db creates to if not exists, and changed the _internal DB instantiation in datasette/app.py:Datasette.__init__ to a path with is_mutable=True.) Super rough, but the pages now load so I can continue testing ideas.

Upon first stab, there's two issues here: - DB/table/row counts (as discussed above). This isn't too bad if the DBs are actually above the MAX limit check. - Populating the internal DB. On first load of a giant set of DBs, it can take 10-20 mins to populate. By altering datasette and persisting the internal DB to disk, this problem is vastly improved, but I'm sure this will cause problems elsewhere.

I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work.

Related: Here's an implementation of a get_metadata() plugin hook by @brandonrobertz next-LI@3fd8ce9

Here's a plugin that implements metadata-within-DBs: next-LI/datasette-live-config

How it works: If a database has a __metadata table, then it gets parsed and included in the global metadata. It also implements a database-action hook with a UI for managing config.

More context: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/init.py#L109-L140

Here's where the plugin hook is called, demonstrating the fallback= argument:

https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472

I'm not convinced of the use-case for passing fallback= to the hook here - is there a reason a plugin might care whether fallback is True or False, seeing as the metadata() method already respects that fallback logic on line 459?

I think you're right. I can't think of a reason why the plugin would care about the fallback parameter since plugins are currently mandated to return a full, global metadata dict.

Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries. @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?

Answering my own question: here's how Brandon implements it in his datasette-live-config plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/init.py#L50-L160

That's using a completely separate SQLite connection (actually wrapped in sqlite-utils) and making blocking synchronous calls to it.

This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.

But... it's weird. Everywhere else in Datasette land uses await db.execute(...) - but here's an example where users are encouraged to use blocking calls instead.

Ideally this hook would be asynchronous, but when I started down that path I quickly realized how large of a change this would be, since metadata gets used synchronously across the entire Datasette codebase. (And calling async code from sync is non-trivial.)

In my live-configuration implementation I use synchronous reads using a persistent sqlite connection. This works pretty well in practice, but I agree it's limiting. My thinking around this was to go with the path of least change as Datasette.metadata() is a critical core function.

If this is a concept ACK then I will move onto fixing the tests (adding new ones) and updating the documentation for the new plugin hook.

Note that if we went with a "update_metadata" hook, the hook signature would look something like this (it would return nothing):

update_metadata( datasette=self, metadata=metadata, key=key, database=database, table=table, fallback=fallback )

The Datasette function _metadata_recursive_update(self, orig, updated) would disappear into the plugins. Doing this, though, we'd lose the easy ability to make the local metadata.yaml immutable (since we'd no longer have the recursive update).