This would be awesome to have for multi-gig tsv and csv files! I'm currently looking at a 10 hour countdown for one such important. Not a problem because I'm lazy and happy to let it run and check on it tomorrow..

/me munches popcorn at a furious rate, utterly entralled

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.02% :tada:

Comparison is base (6ed7908) 92.69% compared to head (4e6a341) 92.72%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage has no change and project coverage change: -0.04% :warning:

Comparison is base (6ed7908) 92.69% compared to head (fe5f881) 92.66%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.05% :warning:

Comparison is base (6ed7908) 92.69% compared to head (f7bdedf) 92.64%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.03% :warning:

Comparison is base (6ed7908) 92.69% compared to head (fc7dbe0) 92.67%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Can confirm that this bug can be reproduced as follows:

docker run datasetteproject/datasette datasette serve --reload

which produces the following output:

Starting monitor for PID 10. Error: Invalid value for '[FILES]...': Path 'serve' does not exist.

This is such a well thought out contribution. I don't think I've seen such a thoroughly considered PR on any project in recent memory.

Codecov Report

Patch coverage: 95.00% and project coverage change: -0.04% :warning:

Comparison is base (a4c96d0) 92.69% compared to head (659dcbd) 92.66%.

:exclamation: Current head 659dcbd differs from pull request most recent head acca338. Consider uploading reports for the commit acca338 to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (5d123f0) 95.77% compared to head (b86374f) 95.77%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (c263704) 92.68% compared to head (e1c5a9d) 92.69%.

:umbrella: View full report in Codecov by Sentry.

:loudspeaker: Have feedback on the report? Share it here.

Makes sense, thanks for explaining!

Codecov Report

Patch and project coverage have no change.

Comparison is base (05707aa) 92.68% compared to head (d75b519) 92.68%. Report is 7 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.

:loudspeaker: Have feedback on the report? Share it here.

@maxhawkins curious why you didn't use the stdlib mailbox to parse the mbox files?

Mailbox parses the entire mbox into memory. Using the lower level library lets us stream the emails in one at a time to support larger archives. Both libraries are in the stdlib.

I ran into the same issue on Windows using

datasette publish cloudrun mydatabase.db --service=my-database do do a google cloud publish.

@Rik-de-Kort your fix worked perfectly! Thanks!

I can always go back and delete the temp directories myself :)

@maxhawkins curious why you didn't use the stdlib mailbox to parse the mbox files?

Codecov Report

Patch and project coverage have no change.

Comparison is base (02e56d1) 95.77% compared to head (eea5151) 95.77%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (05707aa) 92.68% compared to head (7c9df6e) 92.68%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (fd083e3) 92.68% compared to head (a481ebd) 92.68%. Report is 1 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Hey @simonw I hit the same issue as mentionned by @chmaynard on a fresh install, "/mnt/fixtures.db" doesn't seem to exist in the docker image

Codecov Report

Patch coverage: 26.66% and project coverage change: -0.17% :warning:

Comparison is base (30b28c8) 92.84% compared to head (5eddf34) 92.68%.

:exclamation: Current head 5eddf34 differs from pull request most recent head 6321c9c. Consider uploading reports for the commit 6321c9c to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.03% :tada:

Comparison is base (2e28258) 92.82% compared to head (3e49fd3) 92.85%. Report is 3 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.03% :warning:

Comparison is base (2e28258) 92.82% compared to head (73489ca) 92.79%. Report is 1 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (2e28258) 92.82% compared to head (c81b024) 92.82%.

:exclamation: Current head c81b024 differs from pull request most recent head 3529c9c. Consider uploading reports for the commit 3529c9c to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 75.00% and project coverage change: +0.05% :tada:

Comparison is base (527cec6) 92.76% compared to head (a5cbf80) 92.81%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (527cec6) 92.76% compared to head (2e45686) 92.76%.

:exclamation: Current head 2e45686 differs from pull request most recent head 5dfa305. Consider uploading reports for the commit 5dfa305 to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Something to think about, but I hate how long the url is when sharing a custom SQL query. Would it be possible to hash the query and state of a page instead so the url is more manageable? The mapping from hash to query would have to be stored in order to recover/lookup the page after sharing.

It's not uncommon to have things like this currently:

https://global-power-plants.datasettes.com/global-power-plants?sql=select+rowid%2C+country%2C+country_long%2C+name%2C+gppd_idnr%2C+capacity_mw%2C+latitude%2C+longitude%2C+primary_fuel%2C+other_fuel1%2C+other_fuel2%2C+other_fuel3%2C+commissioning_year%2C+owner%2C+source%2C+url%2C+geolocation_source%2C+wepp_id%2C+year_of_capacity_data%2C+generation_gwh_2013%2C+generation_gwh_2014%2C+generation_gwh_2015%2C+generation_gwh_2016%2C+generation_gwh_2017%2C+generation_gwh_2018%2C+generation_gwh_2019%2C+generation_data_source%2C+estimated_generation_gwh_2013%2C+estimated_generation_gwh_2014%2C+estimated_generation_gwh_2015%2C+estimated_generation_gwh_2016%2C+estimated_generation_gwh_2017%2C+estimated_generation_note_2013%2C+estimated_generation_note_2014%2C+estimated_generation_note_2015%2C+estimated_generation_note_2016%2C+estimated_generation_note_2017+from+%5Bglobal-power-plants%5D+order+by+rowid+limit+101

I'm thinking a plugin like https://datasette.io/plugins/datasette-query-files, but could be created and managed from the UI (with the right permissions).

If I may, the "path-like" configuration is great but one thing that would be even greater: allowing the same configuration to be provided using environment variables.

For instance:

datasette -s plugins.datasette-complex-plugin.configs '{"foo": [1,2,3], "bar": "baz"}'

could also be provided using:

export DS_PLUGINS_DATASETTE-COMPLEX-PLUGIN_CONFIGS='{"foo": [1,2,3], "bar": "baz"}' datasette

(I do not like mixing - and _ in env vars but I do not have a best easily reversible example at the moment)

FYI, you could take some inspiration from another great open source data project, Metabase: https://www.metabase.com/docs/latest/configuring-metabase/config-file https://www.metabase.com/docs/latest/configuring-metabase/environment-variables

That's fair. The best idea I can think of is that if a plugin wanted to limit intensive queries, it could add LIMITs or something. A hook that gives you visibility of queries and maybe the option to reject felt a little more limited than the existing plugin hooks, so I was trying to think of what else one might want to do while looking at to-be-run queries.

But without a real motivating example, I see why you don't want to add that.

@simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.

Codecov Report

Patch and project coverage have no change.

Comparison is base (bdf59eb) 92.73% compared to head (b895cd2) 92.73%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage has no change and project coverage change: +0.01% :tada:

Comparison is base (17ec309) 92.71% compared to head (a651d3c) 92.73%. Report is 2 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 76.47% and project coverage change: +0.10% :tada:

Comparison is base (943df09) 92.06% compared to head (db720cd) 92.17%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks for all this! You're totally right that the ASGI option is doable, if a bit low level and coupled to the current URI design. I'm totally fine with that being the final answer.

process_view is interesting and in the general direction of what I had in mind.

A somewhat less powerful idea: Is there value in giving a hook for just the query that's about to be run? Maybe I'm thinking a little narrowly about this problem I decided I wanted to solve, but I could see other uses for a hook of the sketch below:

def prepare_query(database, table, query): """Modify query that is about to be run in some way. Return the (possibly rewritten) query to run, or None to disallow running the query""" (Maybe you actually want to return a tuple so there can be an error message when you disallow, or something.)

Maybe it's too narrowly useful and some of the other pieces of datasette obviate some of these ideas, but off the cuff I could imagine using it to: * Require a LIMIT. Either fail the query or add the limit if it's not there. * Do logging, like my usecase. * Do other analysis on whether you want to allow the query to run; a linter? query complexity?

Definitely feel free to say no, or not now. This is all me just playing around with what datasette and its plugin architecture can do with toy ideas, so don't let me push you to commit to a hook you don't feel confident fits well in the design.

@simonw, since you're referencing "rowid" column by name, I just want to note that there may be an existing rowid column with completely different semantics (https://www.sqlite.org/lang_createtable.html#rowid), which is likely to break this logic. I don't see a good way to detect a proper "rowid" name short of checking if there is a field with that name and using the alternative (_rowid_ or oid), which is not ideal, but may work.

In terms of the original issue, maybe a way to deal with it is to use rowid by default and then use primary key for WITHOUT ROWID tables (as they are guaranteed to be not null), but I suspect it may require significant changes to the API (and doesn't fully address the issue of what value to pass to indicate NULL when editing records). Would it make sense to generate a random string to indicate NULL values when editing?

This also makes it simple to separate out secrets.

datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml

settings.yaml settings: default_page_size: 10 max_returned_rows: 3000 sql_time_limit_ms": 8000 plugins: datasette-ripgrep: path: /usr/local/lib/python3.11/site-packages

secrets.yaml plugins: datasette-auth-github: client_secret: SUCH_SECRET

db-docs.yaml databases: docs: permissions: create-table: id: editor

db-fixtures.yaml databases: fixtures: tables: no_primary_key: hidden: true queries: neighborhood_search: sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood; title: Search neighborhoods description_html: |- <p>This demonstrates <em>basic</em> LIKE search

https://pypi.org/project/deep-chainmap/

https://docs.python.org/3/library/collections.html#collections.ChainMap

To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate "settings" from "configuration" is, I believe, an uphill fight. Even separating db/table-scope settings from pure descriptive metadata is not always easy. Like, do canned queries belong to database metadata or to settings? Do I need two separate files for this?

One pragmatic solution I used in a project is stacking yaml configuration files. Basically, have an arbitrary number of yaml or json settings files that you load in a specified order. Every file adds to the corresponding settings in the earlier-loaded file (if it already existed). I implemented this myself but found later that there is an existing Python "cascading dict" type of thing, I forget what it's called. There is a bit of a challenge deciding whether there is "replacement" or "addition" (I think I pragmatically ran update on the second level of the dict but better solutions are certainly possible).

This way, one allows separation of settings into different blocks, while not imposing a specific idea of what belongs where that might not apply equally to all cases.

Codecov Report

Patch and project coverage have no change.

Comparison is base (943df09) 92.06% compared to head (3a97755) 92.06%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch coverage: 92.85% and project coverage change: -0.07% :warning:

Comparison is base (1dc6b5a) 95.82% compared to head (2915050) 95.76%. Report is 1 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (943df09) 92.06% compared to head (6b617f7) 92.06%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (943df09) 92.06% compared to head (e2be94a) 92.06%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

I see :) yeah, I’m on the stable version installed from homebrew on macOS

It looks very nice now. Finally, no more manual installation of plugins one by one. Thank you, Simon! ❤️

Yes, using this approach(datasette install -r requirements.txt) will result in more consistency.

I'm curious about the results of the datasette plugins --all command. Where will we use the output of this command? Will it include configuration information for these plugins in the future? If so, will we need to consider the configuration of these plugins in addition to installing them on different computers?

You can also install this with pip using this oneliner:

pip install git+https://github.com/RhetTbull/dogsheep-photos.git@update_for_bigsur

Codecov Report

Patch and project coverage have no change.

Comparison is base (adf54f5) 92.38% compared to head (935d336) 92.38%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

And in similar fashion, how can I assign the edit-tiddlywiki permission to my user myuser in metadata.yml / metadata.json?

Codecov Report

Patch and project coverage have no change.

Comparison is base (0818182) 92.38% compared to head (730f554) 92.38%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

+1 to getting this merged down.

For future googlers, I installed by... git clone git@github.com:RhetTbull/dogsheep-photos.git cd dogsheep-photos git checkout update_for_bigsur python setup.py install

Duplicate of https://github.com/dogsheep/dogsheep-photos/pull/36 - closing.

Ran across https://github.com/dogsheep/dogsheep-photos/issues/33 which is the same subject. My PR just fixes docs

Codecov Report

Patch and project coverage have no change.

Comparison is base (0818182) 92.38% compared to head (9388102) 92.38%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (0818182) 92.38% compared to head (3aca01f) 92.38%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report

Patch and project coverage have no change.

Comparison is base (278ac91) 92.38% compared to head (7fc79ba) 92.38%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

My bad, although I could not find how to use urls, I was using a function call syntax for the filters instead of jinja's pipe syntax; closing this issue.

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.03 :tada:

Comparison is base (86a352f) 96.09% compared to head (faf398f) 96.12%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Patch and project coverage have no change.

Comparison is base (3a51ca9) 92.39% compared to head (3d45262) 92.39%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

I must be doing something wrong. On page https://192.168.1.3:5432/datasette-llm/logs/_llm_migrations there's the option to filter the results, there's an Apply button. It also tries to load an URL with an extra URL element in it:

https://192.168.1.3:5432/datasette-llm/datasette-llm/logs/_llm_migrations?_sort=name&name__contains=initial

I forgot to add that I followed these instructions to set up the python llm:

https://simonwillison.net/2023/Jul/18/accessing-llama-2/

The same problem can be seen in the links:

Advanced export

JSON shape: default, array, newline-delimited, object

Came here to report this, but instead I'll confirm the issue across two terminal emulators (Gnome Terminal and Alacritty) on Pop_OS! 22.04 (currently based on Ubuntu/Gnome). Also messes up the formatting of the terminal. Can also confirm that reset fixes it until the next sqlite-utils command.

Is there any working version of datasette/postgresql ?

@asg017 - the docs say that the autodetection only occurs in configuration directory mode. I for one would also be interested in the --settings settings.json feature.

For context, I am developing a large database for use with Datasette, but the database lives in a different network volume than my source code, since the volume in which my source code lives is aggressively backed up, while the location where the database lives is meant for temporary files and is not as aggressively backed up (since the backups would get unreasonably large).

Codecov Report

Patch coverage has no change and project coverage change: -0.02 :warning:

Comparison is base (3feed1f) 92.46% compared to head (cf5a9df) 92.45%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Thanks for the review and the code pointers @simonw - I've made the suggested edits, fixed the renamed variable, and confirmed that the panels still render on the table and database views.

@simonw any chance to get this fixed soon?

I'm not keen on requiring metadata to be within the database. I commonly have multiple DBs, from various sources, and having one config file to provide the metadata works out very well. I use Datasette with databases where I'm not the original source, needing to mutate them to add a metadata table or sqlite-docs makes me uncomfortable.

Ok @hydrosquall a couple things before this PR should be good to go:

Thank you @asg017 ! I've pushed both suggested changes onto this branch.

Not sure how difficult it'll be to inject it server-side

If we are OK with having a build system, it would free me up to do do many things! We could make datasette-manager.js a server-side rendered file as a "template" instead of having it as a static JS file, but I'm not sure it's worth the extra jump in complexity / loss of syntax highlighting in the JS file.

In the short-term, I could see an intermediary solution where a unit test in the preferred language was able to read both version.py and datasette-manager.js, and make sure that the strings versions are in sync. (This assumes that we want the manager and datasette's versions to be synced, and not decoupled). Since the version is not changing very often, a "manual sync" might be good enough.

In terms of how to integrate this into Datasette, a few options I can see working:

This sounds good to me. I'm not sure how to add a settings flag, but will be interested to see the PR that adds support for it.

I'm also curious to see how "plugins for a plugin' would work

I'm comfortable to wait until we have a realistic usecase for this. In the short term, I think we could give plugins a way to grant access to a "public API of other plugins", and also ask to be notified when plugins with other names have loaded, but don't picture the datasette manager getting more involved than that.

here's a list of Simon's Datasette plugins that use "extra_js_urls()"

Neat, thanks for compiling this list! Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?

I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.)

I look forward to trying this out 👍

Codecov Report

Patch and project coverage have no change.

Comparison is base (ede6203) 92.40% compared to head (9785c4f) 92.40%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Patch coverage: 80.55% and project coverage change: -0.15 :warning:

Comparison is base (2747257) 96.36% compared to head (5e7d27e) 96.22%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

piping to jq is good enough usually

... or python -m json.tool[^1], if you don't have jq installed.

[^1]: no fancy colouring, like jq; only pretty-printing

Thanks @cldellow for the thoughtful comments! These are all things that I'll keep in mind as we figure out how/if this API is actually used by plugin authors once it's actually out in the world.

Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.

Ah, I understand now! Thanks for explaining.

I also ran into this recently. See below for a patch for one possible solution (tested via "it works on my machine", but I don't expect that this behavior would vary a whole lot across terminal emulators and shells). Another possible solution might be to subclass click's ProgressBar to keep the logic within the original context manager. Happy to send a PR or for this patch to serve as the basis for a fix that someone else authors.

```patch diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py index 06c1a4c..530a3a3 100644 --- a/sqlite_utils/utils.py +++ b/sqlite_utils/utils.py @@ -147,14 +147,23 @@ def decode_base64_values(doc):

class UpdateWrapper: - def init(self, wrapped, update): + def init(self, wrapped, update, render_finish): self._wrapped = wrapped self._update = update + self._render_finish = render_finish

 def __iter__(self):

• for line in self._wrapped:
• self._update(len(line))
• yield line
• return self +
• def next(self):
• try:
• line = next(self._wrapped)
• except StopIteration as e:
• self._render_finish()
• raise +
• self._update(len(line))

• return line

  def read(self, size=-1): data = self._wrapped.read(size) @@ -178,7 +187,7 @@ def file_progress(file, silent=False, kwargs): else: file_length = os.path.getsize(file.name) with click.progressbar(length=file_length, kwargs) as bar: - yield UpdateWrapper(file, bar.update) + yield UpdateWrapper(file, bar.update, bar.render_finish)

class Format(enum.Enum):

```

Codecov Report

Patch coverage: 95.45% and project coverage change: +0.01 :tada:

Comparison is base (b49fa44) 92.40% compared to head (df5fd73) 92.41%.

:exclamation: Current head df5fd73 differs from pull request most recent head e990fbc. Consider uploading reports for the commit e990fbc to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Oh and for context - this goes away if I use .upsert instead of insert(..., ignore=True), but I don't want to update the value if it's written, just do an insert if it's new. The code is basically:

```py def save_items(table, items): db["users"].insert(build_user(items[0]), pk="id",ignore=True) db[table].insert_all(items)

if comments := fetch_comments(): save_items('comments', comments)

if posts := fetch_posts(): save_items('posts', posts) ```

So either comments or post could create the relevant user if those items exist. In cases where they both exist, I get this error. I need the pk because either call could create the table.

Codecov Report

Patch and project coverage have no change.

Comparison is base (e240133) 96.36% compared to head (0b81794) 96.36%.

:exclamation: Current head 0b81794 differs from pull request most recent head 21036a5. Consider uploading reports for the commit 21036a5 to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Patch coverage: 83.33% and project coverage change: +0.06 :tada:

Comparison is base (b3b100d) 96.30% compared to head (948692a) 96.36%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Patch coverage: 93.75% and no project coverage change.

Comparison is base (b3b100d) 96.30% compared to head (9f23e68) 96.31%.

:exclamation: Current head 9f23e68 differs from pull request most recent head 2eca17d. Consider uploading reports for the commit 2eca17d to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Using this thread and some other resources I managed to cobble together a couple of sqlite-utils lines to add a geometry column for a table that already has a lat/lng column.

```

add a geometry column

sqlite-utils add-geometry-column [db name] [table name] geometry --type POINT --srid 4326

add a point for each row to geometry column

sqlite-utils --load-extension=spatialite [db name] 'update [table name] SET Geometry=MakePoint(longitude, latitude, 4326);' ```

Codecov Report

Patch and project coverage have no change.

Comparison is base (49184c5) 92.40% compared to head (b99e1d3) 92.40%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Hey @cldellow , thanks for the thoughtful feedback and describing the "lazy facets" feature!

It sounds like the postTask API might be relevant for the types of network request scheduling you have in mind.

Addressing your points inline below:

It might also be nice if the plugins could return Promises.

Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like makeColumnActions or makeAboveTablePanelConfigs supported returning a promise of arrays instead only returning plain arrays?

I think what you're describing can be achievable, but I want to make sure I do so in a way that addresses your need / keeps the complexity of the plugin core system at a level this is approachable .

I have a hunch that what you're describing might be achievable without adding Promises to the API with something like

fetch('/api/with-custom-facets').then(myFacets => { // reusing the go() idiom go(manager, myFacets); })

but I'd like to confirm if that's the case before investigating adding support.

bulletproof plugin registration code that is robust against the order in which the script tags load

Yes, I think what you wrote looks right to me! While it looks a little bit verbose compared to the second example, I'm hoping we can mitigate the cost of that during this API incubation phase by making it an easy-to-copy paste code snippet.

I haven't heard of the GA queing pattern before, thanks for the example. I won't have time to implement of proof of concept in the next few weeks, but I took some time to think through the pros/cons to decide whether we may want to add this in a future release:

I can see that this approach brings advantages

• Plugin developers don't need to know the name of the datasette initialization event to start their plugin
• Pushing a function to an array probably is easier (definitely more concise) than adding a document event listener
• One less event listener sitting in memory

It also has some minor costs

• A malicious plugin could choose to (or accidentally) mess with the order of the queue if multiple scripts are lined up
• Some risk in encouraging people to mutate global state
• (not a cost, more a moot point): changing this API may not make a meaningful difference if we're discussing whether people enter 2 vs 5 lines of code, especially if those lines are encapsulated by a function we provide (maybe something that's available on the window provided by Datasette as an inline script tag).

perfect, thank you!

Hey @simonw, urllib3 maintainer here :wave:

Sorry for breaking your CI. I understand you may prefer to pin the Python version, but note that specifying just python: "3" will get you the latest. We use that in urllib3: https://github.com/urllib3/urllib3/blob/main/.readthedocs.yml

I can open PRs to sqlite-utils / datasette if you're interested

Some updates:

I notice that there is an option in the CLI where we can explicitly set immutable mode when spinning up the server

console Options: -i, --immutable PATH Database files to open in immutable mode

Then, the question is - how can I disable immutable mode in the deployed instance on Vercel?

Would love to put our heads together for improvements here.

I think anything that is argname=DEFAULT needs to be typed as argname: str | Default = DEFAULT (replacing str with the appropriate type(s)). We may be able to get clever and tie the types to that key in the _defaults dict (definitely possible in Typescript, but I'm less familiar with advanced python types).

Right now, all args are typed as Default, which means all callers get type errors.

As for table/view, given that they don't have the same methods, it would be nice to be able to get one or the other specifically.

Curiously, after running commands on the database that was litestream-restored, datasette starts to work again, e.g.

```sh litestream restore -o data/db.sqlite s3://mytestbucketxx/db datasette data/db.sqlite

fails (OperationalError described above)

```

```sh litestream restore -o data/db.sqlite s3://mytestbucketxx/db sqlite-utils enable-wal data/db.sqlite datasette data/db.sqlite

works

```

```sh litestream restore -o data/db.sqlite s3://mytestbucketxx/db sqlite-utils optimize data/db.sqlite datasette data/db.sqlite

works

```

```sh litestream restore -o data/db.sqlite s3://mytestbucketxx/db sqlite3 data/db.sqlite ".clone test.db" datasette test.db

works

```

Javascript Plugin Docs (alpha)

Motivation

The Datasette JS Plugin API allows developers to add interactive features to the UI, without having to modify the Python source code.

Setup

No external/NPM dependencies are needed.

Plugin behavior is coordinated by the Datasette manager. Every page has 1 manager.

There are 2 ways to add your plugin to the manager.

1. Read window.__DATASETTE__ if the manager was already loaded.

js const manager = window.__DATASETTE__;

1. Wait for the datasette_init event to fire if your code was loaded before the manager is ready.

```js document.addEventListener("datasette_init", function (evt) { const { detail: manager } = evt;

// register plugin here }); ```

1. Add plugin to the manager by calling manager.registerPlugin in a JS file. Each plugin will supply 1 or more hooks with

2. unique name (YOUR_PLUGIN_NAME)

3. a numeric version (starting at 0.1),
4. configuration value, the details vary by hook. (In this example, getColumnActions takes a function)

js manager.registerPlugin("YOUR_PLUGIN_NAME", { version: 0.1, makeColumnActions: (columnMeta) => { return [ { label: "Copy name to clipboard", // evt = native click event onClick: (evt) => copyToClipboard(columnMeta.column), } ]; }, });

There are 2 plugin hooks available to manager.registerPlugin:

• makeColumnActions - Add items to the cog menu for headers on datasette table pages
• makeAboveTablePanelConfigs - Add items to "tabbed" panel above the <table/> on pages that use the Datasette table template.

While there are additional properties on the manager, but it's not advised to depend on them directly as the shape is subject to change.

1. To make your JS file available as a Datasette plugin from the Python side, you can add a python file resembling this to your plugins directory. Note that you could host your JS file anywhere, it doesn't have to be served from the Datasette statics folder.

I welcome ideas for more hooks, or feedback on the current design!

Examples

See the example plugins file for additional examples.

Hooks API Guide

makeAboveTablePanelConfigs

Provide a function with a list of panel objects. Each panel object should contain

1. A unique string id
2. A string label for the tab
3. A render function. The first argument is reference to an HTML Element.

Example:

js manager.registerPlugin("panel-plugin-graphs", { version: 0.1, makeAboveTablePanelConfigs: () => { return [ { id: 'first-panel', label: "My new panel", render: node => { const description = document.createElement('p'); description.innerText = 'Hello world'; node.appendChild(description); } } ]; }, });

makeColumnActions

Provide a function that returns a list of action objects. Each action object has

1. A string label for the menu dropdown label
2. An onClick render function.

Example:

```js manager.registerPlugin("column-name-plugin", { version: 0.1, getColumnActions: (columnMeta) => {

  // Info about selected column. 
  const { columnName, columnNotNull, columnType, isPk } = columnMeta;

  return [
    {
      label: "Copy name to clipboard",
      onClick: (evt) => copyToClipboard(column),
    }
  ];
},

}); ```

The getColumnActions callback has access to an object with metadata about the clicked column. These fields include:

• columnName: string (name of the column)
• columnNotNull: boolean
• columnType: sqlite datatype enum (text, number, etc)
• isPk: Whether this is the primary key: boolean

You can use this column metadata to customize the action config objects (for example, handling different summaries for text vs number columns).

Thanks for the thoughtful review and generous examples @asg017 ! I'll make the changes you suggested soon. Bonus thoughts inlined below.

comments

These were very much appreciated, it's important to a plugin system that details like this feel right! I'll address them in batch later in the week.

I know TypeScript can be a little controversial

FWIW I am in favor of doing Typescript - I just wanted to keep the initial set of files in this PR as simple as possible to review. Really appreciate you scaffolding this initial set of types + I think it would be a welcome addition to maintain a set of types.d.ts files.

I'm entertaining the idea of writing the actual source code in Typescript as long as the compiled output is readable b/c it can be tricky to keep the types and plain JS files in sync. Curious if you have encountered projects that are good at preventing drift.

Maybe they should have more "action-y" names

This is a great observation. I'm inclined towards something like make* or build* since to me add* make me think the thing the method is attached to is being mutated, but I agree that any of these may be clearer than the current get* setup. I'll go through and update these.

Maybe we can make it easier to do pure-js datasette plugins?

I really like this idea! It'll be easier to get contributors if they don't have to touch the python side at all.

And then do the PERMITTED_VIEWS filtering in JS rather than Python.

One cost of doing this is that pages that won't use the JS would still have to load the unused code (given that I'm not sending up anything complex like lazy loading). But hopefully the manager core size is close to negligible, and it won't be a big deal.

Research notes

• I stuck to the "minimal dependencies" ethos of datasette (no React, Typescript, JS linting, etc).
• Main threads on JS plugin development
• Main: sketch of pluggy-inspired system: https://github.com/simonw/datasette/issues/983
• Main: provide locations in Datasette HTML that are designed for multiple plugins to safely cooperate with each other (starting with the panel, but eventually could extend to "search boxes"): https://github.com/simonw/datasette/issues/1191
• Main: HTML hooks for JS plugin authors: https://github.com/simonw/datasette/issues/987
• Prior threads on JS plugins in Datasette for future design directions
• Idea: pass useful strings to JS plugins: https://github.com/simonw/datasette/issues/1565
• Idea: help with plugin dependency loading: https://github.com/simonw/datasette/issues/1542 . (IMO - the plugin providing the dependency can emit an event once it's done. Other plugins can listen for it, or ask the manager to inform them when the dependency is available).
• Idea: help plugins to manage state in shareable URLs (plugins shouldn't have to interact with the URL directly, should have some basic insulation from clobbering each others' keys): https://github.com/simonw/datasette/issues/1144
• Articles on plugins reviewed
• https://css-tricks.com/designing-a-javascript-plugin-system/
• Plugin/Extension systems reviewed (mostly JS).
• Yarn: https://yarnpkg.com/advanced/plugin-tutorial
• Tappable https://github.com/webpack/tapable (used by Auto, webpack)
• Pluggy: https://pluggy.readthedocs.io/en/stable/
• VSCode: https://code.visualstudio.com/api/get-started/your-first-extension
• Chrome: https://developer.chrome.com/docs/extensions/reference/
• Figma/Figjam Widget: https://www.figma.com/widget-docs/
• Datadog Apps: Programming Model
• Storybook: https://storybook.js.org/docs/react/addons/addons-api

is this change released ? Because when we run docker containers issue still persists for production deployments.

Status?

Notes from 1:1 - it is possible to pass in URL params into a ObservableHQ notebook: https://observablehq.com/@bherbertlc/pass-values-as-url-parameters

👋 Any update on getting this merged?

Alternatively, is there a work around for this issue to unblock myself?

edit to add: huge fan of both this project and osxphotos, thanks so much for your work here 🙏 If I had any experience with Python I would offer to help but somehow I've managed to not write any Python in 10+ years of programming 😅

Edit again to add:

Alternatively, is there a work around for this issue to unblock myself?

Yes, there is. I was able to apply the patch of this PR and it applies (mostly) cleanly and works.

• verified I have a high enough version of osxphotos
• downloaded the .patch of this (by appending .patch to the URL)
• edited the patch to remove the setup.py changes
• cd to the directory containing dogsheep-photos and git apply 31.patch

Just changing the encoding in extract_notes to utf8 seems to fix it for my titles that were messed up.

I tried deploying the fix you submitted, but still getting the same errors. I can past the logs here if needed, but I really don't see anything new in them.

Ruff (written in Rust, not Python) is a 23MB executable so the time to download and pip install it dwarfs its runtime.

Let's run ruff with and without GitHub Actions pip cache side-by-side to see the relative performance.

Once you approve the workflows below, ruff_with_cache should echo cache-hit = false but if you rerun that job hopefully it should echo cache-hit = true. That will be the execution time that we are interested to compare.

There are two great problems in computer science: ;-) 1. Naming things 2. Cache invalidation 3. Off-by-one errors

For 2., https://github.com/actions/setup-python#caching-packages-dependencies is vital reading. Only exactly pinned requirements can be cached. Currently in setup.py the only pinned dependencies are: 1. Sphinx==6.1.3 2. furo==2023.3.27 3. black==23.3.0 4. blacken-docs==1.13.0 # but unpinned elsewhere in setup.py

This means that there will be very few cache hits in the current actions. See the link below to print out cache hits: https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#cache-hit

PyPI uses Fastly's CDN to quickly serve content to end-users, allowing us to minimize our hosting infrastructure and obscure possible downtime. -- https://pypi.org/sponsors

I would be shocked if Fastly does not have beefy CDN nodes in the same datacenters where GitHub Actions run so GHA requests to download ruff probably never hit a PyPI server.

Really interesting how you are using ChatGPT in this.

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (c0251cc) 96.25% compared to head (a75abeb) 96.25%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.