Or you can use https://sphinx-version-warning.readthedocs.io/! 😄

Codecov Report

Merging #1602 (29f5772) into main (cb29119) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1602   +/-   ##
=======================================
  Coverage   92.09%   92.09%           
=======================================
  Files          34       34           
  Lines        4516     4516           
=======================================
  Hits         4159     4159           
  Misses        357      357           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb29119...29f5772. Read the comment docs.

@simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks.

One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table.

Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (gis.py or something) just so it's easier to find later.

Some proposals I'd add to sqlite-utils:

Some version of this, from geojson-to-sqlite:

def init_spatialite(db, lib):
    db.conn.enable_load_extension(True)
    db.conn.load_extension(lib)
    # Initialize SpatiaLite if not yet initialized
    if "spatial_ref_sys" in db.table_names():
        return
    db.conn.execute("select InitSpatialMetadata(1)")

Also a function for creating a spatial index:

db.conn.execute("select CreateSpatialIndex(?, ?)", [table, "geometry"])

I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like:

sqlite-utils spatial-index spatial.db table-name column-name

Came here to add this. I might pick it up.

Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time.

Codecov Report

Merging #1593 (df73ebb) into main (8c401ee) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1593   +/-   ##
=======================================
  Coverage   92.09%   92.09%           
=======================================
  Files          34       34           
  Lines        4516     4516           
=======================================
  Hits         4159     4159           
  Misses        357      357           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c401ee...df73ebb. Read the comment docs.

In my example, path matching happens at the application layer (being the Django channels URLRouter). That might be a somewhat exotic solution that would normally be solved by a proxy like Apache or Nginx. However, in my specific use case, this is a "feature" enabling me to do simple management of databases and metadata from within a Django admin app instance mapped in that same router.

Thank you for the quick reply! Just a quick observation, I am running this locally without a proxy, whereas your fly example seems to be running behind an apache proxy (if the name is accurate). Can it be that the apache proxy strips the prefix before it passes on the request to the daphne backend?

thanks so much! always a pleasure to see how you work through these things

Codecov Report

Merging #377 (f4ea0d3) into main (b6dad08) will decrease coverage by 0.01%.
The diff coverage is 96.00%.

@@            Coverage Diff             @@
##             main     #377      +/-   ##
==========================================
- Coverage   96.52%   96.50%   -0.02%     
==========================================
  Files           6        6              
  Lines        2330     2378      +48     
==========================================
+ Hits         2249     2295      +46     
- Misses         81       83       +2     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6dad08...f4ea0d3. Read the comment docs.

i don’t want to be such a partisan for analyze, but the query planner deciding not to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice.

<s>the original poster in that stack overflow doesn’t say there’s a performance regression </s>

@fgregg my thinking was backwards compatibility. I don't know what people do to their builds, I just wanted a smaller image for my use case.

@simonw any chance to take a look at this? If there is no interest, feel free to close the PR

Codecov Report

Merging #367 (9848eaa) into main (a8f9cc6) will decrease coverage by 0.20%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##             main     #367      +/-   ##
==========================================
- Coverage   96.44%   96.24%   -0.21%     
==========================================
  Files           5        6       +1     
  Lines        2307     2317      +10     
==========================================
+ Hits         2225     2230       +5     
- Misses         82       87       +5     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8f9cc6...9848eaa. Read the comment docs.

or using “ pragma optimize”

the out-of-date scenario you describe could be addressed by automatically adding an analyze to the insert or convert commands if they implicate an index

the table with the query ran so bad was about 50k.

i think the scenario should not be worse than no stats.

i also did not know that sqlite was so different from postgres and needed an explicit analyze call.

for options 2 and 3, i would worry about discoverablity.

in other db’s it is not necessary to explicitly call analyze for most indices. ie for postgres

The system regularly collects statistics on all of a table's columns. Newly-created non-expression indexes can immediately use these statistics to determine an index's usefulness.

i suppose i would propose raising a warning if the stats table is created that explains what is going on and informs users about a —no-analyze argument.

is there a reason to not always use the slim option?

i added an index to one table with sqlite-utils, and then a query that used to take about 1 second started taking hundreds of seconds.

running analyze got me back to sub second speed.

This one got me, today, too. 👍

Codecov Report

Merging #361 (b7f0b88) into main (f3fd861) will decrease coverage by 0.05%.
The diff coverage is 92.85%.

@@            Coverage Diff             @@
##             main     #361      +/-   ##
==========================================
- Coverage   96.49%   96.44%   -0.06%     
==========================================
  Files           5        5              
  Lines        2283     2306      +23     
==========================================
+ Hits         2203     2224      +21     
- Misses         80       82       +2     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3fd861...b7f0b88. Read the comment docs.

@maxhawkins how hard would it be to add an entry to the table that includes the HTML version of the email, if it exists? I just attempted your the PR branch on a very small mbox file, and it worked great. My use case is a research project and I need to access more than just the body plain text.

Shouldn't be hard. The easiest way is probably to remove the if body.content_type == "text/html" clause from utils.py:254 and just return content directly without parsing.

if that is not desirable, it might be good to document that users might want to set up a lifecycle rule to automatically delete these build artifacts. something like https://stackoverflow.com/questions/59937542/can-i-delete-container-images-from-google-cloud-storage-artifacts-bucket

@maxhawkins how hard would it be to add an entry to the table that includes the HTML version of the email, if it exists? I just attempted your the PR branch on a very small mbox file, and it worked great. My use case is a research project and I need to access more than just the body plain text.

just confirmed that this error does not occur when i use my public main account. gets more interesting!

--no-headers does not work?

$ echo 'a,1\nb,2' | sqlite-utils memory --no-headers -t - 'select * from stdin'
a      1                                                                                                                             
---  ---                                                                                                                             
b      2 

I wonder if I'm encountering the same bug (or something related). I had previously been using the .csv feature to run queries and then fetch results for the pandas read_csv() function, but it seems to have stopped working recently.

https://ilsweb.cincinnatilibrary.org/collection-analysis/collection-analysis/current_collection-3d56dbf.csv?sql=select%0D%0A++*%0D%0Afrom%0D%0A++bib%0D%0Alimit%0D%0A++100&_size=max

Datasette v0.59.4

interesting! i love this feature. this + full caching with cloudflare is really super!

This goes beyond the transform() method - the curious methods that create new SQL tables could benefit from the ability to add CHECK constraints too.

I haven't used these myself, do you have any CREATE TABLE examples that use them that you can share?

I'm using them myself for the first time as well, this is a tutorial of how to use (and change) them in sqlite: https://www.sqlitetutorial.net/sqlite-check-constraint/

Codecov Report

Merging #1559 (71af58d) into main (0663d55) will increase coverage by 0.09%.
The diff coverage is 97.97%.

@@            Coverage Diff             @@
##             main    #1559      +/-   ##
==========================================
+ Coverage   91.96%   92.05%   +0.09%     
==========================================
  Files          34       34              
  Lines        4442     4493      +51     
==========================================
+ Hits         4085     4136      +51     
  Misses        357      357              

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0663d55...71af58d. Read the comment docs.

Codecov Report

Merging #1562 (2f008e8) into main (0663d55) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1562   +/-   ##
=======================================
  Coverage   91.96%   91.96%           
=======================================
  Files          34       34           
  Lines        4442     4442           
=======================================
  Hits         4085     4085           
  Misses        357      357           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0663d55...2f008e8. Read the comment docs.

Wow, that was fast, thank you so much @simonw !

I'm also not convinced that this configuration syntax is right. It's a bit weird having a "facets" list that can either by column-name-strings or {"type-of-facet": "column-name"} objects. Maybe there's a better design for this?

I agree that it's not ideal, my initial naive approach was to detect if it's an array, like what is done here:

https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/facets.py#L312-L313

But it requires an extra query to determine the type, which is a bit problematic, especially for big tables I guess.

Taking a look at #510, I wonder if a facet_delimiter should be defined for that kind of columns (that would help our team not to have an intermediary conversion step from foo|bar to ["foo","bar"] for instance).

To be consistent with the --extract-column parameter, maybe an explicit casting/delimiter would be useful: --set-column 'Foo:Array:|'.

Throwing a lot of ideas without knowing the big picture… but sometimes newcomers have superpowers :).

@simonw thank you for your fast answer and your guidance!

While digging into the code, I found an undocumented way of doing it:

facets: ["Facet for a column", {"array": "Facet for an array"}]

The only remaining problem with that solution is here: https://github.com/simonw/datasette/blob/250db8192cb8aba5eb8cd301ccc2a49525bc3d24/datasette/facets.py#L33

We have:

type, metadata_config = metadata_config.items()[0]

But it requires to cast the dict_items as a list prior to access the first element:

type, metadata_config = list(metadata_config.items())[0]

I guess it's an unspotted bug? (I mean, independently of the facets-with-arrays issue.)

Great feature.

But what is the right way to enable this to show up? Currently, it seems I need to edit the URL to add, in the right place, &_facet_size=max

Is there another (easier) way to enable this feature?

the nested query idea is very nice, and i stole if for my client side paginator. However, it won't do the right thing if the original query orders by random().

If you go the nested query route, maybe raise a 4XX status code if the query has such a clause?

these headers would also be relevant for json exports of custom queries

Codecov Report

Merging #1554 (1d08b46) into main (a6ff123) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1554   +/-   ##
=======================================
  Coverage   91.84%   91.84%           
=======================================
  Files          34       34           
  Lines        4437     4437           
=======================================
  Hits         4075     4075           
  Misses        362      362           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a6ff123...1d08b46. Read the comment docs.

would also be good if the header said the what the max row limit was

just came by to open this issue. would make my data analysis in observable a lot better!

that option is not available on custom queries.

wow! that's awesome! thanks so much, @simonw!

Codecov Report

Merging #347 (71b6c38) into main (213a0ff) will increase coverage by 0.00%.
The diff coverage is 100.00%.

:exclamation: Current head 71b6c38 differs from pull request most recent head 1a7ef2f. Consider uploading reports for the commit 1a7ef2f to get more accurate results

@@           Coverage Diff           @@
##             main     #347   +/-   ##
=======================================
  Coverage   96.51%   96.52%           
=======================================
  Files           5        5           
  Lines        2270     2271    +1     
=======================================
+ Hits         2191     2192    +1     
  Misses         79       79           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 213a0ff...1a7ef2f. Read the comment docs.

Codecov Report

Merging #1548 (68383f5) into main (737115e) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1548   +/-   ##
=======================================
  Coverage   91.84%   91.84%           
=======================================
  Files          34       34           
  Lines        4438     4438           
=======================================
  Hits         4076     4076           
  Misses        362      362           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 737115e...68383f5. Read the comment docs.

FWIW I implemented something similar with a bit of plugin code:

@hookimpl
def canned_queries(datasette: Datasette, database: str) -> Mapping[str, str]:
    # load "canned queries" from the filesystem under
    #  www/sql/db/query_name.sql
    queries = {}

    sqldir = Path(__file__).parent.parent / "sql"
    if database:
        sqldir = sqldir / database

    if not sqldir.is_dir():
        return queries

    for f in sqldir.glob('*.sql'):
        try:
            sql = f.read_text('utf8').strip()
            if not len(sql):
                log(f"Skipping empty canned query file: {f}")
                continue
            queries[f.stem] = { "sql": sql }
        except OSError as err:
            log(err)

    return queries

I actually think it would be a useful thing to add support for in datasette. It wouldn't be difficult to unwind an array of params and add the placeholders automatically.

The easiest or most straightforward thing to do is to use named parameters like:

select * where key IN (:p1, :p2, :p3)

And simply construct the list of placeholders dynamically based on the number of values.

Doing this is possible with datasette if you forgo "canned queries" and just use the raw query endpoint and pass the query sql, along with p1, p2 ... in the request.

I was thinking if there were a way to use some sort of sting function to "unpack" the values and convert them into ints... hm

Codecov Report

Merging #1543 (353a851) into main (7c02be2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1543   +/-   ##
=======================================
  Coverage   91.84%   91.84%           
=======================================
  Files          34       34           
  Lines        4438     4438           
=======================================
  Hits         4076     4076           
  Misses        362      362           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c02be2...353a851. Read the comment docs.

You can generate xml site map from the online tools using https://tools4seo.site/xml-sitemap-generator.

@hannseman
I have tried the above suggestion given by you but somehow I'm getting the below error.

note : I'm running my application with Docker.

app_1 | {"event": "Exception in ASGI application\n", "exc_info": ["<class 'RuntimeError'>", "RuntimeError('no running event loop')", "<traceback object at 0x7f9d2f5dffc0>"], "logger": "uvicorn.error", "level": "error", "timestamp": "2021-12-02T12:06:36.011448Z"}

thanks so very much for the prompt attention and fix! Plus, the animated GIF showing the bug is just extra and I love it. Interactions like this are why I love open source.

I'd use this. Right now, I tend to do touch my.db and then enable-wal or whatever else, but I'm never sure if that's a bad idea.

It's a really great API and the documentation is really great too. Honestly, in more than 20 years of professional experience, I haven't worked with any software API that was more of a joy to use.

Codecov Report

Merging #1529 (2cb3dbb) into main (48f1199) will decrease coverage by 0.06%.
The diff coverage is n/a.

:exclamation: Current head 2cb3dbb differs from pull request most recent head 495b726. Consider uploading reports for the commit 495b726 to get more accurate results

@@            Coverage Diff             @@
##             main    #1529      +/-   ##
==========================================
- Coverage   91.90%   91.83%   -0.07%     
==========================================
  Files          34       34              
  Lines        4434     4434              
==========================================
- Hits         4075     4072       -3     
- Misses        359      362       +3     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1beb7d9...495b726. Read the comment docs.

There isn't any way to do this with sqlite as far as I know. The only option is to insert the right number of ? placeholders into the sql template and then provide an array of values.

FWIW I've written some web components that consume the json api and I think it's a really nice way to work with datasette. I like the combination with datasette+sqlite as a back-end feeding data to a front-end that's entirely javascript + html.

Codecov Report

Merging #1537 (fcfaec1) into main (48f1199) will decrease coverage by 0.06%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1537      +/-   ##
==========================================
- Coverage   91.90%   91.83%   -0.07%     
==========================================
  Files          34       34              
  Lines        4434     4434              
==========================================
- Hits         4075     4072       -3     
- Misses        359      362       +3     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 48f1199...fcfaec1. Read the comment docs.

Thanks for your reply @simonw . Tbh, my first attempt was actually the parquet-to-sqlite package but I already had Makefiles that relied on SQLite-utils and it was less intrusive to my workflow. Maybe I'll revisit that decision.
FYI: there's a [sqlite-parquet-vtable](https://github.com/cldellow/sqlite-parquet-vtable)

I don't think plugins make much sense either. Probably defeats the purpose of simplicity: simple database along with a pip-able package.

If you suspect that Cloud Run throttled CPU could be the cause, you can request to have CPU always allocated with gcloud beta run deploy --no-cpu-throttling (read more)

It could also be the Cloud Run sandbox that somehow gets in the way here, in which case I recommend testing with the second generation execution environment: gcloud beta run deploy --execution-environment gen2

Oh, another thing to consider: I believe this would be the first "_file" key in datasette's metadata, compared to other "_url" keys like "license_url" or "about_url". Not too sure what considerations to include with this (ex should missing files cause Datasette to stop before starting, should build scripts bundle these sql files somewhere during datasette package, etc.)

I think another thing would be to make /pages/robots.txt work. That way you can use jinja to generate a desired robots.txt. I'm using it to allow the main index and what it links to to be crawled (but not the database pages directly.)

As a a sanity check, would it be worth looking at trying to push the multi-process container on another provider of a knative / cloud run / tekton ? I have a somewhat similar use case for a future proejct, so i'm been very grateful to you sharing all the progress in this issue.

As I understand it, Scaleway also offer a very similar offering using what appear to be many similar components that might at least see if it's an issue with more than one knative based FaaS provider

https://www.scaleway.com/en/serverless-containers/
https://developers.scaleway.com/en/products/containers/api/#main-features

Codecov Report

Merging #1516 (a82c620) into main (0156c6b) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1516   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4430     4430           
=======================================
  Hits         4068     4068           
  Misses        362      362           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0156c6b...a82c620. Read the comment docs.

Codecov Report

Merging #1514 (b02c35a) into main (0156c6b) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1514   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4430     4430           
=======================================
  Hits         4068     4068           
  Misses        362      362           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0156c6b...b02c35a. Read the comment docs.

Codecov Report

Merging #1512 (8f757da) into main (0156c6b) will decrease coverage by 2.10%.
The diff coverage is 36.20%.

@@            Coverage Diff             @@
##             main    #1512      +/-   ##
==========================================
- Coverage   91.82%   89.72%   -2.11%     
==========================================
  Files          34       36       +2     
  Lines        4430     4604     +174     
==========================================
+ Hits         4068     4131      +63     
- Misses        362      473     +111     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0156c6b...8f757da. Read the comment docs.

A suggestion is to have the option to choose an arbitrary delimiter (and quoting characters )

Codecov Report

Merging #1508 (299774b) into main (502c02f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1508   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4430     4430           
=======================================
  Hits         4068     4068           
  Misses        362      362           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 502c02f...299774b. Read the comment docs.

@glasnt yeah I guess that could be an option. I run datasette on large databases > 75gb and the startup time is a bit slow for me even with -i --inspect-file options. Here's a quick sketch for a plugin that will reload db's in a folder that you set for the plugin in metadata.json. If you request /-reload-db new db's will be added. (You probably want to implement some authentication for this =) )

https://gist.github.com/Segerberg/b96a0e0a5389dce2396497323cda7042

There is no such option see https://github.com/simonw/datasette/issues/43. But you could write a plugin using the datasette.add_database(db, name=None) https://docs.datasette.io/en/stable/internals.html#add-database-db-name-none

i was just reaching for a tool to do this this morning

Smokes, it looks like there was a bug in click 8.0.2 (fixed in 8.0.3: https://github.com/pallets/click/issues/2089). Meaning this PR is not needed. Closing.

I think I only wonder how I would parse the JSON value within such a lambda...

My naive approach would have been $ sqlite-utils convert demo.db statuses statuses 'return value' --multi

I am super interested in this feature.

After reading the other issues you referenced, I think the right way would be to use the current extract feature and then to use sqlite-utils convert to extract the json object into individual columns

Codecov Report

Merging #1500 (1b7f679) into main (2c31d1c) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1500   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4426     4426           
=======================================
  Hits         4064     4064           
  Misses        362      362           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2c31d1c...1b7f679. Read the comment docs.

I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.

Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.

I was thinking about this, while inserting a stream of ~40M line-delimited json docs. Wouldn't a --total-expected flag work ?

That's how tqdm does it

Yeah, I was pressing Ctrl + Enter as well. Came here to open this issue and found out Shift + Enter works.

@simonw Any way to configure this?

@simonw That was working fine. It turned out that I had to use --nl flag since I was passing in jsonl.

@simonw Hey! JSON parsing for me is failing and I'm getting this same error, but I feel that my json is correct. How can I debug this?

okay @simonw, made the requested changes. tests are running locally. i think this is ready for you to look at again.

@mhalle maybe you can try out #333 ?

I've been thinking about this a bit lately, doing a project that involves moving a lot of data in and out of SQLite files, datasette and GeoJSON. This has me leaning toward the idea that something like datasette query would be a better place to do async queries.

I know there's a lot of overlap in sqlite-utils and datasette, and maybe keeping sqlite-utils synchronous would let datasette be entirely async and give a cleaner separation of implementations.

This sounds really ambitious but also really awesome. I like the idea that basically any piece of a page could be selectively replaced.

It sort of sounds like a python asyncio version of https://github.com/observablehq/runtime

Writing the tests will be a bit tricky since we need to confirm that the include_table_top(datasette, database, actor, table) arguments were all passed correctly but the only thing we get back from the plugin is a list of templates. Maybe encode those values into the template names somehow?

Why not return a data structure instead of just a template name?

I've already done some custom hacking to modify datasette but the plugin mechanism you are building here would be much cleaner than what I've built. I'd be happy to help with testing this PR and fleshing it out further if you are still considering merging this.

Codecov Report

Merging #1481 (77542e7) into main (6388617) will decrease coverage by 0.01%.
The diff coverage is n/a.

:exclamation: Current head 77542e7 differs from pull request most recent head 50005bd. Consider uploading reports for the commit 50005bd to get more accurate results

@@            Coverage Diff             @@
##             main    #1481      +/-   ##
==========================================
- Coverage   91.83%   91.82%   -0.02%     
==========================================
  Files          34       34              
  Lines        4421     4426       +5     
==========================================
+ Hits         4060     4064       +4     
- Misses        361      362       +1     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6388617...50005bd. Read the comment docs.

i'm going to take a swing at this today. we'll see.

@simonw I know this is closed, just found this via the annotated release notes, but I wanted to note this one thing:

Not sure how widely used this is, but I've seen CSVW a couple times in the wild. It is trying to address these metadata challenges in a standardised way.

See e.g.
- https://www.w3.org/TR/tabular-data-primer/#h-documentation-columns
- https://w3c.github.io/csvw/tests/

I'm not suggesting you change the syntax you've implemented, just letting you know of this effort by W3C.

Codecov Report

Merging #1494 (7a3e3c3) into main (ff9ccfb) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1494   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4426     4426           
=======================================
  Hits         4064     4064           
  Misses        362      362           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ff9ccfb...7a3e3c3. Read the comment docs.

@simonw I am getting the below issue again now, even after removing branch argument from vercel datasette plugin

module initialization error: init() got an unexpected keyword argument 'config'
module initialization error
init() got an unexpected keyword argument 'config'

Yay! Thank you @simonw!!

It would be cool if --flatten worked with sqlite-utils memory as well.

Codecov Report

Merging #330 (fc3de90) into main (718a8f6) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #330   +/-   ##
=======================================
  Coverage   96.59%   96.59%           
=======================================
  Files           5        5           
  Lines        2230     2230           
=======================================
  Hits         2154     2154           
  Misses         76       76           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 718a8f6...fc3de90. Read the comment docs.

Codecov Report

Merging #1463 (2dd94c5) into main (0d5cc20) will decrease coverage by 0.00%.
The diff coverage is n/a.

:exclamation: Current head 2dd94c5 differs from pull request most recent head 481212c. Consider uploading reports for the commit 481212c to get more accurate results

@@            Coverage Diff             @@
##             main    #1463      +/-   ##
==========================================
- Coverage   91.83%   91.83%   -0.01%     
==========================================
  Files          34       34              
  Lines        4422     4421       -1     
==========================================
- Hits         4061     4060       -1     
  Misses        361      361              

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e1012e7...481212c. Read the comment docs.

Codecov Report

Merging #1471 (01b05ed) into main (0d5cc20) will decrease coverage by 0.00%.
The diff coverage is n/a.

:exclamation: Current head 01b05ed differs from pull request most recent head 847238a. Consider uploading reports for the commit 847238a to get more accurate results

@@            Coverage Diff             @@
##             main    #1471      +/-   ##
==========================================
- Coverage   91.83%   91.83%   -0.01%     
==========================================
  Files          34       34              
  Lines        4422     4421       -1     
==========================================
- Hits         4061     4060       -1     
  Misses        361      361              

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a673a93...847238a. Read the comment docs.

Nevermind, had to remove the branch argument in the workflow to make vercel publish work

I am running into the same problem. Is there any workaround?

Any workaround for making it work with datasette-publish-vercel. Currently getting below error

module initialization error: init() got an unexpected keyword argument 'config'
module initialization error
init() got an unexpected keyword argument 'config'

Codecov Report

Merging #1485 (be9aed9) into main (0d5cc20) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1485   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4422     4422           
=======================================
  Hits         4061     4061           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0d5cc20...be9aed9. Read the comment docs.

Maybe the exif-loader from the SpatiaLite project could be useful as a reference even it is written in C and it also saves images as blobs https://www.gaia-gis.it/fossil/spatialite-tools/file?name=exif_loader.c&ci=tip. The tool is also integrated into the spatialite-gui application.

I found some user documentation from the web archive http://web.archive.org/web/20180629041238/https://www.gaia-gis.it/spatialite-2.3.1/spatialite-exif-2.3.1.html.