Have you seen MakeTypes? Not the exact same thing but it may be relevant.

And it's inspired by the paper "Types from Data: Making Structured Data First-Class Citizens in F#".

Codecov Report

Merging #1763 (fd6a817) into main (00e59ec) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1763   +/-   ##
=======================================
  Coverage   91.67%   91.67%           
=======================================
  Files          36       36           
  Lines        4658     4658           
=======================================
  Hits         4270     4270           
  Misses        388      388           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 00e59ec...fd6a817. Read the comment docs.

Yeah, I think I prefer your suggestion: it seems cleaner than my initial left_name=/right_name= idea. Perhaps one downside is that it's less obvious what the role of each field is: in this example, is people_id_1 a reference to parent or child?

In terms of unique identifiers - could you use values stored in HKMetadataKeySyncIdentifier?

Codecov Report

Merging #1760 (69951ee) into main (00e59ec) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1760   +/-   ##
=======================================
  Coverage   91.67%   91.67%           
=======================================
  Files          36       36           
  Lines        4658     4658           
=======================================
  Hits         4270     4270           
  Misses        388      388           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 00e59ec...69951ee. Read the comment docs.

log.txt

Attempted to test on a machine with a new version of Python, but install failed with an error message for the 'click' package.

C:\WINDOWS\system32>"c:\Program Files\Python310\python.exe"
Python 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

C:\WINDOWS\system32>cd C:\Users\swood\Downloads\sqlite-utils-main-20220621\sqlite-utils-main

C:\Users\swood\Downloads\sqlite-utils-main-20220621\sqlite-utils-main>"c:\Program Files\Python310\python.exe" setup.py install
running install
running bdist_egg
running egg_info

...

Installed c:\program files\python310\lib\site-packages\click_default_group_wheel-1.2.2-py3.10.egg
Searching for click
Downloading https://files.pythonhosted.org/packages/3d/da/f3bbf30f7e71d881585d598f67f4424b2cc4c68f39849542e81183218017/click-default-group-wheel-1.2.2.tar.gz#sha256=e90da42d92c03e88a12ed0c0b69c8a29afb5d36e3dc8d29c423ba4219e6d7747
Best match: click default-group-wheel-1.2.2
Processing click-default-group-wheel-1.2.2.tar.gz
Writing C:\Users\swood\AppData\Local\Temp\easy_install-aiaj0_eh\click-default-group-wheel-1.2.2\setup.cfg
Running click-default-group-wheel-1.2.2\setup.py -q bdist_egg --dist-dir C:\Users\swood\AppData\Local\Temp\easy_install-aiaj0_eh\click-default-group-wheel-1.2.2\egg-dist-tmp-z61a4h8n
zip_safe flag not set; analyzing archive contents...
removing 'c:\program files\python310\lib\site-packages\click_default_group_wheel-1.2.2-py3.10.egg' (and everything under it)
Copying click_default_group_wheel-1.2.2-py3.10.egg to c:\program files\python310\lib\site-packages
click-default-group-wheel 1.2.2 is already the active version in easy-install.pth

Installed c:\program files\python310\lib\site-packages\click_default_group_wheel-1.2.2-py3.10.egg
error: The 'click' distribution was not found and is required by click-default-group-wheel, sqlite-utils

Codecov Report

Merging #1759 (b901bb0) into main (2e97516) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1759   +/-   ##
=======================================
  Coverage   91.67%   91.67%           
=======================================
  Files          36       36           
  Lines        4658     4658           
=======================================
  Hits         4270     4270           
  Misses        388      388           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e97516...b901bb0. Read the comment docs.

That would be handy (additional where filters) but I think the trick with the with statement is already an order of magnitude better than what I had thought of, so my problem is solved by it (plus I got to learn about with today!)

Codecov Report

Merging #1753 (23a8515) into main (2e97516) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1753   +/-   ##
=======================================
  Coverage   91.67%   91.67%           
=======================================
  Files          36       36           
  Lines        4658     4658           
=======================================
  Hits         4270     4270           
  Misses        388      388           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e97516...23a8515. Read the comment docs.

I ran into this. My use case has a JSON file with array of book objects with a key called reviews which is also an array of objects. My JSON is human-edited and does not specify IDs for either books or reviews. Because sqlite-utils does not support inserting nested objects, I instead have to maintain two separate CSV files with id column in books.csv and book_id column in reviews.csv.

I think the right way to declare the relationship while inserting a JSON might be to describe the relationship:

sqlite-utils insert data.db books mydata.json --hasmany reviews --hasone author --manytomany tags

This is relying on the assumption that foreign keys can point to rowid primary key.

Closing in favor of existing issue #1298.

This code works:

import csv
import sqlite_utils
db = sqlite_utils.Database("test.db")
reader = csv.DictReader(open("csv", encoding="utf-16-le").read().split("\r\n"), delimiter=";")
db["test"].insert_all(reader, pk="Id")

I used iconv to change the encoding; sqlite-utils can import the resulting file, even if it stops at 98 %:

sqlite-utils insert --csv test test.db clean 
  [------------------------------------]    0%
  [###################################-]   98%  00:00:00

I can confirm. This only happens with sqlite-utils. I am using gnome-terminal with bash.

Argument were specified in the wrong order. PATH TABLE FILE can be misleading :)

I tried to debug it using pdb, but it looks sqlite-utils catches the exception, so it is not quick to figure out where the failure is happening.

Codecov Report

Merging #1693 (65a5d5e) into main (1465fea) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 65a5d5e differs from pull request most recent head ec2d1e4. Consider uploading reports for the commit ec2d1e4 to get more accurate results

@@           Coverage Diff           @@
##             main    #1693   +/-   ##
=======================================
  Coverage   91.67%   91.67%           
=======================================
  Files          36       36           
  Lines        4658     4658           
=======================================
  Hits         4270     4270           
  Misses        388      388           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d33fd0...ec2d1e4. Read the comment docs.

Hi, I'm trying to deploy my site using elasticbeanstalk and I keep getting this same error :
deterministic=True requires SQLite 3.8.3 or higher

I saw your previous solution that involves editing sqlite-utils/sqlite_utils/db.py file, but I'm curious as to how that will work in production.

I'd be happy to write a PR for this, if you think it's worth having.

user report: I found this issue because the first time I tried to use datasette for real, I displayed a large table, and thought there was no horizontal scroll bar at all. I didn't even consider that I had to scroll all the way to the end of the page to find it.

Just chipping in to say that this confused me, and I didn't even find the scroll bar until after I saw this issue. I don't know what the right answer is, but IMO the UI should suggest to the user that there is a way to view the data that's hidden to the right.

Thanks for addressing this @simonw! However, I just reinstalled sqlite-utils 3.26.1 and get an ParserError: Unknown string format: None:

sqlite-utils --version
sqlite-utils, version 3.26.1

sqlite-utils convert idfpr.db license "Original Issue Date" "r.parsedate(value)"
Traceback (most recent call last):
  File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/db.py", line 2514, in convert_value
    return fn(v)
  File "<string>", line 2, in fn
  File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/recipes.py", line 19, in parsedate
    parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst)
  File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 1374, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 649, in parse
    raise ParserError("Unknown string format: %s", timestr)
dateutil.parser._parser.ParserError: Unknown string format: None
Traceback (most recent call last):
  File "/home/matt/.local/bin/sqlite-utils", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/cli.py", line 2707, in convert
    db[table].convert(
  File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/db.py", line 2530, in convert
    self.db.execute(sql, where_args or [])
  File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/db.py", line 463, in execute
    return self.conn.execute(sql, parameters)
sqlite3.OperationalError: user-defined function raised exception

I definitely have some invalid data in the db. Happy to send a copy if it's helpful.

So, the good news is that it appears that setting one of those PRAGMA statements fixed the issue of table.extract() method call on this large database completing (that I described above.) The bad news is that I'm not sure which one!

I wonder if it's something system / environment specific about SQLite, or maybe something else going on.

I haven’t set one up unfortunately. My time is very limited because we just had a baby.

On Mon, May 2, 2022, at 6:42 PM, Simon Willison wrote:

Thanks, this definitely sounds like a bug. Do you have simple steps to reproduce this?

—
Reply to this email directly, view it on GitHub https://github.com/simonw/datasette/issues/1732#issuecomment-1115533820, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAM3KIY5L6FENZ22XANTHDVICAAXANCNFSM5UYOTKQA.
You are receiving this because you authored the thread.Message ID: @.***>

Also ran into this issue today using datasette package. The stack trace takes up my whole PowerShell history, though (recursionerror), but it also concerns the temporary directory.
Our development machines have a very zealous scanner that appears to insert itself between every call to the filesystem. I suspected that was causing some racing, but this turned out not to be the case: inserting time.sleep(3) on line 451 of datasette/datasette/utils/__init__.py does not make the problem go away. Commenting out the tmp.cleanup() line does.

The next error I get is docker-specific, so that probably does resolve the Datasette error here.

I have verified that the problem with base_url still exists in the latest version 0.61.1. I would need some guidance if my code change suggestion is correct or if base_url should be included in some other code?

Just stumbled across this, wondering why none of my deletes were working.

I should try seeing what happens with WAL mode enabled.

I've only skimmed above but it looks like you're doing mainly read-only queries? WAL mode is about better interactions between writers & readers, primarily.

You don't want to re-use an SQLite connection from multiple threads anyway: https://www.sqlite.org/threadsafe.html

Multiple connections can operate on the file in parallel, but a single connection can't:

Multi-thread. In this mode, SQLite can be safely used by multiple threads provided that no single database connection is used simultaneously in two or more threads.

(emphasis mine)

Codecov Report

Merging #1717 (9b9a314) into main (d57c347) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1717   +/-   ##
=======================================
  Coverage   91.75%   91.75%           
=======================================
  Files          34       34           
  Lines        4574     4575    +1     
=======================================
+ Hits         4197     4198    +1     
  Misses        377      377           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d57c347...9b9a314. Read the comment docs.

tfw you have more ephemeral storage than upstream bandwidth

FROM python:3.10-slim AS base

RUN apt update && apt -y install zstd

ENV DATASETTE_SECRET 'sosecret'
RUN --mount=type=cache,target=/root/.cache/pip
    pip install -U datasette datasette-pretty-json datasette-graphql

ENV PORT 8080
EXPOSE 8080

FROM base AS pack

COPY . /app
WORKDIR /app

RUN datasette inspect --inspect-file inspect-data.json
RUN zstd --rm *.db

FROM base AS unpack

COPY --from=pack /app /app
WORKDIR /app

CMD ["/bin/bash", "-c", "shopt -s nullglob && zstd --rm -d *.db.zst && datasette serve --host 0.0.0.0 --cors --inspect-file inspect-data.json --metadata metadata.json --create --port $PORT *.db"]

feels related to #1402

@simonw : do you think this could be merged ?

What would be an awesome feature as a plugin would be to be able to save a query (and possibly even results) to a github gist. Being able to share results that way would be super fantastic. Possibly even in Jupyter Notebook format (since github and github gists nicely render those)!

I know there's the handy datasette-saved-queries plugin, but a button that could export stuff out and then even possibly import stuff back in (I'm sort of thinking the way that Google Colab allows you to save to github, and then pull the notebook back in is a really great workflow

https://github.com/cincinnatilibrary/collection-analysis/blob/master/reports/colab_datasette_example.ipynb )

Codecov Report

Merging #1703 (73aabe6) into main (90d1be9) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1703   +/-   ##
=======================================
  Coverage   91.75%   91.75%           
=======================================
  Files          34       34           
  Lines        4573     4573           
=======================================
  Hits         4196     4196           
  Misses        377      377           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90d1be9...73aabe6. Read the comment docs.

another way to get the behavior of downloading the file is to use the download attribute of the anchor tag

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#attr-download

@simonw I like that you want to go "slim by default". Do you want another PR for that or should I just wait?

@simonw Thanks for looking into it and documenting the solution!

Codecov Report

Merging #1694 (83ff967) into main (e73fa72) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1694   +/-   ##
=======================================
  Coverage   91.74%   91.74%           
=======================================
  Files          34       34           
  Lines        4565     4565           
=======================================
  Hits         4188     4188           
  Misses        377      377           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e73fa72...83ff967. Read the comment docs.

Sure, it is from the documentation example:
Extracting columns into a separate table

wget "https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true"

sqlite-utils insert global.db power_plants \
    'global_power_plant_database.csv?raw=true' --csv
# Extract those columns:
sqlite-utils extract global.db power_plants country country_long \
    --table countries \
    --fk-column country_id \
    --rename country_long name

Thank you! I went through the cookiecutter template, and published my first package here: https://github.com/hydrosquall/datasette-nteract-data-explorer

Is there a way to serve a static assets when using the plugins/ directory method instead of installing plugins as a new python package?

As a workaround, I found I can serve my statics from a non-plugin specific folder using the --static CLI flag.

datasette ~/Library/Safari/History.db \
  --plugins-dir=plugins/ \
  --static assets:dist/

It's not ideal because it means I'll change the cache pattern path depending on how the plugin is running (via pip install or as a one off script), but it's usable as a workaround.

Codecov Report

Merging #1685 (933ce47) into main (c496f2b) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1685   +/-   ##
=======================================
  Coverage   91.74%   91.74%           
=======================================
  Files          34       34           
  Lines        4565     4565           
=======================================
  Hits         4188     4188           
  Misses        377      377           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c496f2b...933ce47. Read the comment docs.

would be nice if the behavior was

1. try to facet all the columns
2. for bigger tables try to facet the indexed columns
3. for the biggest tables, turn off autofacetting completely

This is based on my assumption that what determines autofaceting is the rarity of unique values. Which may not be true!

this has been addressed by the datasette-hashed-urls plugin

this has been obviated by the datasette-hashed-urls plugin

Codecov Report

Merging #419 (228f736) into main (93fa79d) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #419   +/-   ##
=======================================
  Coverage   96.55%   96.55%           
=======================================
  Files           6        6           
  Lines        2498     2498           
=======================================
  Hits         2412     2412           
  Misses         86       86           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 93fa79d...228f736. Read the comment docs.

That makes sense; just a little hint that points folks towards doing the right thing might be helpful!

fwiw, the reason I was using jq in the first place was just a quick way to extract one attribute from an actual JSON array. When I initially imported it, I got a table with a bunch of embedded JSON values, rather than a native table, because each array entry had two attributes, one with the data I actually wanted. Not sure how common a use-case this is, though (and easily fixed, aside from the jq weirdness!)

This line needs to say "MyActivity.json" instead of "My Activity.json". Google must have changed the file name.

Update: this appears to be because of running the command twice without clearing the DB in between. Tries to insert a Workout that already exists, causing a collision on the (auto-generated) id column. Had a different error with a clean DB, likely due to the workout points format; will make a new issue for that.

I have this issue, too, with a fresh export. None of my Workout entries in export.xml have an id key, though the sample export.xml in the tests folder doesn’t either, so I don’t think this is the culprit. Indeed, it seems @simonw is using the hash_id function from sqlite_utils, which creates a column (id, in this case) based on a hash of the row’s contents.

When I run the script, a workouts table is created, with one entry: my first workout. No workout_points table is created, as I’d expect from utils.py. I then get essentially the same error as noted in this thread:

Importing from HealthKit [###################################-] 98% 00:00:01 Traceback (most recent call last): File "/Users/lchski/.pyenv/versions/3.10.3/bin/healthkit-to-sqlite", line 8, in <module> sys.exit(cli()) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/cli.py", line 57, in cli convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/utils.py", line 34, in convert_xml_to_sqlite workout_to_db(el, db, zipfile) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/healthkit_to_sqlite/utils.py", line 57, in workout_to_db pk = db["workouts"].insert(record, alter=True, hash_id="id").last_pk File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py", line 2822, in insert return self.insert_all( File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py", line 2950, in insert_all self.insert_chunk( File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py", line 2715, in insert_chunk result = self.db.execute(query, params) File "/Users/lchski/.pyenv/versions/3.10.3/lib/python3.10/site-packages/sqlite_utils/db.py", line 458, in execute return self.conn.execute(sql, parameters) sqlite3.IntegrityError: UNIQUE constraint failed: workouts.id

Are there maybe duplicate workouts in the data, which’d cause multiple rows to share the same id? It’s strange, though, that no workout_points is created at all. Export created from iOS 15.3.1.

... unless your data had a column called n?

Exactly, that's highly likely even though I can't double check from this computer just now. Thanks!

Codecov Report

Merging #1659 (85dde28) into main (c10cd48) will increase coverage by 0.03%.
The diff coverage is 100.00%.

:exclamation: Current head 85dde28 differs from pull request most recent head 99b8263. Consider uploading reports for the commit 99b8263 to get more accurate results

@@            Coverage Diff             @@
##             main    #1659      +/-   ##
==========================================
+ Coverage   92.06%   92.10%   +0.03%     
==========================================
  Files          34       34              
  Lines        4576     4584       +8     
==========================================
+ Hits         4213     4222       +9     
+ Misses        363      362       -1     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c10cd48...99b8263. Read the comment docs.

Codecov Report

Merging #1656 (5d9883f) into main (c10cd48) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1656   +/-   ##
=======================================
  Coverage   92.06%   92.06%           
=======================================
  Files          34       34           
  Lines        4576     4576           
=======================================
  Hits         4213     4213           
  Misses        363      363           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c10cd48...5d9883f. Read the comment docs.

Ah, sorry, I didn't get what you were saying you the first time. Using _metadata_local in that way makes total sense -- I agree, refreshing metadata each cell was seeming quite excessive. Now I'm on the same page! :)

Thanks for taking the time to reply @brandonrobertz , this is really helpful info.

See "Many small queries are efficient in sqlite" for more information on the rationale here. Also note that in the datasette-live-config reference plugin, the DB connection is cached, so that eliminated most of the performance worries we had.

Ah, that's nifty! Yeah, then caching on the python side is likely a waste :) I'm new to working with sqlite so this is super good to know the many-small-queries is a common pattern

I tested on very large Datasette deployments (hundreds of DBs, millions of rows).

For my reference, did you include a render_cell plugin calling get_metadata in those tests? I'm less concerned now that I know a little more about sqlite's caching, but that special situation will jump you to a few orders of magnitude above what the sqlite article describes (e.g. 200 vs 20,000 queries+metadata merges for a page displaying 100 rows of a 200 column table). It wouldn't scale with db size as much as # of visible cells being rendered on the page, although they would be identical queries I suppose so will cache well.

(If you didn't test this specific situation, no worries -- I'm just trying to calibrate my intuition on this and can do my own benchmarks at some point.)

Simon talked about eventually making something like this a standard feature of Datasette

Yeah, getting metadata (and static pages as well for that matter) from internal tables definitely has my vote for including as a standard feature! Its really nice to be able to distribute a single *.db with all the metadata and static pages bundled. My metadata are sufficiently complex/domain specific that it makes sense to continue on my own plugin for now, but I'll be thinking about more general parts I can spin off as possible contributions to liveconfig (if you're open to them) or other plugins in this ecosystem.

@simonw

Now I get this:

(app-root) sqlite-utils indexes global.db --table
Error: near "(": syntax error
(app-root) sqlite-utils --version
sqlite-utils, version 3.25.1
(app-root) sqlite3 --version
3.36.0 2021-06-18 18:36:39
(app-root) python --version
Python 3.8.11

Dockerfile

FROM centos/python-38-centos7

USER root

RUN yum update -y
RUN yum upgrade -y


# epel
RUN yum -y install epel-release && yum clean all

# SQLite
RUN yum -y install zlib-devel geos geos-devel proj proj-devel freexl freexl-devel libxml2-devel 

WORKDIR /build/
COPY sqlite-autoconf-3360000.tar.gz ./
RUN tar -zxf sqlite-autoconf-3360000.tar.gz
WORKDIR /build/sqlite-autoconf-3360000
RUN ./configure
RUN make
RUN make install

# 
RUN /opt/app-root/bin/python3.8 -m pip install --upgrade pip
RUN pip install sqlite-utils

Awesome, thanks @brandonrobertz !

The plugin is close, but looks like it only grabs remote metadata, is that right? Instead what I'm wanting is to grab metadata embedded in the attached databases. Rather than extending that plugin, at this point I've realized I need a lot more flexibility in metadata for my data model (esp around formatting cell values and custom file exports) so rather than extending that I'll continue working on a plugin specific to my app.

If I'm understanding your plugin code correctly, you query the db using the sync handle every time get_metdata is called, right? Won't this become a pretty big bottleneck if a hook into render_cell is trying to read metadata / plugin config?

Making the get_metadata async won't improve the situation by itself as only some of the code paths accessing metadata use that hook. The other paths use the internal metadata dict.

I agree -- because things like render_cell will potentially want to read metadata/config, get_metadata should really remain sync and lightweight, which we can do with something like the remote-metadata plugin that could also poll metadata tables in attached databases.

That leaves your app, where it sounds like you want changes made by the user in the browser in to be immediately reflected, rather than have to wait for the next metadata refresh. In this case I wonder if you could have your app make a sync write to the datasette object so the change would have the immediate effect, but then have a separate async polling mechanism to eventually write that change out to the database for long-term persistence. Then you'd have the best of both worlds, I think? But probably not worth the trouble if your use cases are small (and/or you're not reading metadata/config from tight loops like render_cell).

Ok, I'm taking a slightly different approach, which I think is sort of close to the in-memory _metadata table idea.

I'm using a startup hook to load metadata / other info from the database, which I store in the datasette object for later:

@hookimpl
def startup(datasette):
    async def inner():
        datasette._mypluginmetadata = # await db query
    return inner

Then, I can use this in other plugins:

@hookimpl
def render_cell(value, column, table, database, datasette):
    # use datasette._mypluginmetadata

For my app I don't need anything to update dynamically so it's fine to pre-populate everything on startup. It's also good to have things precached especially for a hook like render_cell, which would otherwise require a ton of redundant db queries.

Makes me wonder if we could take a sort of similar caching approach with the internal _metadata table. Like have a little watchdog that could query all of the attached dbs for their _metadata tables every 5min or so, which then could be merged into the in memory _metadata table which then could be accessed sync by the plugins, or something like that.

For most the use cases I can think of, live updates don't need to take into effect immediately; refreshing a cache every 5min or on some other trigger (adjustable w a config setting) would be just fine.

I noticed the image was large when using fly. Is it possible to use a -slim base?

i increased the max_returned_row, because I have some scripts that get CSVs from this site, and this makes doing pagination of CSVs less annoying for many cases. i know that's streaming csvs is something you are hoping to address in 1.0. let me know if there's anything i can do to help with that.

as for what if anything can be done about the size of the dom, I don't have any ideas right now, but i'll poke around.

Looks like I'm late to the party here, but wanted to join the convo if there's still time before this interface is solidified in v1.0. My plugin use case is for education / social science data, which is meta-data heavy in the documentation of measurement scales, instruments, collection procedures, etc. that I want to connect to columns, tables, and dbs (and render in static pages, but looks like I can do that with the jinja plugin hook). I'm still digging in and I think @brandonrobertz 's approach will work for me at least for now, but I want to bump this thread in the meantime -- are there still plans for an async metadata hook at some point in the future? (or are you considering other directions?)

Codecov Report

Merging #1648 (32548b8) into main (7d24fd4) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1648      +/-   ##
==========================================
+ Coverage   92.03%   92.05%   +0.02%     
==========================================
  Files          34       34              
  Lines        4557     4570      +13     
==========================================
+ Hits         4194     4207      +13     
  Misses        363      363              

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7d24fd4...32548b8. Read the comment docs.

Codecov Report

Merging #1649 (59b2c16) into main (0499f17) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1649   +/-   ##
=======================================
  Coverage   92.03%   92.03%           
=======================================
  Files          34       34           
  Lines        4557     4557           
=======================================
  Hits         4194     4194           
  Misses        363      363           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0499f17...59b2c16. Read the comment docs.

Codecov Report

Merging #1602 (9eb0bdf) into main (5010d13) will increase coverage by 0.13%.
The diff coverage is n/a.

:exclamation: Current head 9eb0bdf differs from pull request most recent head a9c69dc. Consider uploading reports for the commit a9c69dc to get more accurate results

@@            Coverage Diff             @@
##             main    #1602      +/-   ##
==========================================
+ Coverage   92.03%   92.16%   +0.13%     
==========================================
  Files          34       34              
  Lines        4557     4531      -26     
==========================================
- Hits         4194     4176      -18     
+ Misses        363      355       -8     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 73f2d25...a9c69dc. Read the comment docs.

Probably too late… but I have just seen this because
http://simonwillison.net/2022/Mar/5/dash-encoding/#atom-everything

And it reminded me of comma tools at W3C.
http://www.w3.org/,tools

Example, the text version of W3C homepage
https://www.w3.org/,text

The challenge comes down to telling the difference between the following:

* `/db/table` - an HTML table page

/db/table

* `/db/table.csv` - the CSV version of `/db/table`

/db/table,csv

* `/db/table.csv` - no this one is actually a database table called `table.csv`

/db/table.csv

* `/db/table.csv.csv` - the CSV version of `/db/table.csv`

/db/table.csv,csv

* `/db/table.csv.csv.csv` and so on...

/db/table.csv.csv,csv

I haven't checked all the cases in the thread.

We do a lot of df.to_sql() to write into sqlite, mostly in this moddule

Thanks @simonw

I will test it after my vacation 👍

Is there already functionality that can be used to validate the metadata.json file? Is there a JSON Schema that defines it? Or a validation that's available via datasette with Python? We're working on automatically building the metadata in CI and when we deploy to cloud run, and it would be nice to be able to check whether the the metadata we're outputting is valid in our tests.

gonna close this for now since i'm not actively working on it.

maybe worth considering adding buttons for paren, asterisk, etc. under the input text box on mobile?

i seem to have fixed this issue by applying for elevated API access

On second thought any kind of quick-to-decompress-on-startup could be helpful if we're paying for the container registry and deployment bandwidth but not ephemeral storage.

On fly.io. This particular database goes from 1.4GB to 200M. Slower, part of that might be having no --inspect-file?

$ datasette publish fly ...   --generate-dir /tmp/deploy-this
...
$ mksquashfs large.db large.squashfs
$ rm large.db # don't accidentally put it in the image
$ cat Dockerfile
FROM python:3.8
COPY . /app
WORKDIR /app

ENV DATASETTE_SECRET 'xyzzy'
RUN pip install -U datasette
# RUN datasette inspect large.db --inspect-file inspect-data.json
ENV PORT 8080
EXPOSE 8080
CMD mount -o loop -t squashfs large.squashfs /mnt; datasette serve --host 0.0.0.0 -i /mnt/large.db --cors --port $PORT

It would also be possible to copy the file onto the ~6GB available on the ephemeral container filesystem on startup. A little against the spirit of the thing? On this example the whole docker image is 2.42 GB and the squashfs version is 1.14 GB.

Wondering if this actually relates to https://github.com/simonw/sqlite-utils/issues/402 ?

I also wonder if this would be a sensible approach for eg registering pint based quantity conversions into and out of the db, perhaps storing the quantity as a serialised magnitude measurement single column string?

My hunch is that the case where you want to consider input from more than one column will actually be pretty rare - the only case I can think of where I would want to do that is for latitude/longitude columns

Other possible pairs: unconventional date/datetime and timezone pairs eg 2022-02-16::17.00, London; or more generally, numerical value and unit of measurement pairs (eg if you want to cast into and out of different measurement units using packages like pint) or currencies etc. Actually, in that case, I guess you may be presenting things that are unit typed already, and so a conversion would need to parse things into an appropriate, possibly two column value, unit format.

The CustomColumnType() approach looks good. This pushes you into the mindspace that you are defining and working with a custom column type.

When creating the table, you could then error, or at least warn, if someone wasn't setting a column on a type or a custom column type, which I guess is where mypy comes in?

Codecov Report

Merging #407 (a974da5) into main (e7f0401) will increase coverage by 0.71%.
The diff coverage is 85.00%.

@@            Coverage Diff             @@
##             main     #407      +/-   ##
==========================================
+ Coverage   95.91%   96.62%   +0.71%     
==========================================
  Files           6        6              
  Lines        2421     2460      +39     
==========================================
+ Hits         2322     2377      +55     
+ Misses         99       83      -16     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e7f0401...a974da5. Read the comment docs.

I would love this merged!

Is this the correct solution to add the base_url row to url_builder.py?

Hi @simonw,

I've received some inquiries over the last year or so about Datasette and how it might be supported by Mangum. I maintain Mangum which is, as far as I know, the only project that provides support for ASGI applications in AWS Lambda.

If there is anything that I can help with here, please let me know because I think what Datasette provides to the community (even beyond OSS) is noble and worthy of special consideration.

Is there any progress elsewhere on the handling of compound / composite foreign keys, or is this PR still effectively open?

dddoooope

you can hack something like this to achieve this result:

sqlite-utils convert my_database my_table rowid "{'id': value}" --multi

if you are curious about prior art, https://github.com/jsnell/json-to-multicsv is really good!

Codecov Report

Merging #1631 (62eed84) into main (03305ea) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1631   +/-   ##
=======================================
  Coverage   92.19%   92.19%           
=======================================
  Files          34       34           
  Lines        4546     4546           
=======================================
  Hits         4191     4191           
  Misses        355      355           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 03305ea...62eed84. Read the comment docs.

Interesting that some accept an SRID and others do not - presumably GeomFromGeoJSON() always uses SRID=4326?

The ewtk/ewkb ones don't accept an SRID is because ewkt encodes the SRID in the string, so you would do this with a wkt string:

GeomFromText('POINT(529090 179645)', 27700)

but for ewkt it would be

GeomFromEWKT('SRID=27700;POINT(529090 179645)')

The specs for KML and GeoJSON specify a Coordinate Reference System for the format

• https://datatracker.ietf.org/doc/html/rfc7946#section-4
• https://docs.opengeospatial.org/is/12-007r2/12-007r2.html#1274

GML can specify the SRID in the XML at feature level e.g:

<gml:Point srsName="EPSG:27700">
  <gml:coordinates>529090, 179645</gml:coordinates>
</gml:Point>

There's a few more obscure formats in there, but broadly I think it is safe to assume an SRID param exists on the function for cases where the SRID is not implied by or specified in the input format.

Codecov Report

Merging #1629 (1c0d848) into main (1af1041) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1629   +/-   ##
=======================================
  Coverage   92.16%   92.16%           
=======================================
  Files          34       34           
  Lines        4531     4531           
=======================================
  Hits         4176     4176           
  Misses        355      355           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1af1041...1c0d848. Read the comment docs.

Codecov Report

Merging #385 (af86b17) into main (74586d3) will decrease coverage by 0.61%.
The diff coverage is 28.00%.

@@            Coverage Diff             @@
##             main     #385      +/-   ##
==========================================
- Coverage   96.52%   95.91%   -0.62%     
==========================================
  Files           6        6              
  Lines        2389     2421      +32     
==========================================
+ Hits         2306     2322      +16     
- Misses         83       99      +16     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74586d3...af86b17. Read the comment docs.

Codecov Report

Merging #1617 (af293c9) into main (2aa686c) will increase coverage by 0.06%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1617      +/-   ##
==========================================
+ Coverage   92.09%   92.16%   +0.06%     
==========================================
  Files          34       34              
  Lines        4518     4531      +13     
==========================================
+ Hits         4161     4176      +15     
+ Misses        357      355       -2     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2aa686c...af293c9. Read the comment docs.

Codecov Report

Merging #1626 (4b4d0e1) into main (b5e6b1a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1626   +/-   ##
=======================================
  Coverage   92.16%   92.16%           
=======================================
  Files          34       34           
  Lines        4531     4531           
=======================================
  Hits         4176     4176           
  Misses        355      355           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b5e6b1a...4b4d0e1. Read the comment docs.

Codecov Report

Merging #1622 (fbaf317) into main (8d5779a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1622   +/-   ##
=======================================
  Coverage   92.11%   92.11%           
=======================================
  Files          34       34           
  Lines        4525     4525           
=======================================
  Hits         4168     4168           
  Misses        357      357           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d5779a...fbaf317. Read the comment docs.

Thanks for looking into this. It might have been nice if explain surfaced these function calls. Looks like explain query plan does, but only for basic queries.

sqlite-utils fixtures.db 'explain query plan select * from pragma_function_list(), pragma_database_list(), pragma_module_list()' -t
  id    parent    notused  detail
----  --------  ---------  ------------------------------------------------
   4         0          0  SCAN pragma_function_list VIRTUAL TABLE INDEX 0:
   8         0          0  SCAN pragma_database_list VIRTUAL TABLE INDEX 0:
  12         0          0  SCAN pragma_module_list VIRTUAL TABLE INDEX 0:

sqlite-utils fixtures.db 'explain query plan select * from pragma_function_list() as fl, pragma_database_list() as dl, pragma_module_list() as ml' -t
  id    parent    notused  detail
----  --------  ---------  ------------------------------
   4         0          0  SCAN fl VIRTUAL TABLE INDEX 0:
   8         0          0  SCAN dl VIRTUAL TABLE INDEX 0:
  12         0          0  SCAN ml VIRTUAL TABLE INDEX 0:

Codecov Report

Merging #1616 (4ebe94b) into main (2aa686c) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1616   +/-   ##
=======================================
  Coverage   92.09%   92.09%           
=======================================
  Files          34       34           
  Lines        4518     4518           
=======================================
  Hits         4161     4161           
  Misses        357      357           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2aa686c...4ebe94b. Read the comment docs.

Oops, it feels like this should perhaps be migrated to GitHub Discussions - sorry! I don't think I have the ability to do that 😅

@simonw, could you review?

Or you can use https://sphinx-version-warning.readthedocs.io/! 😄

Codecov Report

Merging #1593 (df73ebb) into main (8c401ee) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1593   +/-   ##
=======================================
  Coverage   92.09%   92.09%           
=======================================
  Files          34       34           
  Lines        4516     4516           
=======================================
  Hits         4159     4159           
  Misses        357      357           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c401ee...df73ebb. Read the comment docs.

In my example, path matching happens at the application layer (being the Django channels URLRouter). That might be a somewhat exotic solution that would normally be solved by a proxy like Apache or Nginx. However, in my specific use case, this is a "feature" enabling me to do simple management of databases and metadata from within a Django admin app instance mapped in that same router.

Thank you for the quick reply! Just a quick observation, I am running this locally without a proxy, whereas your fly example seems to be running behind an apache proxy (if the name is accurate). Can it be that the apache proxy strips the prefix before it passes on the request to the daphne backend?

thanks so much! always a pleasure to see how you work through these things