issue_comments

@zschira is working with Pydantic while converting between and validating JSON frictionless datapackage descriptors that annotate an SQLite DB (extracted from FERC's XBRL data) and the Datasette YAML metadata so we can publish them with Datasette. Maybe there's some overlap? We've been loving Pydantic.

We do a lot of df.to_sql() to write into sqlite, mostly in this moddule

Is there already functionality that can be used to validate the metadata.json file? Is there a JSON Schema that defines it? Or a validation that's available via datasette with Python? We're working on automatically building the metadata in CI and when we deploy to cloud run, and it would be nice to be able to check whether the the metadata we're outputting is valid in our tests.

This looks awesome. We'll definitely make extensive use of this feature!

On Thu, Aug 12, 2021 at 5:52 PM Simon Willison @.***> wrote:

I like this. Need to solve for mobile though where the cog menu isn't visible - I think I'll do that with a definition list at the top of the page.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/simonw/datasette/issues/942#issuecomment-898022235, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAERSNYEF6QRZO2HRJGRIWDT4RGDFANCNFSM4QEC6ATA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

-- Zane A. Selvans, PhD Chief Data Wrangler Catalyst Cooperative https://catalyst.coop @.*** Signal/WhatsApp/SMS: +1 720 443 1363 Twitter: @ZaneSelvans https://twitter.com/ZaneSelvans PGP https://www.gnupg.org/: 0x64F7B56F3A127B04

Ah, okay so --extra-options applies to both datasette publish and datasette package? There wren't any examples of it being used with publish in the docs, so this tripped me up for a bit.

Are you thinking that those metadata tables would be added to the SQLite DB by Datasette, when you tell it to wrap up the database, with the metadata coming from the metadata.json? Would it be easy to allow the prepopulation of those tables in the database itself? We've been struggling with the best way to make sure that the data is always accompanied by metadata, and baking it all into the database itself would be nice, since then we wouldn't need to worry about separately distributing different files in different contexts.

Are there common patterns for storing column-based metadata inside SQLite itself? I know Postgres allows "comment" fields, which this is kind of trying to replicate. Should the units and description and possibly other per-column metadata fields be combined into a single (tabular?) structure, that would be displayed above the data on the table / query results page?