I'll add issues for both and do a documentation PR.

Interesting. I started a version using metadata like I outlined up top, but I realized that there's no documented way for a plugin to access either metadata or canned queries. Or at least, I couldn't find a way.

There is this method: https://github.com/simonw/datasette/blob/main/datasette/app.py#L472 but I don't want to rely on it if it's not documented. Same with this: https://github.com/simonw/datasette/blob/main/datasette/app.py#L544

If those are safe, I'll build on them. I'm also happy to document them, if that greases the wheels.

Counting any virtual tables can be pretty tricky. On one hand, counting a CSV virtual table would return the number of rows in the CSV, which is helpful (but can be I/O intensive). Counting a FTS5 virtual table would return the number of entries in the FTS index, which is kindof helpful, but can be misleading in some cases.

On the other hand, arbitrarily running COUNT(*) on some virtual tables can be incredibly expensive. SQLite offers new shortcuts/pushdowns on COUNT(*) queries for virtual tables, and instead calls the underlying vtab implementation and iterates through all rows in the table without discretion. For example, a virtual table that's backed by a Postgres table would call select * from pg_table, which would use up a lot of network and CPU calls. Or a virtual table backed by a google sheet would make network/API requests to get all the rows from the sheet just to make a count.

The pragma_table_list pragma tells you when a table is a regular table or virtual (in the type column), but was only added in version 3.37.0 (2021-11-27).

Personally, I wouldnt try to COUNT(*) virtual tables - it depends on how the virtual table is implemented, it requires that the connection has the proper extensions loaded, and it may accientally cause perf issues for new-age extensions. A few extensions that I'm writing have virtual tables that wouldn't benefit much from COUNT(*), and the fact that SQLite iterates through all rows in a table to count just makes things worse.

i wrote up a blog post of how i'm using it! https://bunkum.us/2022/11/20/mgdo-stack.html

Are you tracking the change to send the JSON over to the frontend separately or was that part of this? Something like this is probably pretty close https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25#diff-0c93232bfd5477eeac96382e52769108b41433d960d5277ffcccf2f464e60abdR9

Another idea would be to just not set a min-height and allow the 1 line input to be 1 line heigh

Maybe this is actually a problem in the python sqlite bindings. Given SQLITE's stance on this they should probably use encode('utf-8', 'surrogatepass'). As far as I understand the error here won't actually be resolved by this PR as-is. We would need to modify the data with surrogateescape... :/ or modify the sqlite3 module to use surrogatepass

why close? is the only problem that the _config table that incorrectly says 4 for fts5? if so, that's still something that should be fixed

amazing! thanks @simonw

This is one way to fix it

patch r.html diff --git a/datasette/static/cm-editor-6.0.1.js b/datasette/static/cm-editor-6.0.1.js index c1fd2ab..68cf398 100644 --- a/datasette/static/cm-editor-6.0.1.js +++ b/datasette/static/cm-editor-6.0.1.js @@ -22,7 +22,14 @@ export function editorFromTextArea(textarea, conf = {}) { // https://github.com/codemirror/lang-sql#user-content-sqlconfig.tables let view = new EditorView({ doc: textarea.value, + extensions: [ + EditorView.theme({ + ".cm-content": { + // Height on cm-content ensures the editor is focusable by clicking beyond the height of the text + minHeight: "70px", + }, + }), keymap.of([ { key: "Shift-Enter", diff --git a/datasette/templates/_codemirror.html b/datasette/templates/_codemirror.html index dea4710..c4629ae 100644 --- a/datasette/templates/_codemirror.html +++ b/datasette/templates/_codemirror.html @@ -4,7 +4,6 @@ .cm-editor { resize: both; overflow: hidden; - min-height: 70px; width: 80%; border: 1px solid #ddd; }

I don't love it but it seems to work for the default case. You can still retrigger the bug by resizing the editor to be > 70px high.

The other approach would be to listen for a click on that empty region and move focus to the editor, or something

Should we empty out the fixture schema to avoid fixture autocomplete showing up on live databases in the interim, or are you planning to tackle #1897 shortly?

Alright with https://github.com/simonw/datasette/pull/1893/commits/f254be4b38936e95e7a7f25866e7c6b0520db96f we should be getting autocomplete on fixture data. Give that a test and see what you think

I can push up a commit that uses the static fixtures schema for testing, but given that the query used to generate it is authed we would still need some work to make that work on live data, right? Ideally it could come down to db and query views directly to avoid waiting on an extra xhr and managing that state change.On Nov 16, 2022, at 2:16 PM, Simon Willison @.***> wrote: Honestly I'm not too bothered if table names with weird characters don't work correctly here - I care about those in the Datasette fixtures.db database because Datasette aims to support ANY valid SQLite database, so I need stuff in the test suite that includes weird edge cases like this. But I would hope very few people actually create tables with spaces in their names, so it's not a huge concern to me if autocompletion doesn't work properly for those.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

I think the table completion still has some quirks to work out. Something like

schema: { "[123_starts_with_digits]": ["content"], }

Seems to work alright, although it will append it after any other numbers you've started typing - so you end up with select * from 12[123_starts_with_digits] if you typed "12" to get the completion to appear. This might just be an issue with numeric names, I haven't tested it in a lot of detail.

You can do

searchable: [ { label: "name with . and spaces", apply: "[name with . and spaces]", }, "pk", "text1", "text2", ],

Which is pretty neat and will show the non-escaped string but complete to the escaped one. You can't easily do that with the table names themselves (you can pass a tables array like so https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/src/sql.ts#L121 but it will overwrite the columns from the schema ).

It's buggy enough (bad output for these unusual table names) that I'd suggest that work gets moved into a follow up to the upgrade to 6. That would give space to sort out how to deliver that to the view directly, figure out where name escaping should happen, and have overall testing to uncover bugs and fix papercuts before enabling it.

Alright, added Cmd+Enter to submit (Ctrl+Enter on Windows as well bc of using Meta-Enter on codemirror). We can make that MacOS only by changing the combo to Cmd+Enter specifically but I think it's probably fine to have both.

Or I guess you could return only the escaped table name and then we could derive the unescaped from the client side (removing the outer [] when present)

Nice. And is it possible to include another field which is an escaped table name (only when necessary) - i.e. [123_starts_with_digits]. Or is that easy enough to derive on the client? I'm thinking we'd map those to Completion objects so that CM would show the non escaped text but complete to escaped.

UI issue I see on the autocomplete popup with overlapping icon & text. Screenshot's from Firefox, it seems even a little more pronounced on Safari

I checked and if I empty out app.css the bug goes away, so there's some kind of inheritance issue there. It's hard to debug bc the autocomplete popup goes away on blur (i.e. when trying to inspect it in devtools), but at least it's narrowed down a bit.

For escaped table names it looks like we could pass a Completion object (https://codemirror.net/docs/ref/#autocomplete) instead of a string which would allow the non escaped name to be a label and then the escaped name to actually complete in the editor, which might help with some of the funkiness I was seeing w/ completion

With

patch diff --git a/datasette/templates/_codemirror_foot.html b/datasette/templates/_codemirror_foot.html index ed709b3..74fe18e 100644 --- a/datasette/templates/_codemirror_foot.html +++ b/datasette/templates/_codemirror_foot.html @@ -7,7 +7,11 @@ sqlFormat.hidden = false; } if (sqlInput) { - var editor = (window.editor = cm.editorFromTextArea(sqlInput)); + var editor = (window.editor = cm.editorFromTextArea(sqlInput, { + schema: { + compound_three_primary_keys: ["pk1", "pk2", "pk3", "content"], + }, + }));

we get table autocompletion and column completion if you name the table in the query (see screencast). I do see bugs with escaped table names like "'123_starts_with_digits'": ["col1", "col2"] or "[123_starts_with_digits]": ["col1", "col2"] where it doesn't seem to pick up the column names though. I think it needs some further testing and debugging.

https://user-images.githubusercontent.com/95570/202238521-e613b4e2-ba92-4418-9068-fc022edaee93.mp4

Ha, nice idea! Updating the dialect with that list.

I'm thinking of also adding count to the list since that's a common thing people would want to autocomplete. I notice BQ console highlights count in the same manner as other keywords like select as well.

I added a commit to make our own dialect at https://github.com/simonw/datasette/pull/1893/commits/e273fc8ed5341bdf0b622e722d761bd2acc30a90. Pulled in the full list of keywords from https://www.sqlite.org/lang_keywords.html but haven't gone through and pruned it to only include common select keywords. @simonw you'll have better knowledge than me on that - do you want to take a first shot at narrowing that down to the set that people will be using in the editor?

Have you ever seen CodeMirror correctly auto-completing columns? I'm not entirely sure I believe that the feature works anywhere else.

I was thinking of the BigQuery console, like

But they must be doing something pretty custom & appears to be using Monaco anyway. I suspect some kind of lower level autocomplete integration could make this work, but if the table completion is a good-enough starting point I think it's not too hard. The main issue is that we don't pass the relevant table data down to QueryView.

UI issue I see on the autocomplete popup with overlapping icon & text. Screenshot's from Firefox, it seems even a little more pronounced on Safari

The resize handle doesn't appear on Mobile Safari on iPhone - I don't think that particularly matters though.

The textarea does get a weird border around it when focused on iPhone though.

The default focus styles appear to be

.c1.cm-editor.cm-focused { outline: 1px dotted #212121; }

Which I also see on desktop. Would be nice to changed to whatever the default UA textarea styles are to blend in better but I wouldn't recommend removing it entirely - just to keep the visual indication that the element is focused. Maybe followup material to have a theming pass

If you can get a version of this working with table and column autocompletion just using a static JavaScript object in the source code with the right tables and columns, I'm happy to take on the work of turning that static object into something that Datasette includes in the page itself with all of the correct values.

This version "sort of" works when on the main database page where the template passes the relevant data https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25 by doing this and passing that into the schema object:

``` let TABLES_DATA = []; {% if tables is defined %}
TABLES_DATA = {{ tables | tojson(indent=2) }}; {% endif %}

// Turn into an object, shaped like https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/test/test-complete.ts#L27. const TABLES_SCHEMA = Object.fromEntries( new Map( TABLES_DATA.map((table) => { return [table.name, table.columns]; }) ).entries() ); ```

But there are a number of papercuts with it - it's not escaping table names with spaces (likely be fixable from the data being passed into the view) but mainly it doesn't seem to autocomplete columns. I think it might only want to do it when you first type the table name from my read of https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/test/test-complete.ts#L37. It's possible I'm just passing something wrong, but it may end up being something that needs feature work upstream.

Was just reviewing the SQL options and there's an upperCaseKeywords if we'd rather have SELECT vs select. Datasette seems to prefer lowercase so probably best to keep it as-is

I experimented with autocompleting the actual schema in https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25, but it would need some work (current problems with it listed in the commit message there)

https://github.com/Sphinxxxx/cm-resize isn't compatible with 6. There's a suggestion to try using CSS resize in https://discuss.codemirror.net/t/resizing-codemirror-6/3265/2

extraKeys is done - Shift+Enter is added in the helper function, and it appears that the Tab behavior now defaults to what the Tab: false setting was doing (allowing it to escape to the form)

Should also minify the bundled output

This probably counts as a case study: https://github.com/eyeseast/spatial-data-cooking-show. Even has video.

Seriously, though, this workflow has become integral to my work with reporters and editors across USA TODAY Network. Very often, I get sent a folder of data in mixed formats, with a vague ask of how we should communicate some part of it to users. Datasette and its constellation of tools makes it easy to get a quick look at that data, run exploratory queries, map it and ask questions to figure out what's important to show. And then I export a version of the data that's exactly what I need for display.

I'm realizing I don't know if a virtual table will ever return a count. Maybe it depends on the implementation. For these three, just checking now, it'll always return zero.

That said, I'm not sure there's any downside to having them return zero and caching that. (They're hidden, too.)

It does work, though I think it's probably still worth excluding virtual tables that will always be zero. Here's the same inspection as before, now with --load-extension spatialite:

json { "alltheplaces": { "hash": "0843cfe414439ab903c22d1121b7ddbc643418c35c7f0edbcec82ef1452411df", "size": 963375104, "file": "alltheplaces.db", "tables": { "spatial_ref_sys": { "count": 6215 }, "spatialite_history": { "count": 18 }, "sqlite_sequence": { "count": 2 }, "geometry_columns": { "count": 3 }, "spatial_ref_sys_aux": { "count": 6164 }, "views_geometry_columns": { "count": 0 }, "virts_geometry_columns": { "count": 0 }, "geometry_columns_statistics": { "count": 3 }, "views_geometry_columns_statistics": { "count": 0 }, "virts_geometry_columns_statistics": { "count": 0 }, "geometry_columns_field_infos": { "count": 0 }, "views_geometry_columns_field_infos": { "count": 0 }, "virts_geometry_columns_field_infos": { "count": 0 }, "geometry_columns_time": { "count": 3 }, "geometry_columns_auth": { "count": 3 }, "views_geometry_columns_auth": { "count": 0 }, "virts_geometry_columns_auth": { "count": 0 }, "data_licenses": { "count": 10 }, "sql_statements_log": { "count": 0 }, "states": { "count": 56 }, "counties": { "count": 3234 }, "idx_states_geometry_rowid": { "count": 56 }, "idx_states_geometry_node": { "count": 3 }, "idx_states_geometry_parent": { "count": 2 }, "idx_counties_geometry_rowid": { "count": 3234 }, "idx_counties_geometry_node": { "count": 98 }, "idx_counties_geometry_parent": { "count": 97 }, "idx_places_geometry_rowid": { "count": 1236796 }, "idx_places_geometry_node": { "count": 38163 }, "idx_places_geometry_parent": { "count": 38162 }, "places": { "count": 1332609 }, "SpatialIndex": { "count": 0 }, "ElementaryGeometries": { "count": 0 }, "KNN": { "count": 0 }, "idx_states_geometry": { "count": 56 }, "idx_counties_geometry": { "count": 3234 }, "idx_places_geometry": { "count": 1236796 } } } }

Hi @simonw and thanks for the great tools you're publishing, your dedication is inspiring!

I work for the French Ministry of Culture on a surveying tool for objects protected for their historical value. It is part of a program building modern public services called beta.gouv.fr.

In that context I'm using data published by the Ministry that I have ingested into datasette and published on a free Fly instance : https://collectif-objets-datasette.fly.dev . I have also ingested another data set with infos about french cities on this instance so that I can perform joined queries.

The surveying tool synchronizes its data regularly from this datasette instance, and I also use it to perform queries when asked generic questions about the distribution of objects. (The data is not very accessible as it's undocumented and for internal usage mostly)

Here's how to get a list of virtual tables: https://stackoverflow.com/questions/46617118/how-to-fetch-names-of-virtual-tables

Realized the API explorer doesn't need the API key piece at all - it can work with standard cookie-based auth.

This also reflects how most plugins are likely to use this API, where they'll be adding JavaScript that uses fetch() to call the write API directly.

I agree (that's what I did with the previous insert plugin), maybe a complete example using fetch() in the documentation would be valuable as a “Getting started with the API” or similar?

I ran PRAGMA integrity_check and it returned ok. but then I tried restoring from a backup and I didn't get this IntegrityError: constraint failed error. So I think it was just something wrong with my database. If it happens again I will first try to reindex and see if that fixes the issue

Even more bizarre, the source db only has one record and the target table has no conflicting record:

875 0.3s lb:/ (main|✚2) [0|0]🌺 sqlite-utils tube_71.db 'select * from media where path = "https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz"' | jq [ { "size": null, "time_created": null, "play_count": 1, "language": null, "view_count": null, "width": null, "height": null, "fps": null, "average_rating": null, "live_status": null, "age_limit": null, "uploader": null, "time_played": 0, "path": "https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz", "id": "088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz/074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv", "ie_key": "ArchiveOrg", "playlist_path": "https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz", "duration": 1424.05, "tags": null, "title": "074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv" } ] 875 0.3s lb:/ (main|✚2) [0|0]🥧 sqlite-utils video.db 'select * from media where path = "https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz"' | jq []

I've been able to use this code successfully several times before so not sure what's causing the issue.

I guess the way that I'm handling multiple databases is an issue, though it hasn't ever inserted into the source db, not sure what's different. The only reasonable explanation is that it is trying to insert into the source db from the source db for some reason? Or maybe sqlite3 is checking the source db for primary key violation because the table name is the same

where did you see the limit as 999? I believe the limit has been 32766 for quite some time. If you could detect which one this could speed up batch insert of some types of data significantly

Ideally people could fix their data if they run into this issue.

If you are using filenames try convmv

convmv --preserve-mtimes -f utf8 -t utf8 --notest -i -r .

maybe this script will also help:

```py import argparse, shutil from pathlib import Path

import ftfy

from xklb import utils from xklb.utils import log

def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument("paths", nargs='*') parser.add_argument("--verbose", "-v", action="count", default=0) args = parser.parse_args()

log.info(utils.dict_filter_bool(args.__dict__))
return args

def rename_invalid_paths() -> None: args = parse_args()

for path in args.paths:
    log.info(path)
    for p in sorted([str(p) for p in Path(path).rglob("*")], key=len):
        fixed = ftfy.fix_text(p, uncurl_quotes=False).replace("\r\n", "\n").replace("\r", "\n").replace("\n", "")
        if p != fixed:
            try:
                shutil.move(p, fixed)
            except FileNotFoundError:
                log.warning("FileNotFound. %s", p)
            else:
                log.info(fixed)

if name == "main": rename_invalid_paths() ```

some good concrete use-cases in mind

I actually found myself wanting something like this the past couple days. The use-case was databases with slightly different schema but same table names.

I wonder if something like this would be enough for most people:

``` args = argparse.Namespace(source_dbs = ['dogs.db', 'cats.db']) target_db = sqlite_utils.Database("animals.db")

for source_db in args.source_dbs: db = sqlite_utils.Database(source_db) for table in db.table_names(): db[table].upsert_all(target_db[table].rows, alter=True) ```

The complexity grows from there. I can think of a lot of things that could go wrong. Maybe it would be best to always output to a brand new db so that people don't accidentally destroy their target.db when they make a mistake

here is a full script:

``` import argparse from pathlib import Path

from sqlite_utils import Database

def connect(args, conn=None, kwargs) -> Database: db = Database(conn or args.database, kwargs) with db.conn: db.conn.execute("PRAGMA main.cache_size = 8000") return db

def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument("database") parser.add_argument("dbs_folder") parser.add_argument("--db", "-db", help=argparse.SUPPRESS) parser.add_argument("--verbose", "-v", action="count", default=0) args = parser.parse_args()

if args.db:
    args.database = args.db
Path(args.database).touch()
args.db = connect(args)

return args

def merge_db(args, source_db): source_db = str(Path(source_db).resolve())

s_db = connect(argparse.Namespace(database=source_db, verbose = args.verbose))
for table in s_db.table_names():
    data = s_db[table].rows
    args.db[table].insert_all(data, alter=True, replace=True)

args.db.conn.commit()

def merge_directory(): args = parse_args() source_dbs = list(Path(args.dbs_folder).glob('*.db')) for s_db in source_dbs: merge_db(args, s_db)

if name == 'main': merge_directory() ```

I updated datasette-publish-vercel to 0.14.2 in requirements.txt

And the site is back up!

Is there a way that we can get some sort of notice when something like this will have critical impact on website function?

should this issue be under https://github.com/simonw/datasette-publish-vercel/issues ?

Perhaps I just need to update: datasette-publish-vercel==0.11 in requirements.txt?

I'll try that and see what happens...

Are you saying that I can build a container, but then when I run it and it does datasette serve -i data.db ... it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that serve command?

Somehow, datasette serve -i data.db will lead to the data.db being modified, which will trigger a copy-on-write of data.db into the read-write layer of the container.

I don't understand how that happens.

it kind of feels like a bug in sqlite, but i can't quite follow the sqlite code.

as far as i can tell, this is where the "immutable" argument is used in sqlite:

c pPager->noLock = sqlite3_uri_boolean(pPager->zFilename, "nolock", 0); if( (iDc & SQLITE_IOCAP_IMMUTABLE)!=0 || sqlite3_uri_boolean(pPager->zFilename, "immutable", 0) ){ vfsFlags |= SQLITE_OPEN_READONLY; goto act_like_temp_file; }

so it does set the read only flag, but then has a goto.

Relevant sqlite forum thread: https://www.sqlite.org/forum/forumpost/02f7bda329f41e30451472421cf9ce7f715b768ce3db02797db1768e47950d48

Yeah, every time I see something cool done with triggers, I remember that I need to start using triggers.

This could use a new plugin hook, too. I don't want to complicate your life too much, but for things like GIS, I'd want a way to turn regular JSON into SpatiaLite geometries or combine X/Y coordinates into point geometries and such. Happy to help however I can.

@eyeseast Maybe you could do this with triggers? Like you can insert JSON-friendly data into a "raw" table, and create a trigger that transforms that inserted data into the proper table

Here's an example:

```sql -- meant to be updated from a Datasette insert create table points_raw(longitude int, latitude int);

-- the target table with proper spatliate geometries create table points(point geometry);

CREATE TRIGGER insert_points_raw INSERT ON points_raw BEGIN insert into points(point) values (makepoint(new.longitude, new.latitude)) END; ```

You could then POST a new row to points_raw like this: POST /db/points_raw Authorization: Bearer xxx Content-Type: application/json { "row": { "longitude": 27.64356, "latitude": -47.29384 } }

Then SQLite with run the trigger and insert a new row in points with the correct geometry point. Downside is you'd have duplicated data with points_raw, but maybe it could be a TEMP table (or have a cron that deletes all rows from that table every so often?)

The problem with how it is currently is that the transformed fts table will return incorrect results (unless the table was only 1 row or something), even if create_triggers was enabled previously. Maybe the simplest solution is to disable fts on a transformed table rather than try to recreate it? Thoughts?

That's reasonable. Canned queries and custom endpoints are certainly going to give more room for specific needs.

This could use a new plugin hook, too. I don't want to complicate your life too much, but for things like GIS, I'd want a way to turn regular JSON into SpatiaLite geometries or combine X/Y coordinates into point geometries and such. Happy to help however I can.

nevermind it was because I was running db[table].transform. The fts tables would still be there but the triggers would be dropped

when you are running from docker, you always will want to run as mode=ro because the same thing that is causing duplication in the inspect layer will cause duplication in the final container read/write layer when datasette serve runs.

@simonw, should i open up a new issue for investigating the differences between "immutable=1" and "mode=ro" and possibly switching to "mode=ro". Or would you like to keep that conversation in this issue?

switching from immutable=1 to mode=ro completely addressed this. see https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 for details.

yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in immutable mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!

running a test of that now.

this completely addressed #1480

the only mode that publish cloudrun supports right now is immutable

when i hack the connect method to open non mutable files with "mode=ro" and not "immutable=1" https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/database.py#L79

then:

bash 870 B RUN /bin/sh -c datasette inspect nlrb.db --inspect-file inspect-data.json

the datasette inspect layer is only the size of the json file!

yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in immutable mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!

running a test of that now.

and if we try immutable mode, which is how things are opened by datasette inspect we duplicate the files!!!

```python

test_sql_immutable.py

import sqlite3 import sys

db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}?immutable=1', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ```

fascinatingly, telling python to open sqlite in read only mode makes this layer have a size of 0

```python

test_sql_ro.py

import sqlite3 import sys

db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}?mode=ro', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ```

that's quite weird because setting the file permissions to read only didn't do anything. (on reflection, that chmod isn't doing anything because the dockerfile commands are run as root)

okay, some progress!! running some sql against a database file causes that file to get duplicated even if it doesn't apparently change the file.

make a little test script like this:

```python

test_sql.py

import sqlite3 import sys

db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ```

then

docker RUN python test_sql.py nlrb.db

produced a layer that's the same size as nlrb.db!!

it's not that the inspect command is somehow changing the db files. if i set them to only read-only, the "inspect" layer still has the same very large size.

i thought it was maybe to do with reading through all the files, but that does not seem to be the case

if i make a little test file like:

```python

test_read.py

import hashlib import sys import pathlib

HASH_BLOCK_SIZE = 1024 * 1024

def inspect_hash(path): """Calculate the hash of a database, efficiently.""" m = hashlib.sha256() with path.open("rb") as fp: while True: data = fp.read(HASH_BLOCK_SIZE) if not data: break m.update(data)

return m.hexdigest()

inspect_hash(pathlib.Path(sys.argv[1])) ```

then a line in the Dockerfile like

docker RUN python test_read.py nlrb.db && echo "[]" > /etc/inspect.json

just produes a layer of 3B

thanks @simonw, i'll spend a little more time trying to figure out why this isn't working on cloudrun, and then will flip over to fly if i can't.

as a next step, i'll fetch the docker image from the google registry, and see what memory and disk usage looks like when i run it locally.

running into this as i continue to grow my labor data warehouse.

Here a CloudRun PM says the container size should not count against memory: https://stackoverflow.com/a/56570717

oh weird. it inserts into db2

``` pytest xklb/check.py --pdb

xklb/check.py:11: in test_transaction assert list(db2["t"].rows) == [] E AssertionError: assert [{'foo': 1}] == [] E + where [{'foo': 1}] = list(<generator object Queryable.rows_where at 0x7f2d84d1f0d0>) E + where <generator object Queryable.rows_where at 0x7f2d84d1f0d0> = <Table t (foo)>.rows

entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /home/xk/github/xk/lb/xklb/check.py(11)test_transaction() 9 with db1.conn: 10 db1["t"].insert({"foo": 1}) ---> 11 assert list(db2["t"].rows) == [] 12 assert list(db2["t"].rows) == [{"foo": 1}] ```

It fails because it is already inserted.

btw if you put these two lines in you pyproject.toml you can get ipdb in pytest

[tool.pytest.ini_options] addopts = "--pdbcls=IPython.terminal.debugger:TerminalPdb --ignore=tests/data --capture=tee-sys --log-cli-level=ERROR"

this is likely what you are looking for: https://stackoverflow.com/a/51076749/697964

but yeah I would say just disable smart quotes

Just watched this video which demonstrates the integration of any webapp into JupyterLab: https://youtu.be/FH1dKKmvFtc

Maybe this is the answer?

if you went this route:

python with sqlite_timelimit(conn, time_limit_ms): c.execute(query) for chunk in c.fetchmany(chunk_size): yield from chunk

then time_limit_ms would probably have to be greatly extended, because the time spent in the loop will depend on the downstream processing.

i wonder if this was why you were thinking this feature would need a dedicated connection?

reading more, there's no real limit i can find on the number of active cursors (or more precisely active prepared statements objects, because sqlite doesn't really have cursors).

maybe something like this would be okay?

python with sqlite_timelimit(conn, time_limit_ms): c.execute(query) # step through at least one to evaluate the statement, not sure if this is necessary yield c.execute.fetchone() for chunk in c.fetchmany(chunk_size): yield from chunk

this seems quite weird that there's not more of limit of the number of active prepared statements, but i haven't been able to find one.

for teaching register_output_renderer to stream it seems like the two options are to

1. a nested query technique to paginate through
2. a fetching model that looks like something python with sqlite_timelimit(conn, time_limit_ms): c.execute(query) for chunk in c.fetchmany(chunk_size): yield from chunk currently db.execute is not a generator, so this would probably need a new method?

i think that max_returned_rows is a defense mechanism, just not for connection exhaustion. max_returned_rows is a defense mechanism against memory bombs.

if you are potentially yielding out hundreds of thousands or even millions of rows, you need to be quite careful about data flow to not run out of memory on the server, or on the client.

you have a lot of places in your code that are protective of that right now, but max_returned_rows acts as the final backstop.

so, given that, it makes sense to have removing max_returned_rows altogether be a non-goal, but instead allow for for specific codepaths (like streaming csv's) be able to bypass.

that could dramatically lower the surface area for a memory-bomb attack.

i think this feature would be safe, as its really only the time limit that can, and imo, should protect against long running queries, as it is pretty easy to make very expensive queries that don't return many rows.

moving away from max_returned_rows will requires some thinking about:

1. memory usage and data flows to handle potentially very large result sets
2. how to avoid rendering tens or hundreds of thousands of html rows.

this sql query doesn't trip up maximum_returned_rows but does timeout

sql with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter LIMIT 10 OFFSET 100000000

thanks @simonw, i learned something i didn't know about sqlite's execution model!

Imagine if Datasette CSVs did allow unlimited retrievals. Someone could hit the CSV endpoint for that recursive query and tie up Datasette's SQL connection effectively forever.

why wouldn't the sqlite_timelimit guard prevent that?

on my local version which has the code to turn off truncations for query csv, sqlite_timelimit does protect me.

i agree with that concern! but if i'm understanding the code correctly, maximum_returned_rows does not protect against long-running queries in any way.

the pattern in this PR max_returned_rows control the maximum rows rendered through html and json, and the csv render bypasses that.

i think it would be better to have each of these different query renderers have more direct control for how many rows to fetch, instead of relying on the internals of the execute method.

generally, users will not want to paginate through tens of thousands of results, but often will want to download a full query as json or as csv.

Right. The backup command will copy tables completely, but in the case of conflicting table names, the destination gets overwritten silently. That might not be what you want here.

This might be fairly straightforward using SQLite's backup utility: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.backup

i think the smallest change that gets close to what i want is to change the behavior so that max_returned_rows is not applied in the execute method when we are are asking for a csv of query.

there are some infelicities for that approach, but i'll make a PR to make it easier to discuss.

reading the database execute method i have a few questions.

https://github.com/simonw/datasette/blob/cb1e093fd361b758120aefc1a444df02462389a3/datasette/database.py#L229-L242

unless i'm missing something (which is very likely!!), the max_returned_rows argument doesn't actually offer any protections against running very expensive queries.

It's not like adding a LIMIT max_rows argument. it make sense that it isn't because, the query could already have an LIMIT argument. Doing something like select * from (query) limit {max_returned_rows} might be protective but wouldn't always.

Instead the code executes the full original query, and if still has time it fetches out the first max_rows + 1 rows.

this does offer some protection of memory exhaustion, as you won't hydrate a huge result set into python (however, there are data flow patterns that could avoid that too)

given the current architecture, i don't see how creating a new connection would be use?

If we just removed the max_return_rows limitation, then i think most things would be fine except for the QueryViews. Right now rendering, just 5000 rows takes a lot of client-side memory so some form of pagination would be required.

I think that paginating, even in javascript, could be very helpful. Maybe render json or csv into the page and let javascript loading that into the dom?

from your analysis, it seems like the GIL is blocking on loading of the data from sqlite to python, (particularly in the fetchmany call)

this is probably a simplistic idea, but what if you had the python code in the execute method iterate over the cursor and yield out rows or small chunks of rows.

something like:
python with sqlite_timelimit(conn, time_limit_ms): try: cursor = conn.cursor() cursor.execute(sql, params if params is not None else {}) except: ... max_returned_rows = self.ds.max_returned_rows if max_returned_rows == page_size: max_returned_rows += 1 if max_returned_rows and truncate: for i, row in enumerate(cursor): yield row if i == max_returned_rows - 1: break else: for row in cursor: yield row truncated = False

this kind of thing works well with a postgres server side cursor, but i'm not sure if it will hold for sqlite.

you would still spend about the same amount of time in python and would be contending for the gil, but it would be could be non blocking.

depending on the data flow, this could also some benefit for memory. (data stays in more compact sqlite-land until you need it)

Instead of outputting binary data to stdout the interface might be better like this

sqlite-utils merge animals.db cats.db dogs.db

similar to zip, ogr2ogr, etc

Actually I think this might already be possible within ogr2ogr. I don't believe spatial data is a requirement though it might add an ogc_id column or something

cp cats.db animals.db ogr2ogr -append animals.db dogs.db ogr2ogr -append animals.db another.db

While you are adding features, would you be future-proofing your APIs if you switched over some arguments over to keyword-only arguments or would that be too disruptive?

Thinking out loud:

async def render_template( self, templates, *, context=None, plugin_context=None, request=None, view_name=None ):

hi @simonw, this is becoming more of a bother for my labor data warehouse. Is there any research or a spike i could do that would help you investigate this issue?

When I run reset it fixes my terminal. I suspect it is related to the progress bar

https://linux.die.net/man/1/reset

950 1s /m/d/03_Downloads 🐑 echo $TERM xterm-kitty ▓░▒░ /m/d/03_Downloads 🌏 kitty -v kitty 0.26.2 created by Kovid Goyal $ sqlite-utils insert test.db facility facility-boundary-us-all.csv --csv blah blah blah (no offense) $ <no cursor> $ reset $ <cursor lives again (resurrection [explicit])>

oh and by writing this I just realized the difference: the URL on fly.io is with a custom SQL command whereas the local one is without. It seems that there is no pagination when using custom SQL commands which makes sense

Sorry for this useless issue, maybe this can be useful for someone else / me in the future.

Thanks again for this wonderful project !

A couple+ of possible use case examples:

• someone has a collection of articles indexed with FTS; they want to publish a simple search tool over the results;
• someone has an image collection and they want to be able to search over description text to return images;
• someone has a set of locations with descriptions, and wants to run a query over places and descriptions and get results as a listing or on a map;
• someone has a set of audio or video files with titles, descriptions and/or transcripts, and wants to be able to search over them and return playable versions of returned items.

In many cases, I suspect the raw content will be in one table, but the search table will be a second (eg FTS) table. Generally, the search may be over one or more joined tables, and the results constructed from one or more tables (which may or may not be distinct from the search tables).

Looks like jinja2 is no longer updatable, so this is no longer needed.

Looks like aiofiles is no longer updatable, so this is no longer needed.

Thanks for pointing me to the right place

Should passing alter=True also drop any columns that aren't included in the new table structure?

It could even spot column types that aren't correct and fix those.

Is that consistent with the expectations set by how alter=True works elsewhere?

I would lean towards not dropping them (or making a drop=True or drop_columns=Trueor drop_missing_columns=True) to work with existing tables easier.

I do like that sqlite-utils mostly just works with existing tables but it's also nice to add to existing fields in a few cases.

@simonw to build the extension on ubuntu, you can run:

apt-get update && apt-get install libsqlite3-dev gcc gcc ext.c -fPIC -shared -o ext.so

I'm not the best with Actions, but if you set the cache key to ext.c, run those two commands to download dependencies + compile to ext.so, then the unit test should pick it up and run it correctly. Let me know if you want me to update the PR with that added

Rebased, Read the docs failure should now now fixed

Re docs - ya that's a pretty ambitious page, I'm still not 100% sure what the best practices are/should be... Would be happy to make that page in a future PR

thanks @simonw!

maybe a simpler solution is to set the maxscale to like 2? since datasette is not set up to make use of container scaling anyway?