it probably does not make sense to gzip large sqlite database files on the fly. it can take many seconds to gzip a large file and you either have to have this big thing in memory, or write it to disk, which some deployment environments will not like.

i wonder if it would make sense to gzip the databases as part of the datasette publish process. it would be very cool to statically serve those as if they dynamically zipped (i.e. serve the filename example.db, not example.db.zip, and rely on the browser to expand).

@yejiyang why don’t you move this discussion to my fork to spare simon’s notifications

hi @yejiyang, have your tried using my fork of datasette: https://github.com/fgregg/datasette/tree/no_limit_csv_publish

I also often then run another little script to cast all empty strings to null, but i save that for another issue if this gets accepted.

i'll adopt this PR to make the changes @simonw suggested https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932

see this related discussion to a change in API in sqlite-utils https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932

My original idea for compound foreign keys was to turn both of those columns into links, but that doesn't fit here because database_name is already part of a different foreign key.

it's pretty hard to know what the right thing to do is if a field is part of multiple foreign keys.

but, if that's not the case, what about making each of the columns a link. seems like an improvement over the status quo.

I went ahead and spiked something together, in #2003

@simonw, let me know what you think about this approach!

hi @simonw, this bug bit me today.

the UX for linking from a table to the foreign key seems tough!

the design in the other direction seems a lot easier, for a given primary key detail page, add links back to the tables that refer to the row.

would you be open to a PR that solved the second problem but not the first?

is this issue closed by #1893?

@simonw is this issue closed by #1893?

i wrote up a blog post of how i'm using it! https://bunkum.us/2022/11/20/mgdo-stack.html

amazing! thanks @simonw

Are you saying that I can build a container, but then when I run it and it does datasette serve -i data.db ... it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that serve command?

Somehow, datasette serve -i data.db will lead to the data.db being modified, which will trigger a copy-on-write of data.db into the read-write layer of the container.

I don't understand how that happens.

it kind of feels like a bug in sqlite, but i can't quite follow the sqlite code.

as far as i can tell, this is where the "immutable" argument is used in sqlite:

c pPager->noLock = sqlite3_uri_boolean(pPager->zFilename, "nolock", 0); if( (iDc & SQLITE_IOCAP_IMMUTABLE)!=0 || sqlite3_uri_boolean(pPager->zFilename, "immutable", 0) ){ vfsFlags |= SQLITE_OPEN_READONLY; goto act_like_temp_file; }

so it does set the read only flag, but then has a goto.

Relevant sqlite forum thread: https://www.sqlite.org/forum/forumpost/02f7bda329f41e30451472421cf9ce7f715b768ce3db02797db1768e47950d48

when you are running from docker, you always will want to run as mode=ro because the same thing that is causing duplication in the inspect layer will cause duplication in the final container read/write layer when datasette serve runs.

@simonw, should i open up a new issue for investigating the differences between "immutable=1" and "mode=ro" and possibly switching to "mode=ro". Or would you like to keep that conversation in this issue?

switching from immutable=1 to mode=ro completely addressed this. see https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 for details.

yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in immutable mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!

running a test of that now.

this completely addressed #1480

the only mode that publish cloudrun supports right now is immutable

when i hack the connect method to open non mutable files with "mode=ro" and not "immutable=1" https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/database.py#L79

then:

bash 870 B RUN /bin/sh -c datasette inspect nlrb.db --inspect-file inspect-data.json

the datasette inspect layer is only the size of the json file!

yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in immutable mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!

running a test of that now.

and if we try immutable mode, which is how things are opened by datasette inspect we duplicate the files!!!

```python

test_sql_immutable.py

import sqlite3 import sys

db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}?immutable=1', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ```

fascinatingly, telling python to open sqlite in read only mode makes this layer have a size of 0

```python

test_sql_ro.py

import sqlite3 import sys

db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}?mode=ro', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ```

that's quite weird because setting the file permissions to read only didn't do anything. (on reflection, that chmod isn't doing anything because the dockerfile commands are run as root)

okay, some progress!! running some sql against a database file causes that file to get duplicated even if it doesn't apparently change the file.

make a little test script like this:

```python

test_sql.py

import sqlite3 import sys

db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ```

then

docker RUN python test_sql.py nlrb.db

produced a layer that's the same size as nlrb.db!!

it's not that the inspect command is somehow changing the db files. if i set them to only read-only, the "inspect" layer still has the same very large size.

i thought it was maybe to do with reading through all the files, but that does not seem to be the case

if i make a little test file like:

```python

test_read.py

import hashlib import sys import pathlib

HASH_BLOCK_SIZE = 1024 * 1024

def inspect_hash(path): """Calculate the hash of a database, efficiently.""" m = hashlib.sha256() with path.open("rb") as fp: while True: data = fp.read(HASH_BLOCK_SIZE) if not data: break m.update(data)

return m.hexdigest()

inspect_hash(pathlib.Path(sys.argv[1])) ```

then a line in the Dockerfile like

docker RUN python test_read.py nlrb.db && echo "[]" > /etc/inspect.json

just produes a layer of 3B

thanks @simonw, i'll spend a little more time trying to figure out why this isn't working on cloudrun, and then will flip over to fly if i can't.

as a next step, i'll fetch the docker image from the google registry, and see what memory and disk usage looks like when i run it locally.

running into this as i continue to grow my labor data warehouse.

Here a CloudRun PM says the container size should not count against memory: https://stackoverflow.com/a/56570717

if you went this route:

python with sqlite_timelimit(conn, time_limit_ms): c.execute(query) for chunk in c.fetchmany(chunk_size): yield from chunk

then time_limit_ms would probably have to be greatly extended, because the time spent in the loop will depend on the downstream processing.

i wonder if this was why you were thinking this feature would need a dedicated connection?

reading more, there's no real limit i can find on the number of active cursors (or more precisely active prepared statements objects, because sqlite doesn't really have cursors).

maybe something like this would be okay?

python with sqlite_timelimit(conn, time_limit_ms): c.execute(query) # step through at least one to evaluate the statement, not sure if this is necessary yield c.execute.fetchone() for chunk in c.fetchmany(chunk_size): yield from chunk

this seems quite weird that there's not more of limit of the number of active prepared statements, but i haven't been able to find one.

for teaching register_output_renderer to stream it seems like the two options are to

1. a nested query technique to paginate through
2. a fetching model that looks like something python with sqlite_timelimit(conn, time_limit_ms): c.execute(query) for chunk in c.fetchmany(chunk_size): yield from chunk currently db.execute is not a generator, so this would probably need a new method?

i think that max_returned_rows is a defense mechanism, just not for connection exhaustion. max_returned_rows is a defense mechanism against memory bombs.

if you are potentially yielding out hundreds of thousands or even millions of rows, you need to be quite careful about data flow to not run out of memory on the server, or on the client.

you have a lot of places in your code that are protective of that right now, but max_returned_rows acts as the final backstop.

so, given that, it makes sense to have removing max_returned_rows altogether be a non-goal, but instead allow for for specific codepaths (like streaming csv's) be able to bypass.

that could dramatically lower the surface area for a memory-bomb attack.

i think this feature would be safe, as its really only the time limit that can, and imo, should protect against long running queries, as it is pretty easy to make very expensive queries that don't return many rows.

moving away from max_returned_rows will requires some thinking about:

1. memory usage and data flows to handle potentially very large result sets
2. how to avoid rendering tens or hundreds of thousands of html rows.

this sql query doesn't trip up maximum_returned_rows but does timeout

sql with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter LIMIT 10 OFFSET 100000000

thanks @simonw, i learned something i didn't know about sqlite's execution model!

Imagine if Datasette CSVs did allow unlimited retrievals. Someone could hit the CSV endpoint for that recursive query and tie up Datasette's SQL connection effectively forever.

why wouldn't the sqlite_timelimit guard prevent that?

on my local version which has the code to turn off truncations for query csv, sqlite_timelimit does protect me.

i agree with that concern! but if i'm understanding the code correctly, maximum_returned_rows does not protect against long-running queries in any way.

the pattern in this PR max_returned_rows control the maximum rows rendered through html and json, and the csv render bypasses that.

i think it would be better to have each of these different query renderers have more direct control for how many rows to fetch, instead of relying on the internals of the execute method.

generally, users will not want to paginate through tens of thousands of results, but often will want to download a full query as json or as csv.

i think the smallest change that gets close to what i want is to change the behavior so that max_returned_rows is not applied in the execute method when we are are asking for a csv of query.

there are some infelicities for that approach, but i'll make a PR to make it easier to discuss.

reading the database execute method i have a few questions.

https://github.com/simonw/datasette/blob/cb1e093fd361b758120aefc1a444df02462389a3/datasette/database.py#L229-L242

unless i'm missing something (which is very likely!!), the max_returned_rows argument doesn't actually offer any protections against running very expensive queries.

It's not like adding a LIMIT max_rows argument. it make sense that it isn't because, the query could already have an LIMIT argument. Doing something like select * from (query) limit {max_returned_rows} might be protective but wouldn't always.

Instead the code executes the full original query, and if still has time it fetches out the first max_rows + 1 rows.

this does offer some protection of memory exhaustion, as you won't hydrate a huge result set into python (however, there are data flow patterns that could avoid that too)

given the current architecture, i don't see how creating a new connection would be use?

If we just removed the max_return_rows limitation, then i think most things would be fine except for the QueryViews. Right now rendering, just 5000 rows takes a lot of client-side memory so some form of pagination would be required.

I think that paginating, even in javascript, could be very helpful. Maybe render json or csv into the page and let javascript loading that into the dom?

from your analysis, it seems like the GIL is blocking on loading of the data from sqlite to python, (particularly in the fetchmany call)

this is probably a simplistic idea, but what if you had the python code in the execute method iterate over the cursor and yield out rows or small chunks of rows.

something like:
python with sqlite_timelimit(conn, time_limit_ms): try: cursor = conn.cursor() cursor.execute(sql, params if params is not None else {}) except: ... max_returned_rows = self.ds.max_returned_rows if max_returned_rows == page_size: max_returned_rows += 1 if max_returned_rows and truncate: for i, row in enumerate(cursor): yield row if i == max_returned_rows - 1: break else: for row in cursor: yield row truncated = False

this kind of thing works well with a postgres server side cursor, but i'm not sure if it will hold for sqlite.

you would still spend about the same amount of time in python and would be contending for the gil, but it would be could be non blocking.

depending on the data flow, this could also some benefit for memory. (data stays in more compact sqlite-land until you need it)

hi @simonw, this is becoming more of a bother for my labor data warehouse. Is there any research or a spike i could do that would help you investigate this issue?

thanks @simonw!

maybe a simpler solution is to set the maxscale to like 2? since datasette is not set up to make use of container scaling anyway?

hadley wickham's melt and reshape could be good inspo: http://had.co.nz/reshape/introduction.pdf

for example, i have data on votes that look like this:

| ballot_id | option_id | choice | |-|-|-| | 1 | 1 | 0 | | 1 | 2 | 1 | | 1 | 3 | 0 | | 1 | 4 | 1 | | 2 | 1 | 1 | | 2 | 2 | 0 | | 2 | 3 | 1 | | 2 | 4 | 0 |

and i want to reshape from this long form to this wide form:

| ballot_id | option_id_1 | option_id_2 | option_id_3 | option_id_ 4| |-|-|-|-| -| | 1 | 0 | 1 | 0 | 1 | | 2 | 1 | 0 | 1| 0 |

i could do such a think like this.

sql select ballot_id, sum(choice) filter (where option_id = 1) as option_id_1, sum(choice) filter (where option_id = 2) as option_id_2, sum(choice) filter (where option_id = 3) as option_id_3, sum(choice) filter (where option_id = 4) as option_id_4 from vote group by ballot_id

just ran into this!

feels related to #1402

another way to get the behavior of downloading the file is to use the download attribute of the anchor tag

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#attr-download

would be nice if the behavior was

1. try to facet all the columns
2. for bigger tables try to facet the indexed columns
3. for the biggest tables, turn off autofacetting completely

This is based on my assumption that what determines autofaceting is the rarity of unique values. Which may not be true!

this has been addressed by the datasette-hashed-urls plugin

this has been obviated by the datasette-hashed-urls plugin

i increased the max_returned_row, because I have some scripts that get CSVs from this site, and this makes doing pagination of CSVs less annoying for many cases. i know that's streaming csvs is something you are hoping to address in 1.0. let me know if there's anything i can do to help with that.

as for what if anything can be done about the size of the dom, I don't have any ideas right now, but i'll poke around.

maybe worth considering adding buttons for paren, asterisk, etc. under the input text box on mobile?

dddoooope

you can hack something like this to achieve this result:

sqlite-utils convert my_database my_table rowid "{'id': value}" --multi

if you are curious about prior art, https://github.com/jsnell/json-to-multicsv is really good!

thanks so much! always a pleasure to see how you work through these things

i don’t want to be such a partisan for analyze, but the query planner deciding not to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice.

<s>the original poster in that stack overflow doesn’t say there’s a performance regression </s>

or using “ pragma optimize”

the out-of-date scenario you describe could be addressed by automatically adding an analyze to the insert or convert commands if they implicate an index

the table with the query ran so bad was about 50k.

i think the scenario should not be worse than no stats.

i also did not know that sqlite was so different from postgres and needed an explicit analyze call.

for options 2 and 3, i would worry about discoverablity.

in other db’s it is not necessary to explicitly call analyze for most indices. ie for postgres

The system regularly collects statistics on all of a table's columns. Newly-created non-expression indexes can immediately use these statistics to determine an index's usefulness.

i suppose i would propose raising a warning if the stats table is created that explains what is going on and informs users about a —no-analyze argument.

is there a reason to not always use the slim option?

i added an index to one table with sqlite-utils, and then a query that used to take about 1 second started taking hundreds of seconds.

running analyze got me back to sub second speed.

if that is not desirable, it might be good to document that users might want to set up a lifecycle rule to automatically delete these build artifacts. something like https://stackoverflow.com/questions/59937542/can-i-delete-container-images-from-google-cloud-storage-artifacts-bucket

interesting! i love this feature. this + full caching with cloudflare is really super!

the nested query idea is very nice, and i stole if for my client side paginator. However, it won't do the right thing if the original query orders by random().

If you go the nested query route, maybe raise a 4XX status code if the query has such a clause?

these headers would also be relevant for json exports of custom queries

would also be good if the header said the what the max row limit was

just came by to open this issue. would make my data analysis in observable a lot better!

that option is not available on custom queries.

wow! that's awesome! thanks so much, @simonw!

i was just reaching for a tool to do this this morning

okay @simonw, made the requested changes. tests are running locally. i think this is ready for you to look at again.

i'm going to take a swing at this today. we'll see.

there's a lot of complexity here, that's probably not worth addressing. i got what i needed by patching the dockerfile that cloudrun uses to install a newer version of sqlite.

yes, filter clause on aggregate queries were added to sqlite3 in 3.30

ordered lists didn't work either, btw