issues

Reproducible code below:

```bash

echo 'create table bar (baz text, created_at timestamp default CURRENT_TIMESTAMP)' | sqlite3 foo.db sqlite3 foo.db SQLite version 3.36.0 2021-06-18 18:36:39 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE bar (baz text, created_at timestamp default CURRENT_TIMESTAMP); sqlite> .exit sqlite-utils transform foo.db bar --column-order baz sqlite3 foo.db SQLite version 3.36.0 2021-06-18 18:36:39 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE IF NOT EXISTS "bar" ( [baz] TEXT, [created_at] FLOAT DEFAULT 'CURRENT_TIMESTAMP' ); sqlite> .exit sqlite-utils transform foo.db bar --column-order baz sqlite3 foo.db SQLite version 3.36.0 2021-06-18 18:36:39 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE IF NOT EXISTS "bar" ( [baz] TEXT, [created_at] FLOAT DEFAULT '''CURRENT_TIMESTAMP''' ); ```

datasette allows users to enter in the value for named parameters into a free-text form field.

I think it would add a lot of usability, if the form field could be a drop down of options when query value is already a faceted column.

inspect_data should already include the hash and the db file size, so this PR takes advantage of using those instead of always recalculating. should help a lot on startup with large DBs.

closes #1834

Many apologies, @simonw. My previous PR #1835 did not really solve the problem because the name of the database is often not known to database object in the init method.

I took a cue from how you dealt with this issue and made hash a lazy property and did something similar with size.

:books: Documentation preview :books:: https://datasette--1837.org.readthedocs.build/en/1837/

Relates to #526

This is a minimal set of changes needed for having query CSVs attempt to download all the rows.

What's good about it is the minimalism.

What's bad about it:

1. We are abusing the _size argument to indicate we don't want truncation, which isn't the most obvious thing. Additionally, there are various checks that make sure the "_size" URL parameter is a positive integer, which we are relying on to prevent overloading.
2. The default CSV on a table page will use the max_returned_rows argument. Changing this could be a breaking change, since that's currently a place that has some facilities for pagination. Additionally, i think there's a limit under the hood somewhere which if we removed could lead to sql timeouts
3. There are similar reasons for leaving the current streaming method alone, as the current methods could allow for downloading very large files that could have a sql timeout if we tried to get them in one go.

:books: Documentation preview :books:: https://datasette--1820.org.readthedocs.build/en/1820/

When databases are loaded,

https://github.com/simonw/datasette/blob/cb1e093fd361b758120aefc1a444df02462389a3/datasette/app.py#L257-L260

there is nothing preventing the rehashing of the database for immutable databases.

https://github.com/simonw/datasette/blob/cb1e093fd361b758120aefc1a444df02462389a3/datasette/database.py#L50-L53

what i might expect is that relevant values of inspect_data get passed to the Database class to prevent re-hashing?

With data that is many gigs large, this is a significant start up time.

sqlite "IS" is equivalent to SQL "IS NOT DISTINCT FROM"

closes #423

if you don't set an explicit limit on container scaling, then google defaults to 100

google recently updated the limits on container scaling, such that if you set up datasette to use more memory or cpu, then you need to set the maxScale argument much smaller than 100.

would be nice if datasette publish could do this math for you and set the right maxScale.

Log of an failing publish run.

ERROR: (gcloud.run.deploy) spec.template.spec.containers[0].resources.limits.cpu: Invalid value specified for cpu. For the specified value, maxScale may not exceed 15. Consider running your workload in a region with greater capacity, decreasing your requested cpu-per-instance, or requesting an increase in quota for this region if you are seeing sustained usage near this limit, see https://cloud.google.com/run/quotas. Your project may gain access to further scaling by adding billing information to your account. Traceback (most recent call last): File "/home/runner/.local/bin/datasette", line 8, in <module> sys.exit(cli()) File "/home/runner/.local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/home/runner/.local/lib/python3.8/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/home/runner/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/runner/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/runner/.local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/runner/.local/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/home/runner/.local/lib/python3.8/site-packages/datasette/publish/cloudrun.py", line 160, in cloudrun check_call( File "/usr/lib/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'gcloud run deploy --allow-unauthenticated --platform=managed --image gcr.io/labordata/datasette warehouse --memory 8Gi --cpu 2' returned non-zero exit status 1.

if hashed_urls are on, then a -000 suffix is added to the _memory database, and the cache settings are set just as if it was a normal hashed database.

in particular, this header is set:

cache-control: max-age=31536000

this is not appropriate because the _memory-000 database isn't really hashed based on the contents of the databases (see #1561).

Either the cache-control header should be changed, or the _memory db should have a hash suffix that does depend on the contents of the databases.

This will close #1581.

I couldn't find any unit tests related to the testing hashed urls, and I know that you want to break that code out of the core application (#1561), so I'm not quite sure what you would like me to for testing.

If hashed_url mode is turned on and crossdb is also turned on, then queries to _memory should have a hash_id.

One way that it could work is to have the _memory hash be a hash of all the individual databases.

Otherwise, crossdb queries can get quit out of data if using aggressive caching.

Original title: Add option for adding a new, serial, primary key

sometimes we have tables that don't have primary keys, but ought to have them. we can use rowid for that, but it would often be nicer to have an explicit primary key. using the current value of rowid would be fine.

sqlite's query planner depends upon analyze to make good use of indices. It would be nice if analyze was run as part of the create-index command.

If data is inserted later, things can get out date, but it would still probably be a net win.

sqlite-utils is so nice, but the ergonomics of the multiline code in kind of tough. It's really hard (maybe impossible) to make the newlines play well with Makefiles.

it would be great to write your code fragment in a separate file and direct it into the sqlite-utils

either like

sqlite-utils convert my.db my_table my_column < custom_code.py

or

sqlite-utils convert my.db my_table my_column --custom-code=custom_code.py

Thanks, as ever, for these great tools!