issue_comments

Adding &per_page=100 would reduce the number of API requests by 3x.

Also tested this manually like so:

bash sqlite-utils create-table strict.db strictint id integer size integer --strict sqlite-utils create-table strict.db notstrictint id integer size integer sqlite-utils install sqlite-utils-shell sqlite-utils shell strict.db Attached to strict.db Type 'exit' to exit. sqlite-utils> insert into strictint (size) values (4); 1 row affected sqlite-utils> insert into strictint (size) values ('four'); An error occurred: cannot store TEXT value in INTEGER column strictint.size sqlite-utils> insert into notstrictint (size) values ('four'); 1 row affected sqlite-utils> commit; Done

I'm unable to replicate this issue. This is with a fresh install of sqlite-utils==3.35.2: ``` (base) ~ python3.12 Python 3.12.0 (v3.12.0:0fb18b02c8, Oct 2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin Type "help", "copyright", "credits" or "license" for more information.

import sqlite_utils db = sqlite_utils.Database(memory=True) db["foo"].insert({"bar": 1})

Thanks for opening an issue - this should help future Google searchers figure out what's going on here.

Another approach here could be to store large integers as TEXT in SQLite (or even as BLOB).

Both storing as REAL and storing as TEXT/BLOB feel nasty to me, but it looks like SQLite has a hard upper limit of 9223372036854775807 for integers.

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (9286c1b) 95.77% compared to head (1698a9d) 95.72%. Report is 1 commits behind head on main.

:exclamation: Current head 1698a9d differs from pull request most recent head 61c6e26. Consider uploading reports for the commit 61c6e26 to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

If I uncheck expand labels in the Advanced CSV export dialog, the error does not occur. Re-checking that box and re-running the export does cause the error to occur.

Apologies - I pushed a fix that addresses the mypy failures.

This looks really great on first glance - design is good, implementation is solid, tests and documentation look great.

Looks like a couple of mypy failures in the tests at the moment:

``` mypy sqlite_utils tests

sqlite_utils/db.py:543: error: Incompatible types in assignment (expression has type "type[Table]", variable has type "type[View]") [assignment] tests/test_lookup.py:156: error: Name "test_lookup_new_table" already defined on line 5 [no-redef] Found 2 errors in 2 files (checked 54 source files) Error: Process completed with exit code 1. ```

Updated documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-add-column - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#add-column

I'll add bytes too - float already works. This makes sense because when you are working with the Python API you use str and float and bytes and int to specify column types.

it probably does not make sense to gzip large sqlite database files on the fly. it can take many seconds to gzip a large file and you either have to have this big thing in memory, or write it to disk, which some deployment environments will not like.

i wonder if it would make sense to gzip the databases as part of the datasette publish process. it would be very cool to statically serve those as if they dynamically zipped (i.e. serve the filename example.db, not example.db.zip, and rely on the browser to expand).

Any progress on this? It would be very helpful on my end as well. Thanks!

hello Simon,

I've added more STRICT table support per https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776 in changeset https://github.com/simonw/sqlite-utils/commit/e4b9b582cdb4e48430865f8739f341bc8017c1e4. It also fixes table.transform() to preserve STRICT mode. Please pull if you deem appropriate. Thanks!

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (452a587) 92.69% compared to head (c88414b) 92.69%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Looks like tests are passing now but there is an issue with yaml loading and/or cog.

https://github.com/simonw/datasette/actions/runs/6879299298/job/18710911166?pr=2209

Whoops, looks like I forgot to check for other places where the 'facetable' table is used in the tests.

Without aliases:

The proposed fix in #2209 also works when the 'value' column is actually facetable (just added another value in the 'value' column).

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (452a587) 92.69% compared to head (eec10df) 92.69%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Superseded by #2206.

That keep_blank_values=True is from https://github.com/simonw/datasette/commit/0934844c0b6d124163d0185fb6a41ba5a71433da

Commit message:

request.post_vars() no longer discards empty values

Relevant test:

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/tests/test_internals_request.py#L19-L27

It should return a MultiParams:

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/utils/init.py#L900-L917

Change needs to be made before 1.0.

python return MultiParams(urllib.parse.parse_qs(body.decode("utf-8"))) Need to remember why I was using keep_blank_values=True there and check that using MultiParams doesn't conflict with that reason.

I found this problem while trying to use WTForms with this pattern:

```python choices = [(col, col) for col in await db.table_columns(table)]

class ConfigForm(Form): template = TextAreaField("Template") api_token = PasswordField("OpenAI API token") columns = MultiCheckboxField('Columns', choices=choices) ```

thanks for documenting this @bendnorman! got stuck at exactly the same point gcloud builds submit ... returned non-zero exit status 1, without a clue why this was happening. i now managed to get the github action to deploy datasette by assigning the following roles to the service account: roles/run.admin, roles/storage.admin, roles/cloudbuild.builds.builder, roles/viewer, roles/iam.serviceAccountUser.

I'm going to abandon this PR and ship the 3.12 testing change directly to main.

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (347fdc8) 95.74% compared to head (1f14df1) 95.74%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks!

Manually tested. Before:

After:

And a GIF of the fix after applying: - #598

Here's an animated GIF that demonstrates the bug:

Thanks!

The GIS tests now pass in that container too: bash pytest tests/test_gis.py ``` ======================== test session starts ========================= platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0 rootdir: /tmp/sqlite-utils plugins: hypothesis-6.88.1 collected 12 items

tests/test_gis.py ............ [100%]

========================= 12 passed in 0.48s ========================= ```

See details of how I tested this here: - https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126

Short version: having applied this fix, the following command (on simulated aarch64): bash sqlite-utils memory "select spatialite_version()" --load-extension=spatialite Outputs: json [{"spatialite_version()": "5.0.1"}]

Testing this manually on macOS using Docker Desk top like this: bash docker run -it --rm arm64v8/ubuntu /bin/bash Then inside the container: bash uname -m Outputs: aarch64

Then: bash apt install spatialite-bin libsqlite3-mod-spatialite git python3 python3-venv -y cd /tmp git clone https://github.com/simonw/sqlite-utils cd sqlite-utils python3 -m venv venv source venv/bin/activate pip install -e '.[test]' sqlite-utils memory "select spatialite_version()" --load-extension=spatialite Which output: Traceback (most recent call last): File "/tmp/sqlite-utils/venv/bin/sqlite-utils", line 33, in <module> sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/tmp/sqlite-utils/sqlite_utils/cli.py", line 1959, in memory _load_extensions(db, load_extension) File "/tmp/sqlite-utils/sqlite_utils/cli.py", line 3232, in _load_extensions if ":" in ext: TypeError: argument of type 'NoneType' is not iterable Then I ran this: bash git checkout -b MikeCoats-spatialite-paths-linux-arm main git pull https://github.com/MikeCoats/sqlite-utils.git spatialite-paths-linux-arm And now: bash sqlite-utils memory "select spatialite_version()" --load-extension=spatialite Outputs: json [{"spatialite_version()": "5.0.1"}]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (622c3a5) 95.77% compared to head (b1a6076) 95.77%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

The tests failed because they found a spelling mistake in a completely unrelated area of the code - not sure why that had not been caught before.

Thanks for this!

Datasette is using that now, see: - #1893

FWIW, code mirror 6 now has this standard although if you want table-specific suggestions, you'd have to handle parsing out which table the user is querying yourself.

Using the Counties example, I was able to pull out the MakePoint method as

MakePoint(cast(rm_rnb_history_pres.rx_lng as float), cast(rm_rnb_history_pres.rx_lat as float)) as geometry

which worked, giving me a geometry column.

gave

I believe it's the cast to float that does the trick. Prior to using the cast, I also received a 'wrong number of arguments' eror.

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (452a587) 92.69% compared to head (be4d0f0) 92.69%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Superseded by #2202.

@fgregg, I am happy to do that and just could not find a way to create issues at your fork repo.

@yejiyang why don’t you move this discussion to my fork to spare simon’s notifications

@fgregg Thanks for your reply. I tried to use your fork branch datasette = {url = "https://github.com/fgregg/datasette/archive/refs/heads/no_limit_csv_publish.zip"} and got error - TypeError: 'str' object is not callable. I used the same templates as in your branch here .

INFO: 127.0.0.1:47232 - "GET /-/static/sql-formatter-2.3.3.min.js HTTP/1.1" 200 OK Traceback (most recent call last): File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py", line 1632, in route_path response = await view(request, send) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py", line 1814, in async_view_fn response = await async_call_with_supported_arguments( File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/utils/__init__.py", line 1016, in async_call_with_supported_arguments return await fn(*call_with) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/views/table.py", line 673, in table_view response = await table_view_traced(datasette, request) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/views/table.py", line 822, in table_view_traced await datasette.render_template( File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py", line 1307, in render_template return await template.render_async(template_context) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1324, in render_async return self.environment.handle_exception() File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception raise rewrite_traceback_stack(source=source) File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1321, in <listcomp> [n async for n in self.root_render_func(ctx)] # type: ignore File "templates/table.html", line 1, in top-level template code {% extends "base.html" %} File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/templates/base.html", line 62, in top-level template code {% block content %} File "templates/table.html", line 24, in block 'content' <div class="page-header" style="border-color: #{{ database_color(database) }}"> TypeError: 'str' object is not callable INFO: 127.0.0.1:47228 - "GET /zeropm-v0-0-3/api_ready_query HTTP/1.1" 500 Internal Server Error

hi @yejiyang, have your tried using my fork of datasette: https://github.com/fgregg/datasette/tree/no_limit_csv_publish

Hi @fgregg, I came across this issue and found your setup at labordata.bunkum.us can help me with a research project at https://database.zeropm.eu/. I really like the approach here when dealing with a custom SQL query returning more than 1000 rows: 1) At the table in HTML page, only first 1000 rows displayed; 2) When click the "Download this data as a CSV Spreadsheet(All Rows)" button, a csv with ALL ROWS (could be > 100 Mb) get downloaded.

I am trying to repeat the setup but have yet to be successful so far. What I tried: 1) copy the query.html & table.html templates from this github repo and use it my project 2) use the same datasette version 1.0a2.

Do you know what else I should try to set it correctly? I appreciate your help.

@simonw I would like to use this opportunity to thank you for developing & maintaining such an amazing project. I introduce your datasette to several projects in my institute. I am also interested in your cloud version.

That worked!

I'm landing this despite the cog failures. I'll fix them on main if I have to.

Codecov Report

All modified lines are covered by tests :white_check_mark:

Comparison is base (3feed1f) 92.46% compared to head (8ae479c) 92.69%. Report is 112 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Oh! I think I broke Cog on main and these tests are running against this branch rebased against main.

Weird, the cog check is failing in CI.

Run cog --check docs/*.rst cog --check docs/*.rst shell: /usr/bin/bash -e {0} env: pythonLocation: /opt/hostedtoolcache/Python/3.9.18/x64 PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib/pkgconfig Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib Check failed Checking docs/authentication.rst Checking docs/binary_data.rst Checking docs/changelog.rst Checking docs/cli-reference.rst Checking docs/configuration.rst (changed)

Pushed that incomplete code here: https://github.com/datasette/datasette-upgrade

I can't replicate this bug. Closing, but please re-open if it's still happening.

As an aside, the link I promote is https://datasette.io/discord which redirects:

curl -i 'https://datasette.io/discord' HTTP/2 301 location: https://discord.gg/ktd74dm5mw content-type: text/plain x-cloud-trace-context: 8dcfd08d3d1fa44f7ee78568e0f5305e;o=1 date: Thu, 12 Oct 2023 21:54:17 GMT server: Google Frontend content-length: 0 curl -i 'https://discord.gg/ktd74dm5mw' HTTP/2 301 date: Thu, 12 Oct 2023 21:54:28 GMT content-type: text/plain;charset=UTF-8 content-length: 0 location: https://discord.com/invite/ktd74dm5mw strict-transport-security: max-age=31536000; includeSubDomains; preload permissions-policy: interest-cohort=() x-content-type-options: nosniff x-frame-options: DENY x-robots-tag: noindex, nofollow, noarchive, nocache, noimageindex, noodp x-xss-protection: 1; mode=block report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Dzzrf%2FgGkfFxtzSAQ46slMVDLcFjsH9fsvVkzHtgUUiZ891rXAa6LvTRpHK%2BdSMSQ54F57hS9z1mZXXklIbONZW1bfBuFjSK9J4XmjjLjsFUulMXvpjfCLkB6PI%3D"}],"group":"cf-nel","max_age":604800} nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} server: cloudflare cf-ray: 815294ddff282511-SJC

Demo of that logic: $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-1.yaml New datasette.yaml file will be written to datasette.yaml $ touch metadata-new-1.yaml $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-2.yaml New datasette.yaml file will be written to datasette.yaml $ touch datasette.yaml $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-2.yaml New datasette.yaml file will be written to datasette-new.yaml

Started playing with this plugin idea, now tearing myself away to work on something more important: ```python from datasette import hookimpl import click import pathlib

@hookimpl def register_commands(cli): @cli.group() def upgrade(): """ Apply configuration upgrades to an existing Datasette instance """ pass

@upgrade.command()
@click.argument(
    "metadata", type=click.Path(exists=True)
)
@click.option(
    "new_metadata", "-m", "--new-metadata", help="Path to new metadata.yaml file", type=click.Path(exists=False)
)
@click.option(
    "new_datasette", "-c", "--new-datasette", help="Path to new datasette.yaml file", type=click.Path(exists=False)
)
@click.option(
    "output_dir", "-e", "--output-dir", help="Directory to write new files to", type=click.Path(), default="."
)
def metadata_to_config(metadata, new_metadata, new_datasette, output_dir):
    """
    Upgrade an existing metadata.json/yaml file to the new metadata.yaml and
    datasette.yaml split introduced prior to Datasette 1.0.
    """
    print("Upgrading {} to new metadata.yaml format".format(metadata))
    output_dir = pathlib.Path(output_dir)
    if not new_metadata:
        # Pick a filename for the new metadata.yaml file that does not yet exist
        new_metadata = pick_filename("metadata", output_dir)
    if not new_datasette:
        new_datasette = pick_filename("datasette", output_dir)
    print("New metadata.yaml file will be written to {}".format(new_metadata))
    print("New datasette.yaml file will be written to {}".format(new_datasette))

def pick_filename(base, output_dir): options = ["{}.yaml".format(base), "{}-new.yaml".format(base)] i = 0 while True: option = options.pop(0) option_path = output_dir / option if not option_path.exists(): return option_path # If we ran out if not options: i += 1 options = ["{}-new-{}.yaml".format(base, i)] ```

I think I'm OK with not preserving comments, just because it adds a level of complexity to the tool which I don't think is worth the value it provides.

If people want to keep their comments I'm happy to leave them to copy those over by hand.

I dig it - I was thinking an Observable notebook where you paste your metadata.json/metadata.yaml and it would generate the new metadata + datasette.yaml files, but an extensible datasette upgrade plugin would be nice for future plugins.

One thing to think about: If someone has comments in their original metadata.yaml, could we preserve them in the new files? tbh maybe not too important bc if people cared that much they could just copy + paste, and it might be too distracting

Related idea: how about a datasette-upgrade plugin which adds a datasette upgrade command that can be used to automate this process?

Maybe something like this: bash datasette install datasette-upgrade datasette upgrade metadata-to-config metadata.json This would output two new files: metadata.yaml and datasette.yaml. If files with those names existed already in the current directory they would be called metadata-new.yaml and datasette-new.yaml.

The command would tell you what it did: ``` Your metadata.json file has been rewritten as two files:

metadata-new.yaml
datasette.yaml

Start Datasette like this to try them out:

datasette -m metadata-new.yaml -c datasette.yaml

`` The command isdatasette upgrade metadata-to-configbecausemetadata-to-config` is the name of the upgrade recipe. The first version of the plugin would only have that single recipe, but we could add more recipes in the future for other upgrades.

Some options for where this could go: - Directly in the release notes? I'm not sure about that, those are getting pretty long already. I think the release notes should link to relevant upgrade guides. - On a new page? We could have a "upgrade instructions" page in the documentation. - At the bottom of the new https://docs.datasette.io/en/latest/configuration.html page

I'm leaning towards the third option at the moment.

But... we may also need to provide upgrade instructions for plugin authors. Those could live in a separate area of the documentation though, since issues affecting end-users who configure Datasette and issues affecting plugin authors are unlikely to overlap much.

I'm going to land this and open a new issue for the upgrade instructions.

It would be nice if we could catch that and turn that into a less intimidating Click exception too.

This looks good and works well. The error from this currently looks like:

datasette -m metadata.json -p 8844 Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette", line 33, in <module> sys.exit(load_entry_point('datasette', 'console_scripts', 'datasette')()) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 98, in wrapped return fn(*args, **kwargs) File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 546, in serve metadata_data = fail_if_plugins_in_metadata(parse_metadata(metadata.read())) File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 1282, in fail_if_plugins_in_metadata raise Exception( Exception: Datasette no longer accepts plugin configuration in --metadata. Move your "plugins" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details. With wrapping:

Exception: Datasette no longer accepts plugin configuration in --metadata. Move your "plugins" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.

I think we should link directly to documentation that tells people how to perform this upgrade.

Codecov Report

All modified lines are covered by tests :white_check_mark:

Comparison is base (6ed7908) 92.69% compared to head (0135e7c) 92.68%. Report is 14 commits behind head on main.

:exclamation: Current head 0135e7c differs from pull request most recent head 18b48f8. Consider uploading reports for the commit 18b48f8 to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

https://lite.datasette.io/?install=datasette-packages#/-/packages confirms that Datasette Lite still works (click-default-group-wheel was originally built to allow it to run) and that it's now using these packages: "click-default-group": "1.2.4", "datasette": "0.64.5", Full list: json { "aiofiles": "23.2.1", "anyio": "3.7.1", "asgi-csrf": "0.9", "asgiref": "3.7.2", "certifi": "2022.12.7", "click": "8.1.3", "click-default-group": "1.2.4", "datasette": "0.64.5", "datasette-packages": "0.2", "h11": "0.12.0", "httpcore": "0.15.0", "httpx": "0.23.0", "hupper": "1.12", "idna": "3.4", "itsdangerous": "2.1.2", "janus": "1.0.0", "Jinja2": "3.1.2", "MarkupSafe": "2.1.2", "mergedeep": "1.3.4", "micropip": "0.3.0", "packaging": "23.0", "Pint": "0.22", "pip": "23.2.1", "pluggy": "1.0.0", "pyparsing": "3.0.9", "python-multipart": "0.0.6", "PyYAML": "6.0", "rfc3986": "1.5.0", "setuptools": "67.6.1", "sniffio": "1.3.0", "typing-extensions": "4.5.0", "uvicorn": "0.23.2" }

Confirmed - I ran this in a fresh virtual environment: bash pip install --no-cache datasette And now: bash pip freeze | grep click click==8.1.7 click-default-group==1.2.4 bash datasette --version datasette, version 0.64.5

Replicated this myself: bash cd /tmp mkdir dddd cd dddd pipenv shell pip install datasette pip freeze | grep click click==8.1.7 click-default-group==1.2.4 click-default-group-wheel==1.2.3 Yeah this is bad, I'll ship a 0.64.5 release.

I confirm the bug, as above, and that @jonafato 's patch fixes it for me. However, it's not the right fix. The problem is that ProgressBar is being used in the wrong way. This also results in two lines being printed instead of one, like this:

[#######-----------------------------] 20% [####################################] 100%% The bug is reproducible for me in any terminal, including Gnome Terminal and Guake, and VSCode. With VSCode I can use this launch.json to reproduce it:

```json

{ "version": "0.2.0", "configurations": [ { "name": "Python: Module", "type": "python", "request": "launch", "module": "sqlite_utils", "justMyCode": false, "args": ["insert", "test.db", "test", "--csv", "tests/sniff/example1.csv"] } ] } ```

[edit - deleted my analysis of why the current code is wrong, which was confused and confusing]

Hello! Resurrecting this issue since we're running into something similar with data.catalyst.coop as our database files have ballooned up to several GB. Our Cloud Run revisions now require huge amounts of RAM to start up without receiving a SIGBUS.

I'd love to see this fix merged in. It sounds like we want to make the immutable/read-only mode decision more flexible before doing so, so that we can use ro in Docker and immutable outside. If that sounds right, I'm happy to take a crack at adding that as a command-line flag or something that gets set automatically based on the expected execution environment.

Datasette Cloud really wants this.

@simonw Any chance we can get this tiny patch merged for an upcoming release?

I'm hitting the #2123 issue and I just patched my local version with this and it seems to work fine.

Never mind

When I created the connection using sqlite_utils.Database(path)

I just needed to add the following statement right after and it did the trick

self.db.conn.execute("PRAGMA foreign_keys = ON")

Hope this helps people in the future 👍

This would be awesome to have for multi-gig tsv and csv files! I'm currently looking at a 10 hour countdown for one such important. Not a problem because I'm lazy and happy to let it run and check on it tomorrow..

Worth noting that it already sets --cors automatically without you needing to specify it:

https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/init.py#L374-L374

I wonder if that's actually surprising behaviour that we should change before 1.0.

Maybe I should add --cors and --crossdb to datasette publish cloudrun as well?

Found more when I searched for YAML.

Here's the most interesting: https://github.com/labordata/warehouse/blob/0029a72fc1ceae9091932da6566f891167179012/.github/workflows/build.yml#L59

--extra-options="--crossdb --setting sql_time_limit_ms 100000 --cors --setting facet_time_limit_ms 500 --setting allow_facet off --setting trace_debug 1"

Uses both --cors and --crossdb.

https://github.com/search?q=datasette+publish+extra-options+language%3AShell&type=code&l=Shell shows 17 matches, I'll copy in illustrative examples here:

--extra-options="--setting sql_time_limit_ms 5000" --extra-options="--config default_cache_ttl:3600 --config hash_urls:1" --extra-options "--setting sql_time_limit_ms 3500 --setting default_page_size 20 --setting trace_debug 1" --extra-options="--config default_page_size:50 --config sql_time_limit_ms:30000 --config facet_time_limit_ms:10000" --extra-options="--setting sql_time_limit_ms 5000" --extra-options "--setting suggest_facets off --setting allow_download on --setting truncate_cells_html 0 --setting max_csv_mb 0 --setting sql_time_limit_ms 2000"

I'd really like to remove --extra-options. I think the new design makes that completely obsolete?

Maybe it doesn't. You still need --extra-options for the --crossdb option for example.

Here's the full help for Cloud Run at the moment: bash datasette publish cloudrun --help ``` Usage: datasette publish cloudrun [OPTIONS] [FILES]...

Publish databases to Datasette running on Cloud Run

Options: -m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish --extra-options TEXT Extra options to pass to datasette serve --branch TEXT Install datasette from a GitHub branch e.g. main --template-dir DIRECTORY Path to directory containing custom templates --plugins-dir DIRECTORY Path to directory containing custom plugins --static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/... --install TEXT Additional packages (e.g. plugins) to install --plugin-secret <TEXT TEXT TEXT>... Secrets to pass to plugins, e.g. --plugin- secret datasette-auth-github client_id xxx --version-note TEXT Additional note to show on /-/versions --secret TEXT Secret used for signing secure values, such as signed cookies --title TEXT Title for metadata --license TEXT License label for metadata --license_url TEXT License URL for metadata --source TEXT Source label for metadata --source_url TEXT Source URL for metadata --about TEXT About label for metadata --about_url TEXT About URL for metadata -n, --name TEXT Application name to use when building --service TEXT Cloud Run service to deploy (or over-write) --spatialite Enable SpatialLite extension --show-files Output the generated Dockerfile and metadata.json --memory TEXT Memory to allocate in Cloud Run, e.g. 1Gi --cpu [1|2|4] Number of vCPUs to allocate in Cloud Run --timeout INTEGER Build timeout in seconds --apt-get-install TEXT Additional packages to apt-get install --max-instances INTEGER Maximum Cloud Run instances --min-instances INTEGER Minimum Cloud Run instances --help Show this message and exit. ```

I think the actual design of this is pretty simple. Current help starts like this:

``` Usage: datasette publish cloudrun [OPTIONS] [FILES]...

Publish databases to Datasette running on Cloud Run

Options: -m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish --extra-options TEXT Extra options to pass to datasette serve `` The-sand-c` short options are not being used.

So I think -c/--config can point to a JSON or YAML datasette.yaml file, and -s/--setting key value can mirror the new -s/--setting option in datasette serve itself (a shortcut for populating the config file directly from the CLI).

Here's the relevant help section from datasette serve: -m, --metadata FILENAME Path to JSON/YAML file containing license/source metadata -c, --config FILENAME Path to JSON/YAML Datasette configuration file -s, --setting SETTING... nested.key, value setting to use in Datasette configuration

1.0a7 is out with this fix as well now: https://docs.datasette.io/en/1.0a7/changelog.html#a7-2023-09-21

Tests all pass now.

That worked

https://github.com/simonw/datasette/commit/e4f868801a6633400045f59584cfe650961c3fa6 is the latest commit right now and https://latest.datasette.io/-/versions shows that as the deployed version.

Tested that locally with Python 3.9 from pyenv and it worked.

Still fails in Python 3.9: https://github.com/simonw/datasette/actions/runs/6266752548/job/17018363302 plugin_info["name"] = distinfo.name or distinfo.project_name AttributeError: 'PathDistribution' object has no attribute 'name' Test failed: datasette-json-html should not have been loaded

The test fails as expected now. Closing this issue, will solve the remaining problems in: - #2057

https://github.com/simonw/datasette/actions/runs/6267146158/job/17019594849 failed on 3.9 this time.

plugin_info["name"] = distinfo.name or distinfo.project_name AttributeError: 'PathDistribution' object has no attribute 'name' Test failed: datasette-json-html should not have been loaded

The @add_common_publish_arguments_and_options decorator described here is bad. If I update it to support a new config option all plugins that use it will break.

https://github.com/simonw/datasette/blob/f130c7c0a88e50cea4121ea18d1f6db2431b6fab/docs/plugin_hooks.rst#L347-L355

I want to deprecate it and switch to a different, better design to address the same problem.

As soon as datasette publish cloudrun has this I can re-enable this bit of the demo deploy:

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97

Which should fix this broken demo from https://simonwillison.net/2022/Dec/2/datasette-write-api/

https://todomvc.datasette.io/

I'm going to disable this bit of the deploy for the moment, which will break the demo linked to from https://simonwillison.net/2022/Dec/2/datasette-write-api/

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97

... which raises the challenge that datasette publish doesn't yet know what to do with a config file!

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L114-L122

Correct usage is now: bash python tests/fixtures.py fixtures.db fixtures-config.json fixtures-metadata.json \ plugins --extra-db-filename extra_database.db Test tables written to fixtures.db - metadata written to fixtures-metadata.json - config written to fixtures-config.json Wrote plugin: plugins/register_output_renderer.py Wrote plugin: plugins/view_name.py Wrote plugin: plugins/my_plugin.py Wrote plugin: plugins/messages_output_renderer.py Wrote plugin: plugins/sleep_sql_function.py Wrote plugin: plugins/my_plugin_2.py Test tables written to extra_database.db

New error: "Error: Metadata should end with .json"

https://github.com/simonw/datasette/actions/runs/6266720924/job/17018265851

That does seem to fix the problem!

Hunch: https://pypi.org/project/importlib-metadata/ may help here.

I think I see the problem - it's from here: https://github.com/simonw/datasette/commit/b2ec8717c3619260a1b535eea20e618bf95aa30b#diff-5dbc88d6e5c3615caf10e32a9d6fc6ff683f5b5814948928cb84c3ab91c038b6L770

The config and metadata Click options are the wrong way round:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L785-L786

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L801

Maybe plugins/ does not exist? It should have been created by this line:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L41-L42

This looks to be the step that is failing:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L50-L60

We're planning a breaking change in 1.0a7: - #2191

Since that's a breaking change I'm going to ship 1.0a7 right now with this fix, then ship that breaking change as 1.0a8 instead.

Release 0.64.4: https://docs.datasette.io/en/stable/changelog.html#v0-64-4