I've rebased my patch on the latest main. It should be ready to merge.

Codecov Report

Patch coverage: 97.72% and project coverage change: -0.36 :warning:

Comparison is base (3feed1f) 92.46% compared to head (c8a2904) 92.11%.

:exclamation: Current head c8a2904 differs from pull request most recent head ef25867. Consider uploading reports for the commit ef25867 to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

I also have the same problem when running behind an apache proxy server with base_url. However, I have researched the problem a bit and have come to the conclusion that if you change _facet_date to _facet in the following https://github.com/simonw/datasette/blob/3feed1f66e2b746f349ee56970a62246a18bb164/datasette/facets.py#LL493C57-L493C68 , the facets work. But I'm not sure if this has other consequences?

Codecov Report

Patch coverage has no change and project coverage change: -0.06 :warning:

Comparison is base (3feed1f) 92.46% compared to head (85b9e51) 92.41%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Patch coverage: 87.89% and project coverage change: -4.43 :warning:

Comparison is base (56b0758) 92.15% compared to head (921faae) 87.73%.

:exclamation: Current head 921faae differs from pull request most recent head 69a31cd. Consider uploading reports for the commit 69a31cd to get more accurate results

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Well that's embarrassing. I made a fork using macnotesapp and it's actually slower. This is because the Scripting Bridge sometimes fails to return the folder and thus macnotesapp resorts to AppleScript in this situation. The repeated AppleScript calls on a large library are slower than your "slurp it all in" approach. I've got some ideas about how to improve this--will make another attempt if I can fix the issues.

I keep coming back to this in search for the related exploration, so I'll just link it now:

@simonw has meanwhile researched how to deploy Datasette to AWS Lambda using function URLs and Mangum via https://github.com/simonw/public-notes/issues/6 and concluded that's everything I need to know in order to build a datasette-publish-lambda plugin.

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.03 :tada:

Comparison is base (fc221f9) 96.25% compared to head (0e4750d) 96.29%.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

@simonw - maybe put in some error handling to trap for poorly formed XML (from Apple engineers) so that it suggests that there are problems with export.zip rather than odd looking Python errors :)

Thanks for reporting this and providing a solution -- I was puzzled by this error when I revisited my walking data and experienced this issues. I haven't tried the fix yet.

+1, I have been wishing for this feature (also for use with template-sql). It was requested before here #1304.

I managed to make it work by using nginx's 'exact match' (=) combined with 'prefix match'; that is, match explicitly on /, and redirect to /<db>/<table>, and then have the normal ProxyPath for the unadorned (prefix-matching) /.

location = / { return 302 /<db>/<table>; } location / { proxy_pass http://127.0.0.1:8001/; proxy_set_header Host $host; }

I've implemented the test (thanks for pointing me in the right direction!).

At tmcl-it/datasette:0.64.1+row-view-expand-labels I also have a variant of this patch that applies to the 0.64.x branch. Please let me know if you'd be interested in merging that as well and I'll open another PR.

Codecov Report

Patch and project coverage have no change.

Comparison is base (0b4a286) 92.11% compared to head (a8dde13) 92.11%.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

I have this same issue, which is happening with both json links and facets. It is not happening with column sort links in the gear popup menus, but it is happening with the sort arrow that results after you use one of those links. I'm using Apache as a proxy to Datasette; the relevant configs are:

ProxyPass /datasette/ http://127.0.0.1:8000/datasette/ nocanon ProxyPreserveHost on

{ "base_url": "/datasette/" }

If it would be useful to get a look at the running installation via the Web, Simon, let me know.

I think I see what is happening here, although I haven't quite work out a fix yet. Usually:

• click.progressbar.render_progress() renders the cursor invisible on each invocation (update of the bar)
• When the progress bar goes out of scope, the __exit()__ method is invoked, which calls render_finish() to make the cursor re-appear.

(See terminal escape sequences BEFORE_BAR and AFTER_BAR in click).

However the sqlite-utils utils.file_progress context manager wraps click.progressbar and yields an instance of a helper class:

python @contextlib.contextmanager def file_progress(file, silent=False, **kwargs): ... with click.progressbar(length=file_length, **kwargs) as bar: yield UpdateWrapper(file, bar.update)

The yielded UpdateWrapper goes out of scope quickly and click.progressbar.__exit__() is called. The cursor is made un-invisible. Hoewever bar is still live and so when the caller iterates on the yielded wrapper this invokes the bar's update method, calling render_progress(), each time printing the "make cursor invisible" escape code. The progressbar.__exit__ function is not called again, so the cursor doesn't re-appear.

Thanks @dmick . I chose to create a firewall rule under my GCP to open the port of interest and datasette works.

@gk7279, you had asked in a separate bug about how to redirect web servers in general. The datasette docs actually have pretty good information on this for both nginx and apache2: https://docs.datasette.io/en/stable/deploying.html#running-datasette-behind-a-proxy

yes @dmick . How did you make your public IP redirect to your uvicorn server?

Instead of nginx, I have apache2 on my GCP VM. Any pointers here are helpful too.

Thanks.

Was that query to me, @gk7279?

Can you please help or share your expertise with #2030 ?

Meanwhile, a cheap workaround is to invalidate the registered function cache: python table.convert(...) db._registered_functions = set() table.convert(...)

Is this where we talk about i18n of results? Or is that a separate thread.

e.g. Having country_long show España in the Spanish version of the global power plants demo site instead of Spain.

Thanks for confirming my doubts! I removed it after opening this issue, yup, then had another issue with default_cache_ttl_hashed which I assume was removed at the same time. Sorry for not trying that before opening the issue.

Nulls in primary keys, does it every time.

PR below

I live the patch here for others:

original code shell $ which sqlite-utils | xargs cat ```python

!/usr/bin/python3

-- coding: utf-8 --

import re import sys from sqlite_utils.cli import cli

if name == 'main': sys.argv[0] = re.sub(r'(-script.pyw|.exe)?$', '', sys.argv[0]) sys.exit(cli()) ```

patched/sqlite-utils.py ```python

!/usr/bin/python3

-- coding: utf-8 --

import re import sys from sqlite_utils.cli import cli

New imports

from unittest.mock import patch from sqlite_utils.cli import VALID_COLUMN_TYPES

if name == 'main': # Choices of the option --type cli.commands['transform'].params[2].type.types[1].choices.append('DATETIME')

# The dicts has to be extended with a new type
with patch.dict('sqlite_utils.db.COLUMN_TYPE_MAPPING', {'DATETIME': 'DATETIME'}),\
     patch('sqlite_utils.cli.VALID_COLUMN_TYPES', VALID_COLUMN_TYPES + ("DATETIME", )):

    # Command is unchanged
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(cli())

```

And now it's working ```bash $ sqlite-utils schema events.sqlite cards.chunk.get CREATE TABLE "cards.chunk.get" ( [id] INTEGER PRIMARY KEY NOT NULL, [timestamp] TEXT, )

$ python patched/sqlite-utils.py transform events.sqlite cards.chunk.get --type timestamp DATETIME

$ sqlite-utils schema events.sqlite cards.chunk.get CREATE TABLE "cards.chunk.get" ( [id] INTEGER PRIMARY KEY NOT NULL, [timestamp] DATETIME, ) ```

Hey, isn't this essentially the same issue as #448 ?

lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed

I've got your concern. Let's see if we will be replied on it and i'll close the issue some later.

Ha, yes, I might end up making something very niche. That's OK.

I'm building a UI for Datasette that lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed. And so on. I had hoped that I could just lean on sqlite-utils, but I think it's a little too surprising.

Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178

I suspect you could overwrite the datetime entry to achieve what you want

And thank you for pointing me to it. At least, i can make a monkey patch for my need...

Isn't your suggestion too fundamental for the utility?

The bigger flexibility, the bigger complexity. Your idea make sense defenitely, but how often do you make schema changes? And how many people could benefit from it, what do you think?

I think the bigger issue is that sqlite-utils mixes mechanism (it implements the 12-step way to alter SQLite tables) and policy (it has an opinionated stance on what column types should be used).

That might be a design choice to make it accessible to users by providing a reasonable set of defaults, but it doesn't quite fit my use case.

It might make sense to extract a separate library that provides just the mechanisms, and then sqlite-utils would sit on top of that library with its opinionated set of policies.

That would be a very big change, though.

I might take a stab at extracting the library, but just for the table schema migration piece, not all the other features that sqlite-utils supports. I wouldn't expect sqlite-utils to depend on it.

Part of my motivation is that I want to provide some other abilities, too, like support for CHECK constraints. I see that the issue in this repo (https://github.com/simonw/sqlite-utils/issues/358) proposes a bunch of short-hand constraints, which I wouldn't want to accidentally expose to people -- I want a layer that is a 1:1 mapping to SQLite.

Do you see a way to enable it without affecting existing users or bumping the major version number?

I don't see a clean solution, only extending code with a side variable that tells us we want to apply advanced types instead of basic.

it could be a similiar command like tranform-v2 --type column DATETIME or a cli option transform --adv-type column DATETIME along with a dict that contains the advanced types. Then with knowledge that we run an advanced command we take that dictionary somehow, we can wrap the current and new dictionaries by a superdict and work with it everywhere according to the knowledge. This way shouldn't affect users who are using the previous lib versions and it have to be merged in the next major one.

But this way looks a bad design, too messy.

I'd support that, but I'm not the author of this library.

One challenge is that would be a breaking change. Do you see a way to enable it without affecting existing users or bumping the major version number?

I could, of course.

Doest it worth bringing such the improvement to the library?

Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178

I suspect you could overwrite the datetime entry to achieve what you want

That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work.

It works, but a column becomes TEXT

```python In [1]: import sqlite_utils In [2]: db = sqlite_utils.Database('events.sqlite') In [3]: table = db['cards.chunk.get'] In [4]: table.columns_dict Out[4]: {'id': int, 'timestamp': float, 'data_chunk_number': int, 'user_id': str, 'meta_duplication_source_id': int, 'context_sort_attribute': str, 'context_sort_order': str}

In [5]: from datetime import datetime In [7]: table.transform(types={'timestamp': datetime}) In [8]: table.columns_dict Out[8]: {'id': int, 'timestamp': str, 'data_chunk_number': int, 'user_id': str, 'meta_duplication_source_id': int, 'context_sort_attribute': str, 'context_sort_order': str} ```

bash ❯ sqlite-utils schema events.sqlite cards.chunk.get CREATE TABLE "cards.chunk.get" ( [id] INTEGER PRIMARY KEY NOT NULL, [timestamp] TEXT, ...

I think it's not currently possible: sqlite-utils requires that it be one of integer, text, float, blob (see code)

IMO, this is a bit of friction and it would be nice if it was more permissive. SQLite permits developers to use any data type when creating a table. For example, this is a perfectly cromulent sqlite session that creates a table with columns of type baz and bar:

``` sqlite> create table foo(column1 baz, column2 bar); sqlite> .schema foo CREATE TABLE foo(column1 baz, column2 bar); sqlite> select * from pragma_table_info('foo'); cid name type notnull dflt_value pk

0 column1 baz 0 0
1 column2 bar 0 0
```

The idea is that the application developer will know what meaning to ascribe to those types. For example, I'm working on a plugin to Datasette. Dates are tricky to handle. If you have some existing rows, you can look at the values in them to know how a user is serializing the dates -- as an ISO 8601 string? An RFC 3339 string? With millisecond precision? With timezone offset? But if you don't yet have any rows, you have to guess. If the column is of type TEXT, you don't even know that it's meant to hold a date! In this case, my plugin will look to see if the column is of type DATE or DATETIME, and assume a certain representation when writing.

Perhaps there is an argument that sqlite-utils is trying to conform to SQLite's strict mode, and that is why it limits the choices. In strict mode, SQLite requires that the data type be one of INT, INTEGER, REAL, TEXT, BLOB, ANY. But that can't be the case -- sqlite-utils supports FLOAT, which is not one of the valid types in strict mode, and it rejects INT, REAL and ANY, which are valid.

That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work.

SQLite doesn't have a native DATETIME type. It stores dates internally as strings and then has functions to work with date-like strings. Yes it's weird.

That's correct. But my issue is about the application level libraries that, i suppose, have better data understanding if see a specific type such as DATETIME.

I'm writing data with dataset i've mentioned. The lib changes its behavior depending on a type. I saw different behavior with types DATETIME, FLOAT, TEXT. Dataset, for their part, is built upon Sqlalchemy, you know what it is.

To be honest, i didn't dive into the details of why the behavior changes, but when i altered manually by other util a type of column to DATETIME things got back to normal.

On the matter, can i achieve it with Sqlite Utils at the moment?

Came here to say that I also have this issue.

Codecov Report

Base: 92.11% // Head: 91.78% // Decreases project coverage by -0.34% :warning:

Coverage data is based on head (f529a30) compared to base (e4ebef0). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Base: 92.11% // Head: 92.11% // No change to project coverage :thumbsup:

Coverage data is based on head (dca7634) compared to base (e4ebef0). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

@cldellow glad to hear you tried it, as I got grossed out by my own suggestion ;) If you are on macOS I do have one trick for debugging segfaults using lldb.

Codecov Report

Base: 92.11% // Head: 92.12% // Increases project coverage by +0.01% :tada:

Coverage data is based on head (1e5b42f) compared to base (e4ebef0). Patch coverage: 100.00% of modified lines in pull request are covered.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

The problem (in my particular use case) with using a VIEW is that I'd need one of the columns to be searchable – but that (enable-fts) doesn't work with views :/

__ side-suggestion: I don't know how feasible this might be, but when one column (or table) would be marked as hidden, could the Download SQLite DB link take that into account? 🧐

Me too, on a M1. Not sure if it's compatible?

great initiative, @cldellow :+1:

@simonw, if you want to merge this, that would still be welcome :)

Codecov Report

Base: 92.11% // Head: 92.11% // Increases project coverage by +0.00% :tada:

Coverage data is based on head (96b8b7f) compared to base (6a352e9). Patch coverage: 100.00% of modified lines in pull request are covered.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

You may have just been talking to yourself here, but I found your documentation of the process incredibly useful when I hit the same error myself. Thanks! 🌈

Codecov Report

Base: 92.11% // Head: 92.11% // No change to project coverage :thumbsup:

Coverage data is based on head (bfaa59b) compared to base (6a352e9). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

I just tried this branch and saw some errors.

I installed this PR locally with:

bash pip install https://github.com/RhetTbull/dogsheep-photos/archive/update_for_bigsur.zip

Codecov Report

Base: 92.15% // Head: 92.15% // No change to project coverage :thumbsup:

Coverage data is based on head (a4eebee) compared to base (c41278b). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Base: 92.15% // Head: 92.15% // No change to project coverage :thumbsup:

Coverage data is based on head (af76abc) compared to base (c41278b). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

as of 2023 it appears that tweets: not yet implemented is happening.. pretty important for a twitter export functionality!

this seems an impossible task with twitter liable to switch this around every other day

Codecov Report

Base: 92.10% // Head: 92.10% // No change to project coverage :thumbsup:

Coverage data is based on head (1e213a3) compared to base (572bdb5). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

text columns containing empty strings should not be rewritten to null.

I would actually appreciate an option to do just that for text columns as well.

Just a quick +1 to this one from me - I would like to do a better job of tracking who is reviewing one another's pull requests in repositories, since this is a specific kind of maintenance work that I think often goes unrewarded. I can't seem to figure this out just by looking at the pull_request or issue_comments tables, so I think it would be helpful to support PR reviews natively (even if just for summary statistics). Alternatively if there is a way in the API to tell if an issue comment is part of a review, then perhaps you could quickly calculate the number of unique reviews that an author performed. But that was beyond my SQL-foo :-)

You were super-close on the python version of the test here, changing http to https on 8b73fc6b47dffd8836f5c58aae1e57c1f66a5754 is enough to pass the test:

diff diff --git a/tests/conftest.py b/tests/conftest.py index 69dee68b4a3f..ba07a11d37f6 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -207,7 +207,7 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - wait_until_responds("http://localhost:8042/", verify=client_cert) + wait_until_responds("https://localhost:8042/", verify=client_cert) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert

My speculation about what was happening here: when wait_until_responds() would time out due to SSL connection problems, because .terminate() isn’t in a finally, the datasette process wouldn’t get killed. That could (1) hang CI and (2) cause all your future local test runs to mysteriously fail because they’d be secretly talking to that old datasette process still hanging around from a past test run with an old temporary server certificate, and that old server cert wouldn’t validate against your newly-created ca cert.

A finally for .terminate() would help; a fancier version could be a context manager for running the external datasette process that could: - ensure the process always exited when no longer needed - if you want to be fancy, call terminate(), wait() for a short timeout for the process to exit, then try kill() and wait() again; raise an exception complaining about the seemingly-unkillable process if all that fails - raise an error if the process exited with a non-zero error code; here it’s likely that some datasette executions were secretly failing with [Errno 48] error while attempting to bind on address ('127.0.0.1', 8042): address already in use

A bit late to this, but I have made an app to publish air quality data in Bristol, UK. air quality data in Bristol, UK. Next step to see if I can make a streamlit app based on this to produce some nice charts.

the problem is: .select-wrapper select:focus { outline: none; }

I sometimes add this js: window.addEventListener('keydown', function check_tab(e) { if (e.key === 'Tab') { document.documentElement.classList.add('user-is-tabbing') window.removeEventListener('keydown', check_tab) } })

and then in the css, using a html.user-is-tabbing selector undo any outlines I removed.

Codecov Report

Base: 92.19% // Head: 92.05% // Decreases project coverage by -0.13% :warning:

Coverage data is based on head (770879a) compared to base (0b68996). Patch coverage: 100.00% of modified lines in pull request are covered.

:exclamation: Current head 770879a differs from pull request most recent head f42bca8. Consider uploading reports for the commit f42bca8 to get more accurate results

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

A related historical problem:

https://github.com/tekartik/sqflite/issues/525#issuecomment-714500720

I wonder if the version of Sqlite or Python for Intel chip have defensive mode disabled by default, whereas M1 chips versions have it enabled.

Hi @ryascott, thanks for sharing this! How did you upgrade your sqlite3 version? I'm running into this issue (also on an m1) with

Python ==3.10.7 sqlite3.sqlite_version==3.37.0 sqlite-utils==3.30

Unfortunately, 3.10.8 isn't listed in pyenv so I'm unable to install it.

For me, the trigger is trying to use the add-foreign-key command on its own:

bash sqlite-utils add-foreign-key library.db book_creators creator_id creators id

Some stackoverflow searching suggests that brew installing sqlite may fix it ( https://stackoverflow.com/questions/26345972/how-do-i-upgrade-the-sqlite-version-used-by-pythons-sqlite3-module-on-mac ), but I don't want to risk breaking the version of sqlite used by some other system, I'd only like to upgrade sqlite3 inside my current virtual environment.

Awesome. Thank you @simonw.

i'm also getting bit by this. I'm trying to set up an nginx reverse proxy in front of multiple datasette backends.

When I run it locally or behind the proxy, I see the base_url value added a second time to the path for various action links on table pages (view as JSON, sort by column, etc).

I don't have much experience with Python; however, I wonder if this print statement needs flush=True?

https://github.com/simonw/datasette/blob/fdf7c27b5438f02153c3a7f8ad1b320e4b29e4f4/datasette/cli.py#L621

Hmm, it appears after I kill the process with Ctrl+c:

FYI @simonw, I don't see that message.

I may have found the reason. I don't think the --root option is taking effect.

Visited: http://127.0.0.1:8001/-/permissions

Codecov Report

Base: 92.00% // Head: 92.03% // Increases project coverage by +0.02% :tada:

Coverage data is based on head (a1317ab) compared to base (e539c1c). Patch coverage: 100.00% of modified lines in pull request are covered.

:exclamation: Current head a1317ab differs from pull request most recent head 94e5c75. Consider uploading reports for the commit 94e5c75 to get more accurate results

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Ah, yes! Was just trying to do this and had the same issue. +1 to this!

Codecov Report

Base: 92.00% // Head: 92.01% // Increases project coverage by +0.01% :tada:

Coverage data is based on head (6e35a6b) compared to base (e539c1c). Patch coverage: 100.00% of modified lines in pull request are covered.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Base: 90.42% // Head: 91.77% // Increases project coverage by +1.34% :tada:

Coverage data is based on head (645be0f) compared to base (cab5b60). Patch coverage: 95.55% of modified lines in pull request are covered.

:exclamation: Current head 645be0f differs from pull request most recent head 7cd6fd9. Consider uploading reports for the commit 7cd6fd9 to get more accurate results

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Base: 91.73% // Head: 91.49% // Decreases project coverage by -0.24% :warning:

Coverage data is based on head (e8ae41e) compared to base (93ababe). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Reference: tqdm (https://tqdm.github.io/) shows a progress bar when total is known, and falls back to counting units of work done for streams. File input vs. stdin seems like a similar situation.

Current behavior is different when importing via stdin vs. a file. Imports from a file give a progress bar. For this new request, I'd love to see total imported and total ignored/replaced in both cases.

--verbose or --verbosity=ABC were the flags I looked for. Expected to see them at a global level near --version. But only sharing because that's where I looked first, I don't have a strong opinion on the exact wording/location.

Codecov Report

Base: 90.42% // Head: 90.42% // No change to project coverage :thumbsup:

Coverage data is based on head (9928ff1) compared to base (cab5b60). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

This would be helpful.

Happy birthday to datasette and thank you Simon for your continued effort on this project!

I use datasette (python) as a fast layer on top of search for github projects using https://github.com/dogsheep/github-to-sqlite , and use the JSON API it provides to serve sample data to make Vega-Lite graphing workshop examples that don't require authentication/API keys. It's awesome to have a full SQL API support working without needing to develop any custom API middleware for both filtering and grouping.

I've also enjoyed using it as a teaching tool for working with public dataset in civic data workshops and as a platform for making visualization plugins . I

I'm especially excited about datasette-lite, as it will let people participate in future editions of this workshop without having to install anything to make use of their own tables :)

I guess it is not incorrect when it says the version is 4, though it is confusing. Maybe it doesn't even refer to FTS4/FTS5 versions, but something else? In any case, it's not related to sqlite-utils, but SQLite itself.

Okay, my final observations for the night! I've been pushing and pulling the various levers in utils/__init__.py to see what makes this work without hard-coding in something for arm64 and it seems that if I change /usr/lib/x86_64-linux-gnu/mod_spatialite.so here to just mod_spatialite it's happy.

Unfortunately cannot audit that for x86_64, but maybe that's a solution that'd be cross-arch compatible? It seems like it's the hard-coding of that path that's tripping it up.

(It was actually this comment from back in 2018 in an entirely unrelated repo that nudged me to try this, ha.)

Can confirm that my uname -a returns something different at the end:

root:xnu-8792.41.9~2/RELEASE_ARM64_T6000 arm64

I'm in arm64 land, you're in x86_64. I am admittedly very fuzzy on how this factors into Docker these days. Honestly thought this was one of the things Docker was suppose to help address. 🤔

Is it, uh, possible we are on different architectures? 😅 I'm using an Apple M1 Pro.

I jumped into a bash shell of an unmodified python:3.11.0-slim-bullseye container and manually ran apt-get update and installed libsqlite3-mod-spatialite. I don't end up with with mod_spatialite.so in /usr/lib/x86_64-linux-gnu/ — mine is in /usr/lib/aarch64-linux-gnu/.

I swapped that directory in here in a local copy of datasette and poof — it worked!

Interesting! So I tried this locally using your copy of nps-spatialite.db and I got the same error. 🤔

``` ❯ datasette package nps-spatialite.db --spatialite [+] Building 27.5s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 622B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye 0.9s => [internal] load build context 2.3s => => transferring context: 72.38MB 2.3s => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c 0.0s => [2/6] COPY . /app 0.1s => [3/6] WORKDIR /app 0.0s => [4/6] RUN apt-get update && apt-get install -y python3-dev gcc libsqlite3-mod-spatialite && rm -rf /var/lib/apt/lists/* 18.5s => [5/6] RUN pip install -U datasette 4.9s => ERROR [6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json 0.7s

[6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json:

10 0.681 Traceback (most recent call last):

10 0.681 File "/usr/local/bin/datasette", line 8, in <module>

10 0.681 sys.exit(cli())

10 0.681 ^^^^^

10 0.681 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1130, in call

10 0.682 return self.main(args, *kwargs)

10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.682 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1055, in main

10 0.682 rv = self.invoke(ctx)

10 0.682 ^^^^^^^^^^^^^^^^

10 0.682 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1657, in invoke

10 0.682 return _process_result(sub_ctx.command.invoke(sub_ctx))

10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.682 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke

10 0.682 return ctx.invoke(self.callback, **ctx.params)

10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.682 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 760, in invoke

10 0.682 return __callback(args, *kwargs)

10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/cli.py", line 164, in inspect

10 0.683 inspect_data = loop.run_until_complete(inspect_(files, sqlite_extensions))

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete

10 0.683 return future.result()

10 0.683 ^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/cli.py", line 179, in inspect_

10 0.683 counts = await database.table_counts(limit=3600 * 1000)

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 304, in table_counts

10 0.683 for table in await self.table_names():

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 342, in table_names

10 0.683 results = await self.execute(

10 0.683 ^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 267, in execute

10 0.683 results = await self.execute_fn(sql_operation_in_thread)

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 213, in execute_fn

10 0.683 return await asyncio.get_event_loop().run_in_executor(

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run

10 0.683 result = self.fn(self.args, *self.kwargs)

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 209, in in_thread

10 0.683 self.ds._prepare_connection(conn, self.name)

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/app.py", line 593, in _prepare_connection

10 0.683 conn.execute("SELECT load_extension(?)", [extension])

10 0.683 sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory

executor failed running [/bin/sh -c datasette inspect nps-spatialite.db --inspect-file inspect-data.json]: exit code: 1 ```

The virtual table's _config version: 4 seems to indicate FTS5.

Codecov Report

Base: 92.55% // Head: 92.55% // No change to project coverage :thumbsup:

Coverage data is based on head (5eb4ea4) compared to base (00e233d). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Base: 92.55% // Head: 92.55% // No change to project coverage :thumbsup:

Coverage data is based on head (f254be4) compared to base (6f610e1). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

I appreciate your response @simonw - thanks!

I'll clarify what we need further - let's imagine we have 2000 SQLLite databases (for 2000 tenants), but we only want to run one datasette instance for each of those tenants to query/use datasette against their own database only. This means the "connection" between datasette and the SQLLite database would be dynamic, based on the tenantID that's required on an incoming request.

Is there any specific config or other considerations in this use case, to minimize memory use on a single, efficient VM and serve queries to all these tenants?

cc @muadham

Happy Birthday Datasette!

Thanks Simon!!

I use datasette on everything most notably my flickr metadata SQLite DB to make art.

Datasette lite on my 2019 flickr metadata is super helpful too: https://lite.datasette.io/?csv=https%3A%2F%2Fraw.githubusercontent.com%2Frtanglao%2Frt-flickr-sqlite-csv%2Fmain%2F2019-roland-flickr-metadata.csv

Even better datasette lite on all firefox support questions from 2021: https://lite.datasette.io/?url=https%3A%2F%2Fraw.githubusercontent.com%2Frtanglao%2Frt-kits-api3%2Fmain%2FYEARLY_CSV_FILES%2F2021-firefox-sumo-questions.db

Thanks again Simon! So great! What a gift to the world!!!!!!

I have no particular point, I just want to say being around for the 1.0 release of Datasette seems historic and legendary to witness.

Datasette usage comments for its 5th anniversary celebration:

I use Datasette and related tools for a Cosmology Researcher Talks database app project, which is described in the github Readme

The app hosted on the Google Cloud Run service also uses other Datasette-related tools developed by Simon - datasette-render-markdown, csvs-to-sqlite, datasette-template-sql, and datasette-block-robots. This is one of two apps used for querying the talks database, each has it pros/cons as described in the github Readme.

At present, over 170 different sites that host cosmology talks are scraped to collect new talks for import into the sqlite database. The shot-scraper and sqlite-utils tools are a major help for this.

I also use the Mastodon API to get my favorites, toots, and boosts into a local database so I can do searches on the data. This was done on Twitter and was then extended to the Mastodon data. Again, sqlite-utils is an important tool for this.

Happy Birthday Datasette!

I am a librarian at the Université du Québec à Montréal (UQAM) and I've been using Datasette to publish excerpts of our library data. There are several use cases I'm working with as a proof of concept : 1. New titles list : based on reports of recent acquisitions by subject, discipline, etc. 2. List of all UQAM theses and dissertations : based on an extract of bibliographic records 3. List of all publications by UQAM Authors : based on an extract of bibliographic records

See our prototype under construction here : https://datasette-bib.uqam.ca/ (some bits and pieces have been translated into French)

Datasette is amazing, there is so much potential here for libraries. Thanks to Simon and all the contributors for this outstanding effort. Also sqlite-utils deserves special mention as incredibly handy and useful.

I use Datasette to analyze blocklists by using csv-to-sqlite to pull their contents into a database and Datasette to look around through them. I also use its REST API to query said database as part of filtering out garbage from domains found in those blocklists.

Nothing spectacular yet but I think this falls under "cool/cute application of datasette": improving fakedata performance for fun. tl;dr I used datasette to visualize benchmarking data.

I'm a cryptic crossword enthusiast and have spent a lot of time scraping and parsing cryptic crossword clues from various blogs, forums and publications. The result is over half a million clues from cryptic crosswords over the past twelve years, including the clue, answer, puzzle date, puzzle name and a link to the original source. This is all hosted using Datasette, which has been a delight to use: https://cryptics.georgeho.org/

This dataset is a significant work of crossword archivism and scholarship, as acquiring historical crosswords and structuring their contents require focused effort and tedious cleaning that few are willing to do for such trivial data - for example, according to this 2004 selection guide, the Library of Congress explicitly does not collect crossword puzzles. Anecdotally, I know that many constructors/setters of cryptic crosswords use this dataset as a resource, and some even simply call it "the database" - this is probably one of the most impactful data projects I've worked on!

I work at The Wall Street Journal as a computational journalist and serve as our self-appointed Datasette evangelist. They say that to a hammer everything looks like a nail, but the reality is newsrooms find themselves in a sea of nails!

I've only got a couple public projects that I can share, but happy to offer you a look at some of the internal projects.

More often than not the internal projects stay internal because the reporting doesn't lead anywhere or I can't convince an editor to greenlight it. But imho that's the beauty of datasette: a (relatively) painless mechanism to see if there's any there there.

• WSJ Inflation Tracker
• I scraped the oscars website and turned it into a datasette instance and ran the numbers on best actress/best picture overlap

Hi Simon. I have just started experimenting with datasette in earnest, looking at it's suitability for air quality open data. A bulk upsert \ upsert_all would be very useful for me in enabling real time data to be pushed from a sql server database with FME server to a datasette db.

An hourly process queries the last 2 hours of data and pushes that to my database, inserting new data and updating existing combinations of pk siteid and date_time. This is already implemented on our current open data portal. Excited to see your progress with this! Thank you for this amazing software.