issue_comments

We can use this here: https://click.palletsprojects.com/en/8.1.x/options/#values-from-environment-variables python @click.option(..., envvar="DATASETTE_INTERNAL")

Here's the thinking: setting theDATASETTE_INTERNAL environment variable for your entire machine will cause any time you run datasette x.db to result in a process that shares the same internal.db database as other instances.

You might run more than one instance at once (I often have 4 or 5 going). This would currently break, because they would over-write each other's catalog tables:

https://github.com/simonw/datasette/blob/e4abae3fd7a828625d00c35c316852ffbaa5ef2f/datasette/utils/internal_db.py#L5-L62

The breaking wouldn't be obvious because the catalog tables aren't used by any features yet, but it's still bad.

This convinced us that actually we should move those catalog_ tables OUT of internal.db. The _internal database will be reserved for plugins that want to use it for caching, storing progress, etc.

I think we move them to an in-memory _catalog database which is excluded from ds.databases (like _internal is ) but can be accessed using datasette.get_catalog_database() - similar to datasette.get_internal_database().

So each instance of Datasette gets its own truly private _catalog, which is in-memory and so gets cleared at the end of each process.

An interesting thing that came up about a shared _internal database is that it provides opportunities for things like a notes plugin which allows you to attach notes to any row in any table in a database... where those notes become available to multiple Datasette instances that you might launch on the same laptop.

DATASETTE_INTERNAL would be more consistent with DATASETTE_SECRET.

The test failure was while installing numpy, relating to importing distutils - maybe relevant: - https://github.com/pypa/setuptools/issues/3661

25h Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' ERROR: Exception: Traceback (most recent call last): ... File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_internal/utils/misc.py", line 697, in get_requires_for_build_wheel return super().get_requires_for_build_wheel(config_settings=cs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel return self._call_hook('get_requires_for_build_wheel', { ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook raise BackendUnavailable(data.get('traceback', '')) pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend obj = import_module(mod_path) ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/importlib/__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1381, in _gcd_import File "<frozen importlib._bootstrap>", line 1354, in _find_and_load File "<frozen importlib._bootstrap>", line 1304, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 1381, in _gcd_import File "<frozen importlib._bootstrap>", line 1354, in _find_and_load File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 929, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 994, in exec_module File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "/tmp/pip-build-env-x9nyg3kd/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module> import distutils.core ModuleNotFoundError: No module named 'distutils'

Surprising error: blacken-docs -l 60 docs/*.rst shell: /usr/bin/bash -e {0} env: pythonLocation: /opt/hostedtoolcache/Python/3.11.4/x64 PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib/pkgconfig Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64 Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64 Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64 LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib docs/writing_plugins.rst:365: code block parse error Cannot parse: 8:0: <line number missing in source>

Running tests to see if Pyodide works - that was the reason I switched to my click-default-group-wheel package originally.