New in SQLite 3.37.0, released a few days ago: https://www.sqlite.org/stricttables.html

I keep making this mistake: bash sqlite-utils add-column content.db assets _since int ``` Usage: sqlite-utils add-column [OPTIONS] PATH TABLE COL_NAME [[integer|float|b lob|text|INTEGER|FLOAT|BLOB|TEXT]] Try 'sqlite-utils add-column -h' for help.

Error: Invalid value for '[[integer|float|blob|text|INTEGER|FLOAT|BLOB|TEXT]]': 'int' is not one of 'integer', 'float', 'blob', 'text', 'INTEGER', 'FLOAT', 'BLOB', 'TEXT'. ```

It would also be interesting to try out the SQL hint mode, which can autocomplete against tables and columns. This demo shows how to configure that: https://codemirror.net/mode/sql/

Some missing documentation: https://stackoverflow.com/questions/20023381/codemirror-how-add-tables-to-sql-hint Originally posted by @simonw in https://github.com/simonw/datasette/issues/948#issuecomment-679355426

I've started to encounter a bug where queries to tables inside named in-memory databases sometimes trigger server hangs.

I'm still trying to figure out what's going on here - on one occasion I managed to Ctrl+C the server and saw an exception that mentioned a thread lock, but usually hitting Ctrl+C does nothing and I have to kill -9 the PID instead.

This is all running on my M2 Mac.

I've seen the bug in the Datasette 1.0 alphas and in Datasette 0.64.3 - but reverting to 0.61 appeared to fix it.

Got this running tests against Python 3.11.

../../../.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette/app.py:14: in <module> import pkg_resources ../../../.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/pkg_resources/__init__.py:121: in <module> warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning) E DeprecationWarning: pkg_resources is deprecated as an API

I ran with pytest -Werror --pdb -x to get the debugger for that warning, but it turned out searching the code worked better. It's used in these two places:

https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/plugins.py#L43-L50

https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/app.py#L1037

https://github.com/simonw/datasette/actions/runs/6266449018/job/17017460074

This is a bit of a mystery, I don't think I've changed anything recently that could have broken this.

That passed on 3.8 but should have failed: https://github.com/simonw/datasette/actions/runs/6266341481/job/17017099801 - the "Test DATASETTE_LOAD_PLUGINS" test shows errors but did not fail the CI run.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/2057#issuecomment-1730201226

I just spotted a bug when using https://datasette.io/plugins/datasette-configure-fts and https://datasette.io/plugins/datasette-edit-schema at the same time.

Steps to reproduce:

• Configure FTS for a table, then run a test search
• Edit the schema for that table and change the order of columns
• Run the test search again

I got the wrong search results, which I think is because the _fts table pointed to the first table by rowid but those rowid values were entirely rewritten as a consequence of running table.transform() on the table.

Reconfiguring FTS on the table fixed the problem.

I think table.transform() should be able to preserve rowid values.

In building Datasette Cloud we realized that a bunch of the features we are building need a way of resolving an actor ID to the actual actor, in order to display something more interesting than just an integer ID.

Social plugins in particular need this - comments by X, CSV uploaded by X, that kind of thing.

I think the solution is a new plugin hook: actors_from_ids(datasette, ids) which can return a list of actor dictionaries.

The default implementation can return [{"id": "..."}] for the IDs passed to it.

Pluggy has a firstresult=True option which is relevant here, since this is the first plugin hook we will have implemented where only one plugin should provide an answer.

I'm going to try and get this working through an environment variable, so that you can start Datasette and it will only load a subset of plugins including those that use the register_commands() hook.

Initial research on this: - https://github.com/pytest-dev/pluggy/issues/422

Spotted this problem while working on this plugin: - https://github.com/simonw/datasette-public

It's possible to make a table public to any users - but then you may end up with situations like this:

That table is public, but the foreign key links go to tables that are NOT public.

We're also leaking the names of the values in those private tables here, which we shouldn't do. So this is a tiny bit of an information leak.

Since this only affects people who have configured a table to be public that has foreign keys to a table that is private I don't think this is worth issuing a vulnerability report about - I very much doubt anyone is running Datasette configured in a way that could result in problems because of this.

This test here: https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/tests/test_api.py#L1011-L1020

It failed for me like this:

E AssertionError: assert [('normal', False), ('sqlite_stat1', True), ('sqlite_stat4', True)] in ([('normal', False), ('sqlite_stat1', True)],)

Looks like some builds of SQLite include a sqlite_stat4 table.

I've used this pattern twice now: https://til.simonwillison.net/datasette/register-new-plugin-hooks - in datasette-graphql and datasette-low-disk-space-hook. I should describe the pattern on https://docs.datasette.io/en/stable/writing_plugins.html

I noticed that a token with execute-sql permission alone did not work, because it was not allowed to view the instance of the database.

The BaseView class has a method for cascading permission checks, but it's not easily accessible to plugins.

https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/views/base.py#L75-L99

This leaves plugins like datasette-graphql having to implement their own versions of this logic, which is bad: https://github.com/simonw/datasette-graphql/issues/65

First check view-database - if that says False then disallow access, if it says True then allow access. If it says None check view-instance.

This should become a supported API that plugins are encouraged to use.

Discussed with Alex this morning. We think the American spelling is fine here (it's shorter than catalogue) and that it's a slightly less lazy name than core_.

Follows: - https://github.com/simonw/datasette/issues/2157

Spotted a problem while working on this: if you grant a token access to view table for a specific table but don't also grant view database and view instance permissions, that token is useless.

This was a deliberate design decision in Datasette - it's documented on https://docs.datasette.io/en/1.0a2/authentication.html#access-permissions-in-metadata

If a user cannot access a specific database, they will not be able to access tables, views or queries within that database. If a user cannot access the instance they will not be able to access any of the databases, tables, views or queries.

I'm now second-guessing if this was a good decision.

Originally posted by @simonw in https://github.com/simonw/datasette-auth-tokens/issues/7#issuecomment-1636031702

I experimented with a prototype of this here: - https://github.com/simonw/datasette/issues/2102#issuecomment-1691037971_

Which lets me run requests as if they belonged to a specific actor like this:

bash datasette fixtures.db --get '/fixtures/facetable.json' --actor '{ "_r": { "r": { "fixtures": { "facetable": [ "vt" ] } } }, "a": "user" }'

Really useful for testing actors an _r options. Is this worth adding as a feature?

In thinking about the best way to implement https://github.com/simonw/datasette-auth-passwords/issues/6 (SQL-backed user accounts for datasette-auth-passwords) I realized that there are a few different use-cases where a plugin might want to store data that isn't visible to regular Datasette users:

• Storing password hashes
• Storing API tokens
• Storing secrets that are used for data import integrations (secrets for talking to the Twitter API for example)

Idea: allow one or more private database files to be attached to Datasette, something like this:

datasette github.db linkedin.db -s secrets.db -m metadata.yml

The secrets.db file would not be visible using any of the Datasette's usual interface or API routes - but plugins would be able to run queries against it.

So datasette-auth-passwords might then be configured like this:

yaml plugins: datasette-auth-passwords: database: secrets sql: "select password_hash from passwords where username = :username" The plugin could even refuse to operate against a database that hadn't been loaded as a secret database.

See: - https://github.com/simonw/datasette-configure-fts/issues/14 - https://github.com/simonw/datasette-auth-tokens/issues/12

It's EOL now, as of 2023-06-27 (two days ago): https://devguide.python.org/versions/

I'm going to drop Python 3.7.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1153#issuecomment-1627455892

It's not supported any more: https://devguide.python.org/versions/

It is increasingly being useful configuration options, when it started out as purely metadata.

Could cause confusion with the --config mechanism though - maybe that should be called "settings" instead?

Split off from #274

Metadata should an optional "config" section at root, table or database level.

The TableView and RowView and DatabaseView and BaseView classes could all have a .config("key") method which knows how to resolve the hierarchy of configs.

This will allow individual tables (or databases) to set their own config settings for things like sql_time_limit_ms

Extremely detailed story of how I got to this point:

• https://github.com/simonw/llm/issues/162

Steps to reproduce (only if that pragma is on though): bash python -c ' import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute(""" CREATE TABLE "logs" ( [id] INTEGER PRIMARY KEY, [model] TEXT, [prompt] TEXT, [system] TEXT, [prompt_json] TEXT, [options_json] TEXT, [response] TEXT, [response_json] TEXT, [reply_to_id] INTEGER, [chat_id] INTEGER REFERENCES [log]([id]), [duration_ms] INTEGER, [datetime_utc] TEXT ); """) db["logs"].add_foreign_key("reply_to_id", "logs", "id") ' This succeeds in some environments, fails in others.

https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1165-L1174

This is the only place in the code that attempts to modify sqlite_master directly, which fails on some Python installations.

Could this use the .transform() trick instead?

Or automatically switch to that trick if it hits an error?

The new options added in: - #577 Deserve consideration in the CLI as well.

https://github.com/simonw/sqlite-utils/blob/d2bcdc00c6ecc01a6e8135e775ffdb87572b802b/sqlite_utils/db.py#L1706-L1708

I have a neater pattern for this now - instead of: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_create.py#L472-L475

I now prefer:

https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_create.py#L1163-L1171

For example this: https://github.com/simonw/datasette/blob/4535568f2ce907af646304d0ebce2500ebd55677/docs/authentication.rst?plain=1#L33-L35

The problem with that $ prefix is that it prevents users from copying and pasting the raw command.

https://docs.datasette.io/en/stable/authentication.html#using-the-root-actor

I think this has been a bug for a long time.

https://latest.datasette.io/ currently shows:

Those colors are based on a hash of the database name. But when you click through to https://latest.datasette.io/fixtures

It's red on all sub-pages too.

Spotted this on https://latest.datasette.io/extra_database

```html

This would be useful while working on: - #2114

Or... how about if the on_success_message option could define a SQL query to be executed to generate that message? Maybe on_success_message_sql.

• https://github.com/simonw/datasette/issues/2134

The default JSON just isn't right. I find myself using ?_shape=array for almost everything I build against the API.

118 commits! https://github.com/simonw/datasette/compare/1.0a2...26be9f0445b753fb84c802c356b0791a72269f25

I just noticed that https://latest.datasette.io/fixtures/roadside_attraction_characteristics.json?_labels=on&_size=1 returns: json { "ok": true, "next": "1", "rows": [ { "rowid": 1, "attraction_id": { "value": 1, "label": "The Mystery Spot" }, "characteristic_id": { "value": 2, "label": "Paranormal" } } ], "truncated": false } But https://latest.datasette.io/fixtures.json?sql=select+rowid%2C+attraction_id%2C+characteristic_id+from+roadside_attraction_characteristics+order+by+rowid+limit+1 returns: json { "rows": [ { "rowid": 1, "attraction_id": 1, "characteristic_id": 2 } ], "columns": [ "rowid", "attraction_id", "characteristic_id" ], "ok": true, "truncated": false } The columns key in the query response is inconsistent with the table response.

From Discord: https://discordapp.com/channels/823971286308356157/996877076982415491/1068227071156965486

• 2109

Caused by: - #2112

https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%22pk1%22+%3D+%3Ap0+order+by+pk1%2C+pk2%2C+pk3+limit+101&p0=b

The :p0 form field is missing. Submitting the form results in this error:

In the old code:

https://github.com/simonw/datasette/blob/08181823990a71ffa5a1b57b37259198eaa43e06/datasette/views/database.py#L34-L35

This name class attribute was later used by some of the plugin hooks, passed as view_name: https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/hookspecs.py#L50-L54

Figure out how that should work once I've refactored those classes to view functions instead.

Refs: - #2109

• 2109

This will help make the hook as robust as possible. - #2109

This will be the base that the remaining work builds on top of. Refs: - #2109

While working on: - https://github.com/simonw/datasette/pull/2118

It became clear that the render callback function documented here: https://docs.datasette.io/en/0.64.3/plugin_hooks.html#register-output-renderer-datasette

Needs to grow the ability to be told if an error occurred (an error string) and if the results were truncated (a truncated boolean).

I've been stuck on this for too long. I'm breaking it down into a full milestone:

https://github.com/simonw/datasette/milestone/29

While using sqlite-utils convert I realized it would be handy if you could pass --pdb to have it open the debugger at the first instance of a failed conversion.

Refs: - #2109

The idea here is that hitting /content without a ?sql= will show an empty result set AND default to including a bunch of extras about the list of tables in the database.

Then I won't have to think about /content and /content?sql= as separate pages any more.

A minor refactor, but it is a good starting point for this new branch. Refs: - #2109

As seen in LLM and now in sqlite-utils too: - https://github.com/simonw/sqlite-utils/issues/570

Useful for developing plugins, see tutorial at https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html

Currently the changelog starts at 0.4:

https://sqlite-utils.datasette.io/en/3.34/changelog.html#id115

I want the other releases - according to https://pypi.org/project/sqlite-utils/#history there are three missing:

Plugins affecting the CLI by default makes sense to me.

I'm less confident about them always affecting users of the Python API.

I'm going to have them apply by default, but I'm going to add a mechanism to opt-out on an individual database basis. Basically this:

```python from sqlite_utils import Database db = Database(memory=True, execute_plugins=False)

Anything using db from here on will not execute plugins

``` cc @asg017

Refs: - #567 - #574

Splitting off an issue for prepare_connection() since Alex got the PR in seconds before I shipped 3.34!

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646686424

Also need a design for an option for the .transform() method to indicate that the new table should be created with a new name without dropping the old one.

I think keep_table="name_of_table" is good for this.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657324

Spotted this in the GitHub Actions logs:

... and it turns out running .transform() with no arguments still fixes the format of the schema!

```pycon

db["log"].add_column("foo", str)

For every command/API method there should be a link to the equivalent in the other form factor.

Maybe also link to the API and CLI reference pages too.

I find myself wanting two new features in sqlite-utils: - The ability to have the new transformed table set to a specific name, while keeping the old table around - The ability to rename a table (sqlite-utils doesn't have a table rename function at all right now)

Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1618375042

Found myself using this pattern to quickly prototype a schema:

```python import sqlite_utils db = sqlite_utils.Database(memory=True)

print(db["answers_chunks"].create({ "id": int, "content": str, "embedding_type_id": int, "embedding": bytes, "embedding_content_md5": str, "source": str, }, pk="id", transform=True).schema) ```

Using replace=True to drop and then recreate the table would be neat here, and would be consistent with other places that use replace=True.

I'm going to start by adding the register_command hook using the exact same pattern as Datasette and LLM.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643450

As seen in LLM.

Needed while working on: - #567

First of my CLI tools to use https://til.simonwillison.net/python/pyproject

YAML configuration is much better for multi-line strings, and I'm increasingly adding configuration options to Datasette that benefit from that - fragments of HTML in description_html or SQL queries used to configure things like https://github.com/simonw/datasette-atom for example.

Rather than confusing things by showing both in the documentation, I should switch all of the default examples to use YAML instead.

CSV files with empty cells in (which come through as the empty string) are common and a bit gross.

Having an option that means "and if it's an empty string store null instead) would be cool.

I brainstormed name options here https://chat.openai.com/share/c947b738-ee7d-419c-af90-bc84e90987da

There are still a bunch of bugs in CodeMirror 5 that affect various mobile browsers - see Datasette Discord report here: https://discord.com/channels/823971286308356157/823971286941302908/1013878624992108645

https://user-images.githubusercontent.com/9599/187349269-7b7c0c8c-3894-4810-82f0-de7c1eb940b3.mp4

e.g. in https://github.com/simonw/datasette/actions/runs/5413590227/jobs/9839373852

``` =================================== FAILURES =================================== _____ testhomepage _______ [gw0] linux -- Python 3.7.17 /opt/hostedtoolcache/Python/3.7.17/x64/bin/python

ds_client = <datasette.app.DatasetteClient object at 0x7f85d271ef50>

@pytest.mark.asyncio
async def test_homepage(ds_client):
    response = await ds_client.get("/.json")
    assert response.status_code == 200
    assert "application/json; charset=utf-8" == response.headers["content-type"]
    data = response.json()
    assert data.keys() == {"fixtures": 0}.keys()
    d = data["fixtures"]
    assert d["name"] == "fixtures"
    assert d["tables_count"] == 24
    assert len(d["tables_and_views_truncated"]) == 5
    assert d["tables_and_views_more"] is True
    # 4 hidden FTS tables + no_primary_key (hidden in metadata)
    assert d["hidden_tables_count"] == 6
    # 201 in no_primary_key, plus 6 in other hidden tables:

  assert d["hidden_table_rows_sum"] == 207, data

E AssertionError: {'fixtures': {'color': '9403e5', 'hash': None, 'hidden_table_rows_sum': 0, 'hidden_tables_count': 6, ...}} E assert 0 == 207 ``` My guess is that this is a timing error, where very occasionally the "count rows but stop counting if it exceeds a time limit" thing fails.

https://github.com/simonw/datasette/actions/runs/5413443676/jobs/9838999356 codespell docs/*.rst --ignore-words docs/codespell-ignore-words.txt codespell datasette -S datasette/static --ignore-words docs/codespell-ignore-words.txt shell: /usr/bin/bash -e {0} env: pythonLocation: /opt/hostedtoolcache/Python/3.9.17/x64 LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.17/x64/lib docs/metadata.rst:192: displaing ==> displaying This failure is legit, it found a spelling mistake: https://github.com/simonw/datasette/blob/ede62036180993dbd9d4e5d280fc21c183cda1c3/docs/metadata.rst#L192

I found myself wanting to insert rows from a 849MB CSV file without processing the whole thing: https://huggingface.co/datasets/jerpint-org/HackAPrompt-Playground-Submissions/tree/main

If sqlean is available, use that.

Refs: - https://github.com/nalgeon/sqlean.py/issues/1#issuecomment-1605707788

This will provide a good workaround for: - #235

There are two patterns for implementing views in Datasette at the moment. I want to combine those.

Part of: - #2053

Currently the CORS headers served are:

https://github.com/simonw/datasette/blob/9584879534ff0556e04e4c420262972884cac87b/datasette/utils/init.py#L1139-L1143

Serving Access-Control-Max-Age: 600 would allow browsers to cache that for 10 minutes, avoiding additional CORS pre-flight OPTIONS requests during that time.

https://click.palletsprojects.com/en/8.1.x/shell-completion/

This works for zsh:

eval "$(_SQLITE_UTILS_COMPLETE=zsh_source sqlite-utils)"

This will probably work for bash:

eval "$(_SQLITE_UTILS_COMPLETE=bash_source sqlite-utils)"

Need to add this to the installation docs here: https://sqlite-utils.datasette.io/en/stable/installation.html - along with the pattern for adding that to .zshrc or whatever.

e.g. in this section:

https://sqlite-utils.datasette.io/en/stable/cli.html#running-queries-directly-against-csv-or-json

The little copy button will also copy the $ which breaks the examples when copied.

https://github.com/Textualize/trogon

https://github.com/simonw/sqlite-utils/actions/runs/5039064797/jobs/9036965488

Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.7.16/x64/bin/flake8", line 8, in <module> sys.exit(main()) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/cli.py", line 22, in main app.run(argv) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py", line 363, in run self._run(argv) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py", line 350, in _run self.initialize(argv) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py", line 330, in initialize self.find_plugins(config_finder) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py", line 153, in find_plugins self.check_plugins = plugin_manager.Checkers(local_plugins.extension) File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 357, in __init__ self.namespace, local_plugins=local_plugins File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 238, in __init__ self._load_entrypoint_plugins() File "/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 254, in _load_entrypoint_plugins eps = importlib_metadata.entry_points().get(self.namespace, ()) AttributeError: 'EntryPoints' object has no attribute 'get' Error: Process completed with exit code 1.

Noticed this while testing: - #547

If you pass a non-existent column to -c/--column you don't get an error message.

Noticed this:

``` % sqlite-utils analyze-tables content.db repos -c delete_branch_on_merge --common-limit 20 --no-least repos.delete_branch_on_merge: (1/1)

Total rows: 158 Null rows: 158 Blank rows: 0

Distinct values: 0

Most common: 158: None ```

The 158: None there is duplicate information considering we already know there are 158/158 null rows.

The "least common" section is frequently uninteresting, especially for huge tables with a large number of repeated-once values.

sqlite-utils analyze-tables content.db repos --common-limit 20 --no-least

I prototyped that here: https://github.com/simonw/one-off-actions/blob/main/.github/workflows/deploy-datasette-branch-preview.yml

It deployed the json-extras-query branch here: https://datasette-preview-json-extras-query.vercel.app/

https://github.com/simonw/sqlite-utils/actions/runs/4920084038/jobs/8788501314 python sql = 'update [example] set [dt] = lambda_-9223371942137158589([dt]);' From: - #526

I got this error:

File "/Users/simon/Dropbox/Development/openai-to-sqlite/openai_to_sqlite/cli.py", line 27, in embeddings rows, _ = rows_from_file(input) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/simon/.local/share/virtualenvs/openai-to-sqlite-jt4obeb2/lib/python3.11/site-packages/sqlite_utils/utils.py", line 305, in rows_from_file first_bytes = buffered.peek(2048).strip() ^^^^^^^^^^^^^^^^^^^ From this code: ```python

@cli.command() @click.argument( "db_path", type=click.Path(file_okay=True, dir_okay=False, allow_dash=False), ) @click.option( "-i", "--input", type=click.File("r"), default="-", ) def embeddings(db_path, input): "Store embeddings for one or more text documents" click.echo("Here is some output") db = sqlite_utils.Database(db_path) rows, _ = rows_from_file(input) print(list(rows)) `` The error went away when I changed it totype=click.File("rb")`.

This should either be called out in the documentation or rows_from_file() should be fixed to handle text-mode files in addition to binary files.

https://readthedocs.org/projects/sqlite-utils/builds/20512693/ ``` Running Sphinx v6.2.1

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py", line 442, in load_extension mod = import_module(extname) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 783, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/builders/linkcheck.py", line 20, in <module> from requests import Response File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/requests/init.py", line 43, in <module> import urllib3 File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/urllib3/init.py", line 38, in <module> raise ImportError( ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main app = Sphinx(args.sourcedir, args.confdir, args.outputdir, File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py", line 225, in init self.setup_extension(extension) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py", line 404, in setup_extension self.registry.load_extension(self, extname) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py", line 445, in load_extension raise ExtensionError(__('Could not import extension %s') % extname, sphinx.errors.ExtensionError: Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168)

Extension error: Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168) ```

I wanted to output newline-separated output of the first column of every row in the results - like --row but for more than one line.

https://github.com/mitsuhiko/rye

I tried this:

rye install datasette

But now:

% ~/.rye/shims/datasette Traceback (most recent call last): File "/Users/simon/.rye/shims/datasette", line 5, in <module> from datasette.cli import cli File "/Users/simon/.rye/tools/datasette/lib/python3.11/site-packages/datasette/cli.py", line 17, in <module> from .app import ( File "/Users/simon/.rye/tools/datasette/lib/python3.11/site-packages/datasette/app.py", line 14, in <module> import pkg_resources ModuleNotFoundError: No module named 'pkg_resources' I think that's because setuptools is not included in Rye.

https://github.com/simonw/datasette/actions/runs/4815723640/jobs/8574667731 ``` def urlparse(url: str = "", **kwargs: typing.Optional[str]) -> ParseResult: # Initial basic checks on allowable URLs. # ---------------------------------------

    # Hard limit the maximum allowable URL length.
    if len(url) > MAX_URL_LENGTH:

      raise InvalidURL("URL too long")

E httpx.InvalidURL: URL too long

/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/httpx/_urlparse.py:155: InvalidURL =========================== short test summary info ============================ FAILED tests/test_csv.py::test_max_csv_mb - httpx.InvalidURL: URL too long ```

Got this error while investigating: - #421

Even though I was using the LD_PRELOAD trick from https://til.simonwillison.net/sqlite/ld-preload to use a newer version of SQLite.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354

It looks like Swarm changed their schema and now both venues and checkins have createdAt fields.

Which breaks this view: https://github.com/dogsheep/swarm-to-sqlite/blob/719b6e96a016d0ca8b316d3bed9c2a7a0cb499ee/swarm_to_sqlite/utils.py#L171-L188

https://sqlite-utils.datasette.io/en/2.14.1/ isn't showing a link to the stable release right now.

I should also apply the same fix I used for Datasette in: - https://github.com/simonw/datasette/issues/1608

TIL: https://til.simonwillison.net/readthedocs/link-from-latest-to-stable

It's not currently clear what the difference between https://docs.datasette.io/en/latest/ and https://docs.datasette.io/en/stable/ is - I should fix that.

On Twitter: https://twitter.com/simonw/status/1484285006243528705

Spotted this in: - #1999

POST /db/table currently executes obsolete code for inserting a row - I replaced that with /db/table/-/insert in https://github.com/simonw/datasette/commit/6e788b49edf4f842c0817f006eb9d865778eea5e but forgot to remove the old code.

This came up as a failure running tests for: - #531

Traceback on https://readthedocs.org/projects/sqlite-utils/builds/19749348/

``` File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 889, in interpreted nodes, messages2 = role_fn(role, rawsource, text, lineno, self) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/sphinx/ext/extlinks.py", line 103, in role title = caption % part TypeError: not all arguments converted during string formatting

Exception occurred: File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/sphinx/ext/extlinks.py", line 103, in role title = caption % part TypeError: not all arguments converted during string formatting ```

FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError: [Errno 2] No such file or directory

From https://github.com/simonw/datasette/actions/runs/4348548218/jobs/7597208191

``` =================================== FAILURES =================================== ____ test_install_requirements _______

run_module = <MagicMock name='run_module' id='139768358191936'>

@mock.patch("datasette.cli.run_module")
def test_install_requirements(run_module):
    runner = CliRunner()

  with runner.isolated_filesystem():

/home/runner/work/datasette/datasette/tests/test_cli.py:184:

/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/contextlib.py:119: in enter return next(self.gen)

self = <click.testing.CliRunner object at 0x7f1e5bfb9490>, temp_dir = None

@contextlib.contextmanager
def isolated_filesystem(
    self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None
) -> t.Iterator[str]:
    """A context manager that creates a temporary directory and
    changes the current working directory to it. This isolates tests
    that affect the contents of the CWD to prevent them from
    interfering with each other.

    :param temp_dir: Create the temporary directory under this
        directory. If given, the created directory is not removed
        when exiting.

    .. versionchanged:: 8.0
        Added the ``temp_dir`` parameter.
    """

  cwd = os.getcwd()

E FileNotFoundError: [Errno 2] No such file or directory

/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/click/testing.py:466: FileNotFoundError ``` Not sure why it only affected the "Calculate test coverage" one.

Notes can live in folders. These relationships should be exported too.

As seen in other tools like https://github.com/simonw/git-history

It's going to shell out to osascript as seen in: - #1

I'm going with that option because https://appscript.sourceforge.io/status.html warns against the other potential methods:

Apple eliminated its Mac Automation department in 2016. The future of AppleScript and its related technologies is unclear. Caveat emptor.

But osascript looks pretty stable to me.

Option that doesn't write to the database at all, it just outputs all the notes to stdout as newline-delimited JSON.

I'm using ChatGPT to figure out enough AppleScript to get at my notes data.

See this issue: - https://github.com/simonw/datasette.io/issues/141

Would be useful for cases where you want to install a whole set of plugins in one go, e.g. when running tutorials in GitHub Codespaces.

This means the Datasette 0.59.1 release has not been published to Docker Hub.

Here's where that failed: https://github.com/simonw/datasette/runs/3991043374?check_suite_focus=true

Preparing to unpack .../libc6_2.32-4_amd64.deb ... debconf: unable to initialize frontend: Dialog debconf: (TERM is not set, so the dialog frontend is not usable.) debconf: falling back to frontend: Readline debconf: unable to initialize frontend: Readline debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.28.1 /usr/local/share/perl/5.28.1 /usr/lib/x86_64-linux-gnu/perl5/5.28 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.28 /usr/share/perl/5.28 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.) debconf: falling back to frontend: Teletype Checking for services that may need to be restarted... Checking init scripts... Unpacking libc6:amd64 (2.32-4) over (2.28-10) ... Setting up libc6:amd64 (2.32-4) ... /usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory dpkg: error processing package libc6:amd64 (--configure): installed libc6:amd64 package post-installation script subprocess returned error exit status 127 Errors were encountered while processing: libc6:amd64 E: Sub-process /usr/bin/dpkg returned an error code (1) The command '/bin/sh -c apt-get update && apt-get -y --no-install-recommends install software-properties-common && add-apt-repository "deb http://httpredir.debian.org/debian sid main" && apt-get update && apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && apt-get remove -y software-properties-common && apt clean && rm -rf /var/lib/apt && rm -rf /var/lib/dpkg/info/*' returned a non-zero code: 100 Same problem when I attempted to publish using the "Push specific Docker tag" workflow: https://github.com/simonw/datasette/runs/3991059912?check_suite_focus=true