The new options added in: - #577 Deserve consideration in the CLI as well.

https://github.com/simonw/sqlite-utils/blob/d2bcdc00c6ecc01a6e8135e775ffdb87572b802b/sqlite_utils/db.py#L1706-L1708

Found myself using this pattern to quickly prototype a schema:

```python import sqlite_utils db = sqlite_utils.Database(memory=True)

print(db["answers_chunks"].create({ "id": int, "content": str, "embedding_type_id": int, "embedding": bytes, "embedding_content_md5": str, "source": str, }, pk="id", transform=True).schema) ```

Using replace=True to drop and then recreate the table would be neat here, and would be consistent with other places that use replace=True.

https://sqlite-utils.datasette.io/en/2.14.1/ isn't showing a link to the stable release right now.

I should also apply the same fix I used for Datasette in: - https://github.com/simonw/datasette/issues/1608

TIL: https://til.simonwillison.net/readthedocs/link-from-latest-to-stable

:books: Documentation preview :books:: https://sqlite-utils--486.org.readthedocs.build/en/486/

I pushed a release for https://github.com/simonw/sqlite-utils/releases/tag/3.25.1 but forgot to include the release notes in docs/changelog.rst

This means https://sqlite-utils.datasette.io/en/stable/changelog.html isn't showing them.

Closes #398

This adds SpatiaLite helpers to the CLI.

```sh

init spatialite when creating a database

sqlite-utils create database.db --enable-wal --init-spatialite

add geometry columns

needs a database, table, geometry column name, type, with optional SRID and not-null

this will throw an error if the table doesn't already exist

sqlite-utils add-geometry-column database.db table-name geometry --srid 4326 --not-null

spatial index an existing table/column

this will throw an error it the table and column don't exist

sqlite-utils create-spatial-index database.db table-name geometry ```

Docs and tests are included.

Not sure why this has only just started failing, but I'm getting this: https://github.com/simonw/sqlite-utils/runs/4488687639

``` E sqlite3.DatabaseError: database disk image is malformed

sqlite_utils/db.py:425: DatabaseError ___ test_rebuild_fts[searchable_fts] ___

fresh_db = <Database \<sqlite3.Connection object at 0x1084ea9d0>> table_to_fix = 'searchable_fts'

@pytest.mark.parametrize("table_to_fix", ["searchable", "searchable_fts"])
def test_rebuild_fts(fresh_db, table_to_fix):
    table = fresh_db["searchable"]
    table.insert(search_records[0])
    table.enable_fts(["text", "country"])
    # Run a search
    rows = list(table.search("tanuki"))
    assert len(rows) == 1
    assert {
        "rowid": 1,
        "text": "tanuki are running tricksters",
        "country": "Japan",
        "not_searchable": "foo",
    }.items() <= rows[0].items()
    # Delete from searchable_fts_data
    fresh_db["searchable_fts_data"].delete_where()
    # This should have broken the index
    with pytest.raises(sqlite3.DatabaseError):
        list(table.search("tanuki"))
    # Running rebuild_fts() should fix it

  fresh_db[table_to_fix].rebuild_fts()

```

For https://github.com/simonw/git-history/issues/12 I found myself wanting to pass extra options to lookup() to set the column order, primary key etc.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416785

• Use sniff to detect CSV or TSV (if :tsv or :csv was not specified) and delimiters

Follow-on from #272

Original title: Easier way to update a row returned from .rows

Here's a surprisingly hard problem I ran into while trying to implement #239 - given a row returned by db[table].rows how can you update that row?

The problem is that the db[table].update(...) method requires a primary key. But if you have a row from the db[table].rows iterator it might not even contain the primary key - provided the table is a rowid table.

Instead, currently, you need to introspect the table and, if rowid is a primary key, explicitly include that in the select= argument to table.rows_where(...) - otherwise it will not be returned.

A utility mechanism to make this easier would be very welcome.

Takes my test down from ten minutes to four seconds. Refs #172.

Is there a way to make .insert_all() work properly when new columns are introduced outside the first 100 records (with or without the alter=True argument)?

I'm using .insert_all() to bulk insert ~3-4k records at a time and it is common for records to need to introduce new columns. However, if new columns are introduced after the first 100 records, sqlite_utils doesn't even raise the OperationalError: table ... has no column named ... exception; it just silently drops the extra data and moves on.

It took me a while to find this little snippet in the documentation for .insert_all() (it's not mentioned under Adding columns automatically on insert/update):

The column types used in the CREATE TABLE statement are automatically derived from the types of data in that first batch of rows. Any additional or missing columns in subsequent batches will be ignored.

I tried changing the batch_size argument to the total number of records, but it seems only to effect the number of rows that are committed at a time, and has no influence on this problem.

Is there a way around this that you would suggest? It seems like it should raise an exception at least.

Got this error: File ".../python3.7/site-packages/sqlite_utils/db.py", line 462, in <dictcomp> for column in self.columns KeyError: 'REAL' .columns_dict uses REVERSE_COLUMN_TYPE_MAPPING: https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L457-L463 REVERSE_COLUMN_TYPE_MAPPING defines FLOAT not REALA https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L68-L74

testing some data from european power information (entsoe.eu), the title of the csv contains square brackets. as I am playing with glitch, sqlite-utils are used for creating the db.

Traceback (most recent call last):

File "/app/.local/bin/sqlite-utils", line 8, in <module>

sys.exit(cli())

File "/app/.local/lib/python3.7/site-packages/click/core.py", line 764, in call

return self.main(*args, **kwargs)

File "/app/.local/lib/python3.7/site-packages/click/core.py", line 717, in main

rv = self.invoke(ctx)

File "/app/.local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke

return _process_result(sub_ctx.command.invoke(sub_ctx))

File "/app/.local/lib/python3.7/site-packages/click/core.py", line 956, in invoke

return ctx.invoke(self.callback, **ctx.params)

File "/app/.local/lib/python3.7/site-packages/click/core.py", line 555, in invoke

return callback(*args, **kwargs)

File "/app/.local/lib/python3.7/site-packages/sqlite_utils/cli.py", line 434, in insert

default=default,

File "/app/.local/lib/python3.7/site-packages/sqlite_utils/cli.py", line 384, in insert_upsert_implementation

docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs

File "/app/.local/lib/python3.7/site-packages/sqlite_utils/db.py", line 997, in insert_all

extracts=extracts,

File "/app/.local/lib/python3.7/site-packages/sqlite_utils/db.py", line 618, in create

extracts=extracts,

File "/app/.local/lib/python3.7/site-packages/sqlite_utils/db.py", line 310, in create_table

self.conn.execute(sql)

sqlite3.OperationalError: unrecognized token: "]"

entsoe_2016.csv

renamed to txt for uploading compatibility

entsoe_2016.txt

code is remixed directly from your https://glitch.com/edit/#!/datasette-csvs repo

The foreign_keys= argument to table.insert_all() and friends can be used to specify foreign key relationships that should be created.

It is not yet documented. It also requires you to specify the SQLite type of each column, even though this can be detected by introspecting the referenced table:

cols = [c for c in self.db[other_table].columns if c.name == other_column]
cols[0].type

Relates to #2