issue_comments

Thought: maybe there should be a .reset_counts() method too, for if the table gets out of date with the triggers.

One way that could happen is if a table is dropped and recreated - the counts in the _counts table would likely no longer match the number of rows in that table.

Idea: a db.cached_counts() method that returns a dictionary of data from the _counts table. Call it with a list of tables to get back the counts for just those tables.

https://github.com/simonw/sqlite-utils/blob/9a5c92b63e7917c93cc502478493c51c781b2ecc/sqlite_utils/db.py#L231-L239

I got this error while prototyping this:

too many levels of trigger recursion

It looks like that's because SQLite doesn't like triggers on a table that themselves then update that table - so I'm going to exclude the _counts table from this mechanism.

I'm going to skip virtual tables, which I can identify using this property: https://github.com/simonw/sqlite-utils/blob/1cad7fad3e7a5b734088f5cc545b69a055e636da/sqlite_utils/db.py#L720-L726

Idea: version the metadata scheme. If the table is called _metadata_v1 it gives me a clear path to designing a new scheme in the future.

Here's a prototype: python with db.conn: db.conn.executescript(""" CREATE TABLE IF NOT EXISTS [_counts] ([table] TEXT PRIMARY KEY, [count] INTEGER DEFAULT 0); CREATE TRIGGER IF NOT EXISTS [Street_Tree_List_counts_ai] AFTER INSERT ON [Street_Tree_List] BEGIN INSERT OR REPLACE INTO _counts VALUES ('Street_Tree_List', COALESCE( (SELECT count FROM _counts WHERE [table]='Street_Tree_List'), 0) + 1); END; CREATE TRIGGER IF NOT EXISTS [Street_Tree_List_counts_ad] AFTER DELETE ON [Street_Tree_List] BEGIN INSERT OR REPLACE INTO _counts VALUES ('Street_Tree_List', COALESCE( (SELECT count FROM _counts WHERE [table]='Street_Tree_List'), 0) - 1); END; INSERT OR REPLACE INTO _counts VALUES ('Street_Tree_List', (select count(*) from [Street_Tree_List])); """)

It looks like https://github.com/ofajardo/pyreadr is a good library for this.

I won't add this to sqlite-utils because it's quite a bulky dependency for a relatively small feature.

Normally I'd write a rdata-to-sqlite tool similar to https://pypi.org/project/dbf-to-sqlite/ - but I'm actually working on a new plugin hook for Datasette that might be an even better fit for this. The idea is to allow Datasette plugins to define input formats - such as RData - which would then result in being able to import them on the command-line with datasette insert my.db file.rdata or by uploading a file through the Datasette web interface.

That work is happening over here: https://github.com/simonw/datasette/issues/1160 - I'll close this issue in favour of a sometime-in-the-future datasette-import-rdata plugin.