issue_comments

Oh that's interesting, I hadn't thought about plugins firing events - just responding to events fired by the rest of the application.

A table.has_count_triggers property.

To detect tables, look at the names of the triggers - {table}{counts_table}_insert and {table}{counts_table}_delete.

I think this should be accompanied by a sqlite-utils reset-counts command.

This is a little tricky. We should assume that the existing values in the _counts table cannot be trusted at all when this method is called - so we should probably clear that table entirely and then re-populate it.

But that means we need to figure out which tables in the database have the counts triggers defined.

So if you instantiate the Database() constructor with use_counts_table=True any access to the .count properties will go through this table - otherwise regular count(*) queries will be executed.

I'm having second thoughts about this being the default behaviour. It's pretty weird. I feel like HUGE databases that need this are rare, so having it on by default doesn't make sense.

Idea: a .execute_count() method that never uses the cache.

Another option: on creation of the Database() object, check to see if the _counts table exists and use that as the default for a use_counts_table property. Also flip that property to True if the user calls .enable_counts() at any time.

Alternative implementation: provided db.should_trust_counts is True, try running the query: sql select count from _counts where [table] = ? If the query fails to return a result OR throws an error because the table doesn't exist, run the count(*) query.

Here's the current .count property: https://github.com/simonw/sqlite-utils/blob/036ec6d32313487527c66dea613a3e7118b97459/sqlite_utils/db.py#L597-L609

It's implemented on Queryable which means it's available on both Table and View - the optimization doesn't make sense for views.

I'm a bit cautious about making that property so much more complex. In order to decide if it should try the _counts table first it needs to know:

• Should it be trusting the counts? I'm thinking a .should_trust_counts property on Database which defaults to True would be good - then advanced users can turn that off if they know the counts should not be trusted.
• Does the _counts table exist?
• Are the triggers defined?

Then it can do the query, and if the query fails it can fall back on the count(*). That's quite a lot of extra activity though.

python def cached_counts(self, tables=None): sql = "select [table], count from {}".format(self._counts_table_name) if tables: sql += " where [table] in ({})".format(", ".join("?" for table in tables)) return {r[0]: r[1] for r in self.execute(sql, tables).fetchall()}

% sqlite-utils insert blah.db blah global_power_plant_database.csv Error: Invalid JSON - use --csv for CSV or --tsv for TSV files

Also welcome in that PR would be a bit of documentation for contributors, see #1167 - but no problem if you leave that out, I'm happy to add it later.

And @benpickles yes I would land that pull request straight away as-is. Thanks!

Oh that's so frustrating! I was worried about that - I spotted a few runs that seemed faster and hoped that it meant that the package was coming out of the ~/.npm cache, but evidently that's not the case.

You've convinced me that Datasette itself should have a package.json - the Dependabot argument is a really good one.

But... I'd really love to figure out a general pattern for using npx scripts in GitHub Actions workflows in a cache-friendly way. I have plenty of other projects that I'd love to run Prettier or Uglify or puppeteer-cli in without adding a package.json to them.

Any ideas? The best I can think of is for the workflow itself to write out a package.json file (using echo '{ ... }' > package.json) as part of the run - that way the cache should work (I think) but I don't get a misleading package.json file sitting in the repo.

I thought about using browser events, but they don't quite match the API that I'm looking to provide. In particular, the great thing about Pluggy is that if you have multiple handlers registered for a specific plugin hook each of those handlers can return a value, and Pluggy will combine those values into a list of replies.

This is great for things like plugin hooks that add extra menu items - each plugin can return a menu item (maybe as a label/URL/click-callback object) and the calling code can then add all of those items to the menu. See https://docs.datasette.io/en/stable/plugin_hooks.html#table-actions-datasette-actor-database-table for a Python example.

I'm on the fence about relying on JavaScript modules. I need to think about browser compatibility for them - but I'm already commited to requiring support for () => {} arrow functions so maybe I'm committed to module support too already?

Should this command include a --fts option for configuring full-text search on one-or-more columns?

I thought about doing that for sqlite-utils insert in https://github.com/simonw/sqlite-utils/issues/202 and decided not to because of the need to include extra options covering the FTS version, porter stemming options and whether or not to create triggers.

But maybe I can set sensible defaults for that with datasette insert ... -f title -f body? Worth thinking about a bit more.

I'm not going to implement this, because it actually needs several additional options that already exist on sqlite-utils enable-fts: --fts4 Use FTS4 --fts5 Use FTS5 --tokenize TEXT Tokenizer to use, e.g. porter --create-triggers Create triggers to update the FTS tables when the parent table changes. I'd rather not add all four of those options to sqlite-utils insert just to support this shortcut.

The long version of this can be --fts, same as in csvs-to-sqlite.

Another option: expand the ForeignKey object to have .columns and .other_columns properties in addition to the existing .column and .other_column properties. These new plural properties would always return a tuple, which would be a one-item tuple for a non-compound-foreign-key.

The question then is what should .column and .other_column return for compound foreign keys?

I'd be inclined to say they should return None - which would trigger errors in code that encounters a compound foreign key for the first time, but those errors would at least be a strong indicator as to what had gone wrong.

We can label .column and .other_column as deprecated and then remove them in sqlite-utils 4.0.

Since this would still be a breaking change in some minor edge-cases I'm thinking maybe 4.0 needs to happen in order to land this feature. I'm not opposed to doing that, I was just hoping it might be avoidable.

One way that this could avoid a breaking change would be to have fk.column and fk.other_column remain as strings for non-compound-foreign-keys, but turn into tuples for a compound foreign key.

This is a bit of an ugly API design, and it could still break existing code that encounters a compound foreign key for the first time - but it would leave code working for the more common case of a non-compound-foreign-key.

Sorry for taking so long to review this!

This approach looks great to me - being able to optionally pass a tuple anywhere the API currently expects a column is smart, and it's consistent with how the pk= parameter works elsewhere.

There's just one problem I can see with this: the way it changes the ForeignKey(...) interface to always return a tuple for .column and .other_column, even if that tuple only contains a single item.

This represents a breaking change to the existing API - any code that expects ForeignKey.column to be a single string (which is any code that has been written against that) will break.

As such, I'd have to bump the major version of sqlite-utils to 4.0 in order to ship this.

Ideally I'd like to make this change in a way that doesn't represent an API compatibility break. I need to think a bit harder about how that might be achieved.

Documented here: https://sqlite-utils.datasette.io/en/latest/python-api.html#quoting-strings-for-use-in-sql

Documented here: https://sqlite-utils.datasette.io/en/latest/python-api.html#introspection

Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#listing-triggers