issues

What is the right way to limit a full text search query to some rows of a table?

For example, I have a table that contains the following columns: title, content, owner (each row represents a document). The owner column is a username. It feels right to store all documents in one table, instead of having one table per owner. In particular because I'd like to full text search all documents, only documents owned by one user and documents owned by a set of users.

I tried to combine .rows_where("owner = ?", "1234") and .search() from the Table class but I don't think that is meant to work. I discovered .search_sql() as a way to generate the FTS SQL statement. By hand I can edit it to add a AND [original].[owner] = :owner to the where clause. This seems to do what I want.

My two questions: 1. is adding a AND ... to the where clause actually the right thing to do or should I be doing something else (my SQL skills are low)? 2. is there a built-in to sqlite-utils way to achieve this?

Right now I am thinking I will make my own version of search_sql() that generates a query that contains an additional owner = :owner for my particular use-case.

Bonus question: is this generally useful/something to add to sqlite-utils or too niche?

This issue is about how best to pass additional options to tools used for publishing datasettes. A concrete example is wanting to pass the --tar flag to the heroku CLI tool. I think there are at least two options for doing this: documentation for each publishing tool to explain how to set flags via env variables (if possible) or building a mechanism that lets users pass additional flags through datasette.

When using datasette publish heroku binder-launches.db --extra-options="--config facet_time_limit_ms:35000 --config sql_time_limit_ms:35000" --name=binderlytics --install=datasette-vega to publish https://binderlytics.herokuapp.com/ the following error happens:

``` › Warning: heroku update available from 7.42.1 to 7.43.0. › Warning: heroku update available from 7.42.1 to 7.43.0. › Warning: heroku update available from 7.42.1 to 7.43.0. Setting WEB_CONCURRENCY and restarting ⬢ binderlytics... done, v13 WEB_CONCURRENCY: 1 › Warning: heroku update available from 7.42.1 to 7.43.0. ▸ Couldn't detect GNU tar. Builds could fail due to decompression errors ▸ See https://devcenter.heroku.com/articles/platform-api-deploying-slugs#create-slug-archive ▸ Please install it, or specify the '--tar' option ▸ Falling back to node's built-in compressor buffer.js:358 throw new ERR_INVALID_OPT_VALUE.RangeError('size', size); ^

RangeError [ERR_INVALID_OPT_VALUE]: The value "3303763968" is invalid for option "size" at Function.alloc (buffer.js:367:3) at new Buffer (buffer.js:281:19) at Readable.<anonymous> (/Users/thead/.local/share/heroku/node_modules/archiver-utils/index.js:39:15) at Readable.emit (events.js:322:22) at endReadableNT (/Users/thead/.local/share/heroku/node_modules/readable-stream/lib/_stream_readable.js:1010:12) at processTicksAndRejections (internal/process/task_queues.js:84:21) { code: 'ERR_INVALID_OPT_VALUE' } ```

After installing GNU tar with brew install gnu-tar and modifying datasette/publish/heroku.py to include the --tar=/path/to/gnu-tar publishing works.

I think the problem occurs once your heroku slug reaches a certain size. At least when I add only a few 100 entries to the datasette then the error does not occcur.

datasette version 0.49.1 OSX 10.14.6 (18G103)

I have a script that imports data into a sqlite DB. When I re-run that script I'd like to remove the existing sqlite DB, instead of adding to it. The pragmatic answer is to add the check and file deletion to my script.

However I thought it would be easy and useful for others to add a recreate=True flag to db = sqlite_utils.Database("binder-launches.db"). After taking a look at the code for it I am not so sure any more. This is because the connection string could be a URL (or "connection string") like "file:///tmp/foo.db". I don't know what the equivalent of os.path.exists() is for a connection string or how to detect that something is a connection string and raise an error "can't use recreate=True and conn_string at the same time".

Does anyone have an idea/suggestion where to start investigating?