• https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776

Make table.transform() preserve STRICT mode.

:books: Documentation preview :books:: https://sqlite-utils--604.org.readthedocs.build/en/604/

Just like the Datasette prepare_connection hook, this PR adds a similar hook for the sqlite-utils plugin system.

The sole argument is conn, since I don't believe a database or datasette argument would be relevant here.

I want to do this so I can release sqlite-utils plugins for my SQLite extensions, similar to the Datasette plugins I've release for them.

An example plugin: https://gist.github.com/asg017/d7cdf0d56e2be87efda28cebee27fa3c

```bash $ sqlite-utils install https://gist.github.com/asg017/d7cdf0d56e2be87efda28cebee27fa3c/archive/5f5ad549a40860787629c69ca120a08c32519e99.zip

$ sqlite-utils memory 'select hello("alex") as response' [{"response": "Hello, alex!"}] ``` Refs: - #574

:books: Documentation preview :books:: https://sqlite-utils--573.org.readthedocs.build/en/573/

Summary

When using the API, repeated calls to Table.convert() do not work correctly since all conversions quietly use the callable (function, lambda) from the first call to convert() only. Subsequent invocations with different callables use the callable from the first invocation only.

Example

```python from sqlite_utils import Database

db = Database(memory=True) table = db['table'] col = 'x' table.insert_all([{col: 1}]) print(table.get(1))

table.convert(col, lambda x: x*2) print(table.get(1))

def zeroize(x): return 0

zeroize = lambda x: 0

zeroize.name = 'zeroize'

table.convert(col, zeroize) print(table.get(1)) ```

Output: {'x': 1} {'x': 2} {'x': 4} Expected: {'x': 1} {'x': 2} {'x': 0}

Explanation

This is some relevant documentation.

• Table.convert() takes a Callable to perform data conversion on a column
• The Callable is passed to Database.register_function()
• Database.register_function() uses the callable's __name__ attribute for registration
• (Aside: all lambdas have a __name__ of <lambda>: I thought this was the problem, and it was close, but not quite)
• However convert() first wraps the callable by local function convert_value()
• Consequently register_function() sees name convert_value for all invocations from convert()
• register_function() silently ignores registrations using the same name, retaining only the first such registration

There's a mismatch between the comments and the code: https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L404

but actually the existing function is returned/used instead (as the "registering custom sql functions" doc I linked above says too). Seems like this can be rectified to match the comment?

Suggested fix

I think there are four things: 1. The call to register_function() from convert()should have an explicit name= parameter (to continue using convert_value() and the progress bar). 2. For functions, this name can be the real function name. (I understand the sqlite api needs a name, and it's nice if those are recognizable names where possible). For lambdas would 'lambda-{uuid}' or similar be acceptable? 3. register_function() really should throw an error on repeated attempts to register a duplicate (function, arity)-pair. 4. A test? I haven't looked at the test framework here but seems this should be testable.

See also

• 458

https://readthedocs.org/projects/sqlite-utils/builds/20512693/ ``` Running Sphinx v6.2.1

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py", line 442, in load_extension mod = import_module(extname) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 783, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/builders/linkcheck.py", line 20, in <module> from requests import Response File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/requests/init.py", line 43, in <module> import urllib3 File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/urllib3/init.py", line 38, in <module> raise ImportError( ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main app = Sphinx(args.sourcedir, args.confdir, args.outputdir, File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py", line 225, in init self.setup_extension(extension) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py", line 404, in setup_extension self.registry.load_extension(self, extname) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py", line 445, in load_extension raise ExtensionError(__('Could not import extension %s') % extname, sphinx.errors.ExtensionError: Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168)

Extension error: Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168) ```

I wanted to output newline-separated output of the first column of every row in the results - like --row but for more than one line.

Pylance is generally pretty good at figuring out stuff but sqlite-utils has some quirks which make type hinting kinda useless. Maybe you don't care but I thought I would bring it to your attention.

For example:

db["subs"].insert_all(subs, pk="index")

Cannot access member "insert_all" for type "View" Member "insert_all" is unknown

insert_all and all the other methods show up as a type issues because the program can't know whether something is a View or a Table. Fair enough. But that basically throws all type checking out the window.

pk="index" also shows up as a type issue:

Argument of type "Literal['index']" cannot be assigned to parameter "pk" of type "Default" in function "insert_all" "Literal['index']" is incompatible with "Default"

I think this is because DEFAULT is an empty class?

maybe a few small changes could be made to make the library more type-friendly

The interim solution is of course to turn off type hints completely for the line db["subs"].insert_all(subs, pk="index") # type: ignore

This also passes in the extension path when specified in GIS methods. Wherever we know an extension path, we use db.init_spatialite(find_spatialite() or load_extension).

:books: Documentation preview :books:: https://sqlite-utils--531.org.readthedocs.build/en/531/

https://hachyderm.io/@briandorsey/109468185742876820

I'm fiddling with piping json to insert -ignore I'd love to see the count of records inserted & ignored, but didn't see a way to do that in the help/docs.

Example: xh "https://hachyderm.io/api/v1/timelines/tag/rust?max_id=109443380308326328" | sqlite-utils insert aoc.db aoc - --pk=id --ignore

Reported here: https://github.com/dogsheep/healthkit-to-sqlite/issues/9

It looks like there's a default limit of 999 variables - we need to be smart about that, maybe dynamically lower the batch size based on the number of columns.

Failure here: https://github.com/simonw/sqlite-utils/actions/runs/3275786702/jobs/5391063221

I figured out the problem:

`diff diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index b88e38a..82b4b6c 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -112,11 +112,15 @@ See :ref:`cli_query`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, - simple, textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, + latex, latex_booktabs, latex_longtable, latex_raw, + mediawiki, mixed_grid, mixed_outline, moinmoin, + orgtbl, outline, pipe, plain, presto, pretty, + psql, rounded_grid, rounded_outline, rst, simple, + simple_grid, simple_outline, textile, tsv, + unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row @@ -176,11 +180,15 @@ See :ref:`cli_memory`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, - simple, textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, + latex, latex_booktabs, latex_longtable, latex_raw, + mediawiki, mixed_grid, mixed_outline, moinmoin, + orgtbl, outline, pipe, plain, presto, pretty, + psql, rounded_grid, rounded_outline, rst, simple, + simple_grid, simple_outline, textile, tsv, + unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row @@ -401,11 +409,14 @@ See :ref:`cli_search`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, simple, - textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, latex, + latex_booktabs, latex_longtable, latex_raw, mediawiki, + mixed_grid, mixed_outline, moinmoin, orgtbl, outline, + pipe, plain, presto, pretty, psql, rounded_grid, + rounded_outline, rst, simple, simple_grid, + simple_outline, textile, tsv, unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings --load-extension TEXT Path to SQLite extension, with optional :entrypoint @@ -651,11 +662,14 @@ See :ref:`cli_tables`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, simple, - textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, latex, + latex_booktabs, latex_longtable, latex_raw, mediawiki, + mixed_grid, mixed_outline, moinmoin, orgtbl, outline, + pipe, plain, presto, pretty, psql, rounded_grid, + rounded_outline, rst, simple, simple_grid, + simple_outline, textile, tsv, unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings --columns Include list of columns for each table @@ -689,11 +703,14 @@ See :ref:`cli_views`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, simple, - textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, latex, + latex_booktabs, latex_longtable, latex_raw, mediawiki, + mixed_grid, mixed_outline, moinmoin, orgtbl, outline, + pipe, plain, presto, pretty, psql, rounded_grid, + rounded_outline, rst, simple, simple_grid, + simple_outline, textile, tsv, unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings --columns Include list of columns for each view @@ -732,11 +749,15 @@ See :ref:`cli_rows`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, - simple, textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, + latex, latex_booktabs, latex_longtable, latex_raw, + mediawiki, mixed_grid, mixed_outline, moinmoin, + orgtbl, outline, pipe, plain, presto, pretty, + psql, rounded_grid, rounded_outline, rst, simple, + simple_grid, simple_outline, textile, tsv, + unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings --load-extension TEXT Path to SQLite extension, with optional @@ -768,11 +789,14 @@ See :ref:`cli_triggers`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, simple, - textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, latex, + latex_booktabs, latex_longtable, latex_raw, mediawiki, + mixed_grid, mixed_outline, moinmoin, orgtbl, outline, + pipe, plain, presto, pretty, psql, rounded_grid, + rounded_outline, rst, simple, simple_grid, + simple_outline, textile, tsv, unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings --load-extension TEXT Path to SQLite extension, with optional :entrypoint @@ -804,11 +828,14 @@ See :ref:`cli_indexes`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, simple, - textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, latex, + latex_booktabs, latex_longtable, latex_raw, mediawiki, + mixed_grid, mixed_outline, moinmoin, orgtbl, outline, + pipe, plain, presto, pretty, psql, rounded_grid, + rounded_outline, rst, simple, simple_grid, + simple_outline, textile, tsv, unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings --load-extension TEXT Path to SQLite extension, with optional :entrypoint diff --git a/docs/cli.rst b/docs/cli.rst index 8bc4176..1d67e88 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -187,10 +187,15 @@ Available--fmtoptions are: cog.out("\n" + "\n".join('-{}``'.format(t) for t in tabulate.tabulate_formats) + "\n\n") .. ]]]

+- asciidoc +- double_grid +- double_outline - fancy_grid - fancy_outline - github - grid +- heavy_grid +- heavy_outline - html - jira - latex @@ -198,15 +203,22 @@ Available --fmt options are: - latex_longtable - latex_raw - mediawiki +- mixed_grid +- mixed_outline - moinmoin - orgtbl +- outline - pipe - plain - presto - pretty - psql +- rounded_grid +- rounded_outline - rst - simple +- simple_grid +- simple_outline - textile - tsv - unsafehtml ```

The --flatten implementation isn't currently available to Python code - people have to roll their own implementation. Feedback from a conversation at DjangoCon.

I was looking into how to parse application log files that contain multiline text (e.g. Java stack traces) into sqlite. I can see that at the moment --lines helps, but falls short when processing multi-line texts.

I wonder if this functionality would be useful for sqlite-utils. A similar approach to Elastic logstash/filebeat can be adopted: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html

Potential changes:

• add a --multiline option
• additional properties for
• multiline-pattern (regex expression)
• multiline-negate: true/false
• multiline-what: previous or next

Or if this is achievable in a different way, please share. Thanks!

I haven't tested this yet but wanted to get a heads-up whether this kind of change would be useful or if I should just duplicate the function and tweak it within my code

:books: Documentation preview :books:: https://sqlite-utils--480.org.readthedocs.build/en/480/

Work in progress, partly to test the ReadTheDocs preview link action.

Refs: - #426

:books: Documentation preview :books:: https://readthedocs-preview--460.org.readthedocs.build/en/460/

is there a duplicate table functionality? Otherwise, I'd be happy to submit a PR.

In sqlite3 it would look like:

```python import sqlite3 as sl

con = sl.connect('prompt-tune.db')

def db_duplicate_table(table_name, table_name_new, con=con): # Duplicates table table_name to a new table table_name_new. try: cur = con.cursor() cur.execute(f"""CREATE TABLE {table_name_new} AS SELECT * FROM {table_name}""") except Exception as e: print(e) finally: cur.close()

db_duplicate_table('orig_table', 'new_table') ```

What is the right way to limit a full text search query to some rows of a table?

For example, I have a table that contains the following columns: title, content, owner (each row represents a document). The owner column is a username. It feels right to store all documents in one table, instead of having one table per owner. In particular because I'd like to full text search all documents, only documents owned by one user and documents owned by a set of users.

I tried to combine .rows_where("owner = ?", "1234") and .search() from the Table class but I don't think that is meant to work. I discovered .search_sql() as a way to generate the FTS SQL statement. By hand I can edit it to add a AND [original].[owner] = :owner to the where clause. This seems to do what I want.

My two questions: 1. is adding a AND ... to the where clause actually the right thing to do or should I be doing something else (my SQL skills are low)? 2. is there a built-in to sqlite-utils way to achieve this?

Right now I am thinking I will make my own version of search_sql() that generates a query that contains an additional owner = :owner for my particular use-case.

Bonus question: is this generally useful/something to add to sqlite-utils or too niche?

Hi

I note that you define _hash() to create a hash_id from non-id column values in a table here.

It would be useful to be able to call a complementary function to generate a corresponding _id from a subset of specified columns when adding items to another table, eg to support the creation of foreign keys.

Or is there a better pattern for doing that?

Original title: Add option for adding a new, serial, primary key

sometimes we have tables that don't have primary keys, but ought to have them. we can use rowid for that, but it would often be nicer to have an explicit primary key. using the current value of rowid would be fine.

Is there a reason I've not added --extract as an option for sqlite-utils insert next? There's a extracts= option for the various table.insert() etc methods - last line in this code block:

https://github.com/simonw/sqlite-utils/blob/213a0ff177f23a35f3b235386366ff132eb879f1/sqlite_utils/db.py#L2483-L2495

Here's the traceback if your --convert function doesn't return a dict right now: ``` % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all

Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils", line 33, in <module> sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1137, in call return self.main(args, kwargs) File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 763, in invoke return __callback(args, **kwargs) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py", line 949, in insert insert_upsert_implementation( File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py", line 834, in insert_upsert_implementation db[table].insert_all( File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 2602, in insert_all first_record = next(records) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 3044, in fix_square_braces for record in records: File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py", line 831, in <genexpr> docs = (decode_base64_values(doc) for doc in docs) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py", line 86, in decode_base64_values to_fix = [ File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py", line 89, in <listcomp> if isinstance(doc[k], dict) TypeError: string indices must be integers ``` It would be nicer if that returned a more useful error message.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/361#issuecomment-1006295276

Could add support for --batch-size as seen in insert/upsert too - causing it to break the list up into batches and commit for each one.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876055

Can be maintained using cog from #373. Similar in purpose to the API reference page, but this is for the CLI.

Not sure why I didn't implement this.

Right now you use conversions= like this:

python db["example"].insert({ "name": "The Bigfoot Discovery Museum" }, conversions={"name": "upper(?)"}) How about if you could optionally provide a Python function (or a lambda) like this? python db["example"].insert({ "name": "The Bigfoot Discovery Museum" }, conversions={"name": lambda s: s.upper()}) This would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end.

I think the fix for this is to change the rules about what code is accepted in both the - mode and the literal code string mode: you can pass in a Python expression, OR a fragment that gets turned into a function, OR code that implements its own def convert(value) function. So this would work too: sh sqlite-utils convert my.db mytable col1 ' def convert(value): return value.upper() ' Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381679

For the commits table I feel like I want a version of table.lookup() that can be passed additional columns to populate only if the record does not exist yet.

Originally posted by @simonw in https://github.com/simonw/git-history/issues/12#issuecomment-967455017

~ % gsutil cat gs://ossf-criticality-score/python_top_200.csv | sqlite-utils insert /tmp/crit.db crit - ... File "/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) A nicer error message here would be one that says the JSON is invalid but suggests that maybe you could try --csv.

When I use multiple files with the same name, e.g. in sqlite-utils memory a/bug.csv b/bug.csv, sqlite-utils creates invalid views. Traceback (most recent call last): File "/home/karl/.local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 763, in invoke return __callback(*args, **kwargs) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/sqlite_utils/cli.py", line 1299, in memory db[csv_table].transform(types=tracker.types) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/sqlite_utils/db.py", line 1287, in transform self.db.execute(sql) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/sqlite_utils/db.py", line 421, in execute return self.conn.execute(sql) sqlite3.OperationalError: error in view t1: no such table: main.bug

This can be reproduced with ```sh

!/bin/bash

mkdir foo mkdir bar echo -e 'col1,col2\nval1,val2' > foo/bug.csv echo -e 'col3,col4\nval3,val4' > bar/bug.csv sqlite-utils memory */bug.csv 'SELECT 1' ```

Ideally, the tables would get unique names by including the next path segment until the names are unique. But just making the numbered t* aliases work would be good enough.

This problem can of course be worked around by renaming the files, but it would be nice if this case was handled more gracefully.

Thanks a lot for this great tool!

Thanks for the excellent library, it's very nice to use!

I've been building some in memory search functionality for a data annotation tool i'm making, and I got tripped up a little bit with escaping the full text search queries. First I tried using db.quote(q), which doesn't work, because sqlite FTS has it's own (separate) query syntax. You can see this happening here also:

http://search-24ways.herokuapp.com/24ways-f8f455f/articles?_search=acces%2A

I got around this by aggressively escaping quotes inside the query string like this:

```python quoted = q.replace('"', '""') quoted = f'"{quoted}"' print(quoted) results = db["data"].search(quoted, columns=["id"]) return [x["id"] for x in results]

```

This works in the sense it doesn't crash, but it also removes access to the search query syntax. Given the well specified definition, it might be possible for sqlite-utils to provide a db.quote_query(q) which would intelligently escape a query whilst leaving the syntax intact. This would be very nice!

Using https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html

I'm not a big fan of this kind of documentation because it so often comes in place of narrative documentation - but the library has great narrative documentation now, so the reference documentation can link to it in places.

This will also encourage me to add good docstrings everywhere, useful for IDEs and suchlike.

Hi,

fantastic framework for working with Sqlite3 databases!!!

I tried to update spezific rows in a table and used

for row in db[tablename]: newValue = row["counter"] * row["prize"]
row.update({"Fieldname": newValue}) print(row)

This updates the value in the printet row, but not in the database. So I switched to

db[tablename].update(id, {"Filedname": newValue})

This works fine. But row.update would be nicer, because no need for the id (its that row), no need for the tablename and the db (all defined in the for row ... loop).

Thx

I've touched on this before in #179 - but now that I've added sqlite-utils memory this is much more important - because unlike with sqlite-utils insert the in-memory command doesn't give you the opportunity to fix any types you imported from CSV, so queries like select * from stdin where age > 3 are never going to work correctly against these temporary in-memory tables.

Teaching sqlite-utils insert to detect types for columns in a CSV file would be a backwards-compatibility breaking change. Teaching sqlite-utils memory that trick would not be, since it hasn't been included in a release yet.

It's a little inconsistent, but I'm going to have sqlite-utils memory default to detecting types while sqlite-utils insert does not. In each case this can be controlled by a new command-line option:

cat file.csv | sqlite-utils memory - --no-detect-types

To opt-in for sqlite-utils insert:

cat file.csv | sqlite-utils insert blah.db blah - --detect-types

I'll have short options for these too: -n for --no-detect-types and -d for --detect-types.

Idea from https://github.com/simonw/datasette-edit-tables/issues/13 - provide Python utility methods and accompanying CLI options for detecting the likely types of TEXT columns.

So if you have a text column that actually contained exclusively integer string values, it can let you know and let you run transform against it.

It would be great if the column type could be set to JSON. That would not be different from handling a regular string. It would be something like repr(value) and it would work with both JSON and CSV inputs, no matter if value is a real list or just a string representing a list.

table.schema returns the schema for a table. db.schema should return the schema for the whole databes.

Can do this using select sql from sqlite_master where sql is not null:

https://latest.datasette.io/fixtures?sql=select+sql+from+sqlite_master+where+sql+is+not+null

Feel free to close this issue - I mostly added it for reference for future folks that run into this :)

I have a CSV file with one column that has very long strings. When i try to import this file via the insert command I get the following error: ``` sqlite-utils insert database.db table_name file_with_large_column.csv

Traceback (most recent call last): File "/usr/local/bin/sqlite-utils", line 10, in <module> sys.exit(cli()) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke return callback(args, kwargs) File "/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py", line 774, in insert default=default, File "/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py", line 705, in insert_upsert_implementation docs, pk=pk, batch_size=batch_size, alter=alter, extra_kwargs File "/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py", line 1852, in insert_all first_record = next(records) File "/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py", line 703, in <genexpr> docs = (decode_base64_values(doc) for doc in docs) File "/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py", line 681, in <genexpr> docs = (dict(zip(headers, row)) for row in reader) _csv.Error: field larger than field limit (131072) Built with the docker image `datasetteproject/datasette:0.54` with the following versions:

sqlite-utils --version

sqlite-utils, version 3.4.1

datasette --version

datasette, version 0.54 ``` It appears this is a known issue reading in csv files in python and doesn't look to be modifiable through system / env vars (i may be very wrong on this).

Noting that using sqlite3 import command work without error (not using the python csv reader) sqlite3 database.db sqlite> .mode csv sqlite> .import file_with_large_column.csv table_name Sadly I couldn't see an easy way around this while using the cli as it appears this value needs to be changed in python code. FWIW I've switched to using https://datasette.io/tools/csvs-to-sqlite for importing csv data and it's working well.

Finally, I'm loving https://datasette.io/ thank you very much for an amazing tool and data ecosytem 🙇‍♀️

Thought: maybe there should be a .reset_counts() method too, for if the table gets out of date with the triggers.

One way that could happen is if a table is dropped and recreated - the counts in the _counts table would likely no longer match the number of rows in that table.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753545757

A command which analyzes a table (potentially taking quite a while if the table is large) and outputs information for each column - things like:

• How many unique values does this column have?
• How many null rows?
• How many blank rows? (defined as empty string)
• What are the 10 most common values?
• What are the 10 least common values?

The command can output this information to the terminal, but it should also provide an option for writing the information to a database table so it can be explored later.

The .transform() method in #114 warrants an equivalent CLI tool.

When using this library I often find myself wanting to "add this foreign key, but only if it doesn't exist yet". The ignore=True parameter is increasingly being used for this else where in the library (e.g. in create_view()).

I got this error: % sqlite-utils calands.db 'create table superunits_with_maps_view_concrete as select * from superunits_with_maps_view' Traceback (most recent call last): ... cursor = db.conn.execute(sql, dict(param)) sqlite3.OperationalError: no such function: AsGeoJSON A --load-extension=/usr/local/lib/mod_spatialite.dylib option (imitating the same option for Datasette) would help.

You get an error if you try to run a query that returns data from a BLOB.

In some cases, I want to iterate rows in a table with ORDER BY clause. It would be nice to have a rows_order_by function similar to rows_where. In a more general case, rows_filter function might be added to allow more customized filtering to iterate rows.

I have a script that imports data into a sqlite DB. When I re-run that script I'd like to remove the existing sqlite DB, instead of adding to it. The pragmatic answer is to add the check and file deletion to my script.

However I thought it would be easy and useful for others to add a recreate=True flag to db = sqlite_utils.Database("binder-launches.db"). After taking a look at the code for it I am not so sure any more. This is because the connection string could be a URL (or "connection string") like "file:///tmp/foo.db". I don't know what the equivalent of os.path.exists() is for a connection string or how to detect that something is a connection string and raise an error "can't use recreate=True and conn_string at the same time".

Does anyone have an idea/suggestion where to start investigating?

I used it in geojson-to-sqlite here: https://github.com/simonw/geojson-to-sqlite/blob/f10e44264712dd59ae7dfa2e6fd5a904b682fb33/geojson_to_sqlite/utils.py#L45-L50

It would make more sense for this method to live on the Database rather than the Table - or even to exist as a separate utility method entirely.

Then it should be documented.

This adds the option for a user to set up triggers in the database to keep their FTS table in sync with the parent table.

Ref: https://sqlite.org/fts5.html#external_content_and_contentless_tables

I would prefer to make the creation of triggers the default behavior, but that will break existing usage where people have been calling populate_fts after inserting new rows.

I am happy to make changes to the PR as you see fit.

The foreign_keys= argument currently takes a list of triples: python db["usages"].insert_all( usages_to_insert, foreign_keys=( ("line_id", "lines", "id"), ("definition_id", "definitions", "id"), ), ) As of #16 we have a mechanism for detecting the primary key column (the third item in this triple) - we should use that here too, so foreign keys can be optionally defined as a list of pairs.

Related to #2