Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#creating-an-empty-database

I'm not going to implement --page-size unless someone specifically requests it - I don't like having features that I've never needed to use myself.

The previous code for highlighting errors in syntax (which was already a bit confused thanks to the added return, see https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684 - isn't compatible with this approach at all. I'm going to ditch it and just show a generic Error: Could not compile code message.

This seems to work: ```python def _compile_code(code, imports, variable="value"): locals = {} globals = {"r": recipes, "recipes": recipes} # If user defined a convert() function, return that try: exec(code, globals, locals) return locals["convert"] except (AttributeError, SyntaxError, NameError, KeyError, TypeError): pass

# Try compiling their code as a function instead
body_variants = [code]
# If single line and no 'return', try adding the return
if "\n" not in code and not code.strip().startswith("return "):
    body_variants.insert(0, "return {}".format(code))

for variant in body_variants:
    new_code = ["def fn({}):".format(variable)]
    for line in variant.split("\n"):
        new_code.append("    {}".format(line))
    try:
        code_o = compile("\n".join(new_code), "<string>", "exec")
        break
    except SyntaxError:
        # Try another variant, e.g. for 'return row["column"] = 1'
        continue

for import_ in imports:
    globals[import_.split(".")[0]] = __import__(import_)
exec(code_o, globals, locals)
return locals["fn"]

```

Here's the code in question: https://github.com/simonw/sqlite-utils/blob/b8c134059e89f0fa040b84fb7d0bda25b9a52759/sqlite_utils/utils.py#L288-L299

Tried this test:

python result = CliRunner().invoke( cli.cli, [ "insert", db_path, "rows", "-", "--convert", 'row["is_chicken"] = True', ], input='{"name": "Azi"}', ) And got this error:

E + where 1 = <Result SyntaxError('invalid syntax', ('<string>', 2, 30, ' return row["is_chicken"] = True\n'))>.exit_code

The code snippet compilation isn't currently compatible with this.

In addition to a unit test I manually tested all of the above, e.g. % sqlite-utils indexes global-power-plants.db sqlite_master --fmt rst ======= ============ ======= ===== ====== ====== ====== ===== table index_name seqno cid name desc coll key ======= ============ ======= ===== ====== ====== ====== ===== ======= ============ ======= ===== ====== ====== ====== =====

Commands that support --fmt (via the @output_options decorator) are:

• tables
• views
• query
• memory
• search
• rows
• triggers
• indexes

Better: python if fmt: table = True

Updated TIL: https://til.simonwillison.net/python/cog-to-update-help-in-readme#user-content-cog-for-restructuredtext

Updated docs: https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output

Found an example of using cog in a rST file here: https://github.com/nedbat/coveragepy/blob/f3238eea7e403d13a217b30579b1a1c2cbff62e3/doc/dbschema.rst#L21 ``` .. [[[cog from coverage.sqldata import SCHEMA_VERSION print(".. code::") print() print(f" SCHEMA_VERSION = {SCHEMA_VERSION}") print() .. ]]] .. code::

SCHEMA_VERSION = 7

.. [[[end]]] ```

cat blah.csv | sqlite-utils bulk blah.db - \
   "insert into blah (:foo, :bar)" --csv

i don’t want to be such a partisan for analyze, but the query planner deciding not to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice.

<s>the original poster in that stack overflow doesn’t say there’s a performance regression </s>

@fgregg my thinking was backwards compatibility. I don't know what people do to their builds, I just wanted a smaller image for my use case.

@simonw any chance to take a look at this? If there is no interest, feel free to close the PR

Need to change if table: to if table or fmt: in a few places.

Pathlib says the stem of that would be dogs.and.cats.jpg - best stick with that for consistency. https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.suffix

It calls the last bit suffix - maybe I should use that instead of ext.

Also need to update relevant docs for that example.

Might be a case of modifying this line: https://github.com/simonw/sqlite-utils/blob/e0c476bc380744680c8b7675c24fb0e9f5ec6dcd/sqlite_utils/cli.py#L828

To: python docs = (fn(doc) or doc for doc in docs)

Calling p.stdin.close() and then p.wait() terminates the subprocess.

This is strange. The following: ```pycon

import subprocess p = subprocess.Popen(["sqlite-utils", "insert", "/tmp/stream.db", "stream", "-", "--nl"], stdin=subprocess.PIPE) p.stdin.write(b'\n'.join(b'{"id": %s}' % str(i).encode("utf-8") for i in range(1000))) 11889

At this point /tmp/stream.db is still 0 bytes - but if I then run this:

p.stdin.close()

/tmp/stream.db is now 20K and contains the written data

`` No wait, mystery solved - I can addp.stdin.flush()instead ofp.stdin.close()` and the file suddenly jumps up in size.

I think the query that will help solve this is:

explain query plan select * from ny_times_us_counties where state = 1 and county = 2

In this case, the query planner needs to decide if it should use the index for the state column or the index for the county column. That's where the statistics come into play. In particular:

| tbl | idx | stat | |----------------------|---------------------------------|---------------| | ny_times_us_counties | idx_ny_times_us_counties_date | 2092871 2915 | | ny_times_us_counties | idx_ny_times_us_counties_fips | 2092871 651 | | ny_times_us_counties | idx_ny_times_us_counties_county | 2092871 1085 | | ny_times_us_counties | idx_ny_times_us_counties_state | 2092871 37373 |

Those numbers are explained by this comment in the SQLite C code: https://github.com/sqlite/sqlite/blob/5622c7f97106314719740098cf0854e7eaa81802/src/analyze.c#L41-L55

** There is normally one row per index, with the index identified by the ** name in the idx column. The tbl column is the name of the table to ** which the index belongs. In each such row, the stat column will be ** a string consisting of a list of integers. The first integer in this ** list is the number of rows in the index. (This is the same as the ** number of rows in the table, except for partial indices.) The second ** integer is the average number of rows in the index that have the same ** value in the first column of the index. So that table is telling us that using a value in the county column will filter down to an average of 1,085 rows, whereas filtering on the state column will filter down to an average of 37,373 - so clearly the county index is the better index to use here!

Just one catch: against both mycovid.db and my covid-analyzed.db databases the county index is picked for both of them - so SQLite is somehow guessing that county is a better index even though it doesn't have statistics for that.

Found one report on Stack Overflow from 9 years ago of someone seeing broken performance after running ANALYZE, hard to say that's a trend and not a single weird edge-case though! https://stackoverflow.com/q/12947214/6083

This is probably too fancy. I think maybe the way to do this is with select * from [global-power-plants] where "country_long" = 'United Kingdom' - then mess around with stats to see if I can get it to use the index or not based on them.

Here's the explain for that: https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+*+from+[global-power-plants]+where+%22country_long%22+%3D+%27United+Kingdom%27

sql EXPLAIN QUERY PLAN select country_long, count(*) from [global-power-plants] group by country_long https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+country_long%2C+count%28*%29+from+%5Bglobal-power-plants%5D+group+by+country_long

SCAN TABLE global-power-plants USING COVERING INDEX "global-power-plants_country_long"

Relevant code: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/init.py#L163-L170

https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/init.py#L195-L204

I think this is the fix: python re.compile(r"^explain\s+query\s+plan\s+select\b"),

There are some clues as to what effect ANALYZE has in https://www.sqlite.org/optoverview.html

Some quotes:

SQLite might use a skip-scan on an index if it knows that the first one or more columns contain many duplication values. If there are too few duplicates in the left-most columns of the index, then it would be faster to simply step ahead to the next value, and thus do a full table scan, than to do a binary search on an index to locate the next left-column value.

The only way that SQLite can know that there are many duplicates in the left-most columns of an index is if the ANALYZE command has been run on the database. Without the results of ANALYZE, SQLite has to guess at the "shape" of the data in the table, and the default guess is that there are an average of 10 duplicates for every value in the left-most column of the index. Skip-scan only becomes profitable (it only gets to be faster than a full table scan) when the number of duplicates is about 18 or more. Hence, a skip-scan is never used on a database that has not been analyzed.

And

Join reordering is automatic and usually works well enough that programmers do not have to think about it, especially if ANALYZE has been used to gather statistics about the available indexes, though occasionally some hints from the programmer are needed.

And

The various sqlite_statN tables contain information on how selective the various indexes are. For example, the sqlite_stat1 table might indicate that an equality constraint on column x reduces the search space to 10 rows on average, whereas an equality constraint on column y reduces the search space to 3 rows on average. In that case, SQLite would prefer to use index ex2i2 since that index is more selective.

Didn't manage to spot a meaningful difference with that database either: analyze % python3 -m timeit '__import__("sqlite3").connect("covid.db").execute("select fips, count(*) from [ny_times_us_counties] group by fips").fetchall()' 2 loops, best of 5: 101 msec per loop analyze % python3 -m timeit '__import__("sqlite3").connect("covid-analyzed.db").execute("select fips, count(*) from [ny_times_us_counties] group by fips").fetchall()' 2 loops, best of 5: 103 msec per loop Maybe select fips, count(*) from [ny_times_us_counties] group by fips isn't a good query for testing this?

Basically no difference using this very basic benchmark: analyze % python3 -m timeit '__import__("sqlite3").connect("global-power-plants.db").execute("select country_long, count(*) from [global-power-plants] group by country_long").fetchall()' 100 loops, best of 5: 2.39 msec per loop analyze % python3 -m timeit '__import__("sqlite3").connect("global-power-plants-analyzed.db").execute("select country_long, count(*) from [global-power-plants] group by country_long").fetchall()' 100 loops, best of 5: 2.38 msec per loop I should try this against a much larger database.

https://covid-19.datasettes.com/covid.db is 879MB.

``` analyze % sqlite-utils global-power-plants-analyzed.db 'analyze' [{"rows_affected": -1}] analyze % sqlite-utils tables global-power-plants-analyzed.db [{"table": "global-power-plants"}, {"table": "global-power-plants_fts"}, {"table": "global-power-plants_fts_data"}, {"table": "global-power-plants_fts_idx"}, {"table": "global-power-plants_fts_docsize"}, {"table": "global-power-plants_fts_config"}, {"table": "sqlite_stat1"}] analyze % sqlite-utils rows global-power-plants-analyzed.db sqlite_stat1 -t tbl idx stat

global-power-plants_fts_config global-power-plants_fts_config 1 1 global-power-plants_fts_docsize 33643 global-power-plants_fts_idx global-power-plants_fts_idx 199 40 1 global-power-plants_fts_data 136 global-power-plants "global-power-plants_owner" 33643 4 global-power-plants "global-power-plants_country_long" 33643 202 ```

``` analyze % sqlite-utils indexes global-power-plants.db -t
table index_name seqno cid name desc coll key

global-power-plants "global-power-plants_owner" 0 12 owner 0 BINARY 1 global-power-plants "global-power-plants_country_long" 0 1 country_long 0 BINARY 1 global-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 0 0 segid 0 BINARY 1 global-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 1 1 term 0 BINARY 1 global-power-plants_fts_config sqlite_autoindex_global-power-plants_fts_config_1 0 0 k 0 BINARY 1 ```

I'll start by running some experiments against the 11MB database file from https://global-power-plants.datasettes.com/global-power-plants.db

Refs: - #366 - #365

In Python 3.6: https://docs.python.org/3.6/library/subprocess.html

This does not capture stdout or stderr by default. To do so, pass PIPE for the stdout and/or stderr arguments.

Test failure on Python 3.6:

E TypeError: __init__() got an unexpected keyword argument 'capture_output'

I can now write tests that look like this: https://github.com/simonw/sqlite-utils/blob/539f5ccd90371fa87f946018f8b77d55929e06db/tests/test_cli.py#L2024-L2030

Which means I can write a test that exercises this bug.

Even better, inspired by rich, support python -m sqlite_utils. https://github.com/Textualize/rich/blob/master/rich/main.py

Possible way of running the test: add this to sqlite_utils/cli.py:

python if __name__ == "__main__": cli() Now the tool can be run using python -m sqlite_utils.cli --help

Then in the test use subprocess to call sys.executable (the path to the current Python interpreter) and pass it -m sqlite_utils.cli to run the script!

I'm having trouble figuring out the best way to write a unit test for this. Filed a relevant feature request for Click here: - https://github.com/pallets/click/issues/2171