The issue is that underlying iterator is not fully consumed within the body of the with file_progress() block. Instead, that block creates generator expressions like docs = (dict(zip(headers, row)) for row in reader)

These iterables are consumed later, outside the with file_progress() block, which consumes the underlying iterator, and in turn updates the progress bar.

This means that the ProgressBar.__exit__ method gets called before the last time the ProgressBar.update method gets called. The result is that the code to make the cursor invisible (inside the update() method) is called after the cleanup code to make it visible (in the __exit__ method).

The fix is to move consumption of the docs iterators within the progress bar block. (

(An additional fix, to make ProgressBar more robust against this kind of misuse, would to make it refusing to update after its __exit__ method had been called, just like files cannot be read() after they are closed. That requires a in the click library).

Note that Github diff obscures the simplicity of this diff, it's just indenting a block of code.

:books: Documentation preview :books:: https://sqlite-utils--598.org.readthedocs.build/en/598/

Refs: - #551

:books: Documentation preview :books:: https://sqlite-utils--553.org.readthedocs.build/en/553/

Refs #544

• [x] Documentation for CLI options
• [x] Documentation for new Python API parameters: most_common: bool and least_common: bool
• [x] Tests for CLI
• [x] Tests for Python API

closes #507

https://dwheeler.com/essays/fixing-unix-linux-filenames.html

:books: Documentation preview :books:: https://sqlite-utils--508.org.readthedocs.build/en/508/

https://github.com/simonw/sqlite-utils/pull/498

:books: Documentation preview :books:: https://sqlite-utils--499.org.readthedocs.build/en/499/

alternatively, self.disable_fts()

I was wondering why my all my databases were giving wild search results. Turns out create_trigger was not sticking!

Running sqlite-utils triggers x.db shows [] after running enable-fts using the python api. Looking at the counts trigger it seems that is the right way to save triggers. triggers show up now

:books: Documentation preview :books:: https://sqlite-utils--498.org.readthedocs.build/en/498/

Also testing https://docs.readthedocs.io/en/stable/pull-requests.html

Refs #464

Also testing fix for: - https://github.com/readthedocs/readthedocs-preview/issues/10

:books: Documentation preview :books:: https://sqlite-utils--462.org.readthedocs.build/en/462/

Trying to get this to work with Pyodide.

Refs: https://github.com/simonw/click-default-group-wheel/issues/3

Refs #366

This tests against the latest 3.10 RC, https://www.python.org/downloads/release/python-3100rc2/

Addresses #246 - this is a bit of a kludge because it doesn't actually validate the FTS string, just makes sure that it will not crash when executed, but I figured that building a query parser is a bit out of the scope of sqlite-utils and if you actually want to use the query language, you probably need to parse that yourself.

This appears to be the canonical solution.

Hi,

I ran into a problem when trying to create a database from my Apple Healthkit data using healthkit-to-sqlite. The program crashed because of an invalid insert statement that was generated for table rDistanceCycling.

The actual problem turned out to be in sqlite-utils. Table.insert_all processes the data to be inserted in chunks of rows and checks for every chunk which columns are used, and it will collect all column names in the variable all_columns. The collection of columns is done using a nested list comprehension that is not completely correct.

I'm using a Windows machine and had to make a few adjustments to the tests in order to be able to run them because they had a posix dependency.

Thanks, kind regards,

Frans

```

this is a (condensed) chunk of data from my Apple healthkit export that caused the problem.

the 3 last items in the chunk have additional keys: metadata_HKMetadataKeySyncVersion and metadata_HKMetadataKeySyncIdentifier

chunk = [{'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '7.0.1', 'device': '<<HKDevice: 0x281cf6c70>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:7.0.1>', 'unit': 'km', 'creationDate': '2020-10-10 12:29:09 +0100', 'startDate': '2020-10-10 12:29:06 +0100', 'endDate': '2020-10-10 12:29:07 +0100', 'value': '0.00518016'}, {'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '7.0.1', 'device': '<<HKDevice: 0x281cf6c70>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:7.0.1>', 'unit': 'km', 'creationDate': '2020-10-10 12:29:10 +0100', 'startDate': '2020-10-10 12:29:07 +0100', 'endDate': '2020-10-10 12:29:08 +0100', 'value': '0.00544049'}, {'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '6.2.6', 'device': '<<HKDevice: 0x281cf83e0>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>', 'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:40:50 +0100', 'endDate': '2020-07-15 16:42:49 +0100', 'value': '0.952092', 'metadata_HKMetadataKeySyncVersion': '1', 'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520450.99823:616520569.99360:119'}, {'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '6.2.6', 'device': '<<HKDevice: 0x281cf83e0>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>', 'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:42:49 +0100', 'endDate': '2020-07-15 16:44:51 +0100', 'value': '0.848983', 'metadata_HKMetadataKeySyncVersion': '1', 'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520569.99360:616520691.98826:119'}, {'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '6.2.6', 'device': '<<HKDevice: 0x281cf83e0>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>', 'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:44:51 +0100', 'endDate': '2020-07-15 16:46:50 +0100', 'value': '0.834403', 'metadata_HKMetadataKeySyncVersion': '1', 'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520691.98826:616520810.98305:119'}]

def all_columns_old(): all_columns = [col for col in chunk[0]] all_columns += [column for record in chunk for column in record if column not in all_columns] return all_columns

def all_columns_new(): all_columns = [col for col in chunk[0]] for record in chunk: all_columns += [column for column in record if column not in all_columns] return all_columns

if name == 'main': from pprint import pprint

print('problem: ')
pprint(all_columns_old())
print('\nfix: ')
pprint(all_columns_new())

```

The limit can be passed as a string to the query builder to have an offset. I have tested it using the shorthand limit=f"15, 30", the standard syntax should work too.

Add type conversion for Panda's Timestamp, if Panda library is present in system (thanks for this project, I was about to do the same thing from scratch)

The table names weren't escaped using double quotes in the populate_fts method.

Reproducible case: ```

import sqlite_utils db = sqlite_utils.Database("abc.db") db["http://example.com"].insert_all([ ... {"id": 1, "age": 4, "name": "Cleo"}, ... {"id": 2, "age": 2, "name": "Pancakes"} ... ], pk="id")

Refs #35

Still to do:

• [x] Unit tests
• [x] Switch to using .get()
• [x] Better exceptions, plus unit tests for what happens if pk does not exist
• [x] Documentation
• [x] Ensure compound primary keys work properly
• [x] alter=True support