issue_comments

This is my failed attempt to recreate the bug (plus some extra debugging output): ```diff % git diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index dd49d5c..ea42aea 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -1013,7 +1013,11 @@ class Table(Queryable): assert ( num_columns <= SQLITE_MAX_VARS ), "Rows can have a maximum of {} columns".format(SQLITE_MAX_VARS) + print("default batch_size = ", batch_size) batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns)) + print("new batch_size = {},num_columns = {}, MAX_VARS // num_columns = {}".format( + batch_size, num_columns, SQLITE_MAX_VARS // num_columns + )) self.last_rowid = None self.last_pk = None for chunk in chunks(itertools.chain([first_record], records), batch_size): @@ -1124,6 +1128,9 @@ class Table(Queryable): ) flat_values = list(itertools.chain(*values)) queries_and_params = [(sql, flat_values)] + print(sql.count("?"), len(flat_values)) + + # print(json.dumps(queries_and_params, indent=4))

         with self.db.conn:
             for query, params in queries_and_params:

diff --git a/tests/test_create.py b/tests/test_create.py index 5290cd8..52940df 100644 --- a/tests/test_create.py +++ b/tests/test_create.py @@ -853,3 +853,33 @@ def test_create_with_nested_bytes(fresh_db): record = {"id": 1, "data": {"foo": b"bytes"}} fresh_db["t"].insert(record) assert [{"id": 1, "data": '{"foo": "b\'bytes\'"}'}] == list(fresh_db["t"].rows) + + +def test_create_throws_useful_error_with_increasing_number_of_columns(fresh_db): + # https://github.com/simonw/sqlite-utils/issues/103 + def rows(): + yield {"name": 0} + for i in range(1, 1001): + yield { + "name": i, + "age": i, + "size": i, + "name2": i, + "age2": i, + "size2": i, + "name3": i, + "age3": i, + "size3": i, + "name4": i, + "age4": i, + "size4": i, + "name5": i, + "age5": i, + "size5": i, + "name6": i, + "age6": i, + "size6": i, + } + + fresh_db["t"].insert_all(rows()) + assert 1001 == fresh_db["t"].count ```

@b0b5h4rp13 I'm having trouble creating a test that triggers this bug. Could you share a chunk of code that replicates what you're seeing here?

I'm reconsidering this: I think this is going to happen ANY time someone has at least one row that is wider than the first row. So at the very least I should show a more understandable error message.

Hopefully anyone who runs into this problem in the future will search for and find this issue thread!

I'm not sure what to do about this.

I was thinking the solution would be to look at ALL of the rows in a batch before deciding on the maximum number of columns, but that doesn't work because we calculate batch size based on the number of columns!

I think my recommendation here is to manually pass a batch_size= argument to .insert_all() if you run into this error.

Yup we only take the number of columns in the first record into account at the moment: https://github.com/simonw/sqlite-utils/blob/d56029549acae0b0ea94c5a0f783e3b3895d9218/sqlite_utils/db.py#L1007-L1016

The varying number of columns thing is interesting - I don't think the tests cover that case much if at all.

Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#listing-views

It should only create views if the underlying tables exist.

This will check if the view exists and has the exact same matching definition as the one we want. If it doesn't, we will drop it (if it exists) and recreate it.

Blocked on #37

Blocked on #37

Blocked on #37

Proposed solution: on Cloud Run don't show the "download database" link if the database file is larger than 32MB.

I can do this with a new config setting, max_db_mb, which is automatically set by the publish cloudrun command.

This is consistent with the existing max_csv_mb setting: https://datasette.readthedocs.io/en/stable/config.html#max-csv-mb

I should set max_csv_mb to 32MB on Cloud Run deploys as well.

This sped that query up even more - down to 4ms. sql create index issue_comments_issue on issue_comments(issue);

It sped up this query a lot - 2.5s down to 300ms: sql select repos.full_name, json_object( 'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number, 'label', '#' || issues.number ) as issue, issues.title, users.login, users.id, issues.state, issues.locked, issues.assignee, issues.milestone, issues.comments, issues.created_at, issues.updated_at, issues.closed_at, issues.author_association, issues.pull_request, issues.repo, issues.type from issues join repos on repos.id = issues.repo join users on issues.user = users.id where issues.state = 'open' and issues.user not in (9599, 27856297) and not exists ( select id from issue_comments where issue_comments.user = 9599 and issues.id = issue_comments.issue ) order by issues.updated_at desc;