Have you seen MakeTypes? Not the exact same thing but it may be relevant.

And it's inspired by the paper "Types from Data: Making Structured Data First-Class Citizens in F#".

Codecov Report

Merging #1763 (fd6a817) into main (00e59ec) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1763   +/-   ##
=======================================
  Coverage   91.67%   91.67%           
=======================================
  Files          36       36           
  Lines        4658     4658           
=======================================
  Hits         4270     4270           
  Misses        388      388           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 00e59ec...fd6a817. Read the comment docs.

Superseded by #1763.

Yeah, I think I prefer your suggestion: it seems cleaner than my initial left_name=/right_name= idea. Perhaps one downside is that it's less obvious what the role of each field is: in this example, is people_id_1 a reference to parent or child?

In terms of unique identifiers - could you use values stored in HKMetadataKeySyncIdentifier?

Codecov Report

Merging #1760 (69951ee) into main (00e59ec) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1760   +/-   ##
=======================================
  Coverage   91.67%   91.67%           
=======================================
  Files          36       36           
  Lines        4658     4658           
=======================================
  Hits         4270     4270           
  Misses        388      388           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 00e59ec...69951ee. Read the comment docs.

Superseded by #1760.

log.txt

Attempted to test on a machine with a new version of Python, but install failed with an error message for the 'click' package.

C:\WINDOWS\system32>"c:\Program Files\Python310\python.exe"
Python 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

C:\WINDOWS\system32>cd C:\Users\swood\Downloads\sqlite-utils-main-20220621\sqlite-utils-main

C:\Users\swood\Downloads\sqlite-utils-main-20220621\sqlite-utils-main>"c:\Program Files\Python310\python.exe" setup.py install
running install
running bdist_egg
running egg_info

...

Installed c:\program files\python310\lib\site-packages\click_default_group_wheel-1.2.2-py3.10.egg
Searching for click
Downloading https://files.pythonhosted.org/packages/3d/da/f3bbf30f7e71d881585d598f67f4424b2cc4c68f39849542e81183218017/click-default-group-wheel-1.2.2.tar.gz#sha256=e90da42d92c03e88a12ed0c0b69c8a29afb5d36e3dc8d29c423ba4219e6d7747
Best match: click default-group-wheel-1.2.2
Processing click-default-group-wheel-1.2.2.tar.gz
Writing C:\Users\swood\AppData\Local\Temp\easy_install-aiaj0_eh\click-default-group-wheel-1.2.2\setup.cfg
Running click-default-group-wheel-1.2.2\setup.py -q bdist_egg --dist-dir C:\Users\swood\AppData\Local\Temp\easy_install-aiaj0_eh\click-default-group-wheel-1.2.2\egg-dist-tmp-z61a4h8n
zip_safe flag not set; analyzing archive contents...
removing 'c:\program files\python310\lib\site-packages\click_default_group_wheel-1.2.2-py3.10.egg' (and everything under it)
Copying click_default_group_wheel-1.2.2-py3.10.egg to c:\program files\python310\lib\site-packages
click-default-group-wheel 1.2.2 is already the active version in easy-install.pth

Installed c:\program files\python310\lib\site-packages\click_default_group_wheel-1.2.2-py3.10.egg
error: The 'click' distribution was not found and is required by click-default-group-wheel, sqlite-utils

just -l now does this:

% just -l
Available recipes:
    black         # Apply Black
    cog           # Rebuild docs with cog
    default       # Run tests and linters
    lint          # Run linters: black, flake8, mypy, cog
    test *options # Run pytest with supplied options

Pushed that prototype to a branch.

Built a prototype of --fast for the sqlite-utils memory command:

% time sqlite-utils memory taxi.csv 'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count' --fast
passenger_count  COUNT(*)  AVG(total_amount)
---------------  --------  -----------------
                 128020    32.2371511482553 
0                42228     17.0214016766151 
1                1533197   17.6418833067999 
2                286461    18.0975870711456 
3                72852     17.9153958710923 
4                25510     18.452774990196  
5                50291     17.2709248175672 
6                32623     17.6002964166367 
7                2         87.17            
8                2         95.705           
9                1         113.6            
sqlite-utils memory taxi.csv  --fast  12.71s user 0.48s system 104% cpu 12.627 total

Takes 13s - about the same time as calling sqlite3 :memory: ... directly as seen in https://til.simonwillison.net/sqlite/one-line-csv-operations

Without the --fast option that takes several minutes (262s = 4m20s)!

Here's the prototype so far:

diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
index 86eddfb..1c83ef6 100644
--- a/sqlite_utils/cli.py
+++ b/sqlite_utils/cli.py
@@ -14,6 +14,8 @@ import io
 import itertools
 import json
 import os
+import shutil
+import subprocess
 import sys
 import csv as csv_std
 import tabulate
@@ -1669,6 +1671,7 @@ def query(
     is_flag=True,
     help="Analyze resulting tables and output results",
 )
+@click.option("--fast", is_flag=True, help="Fast mode, only works with CSV and TSV")
 @load_extension_option
 def memory(
     paths,
@@ -1692,6 +1695,7 @@ def memory(
     save,
     analyze,
     load_extension,
+    fast,
 ):
     """Execute SQL query against an in-memory database, optionally populated by imported data

@@ -1719,6 +1723,22 @@ def memory(
     \b
         sqlite-utils memory animals.csv --schema
     """
+    if fast:
+        if (
+            attach
+            or flatten
+            or param
+            or encoding
+            or no_detect_types
+            or analyze
+            or load_extension
+        ):
+            raise click.ClickException(
+                "--fast mode does not support any of the following options: --attach, --flatten, --param, --encoding, --no-detect-types, --analyze, --load-extension"
+            )
+        # TODO: Figure out and pass other supported options
+        memory_fast(paths, sql)
+        return
     db = sqlite_utils.Database(memory=True)
     # If --dump or --save or --analyze used but no paths detected, assume SQL query is a path:
     if (dump or save or schema or analyze) and not paths:
@@ -1791,6 +1811,33 @@ def memory(
     )


+def memory_fast(paths, sql):
+    if not shutil.which("sqlite3"):
+        raise click.ClickException("sqlite3 not found in PATH")
+    args = ["sqlite3", ":memory:", "-cmd", ".mode csv"]
+    table_names = []
+
+    def name(path):
+        base_name = pathlib.Path(path).stem or "t"
+        table_name = base_name
+        prefix = 1
+        while table_name in table_names:
+            prefix += 1
+            table_name = "{}_{}".format(base_name, prefix)
+        return table_name
+
+    for path in paths:
+        table_name = name(path)
+        table_names.append(table_name)
+        args.extend(
+            ["-cmd", ".import {} {}".format(pathlib.Path(path).resolve(), table_name)]
+        )
+
+    args.extend(["-cmd", ".mode column"])
+    args.append(sql)
+    subprocess.run(args)
+
+
 def _execute_query(
     db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols
 ):

That fixed it:

https://user-images.githubusercontent.com/9599/174875556-3a569c90-5c92-48eb-935c-470638deb335.png">

The thing I like about that --fast option is that it could selectively use this alternative mechanism just for the files for which it can work (CSV and TSV files). I could also add a --fast option to sqlite-utils memory which could then kick in only for operations that involve just TSV and CSV files.

Deploying that to https://sqlite-utils.datasette.io/en/latest/cli-reference.html#insert

https://stackoverflow.com/a/44379513 suggests that the fix is:

.. code-block:: text

Or set this in conf.py:

highlight_language = "none"

I like that better - I don't like that all :: blocks default to being treated as Python code.

Since there are all sorts of existing options for sqlite-utils insert that won't work with this, maybe it would be better to have an entirely separate command - this for example:

sqlite-utils fast-insert data.db mytable data.csv

Another implementation option would be to use the CSV virtual table mechanism. This could avoid shelling out to the sqlite3 binary, but requires solving the harder problem of compiling and distributing a loadable SQLite module: https://www.sqlite.org/csv.html

(Would be neat to produce a Python wheel of this, see https://simonwillison.net/2022/May/23/bundling-binary-tools-in-python-wheels/)

This would also help solve the challenge of making this optimization available to the sqlite-utils memory command. That command operates against an in-memory database so it's not obvious how it could shell out to a binary.

Relevant TIL: https://til.simonwillison.net/sqlite/one-line-csv-operations

just now defaults to running the tests and linters.

just test runs the tests - it can take arguments, e.g. just test -k transform

just lint runs all of the linters.

just black applies Black.

In all case it assumes you are using pipenv, at least for the moment.

Also now shows up here: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-rows-from-file

New documentation:

• https://sqlite-utils.datasette.io/en/latest/python-api.html#detecting-column-types-using-typetracker
• https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-typetracker

I also added inline documentation and types: https://github.com/simonw/sqlite-utils/blob/773f2b6b20622bb986984a1c3161d5b3aaa1046b/sqlite_utils/utils.py#L318-L360

Code to document: https://github.com/simonw/sqlite-utils/blob/3fbe8a784cc2f3fa0bfa8612fec9752ff9068a2b/sqlite_utils/utils.py#L318-L331

Codecov Report

Merging #1759 (b901bb0) into main (2e97516) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1759   +/-   ##
=======================================
  Coverage   91.67%   91.67%           
=======================================
  Files          36       36           
  Lines        4658     4658           
=======================================
  Hits         4270     4270           
  Misses        388      388           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e97516...b901bb0. Read the comment docs.

I don't think this change needs any changes to the documentation: https://docs.datasette.io/en/stable/custom_templates.html#custom-templates

This is a great idea.

I'm going to push a branch with my not-yet-working code (which does at least include a test).

Yeah my initial implementation there makes no sense:

            csv_reader_args = {"dialect": dialect}
            if delimiter:
                csv_reader_args["delimiter"] = delimiter
            if quotechar:
                csv_reader_args["quotechar"] = quotechar
            reader = _extra_key_strategy(
                csv_std.reader(decoded, **csv_reader_args), ignore_extras, extras_key
            )
            first_row = next(reader)
            if no_headers:
                headers = ["untitled_{}".format(i + 1) for i in range(len(first_row))]
                reader = itertools.chain([first_row], reader)
            else:
                headers = first_row
            docs = (dict(zip(headers, row)) for row in reader)

Because my _extra_key_strategy() helper function is designed to work against csv.DictReader - not against csv.reader() which returns a sequenc of lists, not a sequence of dictionaries.

In fact, what's happening here is that dict(zip(headers, row)) is ignoring anything in the row that doesn't correspond to a header:

>>> list(zip(["a", "b"], [1, 2, 3]))
[('a', 1), ('b', 2)]

I tried fixing this by using .tell() to read the file position as I was iterating through it:

diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py
index d2ccc5f..29ad12e 100644
--- a/sqlite_utils/utils.py
+++ b/sqlite_utils/utils.py
@@ -149,10 +149,13 @@ class UpdateWrapper:
     def __init__(self, wrapped, update):
         self._wrapped = wrapped
         self._update = update
+        self._tell = wrapped.tell()

     def __iter__(self):
         for line in self._wrapped:
-            self._update(len(line))
+            tell = self._wrapped.tell()
+            self._update(self._tell - tell)
+            self._tell = tell
             yield line
 ```
This did not work - I get this error:

File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py", line 206, in _extra_key_strategy
for row in reader:
File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py", line 156, in iter
tell = self._wrapped.tell()
OSError: telling position disabled by next() call
`` It looks like you can't use.tell()` during iteration: https://stackoverflow.com/questions/29618936/how-to-solve-oserror-telling-position-disabled-by-next-call

This is tricky to implement because of this code: https://github.com/simonw/sqlite-utils/blob/b8af3b96f5c72317cc8783dc296a94f6719987d9/sqlite_utils/cli.py#L938-L945

It's reconstructing each document using the known headers here:

docs = (dict(zip(headers, row)) for row in reader)

So my first attempt at this - the diff here - did not have the desired result:

diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
index 86eddfb..00b920b 100644
--- a/sqlite_utils/cli.py
+++ b/sqlite_utils/cli.py
@@ -6,7 +6,7 @@ import hashlib
 import pathlib
 import sqlite_utils
 from sqlite_utils.db import AlterError, BadMultiValues, DescIndex
-from sqlite_utils.utils import maximize_csv_field_size_limit
+from sqlite_utils.utils import maximize_csv_field_size_limit, _extra_key_strategy
 from sqlite_utils import recipes
 import textwrap
 import inspect
@@ -797,6 +797,15 @@ _import_options = (
         "--encoding",
         help="Character encoding for input, defaults to utf-8",
     ),
+    click.option(
+        "--ignore-extras",
+        is_flag=True,
+        help="If a CSV line has more than the expected number of values, ignore the extras",
+    ),
+    click.option(
+        "--extras-key",
+        help="If a CSV line has more than the expected number of values put them in a list in this column",
+    ),
 )


@@ -885,6 +894,8 @@ def insert_upsert_implementation(
     sniff,
     no_headers,
     encoding,
+    ignore_extras,
+    extras_key,
     batch_size,
     alter,
     upsert,
@@ -909,6 +920,10 @@ def insert_upsert_implementation(
         raise click.ClickException("--flatten cannot be used with --csv or --tsv")
     if encoding and not (csv or tsv):
         raise click.ClickException("--encoding must be used with --csv or --tsv")
+    if ignore_extras and extras_key:
+        raise click.ClickException(
+            "--ignore-extras and --extras-key cannot be used together"
+        )
     if pk and len(pk) == 1:
         pk = pk[0]
     encoding = encoding or "utf-8-sig"
@@ -935,7 +950,9 @@ def insert_upsert_implementation(
                 csv_reader_args["delimiter"] = delimiter
             if quotechar:
                 csv_reader_args["quotechar"] = quotechar
-            reader = csv_std.reader(decoded, **csv_reader_args)
+            reader = _extra_key_strategy(
+                csv_std.reader(decoded, **csv_reader_args), ignore_extras, extras_key
+            )
             first_row = next(reader)
             if no_headers:
                 headers = ["untitled_{}".format(i + 1) for i in range(len(first_row))]
@@ -1101,6 +1118,8 @@ def insert(
     sniff,
     no_headers,
     encoding,
+    ignore_extras,
+    extras_key,
     batch_size,
     alter,
     detect_types,
@@ -1176,6 +1195,8 @@ def insert(
             sniff,
             no_headers,
             encoding,
+            ignore_extras,
+            extras_key,
             batch_size,
             alter=alter,
             upsert=False,
@@ -1214,6 +1235,8 @@ def upsert(
     sniff,
     no_headers,
     encoding,
+    ignore_extras,
+    extras_key,
     alter,
     not_null,
     default,
@@ -1254,6 +1277,8 @@ def upsert(
             sniff,
             no_headers,
             encoding,
+            ignore_extras,
+            extras_key,
             batch_size,
             alter=alter,
             upsert=True,
@@ -1297,6 +1322,8 @@ def bulk(
     sniff,
     no_headers,
     encoding,
+    ignore_extras,
+    extras_key,
     load_extension,
 ):
     """
@@ -1331,6 +1358,8 @@ def bulk(
             sniff=sniff,
             no_headers=no_headers,
             encoding=encoding,
+            ignore_extras=ignore_extras,
+            extras_key=extras_key,
             batch_size=batch_size,
             alter=False,
             upsert=False,

I'm going with --extras-key and --ignore-extras as the two new options.

I think these become part of the _import_options list which is used in a few places:

https://github.com/simonw/sqlite-utils/blob/b8af3b96f5c72317cc8783dc296a94f6719987d9/sqlite_utils/cli.py#L765-L800

It looks like PRAGMA temp_store was the right option to use here: https://www.sqlite.org/pragma.html#pragma_temp_store

temp_store_directory is listed as deprecated here: https://www.sqlite.org/pragma.html#pragma_temp_store_directory

I'm going to turn this into a help-wanted documentation issue.

Since table names can be quoted like this:

CREATE VIRTUAL TABLE "searchable_fts"
    USING FTS4 (text1, text2, [name with . and spaces], content="searchable")

OR like this:

CREATE VIRTUAL TABLE "searchable_fts"
    USING FTS4 (text1, text2, [name with . and spaces], content=[searchable])

This fix looks to be correct to me (copying from the updated test_with_trace() test):

            (
                "SELECT name FROM sqlite_master\n"
                "    WHERE rootpage = 0\n"
                "    AND (\n"
                "        sql LIKE :like\n"
                "        OR sql LIKE :like2\n"
                "        OR (\n"
                "            tbl_name = :table\n"
                "            AND sql LIKE '%VIRTUAL TABLE%USING FTS%'\n"
                "        )\n"
                "    )",
                {
                    "like": "%VIRTUAL TABLE%USING FTS%content=[dogs]%",
                    "like2": '%VIRTUAL TABLE%USING FTS%content="dogs"%',
                    "table": "dogs",
                },
            )

A test that demonstrates the problem:

@pytest.mark.parametrize("reverse_order", (True, False))
def test_detect_fts_similar_tables(fresh_db, reverse_order):
    # https://github.com/simonw/sqlite-utils/issues/434
    table1, table2 = ("demo", "demo2")
    if reverse_order:
        table1, table2 = table2, table1

    fresh_db[table1].insert({"title": "Hello"}).enable_fts(
        ["title"], fts_version="FTS4"
    )
    fresh_db[table2].insert({"title": "Hello"}).enable_fts(
        ["title"], fts_version="FTS4"
    )
    assert fresh_db[table1].detect_fts() == "{}_fts".format(table1)
    assert fresh_db[table2].detect_fts() == "{}_fts".format(table2)

The order matters - so this test currently passes in one direction and fails in the other:

>       assert fresh_db[table2].detect_fts() == "{}_fts".format(table2)
E       AssertionError: assert 'demo2_fts' == 'demo_fts'
E         - demo_fts
E         + demo2_fts
E         ?     +

tests/test_introspect.py:53: AssertionError
========================================================================================= short test summary info =========================================================================================
FAILED tests/test_introspect.py::test_detect_fts_similar_tables[True] - AssertionError: assert 'demo2_fts' == 'demo_fts'
=============================================================================== 1 failed, 1 passed, 855 deselected in 1.00s ===============================================================================

Definitely a bug - thanks for the detailed write-up!

You're right, the code at fault is here:

https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L2213-L2231

I'm going to mark this as "help wanted" and leave it open. I'm glad that it's not actually a bug where errors get swallowed.

The progress bar only works if the file-like object passed to it has a fp.fileno() that isn't 0 (for stdin) - that's how it detects that the file is something which it can measure the size of in order to show progress.

If we know the file size in bytes AND we know the character encoding, can we change UpdateWrapper to update the number of bytes-per-character instead?

I don't think so: I can't see a way of definitively saying "for this encoding the number of bytes per character is X" - and in fact I'm pretty sure that question doesn't even make sense since variable-length encodings exist.

Yes, this is the problem. The progress bar length is set to the length in bytes of the file - os.path.getsize(file.name) - but it's then incremented by the length of each DECODED line in turn.

So if the file is in utf-16-le (twice the size of utf-8) the progress bar will finish at 50%!

Here's the code that implements the progress bar in question: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/cli.py#L918-L932

It calls file_progress() which looks like this:

https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/utils.py#L159-L175

Which uses this:

https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/utils.py#L148-L156

TIL how to use iconv: https://til.simonwillison.net/linux/iconv

Wait! The arguments in that are the wrong way round. This is correct:

sqlite-utils insert --csv --delimiter ";" --encoding "utf-16-le" test.db test csv

It still outputs the following:

[------------------------------------] 0%
[#################-------------------] 49% 00:00:02%

But it creates a test.db file that is 6.2MB.

That database has 3141 rows in it:

% sqlite-utils tables test.db --counts -t
table      count
-------  -------
test        3142

I converted that csv file to utf-8 like so:

iconv -f UTF-16LE -t UTF-8 csv > utf8.csv

And it contains 3142 lines:

% wc -l utf8.csv 
    3142 utf8.csv

So my hunch here is that the problem is actually that the progress bar doesn't know how to correctly measure files in utf-16-le encoding!

Tried this:

% python -i $(which sqlite-utils) insert --csv --delimiter ";" --encoding "utf-16-le" test test.db csv
  [------------------------------------]    0%
  [#################-------------------]   49%  00:00:01Traceback (most recent call last):
  File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1072, in main
    ctx.exit()
  File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 692, in exit
    raise Exit(code)
click.exceptions.Exit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils", line 33, in <module>
    sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())
  File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1090, in main
    sys.exit(e.exit_code)
SystemExit: 0
>>> 

Maybe this isn't a CSV field value problem - I tried this patch and didn't seem to hit the new breakpoints:

diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py
index d2ccc5f..f1b823a 100644
--- a/sqlite_utils/utils.py
+++ b/sqlite_utils/utils.py
@@ -204,13 +204,17 @@ def _extra_key_strategy(
         # DictReader adds a 'None' key with extra row values
         if None not in row:
             yield row
-        elif ignore_extras:
+            continue
+        else:
+            breakpoint()
+        if ignore_extras:
             # ignoring row.pop(none) because of this issue:
             # https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155358637
             row.pop(None)  # type: ignore
             yield row
         elif not extras_key:
             extras = row.pop(None)  # type: ignore
+            breakpoint()
             raise RowError(
                 "Row {} contained these extra values: {}".format(row, extras)
             )

I have a hunch that this crash may be caused by a CSV value which is too long, as addressed at the library level in:
- #440

But not yet addressed in the CLI tool, see:

• 444

Either way though, I really don't like that errors like this are swallowed!

I forgot to add equivalents of extras_key= and ignore_extras= to the CLI tool - will do that in a separate issue.

I can't figure out why that error is being swallowed like that. The most likely culprit was this code:

https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/cli.py#L1021-L1043

But I tried changing it like this:

diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
index 86eddfb..ed26fdd 100644
--- a/sqlite_utils/cli.py
+++ b/sqlite_utils/cli.py
@@ -1023,6 +1023,7 @@ def insert_upsert_implementation(
             docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs
         )
     except Exception as e:
+        raise
         if (
             isinstance(e, sqlite3.OperationalError)
             and e.args

And your steps to reproduce still got to 49% and then failed silently.

Initial idea of how the .table() method would change:

diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index 7a06304..3ecb40b 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -474,11 +474,12 @@ class Database:
             self._tracer(sql, None)
         return self.conn.executescript(sql)

-    def table(self, table_name: str, **kwargs) -> Union["Table", "View"]:
+    def table(self, table_name: str, alias: Optional[str] = None, **kwargs) -> Union["Table", "View"]:
         """
         Return a table object, optionally configured with default options.

         :param table_name: Name of the table
+        :param alias: The database alias to use, if referring to a table in another connected database
         """
         klass = View if table_name in self.view_names() else Table
         return klass(self, table_name, **kwargs)

Implementing this would be a pretty big change - initial instinct is that I'd need to introduce a self.alias property to Queryable (the subclass of Table and View) and a new self.name_with_alias getter which returns alias.tablename if alias is set to a not-None value. Then I'd need to rewrite every piece of code like this:

https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L1161

To look like this instead:

        sql = "select {} from [{}]".format(select, self.name_with_alias)

But some parts would be harder - for example:

https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L1227-L1231

Would have to know to query alias.sqlite_master instead.

The cached table counts logic like this would need a bunch of changes too:

https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L644-L657

Third option, and I think the one I like the best:

rows = db.table("tablename", alias="otherdb").rows_where(alias="otherdb")

The db.table(tablename) method already exists as an alternative to db[tablename]: https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-table-configuration

Another potential fix: add a alias= parameter to rows_where() and other similar methods. Then you could do this:

rows = db["tablename"].rows_where(alias="otherdb")

This feels wrong to me: db["tablename"] is the bit that is supposed to return a table object. Having part of what that table object is exist as a parameter to other methods is confusing.

I don't like the idea of table_names() returning names of tables from connected databases as well, because it feels like it could lead to surprising behaviour - especially if those connected databases turn to have table names that are duplicated in the main connected database.

It would be neat if functions like .rows_where() worked though.

One thought would be to support something like this:

rows = db["otherdb.tablename"].rows_where()

But... . is a valid character in a SQLite table name. So "otherdb.tablename" might ambiguously refer to a table called tablename in a connected database with the alias otherdb, OR a table in the current database with the name otherdb.tablename.

Yeah, I think it would be neat if the library could support self-referential many-to-many in a nice way.

I'm not sure about the left_name/right_name design though. Would it be possible to have this work as the user intends, by spotting that the other table name "people" matches the name of the current table?

db["people"].insert({"name": "Mary"}, pk="name").m2m(
    "people", [{"name": "Michael"}, {"name": "Suzy"}], m2m_table="parent_child", pk="name"
)

The created table could look like this:

CREATE TABLE [parent_child] (
   [people_id_1] TEXT REFERENCES [people]([name]),
   [people_id_2] TEXT REFERENCES [people]([name]),
   PRIMARY KEY ([people_id_1], [people_id_2])
)

I've not thought very hard about this, so the design I'm proposing here might not work.

Are there other reasons people might wan the left_name= and right_name= parameters? If so then I'm much happier with those.

I added where= and where_args= parameters to that .search() method - updated documentation is here: https://sqlite-utils.datasette.io/en/latest/python-api.html#searching-with-table-search

Marking this as help wanted because I can't figure out how to replicate it!

Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#setting-the-maximum-csv-field-size-limit

Here's the commit where I added that originally, including a test: https://github.com/simonw/sqlite-utils/commit/1a93b72ba710ea2271eaabc204685a27d2469374

Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file

New documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file

I'm going to rename restkey to extras_key for consistency with ignore_extras.

That would be handy (additional where filters) but I think the trick with the with statement is already an order of magnitude better than what I had thought of, so my problem is solved by it (plus I got to learn about with today!)

Actually I have a thought for something that could help here: I could add a mechanism for inserting additional where filters and parameters into that .search() method.

Filed an issue against python/typeshed:

• https://github.com/python/typeshed/issues/8075

Here's as far as I got with my initial prototype, in sqlite_utils/pandas.py:

from .db import Database as _Database, Table as _Table, View as _View
import pandas as pd
from typing import (
    Iterable,
    Union,
    Optional,
)


class Database(_Database):
    def query(
        self, sql: str, params: Optional[Union[Iterable, dict]] = None
    ) -> pd.DataFrame:
        return pd.DataFrame(super().query(sql, params))

    def table(self, table_name: str, **kwargs) -> Union["Table", "View"]:
        "Return a table object, optionally configured with default options."
        klass = View if table_name in self.view_names() else Table
        return klass(self, table_name, **kwargs)


class PandasQueryable:
    def rows_where(
        self,
        where: str = None,
        where_args: Optional[Union[Iterable, dict]] = None,
        order_by: str = None,
        select: str = "*",
        limit: int = None,
        offset: int = None,
    ) -> pd.DataFrame:
        return pd.DataFrame(
            super().rows_where(
                where,
                where_args,
                order_by=order_by,
                select=select,
                limit=limit,
                offset=offset,
            )
        )


class Table(PandasQueryable, _Table):
    pass


class View(PandasQueryable, _View):
    pass

Getting this past mypy is really hard!

% mypy sqlite_utils
sqlite_utils/utils.py:189: error: No overload variant of "pop" of "MutableMapping" matches argument type "None"
sqlite_utils/utils.py:189: note: Possible overload variants:
sqlite_utils/utils.py:189: note:     def pop(self, key: str) -> str
sqlite_utils/utils.py:189: note:     def [_T] pop(self, key: str, default: Union[str, _T] = ...) -> Union[str, _T]

That's because of this line:

row.pop(key=None)

Which is legit here - we have a dictionary where one of the keys is None and we want to remove that key. But the baked in type is apparently def pop(self, key: str) -> str.

That broke mypy:

sqlite_utils/utils.py:229: error: Incompatible types in assignment (expression has type "Iterable[Dict[Any, Any]]", variable has type "DictReader[str]")

I think that's unavoidable: it looks like csv.Sniffer only works if you feed it a CSV file with an equal number of values in each row, which is understandable.

Interesting challenge in writing tests for this: if you give csv.Sniffer a short example with an invalid row in it sometimes it picks the wrong delimiter!

id,name\r\n1,Cleo,oops

It decided the delimiter there was e.

The exception will be called RowError.

Decision: I'm going to default to raising an exception if a row has too many values in it.

You'll be able to pass ignore_extras=True to ignore those extra values, or pass restkey="the_rest" to stick them in a list in the restkey column.

Here's the current function signature for rows_from_file():

https://github.com/simonw/sqlite-utils/blob/26e6d2622c57460a24ffdd0128bbaac051d51a5f/sqlite_utils/utils.py#L174-L179

Whatever I decide, I can implement it in rows_from_file(), maybe as an optional parameter - then decide how to call it from the sqlite-utils insert CLI (perhaps with a new option there too).

So I need to make a design decision here: what should sqlite-utils do with CSV files that have rows with more values than there are headings?

Some options:

• Ignore those extra fields entirely - silently drop that data. I'm not keen on this.
• Throw an error. The library does this already, but the error is incomprehensible - it could turn into a useful, human-readable error instead.
• Put the data in a JSON list in a column with a known name (None is not a valid column name, so not that). This could be something like _restkey or _values_with_no_heading. This feels like a better option, but I'd need to carefully pick a name for it - and come up with an answer for the question of what to do if the CSV file being important already uses that heading name for something else.

That weird behaviour is documented here: https://docs.python.org/3/library/csv.html#csv.DictReader

If a row has more fields than fieldnames, the remaining data is put in a list and stored with the fieldname specified by restkey (which defaults to None). If a non-blank row has fewer fields than fieldnames, the missing values are filled-in with the value of restval (which defaults to None).

Aha! I think I see what's happening here. Here's what DictReader does if one of the lines has too many items in it:

>>> import csv, io
>>> list(csv.DictReader(io.StringIO("id,name\n1,Cleo,nohead\n2,Barry")))
[{'id': '1', 'name': 'Cleo', None: ['nohead']}, {'id': '2', 'name': 'Barry'}]

See how that row with too many items gets this:
[{'id': '1', 'name': 'Cleo', None: ['nohead']}

That's a None for the key and (weirdly) a list containing the single item for the value!

Here are full steps to replicate the bug:

from urllib.request import urlopen
import sqlite_utils
db = sqlite_utils.Database(memory=True)
with urlopen("https://artsdatabanken.no/Fab2018/api/export/csv") as fab:
    reader, other = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le")
    db["fab2018"].insert_all(reader, pk="Id")

Fixing that key thing (to ignore any key that is None) revealed a new bug:

File ~/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py:376, in hash_record(record, keys)
    373 if keys is not None:
    374     to_hash = {key: record[key] for key in keys}
    375 return hashlib.sha1(
--> 376     json.dumps(to_hash, separators=(",", ":"), sort_keys=True, default=repr).encode(
    377         "utf8"
    378     )
    379 ).hexdigest()

File ~/.pyenv/versions/3.8.2/lib/python3.8/json/__init__.py:234, in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    232 if cls is None:
    233     cls = JSONEncoder
--> 234 return cls(
    235     skipkeys=skipkeys, ensure_ascii=ensure_ascii,
    236     check_circular=check_circular, allow_nan=allow_nan, indent=indent,
    237     separators=separators, default=default, sort_keys=sort_keys,
    238     **kw).encode(obj)

File ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:199, in JSONEncoder.encode(self, o)
    195         return encode_basestring(o)
    196 # This doesn't pass the iterator directly to ''.join() because the
    197 # exceptions aren't as detailed.  The list call should be roughly
    198 # equivalent to the PySequence_Fast that ''.join() would do.
--> 199 chunks = self.iterencode(o, _one_shot=True)
    200 if not isinstance(chunks, (list, tuple)):
    201     chunks = list(chunks)

File ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:257, in JSONEncoder.iterencode(self, o, _one_shot)
    252 else:
    253     _iterencode = _make_iterencode(
    254         markers, self.default, _encoder, self.indent, floatstr,
    255         self.key_separator, self.item_separator, self.sort_keys,
    256         self.skipkeys, _one_shot)
--> 257 return _iterencode(o, 0)

TypeError: '<' not supported between instances of 'NoneType' and 'str'

I don't understand why that works but calling insert_all() does not.

Steps to demonstrate that sqlite-utils insert is not affected:

curl -o artsdatabanken.csv https://artsdatabanken.no/Fab2018/api/export/csv
sqlite-utils insert arts.db artsdatabanken artsdatabanken.csv --sniff --csv --encoding utf-16le

rows_from_file() isn't part of the documented API but maybe it should be!

Yeah, at the moment the best way to do this is with search_sql(), but you're right it really isn't very intuitive.

Here's how I would do this, using a CTE trick to combine the queries:

search_sql = db["articles"].search_sql(columns=["title", "author"]))
sql = f"""
with search_results as ({search_sql})
select * from search_results where owner = :owner
"""
results = db.query(sql, {"query": "my search query", "owner": "my owner"})

I'm not sure if sqlite-utils should ever evolve to provide a better way of doing this kind of thing to be honest - if it did, it would turn into more of an ORM. Something like PeeWee may be a better option here.

I just put together a version of this in a plugin: https://github.com/eyeseast/datasette-query-files. Happy to have any feedback.

Codecov Report

Merging #1753 (23a8515) into main (2e97516) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1753   +/-   ##
=======================================
  Coverage   91.67%   91.67%           
=======================================
  Files          36       36           
  Lines        4658     4658           
=======================================
  Hits         4270     4270           
  Misses        388      388           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e97516...23a8515. Read the comment docs.

Thanks, this looks like a good idea to me.

I ran into this. My use case has a JSON file with array of book objects with a key called reviews which is also an array of objects. My JSON is human-edited and does not specify IDs for either books or reviews. Because sqlite-utils does not support inserting nested objects, I instead have to maintain two separate CSV files with id column in books.csv and book_id column in reviews.csv.

I think the right way to declare the relationship while inserting a JSON might be to describe the relationship:

sqlite-utils insert data.db books mydata.json --hasmany reviews --hasone author --manytomany tags

This is relying on the assumption that foreign keys can point to rowid primary key.

Thanks!

Closing in favor of existing issue #1298.

This code works:

import csv
import sqlite_utils
db = sqlite_utils.Database("test.db")
reader = csv.DictReader(open("csv", encoding="utf-16-le").read().split("\r\n"), delimiter=";")
db["test"].insert_all(reader, pk="Id")

I used iconv to change the encoding; sqlite-utils can import the resulting file, even if it stops at 98 %:

sqlite-utils insert --csv test test.db clean 
  [------------------------------------]    0%
  [###################################-]   98%  00:00:00

I can confirm. This only happens with sqlite-utils. I am using gnome-terminal with bash.

Argument were specified in the wrong order. PATH TABLE FILE can be misleading :)

I tried to debug it using pdb, but it looks sqlite-utils catches the exception, so it is not quick to figure out where the failure is happening.

Before:

After:

Done: https://sqlite-utils.datasette.io/en/latest/reference.html

That fixed it:

https://user-images.githubusercontent.com/9599/169614893-ec81fe0e-6043-4d7d-b429-7f087dbeaf61.png">

Relevant JavaScript: https://github.com/simonw/datasette/blob/1d33fd03b3c211e0f48a8f3bde83880af89e4e69/docs/_static/js/custom.js#L20-L24

Now live at https://docs.datasette.io/en/latest/ - the JavaScript that adds the banner about that not being the stable version doesn't seem to work though.

Before:

https://user-images.githubusercontent.com/9599/169607254-99bc0358-4a08-43bf-9aac-a24cc2121979.png">

After:

https://user-images.githubusercontent.com/9599/169607296-bcec1ed2-517c-4acc-a9a9-d1119d0cc589.png">

Codecov Report

Merging #1693 (65a5d5e) into main (1465fea) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 65a5d5e differs from pull request most recent head ec2d1e4. Consider uploading reports for the commit ec2d1e4 to get more accurate results

@@           Coverage Diff           @@
##             main    #1693   +/-   ##
=======================================
  Coverage   91.67%   91.67%           
=======================================
  Files          36       36           
  Lines        4658     4658           
=======================================
  Hits         4270     4270           
  Misses        388      388           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d33fd0...ec2d1e4. Read the comment docs.

I think the trick will be to extend the base.html template from Furo using the same trick I used in https://til.simonwillison.net/readthedocs/custom-sphinx-templates

https://github.com/pradyunsg/furo/blob/2022.04.07/src/furo/theme/furo/base.html - the site_meta block looks good.

One last question: how to include the Plausible analytics?

Furo doesn't have any specific tools for this:

• https://github.com/pradyunsg/furo/discussions/243