{"html_url": "https://github.com/simonw/datasette/pull/1693#issuecomment-1168704157", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1693", "id": 1168704157, "node_id": "IC_kwDOBm6k_c5FqQKd", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2022-06-28T13:11:36Z", "updated_at": "2022-06-28T13:11:36Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #1763.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1184850337, "label": "Bump black from 22.1.0 to 22.3.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/431#issuecomment-1164460052", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/431", "id": 1164460052, "node_id": "IC_kwDOCGYnMM5FaEAU", "user": {"value": 738408, "label": "rafguns"}, "created_at": "2022-06-23T14:12:51Z", "updated_at": "2022-06-23T14:12:51Z", "author_association": "NONE", "body": "Yeah, I think I prefer your suggestion: it seems cleaner than my initial `left_name=`/`right_name=` idea. Perhaps one downside is that it's less obvious what the role of each field is: in this example, is `people_id_1` a reference to parent or child?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1227571375, "label": "Allow making m2m relation of a table to itself"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-1163917719", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12", "id": 1163917719, "node_id": "IC_kwDOC8tyDs5FX_mX", "user": {"value": 956433, "label": "Mjboothaus"}, "created_at": "2022-06-23T04:35:02Z", "updated_at": "2022-06-23T04:35:02Z", "author_association": "NONE", "body": "In terms of unique identifiers - could you use values stored in `HKMetadataKeySyncIdentifier`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 727848625, "label": "Some workout columns should be float, not text"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1760#issuecomment-1163097455", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1760", "id": 1163097455, "node_id": "IC_kwDOBm6k_c5FU3Vv", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-06-22T13:27:08Z", "updated_at": "2022-06-22T13:27:08Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1760?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#1760](https://codecov.io/gh/simonw/datasette/pull/1760?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (69951ee) into [main](https://codecov.io/gh/simonw/datasette/commit/00e59ec461dc0150772b999c7cc15fcb9b507d58?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (00e59ec) will **not change** coverage.\n> The diff coverage is `n/a`.\n\n```diff\n@@ Coverage Diff @@\n## main #1760 +/- ##\n=======================================\n Coverage 91.67% 91.67% \n=======================================\n Files 36 36 \n Lines 4658 4658 \n=======================================\n Hits 4270 4270 \n Misses 388 388 \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1760?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1760?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [00e59ec...69951ee](https://codecov.io/gh/simonw/datasette/pull/1760?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1280136357, "label": "Bump furo from 2022.4.7 to 2022.6.21"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1753#issuecomment-1163091750", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1753", "id": 1163091750, "node_id": "IC_kwDOBm6k_c5FU18m", "user": {"value": 49699333, "label": "dependabot[bot]"}, "created_at": "2022-06-22T13:22:34Z", "updated_at": "2022-06-22T13:22:34Z", "author_association": "CONTRIBUTOR", "body": "Superseded by #1760.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1261826957, "label": "Bump furo from 2022.4.7 to 2022.6.4.1"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1162500525", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/448", "id": 1162500525, "node_id": "IC_kwDOCGYnMM5FSlmt", "user": {"value": 236907, "label": "mungewell"}, "created_at": "2022-06-22T00:46:43Z", "updated_at": "2022-06-22T00:46:43Z", "author_association": "NONE", "body": "[log.txt](https://github.com/simonw/sqlite-utils/files/8953589/log.txt)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1279144769, "label": "Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto'"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1162498734", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/448", "id": 1162498734, "node_id": "IC_kwDOCGYnMM5FSlKu", "user": {"value": 236907, "label": "mungewell"}, "created_at": "2022-06-22T00:43:45Z", "updated_at": "2022-06-22T00:43:45Z", "author_association": "NONE", "body": "Attempted to test on a machine with a new version of Python, but install failed with an error message for the 'click' package.\r\n\r\n```\r\nC:\\WINDOWS\\system32>\"c:\\Program Files\\Python310\\python.exe\"\r\nPython 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)] on win32\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n>>> quit()\r\n\r\nC:\\WINDOWS\\system32>cd C:\\Users\\swood\\Downloads\\sqlite-utils-main-20220621\\sqlite-utils-main\r\n\r\nC:\\Users\\swood\\Downloads\\sqlite-utils-main-20220621\\sqlite-utils-main>\"c:\\Program Files\\Python310\\python.exe\" setup.py install\r\nrunning install\r\nrunning bdist_egg\r\nrunning egg_info\r\n\r\n...\r\n\r\nInstalled c:\\program files\\python310\\lib\\site-packages\\click_default_group_wheel-1.2.2-py3.10.egg\r\nSearching for click\r\nDownloading https://files.pythonhosted.org/packages/3d/da/f3bbf30f7e71d881585d598f67f4424b2cc4c68f39849542e81183218017/click-default-group-wheel-1.2.2.tar.gz#sha256=e90da42d92c03e88a12ed0c0b69c8a29afb5d36e3dc8d29c423ba4219e6d7747\r\nBest match: click default-group-wheel-1.2.2\r\nProcessing click-default-group-wheel-1.2.2.tar.gz\r\nWriting C:\\Users\\swood\\AppData\\Local\\Temp\\easy_install-aiaj0_eh\\click-default-group-wheel-1.2.2\\setup.cfg\r\nRunning click-default-group-wheel-1.2.2\\setup.py -q bdist_egg --dist-dir C:\\Users\\swood\\AppData\\Local\\Temp\\easy_install-aiaj0_eh\\click-default-group-wheel-1.2.2\\egg-dist-tmp-z61a4h8n\r\nzip_safe flag not set; analyzing archive contents...\r\nremoving 'c:\\program files\\python310\\lib\\site-packages\\click_default_group_wheel-1.2.2-py3.10.egg' (and everything under it)\r\nCopying click_default_group_wheel-1.2.2-py3.10.egg to c:\\program files\\python310\\lib\\site-packages\r\nclick-default-group-wheel 1.2.2 is already the active version in easy-install.pth\r\n\r\nInstalled c:\\program files\\python310\\lib\\site-packages\\click_default_group_wheel-1.2.2-py3.10.egg\r\nerror: The 'click' distribution was not found and is required by click-default-group-wheel, sqlite-utils\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1279144769, "label": "Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto'"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/446#issuecomment-1162234441", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/446", "id": 1162234441, "node_id": "IC_kwDOCGYnMM5FRkpJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-21T19:28:35Z", "updated_at": "2022-06-21T19:28:35Z", "author_association": "OWNER", "body": "`just -l` now does this:\r\n\r\n```\r\n% just -l\r\nAvailable recipes:\r\n black # Apply Black\r\n cog # Rebuild docs with cog\r\n default # Run tests and linters\r\n lint # Run linters: black, flake8, mypy, cog\r\n test *options # Run pytest with supplied options\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1277328147, "label": "Use Just to automate running tests and linters locally"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1162231111", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1162231111, "node_id": "IC_kwDOCGYnMM5FRj1H", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-21T19:25:44Z", "updated_at": "2022-06-21T19:25:44Z", "author_association": "OWNER", "body": "Pushed that prototype to a branch.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1162223668", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1162223668, "node_id": "IC_kwDOCGYnMM5FRiA0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-21T19:19:22Z", "updated_at": "2022-06-21T19:22:15Z", "author_association": "OWNER", "body": "Built a prototype of `--fast` for the `sqlite-utils memory` command:\r\n\r\n```\r\n% time sqlite-utils memory taxi.csv 'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count' --fast\r\npassenger_count COUNT(*) AVG(total_amount)\r\n--------------- -------- -----------------\r\n 128020 32.2371511482553 \r\n0 42228 17.0214016766151 \r\n1 1533197 17.6418833067999 \r\n2 286461 18.0975870711456 \r\n3 72852 17.9153958710923 \r\n4 25510 18.452774990196 \r\n5 50291 17.2709248175672 \r\n6 32623 17.6002964166367 \r\n7 2 87.17 \r\n8 2 95.705 \r\n9 1 113.6 \r\nsqlite-utils memory taxi.csv --fast 12.71s user 0.48s system 104% cpu 12.627 total\r\n```\r\nTakes 13s - about the same time as calling `sqlite3 :memory: ...` directly as seen in https://til.simonwillison.net/sqlite/one-line-csv-operations\r\n\r\nWithout the `--fast` option that takes several minutes (262s = 4m20s)!\r\n\r\nHere's the prototype so far:\r\n\r\n```diff\r\ndiff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py\r\nindex 86eddfb..1c83ef6 100644\r\n--- a/sqlite_utils/cli.py\r\n+++ b/sqlite_utils/cli.py\r\n@@ -14,6 +14,8 @@ import io\r\n import itertools\r\n import json\r\n import os\r\n+import shutil\r\n+import subprocess\r\n import sys\r\n import csv as csv_std\r\n import tabulate\r\n@@ -1669,6 +1671,7 @@ def query(\r\n is_flag=True,\r\n help=\"Analyze resulting tables and output results\",\r\n )\r\n+@click.option(\"--fast\", is_flag=True, help=\"Fast mode, only works with CSV and TSV\")\r\n @load_extension_option\r\n def memory(\r\n paths,\r\n@@ -1692,6 +1695,7 @@ def memory(\r\n save,\r\n analyze,\r\n load_extension,\r\n+ fast,\r\n ):\r\n \"\"\"Execute SQL query against an in-memory database, optionally populated by imported data\r\n \r\n@@ -1719,6 +1723,22 @@ def memory(\r\n \\b\r\n sqlite-utils memory animals.csv --schema\r\n \"\"\"\r\n+ if fast:\r\n+ if (\r\n+ attach\r\n+ or flatten\r\n+ or param\r\n+ or encoding\r\n+ or no_detect_types\r\n+ or analyze\r\n+ or load_extension\r\n+ ):\r\n+ raise click.ClickException(\r\n+ \"--fast mode does not support any of the following options: --attach, --flatten, --param, --encoding, --no-detect-types, --analyze, --load-extension\"\r\n+ )\r\n+ # TODO: Figure out and pass other supported options\r\n+ memory_fast(paths, sql)\r\n+ return\r\n db = sqlite_utils.Database(memory=True)\r\n # If --dump or --save or --analyze used but no paths detected, assume SQL query is a path:\r\n if (dump or save or schema or analyze) and not paths:\r\n@@ -1791,6 +1811,33 @@ def memory(\r\n )\r\n \r\n \r\n+def memory_fast(paths, sql):\r\n+ if not shutil.which(\"sqlite3\"):\r\n+ raise click.ClickException(\"sqlite3 not found in PATH\")\r\n+ args = [\"sqlite3\", \":memory:\", \"-cmd\", \".mode csv\"]\r\n+ table_names = []\r\n+\r\n+ def name(path):\r\n+ base_name = pathlib.Path(path).stem or \"t\"\r\n+ table_name = base_name\r\n+ prefix = 1\r\n+ while table_name in table_names:\r\n+ prefix += 1\r\n+ table_name = \"{}_{}\".format(base_name, prefix)\r\n+ return table_name\r\n+\r\n+ for path in paths:\r\n+ table_name = name(path)\r\n+ table_names.append(table_name)\r\n+ args.extend(\r\n+ [\"-cmd\", \".import {} {}\".format(pathlib.Path(path).resolve(), table_name)]\r\n+ )\r\n+\r\n+ args.extend([\"-cmd\", \".mode column\"])\r\n+ args.append(sql)\r\n+ subprocess.run(args)\r\n+\r\n+\r\n def _execute_query(\r\n db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols\r\n ):\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/447#issuecomment-1162186856", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/447", "id": 1162186856, "node_id": "IC_kwDOCGYnMM5FRZBo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-21T18:48:46Z", "updated_at": "2022-06-21T18:48:46Z", "author_association": "OWNER", "body": "That fixed it:\r\n\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1278571700, "label": "Incorrect syntax highlighting in docs CLI reference"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1162179354", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1162179354, "node_id": "IC_kwDOCGYnMM5FRXMa", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-21T18:44:03Z", "updated_at": "2022-06-21T18:44:03Z", "author_association": "OWNER", "body": "The thing I like about that `--fast` option is that it could selectively use this alternative mechanism just for the files for which it can work (CSV and TSV files). I could also add a `--fast` option to `sqlite-utils memory` which could then kick in only for operations that involve just TSV and CSV files.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/447#issuecomment-1161869859", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/447", "id": 1161869859, "node_id": "IC_kwDOCGYnMM5FQLoj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-21T15:00:42Z", "updated_at": "2022-06-21T15:00:42Z", "author_association": "OWNER", "body": "Deploying that to https://sqlite-utils.datasette.io/en/latest/cli-reference.html#insert", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1278571700, "label": "Incorrect syntax highlighting in docs CLI reference"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/447#issuecomment-1161857806", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/447", "id": 1161857806, "node_id": "IC_kwDOCGYnMM5FQIsO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-21T14:55:51Z", "updated_at": "2022-06-21T14:58:14Z", "author_association": "OWNER", "body": "https://stackoverflow.com/a/44379513 suggests that the fix is:\r\n\r\n .. code-block:: text\r\n\r\nOr set this in `conf.py`:\r\n\r\n highlight_language = \"none\"\r\n\r\nI like that better - I don't like that all `::` blocks default to being treated as Python code.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1278571700, "label": "Incorrect syntax highlighting in docs CLI reference"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1161849874", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1161849874, "node_id": "IC_kwDOCGYnMM5FQGwS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-21T14:49:12Z", "updated_at": "2022-06-21T14:49:12Z", "author_association": "OWNER", "body": "Since there are all sorts of existing options for `sqlite-utils insert` that won't work with this, maybe it would be better to have an entirely separate command - this for example:\r\n\r\n sqlite-utils fast-insert data.db mytable data.csv ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-882052693", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 882052693, "node_id": "IC_kwDOCGYnMM40kw5V", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-18T12:57:54Z", "updated_at": "2022-06-21T13:17:15Z", "author_association": "OWNER", "body": "Another implementation option would be to use the CSV virtual table mechanism. This could avoid shelling out to the `sqlite3` binary, but requires solving the harder problem of compiling and distributing a loadable SQLite module: https://www.sqlite.org/csv.html\r\n\r\n(Would be neat to produce a Python wheel of this, see https://simonwillison.net/2022/May/23/bundling-binary-tools-in-python-wheels/)\r\n\r\nThis would also help solve the challenge of making this optimization available to the `sqlite-utils memory` command. That command operates against an in-memory database so it's not obvious how it could shell out to a binary.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1160991031", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1160991031, "node_id": "IC_kwDOCGYnMM5FM1E3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-21T00:35:20Z", "updated_at": "2022-06-21T00:35:20Z", "author_association": "OWNER", "body": "Relevant TIL: https://til.simonwillison.net/sqlite/one-line-csv-operations", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/446#issuecomment-1160798645", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/446", "id": 1160798645, "node_id": "IC_kwDOCGYnMM5FMGG1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-20T19:55:34Z", "updated_at": "2022-06-20T19:55:34Z", "author_association": "OWNER", "body": "`just` now defaults to running the tests and linters.\r\n\r\n`just test` runs the tests - it can take arguments, e.g. `just test -k transform`\r\n\r\n`just lint` runs all of the linters.\r\n\r\n`just black` applies Black.\r\n\r\nIn all case it assumes you are using `pipenv`, at least for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1277328147, "label": "Use Just to automate running tests and linters locally"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/443#issuecomment-1160794604", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/443", "id": 1160794604, "node_id": "IC_kwDOCGYnMM5FMFHs", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-20T19:49:37Z", "updated_at": "2022-06-20T19:49:37Z", "author_association": "OWNER", "body": "Also now shows up here: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-rows-from-file", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1269998342, "label": "Make `utils.rows_from_file()` a documented API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/445#issuecomment-1160794175", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/445", "id": 1160794175, "node_id": "IC_kwDOCGYnMM5FMFA_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-20T19:49:02Z", "updated_at": "2022-06-20T19:49:02Z", "author_association": "OWNER", "body": "New documentation:\r\n\r\n- https://sqlite-utils.datasette.io/en/latest/python-api.html#detecting-column-types-using-typetracker\r\n- https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-typetracker", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1277295119, "label": "`sqlite_utils.utils.TypeTracker` should be a documented API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/445#issuecomment-1160793114", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/445", "id": 1160793114, "node_id": "IC_kwDOCGYnMM5FMEwa", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-20T19:47:36Z", "updated_at": "2022-06-20T19:47:36Z", "author_association": "OWNER", "body": "I also added inline documentation and types: https://github.com/simonw/sqlite-utils/blob/773f2b6b20622bb986984a1c3161d5b3aaa1046b/sqlite_utils/utils.py#L318-L360", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1277295119, "label": "`sqlite_utils.utils.TypeTracker` should be a documented API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/445#issuecomment-1160763268", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/445", "id": 1160763268, "node_id": "IC_kwDOCGYnMM5FL9eE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-20T19:09:21Z", "updated_at": "2022-06-20T19:09:21Z", "author_association": "OWNER", "body": "Code to document: https://github.com/simonw/sqlite-utils/blob/3fbe8a784cc2f3fa0bfa8612fec9752ff9068a2b/sqlite_utils/utils.py#L318-L331", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1277295119, "label": "`sqlite_utils.utils.TypeTracker` should be a documented API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1759#issuecomment-1160717784", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1759", "id": 1160717784, "node_id": "IC_kwDOBm6k_c5FLyXY", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-06-20T18:04:46Z", "updated_at": "2022-06-20T18:04:46Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1759?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#1759](https://codecov.io/gh/simonw/datasette/pull/1759?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b901bb0) into [main](https://codecov.io/gh/simonw/datasette/commit/2e9751672d4fe329b3c359d5b7b1992283185820?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2e97516) will **not change** coverage.\n> The diff coverage is `n/a`.\n\n```diff\n@@ Coverage Diff @@\n## main #1759 +/- ##\n=======================================\n Coverage 91.67% 91.67% \n=======================================\n Files 36 36 \n Lines 4658 4658 \n=======================================\n Hits 4270 4270 \n Misses 388 388 \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1759?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1759?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2e97516...b901bb0](https://codecov.io/gh/simonw/datasette/pull/1759?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1275523220, "label": "Extract facet portions of table.html out into included templates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1759#issuecomment-1160717735", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1759", "id": 1160717735, "node_id": "IC_kwDOBm6k_c5FLyWn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-20T18:04:41Z", "updated_at": "2022-06-20T18:04:41Z", "author_association": "OWNER", "body": "I don't think this change needs any changes to the documentation: https://docs.datasette.io/en/stable/custom_templates.html#custom-templates", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1275523220, "label": "Extract facet portions of table.html out into included templates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1759#issuecomment-1160712911", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1759", "id": 1160712911, "node_id": "IC_kwDOBm6k_c5FLxLP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-20T17:58:37Z", "updated_at": "2022-06-20T17:58:37Z", "author_association": "OWNER", "body": "This is a great idea.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1275523220, "label": "Extract facet portions of table.html out into included templates"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155966234", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/444", "id": 1155966234, "node_id": "IC_kwDOCGYnMM5E5qUa", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-15T04:18:05Z", "updated_at": "2022-06-15T04:18:05Z", "author_association": "OWNER", "body": "I'm going to push a branch with my not-yet-working code (which does at least include a test).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1271426387, "label": "CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155953345", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439", "id": 1155953345, "node_id": "IC_kwDOCGYnMM5E5nLB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-15T03:53:43Z", "updated_at": "2022-06-15T03:53:43Z", "author_association": "OWNER", "body": "I tried fixing this by using `.tell()` to read the file position as I was iterating through it:\r\n```diff\r\ndiff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py\r\nindex d2ccc5f..29ad12e 100644\r\n--- a/sqlite_utils/utils.py\r\n+++ b/sqlite_utils/utils.py\r\n@@ -149,10 +149,13 @@ class UpdateWrapper:\r\n def __init__(self, wrapped, update):\r\n self._wrapped = wrapped\r\n self._update = update\r\n+ self._tell = wrapped.tell()\r\n \r\n def __iter__(self):\r\n for line in self._wrapped:\r\n- self._update(len(line))\r\n+ tell = self._wrapped.tell()\r\n+ self._update(self._tell - tell)\r\n+ self._tell = tell\r\n yield line\r\n ```\r\nThis did not work - I get this error:\r\n\r\n```\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 206, in _extra_key_strategy\r\n for row in reader:\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 156, in __iter__\r\n tell = self._wrapped.tell()\r\nOSError: telling position disabled by next() call\r\n```\r\nIt looks like you can't use `.tell()` during iteration: https://stackoverflow.com/questions/29618936/how-to-solve-oserror-telling-position-disabled-by-next-call", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250495688, "label": "Misleading progress bar against utf-16-le CSV input"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155815186", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/444", "id": 1155815186, "node_id": "IC_kwDOCGYnMM5E5FcS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T23:48:16Z", "updated_at": "2022-06-14T23:48:16Z", "author_association": "OWNER", "body": "This is tricky to implement because of this code: https://github.com/simonw/sqlite-utils/blob/b8af3b96f5c72317cc8783dc296a94f6719987d9/sqlite_utils/cli.py#L938-L945\r\n\r\nIt's reconstructing each document using the known headers here:\r\n\r\n`docs = (dict(zip(headers, row)) for row in reader)`\r\n\r\nSo my first attempt at this - the diff here - did not have the desired result:\r\n\r\n```diff\r\ndiff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py\r\nindex 86eddfb..00b920b 100644\r\n--- a/sqlite_utils/cli.py\r\n+++ b/sqlite_utils/cli.py\r\n@@ -6,7 +6,7 @@ import hashlib\r\n import pathlib\r\n import sqlite_utils\r\n from sqlite_utils.db import AlterError, BadMultiValues, DescIndex\r\n-from sqlite_utils.utils import maximize_csv_field_size_limit\r\n+from sqlite_utils.utils import maximize_csv_field_size_limit, _extra_key_strategy\r\n from sqlite_utils import recipes\r\n import textwrap\r\n import inspect\r\n@@ -797,6 +797,15 @@ _import_options = (\r\n \"--encoding\",\r\n help=\"Character encoding for input, defaults to utf-8\",\r\n ),\r\n+ click.option(\r\n+ \"--ignore-extras\",\r\n+ is_flag=True,\r\n+ help=\"If a CSV line has more than the expected number of values, ignore the extras\",\r\n+ ),\r\n+ click.option(\r\n+ \"--extras-key\",\r\n+ help=\"If a CSV line has more than the expected number of values put them in a list in this column\",\r\n+ ),\r\n )\r\n \r\n \r\n@@ -885,6 +894,8 @@ def insert_upsert_implementation(\r\n sniff,\r\n no_headers,\r\n encoding,\r\n+ ignore_extras,\r\n+ extras_key,\r\n batch_size,\r\n alter,\r\n upsert,\r\n@@ -909,6 +920,10 @@ def insert_upsert_implementation(\r\n raise click.ClickException(\"--flatten cannot be used with --csv or --tsv\")\r\n if encoding and not (csv or tsv):\r\n raise click.ClickException(\"--encoding must be used with --csv or --tsv\")\r\n+ if ignore_extras and extras_key:\r\n+ raise click.ClickException(\r\n+ \"--ignore-extras and --extras-key cannot be used together\"\r\n+ )\r\n if pk and len(pk) == 1:\r\n pk = pk[0]\r\n encoding = encoding or \"utf-8-sig\"\r\n@@ -935,7 +950,9 @@ def insert_upsert_implementation(\r\n csv_reader_args[\"delimiter\"] = delimiter\r\n if quotechar:\r\n csv_reader_args[\"quotechar\"] = quotechar\r\n- reader = csv_std.reader(decoded, **csv_reader_args)\r\n+ reader = _extra_key_strategy(\r\n+ csv_std.reader(decoded, **csv_reader_args), ignore_extras, extras_key\r\n+ )\r\n first_row = next(reader)\r\n if no_headers:\r\n headers = [\"untitled_{}\".format(i + 1) for i in range(len(first_row))]\r\n@@ -1101,6 +1118,8 @@ def insert(\r\n sniff,\r\n no_headers,\r\n encoding,\r\n+ ignore_extras,\r\n+ extras_key,\r\n batch_size,\r\n alter,\r\n detect_types,\r\n@@ -1176,6 +1195,8 @@ def insert(\r\n sniff,\r\n no_headers,\r\n encoding,\r\n+ ignore_extras,\r\n+ extras_key,\r\n batch_size,\r\n alter=alter,\r\n upsert=False,\r\n@@ -1214,6 +1235,8 @@ def upsert(\r\n sniff,\r\n no_headers,\r\n encoding,\r\n+ ignore_extras,\r\n+ extras_key,\r\n alter,\r\n not_null,\r\n default,\r\n@@ -1254,6 +1277,8 @@ def upsert(\r\n sniff,\r\n no_headers,\r\n encoding,\r\n+ ignore_extras,\r\n+ extras_key,\r\n batch_size,\r\n alter=alter,\r\n upsert=True,\r\n@@ -1297,6 +1322,8 @@ def bulk(\r\n sniff,\r\n no_headers,\r\n encoding,\r\n+ ignore_extras,\r\n+ extras_key,\r\n load_extension,\r\n ):\r\n \"\"\"\r\n@@ -1331,6 +1358,8 @@ def bulk(\r\n sniff=sniff,\r\n no_headers=no_headers,\r\n encoding=encoding,\r\n+ ignore_extras=ignore_extras,\r\n+ extras_key=extras_key,\r\n batch_size=batch_size,\r\n alter=False,\r\n upsert=False,\r\n\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1271426387, "label": "CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155804591", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/444", "id": 1155804591, "node_id": "IC_kwDOCGYnMM5E5C2v", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T23:28:36Z", "updated_at": "2022-06-14T23:28:36Z", "author_association": "OWNER", "body": "I'm going with `--extras-key` and `--ignore-extras` as the two new options.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1271426387, "label": "CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155804459", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/444", "id": 1155804459, "node_id": "IC_kwDOCGYnMM5E5C0r", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T23:28:18Z", "updated_at": "2022-06-14T23:28:18Z", "author_association": "OWNER", "body": "I think these become part of the `_import_options` list which is used in a few places:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/b8af3b96f5c72317cc8783dc296a94f6719987d9/sqlite_utils/cli.py#L765-L800", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1271426387, "label": "CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/430#issuecomment-1155803262", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/430", "id": 1155803262, "node_id": "IC_kwDOCGYnMM5E5Ch-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T23:26:11Z", "updated_at": "2022-06-14T23:26:11Z", "author_association": "OWNER", "body": "It looks like `PRAGMA temp_store` was the right option to use here: https://www.sqlite.org/pragma.html#pragma_temp_store\r\n\r\n`temp_store_directory` is listed as deprecated here: https://www.sqlite.org/pragma.html#pragma_temp_store_directory\r\n\r\nI'm going to turn this into a help-wanted documentation issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1224112817, "label": "Document how to use `PRAGMA temp_store` to avoid errors when running VACUUM against huge databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/434#issuecomment-1155801812", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/434", "id": 1155801812, "node_id": "IC_kwDOCGYnMM5E5CLU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T23:23:32Z", "updated_at": "2022-06-14T23:23:32Z", "author_association": "OWNER", "body": "Since table names can be quoted like this:\r\n```sql\r\nCREATE VIRTUAL TABLE \"searchable_fts\"\r\n USING FTS4 (text1, text2, [name with . and spaces], content=\"searchable\")\r\n```\r\nOR like this:\r\n```sql\r\nCREATE VIRTUAL TABLE \"searchable_fts\"\r\n USING FTS4 (text1, text2, [name with . and spaces], content=[searchable])\r\n```\r\nThis fix looks to be correct to me (copying from the updated `test_with_trace()` test):\r\n\r\n```python\r\n (\r\n \"SELECT name FROM sqlite_master\\n\"\r\n \" WHERE rootpage = 0\\n\"\r\n \" AND (\\n\"\r\n \" sql LIKE :like\\n\"\r\n \" OR sql LIKE :like2\\n\"\r\n \" OR (\\n\"\r\n \" tbl_name = :table\\n\"\r\n \" AND sql LIKE '%VIRTUAL TABLE%USING FTS%'\\n\"\r\n \" )\\n\"\r\n \" )\",\r\n {\r\n \"like\": \"%VIRTUAL TABLE%USING FTS%content=[dogs]%\",\r\n \"like2\": '%VIRTUAL TABLE%USING FTS%content=\"dogs\"%',\r\n \"table\": \"dogs\",\r\n },\r\n )\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243151184, "label": "`detect_fts()` identifies the wrong table if tables have names that are subsets of each other"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/434#issuecomment-1155794149", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/434", "id": 1155794149, "node_id": "IC_kwDOCGYnMM5E5ATl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T23:09:54Z", "updated_at": "2022-06-14T23:09:54Z", "author_association": "OWNER", "body": "A test that demonstrates the problem:\r\n```python\r\n@pytest.mark.parametrize(\"reverse_order\", (True, False))\r\ndef test_detect_fts_similar_tables(fresh_db, reverse_order):\r\n # https://github.com/simonw/sqlite-utils/issues/434\r\n table1, table2 = (\"demo\", \"demo2\")\r\n if reverse_order:\r\n table1, table2 = table2, table1\r\n\r\n fresh_db[table1].insert({\"title\": \"Hello\"}).enable_fts(\r\n [\"title\"], fts_version=\"FTS4\"\r\n )\r\n fresh_db[table2].insert({\"title\": \"Hello\"}).enable_fts(\r\n [\"title\"], fts_version=\"FTS4\"\r\n )\r\n assert fresh_db[table1].detect_fts() == \"{}_fts\".format(table1)\r\n assert fresh_db[table2].detect_fts() == \"{}_fts\".format(table2)\r\n```\r\nThe order matters - so this test currently passes in one direction and fails in the other:\r\n```\r\n> assert fresh_db[table2].detect_fts() == \"{}_fts\".format(table2)\r\nE AssertionError: assert 'demo2_fts' == 'demo_fts'\r\nE - demo_fts\r\nE + demo2_fts\r\nE ? +\r\n\r\ntests/test_introspect.py:53: AssertionError\r\n========================================================================================= short test summary info =========================================================================================\r\nFAILED tests/test_introspect.py::test_detect_fts_similar_tables[True] - AssertionError: assert 'demo2_fts' == 'demo_fts'\r\n=============================================================================== 1 failed, 1 passed, 855 deselected in 1.00s ===============================================================================\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243151184, "label": "`detect_fts()` identifies the wrong table if tables have names that are subsets of each other"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/434#issuecomment-1155791109", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/434", "id": 1155791109, "node_id": "IC_kwDOCGYnMM5E4_kF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T23:04:40Z", "updated_at": "2022-06-14T23:04:40Z", "author_association": "OWNER", "body": "Definitely a bug - thanks for the detailed write-up!\r\n\r\nYou're right, the code at fault is here:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L2213-L2231", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243151184, "label": "`detect_fts()` identifies the wrong table if tables have names that are subsets of each other"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155789101", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439", "id": 1155789101, "node_id": "IC_kwDOCGYnMM5E4_Et", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T23:00:45Z", "updated_at": "2022-06-14T23:00:45Z", "author_association": "OWNER", "body": "I'm going to mark this as \"help wanted\" and leave it open. I'm glad that it's not actually a bug where errors get swallowed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250495688, "label": "Misleading progress bar against utf-16-le CSV input"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155788944", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439", "id": 1155788944, "node_id": "IC_kwDOCGYnMM5E4_CQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T23:00:24Z", "updated_at": "2022-06-14T23:00:24Z", "author_association": "OWNER", "body": "The progress bar only works if the file-like object passed to it has a `fp.fileno()` that isn't 0 (for stdin) - that's how it detects that the file is something which it can measure the size of in order to show progress.\r\n\r\nIf we know the file size in bytes AND we know the character encoding, can we change `UpdateWrapper` to update the number of bytes-per-character instead?\r\n\r\nI don't think so: I can't see a way of definitively saying \"for this encoding the number of bytes per character is X\" - and in fact I'm pretty sure that question doesn't even make sense since variable-length encodings exist.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250495688, "label": "Misleading progress bar against utf-16-le CSV input"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155784284", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439", "id": 1155784284, "node_id": "IC_kwDOCGYnMM5E495c", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:51:03Z", "updated_at": "2022-06-14T22:52:13Z", "author_association": "OWNER", "body": "Yes, this is the problem. The progress bar length is set to the length in bytes of the file - `os.path.getsize(file.name)` - but it's then incremented by the length of each DECODED line in turn.\r\n\r\nSo if the file is in `utf-16-le` (twice the size of `utf-8`) the progress bar will finish at 50%!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250495688, "label": "Misleading progress bar against utf-16-le CSV input"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155782835", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439", "id": 1155782835, "node_id": "IC_kwDOCGYnMM5E49iz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:48:22Z", "updated_at": "2022-06-14T22:49:53Z", "author_association": "OWNER", "body": "Here's the code that implements the progress bar in question: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/cli.py#L918-L932\r\n\r\nIt calls `file_progress()` which looks like this:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/utils.py#L159-L175\r\n\r\nWhich uses this:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/utils.py#L148-L156", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250495688, "label": "Misleading progress bar against utf-16-le CSV input"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155781399", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439", "id": 1155781399, "node_id": "IC_kwDOCGYnMM5E49MX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:45:41Z", "updated_at": "2022-06-14T22:45:41Z", "author_association": "OWNER", "body": "TIL how to use `iconv`: https://til.simonwillison.net/linux/iconv", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250495688, "label": "Misleading progress bar against utf-16-le CSV input"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155776023", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439", "id": 1155776023, "node_id": "IC_kwDOCGYnMM5E474X", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:36:07Z", "updated_at": "2022-06-14T22:36:07Z", "author_association": "OWNER", "body": "Wait! The arguments in that are the wrong way round. This is correct:\r\n\r\n sqlite-utils insert --csv --delimiter \";\" --encoding \"utf-16-le\" test.db test csv\r\n\r\nIt still outputs the following:\r\n\r\n [------------------------------------] 0%\r\n [#################-------------------] 49% 00:00:02%\r\n\r\nBut it creates a `test.db` file that is 6.2MB.\r\n\r\nThat database has 3141 rows in it:\r\n\r\n```\r\n% sqlite-utils tables test.db --counts -t\r\ntable count\r\n------- -------\r\ntest 3142\r\n```\r\nI converted that `csv` file to utf-8 like so:\r\n\r\n iconv -f UTF-16LE -t UTF-8 csv > utf8.csv\r\n\r\nAnd it contains 3142 lines:\r\n```\r\n% wc -l utf8.csv \r\n 3142 utf8.csv\r\n```\r\nSo my hunch here is that the problem is actually that the progress bar doesn't know how to correctly measure files in `utf-16-le` encoding!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250495688, "label": "Misleading progress bar against utf-16-le CSV input"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155772244", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439", "id": 1155772244, "node_id": "IC_kwDOCGYnMM5E469U", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:30:03Z", "updated_at": "2022-06-14T22:30:03Z", "author_association": "OWNER", "body": "Tried this:\r\n```\r\n% python -i $(which sqlite-utils) insert --csv --delimiter \";\" --encoding \"utf-16-le\" test test.db csv\r\n [------------------------------------] 0%\r\n [#################-------------------] 49% 00:00:01Traceback (most recent call last):\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1072, in main\r\n ctx.exit()\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 692, in exit\r\n raise Exit(code)\r\nclick.exceptions.Exit: 0\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils\", line 33, in \r\n sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1090, in main\r\n sys.exit(e.exit_code)\r\nSystemExit: 0\r\n>>> \r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250495688, "label": "Misleading progress bar against utf-16-le CSV input"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155771462", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439", "id": 1155771462, "node_id": "IC_kwDOCGYnMM5E46xG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:28:38Z", "updated_at": "2022-06-14T22:28:38Z", "author_association": "OWNER", "body": "Maybe this isn't a CSV field value problem - I tried this patch and didn't seem to hit the new breakpoints:\r\n```diff\r\ndiff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py\r\nindex d2ccc5f..f1b823a 100644\r\n--- a/sqlite_utils/utils.py\r\n+++ b/sqlite_utils/utils.py\r\n@@ -204,13 +204,17 @@ def _extra_key_strategy(\r\n # DictReader adds a 'None' key with extra row values\r\n if None not in row:\r\n yield row\r\n- elif ignore_extras:\r\n+ continue\r\n+ else:\r\n+ breakpoint()\r\n+ if ignore_extras:\r\n # ignoring row.pop(none) because of this issue:\r\n # https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155358637\r\n row.pop(None) # type: ignore\r\n yield row\r\n elif not extras_key:\r\n extras = row.pop(None) # type: ignore\r\n+ breakpoint()\r\n raise RowError(\r\n \"Row {} contained these extra values: {}\".format(row, extras)\r\n )\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250495688, "label": "Misleading progress bar against utf-16-le CSV input"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155769216", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439", "id": 1155769216, "node_id": "IC_kwDOCGYnMM5E46OA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:24:49Z", "updated_at": "2022-06-14T22:25:06Z", "author_association": "OWNER", "body": "I have a hunch that this crash may be caused by a CSV value which is too long, as addressed at the library level in:\r\n- #440\r\n\r\nBut not yet addressed in the CLI tool, see:\r\n\r\n- #444\r\n\r\nEither way though, I really don't like that errors like this are swallowed!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250495688, "label": "Misleading progress bar against utf-16-le CSV input"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155767915", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1155767915, "node_id": "IC_kwDOCGYnMM5E455r", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:22:27Z", "updated_at": "2022-06-14T22:22:27Z", "author_association": "OWNER", "body": "I forgot to add equivalents of `extras_key=` and `ignore_extras=` to the CLI tool - will do that in a separate issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155767202", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439", "id": 1155767202, "node_id": "IC_kwDOCGYnMM5E45ui", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:21:10Z", "updated_at": "2022-06-14T22:21:10Z", "author_association": "OWNER", "body": "I can't figure out why that error is being swallowed like that. The most likely culprit was this code: \r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/cli.py#L1021-L1043\r\n\r\nBut I tried changing it like this:\r\n\r\n```diff\r\ndiff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py\r\nindex 86eddfb..ed26fdd 100644\r\n--- a/sqlite_utils/cli.py\r\n+++ b/sqlite_utils/cli.py\r\n@@ -1023,6 +1023,7 @@ def insert_upsert_implementation(\r\n docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs\r\n )\r\n except Exception as e:\r\n+ raise\r\n if (\r\n isinstance(e, sqlite3.OperationalError)\r\n and e.args\r\n```\r\nAnd your steps to reproduce still got to 49% and then failed silently.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250495688, "label": "Misleading progress bar against utf-16-le CSV input"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155764428", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/432", "id": 1155764428, "node_id": "IC_kwDOCGYnMM5E45DM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:16:21Z", "updated_at": "2022-06-14T22:16:21Z", "author_association": "OWNER", "body": "Initial idea of how the `.table()` method would change:\r\n```diff\r\ndiff --git a/sqlite_utils/db.py b/sqlite_utils/db.py\r\nindex 7a06304..3ecb40b 100644\r\n--- a/sqlite_utils/db.py\r\n+++ b/sqlite_utils/db.py\r\n@@ -474,11 +474,12 @@ class Database:\r\n self._tracer(sql, None)\r\n return self.conn.executescript(sql)\r\n \r\n- def table(self, table_name: str, **kwargs) -> Union[\"Table\", \"View\"]:\r\n+ def table(self, table_name: str, alias: Optional[str] = None, **kwargs) -> Union[\"Table\", \"View\"]:\r\n \"\"\"\r\n Return a table object, optionally configured with default options.\r\n \r\n :param table_name: Name of the table\r\n+ :param alias: The database alias to use, if referring to a table in another connected database\r\n \"\"\"\r\n klass = View if table_name in self.view_names() else Table\r\n return klass(self, table_name, **kwargs)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1236693079, "label": "Support `rows_where()`, `delete_where()` etc for attached alias databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155764064", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/432", "id": 1155764064, "node_id": "IC_kwDOCGYnMM5E449g", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:15:44Z", "updated_at": "2022-06-14T22:15:44Z", "author_association": "OWNER", "body": "Implementing this would be a pretty big change - initial instinct is that I'd need to introduce a `self.alias` property to `Queryable` (the subclass of `Table` and `View`) and a new `self.name_with_alias` getter which returns `alias.tablename` if `alias` is set to a not-None value. Then I'd need to rewrite every piece of code like this:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L1161\r\n\r\nTo look like this instead:\r\n```python\r\n sql = \"select {} from [{}]\".format(select, self.name_with_alias)\r\n```\r\nBut some parts would be harder - for example:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L1227-L1231\r\n\r\nWould have to know to query `alias.sqlite_master` instead.\r\n\r\nThe cached table counts logic like this would need a bunch of changes too:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L644-L657", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1236693079, "label": "Support `rows_where()`, `delete_where()` etc for attached alias databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155759857", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/432", "id": 1155759857, "node_id": "IC_kwDOCGYnMM5E437x", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:09:07Z", "updated_at": "2022-06-14T22:09:07Z", "author_association": "OWNER", "body": "Third option, and I think the one I like the best:\r\n```python\r\nrows = db.table(\"tablename\", alias=\"otherdb\").rows_where(alias=\"otherdb\")\r\n```\r\nThe `db.table(tablename)` method already exists as an alternative to `db[tablename]`: https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-table-configuration\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1236693079, "label": "Support `rows_where()`, `delete_where()` etc for attached alias databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155758664", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/432", "id": 1155758664, "node_id": "IC_kwDOCGYnMM5E43pI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:07:50Z", "updated_at": "2022-06-14T22:07:50Z", "author_association": "OWNER", "body": "Another potential fix: add a `alias=` parameter to `rows_where()` and other similar methods. Then you could do this:\r\n\r\n```python\r\nrows = db[\"tablename\"].rows_where(alias=\"otherdb\")\r\n```\r\nThis feels wrong to me: `db[\"tablename\"]` is the bit that is supposed to return a table object. Having part of what that table object is exist as a parameter to other methods is confusing.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1236693079, "label": "Support `rows_where()`, `delete_where()` etc for attached alias databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155756742", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/432", "id": 1155756742, "node_id": "IC_kwDOCGYnMM5E43LG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:05:38Z", "updated_at": "2022-06-14T22:05:49Z", "author_association": "OWNER", "body": "I don't like the idea of `table_names()` returning names of tables from connected databases as well, because it feels like it could lead to surprising behaviour - especially if those connected databases turn to have table names that are duplicated in the main connected database.\r\n\r\nIt would be neat if functions like `.rows_where()` worked though.\r\n\r\nOne thought would be to support something like this:\r\n```python\r\nrows = db[\"otherdb.tablename\"].rows_where()\r\n```\r\nBut... `.` is a valid character in a SQLite table name. So `\"otherdb.tablename\"` might ambiguously refer to a table called `tablename` in a connected database with the alias `otherdb`, OR a table in the current database with the name `otherdb.tablename`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1236693079, "label": "Support `rows_where()`, `delete_where()` etc for attached alias databases"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/431#issuecomment-1155753397", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/431", "id": 1155753397, "node_id": "IC_kwDOCGYnMM5E42W1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T22:01:38Z", "updated_at": "2022-06-14T22:01:38Z", "author_association": "OWNER", "body": "Yeah, I think it would be neat if the library could support self-referential many-to-many in a nice way.\r\n\r\nI'm not sure about the `left_name/right_name` design though. Would it be possible to have this work as the user intends, by spotting that the other table name `\"people\"` matches the name of the current table?\r\n\r\n```python\r\ndb[\"people\"].insert({\"name\": \"Mary\"}, pk=\"name\").m2m(\r\n \"people\", [{\"name\": \"Michael\"}, {\"name\": \"Suzy\"}], m2m_table=\"parent_child\", pk=\"name\"\r\n)\r\n```\r\nThe created table could look like this:\r\n```sql\r\nCREATE TABLE [parent_child] (\r\n [people_id_1] TEXT REFERENCES [people]([name]),\r\n [people_id_2] TEXT REFERENCES [people]([name]),\r\n PRIMARY KEY ([people_id_1], [people_id_2])\r\n)\r\n```\r\nI've not thought very hard about this, so the design I'm proposing here might not work.\r\n\r\nAre there other reasons people might wan the `left_name=` and `right_name=` parameters? If so then I'm much happier with those.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1227571375, "label": "Allow making m2m relation of a table to itself"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1155750270", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/441", "id": 1155750270, "node_id": "IC_kwDOCGYnMM5E41l-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T21:57:57Z", "updated_at": "2022-06-14T21:57:57Z", "author_association": "OWNER", "body": "I added `where=` and `where_args=` parameters to that `.search()` method - updated documentation is here: https://sqlite-utils.datasette.io/en/latest/python-api.html#searching-with-table-search", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1257724585, "label": "Combining `rows_where()` and `search()` to limit which rows are searched"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1155749696", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/433", "id": 1155749696, "node_id": "IC_kwDOCGYnMM5E41dA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T21:57:05Z", "updated_at": "2022-06-14T21:57:05Z", "author_association": "OWNER", "body": "Marking this as help wanted because I can't figure out how to replicate it!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239034903, "label": "CLI eats my cursor"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/442#issuecomment-1155748444", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/442", "id": 1155748444, "node_id": "IC_kwDOCGYnMM5E41Jc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T21:55:15Z", "updated_at": "2022-06-14T21:55:15Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#setting-the-maximum-csv-field-size-limit", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1269886084, "label": "`maximize_csv_field_size_limit()` utility function"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/442#issuecomment-1155714131", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/442", "id": 1155714131, "node_id": "IC_kwDOCGYnMM5E4sxT", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T21:07:50Z", "updated_at": "2022-06-14T21:07:50Z", "author_association": "OWNER", "body": "Here's the commit where I added that originally, including a test: https://github.com/simonw/sqlite-utils/commit/1a93b72ba710ea2271eaabc204685a27d2469374", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1269886084, "label": "`maximize_csv_field_size_limit()` utility function"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155672675", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1155672675, "node_id": "IC_kwDOCGYnMM5E4ipj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T20:19:07Z", "updated_at": "2022-06-14T20:19:07Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 1, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/443#issuecomment-1155672522", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/443", "id": 1155672522, "node_id": "IC_kwDOCGYnMM5E4inK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T20:18:58Z", "updated_at": "2022-06-14T20:18:58Z", "author_association": "OWNER", "body": "New documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1269998342, "label": "Make `utils.rows_from_file()` a documented API"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155666672", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1155666672, "node_id": "IC_kwDOCGYnMM5E4hLw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T20:11:52Z", "updated_at": "2022-06-14T20:11:52Z", "author_association": "OWNER", "body": "I'm going to rename `restkey` to `extras_key` for consistency with `ignore_extras`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1155515426", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/441", "id": 1155515426, "node_id": "IC_kwDOCGYnMM5E38Qi", "user": {"value": 1448859, "label": "betatim"}, "created_at": "2022-06-14T17:53:43Z", "updated_at": "2022-06-14T17:53:43Z", "author_association": "NONE", "body": "That would be handy (additional where filters) but I think the trick with the `with` statement is already an order of magnitude better than what I had thought of, so my problem is solved by it (plus I got to learn about `with` today!)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1257724585, "label": "Combining `rows_where()` and `search()` to limit which rows are searched"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1155421299", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/441", "id": 1155421299, "node_id": "IC_kwDOCGYnMM5E3lRz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T16:23:52Z", "updated_at": "2022-06-14T16:23:52Z", "author_association": "OWNER", "body": "Actually I have a thought for something that could help here: I could add a mechanism for inserting additional where filters and parameters into that `.search()` method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1257724585, "label": "Combining `rows_where()` and `search()` to limit which rows are searched"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155389614", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1155389614, "node_id": "IC_kwDOCGYnMM5E3diu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T15:54:03Z", "updated_at": "2022-06-14T15:54:03Z", "author_association": "OWNER", "body": "Filed an issue against `python/typeshed`:\r\n\r\n- https://github.com/python/typeshed/issues/8075", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1155364367", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/412", "id": 1155364367, "node_id": "IC_kwDOCGYnMM5E3XYP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T15:36:28Z", "updated_at": "2022-06-14T15:36:28Z", "author_association": "OWNER", "body": "Here's as far as I got with my initial prototype, in `sqlite_utils/pandas.py`:\r\n\r\n```python\r\nfrom .db import Database as _Database, Table as _Table, View as _View\r\nimport pandas as pd\r\nfrom typing import (\r\n Iterable,\r\n Union,\r\n Optional,\r\n)\r\n\r\n\r\nclass Database(_Database):\r\n def query(\r\n self, sql: str, params: Optional[Union[Iterable, dict]] = None\r\n ) -> pd.DataFrame:\r\n return pd.DataFrame(super().query(sql, params))\r\n\r\n def table(self, table_name: str, **kwargs) -> Union[\"Table\", \"View\"]:\r\n \"Return a table object, optionally configured with default options.\"\r\n klass = View if table_name in self.view_names() else Table\r\n return klass(self, table_name, **kwargs)\r\n\r\n\r\nclass PandasQueryable:\r\n def rows_where(\r\n self,\r\n where: str = None,\r\n where_args: Optional[Union[Iterable, dict]] = None,\r\n order_by: str = None,\r\n select: str = \"*\",\r\n limit: int = None,\r\n offset: int = None,\r\n ) -> pd.DataFrame:\r\n return pd.DataFrame(\r\n super().rows_where(\r\n where,\r\n where_args,\r\n order_by=order_by,\r\n select=select,\r\n limit=limit,\r\n offset=offset,\r\n )\r\n )\r\n\r\n\r\nclass Table(PandasQueryable, _Table):\r\n pass\r\n\r\n\r\nclass View(PandasQueryable, _View):\r\n pass\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1160182768, "label": "Optional Pandas integration"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155358637", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1155358637, "node_id": "IC_kwDOCGYnMM5E3V-t", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T15:31:34Z", "updated_at": "2022-06-14T15:31:34Z", "author_association": "OWNER", "body": "Getting this past `mypy` is really hard!\r\n\r\n```\r\n% mypy sqlite_utils\r\nsqlite_utils/utils.py:189: error: No overload variant of \"pop\" of \"MutableMapping\" matches argument type \"None\"\r\nsqlite_utils/utils.py:189: note: Possible overload variants:\r\nsqlite_utils/utils.py:189: note: def pop(self, key: str) -> str\r\nsqlite_utils/utils.py:189: note: def [_T] pop(self, key: str, default: Union[str, _T] = ...) -> Union[str, _T]\r\n```\r\nThat's because of this line:\r\n\r\n row.pop(key=None)\r\n\r\nWhich is legit here - we have a dictionary where one of the keys is `None` and we want to remove that key. But the baked in type is apparently `def pop(self, key: str) -> str`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155350755", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1155350755, "node_id": "IC_kwDOCGYnMM5E3UDj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T15:25:18Z", "updated_at": "2022-06-14T15:25:18Z", "author_association": "OWNER", "body": "That broke `mypy`:\r\n\r\n`sqlite_utils/utils.py:229: error: Incompatible types in assignment (expression has type \"Iterable[Dict[Any, Any]]\", variable has type \"DictReader[str]\")`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155317293", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1155317293, "node_id": "IC_kwDOCGYnMM5E3L4t", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T15:04:01Z", "updated_at": "2022-06-14T15:04:01Z", "author_association": "OWNER", "body": "I think that's unavoidable: it looks like `csv.Sniffer` only works if you feed it a CSV file with an equal number of values in each row, which is understandable.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155310521", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1155310521, "node_id": "IC_kwDOCGYnMM5E3KO5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-14T14:58:50Z", "updated_at": "2022-06-14T14:58:50Z", "author_association": "OWNER", "body": "Interesting challenge in writing tests for this: if you give `csv.Sniffer` a short example with an invalid row in it sometimes it picks the wrong delimiter!\r\n\r\n id,name\\r\\n1,Cleo,oops\r\n\r\nIt decided the delimiter there was `e`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154475454", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1154475454, "node_id": "IC_kwDOCGYnMM5Ez-W-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T21:52:03Z", "updated_at": "2022-06-13T21:52:03Z", "author_association": "OWNER", "body": "The exception will be called `RowError`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154474482", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1154474482, "node_id": "IC_kwDOCGYnMM5Ez-Hy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T21:50:59Z", "updated_at": "2022-06-13T21:51:24Z", "author_association": "OWNER", "body": "Decision: I'm going to default to raising an exception if a row has too many values in it.\r\n\r\nYou'll be able to pass `ignore_extras=True` to ignore those extra values, or pass `restkey=\"the_rest\"` to stick them in a list in the `restkey` column.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154457893", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1154457893, "node_id": "IC_kwDOCGYnMM5Ez6El", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T21:29:02Z", "updated_at": "2022-06-13T21:29:02Z", "author_association": "OWNER", "body": "Here's the current function signature for `rows_from_file()`:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/26e6d2622c57460a24ffdd0128bbaac051d51a5f/sqlite_utils/utils.py#L174-L179", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154457028", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1154457028, "node_id": "IC_kwDOCGYnMM5Ez53E", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T21:28:03Z", "updated_at": "2022-06-13T21:28:03Z", "author_association": "OWNER", "body": "Whatever I decide, I can implement it in `rows_from_file()`, maybe as an optional parameter - then decide how to call it from the `sqlite-utils insert` CLI (perhaps with a new option there too).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154456183", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1154456183, "node_id": "IC_kwDOCGYnMM5Ez5p3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T21:26:55Z", "updated_at": "2022-06-13T21:26:55Z", "author_association": "OWNER", "body": "So I need to make a design decision here: what should `sqlite-utils` do with CSV files that have rows with more values than there are headings?\r\n\r\nSome options:\r\n\r\n- Ignore those extra fields entirely - silently drop that data. I'm not keen on this.\r\n- Throw an error. The library does this already, but the error is incomprehensible - it could turn into a useful, human-readable error instead.\r\n- Put the data in a JSON list in a column with a known name (`None` is not a valid column name, so not that). This could be something like `_restkey` or `_values_with_no_heading`. This feels like a better option, but I'd need to carefully pick a name for it - and come up with an answer for the question of what to do if the CSV file being important already uses that heading name for something else.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154454127", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1154454127, "node_id": "IC_kwDOCGYnMM5Ez5Jv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T21:24:18Z", "updated_at": "2022-06-13T21:24:18Z", "author_association": "OWNER", "body": "That weird behaviour is documented here: https://docs.python.org/3/library/csv.html#csv.DictReader\r\n\r\n> If a row has more fields than fieldnames, the remaining data is put in a list and stored with the fieldname specified by *restkey* (which defaults to `None`). If a non-blank row has fewer fields than fieldnames, the missing values are filled-in with the value of *restval* (which defaults to `None`).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154453319", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1154453319, "node_id": "IC_kwDOCGYnMM5Ez49H", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T21:23:16Z", "updated_at": "2022-06-13T21:23:16Z", "author_association": "OWNER", "body": "Aha! I think I see what's happening here. Here's what `DictReader` does if one of the lines has too many items in it:\r\n\r\n```pycon\r\n>>> import csv, io\r\n>>> list(csv.DictReader(io.StringIO(\"id,name\\n1,Cleo,nohead\\n2,Barry\")))\r\n[{'id': '1', 'name': 'Cleo', None: ['nohead']}, {'id': '2', 'name': 'Barry'}]\r\n```\r\nSee how that row with too many items gets this:\r\n`[{'id': '1', 'name': 'Cleo', None: ['nohead']}`\r\n\r\nThat's a `None` for the key and (weirdly) a list containing the single item for the value!\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154449442", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1154449442, "node_id": "IC_kwDOCGYnMM5Ez4Ai", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T21:18:26Z", "updated_at": "2022-06-13T21:20:12Z", "author_association": "OWNER", "body": "Here are full steps to replicate the bug:\r\n```python\r\nfrom urllib.request import urlopen\r\nimport sqlite_utils\r\ndb = sqlite_utils.Database(memory=True)\r\nwith urlopen(\"https://artsdatabanken.no/Fab2018/api/export/csv\") as fab:\r\n reader, other = sqlite_utils.utils.rows_from_file(fab, encoding=\"utf-16le\")\r\n db[\"fab2018\"].insert_all(reader, pk=\"Id\")\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154396400", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1154396400, "node_id": "IC_kwDOCGYnMM5EzrDw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T20:28:25Z", "updated_at": "2022-06-13T20:28:25Z", "author_association": "OWNER", "body": "Fixing that `key` thing (to ignore any key that is `None`) revealed a new bug:\r\n\r\n```\r\nFile ~/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py:376, in hash_record(record, keys)\r\n 373 if keys is not None:\r\n 374 to_hash = {key: record[key] for key in keys}\r\n 375 return hashlib.sha1(\r\n--> 376 json.dumps(to_hash, separators=(\",\", \":\"), sort_keys=True, default=repr).encode(\r\n 377 \"utf8\"\r\n 378 )\r\n 379 ).hexdigest()\r\n\r\nFile ~/.pyenv/versions/3.8.2/lib/python3.8/json/__init__.py:234, in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)\r\n 232 if cls is None:\r\n 233 cls = JSONEncoder\r\n--> 234 return cls(\r\n 235 skipkeys=skipkeys, ensure_ascii=ensure_ascii,\r\n 236 check_circular=check_circular, allow_nan=allow_nan, indent=indent,\r\n 237 separators=separators, default=default, sort_keys=sort_keys,\r\n 238 **kw).encode(obj)\r\n\r\nFile ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:199, in JSONEncoder.encode(self, o)\r\n 195 return encode_basestring(o)\r\n 196 # This doesn't pass the iterator directly to ''.join() because the\r\n 197 # exceptions aren't as detailed. The list call should be roughly\r\n 198 # equivalent to the PySequence_Fast that ''.join() would do.\r\n--> 199 chunks = self.iterencode(o, _one_shot=True)\r\n 200 if not isinstance(chunks, (list, tuple)):\r\n 201 chunks = list(chunks)\r\n\r\nFile ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:257, in JSONEncoder.iterencode(self, o, _one_shot)\r\n 252 else:\r\n 253 _iterencode = _make_iterencode(\r\n 254 markers, self.default, _encoder, self.indent, floatstr,\r\n 255 self.key_separator, self.item_separator, self.sort_keys,\r\n 256 self.skipkeys, _one_shot)\r\n--> 257 return _iterencode(o, 0)\r\n\r\nTypeError: '<' not supported between instances of 'NoneType' and 'str'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154387591", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1154387591, "node_id": "IC_kwDOCGYnMM5Ezo6H", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T20:17:51Z", "updated_at": "2022-06-13T20:17:51Z", "author_association": "OWNER", "body": "I don't understand why that works but calling `insert_all()` does not.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154386795", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1154386795, "node_id": "IC_kwDOCGYnMM5Ezotr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T20:16:53Z", "updated_at": "2022-06-13T20:16:53Z", "author_association": "OWNER", "body": "Steps to demonstrate that `sqlite-utils insert` is not affected:\r\n\r\n```bash\r\ncurl -o artsdatabanken.csv https://artsdatabanken.no/Fab2018/api/export/csv\r\nsqlite-utils insert arts.db artsdatabanken artsdatabanken.csv --sniff --csv --encoding utf-16le\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154385916", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440", "id": 1154385916, "node_id": "IC_kwDOCGYnMM5Ezof8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T20:15:49Z", "updated_at": "2022-06-13T20:15:49Z", "author_association": "OWNER", "body": "`rows_from_file()` isn't part of the documented API but maybe it should be!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250629388, "label": "CSV files with too many values in a row cause errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1154373361", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/441", "id": 1154373361, "node_id": "IC_kwDOCGYnMM5Ezlbx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-06-13T20:01:25Z", "updated_at": "2022-06-13T20:01:25Z", "author_association": "OWNER", "body": "Yeah, at the moment the best way to do this is with `search_sql()`, but you're right it really isn't very intuitive.\r\n\r\nHere's how I would do this, using a CTE trick to combine the queries:\r\n```python\r\nsearch_sql = db[\"articles\"].search_sql(columns=[\"title\", \"author\"]))\r\nsql = f\"\"\"\r\nwith search_results as ({search_sql})\r\nselect * from search_results where owner = :owner\r\n\"\"\"\r\nresults = db.query(sql, {\"query\": \"my search query\", \"owner\": \"my owner\"})\r\n```\r\nI'm not sure if `sqlite-utils` should ever evolve to provide a better way of doing this kind of thing to be honest - if it did, it would turn into more of an ORM. Something like [PeeWee](http://docs.peewee-orm.com/en/latest/) may be a better option here.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1257724585, "label": "Combining `rows_where()` and `search()` to limit which rows are searched"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1528#issuecomment-1151887842", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1528", "id": 1151887842, "node_id": "IC_kwDOBm6k_c5EqGni", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-06-10T03:23:08Z", "updated_at": "2022-06-10T03:23:08Z", "author_association": "CONTRIBUTOR", "body": "I just put together a version of this in a plugin: https://github.com/eyeseast/datasette-query-files. Happy to have any feedback.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1060631257, "label": "Add new `\"sql_file\"` key to Canned Queries in metadata?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1753#issuecomment-1147435032", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1753", "id": 1147435032, "node_id": "IC_kwDOBm6k_c5EZHgY", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-06-06T13:15:11Z", "updated_at": "2022-06-06T13:15:11Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1753?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#1753](https://codecov.io/gh/simonw/datasette/pull/1753?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (23a8515) into [main](https://codecov.io/gh/simonw/datasette/commit/2e9751672d4fe329b3c359d5b7b1992283185820?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2e97516) will **not change** coverage.\n> The diff coverage is `n/a`.\n\n```diff\n@@ Coverage Diff @@\n## main #1753 +/- ##\n=======================================\n Coverage 91.67% 91.67% \n=======================================\n Files 36 36 \n Lines 4658 4658 \n=======================================\n Hits 4270 4270 \n Misses 388 388 \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1753?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1753?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2e97516...23a8515](https://codecov.io/gh/simonw/datasette/pull/1753?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1261826957, "label": "Bump furo from 2022.4.7 to 2022.6.4.1"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1740#issuecomment-1142556455", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1740", "id": 1142556455, "node_id": "IC_kwDOBm6k_c5EGgcn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-31T19:25:49Z", "updated_at": "2022-05-31T19:25:49Z", "author_association": "OWNER", "body": "Thanks, this looks like a good idea to me.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1226106354, "label": "chore: Set permissions for GitHub actions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/26#issuecomment-1141711418", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/26", "id": 1141711418, "node_id": "IC_kwDOCGYnMM5EDSI6", "user": {"value": 19304, "label": "nileshtrivedi"}, "created_at": "2022-05-31T06:21:15Z", "updated_at": "2022-05-31T06:21:15Z", "author_association": "NONE", "body": "I ran into this. My use case has a JSON file with array of `book` objects with a key called `reviews` which is also an array of objects. My JSON is human-edited and does not specify IDs for either books or reviews. Because sqlite-utils does not support inserting nested objects, I instead have to maintain two separate CSV files with `id` column in `books.csv` and `book_id` column in reviews.csv.\r\n\r\nI think the right way to declare the relationship while inserting a JSON might be to describe the relationship:\r\n\r\n`sqlite-utils insert data.db books mydata.json --hasmany reviews --hasone author --manytomany tags`\r\n\r\nThis is relying on the assumption that foreign keys can point to `rowid` primary key.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 455486286, "label": "Mechanism for turning nested JSON into foreign keys / many-to-many"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/437#issuecomment-1141488533", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/437", "id": 1141488533, "node_id": "IC_kwDOCGYnMM5ECbuV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-30T21:32:36Z", "updated_at": "2022-05-30T21:32:36Z", "author_association": "OWNER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1244294227, "label": "docs to dogs"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1751#issuecomment-1140321380", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1751", "id": 1140321380, "node_id": "IC_kwDOBm6k_c5D9-xk", "user": {"value": 408765, "label": "knutwannheden"}, "created_at": "2022-05-28T19:52:17Z", "updated_at": "2022-05-28T19:52:17Z", "author_association": "NONE", "body": "Closing in favor of existing issue #1298.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1251710928, "label": "Add scrollbars to table presentation in default layout"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1139426398", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439", "id": 1139426398, "node_id": "IC_kwDOCGYnMM5D6kRe", "user": {"value": 4068, "label": "frafra"}, "created_at": "2022-05-27T09:04:05Z", "updated_at": "2022-05-27T10:44:54Z", "author_association": "NONE", "body": "This code works:\r\n\r\n```python\r\nimport csv\r\nimport sqlite_utils\r\ndb = sqlite_utils.Database(\"test.db\")\r\nreader = csv.DictReader(open(\"csv\", encoding=\"utf-16-le\").read().split(\"\\r\\n\"), delimiter=\";\")\r\ndb[\"test\"].insert_all(reader, pk=\"Id\")\r\n```\r\n\r\nI used `iconv` to change the encoding; sqlite-utils can import the resulting file, even if it stops at 98 %:\r\n\r\n```\r\nsqlite-utils insert --csv test test.db clean \r\n [------------------------------------] 0%\r\n [###################################-] 98% 00:00:00\r\n```\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250495688, "label": "Misleading progress bar against utf-16-le CSV input"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1139484453", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/433", "id": 1139484453, "node_id": "IC_kwDOCGYnMM5D6ycl", "user": {"value": 4068, "label": "frafra"}, "created_at": "2022-05-27T10:20:08Z", "updated_at": "2022-05-27T10:20:08Z", "author_association": "NONE", "body": "I can confirm. This only happens with sqlite-utils. I am using gnome-terminal with bash.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239034903, "label": "CLI eats my cursor"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/438#issuecomment-1139392769", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/438", "id": 1139392769, "node_id": "IC_kwDOCGYnMM5D6cEB", "user": {"value": 4068, "label": "frafra"}, "created_at": "2022-05-27T08:21:53Z", "updated_at": "2022-05-27T08:21:53Z", "author_association": "NONE", "body": "Argument were specified in the wrong order. `PATH TABLE FILE` can be misleading :)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250161887, "label": "illegal UTF-16 surrogate"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/438#issuecomment-1139379923", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/438", "id": 1139379923, "node_id": "IC_kwDOCGYnMM5D6Y7T", "user": {"value": 4068, "label": "frafra"}, "created_at": "2022-05-27T08:05:01Z", "updated_at": "2022-05-27T08:05:01Z", "author_association": "NONE", "body": "I tried to debug it using `pdb`, but it looks `sqlite-utils` catches the exception, so it is not quick to figure out where the failure is happening.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1250161887, "label": "illegal UTF-16 surrogate"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/435#issuecomment-1133417432", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/435", "id": 1133417432, "node_id": "IC_kwDOCGYnMM5DjpPY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-20T21:56:10Z", "updated_at": "2022-05-20T21:56:10Z", "author_association": "OWNER", "body": "Before:\r\n\r\n![sqlite-utils-datasette-io-en-stable-reference-html](https://user-images.githubusercontent.com/9599/169617623-457b4c01-3713-4e5c-b3c3-8574a2214238.png)\r\n\r\nAfter:\r\n\r\n![sqlite-utils-datasette-io-en-latest-reference-html](https://user-images.githubusercontent.com/9599/169617666-ba388167-36b8-4a3e-be35-931ee2ab6b3b.png)\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243704847, "label": "Switch to Furo documentation theme"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/435#issuecomment-1133416698", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/435", "id": 1133416698, "node_id": "IC_kwDOCGYnMM5DjpD6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-20T21:54:43Z", "updated_at": "2022-05-20T21:54:43Z", "author_association": "OWNER", "body": "Done: https://sqlite-utils.datasette.io/en/latest/reference.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243704847, "label": "Switch to Furo documentation theme"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1746#issuecomment-1133396285", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1746", "id": 1133396285, "node_id": "IC_kwDOBm6k_c5DjkE9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-20T21:28:29Z", "updated_at": "2022-05-20T21:28:29Z", "author_association": "OWNER", "body": "That fixed it:\r\n\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243498298, "label": "Switch documentation theme to Furo"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1746#issuecomment-1133348094", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1746", "id": 1133348094, "node_id": "IC_kwDOBm6k_c5DjYT-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-20T20:40:09Z", "updated_at": "2022-05-20T20:40:09Z", "author_association": "OWNER", "body": "Relevant JavaScript: https://github.com/simonw/datasette/blob/1d33fd03b3c211e0f48a8f3bde83880af89e4e69/docs/_static/js/custom.js#L20-L24", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243498298, "label": "Switch documentation theme to Furo"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1746#issuecomment-1133347051", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1746", "id": 1133347051, "node_id": "IC_kwDOBm6k_c5DjYDr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-20T20:39:17Z", "updated_at": "2022-05-20T20:39:17Z", "author_association": "OWNER", "body": "Now live at https://docs.datasette.io/en/latest/ - the JavaScript that adds the banner about that not being the stable version doesn't seem to work though.\r\n\r\nBefore:\r\n\r\n\"image\"\r\n\r\nAfter:\r\n\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243498298, "label": "Switch documentation theme to Furo"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1693#issuecomment-1081861670", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1693", "id": 1081861670, "node_id": "IC_kwDOBm6k_c5Ae-Ym", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-03-29T13:18:47Z", "updated_at": "2022-05-20T20:36:30Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1693?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#1693](https://codecov.io/gh/simonw/datasette/pull/1693?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (65a5d5e) into [main](https://codecov.io/gh/simonw/datasette/commit/1465fea4798599eccfe7e8f012bd8d9adfac3039?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1465fea) will **not change** coverage.\n> The diff coverage is `n/a`.\n\n> :exclamation: Current head 65a5d5e differs from pull request most recent head ec2d1e4. Consider uploading reports for the commit ec2d1e4 to get more accurate results\n\n```diff\n@@ Coverage Diff @@\n## main #1693 +/- ##\n=======================================\n Coverage 91.67% 91.67% \n=======================================\n Files 36 36 \n Lines 4658 4658 \n=======================================\n Hits 4270 4270 \n Misses 388 388 \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1693?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1693?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1d33fd0...ec2d1e4](https://codecov.io/gh/simonw/datasette/pull/1693?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1184850337, "label": "Bump black from 22.1.0 to 22.3.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1746#issuecomment-1133335940", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1746", "id": 1133335940, "node_id": "IC_kwDOBm6k_c5DjVWE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-20T20:30:29Z", "updated_at": "2022-05-20T20:30:29Z", "author_association": "OWNER", "body": "I think the trick will be to extend the `base.html` template from Furo using the same trick I used in https://til.simonwillison.net/readthedocs/custom-sphinx-templates\r\n\r\nhttps://github.com/pradyunsg/furo/blob/2022.04.07/src/furo/theme/furo/base.html - the `site_meta` block looks good.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243498298, "label": "Switch documentation theme to Furo"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1746#issuecomment-1133333144", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1746", "id": 1133333144, "node_id": "IC_kwDOBm6k_c5DjUqY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-20T20:28:25Z", "updated_at": "2022-05-20T20:28:25Z", "author_association": "OWNER", "body": "One last question: how to include the Plausible analytics?\r\n\r\nFuro doesn't have any specific tools for this:\r\n\r\n- https://github.com/pradyunsg/furo/discussions/243", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243498298, "label": "Switch documentation theme to Furo"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1746#issuecomment-1133331997", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1746", "id": 1133331997, "node_id": "IC_kwDOBm6k_c5DjUYd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-20T20:27:31Z", "updated_at": "2022-05-20T20:27:31Z", "author_association": "OWNER", "body": "I'm going to move my custom JavaScript from `layout.html` into `js/custom.js`, similar to how the custom CSS works.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243498298, "label": "Switch documentation theme to Furo"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1746#issuecomment-1133331564", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1746", "id": 1133331564, "node_id": "IC_kwDOBm6k_c5DjURs", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-20T20:27:12Z", "updated_at": "2022-05-20T20:27:12Z", "author_association": "OWNER", "body": "This seems to work for `brand.html`:\r\n\r\n```html+jinja\r\n
\r\n {% block brand_content %}\r\n
\r\n \"Datasette\"\r\n
\r\n {%- set nav_version = version %}\r\n {% if READTHEDOCS and current_version %}\r\n {%- set nav_version = current_version %}\r\n {% endif %}\r\n {% if nav_version %}\r\n
\r\n {{ nav_version }}\r\n
\r\n {% endif %}\r\n {% endblock brand_content %}\r\n
\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243498298, "label": "Switch documentation theme to Furo"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1746#issuecomment-1133310253", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1746", "id": 1133310253, "node_id": "IC_kwDOBm6k_c5DjPEt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-20T20:11:00Z", "updated_at": "2022-05-20T20:11:00Z", "author_association": "OWNER", "body": "Oh but `rg display_version` is a lot more interesting:\r\n```\r\nlib/python3.10/site-packages/sphinx/builders/html/__init__.py:from sphinx import __display_version__, package_dir\r\nlib/python3.10/site-packages/sphinx/builders/html/__init__.py: 'sphinx_version': __display_version__,\r\nlib/python3.10/site-packages/sphinx/application.py: logger.info(bold(__('Running Sphinx v%s') % sphinx.__display_version__))\r\nlib/python3.10/site-packages/sphinx/application.py: if self.config.needs_sphinx and self.config.needs_sphinx > sphinx.__display_version__:\r\nlib/python3.10/site-packages/sphinx/application.py: if version > sphinx.__display_version__[:3]:\r\nlib/python3.10/site-packages/sphinx/cmd/build.py:from sphinx import __display_version__, package_dir\r\nlib/python3.10/site-packages/sphinx/cmd/build.py: version='%%(prog)s %s' % __display_version__)\r\nlib/python3.10/site-packages/sphinx/cmd/make_mode.py: print(bold(\"Sphinx v%s\" % sphinx.__display_version__))\r\nlib/python3.10/site-packages/sphinx/__init__.py:__display_version__ = __version__ # used for command line version\r\nlib/python3.10/site-packages/sphinx/__init__.py: __display_version__ = __version__\r\nlib/python3.10/site-packages/sphinx/__init__.py: __display_version__ += '/' + ret.stdout.strip()\r\nlib/python3.10/site-packages/sphinx/ext/githubpages.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/sphinx/ext/intersphinx.py: 'version': sphinx.__display_version__,\r\nlib/python3.10/site-packages/sphinx/cmd/quickstart.py:from sphinx import __display_version__, package_dir\r\nlib/python3.10/site-packages/sphinx/cmd/quickstart.py: print(bold(__('Welcome to the Sphinx %s quickstart utility.')) % __display_version__)\r\nlib/python3.10/site-packages/sphinx/cmd/quickstart.py: version='%%(prog)s %s' % __display_version__)\r\nlib/python3.10/site-packages/sphinx/ext/viewcode.py: 'version': sphinx.__display_version__,\r\nlib/python3.10/site-packages/sphinx/util/__init__.py: (sphinx.__display_version__,\r\nlib/python3.10/site-packages/sphinx/ext/ifconfig.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/sphinx/ext/todo.py: 'version': sphinx.__display_version__,\r\nlib/python3.10/site-packages/sphinx/ext/doctest.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/sphinx/ext/autosummary/__init__.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/sphinx/ext/napoleon/__init__.py:from sphinx import __display_version__ as __version__\r\nlib/python3.10/site-packages/sphinx/ext/autosummary/generate.py:from sphinx import __display_version__, package_dir\r\nlib/python3.10/site-packages/sphinx/ext/autosummary/generate.py: version='%%(prog)s %s' % __display_version__)\r\nlib/python3.10/site-packages/sphinx/ext/inheritance_diagram.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/sphinx/ext/imgmath.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/sphinx/ext/linkcode.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/sphinx/ext/coverage.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/sphinx/writers/texinfo.py:from sphinx import __display_version__, addnodes\r\nlib/python3.10/site-packages/sphinx/writers/texinfo.py:@*Generated by Sphinx \"\"\" + __display_version__ + \"\"\".@*\r\nlib/python3.10/site-packages/sphinx/ext/graphviz.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/sphinx/ext/mathjax.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/sphinx/ext/extlinks.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/sphinx/ext/apidoc.py:from sphinx import __display_version__, package_dir\r\nlib/python3.10/site-packages/sphinx/ext/apidoc.py: version='%%(prog)s %s' % __display_version__)\r\nlib/python3.10/site-packages/sphinx/ext/autodoc/type_comment.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/sphinx/ext/autodoc/__init__.py: return {'version': sphinx.__display_version__, 'parallel_read_safe': True}\r\nlib/python3.10/site-packages/pip/_internal/models/target_python.py: display_version = None\r\nlib/python3.10/site-packages/pip/_internal/models/target_python.py: display_version = '.'.join(\r\nlib/python3.10/site-packages/pip/_internal/models/target_python.py: ('version_info', display_version),\r\nlib/python3.10/site-packages/sphinx_rtd_theme/theme.conf:display_version = True\r\nlib/python3.10/site-packages/sphinx_rtd_theme/layout.html: {%- if theme_display_version %}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243498298, "label": "Switch documentation theme to Furo"}, "performed_via_github_app": null}