{"html_url": "https://github.com/simonw/sqlite-utils/issues/529#issuecomment-1592110694", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/529", "id": 1592110694, "node_id": "IC_kwDOCGYnMM5e5a5m", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T23:11:47Z", "updated_at": "2023-06-14T23:12:12Z", "author_association": "CONTRIBUTOR", "body": "sorry i was wrong. `sqlite-utils --raw-lines` works correctly\r\n\r\n```\r\nsqlite-utils --raw-lines :memory: \"SELECT * FROM (VALUES ('test'), ('line2'))\" | cat -A\r\ntest$\r\nline2$\r\n\r\nsqlite-utils --csv --no-headers :memory: \"SELECT * FROM (VALUES ('test'), ('line2'))\" | cat -A\r\ntest$\r\nline2$\r\n```\r\n\r\nI think this was fixed somewhat recently", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1581090327, "label": "Microsoft line endings"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1264218914", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491", "id": 1264218914, "node_id": "IC_kwDOCGYnMM5LWnMi", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-10-01T03:18:36Z", "updated_at": "2023-06-14T22:14:24Z", "author_association": "CONTRIBUTOR", "body": "> some good concrete use-cases in mind\r\n\r\nI actually found myself wanting something like this the past couple days. The use-case was databases with slightly different schema but same table names.\r\n\r\nhere is a full script:\r\n\r\n```\r\nimport argparse\r\nfrom pathlib import Path\r\n\r\nfrom sqlite_utils import Database\r\n\r\n\r\ndef connect(args, conn=None, **kwargs) -> Database:\r\n    db = Database(conn or args.database, **kwargs)\r\n    with db.conn:\r\n        db.conn.execute(\"PRAGMA main.cache_size = 8000\")\r\n    return db\r\n\r\n\r\ndef parse_args() -> argparse.Namespace:\r\n    parser = argparse.ArgumentParser()\r\n    parser.add_argument(\"database\")\r\n    parser.add_argument(\"dbs_folder\")\r\n    parser.add_argument(\"--db\", \"-db\", help=argparse.SUPPRESS)\r\n    parser.add_argument(\"--verbose\", \"-v\", action=\"count\", default=0)\r\n    args = parser.parse_args()\r\n\r\n    if args.db:\r\n        args.database = args.db\r\n    Path(args.database).touch()\r\n    args.db = connect(args)\r\n\r\n    return args\r\n\r\n\r\ndef merge_db(args, source_db):\r\n    source_db = str(Path(source_db).resolve())\r\n\r\n    s_db = connect(argparse.Namespace(database=source_db, verbose = args.verbose))\r\n    for table in s_db.table_names():\r\n        data = s_db[table].rows\r\n        args.db[table].insert_all(data, alter=True, replace=True)\r\n\r\n    args.db.conn.commit()\r\n\r\n\r\ndef merge_directory():\r\n    args = parse_args()\r\n    source_dbs = list(Path(args.dbs_folder).glob('*.db'))\r\n    for s_db in source_dbs:\r\n        merge_db(args, s_db)\r\n\r\n\r\nif __name__ == '__main__':\r\n    merge_directory()\r\n```\r\n\r\nedit: I've made some improvements to this and put it on PyPI:\r\n\r\n```\r\n$ pip install xklb\r\n$ lb merge-db -h\r\nusage: library merge-dbs DEST_DB SOURCE_DB ... [--only-target-columns] [--only-new-rows] [--upsert] [--pk PK ...] [--table TABLE ...]\r\n\r\n    Merge-DBs will insert new rows from source dbs to target db, table by table. If primary key(s) are provided,\r\n    and there is an existing row with the same PK, the default action is to delete the existing row and insert the new row\r\n    replacing all existing fields.\r\n\r\n    Upsert mode will update matching PK rows such that if a source row has a NULL field and\r\n    the destination row has a value then the value will be preserved instead of changed to the source row's NULL value.\r\n\r\n    Ignore mode (--only-new-rows) will insert only rows which don't already exist in the destination db\r\n\r\n    Test first by using temp databases as the destination db.\r\n    Try out different modes / flags until you are satisfied with the behavior of the program\r\n\r\n        library merge-dbs --pk path (mktemp --suffix .db) tv.db movies.db\r\n\r\n    Merge database data and tables\r\n\r\n        library merge-dbs --upsert --pk path video.db tv.db movies.db\r\n        library merge-dbs --only-target-columns --only-new-rows --table media,playlists --pk path audio-fts.db audio.db\r\n\r\n        library merge-dbs --pk id --only-tables subreddits reddit/81_New_Music.db audio.db\r\n        library merge-dbs --only-new-rows --pk subreddit,path --only-tables reddit_posts reddit/81_New_Music.db audio.db -v\r\n\r\npositional arguments:\r\n  database\r\n  source_dbs\r\n```\r\n\r\nAlso if you want to dedupe a table based on a \"business key\" which isn't explicitly your primary key(s) you can run this:\r\n\r\n```\r\n$ lb dedupe-db -h\r\nusage: library dedupe-dbs DATABASE TABLE --bk BUSINESS_KEYS [--pk PRIMARY_KEYS] [--only-columns COLUMNS]\r\n\r\n    Dedupe your database (not to be confused with the dedupe subcommand)\r\n\r\n    It should not need to be said but *backup* your database before trying this tool!\r\n\r\n    Dedupe-DB will help remove duplicate rows based on non-primary-key business keys\r\n\r\n        library dedupe-db ./video.db media --bk path\r\n\r\n    If --primary-keys is not provided table metadata primary keys will be used\r\n    If --only-columns is not provided all non-primary and non-business key columns will be upserted\r\n\r\npositional arguments:\r\n  database\r\n  table\r\n\r\noptions:\r\n  -h, --help            show this help message and exit\r\n  --skip-0\r\n  --only-columns ONLY_COLUMNS\r\n                        Comma separated column names to upsert\r\n  --primary-keys PRIMARY_KEYS, --pk PRIMARY_KEYS\r\n                        Comma separated primary keys\r\n  --business-keys BUSINESS_KEYS, --bk BUSINESS_KEYS\r\n                        Comma separated business keys\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1383646615, "label": "Ability to merge databases and tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592052320", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/535", "id": 1592052320, "node_id": "IC_kwDOCGYnMM5e5Mpg", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T22:05:28Z", "updated_at": "2023-06-14T22:05:28Z", "author_association": "CONTRIBUTOR", "body": "piping to `jq` is good enough usually", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1655860104, "label": "rows: --transpose or psql extended view-like functionality"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/555#issuecomment-1592047502", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/555", "id": 1592047502, "node_id": "IC_kwDOCGYnMM5e5LeO", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T22:00:10Z", "updated_at": "2023-06-14T22:01:57Z", "author_association": "CONTRIBUTOR", "body": "You may want to try doing a performance comparison between this and just selecting all the ids with few constraints and then doing the filtering within python.\r\n\r\nThat might seem like a lazy-programmer, inefficient way but queries with large resultsets are a different profile than what databases like SQLITE are designed for. That is not to say that SQLITE is slow or that python is always faster but when you start reading >20% of an index there is an equilibrium that is reached. Especially when adding in writing extra temp tables and stuff to memory/disk. And especially given the `NOT IN` style of query...\r\n\r\nYou may also try chunking like this:\r\n\r\n```py\r\ndef chunks(lst, n) -> Generator:\r\n    for i in range(0, len(lst), n):\r\n        yield lst[i : i + n]\r\n\r\nSQLITE_PARAM_LIMIT = 32765\r\n\r\ndata = []\r\nchunked = chunks(video_ids, consts.SQLITE_PARAM_LIMIT)\r\nfor ids in chunked:\r\n    data.expand(\r\n        list(\r\n            db.query(\r\n                f\"\"\"SELECT * from videos\r\n                WHERE id in (\"\"\"\r\n                + \",\".join([\"?\"] * len(ids))\r\n                + \")\",\r\n                (*ids,),\r\n            )\r\n        )\r\n    )\r\n```\r\n\r\nbut that actually won't work with your `NOT IN` requirements. You need to query the full resultset to check any row.\r\n\r\nSince you are doing stuff with files/videos in SQLITE you might be interested in my side project: https://github.com/chapmanjacobd/library", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1733198948, "label": "Filter table by a large bunch of ids"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1590531892", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/557", "id": 1590531892, "node_id": "IC_kwDOCGYnMM5ezZc0", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T06:09:21Z", "updated_at": "2023-06-14T06:09:21Z", "author_association": "CONTRIBUTOR", "body": "I put together a [simple script](https://github.com/chapmanjacobd/library/blob/42129c5ebe15f9d74653c0f5ca4ed0c991d383e0/xklb/scripts/dedupe_db.py) to upsert and remove duplicate rows based on business keys. If anyone has similar problems with above this might help\r\n\r\n```\r\nCREATE TABLE my_table (\r\n    id INTEGER PRIMARY KEY,\r\n    column1 TEXT,\r\n    column2 TEXT,\r\n    column3 TEXT\r\n);\r\n\r\nINSERT INTO my_table (column1, column2, column3)\r\nVALUES\r\n    ('Value 1', 'Duplicate 1', 'Duplicate A'),\r\n    ('Value 2', 'Duplicate 2', 'Duplicate B'),\r\n    ('Value 3', 'Duplicate 2', 'Duplicate C'),\r\n    ('Value 4', 'Duplicate 3', 'Duplicate D'),\r\n    ('Value 5', 'Duplicate 3', 'Duplicate E'),\r\n    ('Value 6', 'Duplicate 3', 'Duplicate F');\r\n```\r\n\r\n```\r\nlibrary dedupe-db test.db my_table --bk column2\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1740150327, "label": "Aliased ROWID option for tables created from alter=True commands"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1577355134", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/557", "id": 1577355134, "node_id": "IC_kwDOCGYnMM5eBId-", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-05T19:26:26Z", "updated_at": "2023-06-05T19:26:26Z", "author_association": "CONTRIBUTOR", "body": "this isn't really actionable... I'm just being a whiny baby. I have tasted the milk of being able to use `upsert_all`, `insert_all`, etc without having to write DDL to create tables. The meat of the issue is that SQLITE doesn't make rowid stable between vacuums so it is not possible to take shortcuts", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1740150327, "label": "Aliased ROWID option for tables created from alter=True commands"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/508#issuecomment-1297788531", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/508", "id": 1297788531, "node_id": "IC_kwDOCGYnMM5NWq5z", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-10-31T22:54:33Z", "updated_at": "2022-11-17T15:11:16Z", "author_association": "CONTRIBUTOR", "body": "Maybe this is actually a problem in the python sqlite bindings. Given [SQLITE's stance on this](https://www.sqlite.org/invalidutf.html) they should probably use `encode('utf-8', 'surrogatepass')`. As far as I understand the error here won't actually be resolved by this PR as-is. We would need to modify the data with `surrogateescape`... :/ or modify the sqlite3 module to use `surrogatepass`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1430563092, "label": "Allow surrogates in parameters"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/510#issuecomment-1318777114", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/510", "id": 1318777114, "node_id": "IC_kwDOCGYnMM5OmvEa", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-11-17T15:09:47Z", "updated_at": "2022-11-17T15:09:47Z", "author_association": "CONTRIBUTOR", "body": "why close? is the only problem that the _config table that incorrectly says 4 for fts5? if so, that's still something that should be fixed", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434911255, "label": "Cannot enable FTS5 despite it being available"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304320521", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/511", "id": 1304320521, "node_id": "IC_kwDOCGYnMM5NvloJ", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-11-04T22:54:09Z", "updated_at": "2022-11-04T22:59:54Z", "author_association": "CONTRIBUTOR", "body": "I ran `PRAGMA integrity_check` and it returned `ok`. but then I tried restoring from a backup and I didn't get this `IntegrityError: constraint failed` error. So I think it was just something wrong with my database. If it happens again I will first try to reindex and see if that fixes the issue", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1436539554, "label": "[insert_all, upsert_all] IntegrityError: constraint failed"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304078945", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/511", "id": 1304078945, "node_id": "IC_kwDOCGYnMM5Nuqph", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-11-04T19:38:36Z", "updated_at": "2022-11-04T20:13:17Z", "author_association": "CONTRIBUTOR", "body": "Even more bizarre, the source db only has one record and the target table has no conflicting record:\r\n\r\n```\r\n875 0.3s lb:/ (main|\u271a2) [0|0]\ud83c\udf3a sqlite-utils tube_71.db 'select * from media where path = \"https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz\"' | jq\r\n[\r\n  {\r\n    \"size\": null,\r\n    \"time_created\": null,\r\n    \"play_count\": 1,\r\n    \"language\": null,\r\n    \"view_count\": null,\r\n    \"width\": null,\r\n    \"height\": null,\r\n    \"fps\": null,\r\n    \"average_rating\": null,\r\n    \"live_status\": null,\r\n    \"age_limit\": null,\r\n    \"uploader\": null,\r\n    \"time_played\": 0,\r\n    \"path\": \"https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz\",\r\n    \"id\": \"088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz/074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv\",\r\n    \"ie_key\": \"ArchiveOrg\",\r\n    \"playlist_path\": \"https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz\",\r\n    \"duration\": 1424.05,\r\n    \"tags\": null,\r\n    \"title\": \"074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv\"\r\n  }\r\n]\r\n875 0.3s lb:/ (main|\u271a2) [0|0]\ud83e\udd67 sqlite-utils video.db 'select * from media where path = \"https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz\"' | jq\r\n[]\r\n```\r\n\r\nI've been able to use this code successfully several times before so not sure what's causing the issue.\r\n\r\nI guess the way that I'm handling multiple databases is an issue, though it hasn't ever inserted into the source db, not sure what's different. The only reasonable explanation is that it is trying to insert into the source db from the source db for some reason? Or maybe sqlite3 is checking the source db for primary key violation because the table name is the same", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1436539554, "label": "[insert_all, upsert_all] IntegrityError: constraint failed"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/50#issuecomment-1303660293", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/50", "id": 1303660293, "node_id": "IC_kwDOCGYnMM5NtEcF", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-11-04T14:38:36Z", "updated_at": "2022-11-04T14:38:36Z", "author_association": "CONTRIBUTOR", "body": "where did you see the limit as 999? I believe the limit has been 32766 for quite some time. If you could detect which one this could speed up batch insert of some types of data significantly", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 473083260, "label": "\"Too many SQL variables\" on large inserts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/507#issuecomment-1297859539", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/507", "id": 1297859539, "node_id": "IC_kwDOCGYnMM5NW8PT", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-11-01T00:40:16Z", "updated_at": "2022-11-01T00:40:16Z", "author_association": "CONTRIBUTOR", "body": "Ideally people could fix their data if they run into this issue.\r\n\r\nIf you are using filenames try [convmv](https://linux.die.net/man/1/convmv)\r\n\r\n```\r\nconvmv --preserve-mtimes -f utf8 -t utf8 --notest -i -r .\r\n```\r\n\r\nmaybe this script will also help: \r\n\r\n```py\r\nimport argparse, shutil\r\nfrom pathlib import Path\r\n\r\nimport ftfy\r\n\r\nfrom xklb import utils\r\nfrom xklb.utils import log\r\n\r\n\r\ndef parse_args() -> argparse.Namespace:\r\n    parser = argparse.ArgumentParser()\r\n    parser.add_argument(\"paths\", nargs='*')\r\n    parser.add_argument(\"--verbose\", \"-v\", action=\"count\", default=0)\r\n    args = parser.parse_args()\r\n\r\n    log.info(utils.dict_filter_bool(args.__dict__))\r\n    return args\r\n\r\n\r\ndef rename_invalid_paths() -> None:\r\n    args = parse_args()\r\n\r\n    for path in args.paths:\r\n        log.info(path)\r\n        for p in sorted([str(p) for p in Path(path).rglob(\"*\")], key=len):\r\n            fixed = ftfy.fix_text(p, uncurl_quotes=False).replace(\"\\r\\n\", \"\\n\").replace(\"\\r\", \"\\n\").replace(\"\\n\", \"\")\r\n            if p != fixed:\r\n                try:\r\n                    shutil.move(p, fixed)\r\n                except FileNotFoundError:\r\n                    log.warning(\"FileNotFound. %s\", p)\r\n                else:\r\n                    log.info(fixed)\r\n\r\n\r\nif __name__ == \"__main__\":\r\n    rename_invalid_paths()\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1430325103, "label": "conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/499#issuecomment-1292401308", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/499", "id": 1292401308, "node_id": "IC_kwDOCGYnMM5NCHqc", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-10-26T17:54:26Z", "updated_at": "2022-10-26T17:54:51Z", "author_association": "CONTRIBUTOR", "body": "The problem with how it is currently is that the transformed fts table _will_ return incorrect results (unless the table was only 1 row or something), even if create_triggers was enabled previously. Maybe the simplest solution is to disable fts on a transformed table rather than try to recreate it? Thoughts?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1405196044, "label": "feat: recreate fts triggers after table transform"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/60#issuecomment-1279249898", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/60", "id": 1279249898, "node_id": "IC_kwDODEm0Qs5MP83q", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-10-14T16:58:26Z", "updated_at": "2022-10-14T16:58:26Z", "author_association": "NONE", "body": "You could try using `msys2`. I've had better luck running python CLIs within that system on Windows.\r\n\r\nHere is a guide: https://github.com/chapmanjacobd/lb/blob/main/Windows.md#prep", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1063982712, "label": "Execution on Windows"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-1279224780", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51", "id": 1279224780, "node_id": "IC_kwDODFdgUs5MP2vM", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-10-14T16:34:07Z", "updated_at": "2022-10-14T16:34:07Z", "author_association": "NONE", "body": "also, it says that authenticated requests have a much higher \"rate limit\". Unauthenticated requests only get 60 req/hour ?? seems more like a quota than a \"rate limit\" (although I guess that is semantic equivalence)\r\n\r\nYou would want to use `x-ratelimit-reset`\r\n\r\n```\r\ntime.sleep(r['x-ratelimit-reset'] + 1 - time.time())\r\n```\r\n\r\nBut a more complete solution would bring authenticated requests to the other subcommands. I'm surprised only `github-to-sqlite get` is using the `--auth=` CLI flag", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 703246031, "label": "github-to-sqlite should handle rate limits better"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/498#issuecomment-1274153135", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/498", "id": 1274153135, "node_id": "IC_kwDOCGYnMM5L8giv", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-10-11T06:34:31Z", "updated_at": "2022-10-11T06:34:31Z", "author_association": "CONTRIBUTOR", "body": "nevermind it was because I was running `db[table].transform`. The fts tables would still be there but the triggers would be dropped", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1404013495, "label": "fix: enable-fts permanently save triggers"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1264223554", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/409", "id": 1264223554, "node_id": "IC_kwDOCGYnMM5LWoVC", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-10-01T03:42:50Z", "updated_at": "2022-10-01T03:42:50Z", "author_association": "CONTRIBUTOR", "body": "oh weird. it inserts into db2", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1149661489, "label": "`with db:` for transactions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1264223363", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/409", "id": 1264223363, "node_id": "IC_kwDOCGYnMM5LWoSD", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-10-01T03:41:45Z", "updated_at": "2022-10-01T03:41:45Z", "author_association": "CONTRIBUTOR", "body": "```\r\npytest xklb/check.py --pdb\r\n\r\nxklb/check.py:11: in test_transaction\r\n    assert list(db2[\"t\"].rows) == []\r\nE   AssertionError: assert [{'foo': 1}] == []\r\nE    +  where [{'foo': 1}] = list(<generator object Queryable.rows_where at 0x7f2d84d1f0d0>)\r\nE    +    where <generator object Queryable.rows_where at 0x7f2d84d1f0d0> = <Table t (foo)>.rows\r\n>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\r\n\r\n>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\r\n> /home/xk/github/xk/lb/xklb/check.py(11)test_transaction()\r\n      9     with db1.conn:\r\n     10         db1[\"t\"].insert({\"foo\": 1})\r\n---> 11         assert list(db2[\"t\"].rows) == []\r\n     12     assert list(db2[\"t\"].rows) == [{\"foo\": 1}]\r\n```\r\n\r\nIt fails because it is already inserted.\r\n\r\nbtw if you put these two lines in you pyproject.toml you can get `ipdb` in pytest\r\n\r\n```\r\n[tool.pytest.ini_options]\r\naddopts = \"--pdbcls=IPython.terminal.debugger:TerminalPdb --ignore=tests/data --capture=tee-sys --log-cli-level=ERROR\"\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1149661489, "label": "`with db:` for transactions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1264219650", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/493", "id": 1264219650, "node_id": "IC_kwDOCGYnMM5LWnYC", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-10-01T03:22:50Z", "updated_at": "2022-10-01T03:23:58Z", "author_association": "CONTRIBUTOR", "body": "this is likely what you are looking for: https://stackoverflow.com/a/51076749/697964\r\n\r\nbut yeah I would say just disable smart quotes", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386562662, "label": "Tiny typographical error in install/uninstall docs"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1256858763", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491", "id": 1256858763, "node_id": "IC_kwDOCGYnMM5K6iSL", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-09-24T04:50:59Z", "updated_at": "2022-09-24T04:52:08Z", "author_association": "CONTRIBUTOR", "body": "Instead of outputting binary data to stdout the interface might be better like this\r\n\r\n```\r\nsqlite-utils merge animals.db cats.db dogs.db\r\n```\r\n\r\nsimilar to `zip`, `ogr2ogr`, etc\r\n\r\nActually I think this might already be possible within `ogr2ogr`. I don't believe spatial data is a requirement though it might add an `ogc_id` column or something\r\n\r\n```\r\ncp cats.db animals.db\r\nogr2ogr -append animals.db dogs.db\r\nogr2ogr -append animals.db another.db\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1383646615, "label": "Ability to merge databases and tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1252898131", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/433", "id": 1252898131, "node_id": "IC_kwDOCGYnMM5KrbVT", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-09-20T20:51:21Z", "updated_at": "2022-09-20T20:56:07Z", "author_association": "CONTRIBUTOR", "body": "When I run `reset` it fixes my terminal. I suspect it is related to the progress bar\r\n\r\nhttps://linux.die.net/man/1/reset\r\n\r\n```\r\n950 1s /m/d/03_Downloads \ud83d\udc11 echo $TERM\r\nxterm-kitty\r\n\u2593\u2591\u2592\u2591 /m/d/03_Downloads \ud83c\udf0f kitty -v\r\nkitty 0.26.2 created by Kovid Goyal\r\n$ sqlite-utils insert test.db facility facility-boundary-us-all.csv --csv\r\nblah blah blah (no offense)\r\n$ <no cursor>\r\n$ reset\r\n$ <cursor lives again (resurrection [explicit])>\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239034903, "label": "CLI eats my cursor"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232356302", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/480", "id": 1232356302, "node_id": "IC_kwDOCGYnMM5JdEPO", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-08-31T01:51:49Z", "updated_at": "2022-08-31T01:51:49Z", "author_association": "CONTRIBUTOR", "body": "Thanks for pointing me to the right place", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1355433619, "label": "search_sql add include_rank option"}, "performed_via_github_app": null}