{"id": 504720731, "node_id": "MDU6SXNzdWU1MDQ3MjA3MzE=", "number": 1, "title": "Add more details on how to request data from google takeout correctly.", "user": {"value": 1055831, "label": "dazzag24"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-10-09T15:17:34Z", "updated_at": "2019-10-09T15:17:34Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "The default is to download everything. This can result in an enormous amount of data when you only really need 2 types of data for now:\r\n\r\n- My Activity\r\n- Location History\r\n\r\nIn addition unless you specify that \"My Activity\" is downloaded in JSON format the default is HTML. This then causes the \r\n\r\n`google-takeout-to-sqlite my-activity takeout.db takeout.zip`\r\n\r\ncommand to fail as it only contains html files not json files.\r\n\r\nThanks", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1353411865, "node_id": "I_kwDODEpn8M5Qq20Z", "number": 1, "title": "Problem with my user", "user": {"value": 2467, "label": "fernand0"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-08-28T16:59:37Z", "updated_at": "2022-08-28T16:59:37Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "If I call the program with:\r\n inaturalist-to-sqlite inaturalist.db ftricas\r\nthe program exits with an error:\r\n `Importing 36 observations\r\nTraceback (most recent call last):\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/bin/inaturalist-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/inaturalist_to_sqlite/cli.py\", line 51, in cli\r\n save_observation(observation, db)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/inaturalist_to_sqlite/utils.py\", line 34, in save_observation\r\n db[\"observations\"]\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 2965, in insert\r\n return self.insert_all(\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 3068, in insert_all\r\n self.create(\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 1564, in create\r\n self.db.create_table(\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 951, in create_table\r\n sql = self.create_table_sql(\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 765, in create_table_sql\r\n foreign_keys = self.resolve_foreign_keys(name, foreign_keys or [])\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 702, in resolve_foreign_keys\r\n other_table = table.guess_foreign_table(column)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 2061, in guess_foreign_table\r\n raise NoObviousTable(\r\nsqlite_utils.db.NoObviousTable: No obvious foreign key table for column 'taxon' - tried ['taxon', 'taxons']\r\n`\r\nIf I call the program with your user everything seems to go well and then, I can call the program with my own user without problems. Moreover, I can call the program again with my own user and everything goes well now.\r\n\r\nAdditional info, the command:\r\n \r\n sqlite-utils tables inaturalist.db\r\n\r\nshows that the correct name can be 'taxons'.\r\n\r\nThere is another small problem with a warning:\r\n \r\n warnings.warn(\"urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported \"\r\n\r\n", "repo": {"value": 206202864, "label": "inaturalist-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/inaturalist-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 541274681, "node_id": "MDU6SXNzdWU1NDEyNzQ2ODE=", "number": 2, "title": "Add linkedin-to-sqlite", "user": {"value": 881925, "label": "mnp"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-12-21T03:13:40Z", "updated_at": "2019-12-21T03:13:40Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "There is an API available. https://developer.linkedin.com/docs/rest-api#\r\n\r\nAt the minimum, I would think contact list and messages would be of interest.", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 664793260, "node_id": "MDU6SXNzdWU2NjQ3OTMyNjA=", "number": 2, "title": "Yak shave", "user": {"value": 145425, "label": "ekg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-07-23T22:04:18Z", "updated_at": "2020-07-23T22:04:18Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Just a quick note... The 23andme data is not exactly your genome, but a SNP chip of your genome. It's \"some of your genotypes.\" Or about 0.1% of your genome. Nice work in any case! It deserves to be liberated!!!!!", "repo": {"value": 209590345, "label": "genome-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 769376447, "node_id": "MDU6SXNzdWU3NjkzNzY0NDc=", "number": 2, "title": "killed by oomkiller on large location-history", "user": {"value": 231498, "label": "khimaros"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-12-17T00:32:24Z", "updated_at": "2020-12-17T00:48:32Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "memory seems to grow unbounded and is oom-killed after about 20GB memory usage.\r\n\r\nthis is happening while loading a ~1GB uncompressed location history.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1485017981, "node_id": "I_kwDODEpn8M5Yg5N9", "number": 2, "title": "table identifications has no column named previous_observation_taxon", "user": {"value": 520541, "label": "heaversm"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-12-08T16:47:17Z", "updated_at": "2022-12-08T16:47:17Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Installed successfully with pip and ran `inaturalist-to-sqlite inaturalist.db simonw` and got the error:\r\n\r\n```\r\nsqlite3.OperationalError: table identifications has no column named previous_observation_taxon\r\n```", "repo": {"value": 206202864, "label": "inaturalist-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/inaturalist-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 769397742, "node_id": "MDU6SXNzdWU3NjkzOTc3NDI=", "number": 3, "title": "sqlite-utils error on takeout import", "user": {"value": 231498, "label": "khimaros"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-12-17T01:18:48Z", "updated_at": "2020-12-17T01:19:04Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "```\r\n$ google-takeout-to-sqlite my-activity takeout.db /path/to/zip\r\n...\r\nsqlite3.OperationalError: no such table: main.my_activity\r\n```\r\n\r\nthere is no table create in `utils.py`, unlike other importers such as github-to-sqlite\r\n\r\nadditionally, this package and hackernews-to-sqlite have conflicting `sqlite-utils` dep with datasette and dogsheep-beta", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/3/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 778380836, "node_id": "MDU6SXNzdWU3NzgzODA4MzY=", "number": 4, "title": "Feature Request: Gmail", "user": {"value": 203343, "label": "Btibert3"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-01-04T21:31:09Z", "updated_at": "2021-03-04T20:54:44Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "From takeout, I only exported my Gmail account. Ideally I could parse this into sqlite via this tool.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1205867842, "node_id": "I_kwDODtX3eM5H4BVC", "number": 4, "title": "Retrieve the top-level story for a comment", "user": {"value": 1755789, "label": "telotortium"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-04-15T20:25:39Z", "updated_at": "2022-04-15T20:25:39Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I think that each comment inserted into the database should include a column `onstory` that contains the ID of the story on which the comment was made. This is exactly equivalent to the link after \"on:\" at the top of an HN comment page ([example](https://news.ycombinator.com/item?id=18358028)). We could do this either by directly retrieving the HTML page and using Beautiful Soup to find that link, or alternatively recurse up the tree in the Firebase API using the `parent` field (probably using `functools.lru_cache` in case a person has commented a bunch of times on the same story).", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 629473827, "node_id": "MDU6SXNzdWU2Mjk0NzM4Mjc=", "number": 5, "title": "Set up a demo", "user": {"value": 26745575, "label": "harryvederci"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-06-02T19:56:49Z", "updated_at": "2020-09-01T06:18:43Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "First off, thanks for open sourcing this application! This is a suggestion to increase the amount of people that would make use of it: an example in the readme file would help.\r\n\r\nCurrently, users have to clone the app, install it, authorize through pocket, run a command, an then find out if this application does what they hope it does.\r\n\r\nAnother possibility is to add a file `example-output.db`, containing one (mock) Pocket article.\r\n\r\nKeep up the good work!", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 504238461, "node_id": "MDU6SXNzdWU1MDQyMzg0NjE=", "number": 6, "title": "sqlite3.OperationalError: table users has no column named bio", "user": {"value": 1055831, "label": "dazzag24"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-10-08T19:39:52Z", "updated_at": "2019-10-13T05:31:28Z", "closed_at": "2019-10-13T05:30:19Z", "author_association": "NONE", "pull_request": null, "body": "```\r\n$ github-to-sqlite repos github.db\r\n$ github-to-sqlite starred github.db dazzag24\r\n\r\nTraceback (most recent call last):\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/bin/github-to-sqlite\", line 10, in \r\n sys.exit(cli())\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 764, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 717, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 1137, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 956, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 555, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/cli.py\", line 106, in starred\r\n utils.save_stars(db, user, stars)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py\", line 177, in save_stars\r\n user_id = save_user(db, user)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py\", line 61, in save_user\r\n return db[\"users\"].upsert(to_save, pk=\"id\").last_pk\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py\", line 1067, in upsert\r\n extracts=extracts,\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py\", line 916, in insert\r\n extracts=extracts,\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py\", line 1024, in insert_all\r\n result = self.db.conn.execute(sql, values)\r\nsqlite3.OperationalError: table users has no column named bio\r\n\r\n```\r\n\r\n```\r\n$ pipenv graph\r\ngithub-to-sqlite==0.4\r\n - requests [required: Any, installed: 2.22.0]\r\n - certifi [required: >=2017.4.17, installed: 2019.9.11]\r\n - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]\r\n - idna [required: >=2.5,<2.9, installed: 2.8]\r\n - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]\r\n - sqlite-utils [required: ~=1.11, installed: 1.11]\r\n - click [required: Any, installed: 7.0]\r\n - click-default-group [required: Any, installed: 1.2.2]\r\n - click [required: Any, installed: 7.0]\r\n - tabulate [required: Any, installed: 0.8.5]\r\n\r\nPython 3.6.8\r\n```", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 842765105, "node_id": "MDExOlB1bGxSZXF1ZXN0NjAyMjYxMDky", "number": 6, "title": "Add testres-db tool", "user": {"value": 1151557, "label": "ligurio"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-03-28T15:43:23Z", "updated_at": "2022-02-16T05:12:05Z", "closed_at": "2022-02-16T05:12:05Z", "author_association": "NONE", "pull_request": "dogsheep/dogsheep.github.io/pulls/6", "body": "", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 743297582, "node_id": "MDU6SXNzdWU3NDMyOTc1ODI=", "number": 7, "title": "evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace'", "user": {"value": 42387931, "label": "martinvanwieringen"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-11-15T16:57:28Z", "updated_at": "2021-02-11T22:13:17Z", "closed_at": "2021-02-11T22:13:17Z", "author_association": "NONE", "pull_request": null, "body": "running evernote-to-sqlite 0.2 on windows 10. Command:\r\n\r\nevernote-to-sqlite enex evernote.db MyNotes.enex\r\n\r\nI get the followinng error:\r\n\r\nFile \"C:\\Users\\marti\\AppData\\Roaming\\Python\\Python38\\site-packages\\evernote_to_sqlite\\utils.py\", line 46, in save_note\r\n note_id = db[\"notes\"].insert(row, hash_id=\"id\", replace=True, alter=True).last_pk\r\nTypeError: insert() got an unexpected keyword argument 'replace'\r\n\r\nRemoving replace=True,\r\n\r\nLeads to below error:\r\n\r\n note_id = db[\"notes\"].insert(row, hash_id=\"id\", alter=True).last_pk\r\n File \"C:\\Users\\marti\\AppData\\Roaming\\Python\\Python38\\site-packages\\sqlite_utils\\db.py\", line 924, in insert\r\n return self.insert_all(\r\n File \"C:\\Users\\marti\\AppData\\Roaming\\Python\\Python38\\site-packages\\sqlite_utils\\db.py\", line 1046, in insert_all\r\n result = self.db.conn.execute(sql, values)\r\nsqlite3.IntegrityError: UNIQUE constraint failed: notes.id", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 930946817, "node_id": "MDU6SXNzdWU5MzA5NDY4MTc=", "number": 7, "title": "KeyError: 'accuracy' when processing Location History", "user": {"value": 403152, "label": "davidwilemski"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-06-27T14:39:43Z", "updated_at": "2021-06-27T14:39:43Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I'm new to both the dogsheep tools and datasette but have been experimenting a bit the last few days and these are really cool tools!\r\n\r\nI encountered a problem running my Google location history through this tool running the latest release in a docker container:\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/google-takeout-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/cli.py\", line 49, in my_activity\r\n utils.save_location_history(db, zf)\r\n File \"/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py\", line 27, in save_location_history\r\n db[\"location_history\"].upsert_all(\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1105, in upsert_all\r\n return self.insert_all(\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 990, in insert_all\r\n chunk = list(chunk)\r\n File \"/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py\", line 33, in \r\n \"accuracy\": row[\"accuracy\"],\r\nKeyError: 'accuracy'\r\n```\r\n\r\nIt looks like the tool assumes the `accuracy` key will be in every location history entry. \r\n\r\nMy first attempt at a local patch to get myself going was to convert accessing the `accuracy` key to a `.get` instead to hopefully make the row nullable but I wasn't quite sure what `sqlite_utils` would do there. That did work in that the import happened and so I was going to propose a patch that made that change but in updating the existing test to include an entry with a missing accuracy entry, I noticed the expected type of the field appeared to be changing to a string in the test (and from a quick scan through the sqlite_utils code, probably TEXT in the database). Given this change in column type, it seemed that opening an issue first before proposing a fix seemed warranted. It seems the schema would need to be explicitly specified if you wanted a nullable integer column.\r\n\r\nNow that I've done a successful import run using my initial fix of calling `.get` on the row dict, I can see with datasette that I only have 7 data points (out of ~250k) that have a null accuracy column. They are all from 2011-2012 in an import that includes points spanning ~2010-2016 so perhaps another approach might be to filter those entries out during import if it really is that infrequent?\r\n\r\nI'm happy to provide a PR for a fix but figured I'd ask about which direction is preferred first.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 403922644, "node_id": "MDU6SXNzdWU0MDM5MjI2NDQ=", "number": 8, "title": "Problems handling column names containing spaces or - ", "user": {"value": 82988, "label": "psychemedia"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-01-28T17:23:28Z", "updated_at": "2019-04-14T15:29:33Z", "closed_at": "2019-02-23T21:09:03Z", "author_association": "NONE", "pull_request": null, "body": "Irrrespective of whether using column names containing a space or - character is good practice, SQLite does allow it, but `sqlite-utils` throws an error in the following cases:\r\n\r\n```python\r\nfrom sqlite_utils import Database\r\n\r\ndbname = 'test.db'\r\nDB = Database(sqlite3.connect(dbname))\r\n\r\nimport pandas as pd\r\ndf = pd.DataFrame({'col1':range(3), 'col2':range(3)})\r\n\r\n#Convert pandas dataframe to appropriate list/dict format\r\nDB['test1'].insert_all( df.to_dict(orient='records') )\r\n#Works fine\r\n```\r\n\r\nHowever:\r\n\r\n```python\r\ndf = pd.DataFrame({'col 1':range(3), 'col2':range(3)})\r\nDB['test1'].insert_all(df.to_dict(orient='records'))\r\n```\r\n\r\nthrows:\r\n\r\n```\r\n---------------------------------------------------------------------------\r\nOperationalError Traceback (most recent call last)\r\n in ()\r\n 1 import pandas as pd\r\n 2 df = pd.DataFrame({'col 1':range(3), 'col2':range(3)})\r\n----> 3 DB['test1'].insert_all(df.to_dict(orient='records'))\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)\r\n 327 jsonify_if_needed(record.get(key, None)) for key in all_columns\r\n 328 )\r\n--> 329 result = self.db.conn.execute(sql, values)\r\n 330 self.db.conn.commit()\r\n 331 self.last_id = result.lastrowid\r\n\r\nOperationalError: near \"1\": syntax error\r\n```\r\n\r\nand:\r\n\r\n```python\r\ndf = pd.DataFrame({'col-1':range(3), 'col2':range(3)})\r\nDB['test1'].upsert_all(df.to_dict(orient='records'))\r\n```\r\n\r\nresults in:\r\n\r\n```\r\n---------------------------------------------------------------------------\r\nOperationalError Traceback (most recent call last)\r\n in ()\r\n 1 import pandas as pd\r\n 2 df = pd.DataFrame({'col-1':range(3), 'col2':range(3)})\r\n----> 3 DB['test1'].insert_all(df.to_dict(orient='records'))\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)\r\n 327 jsonify_if_needed(record.get(key, None)) for key in all_columns\r\n 328 )\r\n--> 329 result = self.db.conn.execute(sql, values)\r\n 330 self.db.conn.commit()\r\n 331 self.last_id = result.lastrowid\r\n\r\nOperationalError: near \"-\": syntax error\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 648245071, "node_id": "MDU6SXNzdWU2NDgyNDUwNzE=", "number": 8, "title": "Error thrown: table photos has no column named hasSticker", "user": {"value": 18504, "label": "harperreed"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-06-30T14:54:37Z", "updated_at": "2020-10-12T20:35:06Z", "closed_at": "2020-10-12T20:25:24Z", "author_association": "NONE", "pull_request": null, "body": "While running `swarm-to-sqlite` it throws an error:\r\n\r\n harper@:~/dogsheep/swarm$ swarm-to-sqlite checkins.db --save=checkins.json\r\n Please provide your Foursquare OAuth token:\r\n Importing 8127 checkins [#################-------------------] 49% 00:01:52\r\n Traceback (most recent call last):\r\n File \"/home/harper/.local/bin/swarm-to-sqlite\", line 11, in \r\n sys.exit(cli())\r\n File \"/home/harper/.local/lib/python3.6/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/cli.py\", line 73, in cli\r\n save_checkin(checkin, db)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/utils.py\", line 94, in save_checkin\r\n photos_table.insert(photo, replace=True)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/db.py\", line 963, in insert\r\n alter = self.value_or_default(\"alter\", alter)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/db.py\", line 1142, in insert_all\r\n def upsert_all(\r\n sqlite3.OperationalError: table photos has no column named hasSticker\r\n\r\nWhere should i dig in?", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 797728929, "node_id": "MDU6SXNzdWU3OTc3Mjg5Mjk=", "number": 8, "title": "QUESTION: extract full text", "user": {"value": 417363, "label": "darribas"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-01-31T14:50:10Z", "updated_at": "2021-01-31T14:50:10Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "This may be solved or a feature already, but I couldn't figure it out, is it possible to extract and store also full text from the saved pages? The same way that Pocket parses the text, it'd be amazing to be able to store (and thus make searchable later) the text.\r\n\r\nThank you very much for the project, it's such an amazing idea! ", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 927385540, "node_id": "MDU6SXNzdWU5MjczODU1NDA=", "number": 8, "title": "any guidance / experience on imessage-to-sqlite ?", "user": {"value": 2675621, "label": "Casyfill"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-06-22T15:46:16Z", "updated_at": "2021-06-22T15:46:16Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1617823309, "node_id": "I_kwDOJHON9s5gbgZN", "number": 8, "title": "Increase performance using macnotesapp", "user": {"value": 41546558, "label": "RhetTbull"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-03-09T18:51:05Z", "updated_at": "2023-03-14T22:00:22Z", "closed_at": "2023-03-14T22:00:21Z", "author_association": "NONE", "pull_request": null, "body": "Neat project! You can probably increase performance using my python interface to Notes, [macnotesapp](https://github.com/RhetTbull/macnotesapp), which uses Scripting Bridge and bulk queries for much better performance than AppleScript.\r\n\r\nAnother related project is [PyXA](https://github.com/SKaplanOfficial/PyXA) which uses Scripting Bridge to access Notes (and many other apps) and can return all the notes at once as opposed to calling AppleScript for each note. \r\n\r\nmacnotesapp allows you to access multiple accounts and folders as well.\r\n\r\n```python\r\nfrom macnotesapp import NotesApp\r\n\r\n# NotesApp() provides interface to Notes.app\r\nnotesapp = NotesApp()\r\n\r\n# Get list of notes (Note objects for each note)\r\nnotes = notesapp.notes()\r\nnote = notes[0]\r\nprint(\r\n note.id,\r\n note.account,\r\n note.folder,\r\n note.name,\r\n note.body,\r\n note.plaintext,\r\n note.password_protected,\r\n)\r\n\r\nprint(note.asdict())\r\n\r\n```", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 472429048, "node_id": "MDU6SXNzdWU0NzI0MjkwNDg=", "number": 9, "title": "Too many SQL variables", "user": {"value": 166463, "label": "tholo"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-07-24T18:24:17Z", "updated_at": "2019-07-26T10:01:05Z", "closed_at": "2019-07-26T10:01:05Z", "author_association": "NONE", "pull_request": null, "body": "Decided to try importing my data, and ran into this:\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/Users/tholo/Source/health/bin/healthkit-to-sqlite\", line 10, in \r\n sys.exit(cli())\r\n File \"/Users/tholo/Source/health/lib/python3.7/site-packages/click/core.py\", line 764, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/tholo/Source/health/lib/python3.7/site-packages/click/core.py\", line 717, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/tholo/Source/health/lib/python3.7/site-packages/click/core.py\", line 956, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/tholo/Source/health/lib/python3.7/site-packages/click/core.py\", line 555, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/tholo/Source/health/lib/python3.7/site-packages/healthkit_to_sqlite/cli.py\", line 50, in cli\r\n convert_xml_to_sqlite(fp, db, progress_callback=bar.update)\r\n File \"/Users/tholo/Source/health/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py\", line 41, in convert_xml_to_sqlite\r\n write_records(records, db)\r\n File \"/Users/tholo/Source/health/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py\", line 80, in write_records\r\n column_order=[\"startDate\", \"endDate\", \"value\", \"unit\"],\r\n File \"/Users/tholo/Source/health/lib/python3.7/site-packages/sqlite_utils/db.py\", line 911, in insert_all\r\n result = self.db.conn.execute(sql, values)\r\nsqlite3.OperationalError: too many SQL variables\r\n```\r\n\r\nAdded some debug output in sqlite_utils/db.py, which resulted in:\r\n\r\n```\r\n INSERT INTO [rBodyMassIndex] ([creationDate], [endDate], [metadata_HKWasUserEntered], [metadata_Health Mate App Version], [metadata_Modified Date], [metadata_Withings Link], [metadata_Withings User Identifier], [sourceName], [sourceVersion], [startDate], [unit], [value]) VALUES\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ,\r\n (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\r\n ;\r\n```\r\n\r\nwith the attached data:\r\n\r\n```\r\n['2019-06-27 22:55:10 -0700', '2011-06-22 21:05:53 -0700', '0', '4.4.2', '2011-06-23 04:05:53 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1308801953&type=1', '301293', 'Health Mate', '4040200', '2011-06-22 21:05:53 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2011-06-23 09:36:27 -0700', '0', '4.4.2', '2011-06-23 16:36:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1308846987&type=1', '301293', 'Health Mate', '4040200', '2011-06-23 09:36:27 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2011-06-23 23:54:07 -0700', '0', '4.4.2', '2011-06-24 06:55:19 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1308898447&type=1', '301293', 'Health Mate', '4040200', '2011-06-23 23:54:07 -0700', 'count', '30.679', '2019-06-27 22:55:10 -0700', '2011-06-24 09:13:40 -0700', '0', '4.4.2', '2011-06-24 16:14:35 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1308932020&type=1', '301293', 'Health Mate', '4040200', '2011-06-24 09:13:40 -0700', 'count', '30.3549', '2019-06-27 22:55:10 -0700', '2011-06-25 08:30:08 -0700', '0', '4.4.2', '2011-06-25 15:30:49 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309015808&type=1', '301293', 'Health Mate', '4040200', '2011-06-25 08:30:08 -0700', 'count', '30.3395', '2019-06-27 22:55:10 -0700', '2011-06-26 07:47:51 -0700', '0', '4.4.2', '2011-06-26 14:48:27 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309099671&type=1', '301293', 'Health Mate', '4040200', '2011-06-26 07:47:51 -0700', 'count', '30.2315', '2019-06-27 22:55:10 -0700', '2011-06-28 08:48:26 -0700', '0', '4.4.2', '2011-06-28 15:49:13 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309276106&type=1', '301293', 'Health Mate', '4040200', '2011-06-28 08:48:26 -0700', 'count', '30.0617', '2019-06-27 22:55:10 -0700', '2011-06-29 09:21:16 -0700', '0', '4.4.2', '2011-06-29 16:21:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309364476&type=1', '301293', 'Health Mate', '4040200', '2011-06-29 09:21:16 -0700', 'count', '29.9537', '2019-06-27 22:55:10 -0700', '2011-06-30 08:41:46 -0700', '0', '4.4.2', '2011-06-30 15:42:30 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309448506&type=1', '301293', 'Health Mate', '4040200', '2011-06-30 08:41:46 -0700', 'count', '29.8302', '2019-06-27 22:55:10 -0700', '2011-07-01 09:05:28 -0700', '0', '4.4.2', '2011-07-01 16:06:24 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309536328&type=1', '301293', 'Health Mate', '4040200', '2011-07-01 09:05:28 -0700', 'count', '29.8611', '2019-06-27 22:55:10 -0700', '2011-07-02 08:58:50 -0700', '0', '4.4.2', '2011-07-02 15:59:40 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309622330&type=1', '301293', 'Health Mate', '4040200', '2011-07-02 08:58:50 -0700', 'count', '29.8765', '2019-06-27 22:55:10 -0700', '2011-07-04 09:33:43 -0700', '0', '4.4.2', '2011-07-04 16:34:19 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309797223&type=1', '301293', 'Health Mate', '4040200', '2011-07-04 09:33:43 -0700', 'count', '30.0309', '2019-06-27 22:55:10 -0700', '2011-07-06 09:40:23 -0700', '0', '4.4.2', '2011-07-06 16:41:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309970423&type=1', '301293', 'Health Mate', '4040200', '2011-07-06 09:40:23 -0700', 'count', '30.1852', '2019-06-27 22:55:10 -0700', '2011-07-08 08:08:48 -0700', '0', '4.4.2', '2011-07-08 15:09:51 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310137728&type=1', '301293', 'Health Mate', '4040200', '2011-07-08 08:08:48 -0700', 'count', '30.0309', '2019-06-27 22:55:10 -0700', '2011-07-09 08:31:05 -0700', '0', '4.4.2', '2011-07-09 15:31:48 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310225465&type=1', '301293', 'Health Mate', '4040200', '2011-07-09 08:31:05 -0700', 'count', '29.9537', '2019-06-27 22:55:10 -0700', '2011-07-10 08:14:36 -0700', '0', '4.4.2', '2011-07-10 15:15:12 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310310876&type=1', '301293', 'Health Mate', '4040200', '2011-07-10 08:14:36 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2011-07-12 07:55:21 -0700', '0', '4.4.2', '2011-07-12 14:55:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310482521&type=1', '301293', 'Health Mate', '4040200', '2011-07-12 07:55:21 -0700', 'count', '30.108', '2019-06-27 22:55:10 -0700', '2011-07-13 08:48:05 -0700', '0', '4.4.2', '2011-07-13 15:48:42 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310572085&type=1', '301293', 'Health Mate', '4040200', '2011-07-13 08:48:05 -0700', 'count', '30', '2019-06-27 22:55:10 -0700', '2011-07-14 09:05:16 -0700', '0', '4.4.2', '2011-07-14 16:05:57 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310659516&type=1', '301293', 'Health Mate', '4040200', '2011-07-14 09:05:16 -0700', 'count', '29.9074', '2019-06-27 22:55:10 -0700', '2011-07-15 07:09:56 -0700', '0', '4.4.2', '2011-07-15 14:10:35 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310738996&type=1', '301293', 'Health Mate', '4040200', '2011-07-15 07:09:56 -0700', 'count', '29.9537', '2019-06-27 22:55:10 -0700', '2011-07-16 09:26:04 -0700', '0', '4.4.2', '2011-07-16 16:26:44 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310833564&type=1', '301293', 'Health Mate', '4040200', '2011-07-16 09:26:04 -0700', 'count', '29.7531', '2019-06-27 22:55:10 -0700', '2011-07-17 09:52:59 -0700', '0', '4.4.2', '2011-07-17 16:53:38 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310921579&type=1', '301293', 'Health Mate', '4040200', '2011-07-17 09:52:59 -0700', 'count', '29.8765', '2019-06-27 22:55:10 -0700', '2011-07-19 08:56:16 -0700', '0', '4.4.2', '2011-07-19 15:57:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311090976&type=1', '301293', 'Health Mate', '4040200', '2011-07-19 08:56:16 -0700', 'count', '29.7685', '2019-06-27 22:55:10 -0700', '2011-07-21 08:21:20 -0700', '0', '4.4.2', '2011-07-21 15:22:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311261680&type=1', '301293', 'Health Mate', '4040200', '2011-07-21 08:21:20 -0700', 'count', '29.7685', '2019-06-27 22:55:10 -0700', '2011-07-23 08:49:56 -0700', '0', '4.4.2', '2011-07-23 15:50:40 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311436196&type=1', '301293', 'Health Mate', '4040200', '2011-07-23 08:49:56 -0700', 'count', '29.7222', '2019-06-27 22:55:10 -0700', '2011-07-24 09:17:35 -0700', '0', '4.4.2', '2011-07-24 16:18:14 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311524255&type=1', '301293', 'Health Mate', '4040200', '2011-07-24 09:17:35 -0700', 'count', '29.5833', '2019-06-27 22:55:10 -0700', '2011-07-25 07:51:55 -0700', '0', '4.4.2', '2011-07-25 14:52:48 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311605515&type=1', '301293', 'Health Mate', '4040200', '2011-07-25 07:51:55 -0700', 'count', '29.5525', '2019-06-27 22:55:10 -0700', '2011-08-06 10:04:05 -0700', '0', '4.4.2', '2011-08-06 17:04:47 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1312650245&type=1', '301293', 'Health Mate', '4040200', '2011-08-06 10:04:05 -0700', 'count', '29.7377', '2019-06-27 22:55:10 -0700', '2011-08-08 07:52:22 -0700', '0', '4.4.2', '2011-08-08 14:53:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1312815142&type=1', '301293', 'Health Mate', '4040200', '2011-08-08 07:52:22 -0700', 'count', '29.6605', '2019-06-27 22:55:10 -0700', '2011-08-10 07:57:30 -0700', '0', '4.4.2', '2011-08-10 14:58:12 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1312988250&type=1', '301293', 'Health Mate', '4040200', '2011-08-10 07:57:30 -0700', 'count', '29.7531', '2019-06-27 22:55:10 -0700', '2011-08-12 07:51:14 -0700', '0', '4.4.2', '2011-08-12 14:51:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1313160674&type=1', '301293', 'Health Mate', '4040200', '2011-08-12 07:51:14 -0700', 'count', '29.6914', '2019-06-27 22:55:10 -0700', '2011-08-13 07:45:28 -0700', '0', '4.4.2', '2011-08-13 14:46:08 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1313246728&type=1', '301293', 'Health Mate', '4040200', '2011-08-13 07:45:28 -0700', 'count', '29.5833', '2019-06-27 22:55:10 -0700', '2011-08-17 09:06:20 -0700', '0', '4.4.2', '2011-08-17 16:07:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1313597180&type=1', '301293', 'Health Mate', '4040200', '2011-08-17 09:06:20 -0700', 'count', '29.5679', '2019-06-27 22:55:10 -0700', '2011-08-22 08:28:08 -0700', '0', '4.4.2', '2011-08-22 15:28:57 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1314026888&type=1', '301293', 'Health Mate', '4040200', '2011-08-22 08:28:08 -0700', 'count', '29.9846', '2019-06-27 22:55:10 -0700', '2011-08-25 08:59:30 -0700', '0', '4.4.2', '2011-08-25 16:00:15 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1314287970&type=1', '301293', 'Health Mate', '4040200', '2011-08-25 08:59:30 -0700', 'count', '29.9691', '2019-06-27 22:55:10 -0700', '2011-08-30 08:13:59 -0700', '0', '4.4.2', '2011-08-30 15:46:08 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1314717239&type=1', '301293', 'Health Mate', '4040200', '2011-08-30 08:13:59 -0700', 'count', '29.784', '2019-06-27 22:55:10 -0700', '2011-09-12 08:47:51 -0700', '0', '4.4.2', '2011-09-12 15:48:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1315842471&type=1', '301293', 'Health Mate', '4040200', '2011-09-12 08:47:51 -0700', 'count', '29.7377', '2019-06-27 22:55:10 -0700', '2011-09-13 09:17:27 -0700', '0', '4.4.2', '2011-09-13 16:48:30 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1315930647&type=1', '301293', 'Health Mate', '4040200', '2011-09-13 09:17:27 -0700', 'count', '29.7531', '2019-06-27 22:55:10 -0700', '2011-10-01 09:12:20 -0700', '0', '4.4.2', '2011-10-01 16:13:00 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1317485540&type=1', '301293', 'Health Mate', '4040200', '2011-10-01 09:12:20 -0700', 'count', '29.8148', '2019-06-27 22:55:10 -0700', '2011-10-11 11:14:11 -0700', '0', '4.4.2', '2011-10-11 18:15:14 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1318356851&type=1', '301293', 'Health Mate', '4040200', '2011-10-11 11:14:11 -0700', 'count', '29.7377', '2019-06-27 22:55:10 -0700', '2011-10-16 09:29:47 -0700', '0', '4.4.2', '2011-10-16 16:30:39 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1318782587&type=1', '301293', 'Health Mate', '4040200', '2011-10-16 09:29:47 -0700', 'count', '29.6914', '2019-06-27 22:55:10 -0700', '2011-10-19 09:21:44 -0700', '0', '4.4.2', '2011-10-19 16:22:25 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1319041304&type=1', '301293', 'Health Mate', '4040200', '2011-10-19 09:21:44 -0700', 'count', '29.7685', '2019-06-27 22:55:10 -0700', '2011-10-24 07:04:22 -0700', '0', '4.4.2', '2011-10-24 14:05:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1319465062&type=1', '301293', 'Health Mate', '4040200', '2011-10-24 07:04:22 -0700', 'count', '29.5988', '2019-06-27 22:55:10 -0700', '2011-11-07 09:33:17 -0700', '0', '4.4.2', '2011-11-07 16:33:58 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1320683597&type=1', '301293', 'Health Mate', '4040200', '2011-11-07 09:33:17 -0700', 'count', '29.8611', '2019-06-27 22:55:10 -0700', '2011-11-10 07:59:03 -0700', '0', '4.4.2', '2011-11-10 14:59:48 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1320937143&type=1', '301293', 'Health Mate', '4040200', '2011-11-10 07:59:03 -0700', 'count', '29.9383', '2019-06-27 22:55:10 -0700', '2011-11-13 09:28:31 -0700', '0', '4.4.2', '2011-11-13 16:29:20 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1321201711&type=1', '301293', 'Health Mate', '4040200', '2011-11-13 09:28:31 -0700', 'count', '29.7531', '2019-06-27 22:55:10 -0700', '2011-11-21 08:45:06 -0700', '0', '4.4.2', '2011-11-21 15:46:04 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1321890306&type=1', '301293', 'Health Mate', '4040200', '2011-11-21 08:45:06 -0700', 'count', '29.9691', '2019-06-27 22:55:10 -0700', '2011-11-23 09:55:44 -0700', '0', '4.4.2', '2011-11-23 16:56:18 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1322067344&type=1', '301293', 'Health Mate', '4040200', '2011-11-23 09:55:44 -0700', 'count', '29.8302', '2019-06-27 22:55:10 -0700', '2011-11-29 09:50:44 -0700', '0', '4.4.2', '2011-11-29 16:51:31 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1322585444&type=1', '301293', 'Health Mate', '4040200', '2011-11-29 09:50:44 -0700', 'count', '30.1698', '2019-06-27 22:55:10 -0700', '2011-11-30 11:13:21 -0700', '0', '4.4.2', '2011-11-30 18:14:14 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1322676801&type=1', '301293', 'Health Mate', '4040200', '2011-11-30 11:13:21 -0700', 'count', '30.0617', '2019-06-27 22:55:10 -0700', '2011-12-04 10:24:36 -0700', '0', '4.4.2', '2011-12-04 17:25:24 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1323019476&type=1', '301293', 'Health Mate', '4040200', '2011-12-04 10:24:36 -0700', 'count', '29.9691', '2019-06-27 22:55:10 -0700', '2011-12-10 09:22:18 -0700', '0', '4.4.2', '2011-12-10 16:23:07 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1323534138&type=1', '301293', 'Health Mate', '4040200', '2011-12-10 09:22:18 -0700', 'count', '29.9537', '2019-06-27 22:55:10 -0700', '2011-12-26 10:36:42 -0700', '0', '4.4.2', '2011-12-26 17:37:31 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1324921002&type=1', '301293', 'Health Mate', '4040200', '2011-12-26 10:36:42 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2012-01-11 11:24:13 -0700', '0', '4.4.2', '2012-01-11 18:25:04 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1326306253&type=1', '301293', 'Health Mate', '4040200', '2012-01-11 11:24:13 -0700', 'count', '29.8302', '2019-06-27 22:55:10 -0700', '2012-01-15 10:17:09 -0700', '0', '4.4.2', '2012-01-15 17:17:51 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1326647829&type=1', '301293', 'Health Mate', '4040200', '2012-01-15 10:17:09 -0700', 'count', '29.8302', '2019-06-27 22:55:10 -0700', '2012-01-19 09:24:32 -0700', '0', '4.4.2', '2012-01-19 16:25:21 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1326990272&type=1', '301293', 'Health Mate', '4040200', '2012-01-19 09:24:32 -0700', 'count', '29.7994', '2019-06-27 22:55:10 -0700', '2012-01-29 10:26:13 -0700', '0', '4.4.2', '2012-01-29 17:26:52 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1327857973&type=1', '301293', 'Health Mate', '4040200', '2012-01-29 10:26:13 -0700', 'count', '30.0154', '2019-06-27 22:55:10 -0700', '2012-02-03 10:13:28 -0700', '0', '4.4.2', '2012-02-03 17:15:01 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1328289208&type=1', '301293', 'Health Mate', '4040200', '2012-02-03 10:13:28 -0700', 'count', '29.8457', '2019-06-27 22:55:10 -0700', '2012-02-12 09:23:01 -0700', '0', '4.4.2', '2012-02-12 16:23:53 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1329063781&type=1', '301293', 'Health Mate', '4040200', '2012-02-12 09:23:01 -0700', 'count', '30.1235', '2019-06-27 22:55:10 -0700', '2012-03-03 09:26:06 -0700', '0', '4.4.2', '2012-03-03 16:26:54 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1330791966&type=1', '301293', 'Health Mate', '4040200', '2012-03-03 09:26:06 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2012-03-11 11:23:15 -0700', '0', '4.4.2', '2012-03-11 18:24:16 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1331490195&type=1', '301293', 'Health Mate', '4040200', '2012-03-11 11:23:15 -0700', 'count', '30.2161', '2019-06-27 22:55:10 -0700', '2012-03-16 09:39:36 -0700', '0', '4.4.2', '2012-03-16 16:40:20 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1331915976&type=1', '301293', 'Health Mate', '4040200', '2012-03-16 09:39:36 -0700', 'count', '30.2778', '2019-06-27 22:55:10 -0700', '2012-03-21 08:33:07 -0700', '0', '4.4.2', '2012-03-21 15:34:00 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1332343987&type=1', '301293', 'Health Mate', '4040200', '2012-03-21 08:33:07 -0700', 'count', '30.1389', '2019-06-27 22:55:10 -0700', '2012-04-11 08:49:34 -0700', '0', '4.4.2', '2012-04-11 15:50:18 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1334159374&type=1', '301293', 'Health Mate', '4040200', '2012-04-11 08:49:34 -0700', 'count', '30.0154', '2019-06-27 22:55:10 -0700', '2012-04-13 08:32:06 -0700', '0', '4.4.2', '2012-04-13 15:32:49 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1334331126&type=1', '301293', 'Health Mate', '4040200', '2012-04-13 08:32:06 -0700', 'count', '29.9383', '2019-06-27 22:55:10 -0700', '2012-04-20 08:21:38 -0700', '0', '4.4.2', '2012-04-20 15:52:45 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1334935298&type=1', '301293', 'Health Mate', '4040200', '2012-04-20 08:21:38 -0700', 'count', '30.2006', '2019-06-27 22:55:10 -0700', '2012-04-25 09:00:01 -0700', '0', '4.4.2', '2012-04-25 16:00:42 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1335369601&type=1', '301293', 'Health Mate', '4040200', '2012-04-25 09:00:01 -0700', 'count', '30.2006', '2019-06-27 22:55:10 -0700', '2012-05-04 11:10:18 -0700', '0', '4.4.2', '2012-05-04 18:10:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1336155018&type=1', '301293', 'Health Mate', '4040200', '2012-05-04 11:10:18 -0700', 'count', '30.4321', '2019-06-27 22:55:10 -0700', '2012-05-12 09:35:00 -0700', '0', '4.4.2', '2012-05-12 16:35:43 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1336840500&type=1', '301293', 'Health Mate', '4040200', '2012-05-12 09:35:00 -0700', 'count', '30.1235', '2019-06-27 22:55:10 -0700', '2012-05-22 09:27:53 -0700', '0', '4.4.2', '2012-05-22 16:28:37 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1337704073&type=1', '301293', 'Health Mate', '4040200', '2012-05-22 09:27:53 -0700', 'count', '30.4167', '2019-06-27 22:55:10 -0700', '2012-05-31 09:23:16 -0700', '0', '4.4.2', '2012-05-31 16:24:04 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1338481396&type=1', '301293', 'Health Mate', '4040200', '2012-05-31 09:23:16 -0700', 'count', '30.2006', '2019-06-27 22:55:10 -0700', '2012-06-08 09:29:07 -0700', '0', '4.4.2', '2012-06-08 16:29:52 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1339172947&type=1', '301293', 'Health Mate', '4040200', '2012-06-08 09:29:07 -0700', 'count', '30.5247', '2019-06-27 22:55:10 -0700', '2012-06-21 08:07:33 -0700', '0', '4.4.2', '2012-06-21 15:08:20 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1340291253&type=1', '301293', 'Health Mate', '4040200', '2012-06-21 08:07:33 -0700', 'count', '30.5864', '2019-06-27 22:55:10 -0700', '2012-08-08 10:02:22 -0700', '0', '4.4.2', '2012-08-08 17:03:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1344445342&type=1', '301293', 'Health Mate', '4040200', '2012-08-08 10:02:22 -0700', 'count', '30.6636', '2019-06-27 22:55:10 -0700', '2012-08-17 09:11:32 -0700', '0', '4.4.2', '2012-08-17 16:42:05 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1345219892&type=1', '301293', 'Health Mate', '4040200', '2012-08-17 09:11:32 -0700', 'count', '30.8796', '2019-06-27 22:55:10 -0700', '2012-09-10 08:27:21 -0700', '0', '4.4.2', '2012-09-10 15:28:07 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1347290841&type=1', '301293', 'Health Mate', '4040200', '2012-09-10 08:27:21 -0700', 'count', '31.034', '2019-06-27 22:55:10 -0700', '2012-09-17 08:35:33 -0700', '0', '4.4.2', '2012-09-17 15:35:33 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1347896133&type=1', '301293', 'Health Mate', '4040200', '2012-09-17 08:35:33 -0700', 'count', '30.7099', '2019-06-27 22:55:10 -0700', '2012-09-26 08:59:46 -0700', '0', '4.4.2', '2012-09-26 16:13:18 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1348675186&type=1', '301293', 'Health Mate', '4040200', '2012-09-26 08:59:46 -0700', 'count', '30.679', '2019-06-27 22:55:10 -0700', '2012-10-18 08:51:16 -0700', '0', '4.4.2', '2012-10-18 15:51:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1350575476&type=1', '301293', 'Health Mate', '4040200', '2012-10-18 08:51:16 -0700', 'count', '30.7716', '2019-06-27 22:55:10 -0700', '2012-11-15 08:54:57 -0700', '0', '4.4.2', '2012-11-15 15:55:58 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1352994897&type=1', '301293', 'Health Mate', '4040200', '2012-11-15 08:54:57 -0700', 'count', '31.0802', '2019-06-27 22:55:10 -0700', '2012-12-17 09:13:40 -0700', '0', '4.4.2', '2012-12-17 16:20:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1355760820&type=1', '301293', 'Health Mate', '4040200', '2012-12-17 09:13:40 -0700', 'count', '29.784', '2019-06-27 22:55:10 -0700', '2012-12-19 11:09:55 -0700', '0', '4.4.2', '2012-12-19 18:10:37 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1355940595&type=1', '301293', 'Health Mate', '4040200', '2012-12-19 11:09:55 -0700', 'count', '29.6914', '2019-06-27 22:55:10 -0700', '2012-12-25 10:37:41 -0700', '0', '4.4.2', '2012-12-25 17:38:25 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1356457061&type=1', '301293', 'Health Mate', '4040200', '2012-12-25 10:37:41 -0700', 'count', '29.8765', '2019-06-27 22:55:10 -0700', '2013-01-01 10:44:02 -0700', '0', '4.4.2', '2013-01-01 17:44:46 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1357062242&type=1', '301293', 'Health Mate', '4040200', '2013-01-01 10:44:02 -0700', 'count', '30.0772', '2019-06-27 22:55:10 -0700', '2013-01-15 09:10:46 -0700', '0', '4.4.2', '2013-01-15 16:11:28 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1358266246&type=1', '301293', 'Health Mate', '4040200', '2013-01-15 09:10:46 -0700', 'count', '29.9691', '2019-06-27 22:55:10 -0700', '2013-01-20 11:03:39 -0700', '0', '4.4.2', '2013-01-20 18:04:22 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1358705019&type=1', '301293', 'Health Mate', '4040200', '2013-01-20 11:03:39 -0700', 'count', '30.108', '2019-06-27 22:55:10 -0700', '2013-01-30 08:56:30 -0700', '0', '4.4.2', '2013-01-30 15:57:14 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1359561390&type=1', '301293', 'Health Mate', '4040200', '2013-01-30 08:56:30 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2013-02-04 11:02:35 -0700', '0', '4.4.2', '2013-02-04 18:03:25 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1360000955&type=1', '301293', 'Health Mate', '4040200', '2013-02-04 11:02:35 -0700', 'count', '29.8148', '2019-06-27 22:55:10 -0700', '2013-02-07 09:07:06 -0700', '0', '4.4.2', '2013-02-07 16:07:49 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1360253226&type=1', '301293', 'Health Mate', '4040200', '2013-02-07 09:07:06 -0700', 'count', '30.1389', '2019-06-27 22:55:10 -0700', '2013-02-19 08:49:57 -0700', '0', '4.4.2', '2013-02-19 15:50:39 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1361288997&type=1', '301293', 'Health Mate', '4040200', '2013-02-19 08:49:57 -0700', 'count', '30.1235', '2019-06-27 22:55:10 -0700', '2013-03-02 11:20:54 -0700', '0', '4.4.2', '2013-03-02 18:21:38 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1362248454&type=1', '301293', 'Health Mate', '4040200', '2013-03-02 11:20:54 -0700', 'count', '30', '2019-06-27 22:55:10 -0700', '2013-04-23 08:05:30 -0700', '0', '4.4.2', '2013-04-23 15:06:59 +0000', 'withings-bd2://timeline/measure?user \"\"\"\r\nid=301293&date=1366729530&type=1', '301293', 'Health Mate', '4040200', '2013-04-23 08:05:30 -0700', 'count', '30.5247', '2019-06-27 22:55:10 -0700', '2013-05-09 09:49:18 -0700', '0', '4.4.2', '2013-05-09 16:50:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1368118158&type=1', '301293', 'Health Mate', '4040200', '2013-05-09 09:49:18 -0700', 'count', '30.4167', '2019-06-27 22:55:10 -0700', '2013-06-09 09:28:47 -0700', '0', '4.4.2', '2013-06-09 16:29:30 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1370795327&type=1', '301293', 'Health Mate', '4040200', '2013-06-09 09:28:47 -0700', 'count', '30.8333', '2019-06-27 22:55:10 -0700', '2013-07-09 08:00:17 -0700', '0', '4.4.2', '2013-07-09 15:01:00 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1373382017&type=1', '301293', 'Health Mate', '4040200', '2013-07-09 08:00:17 -0700', 'count', '30.8179', '2019-06-27 22:55:10 -0700', '2013-07-28 09:16:55 -0700', '0', '4.4.2', '2013-07-28 16:17:39 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1375028215&type=1', '301293', 'Health Mate', '4040200', '2013-07-28 09:16:55 -0700', 'count', '30.5556', '2019-06-27 22:55:10 -0700', '2013-09-13 09:22:19 -0700', '0', '4.4.2', '2013-09-13 16:23:08 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1379089339&type=1', '301293', 'Health Mate', '4040200', '2013-09-13 09:22:19 -0700', 'count', '30.9568', '2019-06-27 22:55:10 -0700', '2013-09-24 08:08:23 -0700', '0', '4.4.2', '2013-09-24 15:09:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1380035303&type=1', '301293', 'Health Mate', '4040200', '2013-09-24 08:08:23 -0700', 'count', '31.4352', '2019-06-27 22:55:10 -0700', '2013-10-01 08:15:13 -0700', '0', '4.4.2', '2013-10-01 15:15:57 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1380640513&type=1', '301293', 'Health Mate', '4040200', '2013-10-01 08:15:13 -0700', 'count', '31.2037', '2019-06-27 22:55:10 -0700', '2013-10-23 09:31:25 -0700', '0', '4.4.2', '2013-10-23 16:32:13 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1382545885&type=1', '301293', 'Health Mate', '4040200', '2013-10-23 09:31:25 -0700', 'count', '31.8056']\r\n```", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 801780625, "node_id": "MDU6SXNzdWU4MDE3ODA2MjU=", "number": 9, "title": "SSL Error", "user": {"value": 12669260, "label": "jfeiwell"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-02-05T02:12:56Z", "updated_at": "2021-02-07T18:45:04Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Here's the error I get when running `pip install pocket-to-sqlite`: \r\n\r\n```\r\n Could not fetch URL https://pypi.python.org/simple/pocket-to-sqlite/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661) - skipping\r\n Could not find a version that satisfies the requirement pocket-to-sqlite (from versions: )\r\nNo matching distribution found for pocket-to-sqlite\r\n```\r\n\r\nDoes this require python 3? ", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 411066700, "node_id": "MDU6SXNzdWU0MTEwNjY3MDA=", "number": 10, "title": "Error in upsert if column named 'order'", "user": {"value": 82988, "label": "psychemedia"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-02-16T12:05:18Z", "updated_at": "2019-02-24T16:55:38Z", "closed_at": "2019-02-24T16:55:37Z", "author_association": "NONE", "pull_request": null, "body": "The following works fine:\r\n```\r\nconnX = sqlite3.connect('DELME.db', timeout=10)\r\n\r\ndfX=pd.DataFrame({'col1':range(3),'col2':range(3)})\r\nDBX = Database(connX)\r\nDBX['test'].upsert_all(dfX.to_dict(orient='records'))\r\n```\r\n\r\nBut if a column is named `order`:\r\n```\r\nconnX = sqlite3.connect('DELME.db', timeout=10)\r\n\r\ndfX=pd.DataFrame({'order':range(3),'col2':range(3)})\r\nDBX = Database(connX)\r\nDBX['test'].upsert_all(dfX.to_dict(orient='records'))\r\n```\r\n\r\nit throws an error:\r\n\r\n```\r\n---------------------------------------------------------------------------\r\nOperationalError Traceback (most recent call last)\r\n in \r\n 3 dfX=pd.DataFrame({'order':range(3),'col2':range(3)})\r\n 4 DBX = Database(connX)\r\n----> 5 DBX['test'].upsert_all(dfX.to_dict(orient='records'))\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order)\r\n 347 foreign_keys=foreign_keys,\r\n 348 upsert=True,\r\n--> 349 column_order=column_order,\r\n 350 )\r\n 351 \r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)\r\n 327 jsonify_if_needed(record.get(key, None)) for key in all_columns\r\n 328 )\r\n--> 329 result = self.db.conn.execute(sql, values)\r\n 330 self.db.conn.commit()\r\n 331 self.last_id = result.lastrowid\r\n\r\nOperationalError: near \"order\": syntax error\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/10/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 770712149, "node_id": "MDExOlB1bGxSZXF1ZXN0NTQyNDA2OTEw", "number": 10, "title": "BugFix for encoding and not update info.", "user": {"value": 1277270, "label": "riverzhou"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-12-18T08:58:54Z", "updated_at": "2021-02-11T22:37:56Z", "closed_at": "2021-02-11T22:37:56Z", "author_association": "NONE", "pull_request": "dogsheep/evernote-to-sqlite/pulls/10", "body": "Bugfix 1:\r\n\r\nTraceback (most recent call last):\r\n File \"d:\\anaconda3\\lib\\runpy.py\", line 194, in _run_module_as_main\r\n return _run_code(code, main_globals, None,\r\n File \"d:\\anaconda3\\lib\\runpy.py\", line 87, in _run_code\r\n exec(code, run_globals)\r\n File \"D:\\Anaconda3\\Scripts\\evernote-to-sqlite.exe\\__main__.py\", line 7, in \r\n File \"d:\\anaconda3\\lib\\site-packages\\click\\core.py\", line 829, in __call__\r\n File \"d:\\anaconda3\\lib\\site-packages\\click\\core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"d:\\anaconda3\\lib\\site-packages\\click\\core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"d:\\anaconda3\\lib\\site-packages\\click\\core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"d:\\anaconda3\\lib\\site-packages\\evernote_to_sqlite\\cli.py\", line 30, in enex\r\n for tag, note in find_all_tags(fp, [\"note\"], progress_callback=bar.update):\r\n File \"d:\\anaconda3\\lib\\site-packages\\evernote_to_sqlite\\utils.py\", line 11, in find_all_tags\r\n chunk = fp.read(1024 * 1024)\r\nUnicodeDecodeError: 'gbk' codec can't decode byte 0xa4 in position 383: illegal multibyte sequence\r\n\r\nBugfix 2:\r\n\r\nTraceback (most recent call last):\r\n File \"D:\\Anaconda3\\Scripts\\evernote-to-sqlite-script.py\", line 33, in \r\n sys.exit(load_entry_point('evernote-to-sqlite==0.3', 'console_scripts', 'evernote-to-sqlite')())\r\n File \"D:\\Anaconda3\\lib\\site-packages\\click\\core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"D:\\Anaconda3\\lib\\site-packages\\click\\core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"D:\\Anaconda3\\lib\\site-packages\\click\\core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"D:\\Anaconda3\\lib\\site-packages\\click\\core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"D:\\Anaconda3\\lib\\site-packages\\click\\core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"D:\\Anaconda3\\lib\\site-packages\\evernote_to_sqlite-0.3-py3.8.egg\\evernote_to_sqlite\\cli.py\", line 31, in enex\r\n File \"D:\\Anaconda3\\lib\\site-packages\\evernote_to_sqlite-0.3-py3.8.egg\\evernote_to_sqlite\\utils.py\", line 28, in save_note\r\nAttributeError: 'NoneType' object has no attribute 'text'", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1123393829, "node_id": "I_kwDODFE5qs5C9aEl", "number": 10, "title": "sqlite3.OperationalError: no such table: main.my_activity", "user": {"value": 69208826, "label": "glxblt14"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-02-03T17:59:29Z", "updated_at": "2022-03-20T02:38:07Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hello,\r\nWhen i run the command `google-takeout-to-sqlite my-activity db.db takeout-20220203T174446Z-001.zip`, i get this error :\r\n```\r\nTraceback (most recent call last):\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\runpy.py\", line 197, in _run_module_as_main\r\n return _run_code(code, main_globals, None,\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\runpy.py\", line 87, in _run_code\r\n exec(code, run_globals)\r\n File \"C:\\Users\\julie\\AppData\\Local\\Programs\\Python\\Python39-32\\Scripts\\google-takeout-to-sqlite.exe\\__main__.py\", line 7, in \r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\google_takeout_to_sqlite\\cli.py\", line 31, in my_activity\r\n utils.save_my_activity(db, zf)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\google_takeout_to_sqlite\\utils.py\", line 19, in save_my_activity\r\n db[\"my_activity\"].create_index([\"time\"])\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\sqlite_utils\\db.py\", line 629, in create_index\r\n self.db.conn.execute(sql)\r\nsqlite3.OperationalError: no such table: main.my_activity\r\n```\r\nThank you for your help\r\nSorry for my bad English\r\nEDIT: i used the json format", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/10/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1246826792, "node_id": "I_kwDODLZ_YM5KUREo", "number": 10, "title": "When running `auth` command, don't overwrite an existing auth.json file", "user": {"value": 11887, "label": "ashanan"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-05-24T16:42:20Z", "updated_at": "2022-09-07T15:07:38Z", "closed_at": "2022-08-22T16:17:19Z", "author_association": "NONE", "pull_request": null, "body": "Ran the `auth` command in the same directory I'd previously set up an auth.json file for `twitter-to-sqlite` and it was completely overwritten. Not the biggest issue, but still unexpected. Ideally, for me, the keys would just be added to the existing file, but getting a warning and a chance to back out would be a good solution as well.", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 723838331, "node_id": "MDU6SXNzdWU3MjM4MzgzMzE=", "number": 11, "title": "export.xml file name varies with different language settings", "user": {"value": 572, "label": "jarib"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-10-17T20:07:18Z", "updated_at": "2020-10-17T21:39:15Z", "closed_at": "2020-10-17T21:14:10Z", "author_association": "NONE", "pull_request": null, "body": "The XML file exported from my phone has a Norwegian file name \u2013 `eksport.xml` \ud83d\ude44 \r\n\r\nI can work around this by unpacking the zip and using `--xml`, but then I lose the workout points. \r\n\r\nPerhaps this could be solved by `--localized-xml eksport.xml`? Alternatively just fall back to the first XML file in the root folder of the zip.\r\n\r\n", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 743400216, "node_id": "MDU6SXNzdWU3NDM0MDAyMTY=", "number": 11, "title": "Error thrown: sqlite3.OperationalError: table users has no column named lastName", "user": {"value": 61791, "label": "beaugunderson"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-11-16T01:21:18Z", "updated_at": "2021-01-18T04:35:22Z", "closed_at": "2021-01-18T04:35:22Z", "author_association": "NONE", "pull_request": null, "body": "Just installed `swarm-to-sqlite-0.3.2` and tried according to the docs:\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/swarm-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/swarm_to_sqlite/cli.py\", line 73, in cli\r\n save_checkin(checkin, db)\r\n File \"/usr/local/lib/python3.9/site-packages/swarm_to_sqlite/utils.py\", line 82, in save_checkin\r\n checkins_table.m2m(\"users\", user, m2m_table=\"likes\", pk=\"id\")\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1914, in m2m\r\n id = other_table.insert(record, pk=pk, replace=True).last_pk\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1647, in insert\r\n return self.insert_all(\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1765, in insert_all\r\n self.insert_chunk(\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1575, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 200, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.OperationalError: table users has no column named lastName\r\n```", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 792851444, "node_id": "MDU6SXNzdWU3OTI4NTE0NDQ=", "number": 11, "title": "XML parse error", "user": {"value": 3613583, "label": "dskrad"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-24T17:38:54Z", "updated_at": "2021-02-11T21:18:58Z", "closed_at": "2021-02-11T21:18:48Z", "author_association": "NONE", "pull_request": null, "body": "I am on Windows 10 using Windows Subsystem for Linux, Python 3.8. I installed evernote-to-sqlite via pipx (in a venv). I tried using enex files from the latest version of Evernote for Windows (10.6.9 which only lets you export 50 notes at a time) and from Legacy Evernote (6.25.2.9198 which lets you export all your notes at once). The enex file from latest evernote gives this error:\r\n\r\n File \"/usr/lib/python3.8/xml/etree/ElementTree.py\", line 1320, in XML parser.feed(text)\r\n xml.etree.ElementTree.ParseError: XML or text declaration not at start of entity: line 2, column 6\r\n\r\nThe enex file from Legacy Evernote gives this error:\r\n\r\n File \"/home/david/.local/pipx/venvs/evernote-to-sqlite/lib/python3.8/site-packages/evernote_to_sqlite/utils.py\", line 28, in save_note\r\n updated = note.find(\"updated\").text\r\n AttributeError: 'NoneType' object has no attribute 'text'", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1345452427, "node_id": "I_kwDODLZ_YM5QMfmL", "number": 11, "title": "-a option is used for \"--auth\" and for \"--all\"", "user": {"value": 2467, "label": "fernand0"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-08-21T10:50:48Z", "updated_at": "2022-08-21T21:11:57Z", "closed_at": "2022-08-21T21:11:57Z", "author_association": "NONE", "pull_request": null, "body": "I'm not sure which option is best, instead of -a -all.", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 951817328, "node_id": "MDU6SXNzdWU5NTE4MTczMjg=", "number": 12, "title": "403 when getting token", "user": {"value": 285352, "label": "treyhunner"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-07-23T18:43:26Z", "updated_at": "2021-10-12T18:31:57Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I tried to use https://your-foursquare-oauth-token.glitch.me/ to get my Swarm auth token and got a 403 after I clicked the Allow button:\r\n\r\n![image](https://user-images.githubusercontent.com/285352/126826478-60e53614-263d-40bb-9f1d-c1a676644eb0.png)\r\n\r\nI'm not sure if this is the right repo to report this in", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1557599877, "node_id": "I_kwDODFE5qs5c1xaF", "number": 12, "title": "location history changes", "user": {"value": 14809320, "label": "gerardrbentley"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-01-26T03:57:25Z", "updated_at": "2023-01-26T03:57:25Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "not sure if each download is unique, but I had to change some things to work with the takeout zip I made 2023-01-25\r\n\r\nfilename changed from \"Location History.json\" to \"Records.json\"\r\n\r\n`\"timestampMs\"` is not present, `\"timestamp\"` is roughly iso timestamp\r\n\r\n```py\r\ndef get_timestamp_ms(raw_timestamp):\r\n try:\r\n return datetime.datetime.strptime(raw_timestamp, \"%Y-%m-%dT%H:%M:%SZ\").timestamp()\r\n except ValueError:\r\n return datetime.datetime.strptime(raw_timestamp, \"%Y-%m-%dT%H:%M:%S.%fZ\").timestamp()\r\n\r\ndef save_location_history(db, zf):\r\n location_history = json.load(\r\n zf.open(\"Takeout/Location History/Records.json\")\r\n )\r\n db[\"location_history\"].upsert_all(\r\n (\r\n {\r\n \"id\": id_for_location_history(row),\r\n \"latitude\": row[\"latitudeE7\"] / 1e7,\r\n \"longitude\": row[\"longitudeE7\"] / 1e7,\r\n \"accuracy\": row[\"accuracy\"],\r\n \"timestampMs\": get_timestamp_ms(row[\"timestamp\"]),\r\n \"when\": row[\"timestamp\"],\r\n }\r\n for row in location_history[\"locations\"]\r\n ),\r\n pk=\"id\",\r\n )\r\n\r\n\r\ndef id_for_location_history(row):\r\n # We want an ID that is unique but can be sorted by in\r\n # date order - so we use the isoformat date + the first\r\n # 6 characters of a hash of the JSON\r\n first_six = hashlib.sha1(\r\n json.dumps(row, separators=(\",\", \":\"), sort_keys=True).encode(\"utf8\")\r\n ).hexdigest()[:6]\r\n return \"{}-{}\".format(\r\n row['timestamp'],\r\n first_six,\r\n )\r\n```\r\n\r\nexample locations from mine\r\n\r\n```json\r\n{\r\n \"latitudeE7\": 427220206,\r\n \"longitudeE7\": -923423972,\r\n \"accuracy\": 10,\r\n \"deviceTag\": -1312429967,\r\n \"deviceDesignation\": \"PRIMARY\",\r\n \"timestamp\": \"2019-01-08T23:31:50.867Z\"\r\n }\r\n```\r\n\r\n```json\r\n{\r\n \"latitudeE7\": 427011317,\r\n \"longitudeE7\": -923448300,\r\n \"accuracy\": 5,\r\n \"deviceTag\": -1312429967,\r\n \"deviceDesignation\": \"PRIMARY\",\r\n \"timestamp\": \"2019-01-08T23:33:53Z\"\r\n }, \r\n```", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/12/reactions\", \"total_count\": 2, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 2}", "draft": null, "state_reason": null} {"id": 1650981564, "node_id": "I_kwDOJHON9s5iZ_q8", "number": 12, "title": "Error running pytest", "user": {"value": 14314871, "label": "amlestin"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-04-02T15:02:36Z", "updated_at": "2023-04-02T15:07:10Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "`______________________________________________________ ERROR collecting tests/test_apple_notes_to_sqlite.py _______________________________________________________\r\nImportError while importing test module '/Users/lol/development/apple-notes-to-sqlite/tests/test_apple_notes_to_sqlite.py'.\r\nHint: make sure your test modules/packages have valid Python names.\r\nTraceback:\r\n/opt/homebrew/Cellar/python@3.9/3.9.16/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module\r\n return _bootstrap._gcd_import(name[level:], package, level)\r\ntests/test_apple_notes_to_sqlite.py:2: in \r\n from apple_notes_to_sqlite.cli import cli, COUNT_SCRIPT, FOLDERS_SCRIPT\r\nE ModuleNotFoundError: No module named 'apple_notes_to_sqlite'`\r\n\r\nSolution:\r\nThis is likely a PYTHONPATH issue due to having pytest installed both globally and in the venv. We can guarantee the tests run by adding the current directory to sys.path automatically using\r\n\r\n`python -m pytest`\r\n\r\nThe alternative is to activate the venv, install pytest, deactivate, then activate the venv again (https://stackoverflow.com/questions/35045038/how-do-i-use-pytest-with-virtualenv)", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1373210675, "node_id": "I_kwDODD6af85R2Ygz", "number": 13, "title": "fails before generating views. ERR: table sqlite_master may not be modified", "user": {"value": 116795, "label": "pax"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-09-14T15:41:50Z", "updated_at": "2023-04-11T03:46:17Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "generates checkins.db but seems to fail before generating views \r\n\r\nnote: it worked on an Ubuntu WSL but fails on macOS 12.5.1\r\n\r\nlater edit: I suspect this is a problem with my local set-up, `dogsheep-beta index` also throws the same error\r\n\r\nfull error:\r\n\r\n Importing 2591 checkins [###################################-] 98% 00:00:00\r\n Traceback (most recent call last):\r\n File \"/Users/pax/devbox/envAll/bin/swarm-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/swarm_to_sqlite/cli.py\", line 77, in cli\r\n ensure_foreign_keys(db)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/swarm_to_sqlite/utils.py\", line 145, in ensure_foreign_keys\r\n db[fk.table].add_foreign_key(fk.column, fk.other_table, fk.other_column)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/sqlite_utils/db.py\", line 2123, in add_foreign_key\r\n self.db.add_foreign_keys([(self.name, column, other_table, other_column)])\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/sqlite_utils/db.py\", line 1086, in add_foreign_keys\r\n cursor.execute(\r\n sqlite3.OperationalError: table sqlite_master may not be modified", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 771608692, "node_id": "MDU6SXNzdWU3NzE2MDg2OTI=", "number": 14, "title": "UNIQUE constraint failed: workouts.id", "user": {"value": 1234956, "label": "n8henrie"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-12-20T15:11:20Z", "updated_at": "2023-07-10T14:46:52Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I'm getting an error on my initial attempt to import data:\r\n\r\n```console\r\n$ healthkit-to-sqlite 20201119\\ healthkit\\ export.zip healthkit.db\r\nImporting from HealthKit [###################################-] 98% 00:00:01\r\nTraceback (most recent call last):\r\n File \"venv/bin/healthkit-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"venv/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"venv/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"venv/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"venv/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"venv/lib/python3.9/site-packages/healthkit_to_sqlite/cli.py\", line 57, in cli\r\n convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)\r\n File \"venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py\", line 34, in convert_xml_to_sqlite\r\n workout_to_db(el, db, zipfile)\r\n File \"venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py\", line 57, in workout_to_db\r\n pk = db[\"workouts\"].insert(record, alter=True, hash_id=\"id\").last_pk\r\n File \"venv/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1660, in insert\r\n return self.insert_all(\r\n File \"venv/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1778, in insert_all\r\n self.insert_chunk(\r\n File \"venv/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1588, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"venv/lib/python3.9/site-packages/sqlite_utils/db.py\", line 213, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.IntegrityError: UNIQUE constraint failed: workouts.id\r\n```", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 986829194, "node_id": "MDU6SXNzdWU5ODY4MjkxOTQ=", "number": 14, "title": "xml.etree.ElementTree.Parse Error - mismatched tag", "user": {"value": 46968, "label": "step21"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-09-02T14:46:36Z", "updated_at": "2021-09-02T14:53:11Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "This is an error message I get upon parsing the enex file of my Inbox. Please find the full error message below. Any hints welcome.\r\n\r\n```\r\nImporting from ENEX [##################------------------] 50% 00:00:50\r\nTraceback (most recent call last):\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/bin/evernote-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/evernote_to_sqlite/cli.py\", line 30, in enex\r\n for tag, note in find_all_tags(fp, [\"note\"], progress_callback=bar.update):\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/evernote_to_sqlite/utils.py\", line 17, in find_all_tags\r\n for event, el in parser.read_events():\r\n File \"/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py\", line 1329, in read_events\r\n raise event\r\n File \"/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py\", line 1301, in feed\r\n self._parser.feed(data)\r\nxml.etree.ElementTree.ParseError: mismatched tag: line 6837961, column 2\r\n```\r\n", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/14/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 544571092, "node_id": "MDU6SXNzdWU1NDQ1NzEwOTI=", "number": 15, "title": "Assets table with downloads", "user": {"value": 2029, "label": "garethr"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5225818, "label": "1.0"}, "comments": 4, "created_at": "2020-01-02T13:05:28Z", "updated_at": "2020-03-28T12:17:01Z", "closed_at": "2020-03-23T19:17:32Z", "author_association": "NONE", "pull_request": null, "body": "The `releases` command extracts the releases table, but data about the individual assets are locked up in the JSON document in the `assets` field. My main interest is in individual and aggregate download counts. I was wondering if creating a new table with a record per asset may be useful?\r\nIf so I'm happy to send a PR when I get a moment. Do you have opinions about that simply being part of the `releases` command or would you prefer a separate command as well?", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 546051181, "node_id": "MDU6SXNzdWU1NDYwNTExODE=", "number": 16, "title": "Exception running first command: IndexError: list index out of range", "user": {"value": 15092, "label": "jayvdb"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-01-07T03:01:58Z", "updated_at": "2020-04-14T18:37:21Z", "closed_at": "2020-04-14T18:37:21Z", "author_association": "NONE", "pull_request": null, "body": "Exception running first command without an existing db or auth.\r\n\r\n```py\r\n> mkdir ~/.github/coala\r\n> /usr/bin/github-to-sqlite repos ~/.github/coala coala\r\nTraceback (most recent call last):\r\n File \"/usr/bin/github-to-sqlite\", line 11, in \r\n load_entry_point('github-to-sqlite==0.6', 'console_scripts', 'github-to-sqlite')()\r\n File \"/usr/lib/python3.7/site-packages/click/core.py\", line 764, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/lib/python3.7/site-packages/click/core.py\", line 717, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/lib/python3.7/site-packages/click/core.py\", line 1137, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/lib/python3.7/site-packages/click/core.py\", line 956, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/lib/python3.7/site-packages/click/core.py\", line 555, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/lib/python3.7/site-packages/github_to_sqlite/cli.py\", line 163, in repos\r\n utils.save_repo(db, repo)\r\n File \"/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py\", line 120, in save_repo\r\n to_save[\"owner\"] = save_user(db, to_save[\"owner\"])\r\n File \"/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py\", line 61, in save_user\r\n return db[\"users\"].upsert(to_save, pk=\"id\", alter=True).last_pk\r\n File \"/usr/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1135, in upsert\r\n extracts=extracts,\r\n File \"/usr/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1162, in upsert_all\r\n upsert=True,\r\n File \"/usr/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1105, in insert_all\r\n row = list(self.rows_where(\"rowid = ?\", [self.last_rowid]))[0]\r\nIndexError: list index out of range\r\n```", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1943259395, "node_id": "I_kwDOEhK-wc5z08kD", "number": 16, "title": " time data '2014-11-21T11:44:12.000Z' does not match format '%Y%m%dT%H%M%SZ'", "user": {"value": 3746270, "label": "linonetwo"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-10-14T13:24:39Z", "updated_at": "2023-10-14T13:24:39Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "\r\n```\r\nevernote-to-sqlite enex evernote.db ./\u6211\u7684\u7b14\u8bb0.enex\r\nImporting from ENEX [#####-------------------------------] 14%\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/evernote-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n ^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 1157, in __call__\r\n return self.main(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 1078, in main\r\n rv = self.invoke(ctx)\r\n ^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 1688, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 1434, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 783, in invoke\r\n return __callback(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/cli.py\", line 31, in enex\r\n save_note(db, note)\r\n File \"/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/utils.py\", line 46, in save_note\r\n \"created\": convert_datetime(created),\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/utils.py\", line 111, in convert_datetime\r\n return datetime.datetime.strptime(s, \"%Y%m%dT%H%M%SZ\").isoformat()\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/_strptime.py\", line 568, in _strptime_datetime\r\n tt, fraction, gmtoff_fraction = _strptime(data_string, format)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/_strptime.py\", line 349, in _strptime\r\n raise ValueError(\"time data %r does not match format %r\" %\r\nValueError: time data '2014-11-21T11:44:12.000Z' does not match format '%Y%m%dT%H%M%SZ'\r\n```\r\n\r\nenex is exported by evernote mac client ", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/16/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 836063389, "node_id": "MDU6SXNzdWU4MzYwNjMzODk=", "number": 17, "title": "Datetime columns are not properly formatted to be recognizes as datetime", "user": {"value": 1234956, "label": "n8henrie"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-03-19T14:33:04Z", "updated_at": "2021-03-19T14:33:04Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "\r\n\r\nCurrently, the datetimes are formatted in a way that is not recognized by datasette-vega for plotting with a `Date/time` type for the axis. \r\n\r\nFor example, if you have datasette running locally with `datasette-vega` installed and have a database that includes resting heart rate:\r\n\r\n```\r\nhttp://localhost:8001/healthkit/rRestingHeartRate#g.mark=line&g.x_column=startDate&g.x_type=temporal&g.y_column=value&g.y_type=quantitative\r\n```\r\n\r\nThe plot is blank unless you choose `Label` as the type for the date data.\r\n\r\nThe `startDate` (and `creationDate` and `endDate`) columns appear like: `2019-11-14 18:22:18 -0700`\r\n\r\nIf instead the format for this column is changed slightly: `2019-11-14T18:22:18-07:00` they are recognized as proper dates and the charting works as expected.\r\n\r\nI have a PR that addresses this issue, will submit shortly.", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/17/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 432727685, "node_id": "MDU6SXNzdWU0MzI3Mjc2ODU=", "number": 20, "title": "JSON column values get extraneously quoted ", "user": {"value": 649467, "label": "mhalle"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 4348046, "label": "1.0"}, "comments": 1, "created_at": "2019-04-12T20:15:30Z", "updated_at": "2019-05-25T00:57:19Z", "closed_at": "2019-05-25T00:57:19Z", "author_association": "NONE", "pull_request": null, "body": "If the input to `sqlite-utils insert` includes a column that is a JSON array or object, `sqlite-utils query` will introduce an extra level of quoting on output:\r\n\r\n```\r\n# echo '[{\"key\": [\"one\", \"two\", \"three\"]}]' | sqlite-utils insert t.db t -\r\n\r\n# sqlite-utils t.db 'select * from t'\r\n[{\"key\": \"[\\\"one\\\", \\\"two\\\", \\\"three\\\"]\"}]\r\n\r\n# sqlite3 t.db 'select * from t'\r\n[\"one\", \"two\", \"three\"]\r\n```\r\n\r\nThis might require an imperfect solution, since sqlite3 doesn't have a JSON type. Perhaps fields that start with `[\"` or `{\"` and end with `\"]` or `\"}` could be detected, with a flag to turn off that behavior for weird text fields (or vice versa).", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/20/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 697162939, "node_id": "MDU6SXNzdWU2OTcxNjI5Mzk=", "number": 20, "title": "Add more tags so people can find your project.", "user": {"value": 7902810, "label": "ran88dom99"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-09T21:14:09Z", "updated_at": "2020-09-09T21:14:09Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "quantified-self habit-tracking google-fit time-tracking wearables quantifiedself \r\nfor example", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/20/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 1, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 977128935, "node_id": "MDU6SXNzdWU5NzcxMjg5MzU=", "number": 21, "title": "Duplicate Column", "user": {"value": 32016596, "label": "FabianHertwig"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-08-23T15:00:44Z", "updated_at": "2021-08-23T17:00:59Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hey, thank you for this repo!\r\n\r\nWhen I try to convert my export, I get a multiple column error. Here is the stack trace:\r\n\r\n```sh\r\n(.venv) (base) computer:bodyweight_app user$ healthkit-to-sqlite ./data/Health_export.zip ./data/healthkit.db\r\nImporting from HealthKit [###############################-----] 87% 00:00:22\r\nTraceback (most recent call last):\r\n File \"/MyProject/.venv/bin/healthkit-to-sqlite\", line 10, in \r\n sys.exit(cli())\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/cli.py\", line 57, in cli\r\n convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py\", line 41, in convert_xml_to_sqlite\r\n write_records(records, db)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py\", line 146, in write_records\r\n batch_size=50,\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py\", line 2579, in insert_all\r\n extracts=extracts,\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1246, in create\r\n extracts=extracts,\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py\", line 767, in create_table\r\n self.execute(sql)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py\", line 421, in execute\r\n return self.conn.execute(sql)\r\nsqlite3.OperationalError: duplicate column name: metadata_Meal\r\n```", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/21/reactions\", \"total_count\": 5, \"+1\": 5, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 449818897, "node_id": "MDU6SXNzdWU0NDk4MTg4OTc=", "number": 24, "title": "Additional Column Constraints?", "user": {"value": 98555, "label": "IgnoredAmbience"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2019-05-29T13:47:03Z", "updated_at": "2019-06-13T06:47:17Z", "closed_at": "2019-06-13T06:30:26Z", "author_association": "NONE", "pull_request": null, "body": "I'm looking to import data from XML with a pre-defined schema that maps fairly closely to a relational database.\r\nIn particular, it has explicit annotations for when fields are required, optional, or when a default value should be inferred.\r\n\r\nWould there be value in adding the ability to define `NOT NULL` and `DEFAULT` column constraints to sqlite-utils?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/24/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1515883470, "node_id": "I_kwDOC8tyDs5aWovO", "number": 24, "title": "DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 ", "user": {"value": 6231413, "label": "mmngreco"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-01-01T23:00:38Z", "updated_at": "2023-03-30T10:17:31Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi @simonw \r\n\r\nI hope you find this issue ok, the idea is provide some documentation to other users like me about how to solve this problem and save some time.\r\n\r\nFollowing the instructions from the `README.md` I've faced this error:\r\n\r\n```bash\r\n(venv) mgreco@pop-os apple-health master* (23:44|0s)\r\n$ healthkit-to-sqlite apple_health_export/export.xml healthkit.db --xml\r\nImporting from HealthKit [------------------------------------] 0%\r\nTraceback (most recent call last):\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/bin/healthkit-to-sqlite\", line 33, in \r\n sys.exit(load_entry_point('healthkit-to-sqlite', 'console_scripts', 'healthkit-to-sqlite')())\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/cli.py\", line 57, in cli\r\n convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)\r\n File \"/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/utils.py\", line 25, in convert_xml_to_sqlite\r\n for tag, el in find_all_tags(\r\n File \"/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/utils.py\", line 12, in find_all_tags\r\n for event, el in parser.read_events():\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/xml/etree/ElementTree.py\", line 1324, in read_events\r\n raise event\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/xml/etree/ElementTree.py\", line 1296, in feed\r\n self._parser.feed(data)\r\nxml.etree.ElementTree.ParseError: syntax error: line 156, column 0\r\n```\r\n\r\nSo, after debugging and searching on internet I found this useful link: https://discussions.apple.com/thread/254202523 (etresoft, the real hero). Which basically says that the xml given by the health app (healthkit version 12) has some bugs but fortunately, they can be solved with a couple of commads:\r\n\r\n1. Uncompress the zip and move the new folder where `export.xml` is.\r\n1. Create a `patch.txt` with the following content\r\n\r\n ```diff\r\n --- export.xml\t2022-09-18 15:17:09.000000000 -0400\r\n +++ export-fixed.xml\t2022-09-18 16:37:08.000000000 -0400\r\n @@ -15,6 +15,7 @@\r\n HKCharacteristicTypeIdentifierBiologicalSex CDATA #REQUIRED\r\n HKCharacteristicTypeIdentifierBloodType CDATA #REQUIRED\r\n HKCharacteristicTypeIdentifierFitzpatrickSkinType CDATA #REQUIRED\r\n + HKCharacteristicTypeIdentifierCardioFitnessMedicationsUse CDATA #IMPLIED\r\n >\r\n \r\n \r\n -\r\n +\r\n \r\n -\r\n +\r\n \r\n \r\n \r\n \r\n \r\n - device CDATA #IMPLIED\r\n -\r\n -\r\n ->\r\n ]>\r\n \r\n \r\n ```\r\n1. Apply the path with the command: `patch < patch.txt`\r\n1. Fix endDates with the command `sed 's/startDate/endDate/2' export.xml > export-fixed.xml`\r\n1. Try again `healthkit-to-sqlite export-fixed.xml healthkit.db --xml`", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 624490929, "node_id": "MDU6SXNzdWU2MjQ0OTA5Mjk=", "number": 28, "title": "Invalid SQL no such table: main.uploads", "user": {"value": 41439, "label": "dmd"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-25T21:25:39Z", "updated_at": "2020-12-24T22:26:22Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "http://127.0.0.1:8001/photos/photos_with_apple_metadata gives \"Invalid SQL no such table: main.uploads\"", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/28/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 803333769, "node_id": "MDU6SXNzdWU4MDMzMzM3Njk=", "number": 32, "title": "KeyError: 'Contents' on running upload", "user": {"value": 11855322, "label": "robmarkcole"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-02-08T08:36:37Z", "updated_at": "2021-07-22T06:40:25Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Following the readme, on big sur, and having entered my auth creds via `dogsheep-photos s3-auth`:\r\n\r\n```\r\n(venv) (base) Robins-MacBook:datasette robin$ dogsheep-photos upload photos.db ~/Pictures/Photos\\ /Users/robin/Pictures/Library.photoslibrary --dry-run\r\n\r\nFetching existing keys from S3...\r\nTraceback (most recent call last):\r\n File \"/Users/robin/datasette/venv/bin/dogsheep-photos\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/cli.py\", line 96, in upload\r\n key.split(\".\")[0] for key in get_all_keys(client, creds[\"photos_s3_bucket\"])\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/utils.py\", line 46, in get_all_keys\r\n for row in page[\"Contents\"]:\r\nKeyError: 'Contents'\r\n```\r\n\r\nPossibly since the bucket is in `EU (London) eu-west-2` and this into is not requested?", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 609950090, "node_id": "MDU6SXNzdWU2MDk5NTAwOTA=", "number": 33, "title": "Fall back to authentication via ENV", "user": {"value": 2029, "label": "garethr"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-04-30T12:58:14Z", "updated_at": "2020-05-02T18:46:10Z", "closed_at": "2020-05-02T18:45:37Z", "author_association": "NONE", "pull_request": null, "body": "Would you accept a PR that falls back to looking for an environment variable for the GitHub token? Specifically a change here:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/c34d5a18bfc41fa08755ba3d5cf9fe09ff204238/github_to_sqlite/cli.py#L271\r\n\r\nI'd like to use `github-to-sqlite` in a GitHub Action workflow and this would be simpler than trying to fill out the prompt or generate a file with sensitive content.\r\n\r\nWanted to check first, I'm happy to submit a PR with tests and updates to the docs. ", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 803338729, "node_id": "MDU6SXNzdWU4MDMzMzg3Mjk=", "number": 33, "title": "photo-to-sqlite: command not found", "user": {"value": 11855322, "label": "robmarkcole"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-02-08T08:42:57Z", "updated_at": "2021-02-12T15:00:44Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Having installed in a venv I get:\r\n```\r\n(venv) (base) Robins-MacBook:datasette robin$ photo-to-sqlite apple-photos photos.db\r\n\r\n-bash: photo-to-sqlite: command not found\r\n```", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 830283447, "node_id": "MDU6SXNzdWU4MzAyODM0NDc=", "number": 34, "title": "bucket name", "user": {"value": 6213, "label": "dsisnero"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-03-12T16:40:57Z", "updated_at": "2021-03-12T16:40:57Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I followed the instructions to setup credentials but I am getting a invalid bucket name. Can you put a sample auth.json file in the base that shows the correct format for this? Thanks", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/34/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 983221851, "node_id": "MDU6SXNzdWU5ODMyMjE4NTE=", "number": 34, "title": "Data folder as index command parameter", "user": {"value": 1223625, "label": "humrochagf"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-08-30T21:29:33Z", "updated_at": "2021-08-30T21:29:33Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi,\r\n\r\nFirst of all, thank you for this wonderful project :smile:\r\n\r\nI started to use dogsheep to make my personal data searchable, and by using the project I noticed an issue with the index command.\r\n\r\nIt always expects you are running it from the root folder from where the data is located, so I got some errors while trying to make it work on my setup.\r\n\r\nI separate all databases inside a `data` folder (I published my setup to be easier to follow: https://github.com/humrochagf/my-dogsheep)\r\n\r\nBefore, I configured `dogsheep.yml` to add the data folder to its path like this:\r\n\r\n```yml\r\ndata/twitter.db:\r\n tweets:\r\n sql: |-\r\n...\r\n```\r\n\r\nAnd running the index command like this:\r\n\r\n```\r\ndogsheep-beta index data/dogsheep.db dogsheep.yml\r\n```\r\n\r\nIt worked to the normal search feature with no problem this way, but when I started adding `display_sql` rules the app started to crash, because at datasette `get_database` it was looking for `data/twitter` and it only had a db called `twitter` there.\r\n\r\nSo my workaround to that was to cd into the data folder and run the indexer. You can check the way I'm doing it at this line of the makefile: https://github.com/humrochagf/my-dogsheep/blob/main/makefile#L3\r\n\r\nIt works but it would be nice to have an option to pass the path where the data is located to the index function.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/34/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 842695374, "node_id": "MDU6SXNzdWU4NDI2OTUzNzQ=", "number": 35, "title": "Support to annotate photos on other than macOS OSes", "user": {"value": 1151557, "label": "ligurio"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-03-28T09:01:25Z", "updated_at": "2021-04-05T07:37:57Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "dogsheep-photos allows to annotate photos using Apple Photo's db. It would be nice to have such ability on other OSes too. For example using trained local model or using Google Vision API (see #14).", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/35/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1751214236, "node_id": "I_kwDOC8SPRc5oYWic", "number": 36, "title": "Getting sqlite_master may not be modified when creating dogsheep index", "user": {"value": 8711912, "label": "khushmeeet"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-06-11T03:21:53Z", "updated_at": "2023-06-11T03:21:53Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "When creating a `dogsheep` index from `config.yml` file on pocket.db (created using pocket-to-sqlite), I am getting this error\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/bin/dogsheep-beta\", line 8, in \r\n sys.exit(cli())\r\n ^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n ^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 1657, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/cli.py\", line 36, in index\r\n run_indexer(\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py\", line 32, in run_indexer\r\n ensure_table_and_indexes(db, tokenize)\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py\", line 91, in ensure_table_and_indexes\r\n table.add_foreign_key(*fk)\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py\", line 2155, in add_foreign_key\r\n self.db.add_foreign_keys([(self.name, column, other_table, other_column)])\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py\", line 1116, in add_foreign_keys\r\n cursor.execute(\r\nsqlite3.OperationalError: table sqlite_master may not be modified\r\n```\r\n\r\nCommand I ran to get this error\r\n```\r\ndogsheep-beta index pocket.db config.yml\r\n```\r\n\r\nDogsheep version\r\n```\r\ndogsheep-beta, version 0.10.2\r\n```\r\n\r\nPython version \r\n```\r\nPython 3.11.2\r\n```", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/36/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1817281557, "node_id": "I_kwDOC8SPRc5sUYQV", "number": 37, "title": "cannot use jinja filters in display?", "user": {"value": 10352819, "label": "rprimet"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-07-23T20:09:54Z", "updated_at": "2023-07-23T20:18:27Z", "closed_at": "2023-07-23T20:18:26Z", "author_association": "NONE", "pull_request": null, "body": "Hi, I'm trying to have a display function in Dogsheep's `config.yml` that includes something like this:\r\n\r\n``` \r\n

\r\n {{ display.title }}\r\n (source)\r\n

\r\n

{{ display.snippet|safe }}

\r\n```\r\n\r\nUnfortunately, rendering fails with a message 'urls is undefined'.\r\n\r\nThe same happens if I'm trying to build a row URL manually, using filters like `quote_plus` (as my keys are URLs).\r\n\r\nAny hints?\r\n\r\nThanks!", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/37/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611284481, "node_id": "MDU6SXNzdWU2MTEyODQ0ODE=", "number": 38, "title": "[Feature Request] Support Repo Name in Search \ud83e\udd7a", "user": {"value": 5779832, "label": "zzeleznick"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-05-02T22:08:51Z", "updated_at": "2020-05-03T02:34:32Z", "closed_at": "2020-05-02T23:15:11Z", "author_association": "NONE", "pull_request": null, "body": "## Description\r\nPer your [v2.2 release tweet](https://twitter.com/simonw/status/1256700238099693568) I played with the demo, but the output did not match my expectations.\r\n\r\n## Expected Behavior\r\nExpected a search query for \"twitter\" contained within the `repo` column to return non-zero results.\r\n\r\n## Actual Behavior\r\n\ud83d\ude2d [0 rows where repo contains \"twitter\" sorted by starred_at descending](https://github-to-sqlite.dogsheep.net/github/stars?repo__contains=twitter&_sort_desc=starred_at) \r\n\r\n## Best Explanation\r\nPer the table schema (see appendix) `repo` is of type `INTEGER` which built from `repo_id` and does not expose the repo name in search.\r\n\r\n## Desired Behavior\r\nGiven that searching for \"206156866\" is less intuitive than \"twitter\", it would be great to support this via extending the search capabilities or by adding an additional column.\r\n\r\n\u2705 104 rows where repo contains \"twitter\"\r\n\u274c [104 rows where repo contains \"206156866\" sorted by starred_at descending](https://github-to-sqlite.dogsheep.net/github/stars?repo__contains=206156866&_sort_desc=starred_at) \r\n\r\n## Appendix\r\n```\r\nCREATE TABLE [stars] (\r\n [user] INTEGER REFERENCES [users]([id]),\r\n [repo] INTEGER REFERENCES [repos]([id]),\r\n [starred_at] TEXT,\r\n PRIMARY KEY ([user], [repo])\r\n);\r\nCREATE INDEX [idx_stars_repo]\r\n ON [stars] ([repo]);\r\nCREATE INDEX [idx_stars_user]\r\n ON [stars] ([user]);\r\n```", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1871935751, "node_id": "I_kwDOD079W85vk3kH", "number": 40, "title": " ImportError: cannot import name 'formatargspec' from 'inspect'", "user": {"value": 36752421, "label": "hosslikw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-08-29T15:36:31Z", "updated_at": "2023-08-31T03:18:07Z", "closed_at": "2023-08-31T03:18:06Z", "author_association": "NONE", "pull_request": null, "body": "I get the following error when running \"pip3 install dogsheep-photos\"\r\n\" from inspect import ismethod, isclass, formatargspec\r\n ImportError: cannot import name 'formatargspec' from 'inspect' (/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/inspect.py). Did you mean: 'formatargvalues'?\"\r\n \r\nPython 3.12.0rc1\r\nsqlite 3.43.0\r\ndatasette, version 0.64.3", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/40/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 664485022, "node_id": "MDU6SXNzdWU2NjQ0ODUwMjI=", "number": 46, "title": "Feature: pull request reviews and comments", "user": {"value": 1326704, "label": "bhrutledge"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-07-23T13:43:45Z", "updated_at": "2022-12-20T14:40:15Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi there! I saw your [presentation at Boston Python](https://www.meetup.com/bostonpython/events/271887195). I'm already a light user of Datasette (thank you!), but wasn't aware of this project.\r\n\r\nI've been working on a \"pull request dashboard\" to get a comprehensive view of the state of open PR's, esp. related to reviews (i.e., pending, approved, changes requested). Currently it's a CLI command, but I thought a Datasette UI might be fun.\r\n\r\nI see that PR's are available from the `issues` command, but I don't see reviews anywhere. From the [API docs](https://docs.github.com/en/rest/reference/pulls#reviews), it looks like there are separate endpoints for those (as well as pull requests in general). What do you think about adding that? Would you accept a PR? Any sense of the level of effort?", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 639542974, "node_id": "MDU6SXNzdWU2Mzk1NDI5NzQ=", "number": 47, "title": "Fall back to FTS4 if FTS5 is not available", "user": {"value": 73579, "label": "hpk42"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-06-16T10:11:23Z", "updated_at": "2020-06-17T20:13:48Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "got this with version 0.21.1 from pypi. twitter-to-sqlite auth worked but then \"twitter-to-sqlite user-timeline USER.db\" produced a tracekback ending in \"no such module: FTS5\". ", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 745393298, "node_id": "MDU6SXNzdWU3NDUzOTMyOTg=", "number": 52, "title": "Discussion: Adding support for fetching only fresh tweets", "user": {"value": 4169772, "label": "fatihky"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-11-18T07:01:48Z", "updated_at": "2020-11-18T07:12:45Z", "closed_at": "2020-11-18T07:12:45Z", "author_association": "NONE", "pull_request": null, "body": "I think it'd be very useful if this tool has an option like `--incremental` to fetch only newer tweets. This way operations could complete very fast in sequential runs. I'd want to try to implement this feature if it seems OK for this tool's purpose. ", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/52/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 771324837, "node_id": "MDU6SXNzdWU3NzEzMjQ4Mzc=", "number": 53, "title": "--since support for favorites", "user": {"value": 27, "label": "anotherjesse"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-12-19T07:08:23Z", "updated_at": "2020-12-19T07:47:11Z", "closed_at": "2020-12-19T07:47:11Z", "author_association": "NONE", "pull_request": null, "body": "Having support for `--since` for updating your favorites would be ideal as the api is both slow and it only returns ~3k most recent favorites.\r\n\r\nhttps://twittercommunity.com/t/cant-get-all-favorite-tweets-by-rest-api/22007/3\r\n\r\nThe api seems to take an optional `since_id` parameter - https://developer.twitter.com/en/docs/twitter-api/v1/tweets/post-and-engage/api-reference/get-favorites-list", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/53/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 480961330, "node_id": "MDU6SXNzdWU0ODA5NjEzMzA=", "number": 54, "title": "Ability to list views, and to access db[\"view_name\"].rows / rows_where / etc", "user": {"value": 20264, "label": "ftrain"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2019-08-15T02:00:28Z", "updated_at": "2019-08-23T12:41:09Z", "closed_at": "2019-08-23T12:20:15Z", "author_association": "NONE", "pull_request": null, "body": "The docs show me how to create a view via `db.create_view()` but I can't seem to get back to that view post-creation; if I query it as a table it returns `None`, and it doesn't appear in the table listing, even though querying the view works fine from inside the sqlite3 command-line.\r\n\r\nIt'd be great to have the view as a pseudo-table, or if the python/sqlite3 module makes that hard to pull off (I couldn't figure it out), to have that edge-case documented next to the `db.create_view()` docs.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/54/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 796736607, "node_id": "MDU6SXNzdWU3OTY3MzY2MDc=", "number": 56, "title": "Not all quoted statuses get fetched?", "user": {"value": 42315895, "label": "gsajko"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-01-29T09:48:44Z", "updated_at": "2021-02-03T10:36:36Z", "closed_at": "2021-02-03T10:36:36Z", "author_association": "NONE", "pull_request": null, "body": "\r\n![image](https://user-images.githubusercontent.com/42315895/106259325-5f75dc80-621f-11eb-8311-db8f2fe2a257.png)\r\n\r\nIn my database I have 13300 quote tweets, but eta 3600 have `quoted_status` empty.\r\n\r\nI fetched some of them using `https://api.twitter.com/1.1/statuses/show.json?id=xx` and they did have ids of quoted tweets.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1063982712, "node_id": "I_kwDODEm0Qs4_axZ4", "number": 60, "title": "Execution on Windows", "user": {"value": 1733616, "label": "bernard01"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-11-26T00:24:34Z", "updated_at": "2022-10-14T16:58:27Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "My installation on Windows using pip has been successful. I have Python 3.6.\r\n\r\nHow do I run twitter-to-sqlite? I cannot even figure out how \"auth\" is a command. I have python on my path:\r\nC:\\prog\\python\\Python36;C:\\prog\\python\\Python36\\Scripts\r\n\r\nWhere should the commands be executed, and where are the files created?\r\n\r\nCould some basics please be added to the documentation to get beginners started?", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/60/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 491219910, "node_id": "MDU6SXNzdWU0OTEyMTk5MTA=", "number": 61, "title": "importing CSV to SQLite as library", "user": {"value": 17739, "label": "witeshadow"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-09-09T17:12:40Z", "updated_at": "2019-11-04T16:25:01Z", "closed_at": "2019-11-04T16:25:01Z", "author_association": "NONE", "pull_request": null, "body": "CSV can be imported to SQLite when used CLI, but I don't see documentation for when using as library. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/61/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1077560091, "node_id": "I_kwDODEm0Qs5AOkMb", "number": 61, "title": "Data Pull fails for \"Essential\" level access to the Twitter API (for Documentation)", "user": {"value": 57161638, "label": "jmnickerson05"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-11T14:59:41Z", "updated_at": "2022-10-31T14:47:58Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Per Twitter documentation:\r\nhttps://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api#v2-access-leve\r\n\r\nThis isn't any fault of twitter-to-sqlite of course, but it should probably be documented as a side-note.\r\n\r\n![image](https://user-images.githubusercontent.com/57161638/145681272-8c85b3b9-be95-44ff-9760-1bafa4917ce2.png)\r\n\r\nAnd this is how I'm surfacing the message from utils.py:\r\n![image](https://user-images.githubusercontent.com/57161638/145681005-2776c0ad-9822-4461-b43a-450ab2e828eb.png)\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/61/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 500783373, "node_id": "MDU6SXNzdWU1MDA3ODMzNzM=", "number": 62, "title": "[enhancement] Method to delete a row in python", "user": {"value": 4454869, "label": "Sergeileduc"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2019-10-01T09:45:47Z", "updated_at": "2019-11-04T16:30:34Z", "closed_at": "2019-11-04T16:18:18Z", "author_association": "NONE", "pull_request": null, "body": "Hi !\r\nThanks for the lib !\r\n\r\nObviously, every possible sql queries won't have a dedicated method.\r\n\r\nBut I was thinking : a method to delete a row (I'm terrible with names, maybe `delete_where()` or something, would be useful.\r\n\r\nI have a Database, with primary key.\r\n\r\nFor the moment, I use :\r\n\r\n```Python3\r\ndb.conn.execute(f\"DELETE FROM table WHERE key = {key_id}\")\r\ndb.conn.commit()\r\n```\r\nto delete a row I don't need anymore, giving his primary key.\r\n\r\nWorks like a charm.\r\n\r\nJust an idea :\r\n\r\n```Python3\r\ntable.delete_where_pkey({'key': key_id})\r\n```\r\nor something (I know, I'm terrible at naming methods...).\r\n\r\nPros : well, no need to write SQL query.\r\n\r\nCons : WHERE normally allows to do many more things (operators =, <>, >, <, BETWEEN), not to mention AND, OR, etc...\r\nMethod is maybe to specific, and/or a pain to render more flexible.\r\n\r\nAgain, just a thought. Writing his own sql works too, so...\r\n\r\nThanks again.\r\nSee yah.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/62/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1088816961, "node_id": "I_kwDODEm0Qs5A5gdB", "number": 62, "title": "KeyError: 'created_at' for private accounts?", "user": {"value": 6764957, "label": "swyxio"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-12-26T17:51:51Z", "updated_at": "2022-03-12T02:36:32Z", "closed_at": "2022-02-24T18:10:18Z", "author_association": "NONE", "pull_request": null, "body": "hey Simon!\r\n\r\ni was running `twitter-to-sqlite user-timeline twitter.db` for [my private alt](https://twitter.com/swyxio) and ran into this error:\r\n\r\n
\r\n\r\n\r\n![image](https://user-images.githubusercontent.com/6764957/147416165-46b69c30-100a-406f-8534-8612b75547ae.png)\r\n\r\n\r\n\r\n\r\n\r\n```bash\r\nTraceback (most recent call last):\r\n File \"/Users/swyx/Work/datasette/env/bin/twitter-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/cli.py\", line 291, in user_timeline\r\n profile = utils.get_profile(db, session, **kwargs)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py\", line 133, in get_profile\r\n save_users(db, [profile])\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py\", line 453, in save_users\r\n transform_user(user)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py\", line 285, in transform_user\r\n user[\"created_at\"] = parser.parse(user[\"created_at\"])\r\nKeyError: 'created_at'\r\n```\r\n\r\n
\r\n\r\n\r\nthis looks awfully like #37 but it can't be, because i'm authed into my account and obviously i have perms to read my own account. wonder if there's any diagnostic methods i should apply here? just filing an issue for others to find while i investigate.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/62/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1091850530, "node_id": "I_kwDODEm0Qs5BFFEi", "number": 63, "title": "Import archive error 'withheld_in_countries'", "user": {"value": 521097, "label": "pauloxnet"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-01T16:58:59Z", "updated_at": "2022-01-01T16:58:59Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Importing the twitter archive I received this error:\r\n```bash\r\n$ twitter-to-sqlite import archive.db twitter-2021-12-31-.zip \r\nbirdwatch-note-rating: not yet implemented\r\nbirdwatch-note: not yet implemented\r\nbranch-links: not yet implemented\r\ncommunity-tweet: not yet implemented\r\ncontact: not yet implemented\r\ndevice-token: not yet implemented\r\ndirect-message-mute: not yet implemented\r\nmute: not yet implemented\r\nperiscope-account-information: not yet implemented\r\nperiscope-ban-information: not yet implemented\r\nperiscope-broadcast-metadata: not yet implemented\r\nperiscope-comments-made-by-user: not yet implemented\r\nperiscope-expired-broadcasts: not yet implemented\r\nperiscope-followers: not yet implemented\r\nperiscope-profile-description: not yet implemented\r\nprofessional-data: not yet implemented\r\nprotected-history: not yet implemented\r\nreply-prompt: not yet implemented\r\nscreen-name-change: not yet implemented\r\nsmartblock: not yet implemented\r\nspaces-metadata: not yet implemented\r\nsso: not yet implemented\r\nTraceback (most recent call last):\r\n File \"/home/paulox/.virtualenvs/dogsheep/bin/twitter-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/cli.py\", line 759, in import_\r\n archive.import_from_file(db, filename, content)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/archive.py\", line 246, in import_from_file\r\n db[table_name].insert_all(rows, pk=pk, replace=True)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2625, in insert_all\r\n self.insert_chunk(\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2406, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 422, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.OperationalError: table archive_tweet has no column named withheld_in_countries\r\n```\r\n\r\nI found only a single tweet with the key `withheld_in_countries` in `tweet.js` that seems the problems:\r\n```JSON\r\n[\r\n{\r\n \"tweet\" : {\r\n \"retweeted\" : false,\r\n \"source\" : \"Twitter for Android\",\r\n \"entities\" : {\r\n \"hashtags\" : [\r\n {\r\n \"text\" : \"NowOnAndroid\",\r\n \"indices\" : [\r\n \"64\",\r\n \"77\"\r\n ]\r\n }\r\n ],\r\n \"symbols\" : [ ],\r\n \"user_mentions\" : [\r\n {\r\n \"name\" : \"Periscope\",\r\n \"screen_name\" : \"PeriscopeCo\",\r\n \"indices\" : [\r\n \"3\",\r\n \"15\"\r\n ],\r\n \"id_str\" : \"1111111111\",\r\n \"id\" : \"222222222\"\r\n }\r\n ],\r\n \"urls\" : [\r\n {\r\n \"url\" : \"https://t.co/xxxxxxxxx\",\r\n \"expanded_url\" : \"https://vine.co/v/xxxxxxxxx\",\r\n \"display_url\" : \"vine.co/v/xxxxxxxxxx\",\r\n \"indices\" : [\r\n \"78\",\r\n \"101\"\r\n ]\r\n }\r\n ]\r\n },\r\n \"display_text_range\" : [\r\n \"0\",\r\n \"101\"\r\n ],\r\n \"favorite_count\" : \"0\",\r\n \"id_str\" : \"1111111111111111111111\",\r\n \"truncated\" : false,\r\n \"retweet_count\" : \"0\",\r\n \"withheld_in_countries\" : [\r\n \"TR\"\r\n ],\r\n \"id\" : \"000000000000000000\",\r\n \"possibly_sensitive\" : false,\r\n \"created_at\" : \"Fri Aug 14 06:04:03 +0000 2015\",\r\n \"favorited\" : false,\r\n \"full_text\" : \"RT @periscopeco: Travel the world. LIVE. The Global Map is here #NowOnAndroid https://t.co/NZXdsPWROk\",\r\n \"lang\" : \"en\"\r\n }\r\n }\r\n ]\r\n```\r\n\r\nI solved the error removing the key from the `tweet.js` but I'm reporting this error to improve the project.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/63/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 920636216, "node_id": "MDU6SXNzdWU5MjA2MzYyMTY=", "number": 64, "title": "feature: support \"events\"", "user": {"value": 231498, "label": "khimaros"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-06-14T17:42:49Z", "updated_at": "2021-06-15T00:48:37Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "the GitHub API provides the ability to fetch all events for a given user, organization, or repository: https://docs.github.com/en/rest/reference/activity#list-events-for-the-authenticated-user\r\n\r\nthis would allow users to export all of the issue comments, new issues, etc. that they created. something which is currently missing from the GitHub takeout exports.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1097332098, "node_id": "I_kwDODEm0Qs5BZ_WC", "number": 64, "title": "Include all entities for tweets", "user": {"value": 111631, "label": "max"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-09T23:35:28Z", "updated_at": "2022-01-09T23:35:28Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Per our conversation [on Twitter](https://twitter.com/mschoening/status/1480312477246054401):\r\n\r\nIt would be neat if all entities (including URLs) were captured. This way you can ensure, that URLs are parsed out exactly the same way Twitter parses URLs \u2013 we all know parsing URLs with a regex ain't fun.\r\n\r\nRight now, I believe the tool filters out all entities that are not of type `media`.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/64/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 534507142, "node_id": "MDU6SXNzdWU1MzQ1MDcxNDI=", "number": 69, "title": "Feature request: enable extensions loading", "user": {"value": 30607, "label": "aborruso"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-12-08T08:06:25Z", "updated_at": "2022-02-05T00:04:25Z", "closed_at": "2020-10-16T18:42:49Z", "author_association": "NONE", "pull_request": null, "body": "Hi, it would be great to add a parameter that enables the load of a sqlite extension you need.\r\n\r\nSomething like \"-ext modspatialite\".\r\n\r\nIn this way your great tool would be even more comfortable and powerful.\r\n\r\n\r\nThank you very much", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/69/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 539204432, "node_id": "MDU6SXNzdWU1MzkyMDQ0MzI=", "number": 70, "title": "Implement ON DELETE and ON UPDATE actions for foreign keys", "user": {"value": 26292069, "label": "LucasElArruda"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-12-17T17:19:10Z", "updated_at": "2020-02-27T04:18:53Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi! I did not find any mention on the library about ON DELETE and ON UPDATE actions for foreign keys. Are those expected to be implemented? If not, it would be a nice thing to include!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/70/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1177059481, "node_id": "I_kwDODFdgUs5GKICZ", "number": 71, "title": "Store commit parents", "user": {"value": 64686, "label": "carltongibson"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-03-22T17:06:48Z", "updated_at": "2022-04-22T12:44:04Z", "closed_at": "2022-04-22T12:44:04Z", "author_association": "NONE", "pull_request": null, "body": "Hi @simonw \ud83d\udc4b \r\n\r\nCurrently, stored commit data doesn't quite give me the information I'm needing... \r\n\r\nCommitter date and author date are not 100% reliable for dividing a commit history up by release or branch. \r\nA PR created before a release but merged after can have earlier dates\u2026 \u2014 this can be quite frustrating if you're trying to pin down commits for a release: _It should be there!_, but then isn't. (This gets worse using release branches.) \r\n\r\nWould you be open to adding the `sha` of a `parent` of a commit to the commit table? (As an FK? \ud83e\udd14 \u2014\u00a0likely not feasible.)\r\n\r\nIt's part of the [response body](https://docs.github.com/en/rest/reference/commits#get-a-commit): \r\n\r\n```\r\n \"parents\": [\r\n {\r\n \"url\": \"https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e\",\r\n \"sha\": \"6dcb09b5b57875f334f61aebed695e2e4193db5e\"\r\n }\r\n ],\r\n```\r\n\r\nI think this list should only have a single entry. (\ud83e\udd14 \u2014\u00a0not sure why it's a list then...)\r\n\r\nWith this it would be possible to build/reconstruct a chain of commits from the history, that I don't **think** is available as yet (unless you know a better way). \r\n\r\nIt is certainly possible to get sequential lists of commits out of git directly, so the same would be possible combining tools, but wondering if a single tool could do it. \r\n\r\nWhat do you think? \r\nThanks! \ud83c\udfc5 \r\n\r\n", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/71/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1211283427, "node_id": "I_kwDODFdgUs5IMrfj", "number": 72, "title": "feature: display progress bar when downloading multi-page responses", "user": {"value": 9020979, "label": "hydrosquall"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-04-21T16:37:12Z", "updated_at": "2022-04-21T17:29:31Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "## Motivation\r\n\r\nFor a long running command (longer than 1 minute) for a big table (like pull requests or commits), it can be tricky to know if the script is still running, or if a rate limit/error was encountered\r\n\r\nWe know how many pages there are, so it may be possible to indicate how many remain.\r\n\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/a6e237f75a4b86963d91dcb5c9582e3a1b3349d6/github_to_sqlite/utils.py#L367\r\n\r\n## Resources\r\n\r\n- Using the existing Click API: \r\n - https://click.palletsprojects.com/en/5.x/utils/#showing-progress-bars\r\n- Loading spinner: https://github.com/pavdmyt/yaspin\r\n- Progress bar: https://github.com/tqdm/tqdm", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/72/reactions\", \"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1524431805, "node_id": "I_kwDODEm0Qs5a3Pu9", "number": 72, "title": "Import thread, including self- and others' replies", "user": {"value": 601708, "label": "mcint"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-01-08T09:51:06Z", "updated_at": "2023-01-08T09:51:06Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "statuses-lookup, home-timeline, mentions (only for auth'ed user) don't cover this.\r\n\r\n`twitter-to-sqlite fetch-thread tw-group1.db 1234123412341234`\r\n\r\ntwitter-to-sqlite focuses on archiving users, but does not easily support archiving conversations or community activity.\r\n\r\nFor reference, this is [implemented in twarc](https://sourcegraph.com/github.com/DocNow/twarc/-/blob/twarc/client.py?L708-766&subtree=true), using a search, optionally recursively.\r\n\r\nOther research suggests that this formerly, or currently, requires a [search query](https://stackoverflow.com/a/30480103/1020467), use of [undocumented `related_results` api](https://stackoverflow.com/a/9419346/1020467), or with requested inclusion of [newer conversation_id](https://stackoverflow.com/a/68115718/1020467) with subsequent query.\r\n\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/72/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 545407916, "node_id": "MDU6SXNzdWU1NDU0MDc5MTY=", "number": 73, "title": "upsert_all() throws issue when upserting to empty table", "user": {"value": 82988, "label": "psychemedia"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-01-05T11:58:57Z", "updated_at": "2020-01-31T14:21:09Z", "closed_at": "2020-01-05T17:20:18Z", "author_association": "NONE", "pull_request": null, "body": "If I try to add a list of `dict`s to an empty table using `upsert_all`, I get an error:\r\n\r\n```python\r\nimport sqlite3\r\nfrom sqlite_utils import Database\r\nimport pandas as pd\r\n\r\nconx = sqlite3.connect(':memory')\r\ncx = conx.cursor()\r\ncx.executescript('CREATE TABLE \"test\" (\"Col1\" TEXT);')\r\n\r\nq=\"SELECT * FROM test;\"\r\npd.read_sql(q, conx) #shows empty table\r\n\r\ndb = Database(conx)\r\ndb['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])\r\n\r\n---------------------------------------------------------------------------\r\nTypeError Traceback (most recent call last)\r\n in \r\n 1 db = Database(conx)\r\n----> 2 db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts)\r\n 1157 alter=alter,\r\n 1158 extracts=extracts,\r\n-> 1159 upsert=True,\r\n 1160 )\r\n 1161 \r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert)\r\n 1040 sql = \"INSERT OR IGNORE INTO [{table}]({pks}) VALUES({pk_placeholders});\".format(\r\n 1041 table=self.name,\r\n-> 1042 pks=\", \".join([\"[{}]\".format(p) for p in pks]),\r\n 1043 pk_placeholders=\", \".join([\"?\" for p in pks]),\r\n 1044 )\r\n\r\nTypeError: 'NoneType' object is not iterable\r\n\r\n```\r\n\r\nA hacky workaround in use is:\r\n\r\n```python\r\ntry:\r\n db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])\r\nexcept:\r\n db['test'].insert_all([{'Col1':'a'},{'Col1':'b'}])\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/73/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1816830546, "node_id": "I_kwDODEm0Qs5sSqJS", "number": 73, "title": "Twitter v1 API shutdown", "user": {"value": 6341745, "label": "david-perez"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-07-22T16:57:41Z", "updated_at": "2023-07-22T16:57:41Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I've been using this project reliably over the past two years to periodically download my liked tweets, but unfortunately since 19th July I get:\r\n\r\n```\r\n[2023-07-19 21:00:04.937536] File \"/home/pi/code/liked-tweets/lib/python3.7/site-packages/twitter_to_sqlite/utils.py\", line 202, in fetch_timeline\r\n[2023-07-19 21:00:04.937606] raise Exception(str(tweets[\"errors\"]))\r\n[2023-07-19 21:00:04.937678] Exception: [{'message': 'You currently have access to a subset of Twitter API v2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only. If you need access to this endpoint, you may need a different access level. You can learn more here: https://developer.twitter.com/en/portal/product', 'code': 453}]\r\n```\r\n\r\nIt appears like Twitter has now shut down their v1 endpoints, which is rather gracious of them, considering they [announced they'd be deprecated on 29th April](https://twittercommunity.com/t/reminder-to-migrate-to-the-new-free-basic-or-enterprise-plans-of-the-twitter-api/189737).\r\n\r\nUnfortunately [retrieving likes using the v2 API](https://developer.twitter.com/en/docs/twitter-api/tweets/likes/introduction) is not part of their [free plan](https://developer.twitter.com/en/portal/products). In fact, with the free plan one can only post and delete tweets and retrieve information about oneself.\r\n\r\nSo I'm afraid this is the end of this very nice project. It was very useful, thank you!\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/73/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "draft": null, "state_reason": null} {"id": 1363244199, "node_id": "I_kwDODFdgUs5RQXSn", "number": 75, "title": "Fetch repos doesn't support organisations", "user": {"value": 2757699, "label": "OverkillGuy"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-09-06T12:55:06Z", "updated_at": "2022-09-06T12:55:06Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Say I want to get all my Github Org's repos info, for data analysis. Not just the public repos, but also the private/internal repos.\r\n\r\nThe endpoints are different for organisation, and this tool doesn't take it into account:\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/ace13ec3d98090d99bd71871c286a4a612c96a50/github_to_sqlite/utils.py#L453\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/ace13ec3d98090d99bd71871c286a4a612c96a50/github_to_sqlite/utils.py#L455\r\n\r\nThe endpoints for organisation repos is instead ([source](https://docs.github.com/en/rest/repos/repos#list-organization-repositories)):\r\n`url = \"https://api.github.com/orgs/{}/repos\".format(username)`\r\n\r\nLet's add support for organisations repo scraping.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/75/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 549287310, "node_id": "MDU6SXNzdWU1NDkyODczMTA=", "number": 76, "title": "order_by mechanism", "user": {"value": 10501166, "label": "metab0t"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-01-14T02:06:03Z", "updated_at": "2020-04-16T06:23:29Z", "closed_at": "2020-04-16T03:13:06Z", "author_association": "NONE", "pull_request": null, "body": "In some cases, I want to iterate rows in a table with `ORDER BY` clause. It would be nice to have a `rows_order_by` function similar to `rows_where`.\r\nIn a more general case, `rows_filter` function might be added to allow more customized filtering to iterate rows.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/76/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1505411725, "node_id": "I_kwDODFdgUs5ZusKN", "number": 78, "title": "self-hosted or corp github enterprise", "user": {"value": 549431, "label": "ebdavison"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-12-20T22:51:45Z", "updated_at": "2022-12-20T22:51:45Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "We use github enterprise at work and I would like to use this tool to pull info from that site rather than the public github.com instance. Is there an option for this? If not, can one be added for a custom repo URL?", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/78/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 559197745, "node_id": "MDU6SXNzdWU1NTkxOTc3NDU=", "number": 82, "title": "Tutorial command no longer works", "user": {"value": 10350886, "label": "petey284"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-02-03T16:36:11Z", "updated_at": "2020-02-27T04:16:43Z", "closed_at": "2020-02-27T04:16:30Z", "author_association": "NONE", "pull_request": null, "body": "Issue with command on [tutorial](https://simonwillison.net/2019/Feb/25/sqlite-utils/) on Simon's site.\r\n\r\nThe following command no longer works, and breaks with the previous too many variables error: #50\r\n\r\n``` cmd\r\n> curl \"https://data.nasa.gov/resource/y77d-th95.json\" | \\\r\n sqlite-utils insert meteorites.db meteorites - --pk=id\r\n```\r\n\r\nOutput:\r\n``` cmd\r\nTraceback (most recent call last):\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\runpy.py\", line 193, in _run_module_as_main\r\n \"__main__\", mod_spec)\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\runpy.py\", line 85, in _run_code\r\n exec(code, run_globals)\r\n File \"Continuum\\miniconda3\\envs\\main\\Scripts\\sqlite-utils.exe\\__main__.py\", line 9, in \r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\click\\core.py\", line 764, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\click\\core.py\", line 717, in main\r\n rv = self.invoke(ctx)\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\click\\core.py\", line 1137, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\click\\core.py\", line 956, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\click\\core.py\", line 555, in invoke\r\n return callback(*args, **kwargs)\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\sqlite_utils\\cli.py\", line 434, in insert\r\n default=default,\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\sqlite_utils\\cli.py\", line 384, in insert_upsert_implementation\r\n docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs\r\n File \"continuum\\miniconda3\\envs\\main\\lib\\site-packages\\sqlite_utils\\db.py\", line 1081, in insert_all\r\n result = self.db.conn.execute(query, params)\r\nsqlite3.OperationalError: too many SQL variables\r\n```\r\n\r\nMy thought is that maybe the dataset grew over the last few years and so didn't run into this issue before.\r\n\r\nNo error when I reduce the count of entries to 83. Once the number of entries hits 84 the command fails.\r\n\r\n// This passes\r\n``` cmd\r\ntype meteorite_83.txt | sqlite-utils insert meteorites.db meteorites - --pk=id\r\n```\r\n\r\n// But this fails\r\n``` cmd\r\ntype meteorite_84.txt | sqlite-utils insert meteorites.db meteorites - --pk=id\r\n```\r\n\r\nA potential fix might be to chunk the incoming data? I can work on a PR if pointed in right direction.\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/82/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 564579430, "node_id": "MDU6SXNzdWU1NjQ1Nzk0MzA=", "number": 86, "title": "Problem with square bracket in CSV column name", "user": {"value": 8149512, "label": "foscoj"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-02-13T10:19:57Z", "updated_at": "2020-02-27T04:16:08Z", "closed_at": "2020-02-27T04:16:07Z", "author_association": "NONE", "pull_request": null, "body": "testing some data from european power information (entsoe.eu), the title of the csv contains square brackets.\r\nas I am playing with glitch, sqlite-utils are used for creating the db.\r\n\r\nTraceback (most recent call last):\r\n\r\n File \"/app/.local/bin/sqlite-utils\", line 8, in \r\n\r\n sys.exit(cli())\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/click/core.py\", line 764, in __call__\r\n\r\n return self.main(*args, **kwargs)\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/click/core.py\", line 717, in main\r\n\r\n rv = self.invoke(ctx)\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/click/core.py\", line 1137, in invoke\r\n\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/click/core.py\", line 956, in invoke\r\n\r\n return ctx.invoke(self.callback, **ctx.params)\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/click/core.py\", line 555, in invoke\r\n\r\n return callback(*args, **kwargs)\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/sqlite_utils/cli.py\", line 434, in insert\r\n\r\n default=default,\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/sqlite_utils/cli.py\", line 384, in insert_upsert_implementation\r\n\r\n docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/sqlite_utils/db.py\", line 997, in insert_all\r\n\r\n extracts=extracts,\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/sqlite_utils/db.py\", line 618, in create\r\n\r\n extracts=extracts,\r\n\r\n File \"/app/.local/lib/python3.7/site-packages/sqlite_utils/db.py\", line 310, in create_table\r\n\r\n self.conn.execute(sql)\r\n\r\nsqlite3.OperationalError: unrecognized token: \"]\"\r\n\r\nentsoe_2016.csv\r\n\r\nrenamed to txt for uploading compatibility\r\n\r\n[entsoe_2016.txt](https://github.com/simonw/sqlite-utils/files/4197688/entsoe_2016.txt)\r\n\r\ncode is remixed directly from your https://glitch.com/edit/#!/datasette-csvs repo\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/86/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 577302229, "node_id": "MDU6SXNzdWU1NzczMDIyMjk=", "number": 91, "title": "Enable ordering FTS results by rank", "user": {"value": 416374, "label": "gfrmin"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 1, "created_at": "2020-03-07T08:43:51Z", "updated_at": "2020-11-06T23:53:26Z", "closed_at": "2020-11-06T23:53:25Z", "author_association": "NONE", "pull_request": null, "body": "According to https://www.sqlite.org/fts5.html (not sure about FTS4) results can be sorted by relevance. At the moment results are returned by default by `rowid`. Perhaps a flag can be added to the `search` method?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/91/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 273944952, "node_id": "MDU6SXNzdWUyNzM5NDQ5NTI=", "number": 93, "title": "Package as standalone binary", "user": {"value": 67420, "label": "atomotic"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 18, "created_at": "2017-11-14T21:14:07Z", "updated_at": "2021-11-21T07:00:23Z", "closed_at": "2021-11-21T07:00:23Z", "author_association": "NONE", "pull_request": null, "body": "hint: more than the docker image a standalone and multiplatform binary (containing the app and the database) could be simpler to distribute.\r\n\r\ni would like to investigate the possibility to package everything with [pyinstaller](http://www.pyinstaller.org/) adding the database as a [data file](https://pythonhosted.org/PyInstaller/spec-files.html#adding-data-files)", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/93/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 593751293, "node_id": "MDU6SXNzdWU1OTM3NTEyOTM=", "number": 97, "title": "Adding a \"recreate\" flag to the `Database` constructor", "user": {"value": 1448859, "label": "betatim"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-04-04T05:41:10Z", "updated_at": "2020-04-15T14:29:31Z", "closed_at": "2020-04-13T03:52:29Z", "author_association": "NONE", "pull_request": null, "body": "I have a [script](https://github.com/betatim/binder-datasette/blob/master/create-db.ipynb) that imports data into a sqlite DB. When I re-run that script I'd like to remove the existing sqlite DB, instead of adding to it. The pragmatic answer is to add the check and file deletion to my script.\r\n\r\nHowever I thought it would be easy and useful for others to add a `recreate=True` flag to `db = sqlite_utils.Database(\"binder-launches.db\")`. After taking a look at the code for it I am not so sure any more. This is because the connection string could be a URL (or \"connection string\") like `\"file:///tmp/foo.db\"`. I don't know what the equivalent of `os.path.exists()` is for a connection string or how to detect that something is a connection string and raise an error \"can't use recreate=True and conn_string at the same time\".\r\n\r\nDoes anyone have an idea/suggestion where to start investigating?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/97/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 274160723, "node_id": "MDU6SXNzdWUyNzQxNjA3MjM=", "number": 100, "title": "TemplateAssertionError: no filter named 'tojson'", "user": {"value": 13304454, "label": "coisnepe"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-15T13:43:41Z", "updated_at": "2017-11-16T09:25:10Z", "closed_at": "2017-11-16T00:14:13Z", "author_association": "NONE", "pull_request": null, "body": "A 500 error is raised upon clicking on the name of a table on the homepage, say _http://0.0.0.0:8001/_ to _http://0.0.0.0:8001/test_check-c1f4771/users_ The API part seems to function as intended, though...\r\n\r\n```\r\n2017-11-15 14:33:57 - (sanic)[ERROR]: Traceback (most recent call last):\r\n File \"/usr/local/lib/python3.5/dist-packages/sanic/app.py\", line 503, in handle_request\r\n response = await response\r\n File \"/usr/local/lib/python3.5/dist-packages/datasette/app.py\", line 155, in get\r\n return await self.view_get(request, name, hash, **kwargs)\r\n File \"/usr/local/lib/python3.5/dist-packages/datasette/app.py\", line 219, in view_get\r\n **context,\r\n File \"/usr/local/lib/python3.5/dist-packages/sanic_jinja2/__init__.py\", line 84, in render\r\n return html(self.render_string(template, request, **context))\r\n File \"/usr/local/lib/python3.5/dist-packages/sanic_jinja2/__init__.py\", line 81, in render_string\r\n return self.env.get_template(template).render(**context)\r\n File \"/usr/lib/python3/dist-packages/jinja2/environment.py\", line 812, in get_template\r\n return self._load_template(name, self.make_globals(globals))\r\n File \"/usr/lib/python3/dist-packages/jinja2/environment.py\", line 786, in _load_template\r\n template = self.loader.load(self, name, globals)\r\n File \"/usr/lib/python3/dist-packages/jinja2/loaders.py\", line 125, in load\r\n code = environment.compile(source, name, filename)\r\n File \"/usr/lib/python3/dist-packages/jinja2/environment.py\", line 565, in compile\r\n self.handle_exception(exc_info, source_hint=source_hint)\r\n File \"/usr/lib/python3/dist-packages/jinja2/environment.py\", line 754, in handle_exception\r\n reraise(exc_type, exc_value, tb)\r\n File \"/usr/lib/python3/dist-packages/jinja2/_compat.py\", line 37, in reraise\r\n raise value.with_traceback(tb)\r\n File \"/usr/local/lib/python3.5/dist-packages/datasette/templates/table.html\", line 29, in template\r\n 
params = {{ query.params|tojson(4) }}
\r\n File \"/usr/lib/python3/dist-packages/jinja2/environment.py\", line 515, in _generate\r\n return generate(source, self, name, filename, defer_init=defer_init)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 62, in generate\r\n generator.visit(node)\r\n File \"/usr/lib/python3/dist-packages/jinja2/visitor.py\", line 38, in visit\r\n return f(node, *args, **kwargs)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 849, in visit_Template\r\n self.blockvisit(block.body, block_frame)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 492, in blockvisit\r\n self.visit(node, frame)\r\n File \"/usr/lib/python3/dist-packages/jinja2/visitor.py\", line 38, in visit\r\n return f(node, *args, **kwargs)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 1172, in visit_If\r\n self.blockvisit(node.body, if_frame)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 492, in blockvisit\r\n self.visit(node, frame)\r\n File \"/usr/lib/python3/dist-packages/jinja2/visitor.py\", line 38, in visit\r\n return f(node, *args, **kwargs)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 1353, in visit_Output\r\n self.visit(argument, frame)\r\n File \"/usr/lib/python3/dist-packages/jinja2/visitor.py\", line 38, in visit\r\n return f(node, *args, **kwargs)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 1565, in visit_Filter\r\n self.fail('no filter named %r' % node.name, node.lineno)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 427, in fail\r\n raise TemplateAssertionError(msg, lineno, self.name, self.filename)\r\njinja2.exceptions.TemplateAssertionError: no filter named 'tojson'\r\n\r\n2017-11-15 14:33:57 - (network)[INFO][127.0.0.1:41316]: GET http://0.0.0.0:8001/test_check-c1f4771/users 500 144\r\n2017-11-15 14:33:57 - (network)[INFO][127.0.0.1:41316]: GET http://0.0.0.0:8001/favicon.ico 200 0\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/100/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 274161964, "node_id": "MDU6SXNzdWUyNzQxNjE5NjQ=", "number": 101, "title": "TemplateAssertionError: no filter named 'tojson'", "user": {"value": 450244, "label": "eaubin"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2017-11-15T13:47:32Z", "updated_at": "2017-11-15T13:48:55Z", "closed_at": "2017-11-15T13:48:55Z", "author_association": "NONE", "pull_request": null, "body": "I get an exception clicking on the table link: \r\n\r\n```\r\n2017-11-15 08:40:10 - (sanic)[ERROR]: Traceback (most recent call last):\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/sanic/app.py\", line 503, in handle_request\r\n response = await response\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/datasette/app.py\", line 155, in get\r\n return await self.view_get(request, name, hash, **kwargs)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/datasette/app.py\", line 219, in view_get\r\n **context,\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/sanic_jinja2/__init__.py\", line 84, in render\r\n return html(self.render_string(template, request, **context))\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/sanic_jinja2/__init__.py\", line 81, in render_string\r\n return self.env.get_template(template).render(**context)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py\", line 812, in get_template\r\n return self._load_template(name, self.make_globals(globals))\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py\", line 786, in _load_template\r\n template = self.loader.load(self, name, globals)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/loaders.py\", line 125, in load\r\n code = environment.compile(source, name, filename)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py\", line 565, in compile\r\n self.handle_exception(exc_info, source_hint=source_hint)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py\", line 754, in handle_exception\r\n reraise(exc_type, exc_value, tb)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/_compat.py\", line 37, in reraise\r\n raise value.with_traceback(tb)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/datasette/templates/table.html\", line 29, in template\r\n 
params = {{ query.params|tojson(4) }}
\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py\", line 515, in _generate\r\n return generate(source, self, name, filename, defer_init=defer_init)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py\", line 62, in generate\r\n generator.visit(node)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/visitor.py\", line 38, in visit\r\n return f(node, *args, **kwargs)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py\", line 849, in visit_Template\r\n self.blockvisit(block.body, block_frame)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py\", line 492, in blockvisit\r\n self.visit(node, frame)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/visitor.py\", line 38, in visit\r\n return f(node, *args, **kwargs)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py\", line 1172, in visit_If\r\n self.blockvisit(node.body, if_frame)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py\", line 492, in blockvisit\r\n self.visit(node, frame)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/visitor.py\", line 38, in visit\r\n return f(node, *args, **kwargs)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py\", line 1353, in visit_Output\r\n self.visit(argument, frame)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/visitor.py\", line 38, in visit\r\n return f(node, *args, **kwargs)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py\", line 1565, in visit_Filter\r\n self.fail('no filter named %r' % node.name, node.lineno)\r\n File \"/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py\", line 427, in fail\r\n raise TemplateAssertionError(msg, lineno, self.name, self.filename)\r\njinja2.exceptions.TemplateAssertionError: no filter named 'tojson'\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/101/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 613755043, "node_id": "MDU6SXNzdWU2MTM3NTUwNDM=", "number": 110, "title": "Support decimal.Decimal type", "user": {"value": 134771, "label": "dvhthomas"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-05-07T03:57:19Z", "updated_at": "2020-05-11T01:58:20Z", "closed_at": "2020-05-11T01:50:11Z", "author_association": "NONE", "pull_request": null, "body": "Decimal types in Postgres cause a failure in db.py data type selection\r\n---\r\nI have a Django app using a MoneyField, which uses a `numeric(14,0)` data type in Postgres (https://www.postgresql.org/docs/9.3/datatype-numeric.html). When attempting to export that table I get the following error:\r\n\r\n```bash\r\n$ db-to-sqlite --table isaweb_proposal \"postgres://connection\" test.db\r\n....\r\n column_type=COLUMN_TYPE_MAPPING[column_type],\r\nKeyError: \r\n```\r\n\r\nLooking at `sql_utils.db.py` at 292-ish it's clear that there is no matching type for what I assume SQLAlchemy interprets as Python decimal.Decimal.\r\n\r\nFrom the [SQLite docs](https://www.sqlite.org/datatype3.html#affinity_name_examples) it looks like DECIMAL in other DBs are considered numeric.\r\n\r\nI'm not quite sure if it's as simple as adding a data type to that list or if there are repercussions beyond it.\r\n\r\nThanks for a great tool!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/110/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 276091279, "node_id": "MDU6SXNzdWUyNzYwOTEyNzk=", "number": 144, "title": "apsw as alternative sqlite3 binding (for full text search)", "user": {"value": 649467, "label": "mhalle"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2017-11-22T14:40:39Z", "updated_at": "2018-05-28T21:29:42Z", "closed_at": "2018-05-28T21:29:42Z", "author_association": "NONE", "pull_request": null, "body": "Hey there,\r\n\r\nHave you considered providing apsw support as an alternative to stock python sqlite3? I use apsw because it keeps up with sqlite3 and is straightforward to bring in extensions like FTS5. FTS really accelerates the kind of searching often done by web clients.\r\n\r\nI may be able to help (it shouldn't be much code), but there are a couple of stylistic questions that come up when supporting an optional package. Also, apsw is tricky in that it doesn't have a pypi package (author says limitations in providing options to setup.py). ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/144/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 276842536, "node_id": "MDU6SXNzdWUyNzY4NDI1MzY=", "number": 153, "title": "Ability to customize presentation of specific columns in HTML view", "user": {"value": 20264, "label": "ftrain"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2949431, "label": "Custom templates edition"}, "comments": 14, "created_at": "2017-11-26T17:46:11Z", "updated_at": "2017-12-10T02:08:45Z", "closed_at": "2017-12-07T06:17:33Z", "author_association": "NONE", "pull_request": null, "body": "This ties into https://github.com/simonw/datasette/issues/3 in some ways. It would be great to have some adaptability in the HTML views and to specific some columns as displaying in certain ways.\r\n\r\n- [x] 1. **Auto-parsing URIs into in-browser links.** Why? Lots of public data around cultural commons stuff links to a specific URL. This would be a great utility to turn on at the command line, just parse everything for URLs. Maybe they need to be underlined or represented in a different way than internal URLs.\r\n- [x] 2. **Ability to identify a column as plain/preformatted text.** Why? Was trying to import the Enron emails, the body collapses. Hard to read. These fields also tend to screw up the ability to scan a table view. If you knew it was text the system could set an `overflow` property on the relevant CSS, so you could still scan.\r\n- [x] 3. **Ability to identify a column as HTML.** Why? I want to spider some stuff and drop sections into SQLite, and just keep them as HTML.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/153/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 277589569, "node_id": "MDU6SXNzdWUyNzc1ODk1Njk=", "number": 155, "title": "A primary key column that has foreign key restriction associated won't rendering label column", "user": {"value": 388154, "label": "wsxiaoys"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2949431, "label": "Custom templates edition"}, "comments": 4, "created_at": "2017-11-29T00:40:02Z", "updated_at": "2017-12-07T05:39:53Z", "closed_at": "2017-12-07T05:39:53Z", "author_association": "NONE", "pull_request": null, "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/155/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 702386948, "node_id": "MDU6SXNzdWU3MDIzODY5NDg=", "number": 159, "title": ".delete_where() does not auto-commit (unlike .insert() or .upsert())", "user": {"value": 11712349, "label": "spdkils"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-09-16T01:55:52Z", "updated_at": "2023-04-01T17:21:05Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "When you use the delete_where() function on a table, it never commits....\r\n\r\nIs that intentional?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/159/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 278814220, "node_id": "MDU6SXNzdWUyNzg4MTQyMjA=", "number": 161, "title": "Support WITH query ", "user": {"value": 388154, "label": "wsxiaoys"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2017-12-03T20:00:40Z", "updated_at": "2017-12-08T06:18:12Z", "closed_at": "2017-12-04T04:52:41Z", "author_association": "NONE", "pull_request": null, "body": "Currently datasettle failed with error message: Statement must begin with SELECT\r\n\r\nExample query\r\n```sql\r\nWITH RECURSIVE\r\n cnt(x) AS (\r\n SELECT 1\r\n UNION ALL\r\n SELECT x+1 FROM cnt\r\n LIMIT 1000000\r\n )\r\nSELECT x FROM cnt;\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/161/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 707407567, "node_id": "MDU6SXNzdWU3MDc0MDc1Njc=", "number": 171, "title": "Idea: transitive closure tables for tree structures", "user": {"value": 649467, "label": "mhalle"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-23T14:17:33Z", "updated_at": "2020-10-22T04:38:35Z", "closed_at": "2020-10-22T04:07:14Z", "author_association": "NONE", "pull_request": null, "body": "I just read that sqlite has a transitive closure table extension using a virtual table in order to represent trees:\r\n\r\nhttps://charlesleifer.com/blog/querying-tree-structures-in-sqlite-using-python-and-the-transitive-closure-extension/\r\n\r\nEven without this extension, though, a util to build a transitive closure table would allow trees to be queried easily. Since it relies on self-referential foreign keys, the relationships might even be able to be automatically detected. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/171/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 281110295, "node_id": "MDU6SXNzdWUyODExMTAyOTU=", "number": 173, "title": "I18n and L10n support", "user": {"value": 50138, "label": "janimo"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-12-11T17:49:58Z", "updated_at": "2021-04-26T12:10:01Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "It would be less geeky and more user friendly if the display strings in the filter menu and possibly other parts could be localized.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/173/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 282971961, "node_id": "MDU6SXNzdWUyODI5NzE5NjE=", "number": 175, "title": "Add project topic \"automatic-api\"", "user": {"value": 3179832, "label": "dbohdan"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2017-12-18T18:09:17Z", "updated_at": "2017-12-21T18:33:55Z", "closed_at": "2017-12-21T18:33:55Z", "author_association": "NONE", "pull_request": null, "body": "Hi there! Could you add the ~~tag~~ topic `automatic-api` to your repository? I am [making a list](https://github.com/dbohdan/automatic-api) of all projects that automatically expose APIs to databases. (Your Show HN made me do it. :-) I knew about PostgREST and PostGraphQL, but it took adding Datasette to sell me on the concept.) They will be easier to discover if there is a standard GitHub tag, and `automatic-api` seems as good a candidate as any. Two projects [already use it](https://github.com/topics/automatic-api).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/175/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 285168503, "node_id": "MDU6SXNzdWUyODUxNjg1MDM=", "number": 176, "title": "Add GraphQL endpoint", "user": {"value": 173848, "label": "yozlet"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2017-12-29T23:21:01Z", "updated_at": "2020-04-21T14:16:24Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Would make it much easier to build React & similar frontends. Maybe with https://github.com/graphql-python/sanic-graphql ?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/176/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 289425975, "node_id": "MDExOlB1bGxSZXF1ZXN0MTYzNTYxODMw", "number": 181, "title": "add \"format sql\" button to query page, uses sql-formatter", "user": {"value": 1957344, "label": "bsmithgall"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2018-01-17T21:50:04Z", "updated_at": "2019-11-11T03:08:25Z", "closed_at": "2019-11-11T03:08:25Z", "author_association": "NONE", "pull_request": "simonw/datasette/pulls/181", "body": "Cool project!\r\n\r\nThis fixes #136 using the suggested [sql formatter](https://github.com/zeroturnaround/sql-formatter) library. I included the minified version in the bundle and added the relevant scripts to the codemirror includes instead of adding new files, though I could also add new files. I wanted to keep it all together, since the result of the format needs access to the editor in order to properly update the codemirror instance.", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/181/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 711649325, "node_id": "MDU6SXNzdWU3MTE2NDkzMjU=", "number": 182, "title": "Better handling of encodings other than utf-8 for \"sqlite-utils insert\"", "user": {"value": 765871, "label": "kaihendry"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-09-30T05:43:48Z", "updated_at": "2020-10-16T17:20:41Z", "closed_at": "2020-10-16T17:18:52Z", "author_association": "NONE", "pull_request": null, "body": "Makefile:\r\n```\r\ndata.db:\r\n curl -O http://maps.natalian.org/data.txt\r\n go run csv-write.go > data.csv\r\n sqlite-utils insert data.db travels data.csv --csv\r\n\r\nclean:\r\n rm data*\r\n```\r\n[csv-write.go](https://gist.github.com/kaihendry/dff2442de20d73f900026d13bf7a11d9)\r\n\r\n\r\nError message is:\r\n\r\n```\r\nsqlite-utils insert data.db travels data.csv --csv\r\nTraceback (most recent call last):\r\n File \"/home/hendry/.local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 614, in insert\r\n insert_upsert_implementation(\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 553, in insert_upsert_implementation\r\n headers = next(reader)\r\n File \"/usr/lib/python3.8/codecs.py\", line 322, in decode\r\n (result, consumed) = self._buffer_decode(data, self.errors, final)\r\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xe3 in position 1234: invalid continuation byte\r\nmake: *** [Makefile:4: data.db] Error 1\r\n[hendry@t14s datasette-map]$ sqlite-utils --version\r\nsqlite-utils, version 2.19\r\n```\r\n\r\nLittle bit surprised if Go is spewing out bad Unicode, but I'm not sure how to grok `position 1234`..\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/182/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 292011379, "node_id": "MDU6SXNzdWUyOTIwMTEzNzk=", "number": 184, "title": "500 from missing table name", "user": {"value": 222245, "label": "carlmjohnson"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-01-26T19:46:45Z", "updated_at": "2019-05-21T16:17:29Z", "closed_at": "2018-04-13T18:18:59Z", "author_association": "NONE", "pull_request": null, "body": "https://github.com/simonw/datasette/blob/56623e48da5412b25fb39cc26b9c743b684dd968/datasette/app.py#L517-L519 throws an error if it gets an empty list back. Simplest solution is to write a helper func that just says \r\n\r\n```python\r\nresult = list(await self.execute(name, sql, params)\r\nif result:\r\n return result[0][0]\r\n```\r\n\r\nand use it anywhere `[0][0]` is now.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/184/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 299760684, "node_id": "MDU6SXNzdWUyOTk3NjA2ODQ=", "number": 185, "title": "Metadata should be a nested arbitrary KV store", "user": {"value": 222245, "label": "carlmjohnson"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2018-02-23T16:02:07Z", "updated_at": "2019-05-13T18:33:33Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I started using the metadata feature and was surprised to find that values are not inherited from the root object down to specific databases and tables. This makes metadata much less useful and requires a lot of pointless duplication.\r\n\r\nIdeally, metadata should allow arbitrary key-value pairs, and there should be a way of accessing metadata either in an inherited or non-inherited manner. Something like `metadata.page.key` vs. `metadata.this.key` might work as an interface.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/185/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}