{"id": 496415321, "node_id": "MDU6SXNzdWU0OTY0MTUzMjE=", "number": 1, "title": "Figure out some interesting example SQL queries", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2019-09-20T15:28:07Z", "updated_at": "2021-05-03T03:46:23Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "My knowledge of genetics has left me short here. I'd love to be able to provide some interesting example SELECT queries - maybe one that spots if you are [likely to have red hair?](https://www.snpedia.com/index.php/Rs1805007)", "repo": {"value": 209590345, "label": "genome-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 504720731, "node_id": "MDU6SXNzdWU1MDQ3MjA3MzE=", "number": 1, "title": "Add more details on how to request data from google takeout correctly.", "user": {"value": 1055831, "label": "dazzag24"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-10-09T15:17:34Z", "updated_at": "2019-10-09T15:17:34Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "The default is to download everything. This can result in an enormous amount of data when you only really need 2 types of data for now:\r\n\r\n- My Activity\r\n- Location History\r\n\r\nIn addition unless you specify that \"My Activity\" is downloaded in JSON format the default is HTML. This then causes the \r\n\r\n`google-takeout-to-sqlite my-activity takeout.db takeout.zip`\r\n\r\ncommand to fail as it only contains html files not json files.\r\n\r\nThanks", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 602533300, "node_id": "MDU6SXNzdWU2MDI1MzMzMDA=", "number": 1, "title": "Import photo metadata from Apple Photos into SQLite", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 5324096, "label": "Apple Photos online and securely browsable"}, "comments": 8, "created_at": "2020-04-18T19:23:26Z", "updated_at": "2020-05-04T02:41:40Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Faces, albums, locations, that kind of thing.", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 718934942, "node_id": "MDU6SXNzdWU3MTg5MzQ5NDI=", "number": 1, "title": "Documentation on how to use this with Datasette", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-10-11T21:56:27Z", "updated_at": "2020-10-11T22:14:00Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "In particular how to use `datasette-render-images` to see the images.", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1353411865, "node_id": "I_kwDODEpn8M5Qq20Z", "number": 1, "title": "Problem with my user", "user": {"value": 2467, "label": "fernand0"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-08-28T16:59:37Z", "updated_at": "2022-08-28T16:59:37Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "If I call the program with:\r\n inaturalist-to-sqlite inaturalist.db ftricas\r\nthe program exits with an error:\r\n `Importing 36 observations\r\nTraceback (most recent call last):\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/bin/inaturalist-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/inaturalist_to_sqlite/cli.py\", line 51, in cli\r\n save_observation(observation, db)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/inaturalist_to_sqlite/utils.py\", line 34, in save_observation\r\n db[\"observations\"]\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 2965, in insert\r\n return self.insert_all(\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 3068, in insert_all\r\n self.create(\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 1564, in create\r\n self.db.create_table(\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 951, in create_table\r\n sql = self.create_table_sql(\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 765, in create_table_sql\r\n foreign_keys = self.resolve_foreign_keys(name, foreign_keys or [])\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 702, in resolve_foreign_keys\r\n other_table = table.guess_foreign_table(column)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 2061, in guess_foreign_table\r\n raise NoObviousTable(\r\nsqlite_utils.db.NoObviousTable: No obvious foreign key table for column 'taxon' - tried ['taxon', 'taxons']\r\n`\r\nIf I call the program with your user everything seems to go well and then, I can call the program with my own user without problems. Moreover, I can call the program again with my own user and everything goes well now.\r\n\r\nAdditional info, the command:\r\n \r\n sqlite-utils tables inaturalist.db\r\n\r\nshows that the correct name can be 'taxons'.\r\n\r\nThere is another small problem with a warning:\r\n \r\n warnings.warn(\"urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported \"\r\n\r\n", "repo": {"value": 206202864, "label": "inaturalist-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/inaturalist-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 541274681, "node_id": "MDU6SXNzdWU1NDEyNzQ2ODE=", "number": 2, "title": "Add linkedin-to-sqlite", "user": {"value": 881925, "label": "mnp"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-12-21T03:13:40Z", "updated_at": "2019-12-21T03:13:40Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "There is an API available. https://developer.linkedin.com/docs/rest-api#\r\n\r\nAt the minimum, I would think contact list and messages would be of interest.", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 664793260, "node_id": "MDU6SXNzdWU2NjQ3OTMyNjA=", "number": 2, "title": "Yak shave", "user": {"value": 145425, "label": "ekg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-07-23T22:04:18Z", "updated_at": "2020-07-23T22:04:18Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Just a quick note... The 23andme data is not exactly your genome, but a SNP chip of your genome. It's \"some of your genotypes.\" Or about 0.1% of your genome. Nice work in any case! It deserves to be liberated!!!!!", "repo": {"value": 209590345, "label": "genome-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 769376447, "node_id": "MDU6SXNzdWU3NjkzNzY0NDc=", "number": 2, "title": "killed by oomkiller on large location-history", "user": {"value": 231498, "label": "khimaros"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-12-17T00:32:24Z", "updated_at": "2020-12-17T00:48:32Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "memory seems to grow unbounded and is oom-killed after about 20GB memory usage.\r\n\r\nthis is happening while loading a ~1GB uncompressed location history.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 952179830, "node_id": "MDU6SXNzdWU5NTIxNzk4MzA=", "number": 2, "title": "Command for fetching Hacker News threads from the search API", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-07-25T02:00:45Z", "updated_at": "2021-07-25T03:12:57Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I want to be able to fetch every item for a domain, e.g. https://news.ycombinator.com/from?site=simonwillison.net", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1485017981, "node_id": "I_kwDODEpn8M5Yg5N9", "number": 2, "title": "table identifications has no column named previous_observation_taxon", "user": {"value": 520541, "label": "heaversm"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-12-08T16:47:17Z", "updated_at": "2022-12-08T16:47:17Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Installed successfully with pip and ran `inaturalist-to-sqlite inaturalist.db simonw` and got the error:\r\n\r\n```\r\nsqlite3.OperationalError: table identifications has no column named previous_observation_taxon\r\n```", "repo": {"value": 206202864, "label": "inaturalist-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/inaturalist-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 267515678, "node_id": "MDU6SXNzdWUyNjc1MTU2Nzg=", "number": 3, "title": "Make individual column valuables addressable, with smart content types", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2017-10-23T01:11:32Z", "updated_at": "2017-12-10T03:11:58Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Some SQLite databases embed images in columns. It would be cool if these had URLs.\r\n\r\n /database-name-7sha256/table-name/compound-pk/column\r\n /database-name-7sha256/table-name/compound-pk/column.json\r\n /database-name-7sha256/table-name/compound-pk/column.png\r\n /database-name-7sha256/table-name/compound-pk/column.gif\r\n /database-name-7sha256/table-name/compound-pk/column.txt\r\n\r\nThe one without an explicit file extension auto-detects the correct extension.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 503243784, "node_id": "MDU6SXNzdWU1MDMyNDM3ODQ=", "number": 3, "title": "Extract images into separate tables", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-10-07T05:43:01Z", "updated_at": "2020-09-01T06:17:45Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "As already done with authors. Slightly harder because images do not have a universally unique ID. Also need to figure out what to do about there being columns for both `image` and `images`.\r\n\r\n\"memory__items\"\r\n", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 602533481, "node_id": "MDU6SXNzdWU2MDI1MzM0ODE=", "number": 3, "title": "Import EXIF data into SQLite - lens used, ISO, aperture etc", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 5324096, "label": "Apple Photos online and securely browsable"}, "comments": 2, "created_at": "2020-04-18T19:24:31Z", "updated_at": "2021-10-05T12:38:24Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 769397742, "node_id": "MDU6SXNzdWU3NjkzOTc3NDI=", "number": 3, "title": "sqlite-utils error on takeout import", "user": {"value": 231498, "label": "khimaros"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-12-17T01:18:48Z", "updated_at": "2020-12-17T01:19:04Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "```\r\n$ google-takeout-to-sqlite my-activity takeout.db /path/to/zip\r\n...\r\nsqlite3.OperationalError: no such table: main.my_activity\r\n```\r\n\r\nthere is no table create in `utils.py`, unlike other importers such as github-to-sqlite\r\n\r\nadditionally, this package and hackernews-to-sqlite have conflicting `sqlite-utils` dep with datasette and dogsheep-beta", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/3/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 952189173, "node_id": "MDU6SXNzdWU5NTIxODkxNzM=", "number": 3, "title": "Use HN algolia endpoint to retrieve trees", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-07-25T03:35:27Z", "updated_at": "2021-07-25T18:41:17Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "The `trees` command currently has to make a request for every single comment. Algolia have an endpoint that bundles the entire thread together into a single request.\r\n\r\n`https://hn.algolia.com/api/v1/items/ID`\r\n\r\nHere's an example that loads quickly, with about 50 comments: https://hn.algolia.com/api/v1/items/27941108\r\n\r\nIt doesn't appear to use pagination at all - if a thread is big then the response is big.\r\n\r\nI ran this search to find some stories with more than 1000 comments: https://hn.algolia.com/api/v1/search?tags=story&numericFilters=num_comments%3E=1000\r\n\r\nHere's one: https://news.ycombinator.com/item?id=25015967 with 4759 comments. Hitting the API takes 41s and returns 3.7 MB of JSON!\r\n```\r\nwget 'https://hn.algolia.com/api/v1/items/25015967' 0.03s user 0.04s system 0% cpu 41.368 total\r\n/tmp % ls -lah 25015967 \r\n-rw-r--r-- 1 simon wheel 3.7M Jul 24 20:31 25015967\r\n```", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 778380836, "node_id": "MDU6SXNzdWU3NzgzODA4MzY=", "number": 4, "title": "Feature Request: Gmail", "user": {"value": 203343, "label": "Btibert3"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-01-04T21:31:09Z", "updated_at": "2021-03-04T20:54:44Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "From takeout, I only exported my Gmail account. Ideally I could parse this into sqlite via this tool.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1205867842, "node_id": "I_kwDODtX3eM5H4BVC", "number": 4, "title": "Retrieve the top-level story for a comment", "user": {"value": 1755789, "label": "telotortium"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-04-15T20:25:39Z", "updated_at": "2022-04-15T20:25:39Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I think that each comment inserted into the database should include a column `onstory` that contains the ID of the story on which the comment was made. This is exactly equivalent to the link after \"on:\" at the top of an HN comment page ([example](https://news.ycombinator.com/item?id=18358028)). We could do this either by directly retrieving the HTML page and using Beautiful Soup to find that link, or alternatively recurse up the tree in the Firebase API using the `parent` field (probably using `functools.lru_cache` in case a person has commented a bunch of times on the same story).", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1616429236, "node_id": "I_kwDOJHON9s5gWMC0", "number": 4, "title": "Support incremental updates", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-03-09T05:14:00Z", "updated_at": "2023-03-09T18:20:56Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Running this script can take several hours against a large notes database.\r\n\r\nWould be neat if it could run against just the notes that have been modified since it last ran. Could pull the max `updated` date and then keep on looping until it finds one modified before then.\r\n\r\nProblem is I don't actually know what order it iterates over the notes in.", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 488874815, "node_id": "MDU6SXNzdWU0ODg4NzQ4MTU=", "number": 5, "title": "Write tests that simulate the Twitter API", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-09-03T23:55:35Z", "updated_at": "2019-09-03T23:56:28Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I can use betamax for this: https://pypi.org/project/betamax/", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 629473827, "node_id": "MDU6SXNzdWU2Mjk0NzM4Mjc=", "number": 5, "title": "Set up a demo", "user": {"value": 26745575, "label": "harryvederci"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-06-02T19:56:49Z", "updated_at": "2020-09-01T06:18:43Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "First off, thanks for open sourcing this application! This is a suggestion to increase the amount of people that would make use of it: an example in the readme file would help.\r\n\r\nCurrently, users have to clone the app, install it, authorize through pocket, run a command, an then find out if this application does what they hope it does.\r\n\r\nAnother possibility is to add a file `example-output.db`, containing one (mock) Pocket article.\r\n\r\nKeep up the good work!", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 718938889, "node_id": "MDU6SXNzdWU3MTg5Mzg4ODk=", "number": 5, "title": "Figure out how to display images from tags inline in Datasette", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-10-11T22:17:03Z", "updated_at": "2020-10-16T20:16:28Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Relates to #1. Evernote XML looks like this:\r\n\r\n```xml\r\n\r\n\r\n
This note includes two images.
\r\n
\r\n The Python logo\r\n
\r\n
\r\n \r\n
\r\n
\r\n The Evernote logo\r\n
\r\n
\r\n \r\n
\r\n\r\n```\r\nThat hash is the md5 we use to store resources. It should be possible to turn these into embedded image tags, especially if done in conjunction with the https://github.com/simonw/datasette-media plugin.", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1616440856, "node_id": "I_kwDOJHON9s5gWO4Y", "number": 5, "title": "Configure full text search", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-03-09T05:20:46Z", "updated_at": "2023-03-09T05:20:46Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "FTS would be useful.\r\n\r\nMaybe even extract the plain text from the notes to make that index easier to create, rather than creating it against the HTML. Can use the `plaintext` property for that.", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 689848827, "node_id": "MDU6SXNzdWU2ODk4NDg4Mjc=", "number": 6, "title": "ISO timestamps", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-01T06:16:42Z", "updated_at": "2020-09-01T06:16:42Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "The `time_added`, `time_updated` and `time_read` columns currently store data like this:\r\n\r\n September 19, 2019 - 00:30:30 UTC\r\n\r\nShould use ISO instead, e.g. `2020-07-26T01:05:24+00:00`", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 689850810, "node_id": "MDU6SXNzdWU2ODk4NTA4MTA=", "number": 6, "title": "Set up a demo instance", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-01T06:20:24Z", "updated_at": "2020-09-01T06:20:24Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Once I've got the Datasette plugin to a state where it's worth building a demo: #3\r\n\r\nI can use data from my public https://github-to-sqlite.dogsheep.net/ demo plus the Pocket data subset I use for the demo in https://github.com/dogsheep/pocket-to-sqlite/issues/5 - I could pull in the https://dogsheep-photos.dogsheep.net/ photos data too.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 821841046, "node_id": "MDU6SXNzdWU4MjE4NDEwNDY=", "number": 6, "title": "Upgrade to latest sqlite-utils", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-03-04T07:21:54Z", "updated_at": "2021-03-04T07:22:51Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "This is pinned to v1 at the moment.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1617602868, "node_id": "I_kwDOJHON9s5gaqk0", "number": 6, "title": "Character encoding problem", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-03-09T16:44:34Z", "updated_at": "2023-04-14T15:22:09Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I ran against a recent note with this in it:\r\n\r\n> Or just \"Actions \u2699\ufe0f \"\r\n\r\nAnd got back:\r\n\r\n\"image\"\r\n\r\n> `Actions \u201a\u00f6\u00f4\u00d4\u220f\u00e8`\r\n\r\nPasting that into https://ftfy.vercel.app/?s=Actions+%E2%80%9A%C3%B6%C3%B4%C3%94%E2%88%8F%C3%A8+ gives this:\r\n\r\n```python\r\ns = 'Actions \u00e2\\x80\\x9a\u00c3\u00b6\u00c3\u00b4\u00c3\\x94\u00e2\\x88\\x8f\u00c3\u00a8'\r\ns = s.encode('latin-1')\r\ns = s.decode('utf-8')\r\ns = s.encode('macroman')\r\ns = s.decode('utf-8')\r\nprint(s)\r\n```\r\n\"image\"\r\n", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 602585497, "node_id": "MDU6SXNzdWU2MDI1ODU0OTc=", "number": 7, "title": "Integrate image content hashing", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-19T00:36:58Z", "updated_at": "2021-08-26T02:01:01Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "To spot duplicate images (where the file content differs such that the sha256 is no longer a match) it would be useful to calculate and store perceptual hashes of some sort.", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 930946817, "node_id": "MDU6SXNzdWU5MzA5NDY4MTc=", "number": 7, "title": "KeyError: 'accuracy' when processing Location History", "user": {"value": 403152, "label": "davidwilemski"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-06-27T14:39:43Z", "updated_at": "2021-06-27T14:39:43Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I'm new to both the dogsheep tools and datasette but have been experimenting a bit the last few days and these are really cool tools!\r\n\r\nI encountered a problem running my Google location history through this tool running the latest release in a docker container:\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/google-takeout-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/cli.py\", line 49, in my_activity\r\n utils.save_location_history(db, zf)\r\n File \"/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py\", line 27, in save_location_history\r\n db[\"location_history\"].upsert_all(\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1105, in upsert_all\r\n return self.insert_all(\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 990, in insert_all\r\n chunk = list(chunk)\r\n File \"/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py\", line 33, in \r\n \"accuracy\": row[\"accuracy\"],\r\nKeyError: 'accuracy'\r\n```\r\n\r\nIt looks like the tool assumes the `accuracy` key will be in every location history entry. \r\n\r\nMy first attempt at a local patch to get myself going was to convert accessing the `accuracy` key to a `.get` instead to hopefully make the row nullable but I wasn't quite sure what `sqlite_utils` would do there. That did work in that the import happened and so I was going to propose a patch that made that change but in updating the existing test to include an entry with a missing accuracy entry, I noticed the expected type of the field appeared to be changing to a string in the test (and from a quick scan through the sqlite_utils code, probably TEXT in the database). Given this change in column type, it seemed that opening an issue first before proposing a fix seemed warranted. It seems the schema would need to be explicitly specified if you wanted a nullable integer column.\r\n\r\nNow that I've done a successful import run using my initial fix of calling `.get` on the row dict, I can see with datasette that I only have 7 data points (out of ~250k) that have a null accuracy column. They are all from 2011-2012 in an import that includes points spanning ~2010-2016 so perhaps another approach might be to filter those entries out during import if it really is that infrequent?\r\n\r\nI'm happy to provide a PR for a fix but figured I'd ask about which direction is preferred first.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 797728929, "node_id": "MDU6SXNzdWU3OTc3Mjg5Mjk=", "number": 8, "title": "QUESTION: extract full text", "user": {"value": 417363, "label": "darribas"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-01-31T14:50:10Z", "updated_at": "2021-01-31T14:50:10Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "This may be solved or a feature already, but I couldn't figure it out, is it possible to extract and store also full text from the saved pages? The same way that Pocket parses the text, it'd be amazing to be able to store (and thus make searchable later) the text.\r\n\r\nThank you very much for the project, it's such an amazing idea! ", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 927385540, "node_id": "MDU6SXNzdWU5MjczODU1NDA=", "number": 8, "title": "any guidance / experience on imessage-to-sqlite ?", "user": {"value": 2675621, "label": "Casyfill"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-06-22T15:46:16Z", "updated_at": "2021-06-22T15:46:16Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 673602857, "node_id": "MDU6SXNzdWU2NzM2MDI4NTc=", "number": 9, "title": "Define a view that displays photos correctly", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-08-05T14:53:39Z", "updated_at": "2020-08-05T14:53:39Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "The `photos` table stores data like this:\r\n\r\nid | createdAt | source | prefix | suffix | width | height | visibility | created\u00a0\u25b2 | user\r\n-- | -- | -- | -- | -- | -- | -- | -- | -- | --\r\n5e12c9708506bc000840262a | January 06, 2020 - 05:45:20 UTC | Swarm for iOS\u00a01 | https://fastly.4sqi.net/img/general/ | /15889193_AXxGk4I1nbzUZuyYqObgbXdJNyEHiwj6AUDq0tPZWtw.jpg | 1920 | 1440 | public | 2020-01-06T05:45:20 | 15889193\r\n\r\nThe photo URL can be derived from those pieces - define a SQL view which does that (using `datasette-json-html` to display the pictures)", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/9/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 801780625, "node_id": "MDU6SXNzdWU4MDE3ODA2MjU=", "number": 9, "title": "SSL Error", "user": {"value": 12669260, "label": "jfeiwell"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-02-05T02:12:56Z", "updated_at": "2021-02-07T18:45:04Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Here's the error I get when running `pip install pocket-to-sqlite`: \r\n\r\n```\r\n Could not fetch URL https://pypi.python.org/simple/pocket-to-sqlite/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661) - skipping\r\n Could not find a version that satisfies the requirement pocket-to-sqlite (from versions: )\r\nNo matching distribution found for pocket-to-sqlite\r\n```\r\n\r\nDoes this require python 3? ", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1617938730, "node_id": "I_kwDOJHON9s5gb8kq", "number": 9, "title": "Default to just storing plaintext, store HTML if `--html` is passed", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-03-09T20:19:06Z", "updated_at": "2023-03-09T20:19:06Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "The full `body` version of the notes can get HUGE, due to embedded images. It turns out for my own purposes I'm usually happy with just the `plaintext` version.\r\n\r\nI'm tempted to say you don't get HTML unless you pass a `--html` option.", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/9/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1123393829, "node_id": "I_kwDODFE5qs5C9aEl", "number": 10, "title": "sqlite3.OperationalError: no such table: main.my_activity", "user": {"value": 69208826, "label": "glxblt14"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-02-03T17:59:29Z", "updated_at": "2022-03-20T02:38:07Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hello,\r\nWhen i run the command `google-takeout-to-sqlite my-activity db.db takeout-20220203T174446Z-001.zip`, i get this error :\r\n```\r\nTraceback (most recent call last):\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\runpy.py\", line 197, in _run_module_as_main\r\n return _run_code(code, main_globals, None,\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\runpy.py\", line 87, in _run_code\r\n exec(code, run_globals)\r\n File \"C:\\Users\\julie\\AppData\\Local\\Programs\\Python\\Python39-32\\Scripts\\google-takeout-to-sqlite.exe\\__main__.py\", line 7, in \r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\click\\core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\google_takeout_to_sqlite\\cli.py\", line 31, in my_activity\r\n utils.save_my_activity(db, zf)\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\google_takeout_to_sqlite\\utils.py\", line 19, in save_my_activity\r\n db[\"my_activity\"].create_index([\"time\"])\r\n File \"c:\\users\\julie\\appdata\\local\\programs\\python\\python39-32\\lib\\site-packages\\sqlite_utils\\db.py\", line 629, in create_index\r\n self.db.conn.execute(sql)\r\nsqlite3.OperationalError: no such table: main.my_activity\r\n```\r\nThank you for your help\r\nSorry for my bad English\r\nEDIT: i used the json format", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/10/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1618130434, "node_id": "I_kwDOJHON9s5gcrYC", "number": 11, "title": "Implement a SQL view to make it easier to query files in a nested folder", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-03-09T23:19:28Z", "updated_at": "2023-03-09T23:24:01Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Working with nested data in SQL is tricky, can I make it easier with a view or canned query?", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 606033104, "node_id": "MDU6SXNzdWU2MDYwMzMxMDQ=", "number": 12, "title": "If less than 500MB, show size in MB not GB", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-04-24T04:35:01Z", "updated_at": "2020-04-24T04:35:25Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Just saw this:\r\n```\r\nUploading 0.05 GB\r\n```", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 692202408, "node_id": "MDU6SXNzdWU2OTIyMDI0MDg=", "number": 12, "title": "Idea: maps and GeoJSON support", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-03T18:47:10Z", "updated_at": "2020-09-04T01:45:03Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "It would be cool if the `display_sql` could return a column populated with GeoJSON which would the automatically be displayed on a map in the results (or maybe default JS would look for a `class=\"geojson\"` element output by the `display` template) - ala https://github.com/simonw/datasette-leaflet-geojson\r\n\r\nThen I could render workout routes on a map, or Swarm checkin points.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 727848625, "node_id": "MDU6SXNzdWU3Mjc4NDg2MjU=", "number": 12, "title": "Some workout columns should be float, not text", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-10-23T02:47:02Z", "updated_at": "2022-06-23T04:35:02Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Columns `duration`, `totalDistance` and `totalEnergyBurned` should be converted to float.\r\n\r\nhttps://github.com/dogsheep/healthkit-to-sqlite/blob/71e36e1cf034b96de2a8e6652265d782d3fdf63b/healthkit_to_sqlite/utils.py#L50-L57", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 951817328, "node_id": "MDU6SXNzdWU5NTE4MTczMjg=", "number": 12, "title": "403 when getting token", "user": {"value": 285352, "label": "treyhunner"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-07-23T18:43:26Z", "updated_at": "2021-10-12T18:31:57Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I tried to use https://your-foursquare-oauth-token.glitch.me/ to get my Swarm auth token and got a 403 after I clicked the Allow button:\r\n\r\n![image](https://user-images.githubusercontent.com/285352/126826478-60e53614-263d-40bb-9f1d-c1a676644eb0.png)\r\n\r\nI'm not sure if this is the right repo to report this in", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1557599877, "node_id": "I_kwDODFE5qs5c1xaF", "number": 12, "title": "location history changes", "user": {"value": 14809320, "label": "gerardrbentley"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-01-26T03:57:25Z", "updated_at": "2023-01-26T03:57:25Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "not sure if each download is unique, but I had to change some things to work with the takeout zip I made 2023-01-25\r\n\r\nfilename changed from \"Location History.json\" to \"Records.json\"\r\n\r\n`\"timestampMs\"` is not present, `\"timestamp\"` is roughly iso timestamp\r\n\r\n```py\r\ndef get_timestamp_ms(raw_timestamp):\r\n try:\r\n return datetime.datetime.strptime(raw_timestamp, \"%Y-%m-%dT%H:%M:%SZ\").timestamp()\r\n except ValueError:\r\n return datetime.datetime.strptime(raw_timestamp, \"%Y-%m-%dT%H:%M:%S.%fZ\").timestamp()\r\n\r\ndef save_location_history(db, zf):\r\n location_history = json.load(\r\n zf.open(\"Takeout/Location History/Records.json\")\r\n )\r\n db[\"location_history\"].upsert_all(\r\n (\r\n {\r\n \"id\": id_for_location_history(row),\r\n \"latitude\": row[\"latitudeE7\"] / 1e7,\r\n \"longitude\": row[\"longitudeE7\"] / 1e7,\r\n \"accuracy\": row[\"accuracy\"],\r\n \"timestampMs\": get_timestamp_ms(row[\"timestamp\"]),\r\n \"when\": row[\"timestamp\"],\r\n }\r\n for row in location_history[\"locations\"]\r\n ),\r\n pk=\"id\",\r\n )\r\n\r\n\r\ndef id_for_location_history(row):\r\n # We want an ID that is unique but can be sorted by in\r\n # date order - so we use the isoformat date + the first\r\n # 6 characters of a hash of the JSON\r\n first_six = hashlib.sha1(\r\n json.dumps(row, separators=(\",\", \":\"), sort_keys=True).encode(\"utf8\")\r\n ).hexdigest()[:6]\r\n return \"{}-{}\".format(\r\n row['timestamp'],\r\n first_six,\r\n )\r\n```\r\n\r\nexample locations from mine\r\n\r\n```json\r\n{\r\n \"latitudeE7\": 427220206,\r\n \"longitudeE7\": -923423972,\r\n \"accuracy\": 10,\r\n \"deviceTag\": -1312429967,\r\n \"deviceDesignation\": \"PRIMARY\",\r\n \"timestamp\": \"2019-01-08T23:31:50.867Z\"\r\n }\r\n```\r\n\r\n```json\r\n{\r\n \"latitudeE7\": 427011317,\r\n \"longitudeE7\": -923448300,\r\n \"accuracy\": 5,\r\n \"deviceTag\": -1312429967,\r\n \"deviceDesignation\": \"PRIMARY\",\r\n \"timestamp\": \"2019-01-08T23:33:53Z\"\r\n }, \r\n```", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/12/reactions\", \"total_count\": 2, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 2}", "draft": null, "state_reason": null} {"id": 1650981564, "node_id": "I_kwDOJHON9s5iZ_q8", "number": 12, "title": "Error running pytest", "user": {"value": 14314871, "label": "amlestin"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-04-02T15:02:36Z", "updated_at": "2023-04-02T15:07:10Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "`______________________________________________________ ERROR collecting tests/test_apple_notes_to_sqlite.py _______________________________________________________\r\nImportError while importing test module '/Users/lol/development/apple-notes-to-sqlite/tests/test_apple_notes_to_sqlite.py'.\r\nHint: make sure your test modules/packages have valid Python names.\r\nTraceback:\r\n/opt/homebrew/Cellar/python@3.9/3.9.16/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module\r\n return _bootstrap._gcd_import(name[level:], package, level)\r\ntests/test_apple_notes_to_sqlite.py:2: in \r\n from apple_notes_to_sqlite.cli import cli, COUNT_SCRIPT, FOLDERS_SCRIPT\r\nE ModuleNotFoundError: No module named 'apple_notes_to_sqlite'`\r\n\r\nSolution:\r\nThis is likely a PYTHONPATH issue due to having pytest installed both globally and in the venv. We can guarantee the tests run by adding the current directory to sys.path automatically using\r\n\r\n`python -m pytest`\r\n\r\nThe alternative is to activate the venv, install pytest, deactivate, then activate the venv again (https://stackoverflow.com/questions/35045038/how-do-i-use-pytest-with-virtualenv)", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 607888367, "node_id": "MDU6SXNzdWU2MDc4ODgzNjc=", "number": 13, "title": "Also upload movie files", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-27T22:11:25Z", "updated_at": "2020-04-28T00:39:45Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "The `upload` command currently only handles static images:\r\n\r\nhttps://github.com/dogsheep/photos-to-sqlite/blob/d939455af00e07866686457ee2fcb9b2d1b7194e/photos_to_sqlite/utils.py#L26-L33\r\n\r\nNeed to cover movies taken by my phone and DSLR too.", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1373210675, "node_id": "I_kwDODD6af85R2Ygz", "number": 13, "title": "fails before generating views. ERR: table sqlite_master may not be modified", "user": {"value": 116795, "label": "pax"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-09-14T15:41:50Z", "updated_at": "2023-04-11T03:46:17Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "generates checkins.db but seems to fail before generating views \r\n\r\nnote: it worked on an Ubuntu WSL but fails on macOS 12.5.1\r\n\r\nlater edit: I suspect this is a problem with my local set-up, `dogsheep-beta index` also throws the same error\r\n\r\nfull error:\r\n\r\n Importing 2591 checkins [###################################-] 98% 00:00:00\r\n Traceback (most recent call last):\r\n File \"/Users/pax/devbox/envAll/bin/swarm-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/swarm_to_sqlite/cli.py\", line 77, in cli\r\n ensure_foreign_keys(db)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/swarm_to_sqlite/utils.py\", line 145, in ensure_foreign_keys\r\n db[fk.table].add_foreign_key(fk.column, fk.other_table, fk.other_column)\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/sqlite_utils/db.py\", line 2123, in add_foreign_key\r\n self.db.add_foreign_keys([(self.name, column, other_table, other_column)])\r\n File \"/Users/pax/devbox/envAll/lib/python3.8/site-packages/sqlite_utils/db.py\", line 1086, in add_foreign_keys\r\n cursor.execute(\r\n sqlite3.OperationalError: table sqlite_master may not be modified", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 530491074, "node_id": "MDU6SXNzdWU1MzA0OTEwNzQ=", "number": 14, "title": "Command for importing events", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-11-29T21:28:58Z", "updated_at": "2020-04-14T19:38:34Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Eg from https://api.github.com/users/simonw/events\r\n\r\nDocs here: https://developer.github.com/v3/activity/events/#list-events-performed-by-a-user", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 608512747, "node_id": "MDU6SXNzdWU2MDg1MTI3NDc=", "number": 14, "title": "Annotate photos using the Google Cloud Vision API", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-04-28T18:09:03Z", "updated_at": "2020-04-28T18:19:06Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "It can detect faces, run OCR, do image labeling (it knows what a lemur is!) and do object localization where it identifies objects and returns bounding polygons for them.", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14/reactions\", \"total_count\": 3, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 771608692, "node_id": "MDU6SXNzdWU3NzE2MDg2OTI=", "number": 14, "title": "UNIQUE constraint failed: workouts.id", "user": {"value": 1234956, "label": "n8henrie"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-12-20T15:11:20Z", "updated_at": "2023-07-10T14:46:52Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I'm getting an error on my initial attempt to import data:\r\n\r\n```console\r\n$ healthkit-to-sqlite 20201119\\ healthkit\\ export.zip healthkit.db\r\nImporting from HealthKit [###################################-] 98% 00:00:01\r\nTraceback (most recent call last):\r\n File \"venv/bin/healthkit-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"venv/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"venv/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"venv/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"venv/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"venv/lib/python3.9/site-packages/healthkit_to_sqlite/cli.py\", line 57, in cli\r\n convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)\r\n File \"venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py\", line 34, in convert_xml_to_sqlite\r\n workout_to_db(el, db, zipfile)\r\n File \"venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py\", line 57, in workout_to_db\r\n pk = db[\"workouts\"].insert(record, alter=True, hash_id=\"id\").last_pk\r\n File \"venv/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1660, in insert\r\n return self.insert_all(\r\n File \"venv/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1778, in insert_all\r\n self.insert_chunk(\r\n File \"venv/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1588, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"venv/lib/python3.9/site-packages/sqlite_utils/db.py\", line 213, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.IntegrityError: UNIQUE constraint failed: workouts.id\r\n```", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 986829194, "node_id": "MDU6SXNzdWU5ODY4MjkxOTQ=", "number": 14, "title": "xml.etree.ElementTree.Parse Error - mismatched tag", "user": {"value": 46968, "label": "step21"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-09-02T14:46:36Z", "updated_at": "2021-09-02T14:53:11Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "This is an error message I get upon parsing the enex file of my Inbox. Please find the full error message below. Any hints welcome.\r\n\r\n```\r\nImporting from ENEX [##################------------------] 50% 00:00:50\r\nTraceback (most recent call last):\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/bin/evernote-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/evernote_to_sqlite/cli.py\", line 30, in enex\r\n for tag, note in find_all_tags(fp, [\"note\"], progress_callback=bar.update):\r\n File \"/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/evernote_to_sqlite/utils.py\", line 17, in find_all_tags\r\n for event, el in parser.read_events():\r\n File \"/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py\", line 1329, in read_events\r\n raise event\r\n File \"/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py\", line 1301, in feed\r\n self._parser.feed(data)\r\nxml.etree.ElementTree.ParseError: mismatched tag: line 6837961, column 2\r\n```\r\n", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/14/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 694136490, "node_id": "MDU6SXNzdWU2OTQxMzY0OTA=", "number": 15, "title": "Add a bunch of config examples", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-05T17:58:43Z", "updated_at": "2020-09-18T23:17:39Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I can bring these over from my personal Dogsheep.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/15/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 505673645, "node_id": "MDU6SXNzdWU1MDU2NzM2NDU=", "number": 16, "title": "Do a better job with archived direct message threads", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-10-11T06:55:21Z", "updated_at": "2019-10-11T06:55:27Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "https://github.com/dogsheep/twitter-to-sqlite/blob/fb2698086d766e0333a55bb73435e7283feeb438/twitter_to_sqlite/archive.py#L98-L99", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/16/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 612287234, "node_id": "MDU6SXNzdWU2MTIyODcyMzQ=", "number": 16, "title": "Import machine-learning detected labels (dog, llama etc) from Apple Photos", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 13, "created_at": "2020-05-05T02:45:43Z", "updated_at": "2020-05-05T05:38:16Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Follow-on from #1. Apple Photos runs some very sophisticated machine learning on-device to figure out if photos are of dogs, llamas and so on. I really want to extract those labels out into my own database.", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16/reactions\", \"total_count\": 2, \"+1\": 0, \"-1\": 0, \"laugh\": 1, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 694493566, "node_id": "MDU6SXNzdWU2OTQ0OTM1NjY=", "number": 16, "title": "Timeline view", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-09-06T19:13:58Z", "updated_at": "2020-09-21T02:42:29Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Ability to browse (and facet) by date.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1943259395, "node_id": "I_kwDOEhK-wc5z08kD", "number": 16, "title": " time data '2014-11-21T11:44:12.000Z' does not match format '%Y%m%dT%H%M%SZ'", "user": {"value": 3746270, "label": "linonetwo"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-10-14T13:24:39Z", "updated_at": "2023-10-14T13:24:39Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "\r\n```\r\nevernote-to-sqlite enex evernote.db ./\u6211\u7684\u7b14\u8bb0.enex\r\nImporting from ENEX [#####-------------------------------] 14%\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/evernote-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n ^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 1157, in __call__\r\n return self.main(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 1078, in main\r\n rv = self.invoke(ctx)\r\n ^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 1688, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 1434, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/click/core.py\", line 783, in invoke\r\n return __callback(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/cli.py\", line 31, in enex\r\n save_note(db, note)\r\n File \"/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/utils.py\", line 46, in save_note\r\n \"created\": convert_datetime(created),\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/utils.py\", line 111, in convert_datetime\r\n return datetime.datetime.strptime(s, \"%Y%m%dT%H%M%SZ\").isoformat()\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/_strptime.py\", line 568, in _strptime_datetime\r\n tt, fraction, gmtoff_fraction = _strptime(data_string, format)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/_strptime.py\", line 349, in _strptime\r\n raise ValueError(\"time data %r does not match format %r\" %\r\nValueError: time data '2014-11-21T11:44:12.000Z' does not match format '%Y%m%dT%H%M%SZ'\r\n```\r\n\r\nenex is exported by evernote mac client ", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/16/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 836063389, "node_id": "MDU6SXNzdWU4MzYwNjMzODk=", "number": 17, "title": "Datetime columns are not properly formatted to be recognizes as datetime", "user": {"value": 1234956, "label": "n8henrie"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-03-19T14:33:04Z", "updated_at": "2021-03-19T14:33:04Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "\r\n\r\nCurrently, the datetimes are formatted in a way that is not recognized by datasette-vega for plotting with a `Date/time` type for the axis. \r\n\r\nFor example, if you have datasette running locally with `datasette-vega` installed and have a database that includes resting heart rate:\r\n\r\n```\r\nhttp://localhost:8001/healthkit/rRestingHeartRate#g.mark=line&g.x_column=startDate&g.x_type=temporal&g.y_column=value&g.y_type=quantitative\r\n```\r\n\r\nThe plot is blank unless you choose `Label` as the type for the date data.\r\n\r\nThe `startDate` (and `creationDate` and `endDate`) columns appear like: `2019-11-14 18:22:18 -0700`\r\n\r\nIf instead the format for this column is changed slightly: `2019-11-14T18:22:18-07:00` they are recognized as proper dates and the charting works as expected.\r\n\r\nI have a PR that addresses this issue, will submit shortly.", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/17/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 612860758, "node_id": "MDU6SXNzdWU2MTI4NjA3NTg=", "number": 18, "title": "Switch CI solution to GitHub Actions with a macOS runner", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-05T20:03:50Z", "updated_at": "2020-05-05T23:49:18Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Refs #17.", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 695553522, "node_id": "MDU6SXNzdWU2OTU1NTM1MjI=", "number": 18, "title": "Deleted records stay in the search index", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-08T05:14:23Z", "updated_at": "2020-09-08T05:15:51Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Here's why: https://github.com/dogsheep/dogsheep-beta/blob/24f7898d41a39218058f174c75ba62f7c0fcfff6/dogsheep_beta/utils.py#L44-L53\r\n\r\nThat should probably do `DELETE FROM index1.search_index WHERE [table] = ?` first.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 695556681, "node_id": "MDU6SXNzdWU2OTU1NTY2ODE=", "number": 19, "title": "Figure out incremental re-indexing", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-08T05:23:31Z", "updated_at": "2020-09-08T05:27:07Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "As tables get bigger reindexing everything on a schedule (essentially recreating the entire index from scratch) will start to become a performance bottleneck.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 697162939, "node_id": "MDU6SXNzdWU2OTcxNjI5Mzk=", "number": 20, "title": "Add more tags so people can find your project.", "user": {"value": 7902810, "label": "ran88dom99"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-09T21:14:09Z", "updated_at": "2020-09-09T21:14:09Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "quantified-self habit-tracking google-fit time-tracking wearables quantifiedself \r\nfor example", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/20/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 1, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 975166271, "node_id": "MDU6SXNzdWU5NzUxNjYyNzE=", "number": 20, "title": "Add index on workout_points.date", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-08-20T01:08:04Z", "updated_at": "2021-08-20T01:12:48Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Sorting that by date makes sense for seeing most recent points, and my DB has 2.5m points in so it's an expensive sort!", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 977128935, "node_id": "MDU6SXNzdWU5NzcxMjg5MzU=", "number": 21, "title": "Duplicate Column", "user": {"value": 32016596, "label": "FabianHertwig"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-08-23T15:00:44Z", "updated_at": "2021-08-23T17:00:59Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hey, thank you for this repo!\r\n\r\nWhen I try to convert my export, I get a multiple column error. Here is the stack trace:\r\n\r\n```sh\r\n(.venv) (base) computer:bodyweight_app user$ healthkit-to-sqlite ./data/Health_export.zip ./data/healthkit.db\r\nImporting from HealthKit [###############################-----] 87% 00:00:22\r\nTraceback (most recent call last):\r\n File \"/MyProject/.venv/bin/healthkit-to-sqlite\", line 10, in \r\n sys.exit(cli())\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/cli.py\", line 57, in cli\r\n convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py\", line 41, in convert_xml_to_sqlite\r\n write_records(records, db)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py\", line 146, in write_records\r\n batch_size=50,\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py\", line 2579, in insert_all\r\n extracts=extracts,\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1246, in create\r\n extracts=extracts,\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py\", line 767, in create_table\r\n self.execute(sql)\r\n File \"/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py\", line 421, in execute\r\n return self.conn.execute(sql)\r\nsqlite3.OperationalError: duplicate column name: metadata_Meal\r\n```", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/21/reactions\", \"total_count\": 5, \"+1\": 5, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 615626118, "node_id": "MDU6SXNzdWU2MTU2MjYxMTg=", "number": 22, "title": "Try out ExifReader", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-05-11T06:32:13Z", "updated_at": "2020-05-14T05:59:53Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "https://pypi.org/project/ExifReader/\r\n\r\nNew fork that should be able to handle EXIF in HEIC files.\r\n\r\nForked here: https://github.com/ianare/exif-py/issues/102#issuecomment-626376522\r\n\r\nRefs #3 ", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 599776345, "node_id": "MDU6SXNzdWU1OTk3NzYzNDU=", "number": 24, "title": "Feature idea: github-to-sqlite everything ...", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-04-14T18:34:00Z", "updated_at": "2020-04-14T18:34:00Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "At the moment if you want to pull all your repos, issues, issues comments etc you have to do it with a sequence of separate commands.\r\n\r\nConsider adding a `everything` or `all` command which fetches everything that the tool knows how to fetch, and is designed to be run on a cron in a way that fetches just new stuff each time.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/24/reactions\", \"total_count\": 7, \"+1\": 7, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 621323348, "node_id": "MDU6SXNzdWU2MjEzMjMzNDg=", "number": 24, "title": "Configurable URL for images", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-19T22:25:56Z", "updated_at": "2020-05-20T06:00:29Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "This is hard-coded at the moment, which is bad:\r\nhttps://github.com/dogsheep/photos-to-sqlite/blob/d5d69b9019703c47bc251444838578dd752801e2/photos_to_sqlite/cli.py#L269-L272", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/24/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1515883470, "node_id": "I_kwDOC8tyDs5aWovO", "number": 24, "title": "DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 ", "user": {"value": 6231413, "label": "mmngreco"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-01-01T23:00:38Z", "updated_at": "2023-03-30T10:17:31Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi @simonw \r\n\r\nI hope you find this issue ok, the idea is provide some documentation to other users like me about how to solve this problem and save some time.\r\n\r\nFollowing the instructions from the `README.md` I've faced this error:\r\n\r\n```bash\r\n(venv) mgreco@pop-os apple-health master* (23:44|0s)\r\n$ healthkit-to-sqlite apple_health_export/export.xml healthkit.db --xml\r\nImporting from HealthKit [------------------------------------] 0%\r\nTraceback (most recent call last):\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/bin/healthkit-to-sqlite\", line 33, in \r\n sys.exit(load_entry_point('healthkit-to-sqlite', 'console_scripts', 'healthkit-to-sqlite')())\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/cli.py\", line 57, in cli\r\n convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)\r\n File \"/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/utils.py\", line 25, in convert_xml_to_sqlite\r\n for tag, el in find_all_tags(\r\n File \"/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/utils.py\", line 12, in find_all_tags\r\n for event, el in parser.read_events():\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/xml/etree/ElementTree.py\", line 1324, in read_events\r\n raise event\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/xml/etree/ElementTree.py\", line 1296, in feed\r\n self._parser.feed(data)\r\nxml.etree.ElementTree.ParseError: syntax error: line 156, column 0\r\n```\r\n\r\nSo, after debugging and searching on internet I found this useful link: https://discussions.apple.com/thread/254202523 (etresoft, the real hero). Which basically says that the xml given by the health app (healthkit version 12) has some bugs but fortunately, they can be solved with a couple of commads:\r\n\r\n1. Uncompress the zip and move the new folder where `export.xml` is.\r\n1. Create a `patch.txt` with the following content\r\n\r\n ```diff\r\n --- export.xml\t2022-09-18 15:17:09.000000000 -0400\r\n +++ export-fixed.xml\t2022-09-18 16:37:08.000000000 -0400\r\n @@ -15,6 +15,7 @@\r\n HKCharacteristicTypeIdentifierBiologicalSex CDATA #REQUIRED\r\n HKCharacteristicTypeIdentifierBloodType CDATA #REQUIRED\r\n HKCharacteristicTypeIdentifierFitzpatrickSkinType CDATA #REQUIRED\r\n + HKCharacteristicTypeIdentifierCardioFitnessMedicationsUse CDATA #IMPLIED\r\n >\r\n \r\n \r\n -\r\n +\r\n \r\n -\r\n +\r\n \r\n \r\n \r\n \r\n \r\n - device CDATA #IMPLIED\r\n -\r\n -\r\n ->\r\n ]>\r\n \r\n \r\n ```\r\n1. Apply the path with the command: `patch < patch.txt`\r\n1. Fix endDates with the command `sed 's/startDate/endDate/2' export.xml > export-fixed.xml`\r\n1. Try again `healthkit-to-sqlite export-fixed.xml healthkit.db --xml`", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 455486286, "node_id": "MDU6SXNzdWU0NTU0ODYyODY=", "number": 26, "title": "Mechanism for turning nested JSON into foreign keys / many-to-many", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 14, "created_at": "2019-06-13T00:52:06Z", "updated_at": "2022-06-29T23:35:29Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The GitHub JSON APIs have a really interesting convention with respect to related objects.\r\n\r\nConsider https://api.github.com/repos/simonw/sqlite-utils/issues - here's a truncated subset:\r\n```json\r\n {\r\n \"id\": 449818897,\r\n \"node_id\": \"MDU6SXNzdWU0NDk4MTg4OTc=\",\r\n \"number\": 24,\r\n \"title\": \"Additional Column Constraints?\",\r\n \"user\": {\r\n \"login\": \"IgnoredAmbience\",\r\n \"id\": 98555,\r\n \"node_id\": \"MDQ6VXNlcjk4NTU1\",\r\n \"avatar_url\": \"https://avatars0.githubusercontent.com/u/98555?v=4\",\r\n \"gravatar_id\": \"\"\r\n },\r\n \"labels\": [\r\n {\r\n \"id\": 993377884,\r\n \"node_id\": \"MDU6TGFiZWw5OTMzNzc4ODQ=\",\r\n \"url\": \"https://api.github.com/repos/simonw/sqlite-utils/labels/enhancement\",\r\n \"name\": \"enhancement\",\r\n \"color\": \"a2eeef\",\r\n \"default\": true\r\n }\r\n ],\r\n \"state\": \"open\"\r\n }\r\n```\r\nThe `user` column lists a complete user. The `labels` column has a list of labels.\r\n\r\nSince both user and label have populated `id` field this is actually enough information for us to create records for them AND set up the corresponding foreign key (for user) and m2m relationships (for labels).\r\n\r\nIt would be really neat if `sqlite-utils` had some kind of mechanism for correctly processing these kind of patterns.\r\n\r\nThanks to `jq` there's not much need for extra customization of the shape here - if we support a narrowly defined structure users can use `jq` to reshape arbitrary JSON to match.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/26/reactions\", \"total_count\": 4, \"+1\": 4, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 705215230, "node_id": "MDU6SXNzdWU3MDUyMTUyMzA=", "number": 26, "title": "Pagination", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-09-21T00:14:37Z", "updated_at": "2020-09-21T02:55:54Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Useful for #16 (timeline view) since you can now filter to just the items on a specific day - but if there are more than 50 items you can't see them all.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 621486115, "node_id": "MDU6SXNzdWU2MjE0ODYxMTU=", "number": 27, "title": "photos_with_apple_metadata view should include labels", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-05-20T06:06:17Z", "updated_at": "2020-05-20T06:06:17Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "https://dogsheep-photos.dogsheep.net/public/photos_with_apple_metadata?place_city=New+Orleans&_facet=place_city&_facet_array=albums&_facet_array=persons\r\n\r\nHere's one way to add that:\r\n```sql\r\n select\r\n rowid,\r\n photo,\r\n (\r\n select\r\n json_group_array(\r\n json_object(\r\n 'label',\r\n normalized_string,\r\n 'href',\r\n '/photos/labelled?_hide_sql=1&label=' || normalized_string\r\n )\r\n )\r\n from\r\n labels\r\n where\r\n labels.uuid = photos_with_apple_metadata.uuid\r\n ) as labels,\r\n date,\r\n```", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/27/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 709789634, "node_id": "MDU6SXNzdWU3MDk3ODk2MzQ=", "number": 27, "title": "Sort order is not persisted by facet filter links", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-27T18:22:07Z", "updated_at": "2020-09-27T18:22:07Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "A link to `/-/beta?category=1×tamp__date=2018-08-01&q=swedish` should be to `/-/beta?category=1×tamp__date=2018-08-01&q=swedish&sort=newest`", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/27/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 624490929, "node_id": "MDU6SXNzdWU2MjQ0OTA5Mjk=", "number": 28, "title": "Invalid SQL no such table: main.uploads", "user": {"value": 41439, "label": "dmd"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-25T21:25:39Z", "updated_at": "2020-12-24T22:26:22Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "http://127.0.0.1:8001/photos/photos_with_apple_metadata gives \"Invalid SQL no such table: main.uploads\"", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/28/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 724759588, "node_id": "MDU6SXNzdWU3MjQ3NTk1ODg=", "number": 29, "title": "Add search highlighting snippets", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-10-19T16:00:48Z", "updated_at": "2021-08-26T20:23:11Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Like on https://til.simonwillison.net/til/search?q=Snippet", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 803333769, "node_id": "MDU6SXNzdWU4MDMzMzM3Njk=", "number": 32, "title": "KeyError: 'Contents' on running upload", "user": {"value": 11855322, "label": "robmarkcole"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-02-08T08:36:37Z", "updated_at": "2021-07-22T06:40:25Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Following the readme, on big sur, and having entered my auth creds via `dogsheep-photos s3-auth`:\r\n\r\n```\r\n(venv) (base) Robins-MacBook:datasette robin$ dogsheep-photos upload photos.db ~/Pictures/Photos\\ /Users/robin/Pictures/Library.photoslibrary --dry-run\r\n\r\nFetching existing keys from S3...\r\nTraceback (most recent call last):\r\n File \"/Users/robin/datasette/venv/bin/dogsheep-photos\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/cli.py\", line 96, in upload\r\n key.split(\".\")[0] for key in get_all_keys(client, creds[\"photos_s3_bucket\"])\r\n File \"/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/utils.py\", line 46, in get_all_keys\r\n for row in page[\"Contents\"]:\r\nKeyError: 'Contents'\r\n```\r\n\r\nPossibly since the bucket is in `EU (London) eu-west-2` and this into is not requested?", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 836923194, "node_id": "MDU6SXNzdWU4MzY5MjMxOTQ=", "number": 32, "title": "JSON API for search results", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-03-20T22:21:36Z", "updated_at": "2021-03-20T22:21:36Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Refs https://github.com/simonw/datasette/issues/878", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/32/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 268110769, "node_id": "MDU6SXNzdWUyNjgxMTA3Njk=", "number": 33, "title": "Use locust for benchmarking and load tests", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2017-10-24T17:00:09Z", "updated_at": "2017-12-10T03:12:16Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "https://github.com/locustio/locust\r\n\r\nNeeded for #32 ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/33/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 803338729, "node_id": "MDU6SXNzdWU4MDMzMzg3Mjk=", "number": 33, "title": "photo-to-sqlite: command not found", "user": {"value": 11855322, "label": "robmarkcole"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-02-08T08:42:57Z", "updated_at": "2021-02-12T15:00:44Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Having installed in a venv I get:\r\n```\r\n(venv) (base) Robins-MacBook:datasette robin$ photo-to-sqlite apple-photos photos.db\r\n\r\n-bash: photo-to-sqlite: command not found\r\n```", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 830283447, "node_id": "MDU6SXNzdWU4MzAyODM0NDc=", "number": 34, "title": "bucket name", "user": {"value": 6213, "label": "dsisnero"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-03-12T16:40:57Z", "updated_at": "2021-03-12T16:40:57Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I followed the instructions to setup credentials but I am getting a invalid bucket name. Can you put a sample auth.json file in the base that shows the correct format for this? Thanks", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/34/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 983221851, "node_id": "MDU6SXNzdWU5ODMyMjE4NTE=", "number": 34, "title": "Data folder as index command parameter", "user": {"value": 1223625, "label": "humrochagf"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-08-30T21:29:33Z", "updated_at": "2021-08-30T21:29:33Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi,\r\n\r\nFirst of all, thank you for this wonderful project :smile:\r\n\r\nI started to use dogsheep to make my personal data searchable, and by using the project I noticed an issue with the index command.\r\n\r\nIt always expects you are running it from the root folder from where the data is located, so I got some errors while trying to make it work on my setup.\r\n\r\nI separate all databases inside a `data` folder (I published my setup to be easier to follow: https://github.com/humrochagf/my-dogsheep)\r\n\r\nBefore, I configured `dogsheep.yml` to add the data folder to its path like this:\r\n\r\n```yml\r\ndata/twitter.db:\r\n tweets:\r\n sql: |-\r\n...\r\n```\r\n\r\nAnd running the index command like this:\r\n\r\n```\r\ndogsheep-beta index data/dogsheep.db dogsheep.yml\r\n```\r\n\r\nIt worked to the normal search feature with no problem this way, but when I started adding `display_sql` rules the app started to crash, because at datasette `get_database` it was looking for `data/twitter` and it only had a db called `twitter` there.\r\n\r\nSo my workaround to that was to cd into the data folder and run the indexer. You can check the way I'm doing it at this line of the makefile: https://github.com/humrochagf/my-dogsheep/blob/main/makefile#L3\r\n\r\nIt works but it would be nice to have an option to pass the path where the data is located to the index function.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/34/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 842695374, "node_id": "MDU6SXNzdWU4NDI2OTUzNzQ=", "number": 35, "title": "Support to annotate photos on other than macOS OSes", "user": {"value": 1151557, "label": "ligurio"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-03-28T09:01:25Z", "updated_at": "2021-04-05T07:37:57Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "dogsheep-photos allows to annotate photos using Apple Photo's db. It would be nice to have such ability on other OSes too. For example using trained local model or using Google Vision API (see #14).", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/35/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1751214236, "node_id": "I_kwDOC8SPRc5oYWic", "number": 36, "title": "Getting sqlite_master may not be modified when creating dogsheep index", "user": {"value": 8711912, "label": "khushmeeet"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-06-11T03:21:53Z", "updated_at": "2023-06-11T03:21:53Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "When creating a `dogsheep` index from `config.yml` file on pocket.db (created using pocket-to-sqlite), I am getting this error\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/bin/dogsheep-beta\", line 8, in \r\n sys.exit(cli())\r\n ^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n ^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 1657, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/cli.py\", line 36, in index\r\n run_indexer(\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py\", line 32, in run_indexer\r\n ensure_table_and_indexes(db, tokenize)\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py\", line 91, in ensure_table_and_indexes\r\n table.add_foreign_key(*fk)\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py\", line 2155, in add_foreign_key\r\n self.db.add_foreign_keys([(self.name, column, other_table, other_column)])\r\n File \"/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py\", line 1116, in add_foreign_keys\r\n cursor.execute(\r\nsqlite3.OperationalError: table sqlite_master may not be modified\r\n```\r\n\r\nCommand I ran to get this error\r\n```\r\ndogsheep-beta index pocket.db config.yml\r\n```\r\n\r\nDogsheep version\r\n```\r\ndogsheep-beta, version 0.10.2\r\n```\r\n\r\nPython version \r\n```\r\nPython 3.11.2\r\n```", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/36/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1888477283, "node_id": "I_kwDOC8SPRc5wj-Bj", "number": 38, "title": "Run `rebuild_fts` after building the index", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-09-08T23:17:45Z", "updated_at": "2023-09-08T23:17:45Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "In:\r\n- https://github.com/simonw/datasette.io/issues/152#issuecomment-1712323347\r\n\r\nThis turned out to be the fix:\r\n\r\n```bash\r\ndogsheep-beta index dogsheep-index.db templates/dogsheep-beta.yml\r\nsqlite-utils rebuild-fts dogsheep-index.db\r\n```", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/38/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 602619330, "node_id": "MDU6SXNzdWU2MDI2MTkzMzA=", "number": 45, "title": "Use raise_for_status() everywhere", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-04-19T04:38:28Z", "updated_at": "2020-04-19T04:39:22Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I keep seeing errors which I think are caused by authentication or rate limit problems but which appear to be unexpected JSON responses - presumably because they are actually an error message.\r\n\r\nRecent example: https://github.com/simonw/jsk-fellows-on-twitter/runs/598892575\r\n\r\nUsing `response.raise_for_status()` everywhere will make these errors less confusing.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 664485022, "node_id": "MDU6SXNzdWU2NjQ0ODUwMjI=", "number": 46, "title": "Feature: pull request reviews and comments", "user": {"value": 1326704, "label": "bhrutledge"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-07-23T13:43:45Z", "updated_at": "2022-12-20T14:40:15Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi there! I saw your [presentation at Boston Python](https://www.meetup.com/bostonpython/events/271887195). I'm already a light user of Datasette (thank you!), but wasn't aware of this project.\r\n\r\nI've been working on a \"pull request dashboard\" to get a comprehensive view of the state of open PR's, esp. related to reviews (i.e., pending, approved, changes requested). Currently it's a CLI command, but I thought a Datasette UI might be fun.\r\n\r\nI see that PR's are available from the `issues` command, but I don't see reviews anywhere. From the [API docs](https://docs.github.com/en/rest/reference/pulls#reviews), it looks like there are separate endpoints for those (as well as pull requests in general). What do you think about adding that? Would you accept a PR? Any sense of the level of effort?", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 639542974, "node_id": "MDU6SXNzdWU2Mzk1NDI5NzQ=", "number": 47, "title": "Fall back to FTS4 if FTS5 is not available", "user": {"value": 73579, "label": "hpk42"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-06-16T10:11:23Z", "updated_at": "2020-06-17T20:13:48Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "got this with version 0.21.1 from pypi. twitter-to-sqlite auth worked but then \"twitter-to-sqlite user-timeline USER.db\" produced a tracekback ending in \"no such module: FTS5\". ", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 472115381, "node_id": "MDU6SXNzdWU0NzIxMTUzODE=", "number": 49, "title": "extracts= should support multiple-column extracts", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2019-07-24T07:06:41Z", "updated_at": "2020-10-16T19:18:19Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Lookup tables can be constructed on compound columns, but the `extracts=` option doesn't currently support that.\r\n\r\nRight now extracts can be defined in two ways:\r\n```python\r\n# Extract these columns into tables with the same name:\r\ndogs = db.table(\"dogs\", extracts=[\"breed\", \"most_recent_trophy\"])\r\n\r\n# Same as above but with custom table names:\r\ndogs = db.table(\"dogs\", extracts={\"breed\": \"Breeds\", \"most_recent_trophy\": \"Trophies\"})\r\n```\r\nNeed some kind of syntax for much more complicated extractions, like when two columns (say \"source\" and \"source_version\") are extracted into a single table.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/49/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 703216044, "node_id": "MDU6SXNzdWU3MDMyMTYwNDQ=", "number": 49, "title": "Feature: gists and starred gists", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-17T02:30:52Z", "updated_at": "2020-09-17T02:30:52Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "https://developer.github.com/v3/gists/#list-starred-gists", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/49/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 698791218, "node_id": "MDU6SXNzdWU2OTg3OTEyMTg=", "number": 50, "title": "favorites --stop_after=N stops after min(N, 200)", "user": {"value": 370930, "label": "mikepqr"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-11T03:38:14Z", "updated_at": "2020-09-13T05:11:14Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "For any number greater than 200, `favorites --stop_after` stops after getting 200 tweets, e.g.\r\n```\r\n$ twitter-to-sqlite favorites tweets.db --stop_after=300\r\nImporting favorites [####################################] 199\r\n$\r\n```\r\nI don't _think_ this is a limitation of the API (if you omit `--stop_after` you get some very large number, possibly all of them), so I _think_ this is a bug.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 703218756, "node_id": "MDU6SXNzdWU3MDMyMTg3NTY=", "number": 50, "title": "Commands for making authenticated API calls", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-09-17T02:39:07Z", "updated_at": "2020-10-19T05:01:29Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Similar to `twitter-to-sqlite fetch`, see https://github.com/dogsheep/twitter-to-sqlite/issues/51", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 703218448, "node_id": "MDU6SXNzdWU3MDMyMTg0NDg=", "number": 51, "title": "Documentation for twitter-to-sqlite fetch", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-17T02:38:10Z", "updated_at": "2020-09-17T02:38:10Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "It's mentioned in passing in the README but it deserves its own section:\r\n```\r\n$ twitter-to-sqlite fetch \\\r\n \"https://api.twitter.com/1.1/account/verify_credentials.json\" \\\r\n | grep '\"id\"' | head -n 1\r\n```", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/51/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 703246031, "node_id": "MDU6SXNzdWU3MDMyNDYwMzE=", "number": 51, "title": "github-to-sqlite should handle rate limits better", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-17T04:01:50Z", "updated_at": "2022-10-14T16:34:07Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "From #50 - right now it will crash with an error of it hits the rate limit. Since the rate limit information (including reset time) is available in the headers it could automatically sleep and try again instead.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 753000405, "node_id": "MDU6SXNzdWU3NTMwMDA0MDU=", "number": 53, "title": "Command for fetching file contents", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-11-29T20:31:04Z", "updated_at": "2020-11-30T00:36:09Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Something like this:\r\n\r\n github-to-sqlite files github.db simonw/datasette\r\n\r\nThis would fetch all files from the `main` branch into a `files` table.\r\n\r\nAdditional options could handle things like pulling files from a branch or tag, or just pulling files that match a specific glob or that exist in a specific directory.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/53/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 779088071, "node_id": "MDU6SXNzdWU3NzkwODgwNzE=", "number": 54, "title": "Archive import appears to be broken on recent exports", "user": {"value": 21148, "label": "jacobian"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-01-05T14:18:01Z", "updated_at": "2023-01-04T11:06:55Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I requested a Twitter export yesterday, and unfortunately they seem to have changed it such that `twitter-to-sqlite import` can't handle it anymore \ud83d\ude22 \r\n\r\nSo far I've ran into two issues. The first was easy to work around, but the second will take more investigation. If I can find the time I'll keep working on it and update this issue accordingly.\r\n\r\nThe issues (so far):\r\n\r\n### 1. Data seems to have moved to a `data/` subdirectory\r\n\r\nRunning `twitter-to-sqlite import` on the raw zip file reports a bunch of \"not yet implemented\" errors, and then exits without actually importing anything:\r\n\r\n```\r\n\u276f twitter-to-sqlite import tarchive.db twitter.zip\r\n...\r\ndata/manifest: not yet implemented\r\ndata/account-creation-ip: not yet implemented\r\ndata/account-suspension: not yet implemented\r\n... (dozens of more lines like this, including critical stuff like data/tweets) ...\r\n```\r\n\r\n(`tarchive.db` now exists, but is empty)\r\n\r\nWorkaround: unpack the zip file, and run `twitter-to-sqlite import tarchive.db path/to/archive/data`\r\n\r\nThat gets further, but:\r\n\r\n### 2. Some schema(s?) have changed\r\n\r\nAt least, the `blocks` schema seems different now:\r\n\r\n```\r\n\u276f twitter-to-sqlite import tarchive.db archive/data\r\ndirect-messages-group: not yet implemented\r\nbranch-links: not yet implemented\r\nperiscope-expired-broadcasts: not yet implemented\r\ndirect-messages: not yet implemented\r\nmute: not yet implemented\r\nTraceback (most recent call last):\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/bin/twitter-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/cli.py\", line 772, in import_\r\n archive.import_from_file(db, filepath.name, open(filepath, \"rb\").read())\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py\", line 215, in import_from_file\r\n to_insert = transformer(data)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py\", line 115, in lists_member\r\n return {\"lists-member\": _list_from_common(data)}\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py\", line 200, in _list_from_common\r\n for url in block[\"userListInfo\"][\"urls\"]:\r\nKeyError: 'urls'\r\n```\r\n\r\nThat's as far as I got before I needed to work on something else. I'll report back if I get further!", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 797097140, "node_id": "MDU6SXNzdWU3OTcwOTcxNDA=", "number": 60, "title": "Use Data from SQLite in other commands", "user": {"value": 22578954, "label": "daniel-butler"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-01-29T18:35:52Z", "updated_at": "2021-02-12T18:29:43Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "As a total beginner here how could you access data from the sqlite table to run other commands.\r\n\r\nWhat I am thinking is I want to get all the repos in an organization then using the repo list pull all the commit messages for each repo. \r\n\r\nI love this project by the way!", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1063982712, "node_id": "I_kwDODEm0Qs4_axZ4", "number": 60, "title": "Execution on Windows", "user": {"value": 1733616, "label": "bernard01"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-11-26T00:24:34Z", "updated_at": "2022-10-14T16:58:27Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "My installation on Windows using pip has been successful. I have Python 3.6.\r\n\r\nHow do I run twitter-to-sqlite? I cannot even figure out how \"auth\" is a command. I have python on my path:\r\nC:\\prog\\python\\Python36;C:\\prog\\python\\Python36\\Scripts\r\n\r\nWhere should the commands be executed, and where are the files created?\r\n\r\nCould some basics please be added to the documentation to get beginners started?", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/60/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1077560091, "node_id": "I_kwDODEm0Qs5AOkMb", "number": 61, "title": "Data Pull fails for \"Essential\" level access to the Twitter API (for Documentation)", "user": {"value": 57161638, "label": "jmnickerson05"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-11T14:59:41Z", "updated_at": "2022-10-31T14:47:58Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Per Twitter documentation:\r\nhttps://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api#v2-access-leve\r\n\r\nThis isn't any fault of twitter-to-sqlite of course, but it should probably be documented as a side-note.\r\n\r\n![image](https://user-images.githubusercontent.com/57161638/145681272-8c85b3b9-be95-44ff-9760-1bafa4917ce2.png)\r\n\r\nAnd this is how I'm surfacing the message from utils.py:\r\n![image](https://user-images.githubusercontent.com/57161638/145681005-2776c0ad-9822-4461-b43a-450ab2e828eb.png)\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/61/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 797784080, "node_id": "MDU6SXNzdWU3OTc3ODQwODA=", "number": 62, "title": "Stargazers and workflows commands always require an auth file when using GITHUB_TOKEN ", "user": {"value": 631242, "label": "frosencrantz"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-01-31T18:56:05Z", "updated_at": "2021-01-31T18:56:05Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Requested fix in https://github.com/dogsheep/github-to-sqlite/pull/59\r\n\r\nThe stargazers and workflows commands always require an auth file, even when using a `GITHUB_TOKEN`. Other commands don't require the auth file.\r\n\r\n", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/62/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 897212458, "node_id": "MDU6SXNzdWU4OTcyMTI0NTg=", "number": 63, "title": "Ability to fetch commits from branches other than the default", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-05-20T17:58:08Z", "updated_at": "2021-05-20T17:58:08Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "This tool is currently almost entirely ignorant of the concept of branches. One example: you can't retrieve commits from any branch other than the default (usually main).", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/63/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1091850530, "node_id": "I_kwDODEm0Qs5BFFEi", "number": 63, "title": "Import archive error 'withheld_in_countries'", "user": {"value": 521097, "label": "pauloxnet"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-01T16:58:59Z", "updated_at": "2022-01-01T16:58:59Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Importing the twitter archive I received this error:\r\n```bash\r\n$ twitter-to-sqlite import archive.db twitter-2021-12-31-.zip \r\nbirdwatch-note-rating: not yet implemented\r\nbirdwatch-note: not yet implemented\r\nbranch-links: not yet implemented\r\ncommunity-tweet: not yet implemented\r\ncontact: not yet implemented\r\ndevice-token: not yet implemented\r\ndirect-message-mute: not yet implemented\r\nmute: not yet implemented\r\nperiscope-account-information: not yet implemented\r\nperiscope-ban-information: not yet implemented\r\nperiscope-broadcast-metadata: not yet implemented\r\nperiscope-comments-made-by-user: not yet implemented\r\nperiscope-expired-broadcasts: not yet implemented\r\nperiscope-followers: not yet implemented\r\nperiscope-profile-description: not yet implemented\r\nprofessional-data: not yet implemented\r\nprotected-history: not yet implemented\r\nreply-prompt: not yet implemented\r\nscreen-name-change: not yet implemented\r\nsmartblock: not yet implemented\r\nspaces-metadata: not yet implemented\r\nsso: not yet implemented\r\nTraceback (most recent call last):\r\n File \"/home/paulox/.virtualenvs/dogsheep/bin/twitter-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/cli.py\", line 759, in import_\r\n archive.import_from_file(db, filename, content)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/archive.py\", line 246, in import_from_file\r\n db[table_name].insert_all(rows, pk=pk, replace=True)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2625, in insert_all\r\n self.insert_chunk(\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2406, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 422, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.OperationalError: table archive_tweet has no column named withheld_in_countries\r\n```\r\n\r\nI found only a single tweet with the key `withheld_in_countries` in `tweet.js` that seems the problems:\r\n```JSON\r\n[\r\n{\r\n \"tweet\" : {\r\n \"retweeted\" : false,\r\n \"source\" : \"Twitter for Android\",\r\n \"entities\" : {\r\n \"hashtags\" : [\r\n {\r\n \"text\" : \"NowOnAndroid\",\r\n \"indices\" : [\r\n \"64\",\r\n \"77\"\r\n ]\r\n }\r\n ],\r\n \"symbols\" : [ ],\r\n \"user_mentions\" : [\r\n {\r\n \"name\" : \"Periscope\",\r\n \"screen_name\" : \"PeriscopeCo\",\r\n \"indices\" : [\r\n \"3\",\r\n \"15\"\r\n ],\r\n \"id_str\" : \"1111111111\",\r\n \"id\" : \"222222222\"\r\n }\r\n ],\r\n \"urls\" : [\r\n {\r\n \"url\" : \"https://t.co/xxxxxxxxx\",\r\n \"expanded_url\" : \"https://vine.co/v/xxxxxxxxx\",\r\n \"display_url\" : \"vine.co/v/xxxxxxxxxx\",\r\n \"indices\" : [\r\n \"78\",\r\n \"101\"\r\n ]\r\n }\r\n ]\r\n },\r\n \"display_text_range\" : [\r\n \"0\",\r\n \"101\"\r\n ],\r\n \"favorite_count\" : \"0\",\r\n \"id_str\" : \"1111111111111111111111\",\r\n \"truncated\" : false,\r\n \"retweet_count\" : \"0\",\r\n \"withheld_in_countries\" : [\r\n \"TR\"\r\n ],\r\n \"id\" : \"000000000000000000\",\r\n \"possibly_sensitive\" : false,\r\n \"created_at\" : \"Fri Aug 14 06:04:03 +0000 2015\",\r\n \"favorited\" : false,\r\n \"full_text\" : \"RT @periscopeco: Travel the world. LIVE. The Global Map is here #NowOnAndroid https://t.co/NZXdsPWROk\",\r\n \"lang\" : \"en\"\r\n }\r\n }\r\n ]\r\n```\r\n\r\nI solved the error removing the key from the `tweet.js` but I'm reporting this error to improve the project.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/63/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 920636216, "node_id": "MDU6SXNzdWU5MjA2MzYyMTY=", "number": 64, "title": "feature: support \"events\"", "user": {"value": 231498, "label": "khimaros"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-06-14T17:42:49Z", "updated_at": "2021-06-15T00:48:37Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "the GitHub API provides the ability to fetch all events for a given user, organization, or repository: https://docs.github.com/en/rest/reference/activity#list-events-for-the-authenticated-user\r\n\r\nthis would allow users to export all of the issue comments, new issues, etc. that they created. something which is currently missing from the GitHub takeout exports.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1097332098, "node_id": "I_kwDODEm0Qs5BZ_WC", "number": 64, "title": "Include all entities for tweets", "user": {"value": 111631, "label": "max"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-09T23:35:28Z", "updated_at": "2022-01-09T23:35:28Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Per our conversation [on Twitter](https://twitter.com/mschoening/status/1480312477246054401):\r\n\r\nIt would be neat if all entities (including URLs) were captured. This way you can ensure, that URLs are parsed out exactly the same way Twitter parses URLs \u2013 we all know parsing URLs with a regex ain't fun.\r\n\r\nRight now, I believe the tool filters out all entities that are not of type `media`.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/64/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1071071397, "node_id": "I_kwDODFdgUs4_10Cl", "number": 69, "title": "View that combines issues and issue comments", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-04T00:34:33Z", "updated_at": "2021-12-04T00:34:52Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I want to see a reverse chronologically ordered interface onto both issues and comments - essentially a unified log of comments and issues opened across one or multiple projects.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 539204432, "node_id": "MDU6SXNzdWU1MzkyMDQ0MzI=", "number": 70, "title": "Implement ON DELETE and ON UPDATE actions for foreign keys", "user": {"value": 26292069, "label": "LucasElArruda"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-12-17T17:19:10Z", "updated_at": "2020-02-27T04:18:53Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi! I did not find any mention on the library about ON DELETE and ON UPDATE actions for foreign keys. Are those expected to be implemented? If not, it would be a nice thing to include!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/70/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1211283427, "node_id": "I_kwDODFdgUs5IMrfj", "number": 72, "title": "feature: display progress bar when downloading multi-page responses", "user": {"value": 9020979, "label": "hydrosquall"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-04-21T16:37:12Z", "updated_at": "2022-04-21T17:29:31Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "## Motivation\r\n\r\nFor a long running command (longer than 1 minute) for a big table (like pull requests or commits), it can be tricky to know if the script is still running, or if a rate limit/error was encountered\r\n\r\nWe know how many pages there are, so it may be possible to indicate how many remain.\r\n\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/a6e237f75a4b86963d91dcb5c9582e3a1b3349d6/github_to_sqlite/utils.py#L367\r\n\r\n## Resources\r\n\r\n- Using the existing Click API: \r\n - https://click.palletsprojects.com/en/5.x/utils/#showing-progress-bars\r\n- Loading spinner: https://github.com/pavdmyt/yaspin\r\n- Progress bar: https://github.com/tqdm/tqdm", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/72/reactions\", \"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}