{"id": 267513424, "node_id": "MDU6SXNzdWUyNjc1MTM0MjQ=", "number": 1, "title": "Addressable pages for every row in a table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 6, "created_at": "2017-10-23T00:44:16Z", "updated_at": "2017-10-24T14:11:04Z", "closed_at": "2017-10-24T14:11:03Z", "author_association": "OWNER", "pull_request": null, "body": " /database-name-7sha256/table-name/compound-pk\r\n /database-name-7sha256/table-name/compound-pk.json\r\n\r\nTricky part will be figuring out what the private key is - especially since it could be a compound primary key and it might involve different data types.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 347058326, "node_id": "MDExOlB1bGxSZXF1ZXN0MjA1NzcwOTk2", "number": 1, "title": "Make .indexes compatible with older SQLite versions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2018-08-02T15:17:05Z", "updated_at": "2018-08-02T15:17:30Z", "closed_at": "2018-08-02T15:17:30Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/1", "body": "Older SQLite versions return a different set of columns from the PRAGMA we are using.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 470637068, "node_id": "MDU6SXNzdWU0NzA2MzcwNjg=", "number": 1, "title": "Use XML Analyser to figure out the structure of the export XML", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-07-20T05:19:02Z", "updated_at": "2019-07-20T05:20:09Z", "closed_at": "2019-07-20T05:20:09Z", "author_association": "MEMBER", "pull_request": null, "body": "https://github.com/simonw/xml_analyser", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 487598042, "node_id": "MDU6SXNzdWU0ODc1OTgwNDI=", "number": 1, "title": "Implement code to pull checkins from the Foursquare API", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-08-30T17:40:02Z", "updated_at": "2019-08-30T18:23:24Z", "closed_at": "2019-08-30T18:23:24Z", "author_association": "MEMBER", "pull_request": null, "body": "The tool currently only works with a pre-prepared JSON file of checkins.\r\n\r\nWhen called without options, it should prompt the user to paste in a Foursquare OAuth token.\r\n\r\nThe `--token=` option should work too, and should be backed up by an optional environment variable.", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 488833136, "node_id": "MDU6SXNzdWU0ODg4MzMxMzY=", "number": 1, "title": "Imported followers should go in \"users\", relationships in \"following\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-09-03T21:27:37Z", "updated_at": "2019-09-04T20:23:04Z", "closed_at": "2019-09-04T20:23:04Z", "author_association": "MEMBER", "pull_request": null, "body": "Right now `twitter-to-sqlite followers` dumps everything in a `followers` table, and doesn't actually record which account they are following!\r\n\r\nIt should instead save them all in a global `users` table and then set up m2m relationships in a `following` table. This also means it should create a record for the specified user in order to record both sides of each relationship.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 493599818, "node_id": "MDU6SXNzdWU0OTM1OTk4MTg=", "number": 1, "title": "Command for fetching starred repos", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-09-14T08:36:29Z", "updated_at": "2019-09-14T21:30:48Z", "closed_at": "2019-09-14T21:30:48Z", "author_association": "MEMBER", "pull_request": null, "body": "", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 496415321, "node_id": "MDU6SXNzdWU0OTY0MTUzMjE=", "number": 1, "title": "Figure out some interesting example SQL queries", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2019-09-20T15:28:07Z", "updated_at": "2021-05-03T03:46:23Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "My knowledge of genetics has left me short here. I'd love to be able to provide some interesting example SELECT queries - maybe one that spots if you are [likely to have red hair?](https://www.snpedia.com/index.php/Rs1805007)", "repo": {"value": 209590345, "label": "genome-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 503233021, "node_id": "MDU6SXNzdWU1MDMyMzMwMjE=", "number": 1, "title": "Use better pagination (and implement progress bar)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-10-07T04:58:11Z", "updated_at": "2020-03-27T22:13:57Z", "closed_at": "2020-03-27T22:13:57Z", "author_association": "MEMBER", "pull_request": null, "body": "Right now we attempt to load everything at once - which caps out at 5,000 items and is really slow.\r\n\r\nWe can do better by implementing pagination using count and offset.", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 504720731, "node_id": "MDU6SXNzdWU1MDQ3MjA3MzE=", "number": 1, "title": "Add more details on how to request data from google takeout correctly.", "user": {"value": 1055831, "label": "dazzag24"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-10-09T15:17:34Z", "updated_at": "2019-10-09T15:17:34Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "The default is to download everything. This can result in an enormous amount of data when you only really need 2 types of data for now:\r\n\r\n- My Activity\r\n- Location History\r\n\r\nIn addition unless you specify that \"My Activity\" is downloaded in JSON format the default is HTML. This then causes the \r\n\r\n`google-takeout-to-sqlite my-activity takeout.db takeout.zip`\r\n\r\ncommand to fail as it only contains html files not json files.\r\n\r\nThanks", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 519979091, "node_id": "MDExOlB1bGxSZXF1ZXN0MzM4NjQ3Mzc4", "number": 1, "title": "Add parkrun-to-sqlite", "user": {"value": 1101318, "label": "mrw34"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-11-08T12:05:32Z", "updated_at": "2020-10-12T00:35:16Z", "closed_at": "2020-10-12T00:35:16Z", "author_association": "CONTRIBUTOR", "pull_request": "dogsheep/dogsheep.github.io/pulls/1", "body": "", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 585526292, "node_id": "MDU6SXNzdWU1ODU1MjYyOTI=", "number": 1, "title": "Set up full text search", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-03-21T15:57:35Z", "updated_at": "2020-03-21T19:47:46Z", "closed_at": "2020-03-21T19:45:52Z", "author_association": "MEMBER", "pull_request": null, "body": "Should run against `title` and `text` in `items`, and `about` and `id` in `users`.", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 602533300, "node_id": "MDU6SXNzdWU2MDI1MzMzMDA=", "number": 1, "title": "Import photo metadata from Apple Photos into SQLite", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 5324096, "label": "Apple Photos online and securely browsable"}, "comments": 8, "created_at": "2020-04-18T19:23:26Z", "updated_at": "2020-05-04T02:41:40Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Faces, albums, locations, that kind of thing.", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 689800307, "node_id": "MDU6SXNzdWU2ODk4MDAzMDc=", "number": 1, "title": "Add an index on the timestamp column", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-01T04:33:37Z", "updated_at": "2020-09-01T04:49:23Z", "closed_at": "2020-09-01T04:49:23Z", "author_association": "MEMBER", "pull_request": null, "body": "Since default view will likely be ordered by timestamp descending.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 718934942, "node_id": "MDU6SXNzdWU3MTg5MzQ5NDI=", "number": 1, "title": "Documentation on how to use this with Datasette", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-10-11T21:56:27Z", "updated_at": "2020-10-11T22:14:00Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "In particular how to use `datasette-render-images` to see the images.", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1353411865, "node_id": "I_kwDODEpn8M5Qq20Z", "number": 1, "title": "Problem with my user", "user": {"value": 2467, "label": "fernand0"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-08-28T16:59:37Z", "updated_at": "2022-08-28T16:59:37Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "If I call the program with:\r\n inaturalist-to-sqlite inaturalist.db ftricas\r\nthe program exits with an error:\r\n `Importing 36 observations\r\nTraceback (most recent call last):\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/bin/inaturalist-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/inaturalist_to_sqlite/cli.py\", line 51, in cli\r\n save_observation(observation, db)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/inaturalist_to_sqlite/utils.py\", line 34, in save_observation\r\n db[\"observations\"]\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 2965, in insert\r\n return self.insert_all(\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 3068, in insert_all\r\n self.create(\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 1564, in create\r\n self.db.create_table(\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 951, in create_table\r\n sql = self.create_table_sql(\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 765, in create_table_sql\r\n foreign_keys = self.resolve_foreign_keys(name, foreign_keys or [])\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 702, in resolve_foreign_keys\r\n other_table = table.guess_foreign_table(column)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py\", line 2061, in guess_foreign_table\r\n raise NoObviousTable(\r\nsqlite_utils.db.NoObviousTable: No obvious foreign key table for column 'taxon' - tried ['taxon', 'taxons']\r\n`\r\nIf I call the program with your user everything seems to go well and then, I can call the program with my own user without problems. Moreover, I can call the program again with my own user and everything goes well now.\r\n\r\nAdditional info, the command:\r\n \r\n sqlite-utils tables inaturalist.db\r\n\r\nshows that the correct name can be 'taxons'.\r\n\r\nThere is another small problem with a warning:\r\n \r\n warnings.warn(\"urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported \"\r\n\r\n", "repo": {"value": 206202864, "label": "inaturalist-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/inaturalist-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1616347574, "node_id": "I_kwDOJHON9s5gV4G2", "number": 1, "title": "Initial proof of concept with ChatGPT", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-03-09T03:44:39Z", "updated_at": "2023-03-09T03:51:55Z", "closed_at": "2023-03-09T03:51:55Z", "author_association": "MEMBER", "pull_request": null, "body": "I'm using ChatGPT to figure out enough AppleScript to get at my notes data.", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 267513523, "node_id": "MDU6SXNzdWUyNjc1MTM1MjM=", "number": 2, "title": "Initial proof-of-concept", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 0, "created_at": "2017-10-23T00:45:37Z", "updated_at": "2017-10-23T01:26:39Z", "closed_at": "2017-10-23T00:45:53Z", "author_association": "OWNER", "pull_request": null, "body": "Implemented in https://github.com/simonw/stateless-datasets/commit/de04d7a854d71003ffcf98028eab976a936c2dba", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 349850687, "node_id": "MDU6SXNzdWUzNDk4NTA2ODc=", "number": 2, "title": "Mechanism for adding foreign keys to an existing table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2018-08-12T22:50:56Z", "updated_at": "2019-02-24T21:34:41Z", "closed_at": "2019-02-24T21:34:41Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite does not have ALTER TABLE support for adding new foreign keys... but it turns out it's possible to make these changes without having to duplicate the entire table by carefully running `UPDATE sqlite_master SET sql=... WHERE type='table' AND name='X';`\r\n\r\nHere's how Django does it: https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/db/backends/sqlite3/schema.py#L103-L125\r\n\r\nAnd here's the official documentation about this: https://sqlite.org/lang_altertable.html#otheralter (scroll to the very bottom of the page)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 470637152, "node_id": "MDU6SXNzdWU0NzA2MzcxNTI=", "number": 2, "title": "Import workouts", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-07-20T05:20:21Z", "updated_at": "2019-07-20T06:21:41Z", "closed_at": "2019-07-20T06:21:41Z", "author_association": "MEMBER", "pull_request": null, "body": "From #1", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 487598468, "node_id": "MDU6SXNzdWU0ODc1OTg0Njg=", "number": 2, "title": "--save option to dump checkins to a JSON file on disk", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-08-30T17:41:06Z", "updated_at": "2019-08-31T02:40:21Z", "closed_at": "2019-08-31T02:40:21Z", "author_association": "MEMBER", "pull_request": null, "body": "This is a complement to the `--load` option - mainly useful for development purposes.\r\n\r\n(I'll rename `--file` to `--load` as part of this issue).", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 488833698, "node_id": "MDU6SXNzdWU0ODg4MzM2OTg=", "number": 2, "title": "\"twitter-to-sqlite user-timeline\" command for pulling tweets by a specific user", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-09-03T21:29:12Z", "updated_at": "2019-09-04T20:02:11Z", "closed_at": "2019-09-04T20:02:11Z", "author_association": "MEMBER", "pull_request": null, "body": "Twitter only allows up to 3,200 tweets to be retrieved from https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html\r\n\r\nI'm going to do:\r\n\r\n $ twitter-to-sqlite tweets simonw\r\n\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 493668862, "node_id": "MDU6SXNzdWU0OTM2Njg4NjI=", "number": 2, "title": "Extract licenses from repos into a separate table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-09-14T21:33:41Z", "updated_at": "2019-09-14T21:46:58Z", "closed_at": "2019-09-14T21:46:58Z", "author_association": "MEMBER", "pull_request": null, "body": "\"starred3__repos__800_rows\"\r\n", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 503234169, "node_id": "MDU6SXNzdWU1MDMyMzQxNjk=", "number": 2, "title": "Track and use the 'since' value", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-10-07T05:02:59Z", "updated_at": "2020-03-27T22:22:30Z", "closed_at": "2020-03-27T22:22:30Z", "author_association": "MEMBER", "pull_request": null, "body": "Pocket says:\r\n\r\n> Whenever possible, you should use the since parameter, or count and and offset parameters when retrieving a user's list. After retrieving the list, you should store the current time (which is provided along with the list response) and pass that in the next request for the list. This way the server only needs to return a small set (changes since that time) instead of the user's entire list every time.\r\n\r\nAt the bottom of https://getpocket.com/developer/docs/v3/retrieve", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 541274681, "node_id": "MDU6SXNzdWU1NDEyNzQ2ODE=", "number": 2, "title": "Add linkedin-to-sqlite", "user": {"value": 881925, "label": "mnp"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-12-21T03:13:40Z", "updated_at": "2019-12-21T03:13:40Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "There is an API available. https://developer.linkedin.com/docs/rest-api#\r\n\r\nAt the minimum, I would think contact list and messages would be of interest.", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 602533352, "node_id": "MDU6SXNzdWU2MDI1MzMzNTI=", "number": 2, "title": "Ability to convert HEIC images to JPEG", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5324096, "label": "Apple Photos online and securely browsable"}, "comments": 1, "created_at": "2020-04-18T19:23:43Z", "updated_at": "2020-04-28T16:47:21Z", "closed_at": "2020-04-28T16:47:21Z", "author_association": "MEMBER", "pull_request": null, "body": "", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 664793260, "node_id": "MDU6SXNzdWU2NjQ3OTMyNjA=", "number": 2, "title": "Yak shave", "user": {"value": 145425, "label": "ekg"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-07-23T22:04:18Z", "updated_at": "2020-07-23T22:04:18Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Just a quick note... The 23andme data is not exactly your genome, but a SNP chip of your genome. It's \"some of your genotypes.\" Or about 0.1% of your genome. Nice work in any case! It deserves to be liberated!!!!!", "repo": {"value": 209590345, "label": "genome-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 689809225, "node_id": "MDU6SXNzdWU2ODk4MDkyMjU=", "number": 2, "title": "Apply porter stemming", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-01T04:57:55Z", "updated_at": "2020-09-01T20:42:00Z", "closed_at": "2020-09-01T20:40:24Z", "author_association": "MEMBER", "pull_request": null, "body": "This can be on by default. You can turn it off for a table in the config file using `stemming: none` - or maybe `tokenize: none` to match the terminology used by SQLite and `sqlite-utils`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#enabling-full-text-search", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 718938046, "node_id": "MDU6SXNzdWU3MTg5MzgwNDY=", "number": 2, "title": "Convert dates to a better format", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-10-11T22:12:33Z", "updated_at": "2020-10-11T23:15:03Z", "closed_at": "2020-10-11T23:15:03Z", "author_association": "MEMBER", "pull_request": null, "body": "They currently look like this:\r\n\r\nhttps://github.com/dogsheep/evernote-to-sqlite/blob/9d8efd17580f6ddf76745c145d1e69dd24e52b64/tests/test_evernote_to_sqlite.py#L35-L36", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 769376447, "node_id": "MDU6SXNzdWU3NjkzNzY0NDc=", "number": 2, "title": "killed by oomkiller on large location-history", "user": {"value": 231498, "label": "khimaros"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-12-17T00:32:24Z", "updated_at": "2020-12-17T00:48:32Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "memory seems to grow unbounded and is oom-killed after about 20GB memory usage.\r\n\r\nthis is happening while loading a ~1GB uncompressed location history.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 952179830, "node_id": "MDU6SXNzdWU5NTIxNzk4MzA=", "number": 2, "title": "Command for fetching Hacker News threads from the search API", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-07-25T02:00:45Z", "updated_at": "2021-07-25T03:12:57Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I want to be able to fetch every item for a domain, e.g. https://news.ycombinator.com/from?site=simonwillison.net", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1485017981, "node_id": "I_kwDODEpn8M5Yg5N9", "number": 2, "title": "table identifications has no column named previous_observation_taxon", "user": {"value": 520541, "label": "heaversm"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-12-08T16:47:17Z", "updated_at": "2022-12-08T16:47:17Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Installed successfully with pip and ran `inaturalist-to-sqlite inaturalist.db simonw` and got the error:\r\n\r\n```\r\nsqlite3.OperationalError: table identifications has no column named previous_observation_taxon\r\n```", "repo": {"value": 206202864, "label": "inaturalist-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/inaturalist-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1616354999, "node_id": "I_kwDOJHON9s5gV563", "number": 2, "title": "First working version", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2023-03-09T03:53:00Z", "updated_at": "2023-03-09T05:10:22Z", "closed_at": "2023-03-09T05:10:22Z", "author_association": "MEMBER", "pull_request": null, "body": "It's going to shell out to `osascript` as seen in:\r\n- #1\r\n\r\nI'm going with that option because https://appscript.sourceforge.io/status.html warns against the other potential methods:\r\n\r\n> Apple eliminated its Mac Automation department in 2016. The future of AppleScript and its related technologies is unclear. Caveat emptor.\r\n\r\nBut `osascript` looks pretty stable to me.", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 267515678, "node_id": "MDU6SXNzdWUyNjc1MTU2Nzg=", "number": 3, "title": "Make individual column valuables addressable, with smart content types", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2017-10-23T01:11:32Z", "updated_at": "2017-12-10T03:11:58Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Some SQLite databases embed images in columns. It would be cool if these had URLs.\r\n\r\n /database-name-7sha256/table-name/compound-pk/column\r\n /database-name-7sha256/table-name/compound-pk/column.json\r\n /database-name-7sha256/table-name/compound-pk/column.png\r\n /database-name-7sha256/table-name/compound-pk/column.gif\r\n /database-name-7sha256/table-name/compound-pk/column.txt\r\n\r\nThe one without an explicit file extension auto-detects the correct extension.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 351845423, "node_id": "MDU6SXNzdWUzNTE4NDU0MjM=", "number": 3, "title": "Experiment with contentless FTS tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2018-08-18T19:31:01Z", "updated_at": "2019-07-22T20:58:55Z", "closed_at": "2019-07-22T20:58:55Z", "author_association": "OWNER", "pull_request": null, "body": "Could greatly reduce size of resulting database for large datasets: http://cocoamine.net/blog/2015/09/07/contentless-fts4-for-large-immutable-documents/", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 470637206, "node_id": "MDU6SXNzdWU0NzA2MzcyMDY=", "number": 3, "title": "Import ActivitySummary", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-07-20T05:21:00Z", "updated_at": "2019-07-20T05:58:07Z", "closed_at": "2019-07-20T05:58:07Z", "author_association": "MEMBER", "pull_request": null, "body": "From #1\r\n\r\n```python\r\n 'ActivitySummary': {'attr_counts': {'activeEnergyBurned': 980,\r\n 'activeEnergyBurnedGoal': 980,\r\n 'activeEnergyBurnedUnit': 980,\r\n 'appleExerciseTime': 980,\r\n 'appleExerciseTimeGoal': 980,\r\n 'appleStandHours': 980,\r\n 'appleStandHoursGoal': 980,\r\n 'dateComponents': 980},\r\n 'child_counts': {},\r\n 'count': 980,\r\n 'parent_counts': {'HealthData': 980}},\r\n```", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 487600595, "node_id": "MDU6SXNzdWU0ODc2MDA1OTU=", "number": 3, "title": "Option to fetch only checkins more recent than the current max checkin", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-08-30T17:46:45Z", "updated_at": "2019-10-16T20:41:23Z", "closed_at": "2019-10-16T20:39:59Z", "author_association": "MEMBER", "pull_request": null, "body": "The Foursquare checkins API supports \"return every checkin occurring after this point\" - I can pass it the maximum createdAt date currently stored in the database. This will allow for quick incremental fetches via a cron.", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 488833975, "node_id": "MDU6SXNzdWU0ODg4MzM5NzU=", "number": 3, "title": "Command for running a search and saving tweets for that search", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2019-09-03T21:29:56Z", "updated_at": "2019-11-04T05:31:56Z", "closed_at": "2019-11-04T05:31:16Z", "author_association": "MEMBER", "pull_request": null, "body": " $ twitter-to-sqlite search dogsheep", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 493670426, "node_id": "MDU6SXNzdWU0OTM2NzA0MjY=", "number": 3, "title": "Command to fetch all repos belonging to a user or organization", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-09-14T21:54:21Z", "updated_at": "2019-09-17T00:17:53Z", "closed_at": "2019-09-17T00:17:53Z", "author_association": "MEMBER", "pull_request": null, "body": "How about this:\r\n\r\n $ github-to-sqlite repos simonw", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 503243784, "node_id": "MDU6SXNzdWU1MDMyNDM3ODQ=", "number": 3, "title": "Extract images into separate tables", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-10-07T05:43:01Z", "updated_at": "2020-09-01T06:17:45Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "As already done with authors. Slightly harder because images do not have a universally unique ID. Also need to figure out what to do about there being columns for both `image` and `images`.\r\n\r\n\"memory__items\"\r\n", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 543717994, "node_id": "MDExOlB1bGxSZXF1ZXN0MzU3OTc0MzI2", "number": 3, "title": "Add todoist-to-sqlite", "user": {"value": 706257, "label": "bcongdon"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-12-30T04:02:59Z", "updated_at": "2020-10-12T00:35:58Z", "closed_at": "2020-10-12T00:35:57Z", "author_association": "CONTRIBUTOR", "pull_request": "dogsheep/dogsheep.github.io/pulls/3", "body": "Really enjoying getting into the dogsheep/datasette ecosystem. I made a downloader for Todoist, and I think/hope others might find this useful", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 602533481, "node_id": "MDU6SXNzdWU2MDI1MzM0ODE=", "number": 3, "title": "Import EXIF data into SQLite - lens used, ISO, aperture etc", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 5324096, "label": "Apple Photos online and securely browsable"}, "comments": 2, "created_at": "2020-04-18T19:24:31Z", "updated_at": "2021-10-05T12:38:24Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 689810340, "node_id": "MDU6SXNzdWU2ODk4MTAzNDA=", "number": 3, "title": "Datasette plugin to provide custom page for running faceted, ranked searches", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-09-01T05:00:22Z", "updated_at": "2020-09-03T21:01:41Z", "closed_at": "2020-09-03T21:01:41Z", "author_association": "MEMBER", "pull_request": null, "body": "This will be a page at `/-/beta` which renders using a custom template.\r\n\r\nIt will offer a default timeline view plus search and facet by type/date.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 718938321, "node_id": "MDU6SXNzdWU3MTg5MzgzMjE=", "number": 3, "title": "Use a content hash for the note IDs", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-10-11T22:13:46Z", "updated_at": "2020-10-11T23:15:04Z", "closed_at": "2020-10-11T23:15:04Z", "author_association": "MEMBER", "pull_request": null, "body": "Without a GUID note IDs are pretty ineffective, but using a hash of the contents will at least avoid creating identical duplicates in the future.\r\n\r\nhttps://sqlite-utils.readthedocs.io/en/stable/python-api.html#setting-an-id-based-on-the-hash-of-the-row-contents", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 769397742, "node_id": "MDU6SXNzdWU3NjkzOTc3NDI=", "number": 3, "title": "sqlite-utils error on takeout import", "user": {"value": 231498, "label": "khimaros"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-12-17T01:18:48Z", "updated_at": "2020-12-17T01:19:04Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "```\r\n$ google-takeout-to-sqlite my-activity takeout.db /path/to/zip\r\n...\r\nsqlite3.OperationalError: no such table: main.my_activity\r\n```\r\n\r\nthere is no table create in `utils.py`, unlike other importers such as github-to-sqlite\r\n\r\nadditionally, this package and hackernews-to-sqlite have conflicting `sqlite-utils` dep with datasette and dogsheep-beta", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/3/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 952189173, "node_id": "MDU6SXNzdWU5NTIxODkxNzM=", "number": 3, "title": "Use HN algolia endpoint to retrieve trees", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-07-25T03:35:27Z", "updated_at": "2021-07-25T18:41:17Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "The `trees` command currently has to make a request for every single comment. Algolia have an endpoint that bundles the entire thread together into a single request.\r\n\r\n`https://hn.algolia.com/api/v1/items/ID`\r\n\r\nHere's an example that loads quickly, with about 50 comments: https://hn.algolia.com/api/v1/items/27941108\r\n\r\nIt doesn't appear to use pagination at all - if a thread is big then the response is big.\r\n\r\nI ran this search to find some stories with more than 1000 comments: https://hn.algolia.com/api/v1/search?tags=story&numericFilters=num_comments%3E=1000\r\n\r\nHere's one: https://news.ycombinator.com/item?id=25015967 with 4759 comments. Hitting the API takes 41s and returns 3.7 MB of JSON!\r\n```\r\nwget 'https://hn.algolia.com/api/v1/items/25015967' 0.03s user 0.04s system 0% cpu 41.368 total\r\n/tmp % ls -lah 25015967 \r\n-rw-r--r-- 1 simon wheel 3.7M Jul 24 20:31 25015967\r\n```", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1616422013, "node_id": "I_kwDOJHON9s5gWKR9", "number": 3, "title": "`apple-notes-to-sqlite --dump` option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-03-09T05:05:49Z", "updated_at": "2023-03-09T05:06:14Z", "closed_at": "2023-03-09T05:06:14Z", "author_association": "MEMBER", "pull_request": null, "body": "Option that doesn't write to the database at all, it just outputs all the notes to stdout as newline-delimited JSON.", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 267515836, "node_id": "MDU6SXNzdWUyNjc1MTU4MzY=", "number": 4, "title": "Make URLs immutable", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 8, "created_at": "2017-10-23T01:13:30Z", "updated_at": "2017-10-24T02:38:24Z", "closed_at": "2017-10-24T02:38:24Z", "author_association": "OWNER", "pull_request": null, "body": "Absolutely everything should have a far-future expires header\r\n\r\nPart of the URL will be the truncated sha1 hash of the database file itself, calculated at build time", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 403028630, "node_id": "MDExOlB1bGxSZXF1ZXN0MjQ3NTc2OTQy", "number": 4, "title": "Fts5", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-01-25T06:54:05Z", "updated_at": "2019-01-25T06:54:33Z", "closed_at": "2019-01-25T06:54:33Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/4", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 470640505, "node_id": "MDU6SXNzdWU0NzA2NDA1MDU=", "number": 4, "title": "Import Records", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-07-20T06:11:20Z", "updated_at": "2019-07-20T06:21:41Z", "closed_at": "2019-07-20T06:21:41Z", "author_association": "MEMBER", "pull_request": null, "body": "From #1:\r\n```python\r\n 'Record': {'attr_counts': {'creationDate': 2672233,\r\n 'device': 2665111,\r\n 'endDate': 2672233,\r\n 'sourceName': 2672233,\r\n 'sourceVersion': 2671779,\r\n 'startDate': 2672233,\r\n 'type': 2672233,\r\n 'unit': 2650012,\r\n 'value': 2672232},\r\n 'child_counts': {'HeartRateVariabilityMetadataList': 2318,\r\n 'MetadataEntry': 287974},\r\n 'count': 2672233,\r\n 'parent_counts': {'Correlation': 2, 'HealthData': 2672231}},\r\n```", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 487601121, "node_id": "MDU6SXNzdWU0ODc2MDExMjE=", "number": 4, "title": "Online tool for getting a Foursquare OAuth token", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-08-30T17:48:14Z", "updated_at": "2019-08-31T18:07:26Z", "closed_at": "2019-08-31T18:07:26Z", "author_association": "MEMBER", "pull_request": null, "body": "I will link to this from the documentation. See also this conversation on Twitter: https://twitter.com/simonw/status/1166822603023011840\r\n\r\nI've decided to go with \"copy and paste in a token\" rather than hooking up a local web server that can have tokens passed to it.", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 488835586, "node_id": "MDU6SXNzdWU0ODg4MzU1ODY=", "number": 4, "title": "Command for importing data from a Twitter Export file", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-09-03T21:34:13Z", "updated_at": "2019-10-11T06:45:02Z", "closed_at": "2019-10-11T06:45:02Z", "author_association": "MEMBER", "pull_request": null, "body": "Twitter lets you export all of your data as an archive file: https://twitter.com/settings/your_twitter_data\r\n\r\nA command for importing this data into SQLite would be extremely useful.\r\n\r\n $ twitter-to-sqlite import twitter.db path-to-archive.zip\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 493670730, "node_id": "MDU6SXNzdWU0OTM2NzA3MzA=", "number": 4, "title": "Command to fetch stargazers for one or more repos", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2019-09-14T21:58:22Z", "updated_at": "2020-05-02T21:30:27Z", "closed_at": "2020-05-02T21:30:27Z", "author_association": "MEMBER", "pull_request": null, "body": "Maybe this:\r\n\r\n $ github-to-sqlite stargazers github.db simonw/datasette\r\n\r\nIt could accept more than one repos.\r\n\r\nMaybe have options similar to `--sql` in [twitter-to-sqlite](https://github.com/dogsheep/twitter-to-sqlite) so you can e.g. fetch all stargazers for all of the repos you have fetched into the database already (or all of the repos belonging to owner X)", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 558715564, "node_id": "MDExOlB1bGxSZXF1ZXN0MzcwMDI0Njk3", "number": 4, "title": "Add beeminder-to-sqlite", "user": {"value": 706257, "label": "bcongdon"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-02-02T15:51:36Z", "updated_at": "2020-10-12T00:36:16Z", "closed_at": "2020-10-12T00:36:16Z", "author_association": "CONTRIBUTOR", "pull_request": "dogsheep/dogsheep.github.io/pulls/4", "body": "", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 589402939, "node_id": "MDU6SXNzdWU1ODk0MDI5Mzk=", "number": 4, "title": "Store authentication information as \"pocket_access_token\" etc", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-03-27T20:43:22Z", "updated_at": "2020-03-27T20:43:59Z", "closed_at": "2020-03-27T20:43:59Z", "author_association": "MEMBER", "pull_request": null, "body": "The `pocket_` prefix will mean that the same `auth.json` file can be used for other Dogsheep tools without Pocket over-riding a value set by some other tool.", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 602533539, "node_id": "MDU6SXNzdWU2MDI1MzM1Mzk=", "number": 4, "title": "Upload all my photos to a secure S3 bucket", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5324096, "label": "Apple Photos online and securely browsable"}, "comments": 14, "created_at": "2020-04-18T19:24:50Z", "updated_at": "2020-04-18T21:58:11Z", "closed_at": "2020-04-18T21:57:13Z", "author_association": "MEMBER", "pull_request": null, "body": "- [x] Create a bucket with bucket credentials\r\n- [x] Programmatically upload some recent photos to it (from a notebook)\r\n- [x] Turn this into a script", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 689839399, "node_id": "MDU6SXNzdWU2ODk4MzkzOTk=", "number": 4, "title": "Optimize the FTS table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-01T05:58:17Z", "updated_at": "2020-09-01T06:10:08Z", "closed_at": "2020-09-01T06:10:08Z", "author_association": "MEMBER", "pull_request": null, "body": "", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 718938508, "node_id": "MDU6SXNzdWU3MTg5Mzg1MDg=", "number": 4, "title": "Configure FTS + add an index on the date columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-10-11T22:14:40Z", "updated_at": "2020-10-11T23:41:29Z", "closed_at": "2020-10-11T23:41:29Z", "author_association": "MEMBER", "pull_request": null, "body": "Sort by date descending is likely the most common way of sorting, so that column should be indexed.\r\n\r\nAlso add FTS configuration for both notes and the OCR column on resources.", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 778380836, "node_id": "MDU6SXNzdWU3NzgzODA4MzY=", "number": 4, "title": "Feature Request: Gmail", "user": {"value": 203343, "label": "Btibert3"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-01-04T21:31:09Z", "updated_at": "2021-03-04T20:54:44Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "From takeout, I only exported my Gmail account. Ideally I could parse this into sqlite via this tool.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1205867842, "node_id": "I_kwDODtX3eM5H4BVC", "number": 4, "title": "Retrieve the top-level story for a comment", "user": {"value": 1755789, "label": "telotortium"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-04-15T20:25:39Z", "updated_at": "2022-04-15T20:25:39Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I think that each comment inserted into the database should include a column `onstory` that contains the ID of the story on which the comment was made. This is exactly equivalent to the link after \"on:\" at the top of an HN comment page ([example](https://news.ycombinator.com/item?id=18358028)). We could do this either by directly retrieving the HTML page and using Beautiful Soup to find that link, or alternatively recurse up the tree in the Firebase API using the `parent` field (probably using `functools.lru_cache` in case a person has commented a bunch of times on the same story).", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1616429236, "node_id": "I_kwDOJHON9s5gWMC0", "number": 4, "title": "Support incremental updates", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-03-09T05:14:00Z", "updated_at": "2023-03-09T18:20:56Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Running this script can take several hours against a large notes database.\r\n\r\nWould be neat if it could run against just the notes that have been modified since it last ran. Could pull the max `updated` date and then keep on looping until it finds one modified before then.\r\n\r\nProblem is I don't actually know what order it iterates over the notes in.", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 267516066, "node_id": "MDU6SXNzdWUyNjc1MTYwNjY=", "number": 5, "title": "Implement sensible query pagination", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 3, "created_at": "2017-10-23T01:16:00Z", "updated_at": "2017-11-10T20:41:39Z", "closed_at": "2017-11-10T20:41:39Z", "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 403396009, "node_id": "MDExOlB1bGxSZXF1ZXN0MjQ3ODYxNDE5", "number": 5, "title": "Run Travis tests against Python 3.8-dev", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-01-26T02:30:55Z", "updated_at": "2019-01-26T02:37:54Z", "closed_at": "2019-01-26T02:37:54Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/5", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 470691622, "node_id": "MDU6SXNzdWU0NzA2OTE2MjI=", "number": 5, "title": "Add progress bar", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-07-20T16:29:07Z", "updated_at": "2019-07-22T03:30:13Z", "closed_at": "2019-07-22T02:49:22Z", "author_association": "MEMBER", "pull_request": null, "body": "Showing a progress bar would be nice, using Click.\r\n\r\nThe easiest way to do this would probably be be to hook it up to the length of the compressed content, and update it as this code pushes more XML bytes through the parser:\r\n\r\nhttps://github.com/dogsheep/healthkit-to-sqlite/blob/d64299765064501f4efdd9a0b21dbdba9ec4287f/healthkit_to_sqlite/utils.py#L6-L10", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 487721884, "node_id": "MDU6SXNzdWU0ODc3MjE4ODQ=", "number": 5, "title": "Treat Foursquare timestamps as UTC", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-08-31T02:44:47Z", "updated_at": "2019-08-31T02:50:41Z", "closed_at": "2019-08-31T02:50:41Z", "author_association": "MEMBER", "pull_request": null, "body": "Current test failure is due to timezone differences between my laptop and Circle CI:\r\n\r\nhttps://circleci.com/gh/dogsheep/swarm-to-sqlite/3\r\n\r\n```\r\nE Full diff:\r\nE - [{'created': '2018-07-01T04:48:19',\r\nE ? ^\r\nE + [{'created': '2018-07-01T02:48:19',\r\nE ? ^\r\nE 'createdAt': 1530413299,\r\n```\r\n\r\nThe timestamps I store in `created` should always be UTC.", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 488874815, "node_id": "MDU6SXNzdWU0ODg4NzQ4MTU=", "number": 5, "title": "Write tests that simulate the Twitter API", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-09-03T23:55:35Z", "updated_at": "2019-09-03T23:56:28Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I can use betamax for this: https://pypi.org/project/betamax/", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 493671014, "node_id": "MDU6SXNzdWU0OTM2NzEwMTQ=", "number": 5, "title": "Add \"incomplete\" boolean to users table for incomplete profiles", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-09-14T22:01:50Z", "updated_at": "2020-03-23T19:23:31Z", "closed_at": "2020-03-23T19:23:30Z", "author_association": "MEMBER", "pull_request": null, "body": "User profiles that are fetched from e.g. stargazers (#4) are incomplete - they have a login but they don't have name, company etc. \r\n\r\nAdd a `incomplete` boolean flag to the `users` table to record this. Then later I can add a `backfill-users` command which loops through and fetches missing data for those incomplete profiles.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 602551638, "node_id": "MDU6SXNzdWU2MDI1NTE2Mzg=", "number": 5, "title": "photos-to-sqlite s3-auth command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-04-18T21:05:25Z", "updated_at": "2020-04-18T21:08:44Z", "closed_at": "2020-04-18T21:08:44Z", "author_association": "MEMBER", "pull_request": null, "body": "Modeled on `github-to-sqlite auth` - prompts the user for their S3 credentials and saves them to `auth.json`.", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 629473827, "node_id": "MDU6SXNzdWU2Mjk0NzM4Mjc=", "number": 5, "title": "Set up a demo", "user": {"value": 26745575, "label": "harryvederci"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-06-02T19:56:49Z", "updated_at": "2020-09-01T06:18:43Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "First off, thanks for open sourcing this application! This is a suggestion to increase the amount of people that would make use of it: an example in the readme file would help.\r\n\r\nCurrently, users have to clone the app, install it, authorize through pocket, run a command, an then find out if this application does what they hope it does.\r\n\r\nAnother possibility is to add a file `example-output.db`, containing one (mock) Pocket article.\r\n\r\nKeep up the good work!", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 689847361, "node_id": "MDU6SXNzdWU2ODk4NDczNjE=", "number": 5, "title": "Add a context column that's not searchable", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-01T06:13:42Z", "updated_at": "2020-09-03T18:43:50Z", "closed_at": "2020-09-03T18:43:50Z", "author_association": "MEMBER", "pull_request": null, "body": "I sometimes like to configure titles that are things like \"Comment on issue X\" or \"Photo in Golden Gate Park\" - these shouldn't be included in the search index but should be stored so they can be displayed to provide context.\r\n\r\nAdd a column for this - probably called `context` - and make it so it can be populated.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 718938889, "node_id": "MDU6SXNzdWU3MTg5Mzg4ODk=", "number": 5, "title": "Figure out how to display images from tags inline in Datasette", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-10-11T22:17:03Z", "updated_at": "2020-10-16T20:16:28Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Relates to #1. Evernote XML looks like this:\r\n\r\n```xml\r\n\r\n\r\n
This note includes two images.
\r\n
\r\n The Python logo\r\n
\r\n
\r\n \r\n
\r\n
\r\n The Evernote logo\r\n
\r\n
\r\n \r\n
\r\n\r\n```\r\nThat hash is the md5 we use to store resources. It should be possible to turn these into embedded image tags, especially if done in conjunction with the https://github.com/simonw/datasette-media plugin.", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 723499985, "node_id": "MDExOlB1bGxSZXF1ZXN0NTA1MDc2NDE4", "number": 5, "title": "Add fitbit-to-sqlite", "user": {"value": 4632208, "label": "mrphil007"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-10-16T20:04:05Z", "updated_at": "2020-10-16T20:04:05Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/dogsheep.github.io/pulls/5", "body": "", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 813880401, "node_id": "MDExOlB1bGxSZXF1ZXN0NTc3OTUzNzI3", "number": 5, "title": "WIP: Add Gmail takeout mbox import", "user": {"value": 306240, "label": "UtahDave"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 25, "created_at": "2021-02-22T21:30:40Z", "updated_at": "2021-07-28T07:18:56Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/google-takeout-to-sqlite/pulls/5", "body": "WIP\r\n\r\nThis PR adds the ability to import emails from a Gmail mbox export from Google Takeout.\r\n\r\nThis is my first PR to a datasette/dogsheep repo. I've tested this on my personal Google Takeout mbox with ~520,000 emails going back to 2004. This took around ~20 minutes to process.\r\n\r\nTo provide some feedback on the progress of the import I added the \"rich\" python module. I'm happy to remove that if adding a dependency is discouraged. However, I think it makes a nice addition to give feedback on the progress of a long import.\r\n\r\nDo we want to log emails that have errors when trying to import them?\r\n\r\nDealing with encodings with emails is a bit tricky. I'm very open to feedback on how to deal with those better. As well as any other feedback for improvements.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1353418822, "node_id": "PR_kwDODtX3eM497MOV", "number": 5, "title": "The program fails when the user has no submissions", "user": {"value": 2467, "label": "fernand0"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-08-28T17:25:45Z", "updated_at": "2022-08-28T17:25:45Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/hacker-news-to-sqlite/pulls/5", "body": "Tested with:\r\n \r\n hacker-news-to-sqlite user hacker-news.db fernand0\r\n\r\nResult:\r\n`\r\nTraceback (most recent call last):\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/bin/hacker-news-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1657, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/hacker_news_to_sqlite/cli.py\", line 27, in user\r\n submitted = user.pop(\"submitted\", None) or []\r\nAttributeError: 'NoneType' object has no attribute 'pop'\r\n`\r\n\r\nThere is a problem of style with the patch (but not sure what to do) because with the new inicialization ( submitted = []) the part \r\n\r\n or []\r\n\r\nis not needed. Maybe there is a more adequate way of doing this.", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1616440856, "node_id": "I_kwDOJHON9s5gWO4Y", "number": 5, "title": "Configure full text search", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-03-09T05:20:46Z", "updated_at": "2023-03-09T05:20:46Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "FTS would be useful.\r\n\r\nMaybe even extract the plain text from the notes to make that index easier to create, rather than creating it against the HTML. Can use the `plaintext` property for that.", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 267516329, "node_id": "MDU6SXNzdWUyNjc1MTYzMjk=", "number": 6, "title": "Better JSON response options", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 0, "created_at": "2017-10-23T01:18:47Z", "updated_at": "2017-10-24T15:07:58Z", "closed_at": "2017-10-24T15:07:58Z", "author_association": "OWNER", "pull_request": null, "body": "Default returns this:\r\n\r\n {\r\n \u201cColumns\u201d: [\u201cid\u201d, \u201cname\u201d, \u201cage\u201d],\r\n \u201cRows\u201d: [\r\n [45, \u201cSimon\u201d, 36]\r\n ]\r\n }\r\n\r\n.jsono instead returns a list of objects each duplicating the headers in its keys.\r\n\r\nThey both probably share the same pagination mechanism so it might not be a jsono flat list.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 403624090, "node_id": "MDU6SXNzdWU0MDM2MjQwOTA=", "number": 6, "title": "\"sqlite-utils insert\" should support newline-delimited JSON", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-01-28T02:00:02Z", "updated_at": "2019-01-28T02:17:45Z", "closed_at": "2019-01-28T02:17:45Z", "author_association": "OWNER", "pull_request": null, "body": "We can already export newline delimited JSON. We should learn to import it as well.\r\n\r\nThe neat thing about importing it is that you can import GBs of data without having to read the whole lot into memory in order to decode the wrapping JSON array.\r\n\r\nDatasette can export it now: https://github.com/simonw/datasette/issues/405\r\n\r\nDemo: https://latest.datasette.io/fixtures/facetable.json?_shape=array&_nl=on\r\n\r\nIt should be possible to do this:\r\n\r\n $ curl \"https://latest.datasette.io/fixtures/facetable.json?_shape=array&_nl=on\" \\\r\n | sqlite-utils insert data.db facetable - --nl\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 470856782, "node_id": "MDU6SXNzdWU0NzA4NTY3ODI=", "number": 6, "title": "Break up records into different tables for each type", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-07-22T01:54:59Z", "updated_at": "2019-07-22T03:28:55Z", "closed_at": "2019-07-22T03:28:50Z", "author_association": "MEMBER", "pull_request": null, "body": "I don't think there's much benefit to having all of the different record types stored in the same enormous table. Here's what I get when I use `_facet=type`:\r\n\r\n\"hello2__records__2_672_233_rows\"\r\n\r\nI'm going to try splitting these up into separate tables - so `HKQuantityTypeIdentifierBodyMassIndex` becomes a table called `rBodyMassIndex` - and see if that's nicer to work with.", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 489419782, "node_id": "MDU6SXNzdWU0ODk0MTk3ODI=", "number": 6, "title": "Extract extended_entities into a media table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-09-04T21:59:10Z", "updated_at": "2019-09-04T22:08:01Z", "closed_at": "2019-09-04T22:08:01Z", "author_association": "MEMBER", "pull_request": null, "body": "\"cleo__select___from_tweets_order_by_id_limit_101\"\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 504238461, "node_id": "MDU6SXNzdWU1MDQyMzg0NjE=", "number": 6, "title": "sqlite3.OperationalError: table users has no column named bio", "user": {"value": 1055831, "label": "dazzag24"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-10-08T19:39:52Z", "updated_at": "2019-10-13T05:31:28Z", "closed_at": "2019-10-13T05:30:19Z", "author_association": "NONE", "pull_request": null, "body": "```\r\n$ github-to-sqlite repos github.db\r\n$ github-to-sqlite starred github.db dazzag24\r\n\r\nTraceback (most recent call last):\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/bin/github-to-sqlite\", line 10, in \r\n sys.exit(cli())\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 764, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 717, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 1137, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 956, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 555, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/cli.py\", line 106, in starred\r\n utils.save_stars(db, user, stars)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py\", line 177, in save_stars\r\n user_id = save_user(db, user)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py\", line 61, in save_user\r\n return db[\"users\"].upsert(to_save, pk=\"id\").last_pk\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py\", line 1067, in upsert\r\n extracts=extracts,\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py\", line 916, in insert\r\n extracts=extracts,\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py\", line 1024, in insert_all\r\n result = self.db.conn.execute(sql, values)\r\nsqlite3.OperationalError: table users has no column named bio\r\n\r\n```\r\n\r\n```\r\n$ pipenv graph\r\ngithub-to-sqlite==0.4\r\n - requests [required: Any, installed: 2.22.0]\r\n - certifi [required: >=2017.4.17, installed: 2019.9.11]\r\n - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]\r\n - idna [required: >=2.5,<2.9, installed: 2.8]\r\n - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]\r\n - sqlite-utils [required: ~=1.11, installed: 1.11]\r\n - click [required: Any, installed: 7.0]\r\n - click-default-group [required: Any, installed: 1.2.2]\r\n - click [required: Any, installed: 7.0]\r\n - tabulate [required: Any, installed: 0.8.5]\r\n\r\nPython 3.6.8\r\n```", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 543355051, "node_id": "MDExOlB1bGxSZXF1ZXN0MzU3NjQwMTg2", "number": 6, "title": "don't break if source is missing", "user": {"value": 78035, "label": "mfa"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-12-29T10:46:47Z", "updated_at": "2020-03-28T02:28:11Z", "closed_at": "2020-03-28T02:28:11Z", "author_association": "CONTRIBUTOR", "pull_request": "dogsheep/swarm-to-sqlite/pulls/6", "body": "broke for me. very old checkins in 2010 had no source set.", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 602575575, "node_id": "MDU6SXNzdWU2MDI1NzU1NzU=", "number": 6, "title": "Add progress bar to upload command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-18T23:32:41Z", "updated_at": "2020-04-19T00:15:24Z", "closed_at": "2020-04-19T00:15:24Z", "author_association": "MEMBER", "pull_request": null, "body": "Upload was added in #4 ", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 689848827, "node_id": "MDU6SXNzdWU2ODk4NDg4Mjc=", "number": 6, "title": "ISO timestamps", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-01T06:16:42Z", "updated_at": "2020-09-01T06:16:42Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "The `time_added`, `time_updated` and `time_read` columns currently store data like this:\r\n\r\n September 19, 2019 - 00:30:30 UTC\r\n\r\nShould use ISO instead, e.g. `2020-07-26T01:05:24+00:00`", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 689850810, "node_id": "MDU6SXNzdWU2ODk4NTA4MTA=", "number": 6, "title": "Set up a demo instance", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-01T06:20:24Z", "updated_at": "2020-09-01T06:20:24Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Once I've got the Datasette plugin to a state where it's worth building a demo: #3\r\n\r\nI can use data from my public https://github-to-sqlite.dogsheep.net/ demo plus the Pocket data subset I use for the demo in https://github.com/dogsheep/pocket-to-sqlite/issues/5 - I could pull in the https://dogsheep-photos.dogsheep.net/ photos data too.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 718949182, "node_id": "MDU6SXNzdWU3MTg5NDkxODI=", "number": 6, "title": "Better handling of OCR data", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-10-11T23:20:52Z", "updated_at": "2020-10-12T00:04:10Z", "closed_at": "2020-10-12T00:04:10Z", "author_association": "MEMBER", "pull_request": null, "body": "> I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought.\r\n\r\n_Originally posted by @simonw in https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706784028_", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 821841046, "node_id": "MDU6SXNzdWU4MjE4NDEwNDY=", "number": 6, "title": "Upgrade to latest sqlite-utils", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-03-04T07:21:54Z", "updated_at": "2021-03-04T07:22:51Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "This is pinned to v1 at the moment.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 842765105, "node_id": "MDExOlB1bGxSZXF1ZXN0NjAyMjYxMDky", "number": 6, "title": "Add testres-db tool", "user": {"value": 1151557, "label": "ligurio"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-03-28T15:43:23Z", "updated_at": "2022-02-16T05:12:05Z", "closed_at": "2022-02-16T05:12:05Z", "author_association": "NONE", "pull_request": "dogsheep/dogsheep.github.io/pulls/6", "body": "", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1617602868, "node_id": "I_kwDOJHON9s5gaqk0", "number": 6, "title": "Character encoding problem", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-03-09T16:44:34Z", "updated_at": "2023-04-14T15:22:09Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I ran against a recent note with this in it:\r\n\r\n> Or just \"Actions \u2699\ufe0f \"\r\n\r\nAnd got back:\r\n\r\n\"image\"\r\n\r\n> `Actions \u201a\u00f6\u00f4\u00d4\u220f\u00e8`\r\n\r\nPasting that into https://ftfy.vercel.app/?s=Actions+%E2%80%9A%C3%B6%C3%B4%C3%94%E2%88%8F%C3%A8+ gives this:\r\n\r\n```python\r\ns = 'Actions \u00e2\\x80\\x9a\u00c3\u00b6\u00c3\u00b4\u00c3\\x94\u00e2\\x88\\x8f\u00c3\u00a8'\r\ns = s.encode('latin-1')\r\ns = s.decode('utf-8')\r\ns = s.encode('macroman')\r\ns = s.decode('utf-8')\r\nprint(s)\r\n```\r\n\"image\"\r\n", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1641117021, "node_id": "PR_kwDODtX3eM5M66op", "number": 6, "title": "Add permalink virtual field to items table", "user": {"value": 1231935, "label": "xavdid"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-03-26T22:22:38Z", "updated_at": "2023-03-29T18:38:52Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/hacker-news-to-sqlite/pulls/6", "body": "I added a virtual column (no storage overhead) to the output that easily links back to the source. It works nicely out of the box with datasette:\r\n\r\n![](https://cdn.zappy.app/faf43661d539ee0fee02c0421de22d65.png)\r\n\r\nI got bit a bit by https://github.com/simonw/sqlite-utils/issues/411, so I went with a manual `table_xinfo` and creating the table via execute. Happy to adjust if that issue moves, but this seems like it works.\r\n\r\nI also added my best-guess instructions for local development on this package. I'm shooting in the dark, so feel free to replace with how you work on it locally.", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 267516650, "node_id": "MDU6SXNzdWUyNjc1MTY2NTA=", "number": 7, "title": "Framework where by every page is JSON plus a template", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 1, "created_at": "2017-10-23T01:22:03Z", "updated_at": "2017-10-24T02:27:25Z", "closed_at": "2017-10-24T02:27:25Z", "author_association": "OWNER", "pull_request": null, "body": "Every single page of my interface should be implemented as a function that returns JSON.\r\n\r\nI can then build my jinja templates on top of the exact data that would be returned by the API version.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 403625674, "node_id": "MDU6SXNzdWU0MDM2MjU2NzQ=", "number": 7, "title": ".insert_all() should accept a generator and process it efficiently", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-01-28T02:11:58Z", "updated_at": "2019-01-28T06:26:53Z", "closed_at": "2019-01-28T06:26:53Z", "author_association": "OWNER", "pull_request": null, "body": "Right now you have to load every record into memory before passing the list to `.insert_all()` and friends.\r\n\r\nIf you want to process millions of rows, this is inefficient. Python has generators - we should use them!\r\n\r\nThe only catch here is that part of the magic of `sqlite-utils` is that it guesses the column types and creates the table for you. This code will need to be updated to notice if the table needs creating and, if it does, create it using the first X (where x=1,000 but can be customized) records.\r\n\r\nIf a record outside of those first 1,000 has a rogue column, we can crash with an error.\r\n\r\nThis will free us up to make the `--nl` option added in #6 much more efficient.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 472097220, "node_id": "MDU6SXNzdWU0NzIwOTcyMjA=", "number": 7, "title": "Script uses a lot of RAM", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-07-24T06:11:11Z", "updated_at": "2019-07-24T06:35:52Z", "closed_at": "2019-07-24T06:35:52Z", "author_association": "MEMBER", "pull_request": null, "body": "I'm using an XML pull parser which should avoid the need to slurp the whole XML file into memory, but it's not working - the script still uses over 1GB of RAM when it runs according to Activity Monitor.\r\n\r\nI think this is because I'm still causing the full root element to be incrementally loaded into memory just in case I try and access it later.\r\n\r\nhttp://effbot.org/elementtree/iterparse.htm says I should use `elem.clear()` as I go. It also says:\r\n\r\n> The above pattern has one drawback; it does not clear the root element, so you will end up with a single element with lots of empty child elements. If your files are huge, rather than just large, this might be a problem. To work around this, you need to get your hands on the root element.\r\n\r\nSo I will try that recipe and see if it helps.", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 490798130, "node_id": "MDU6SXNzdWU0OTA3OTgxMzA=", "number": 7, "title": "users-lookup command for fetching users", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-09-08T19:47:59Z", "updated_at": "2019-09-08T20:32:13Z", "closed_at": "2019-09-08T20:32:13Z", "author_association": "MEMBER", "pull_request": null, "body": "https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-users-lookup\r\n```\r\nhttps://api.twitter.com/1.1/users/lookup.json?user_id=783214,6253282\r\nhttps://api.twitter.com/1.1/users/lookup.json?screen_name=simonw,cleopaws\r\n```\r\nCLI design:\r\n```\r\n$ twitter-to-sqlite users-lookup simonw cleopaws\r\n$ twitter-to-sqlite users-lookup 783214 6253282 --ids\r\n```", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 506276893, "node_id": "MDU6SXNzdWU1MDYyNzY4OTM=", "number": 7, "title": "issue-comments command for importing issue comments", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-10-13T05:23:58Z", "updated_at": "2019-10-14T14:44:12Z", "closed_at": "2019-10-13T05:24:30Z", "author_association": "MEMBER", "pull_request": null, "body": "Using this API: https://developer.github.com/v3/issues/comments/", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 589491711, "node_id": "MDU6SXNzdWU1ODk0OTE3MTE=", "number": 7, "title": "Upgrade to sqlite-utils 2.x", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-03-28T02:24:51Z", "updated_at": "2020-03-28T02:25:03Z", "closed_at": "2020-03-28T02:25:03Z", "author_association": "MEMBER", "pull_request": null, "body": "", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 602585497, "node_id": "MDU6SXNzdWU2MDI1ODU0OTc=", "number": 7, "title": "Integrate image content hashing", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-19T00:36:58Z", "updated_at": "2021-08-26T02:01:01Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "To spot duplicate images (where the file content differs such that the sha256 is no longer a match) it would be useful to calculate and store perceptual hashes of some sort.", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 691265198, "node_id": "MDU6SXNzdWU2OTEyNjUxOTg=", "number": 7, "title": "Mechanism for differentiating between \"by me\" and \"liked by me\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-09-02T17:44:37Z", "updated_at": "2020-09-02T21:06:28Z", "closed_at": "2020-09-02T21:06:28Z", "author_association": "MEMBER", "pull_request": null, "body": "Some of the content I'm indexing is by me - photos I've taken, tweets I wrote, commits, comments I posted.\r\n\r\nSome of it is stuff that I've \"liked\" or \"bookmarked\" in some way - favourited tweets, Pocket articles, starred GitHub repos.\r\n\r\nIt woud be useful to be able to differentiate between the two.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 743297582, "node_id": "MDU6SXNzdWU3NDMyOTc1ODI=", "number": 7, "title": "evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace'", "user": {"value": 42387931, "label": "martinvanwieringen"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-11-15T16:57:28Z", "updated_at": "2021-02-11T22:13:17Z", "closed_at": "2021-02-11T22:13:17Z", "author_association": "NONE", "pull_request": null, "body": "running evernote-to-sqlite 0.2 on windows 10. Command:\r\n\r\nevernote-to-sqlite enex evernote.db MyNotes.enex\r\n\r\nI get the followinng error:\r\n\r\nFile \"C:\\Users\\marti\\AppData\\Roaming\\Python\\Python38\\site-packages\\evernote_to_sqlite\\utils.py\", line 46, in save_note\r\n note_id = db[\"notes\"].insert(row, hash_id=\"id\", replace=True, alter=True).last_pk\r\nTypeError: insert() got an unexpected keyword argument 'replace'\r\n\r\nRemoving replace=True,\r\n\r\nLeads to below error:\r\n\r\n note_id = db[\"notes\"].insert(row, hash_id=\"id\", alter=True).last_pk\r\n File \"C:\\Users\\marti\\AppData\\Roaming\\Python\\Python38\\site-packages\\sqlite_utils\\db.py\", line 924, in insert\r\n return self.insert_all(\r\n File \"C:\\Users\\marti\\AppData\\Roaming\\Python\\Python38\\site-packages\\sqlite_utils\\db.py\", line 1046, in insert_all\r\n result = self.db.conn.execute(sql, values)\r\nsqlite3.IntegrityError: UNIQUE constraint failed: notes.id", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 750141615, "node_id": "MDExOlB1bGxSZXF1ZXN0NTI2ODQ3ODIz", "number": 7, "title": "Fixed conflicting CLI flags", "user": {"value": 8944, "label": "tlockney"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-11-24T23:25:12Z", "updated_at": "2022-08-21T21:11:56Z", "closed_at": "2022-08-21T21:11:56Z", "author_association": "CONTRIBUTOR", "pull_request": "dogsheep/pocket-to-sqlite/pulls/7", "body": "The `-a` used for the auth credentials and the shortened form of the `--all` flags were in conflict on the `fetch` command. To be consistent with other `-to-sqlite` libraries in the Dogsheep ecosystem, I removed the shortened form of the `--all` flag.", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 925384329, "node_id": "MDExOlB1bGxSZXF1ZXN0NjczODcyOTc0", "number": 7, "title": "Add instagram-to-sqlite", "user": {"value": 36654812, "label": "gavindsouza"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-06-19T12:26:16Z", "updated_at": "2021-07-28T07:58:59Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/dogsheep.github.io/pulls/7", "body": "The tool covers only chat imports at the time of opening this PR but I'm planning to import everything else that I feel inquisitive about\r\n\r\nref: https://github.com/gavindsouza/instagram-to-sqlite", "repo": {"value": 214746582, "label": "dogsheep.github.io"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 930946817, "node_id": "MDU6SXNzdWU5MzA5NDY4MTc=", "number": 7, "title": "KeyError: 'accuracy' when processing Location History", "user": {"value": 403152, "label": "davidwilemski"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-06-27T14:39:43Z", "updated_at": "2021-06-27T14:39:43Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I'm new to both the dogsheep tools and datasette but have been experimenting a bit the last few days and these are really cool tools!\r\n\r\nI encountered a problem running my Google location history through this tool running the latest release in a docker container:\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/google-takeout-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/cli.py\", line 49, in my_activity\r\n utils.save_location_history(db, zf)\r\n File \"/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py\", line 27, in save_location_history\r\n db[\"location_history\"].upsert_all(\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1105, in upsert_all\r\n return self.insert_all(\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 990, in insert_all\r\n chunk = list(chunk)\r\n File \"/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py\", line 33, in \r\n \"accuracy\": row[\"accuracy\"],\r\nKeyError: 'accuracy'\r\n```\r\n\r\nIt looks like the tool assumes the `accuracy` key will be in every location history entry. \r\n\r\nMy first attempt at a local patch to get myself going was to convert accessing the `accuracy` key to a `.get` instead to hopefully make the row nullable but I wasn't quite sure what `sqlite_utils` would do there. That did work in that the import happened and so I was going to propose a patch that made that change but in updating the existing test to include an entry with a missing accuracy entry, I noticed the expected type of the field appeared to be changing to a string in the test (and from a quick scan through the sqlite_utils code, probably TEXT in the database). Given this change in column type, it seemed that opening an issue first before proposing a fix seemed warranted. It seems the schema would need to be explicitly specified if you wanted a nullable integer column.\r\n\r\nNow that I've done a successful import run using my initial fix of calling `.get` on the row dict, I can see with datasette that I only have 7 data points (out of ~250k) that have a null accuracy column. They are all from 2011-2012 in an import that includes points spanning ~2010-2016 so perhaps another approach might be to filter those entries out during import if it really is that infrequent?\r\n\r\nI'm happy to provide a PR for a fix but figured I'd ask about which direction is preferred first.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}