{"id": 689809225, "node_id": "MDU6SXNzdWU2ODk4MDkyMjU=", "number": 2, "title": "Apply porter stemming", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-01T04:57:55Z", "updated_at": "2020-09-01T20:42:00Z", "closed_at": "2020-09-01T20:40:24Z", "author_association": "MEMBER", "pull_request": null, "body": "This can be on by default. You can turn it off for a table in the config file using `stemming: none` - or maybe `tokenize: none` to match the terminology used by SQLite and `sqlite-utils`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#enabling-full-text-search", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 769376447, "node_id": "MDU6SXNzdWU3NjkzNzY0NDc=", "number": 2, "title": "killed by oomkiller on large location-history", "user": {"value": 231498, "label": "khimaros"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-12-17T00:32:24Z", "updated_at": "2020-12-17T00:48:32Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "memory seems to grow unbounded and is oom-killed after about 20GB memory usage.\r\n\r\nthis is happening while loading a ~1GB uncompressed location history.", "repo": {"value": 206649770, "label": "google-takeout-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 493670426, "node_id": "MDU6SXNzdWU0OTM2NzA0MjY=", "number": 3, "title": "Command to fetch all repos belonging to a user or organization", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-09-14T21:54:21Z", "updated_at": "2019-09-17T00:17:53Z", "closed_at": "2019-09-17T00:17:53Z", "author_association": "MEMBER", "pull_request": null, "body": "How about this:\r\n\r\n $ github-to-sqlite repos simonw", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 602533481, "node_id": "MDU6SXNzdWU2MDI1MzM0ODE=", "number": 3, "title": "Import EXIF data into SQLite - lens used, ISO, aperture etc", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": {"value": 5324096, "label": "Apple Photos online and securely browsable"}, "comments": 2, "created_at": "2020-04-18T19:24:31Z", "updated_at": "2021-10-05T12:38:24Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 488835586, "node_id": "MDU6SXNzdWU0ODg4MzU1ODY=", "number": 4, "title": "Command for importing data from a Twitter Export file", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-09-03T21:34:13Z", "updated_at": "2019-10-11T06:45:02Z", "closed_at": "2019-10-11T06:45:02Z", "author_association": "MEMBER", "pull_request": null, "body": "Twitter lets you export all of your data as an archive file: https://twitter.com/settings/your_twitter_data\r\n\r\nA command for importing this data into SQLite would be extremely useful.\r\n\r\n $ twitter-to-sqlite import twitter.db path-to-archive.zip\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 718938508, "node_id": "MDU6SXNzdWU3MTg5Mzg1MDg=", "number": 4, "title": "Configure FTS + add an index on the date columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-10-11T22:14:40Z", "updated_at": "2020-10-11T23:41:29Z", "closed_at": "2020-10-11T23:41:29Z", "author_association": "MEMBER", "pull_request": null, "body": "Sort by date descending is likely the most common way of sorting, so that column should be indexed.\r\n\r\nAlso add FTS configuration for both notes and the OCR column on resources.", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1616429236, "node_id": "I_kwDOJHON9s5gWMC0", "number": 4, "title": "Support incremental updates", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-03-09T05:14:00Z", "updated_at": "2023-03-09T18:20:56Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Running this script can take several hours against a large notes database.\r\n\r\nWould be neat if it could run against just the notes that have been modified since it last ran. Could pull the max `updated` date and then keep on looping until it finds one modified before then.\r\n\r\nProblem is I don't actually know what order it iterates over the notes in.", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 470691622, "node_id": "MDU6SXNzdWU0NzA2OTE2MjI=", "number": 5, "title": "Add progress bar", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-07-20T16:29:07Z", "updated_at": "2019-07-22T03:30:13Z", "closed_at": "2019-07-22T02:49:22Z", "author_association": "MEMBER", "pull_request": null, "body": "Showing a progress bar would be nice, using Click.\r\n\r\nThe easiest way to do this would probably be be to hook it up to the length of the compressed content, and update it as this code pushes more XML bytes through the parser:\r\n\r\nhttps://github.com/dogsheep/healthkit-to-sqlite/blob/d64299765064501f4efdd9a0b21dbdba9ec4287f/healthkit_to_sqlite/utils.py#L6-L10", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 493671014, "node_id": "MDU6SXNzdWU0OTM2NzEwMTQ=", "number": 5, "title": "Add \"incomplete\" boolean to users table for incomplete profiles", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-09-14T22:01:50Z", "updated_at": "2020-03-23T19:23:31Z", "closed_at": "2020-03-23T19:23:30Z", "author_association": "MEMBER", "pull_request": null, "body": "User profiles that are fetched from e.g. stargazers (#4) are incomplete - they have a login but they don't have name, company etc. \r\n\r\nAdd a `incomplete` boolean flag to the `users` table to record this. Then later I can add a `backfill-users` command which loops through and fetches missing data for those incomplete profiles.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 504238461, "node_id": "MDU6SXNzdWU1MDQyMzg0NjE=", "number": 6, "title": "sqlite3.OperationalError: table users has no column named bio", "user": {"value": 1055831, "label": "dazzag24"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-10-08T19:39:52Z", "updated_at": "2019-10-13T05:31:28Z", "closed_at": "2019-10-13T05:30:19Z", "author_association": "NONE", "pull_request": null, "body": "```\r\n$ github-to-sqlite repos github.db\r\n$ github-to-sqlite starred github.db dazzag24\r\n\r\nTraceback (most recent call last):\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/bin/github-to-sqlite\", line 10, in \r\n sys.exit(cli())\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 764, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 717, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 1137, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 956, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py\", line 555, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/cli.py\", line 106, in starred\r\n utils.save_stars(db, user, stars)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py\", line 177, in save_stars\r\n user_id = save_user(db, user)\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py\", line 61, in save_user\r\n return db[\"users\"].upsert(to_save, pk=\"id\").last_pk\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py\", line 1067, in upsert\r\n extracts=extracts,\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py\", line 916, in insert\r\n extracts=extracts,\r\n File \"/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py\", line 1024, in insert_all\r\n result = self.db.conn.execute(sql, values)\r\nsqlite3.OperationalError: table users has no column named bio\r\n\r\n```\r\n\r\n```\r\n$ pipenv graph\r\ngithub-to-sqlite==0.4\r\n - requests [required: Any, installed: 2.22.0]\r\n - certifi [required: >=2017.4.17, installed: 2019.9.11]\r\n - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]\r\n - idna [required: >=2.5,<2.9, installed: 2.8]\r\n - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]\r\n - sqlite-utils [required: ~=1.11, installed: 1.11]\r\n - click [required: Any, installed: 7.0]\r\n - click-default-group [required: Any, installed: 1.2.2]\r\n - click [required: Any, installed: 7.0]\r\n - tabulate [required: Any, installed: 0.8.5]\r\n\r\nPython 3.6.8\r\n```", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 602575575, "node_id": "MDU6SXNzdWU2MDI1NzU1NzU=", "number": 6, "title": "Add progress bar to upload command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-18T23:32:41Z", "updated_at": "2020-04-19T00:15:24Z", "closed_at": "2020-04-19T00:15:24Z", "author_association": "MEMBER", "pull_request": null, "body": "Upload was added in #4 ", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 718949182, "node_id": "MDU6SXNzdWU3MTg5NDkxODI=", "number": 6, "title": "Better handling of OCR data", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-10-11T23:20:52Z", "updated_at": "2020-10-12T00:04:10Z", "closed_at": "2020-10-12T00:04:10Z", "author_association": "MEMBER", "pull_request": null, "body": "> I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought.\r\n\r\n_Originally posted by @simonw in https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706784028_", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1617602868, "node_id": "I_kwDOJHON9s5gaqk0", "number": 6, "title": "Character encoding problem", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-03-09T16:44:34Z", "updated_at": "2023-04-14T15:22:09Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "I ran against a recent note with this in it:\r\n\r\n> Or just \"Actions \u2699\ufe0f \"\r\n\r\nAnd got back:\r\n\r\n\"image\"\r\n\r\n> `Actions \u201a\u00f6\u00f4\u00d4\u220f\u00e8`\r\n\r\nPasting that into https://ftfy.vercel.app/?s=Actions+%E2%80%9A%C3%B6%C3%B4%C3%94%E2%88%8F%C3%A8+ gives this:\r\n\r\n```python\r\ns = 'Actions \u00e2\\x80\\x9a\u00c3\u00b6\u00c3\u00b4\u00c3\\x94\u00e2\\x88\\x8f\u00c3\u00a8'\r\ns = s.encode('latin-1')\r\ns = s.decode('utf-8')\r\ns = s.encode('macroman')\r\ns = s.decode('utf-8')\r\nprint(s)\r\n```\r\n\"image\"\r\n", "repo": {"value": 611552758, "label": "apple-notes-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 602585497, "node_id": "MDU6SXNzdWU2MDI1ODU0OTc=", "number": 7, "title": "Integrate image content hashing", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-19T00:36:58Z", "updated_at": "2021-08-26T02:01:01Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "To spot duplicate images (where the file content differs such that the sha256 is no longer a match) it would be useful to calculate and store perceptual hashes of some sort.", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 605147638, "node_id": "MDU6SXNzdWU2MDUxNDc2Mzg=", "number": 8, "title": "Should I have used MD5 instead of SHA256?", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-23T00:02:08Z", "updated_at": "2020-04-23T00:03:35Z", "closed_at": "2020-04-23T00:03:35Z", "author_association": "MEMBER", "pull_request": null, "body": "https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html\r\n\r\n> Objects created by the PUT Object, POST Object, or Copy operation, or through the AWS Management Console, and are encrypted by SSE-S3 or plaintext, have ETags that are an MD5 digest of their object data.\r\n", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 648245071, "node_id": "MDU6SXNzdWU2NDgyNDUwNzE=", "number": 8, "title": "Error thrown: table photos has no column named hasSticker", "user": {"value": 18504, "label": "harperreed"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-06-30T14:54:37Z", "updated_at": "2020-10-12T20:35:06Z", "closed_at": "2020-10-12T20:25:24Z", "author_association": "NONE", "pull_request": null, "body": "While running `swarm-to-sqlite` it throws an error:\r\n\r\n harper@:~/dogsheep/swarm$ swarm-to-sqlite checkins.db --save=checkins.json\r\n Please provide your Foursquare OAuth token:\r\n Importing 8127 checkins [#################-------------------] 49% 00:01:52\r\n Traceback (most recent call last):\r\n File \"/home/harper/.local/bin/swarm-to-sqlite\", line 11, in \r\n sys.exit(cli())\r\n File \"/home/harper/.local/lib/python3.6/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/cli.py\", line 73, in cli\r\n save_checkin(checkin, db)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/utils.py\", line 94, in save_checkin\r\n photos_table.insert(photo, replace=True)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/db.py\", line 963, in insert\r\n alter = self.value_or_default(\"alter\", alter)\r\n File \"/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/db.py\", line 1142, in insert_all\r\n def upsert_all(\r\n sqlite3.OperationalError: table photos has no column named hasSticker\r\n\r\nWhere should i dig in?", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 267517348, "node_id": "MDU6SXNzdWUyNjc1MTczNDg=", "number": 9, "title": "Initial test suite", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 2, "created_at": "2017-10-23T01:28:46Z", "updated_at": "2017-10-24T05:55:33Z", "closed_at": "2017-10-24T05:55:33Z", "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/9/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 605938063, "node_id": "MDU6SXNzdWU2MDU5MzgwNjM=", "number": 9, "title": "upload command should be resumable, should only upload photos not already uploaded", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-23T23:31:08Z", "updated_at": "2020-04-23T23:39:14Z", "closed_at": "2020-04-23T23:39:14Z", "author_association": "MEMBER", "pull_request": null, "body": "Follow on from #4. ", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 801780625, "node_id": "MDU6SXNzdWU4MDE3ODA2MjU=", "number": 9, "title": "SSL Error", "user": {"value": 12669260, "label": "jfeiwell"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-02-05T02:12:56Z", "updated_at": "2021-02-07T18:45:04Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Here's the error I get when running `pip install pocket-to-sqlite`: \r\n\r\n```\r\n Could not fetch URL https://pypi.python.org/simple/pocket-to-sqlite/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661) - skipping\r\n Could not find a version that satisfies the requirement pocket-to-sqlite (from versions: )\r\nNo matching distribution found for pocket-to-sqlite\r\n```\r\n\r\nDoes this require python 3? ", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 413740684, "node_id": "MDU6SXNzdWU0MTM3NDA2ODQ=", "number": 11, "title": "Detect numpy types when creating tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-02-23T21:09:35Z", "updated_at": "2019-02-24T04:02:20Z", "closed_at": "2019-02-24T04:02:20Z", "author_association": "OWNER", "pull_request": null, "body": "Inspired by #8", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/11/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 743400216, "node_id": "MDU6SXNzdWU3NDM0MDAyMTY=", "number": 11, "title": "Error thrown: sqlite3.OperationalError: table users has no column named lastName", "user": {"value": 61791, "label": "beaugunderson"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-11-16T01:21:18Z", "updated_at": "2021-01-18T04:35:22Z", "closed_at": "2021-01-18T04:35:22Z", "author_association": "NONE", "pull_request": null, "body": "Just installed `swarm-to-sqlite-0.3.2` and tried according to the docs:\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/swarm-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/site-packages/swarm_to_sqlite/cli.py\", line 73, in cli\r\n save_checkin(checkin, db)\r\n File \"/usr/local/lib/python3.9/site-packages/swarm_to_sqlite/utils.py\", line 82, in save_checkin\r\n checkins_table.m2m(\"users\", user, m2m_table=\"likes\", pk=\"id\")\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1914, in m2m\r\n id = other_table.insert(record, pk=pk, replace=True).last_pk\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1647, in insert\r\n return self.insert_all(\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1765, in insert_all\r\n self.insert_chunk(\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1575, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py\", line 200, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.OperationalError: table users has no column named lastName\r\n```", "repo": {"value": 205429375, "label": "swarm-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 792851444, "node_id": "MDU6SXNzdWU3OTI4NTE0NDQ=", "number": 11, "title": "XML parse error", "user": {"value": 3613583, "label": "dskrad"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-24T17:38:54Z", "updated_at": "2021-02-11T21:18:58Z", "closed_at": "2021-02-11T21:18:48Z", "author_association": "NONE", "pull_request": null, "body": "I am on Windows 10 using Windows Subsystem for Linux, Python 3.8. I installed evernote-to-sqlite via pipx (in a venv). I tried using enex files from the latest version of Evernote for Windows (10.6.9 which only lets you export 50 notes at a time) and from Legacy Evernote (6.25.2.9198 which lets you export all your notes at once). The enex file from latest evernote gives this error:\r\n\r\n File \"/usr/lib/python3.8/xml/etree/ElementTree.py\", line 1320, in XML parser.feed(text)\r\n xml.etree.ElementTree.ParseError: XML or text declaration not at start of entity: line 2, column 6\r\n\r\nThe enex file from Legacy Evernote gives this error:\r\n\r\n File \"/home/david/.local/pipx/venvs/evernote-to-sqlite/lib/python3.8/site-packages/evernote_to_sqlite/utils.py\", line 28, in save_note\r\n updated = note.find(\"updated\").text\r\n AttributeError: 'NoneType' object has no attribute 'text'", "repo": {"value": 303218369, "label": "evernote-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1795187493, "node_id": "I_kwDODLZ_YM5rAGMl", "number": 12, "title": "Switch to pyproject.toml", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-07-09T01:06:56Z", "updated_at": "2023-07-09T01:19:43Z", "closed_at": "2023-07-09T01:19:42Z", "author_association": "MEMBER", "pull_request": null, "body": "First of my CLI tools to use https://til.simonwillison.net/python/pyproject", "repo": {"value": 213286752, "label": "pocket-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 607888367, "node_id": "MDU6SXNzdWU2MDc4ODgzNjc=", "number": 13, "title": "Also upload movie files", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-27T22:11:25Z", "updated_at": "2020-04-28T00:39:45Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "The `upload` command currently only handles static images:\r\n\r\nhttps://github.com/dogsheep/photos-to-sqlite/blob/d939455af00e07866686457ee2fcb9b2d1b7194e/photos_to_sqlite/utils.py#L26-L33\r\n\r\nNeed to cover movies taken by my phone and DSLR too.", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 505674949, "node_id": "MDU6SXNzdWU1MDU2NzQ5NDk=", "number": 17, "title": "import command should empty all archive-* tables first", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-10-11T06:58:43Z", "updated_at": "2019-10-11T15:40:08Z", "closed_at": "2019-10-11T15:40:08Z", "author_association": "MEMBER", "pull_request": null, "body": "Can have a CLI option for NOT doing that.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 578883725, "node_id": "MDU6SXNzdWU1Nzg4ODM3MjU=", "number": 17, "title": "Command for importing commits", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-03-10T21:55:12Z", "updated_at": "2020-03-11T02:47:37Z", "closed_at": "2020-03-11T02:47:37Z", "author_association": "MEMBER", "pull_request": null, "body": "Using this API: https://api.github.com/repos/dogsheep/github-to-sqlite/commits", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 694500679, "node_id": "MDU6SXNzdWU2OTQ1MDA2Nzk=", "number": 17, "title": "Rename \"table\" to \"type\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-06T19:34:41Z", "updated_at": "2020-09-09T03:03:22Z", "closed_at": "2020-09-09T03:03:22Z", "author_association": "MEMBER", "pull_request": null, "body": "I think \"table\" is the wrong name for the concept I'm using it for here.\r\n\r\nTwo reasons: firstly, `table` is a reserved word in SQLite. More importantly, it turns out there's not a direct mapping from tables to types of search result. In particular, for GitHub I ended up having two different \"tables\" of repositories - one for repos created by me, another for repos that I have starred.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413871266, "node_id": "MDU6SXNzdWU0MTM4NzEyNjY=", "number": 18, "title": ".insert/.upsert/.insert_all/.upsert_all should add missing columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 4348046, "label": "1.0"}, "comments": 2, "created_at": "2019-02-24T21:36:11Z", "updated_at": "2019-05-25T00:42:11Z", "closed_at": "2019-05-25T00:42:11Z", "author_association": "OWNER", "pull_request": null, "body": "This is a larger change, but it would be incredibly useful: if you attempt to insert or update a document with a field that does not currently exist in the underlying table, sqlite-utils should add the appropriate column for you.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/18/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695553522, "node_id": "MDU6SXNzdWU2OTU1NTM1MjI=", "number": 18, "title": "Deleted records stay in the search index", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-08T05:14:23Z", "updated_at": "2020-09-08T05:15:51Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Here's why: https://github.com/dogsheep/dogsheep-beta/blob/24f7898d41a39218058f174c75ba62f7c0fcfff6/dogsheep_beta/utils.py#L44-L53\r\n\r\nThat should probably do `DELETE FROM index1.search_index WHERE [table] = ?` first.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 585850715, "node_id": "MDU6SXNzdWU1ODU4NTA3MTU=", "number": 19, "title": "Enable full-text search for more stuff (like commits, issues and issue_comments)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5225818, "label": "1.0"}, "comments": 2, "created_at": "2020-03-23T00:19:56Z", "updated_at": "2020-03-23T19:06:39Z", "closed_at": "2020-03-23T19:06:39Z", "author_association": "MEMBER", "pull_request": null, "body": "Currently FTS is only enabled for repos and releases.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695556681, "node_id": "MDU6SXNzdWU2OTU1NTY2ODE=", "number": 19, "title": "Figure out incremental re-indexing", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-08T05:23:31Z", "updated_at": "2020-09-08T05:27:07Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "As tables get bigger reindexing everything on a schedule (essentially recreating the entire index from scratch) will start to become a performance bottleneck.", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 975166271, "node_id": "MDU6SXNzdWU5NzUxNjYyNzE=", "number": 20, "title": "Add index on workout_points.date", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-08-20T01:08:04Z", "updated_at": "2021-08-20T01:12:48Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "Sorting that by date makes sense for seeing most recent points, and my DB has 2.5m points in so it's an expensive sort!", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 586561727, "node_id": "MDU6SXNzdWU1ODY1NjE3Mjc=", "number": 21, "title": "Turn GitHub API errors into exceptions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5225818, "label": "1.0"}, "comments": 2, "created_at": "2020-03-23T22:37:24Z", "updated_at": "2020-03-23T23:48:23Z", "closed_at": "2020-03-23T23:48:22Z", "author_association": "MEMBER", "pull_request": null, "body": "This would have really helped in debugging the mess in #13. Running with this `auth.json` is a useful demo:\r\n```json\r\n{\"github_personal_token\": \"\"}\r\n```", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 448395665, "node_id": "MDU6SXNzdWU0NDgzOTU2NjU=", "number": 22, "title": "Release notes for 1.0", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 4348046, "label": "1.0"}, "comments": 2, "created_at": "2019-05-25T00:58:03Z", "updated_at": "2019-05-25T01:18:27Z", "closed_at": "2019-05-25T01:06:52Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/compare/0.14...251e473", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/22/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 508190730, "node_id": "MDU6SXNzdWU1MDgxOTA3MzA=", "number": 23, "title": "Extremely simple migration system", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-10-17T02:13:57Z", "updated_at": "2019-10-17T16:57:17Z", "closed_at": "2019-10-17T16:57:17Z", "author_association": "MEMBER", "pull_request": null, "body": "Needed for #12. This is going to be an incredibly simple version of the Django migration system.\r\n\r\n* A `migrations` table, keeping track of which migrations were applied (and when)\r\n* A `migrate()` function which applies any pending migrations\r\n* A `MIGRATIONS` constant which is a list of functions to be applied\r\n\r\nThe function names will be detected and used as the names of the migrations.\r\n\r\nEvery time you run the CLI tool it will call the `migrate()` function before doing anything else.\r\n\r\nNeeds to take into account that there might be no tables at all. As such, migration functions should sanity check that the tables they are going to work on actually exist.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 267828746, "node_id": "MDU6SXNzdWUyNjc4Mjg3NDY=", "number": 24, "title": "Implement full URL design", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 2, "created_at": "2017-10-23T21:49:05Z", "updated_at": "2017-10-24T14:12:00Z", "closed_at": "2017-10-24T14:12:00Z", "author_association": "OWNER", "pull_request": null, "body": "Full URL design:\r\n\r\n /database-name\r\n /database-name.json\r\n /database-name-7sha256\r\n /database-name-7sha256.json\r\n /database-name/table-name\r\n /database-name/table-name.json\r\n /database-name-7sha256/table-name\r\n /database-name-7sha256/table-name.json\r\n /database-name-7sha256/table-name/compound-pk\r\n /database-name-7sha256/table-name/compound-pk.json\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/24/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1515883470, "node_id": "I_kwDOC8tyDs5aWovO", "number": 24, "title": "DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 ", "user": {"value": 6231413, "label": "mmngreco"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-01-01T23:00:38Z", "updated_at": "2023-03-30T10:17:31Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi @simonw \r\n\r\nI hope you find this issue ok, the idea is provide some documentation to other users like me about how to solve this problem and save some time.\r\n\r\nFollowing the instructions from the `README.md` I've faced this error:\r\n\r\n```bash\r\n(venv) mgreco@pop-os apple-health master* (23:44|0s)\r\n$ healthkit-to-sqlite apple_health_export/export.xml healthkit.db --xml\r\nImporting from HealthKit [------------------------------------] 0%\r\nTraceback (most recent call last):\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/bin/healthkit-to-sqlite\", line 33, in \r\n sys.exit(load_entry_point('healthkit-to-sqlite', 'console_scripts', 'healthkit-to-sqlite')())\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 1130, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 1055, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py\", line 760, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/cli.py\", line 57, in cli\r\n convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)\r\n File \"/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/utils.py\", line 25, in convert_xml_to_sqlite\r\n for tag, el in find_all_tags(\r\n File \"/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/utils.py\", line 12, in find_all_tags\r\n for event, el in parser.read_events():\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/xml/etree/ElementTree.py\", line 1324, in read_events\r\n raise event\r\n File \"/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/xml/etree/ElementTree.py\", line 1296, in feed\r\n self._parser.feed(data)\r\nxml.etree.ElementTree.ParseError: syntax error: line 156, column 0\r\n```\r\n\r\nSo, after debugging and searching on internet I found this useful link: https://discussions.apple.com/thread/254202523 (etresoft, the real hero). Which basically says that the xml given by the health app (healthkit version 12) has some bugs but fortunately, they can be solved with a couple of commads:\r\n\r\n1. Uncompress the zip and move the new folder where `export.xml` is.\r\n1. Create a `patch.txt` with the following content\r\n\r\n ```diff\r\n --- export.xml\t2022-09-18 15:17:09.000000000 -0400\r\n +++ export-fixed.xml\t2022-09-18 16:37:08.000000000 -0400\r\n @@ -15,6 +15,7 @@\r\n HKCharacteristicTypeIdentifierBiologicalSex CDATA #REQUIRED\r\n HKCharacteristicTypeIdentifierBloodType CDATA #REQUIRED\r\n HKCharacteristicTypeIdentifierFitzpatrickSkinType CDATA #REQUIRED\r\n + HKCharacteristicTypeIdentifierCardioFitnessMedicationsUse CDATA #IMPLIED\r\n >\r\n \r\n \r\n -\r\n +\r\n \r\n -\r\n +\r\n \r\n \r\n \r\n \r\n \r\n - device CDATA #IMPLIED\r\n -\r\n -\r\n ->\r\n ]>\r\n \r\n \r\n ```\r\n1. Apply the path with the command: `patch < patch.txt`\r\n1. Fix endDates with the command `sed 's/startDate/endDate/2' export.xml > export-fixed.xml`\r\n1. Try again `healthkit-to-sqlite export-fixed.xml healthkit.db --xml`", "repo": {"value": 197882382, "label": "healthkit-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 267857622, "node_id": "MDU6SXNzdWUyNjc4NTc2MjI=", "number": 25, "title": "Endpoint that returns SQL ready to be piped into DB", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-10-24T00:19:26Z", "updated_at": "2017-11-15T05:11:12Z", "closed_at": "2017-11-15T05:11:11Z", "author_association": "OWNER", "pull_request": null, "body": "It would be cool if I could figure out a way to generate both the create table statements and the inserts for an individual table or the entire database and then stream them down to the client.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/25/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 508578780, "node_id": "MDU6SXNzdWU1MDg1Nzg3ODA=", "number": 25, "title": "Ensure migrations don't accidentally create foreign key twice", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-10-17T16:08:50Z", "updated_at": "2019-10-17T16:56:47Z", "closed_at": "2019-10-17T16:56:47Z", "author_association": "MEMBER", "pull_request": null, "body": "Is it possible for these lines to run against a database table that already has these foreign keys?\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/c9295233f219c446fa2085cace987067488a31b9/twitter_to_sqlite/migrations.py#L21-L22", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 704685890, "node_id": "MDU6SXNzdWU3MDQ2ODU4OTA=", "number": 25, "title": "template_debug mechanism", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-18T22:11:09Z", "updated_at": "2020-09-18T22:12:21Z", "closed_at": "2020-09-18T22:12:03Z", "author_association": "MEMBER", "pull_request": null, "body": "> I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible.\r\n_Originally posted by @simonw in https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584_", "repo": {"value": 197431109, "label": "dogsheep-beta"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 601271612, "node_id": "MDU6SXNzdWU2MDEyNzE2MTI=", "number": 26, "title": "Topics are missing from repositories", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-16T17:36:32Z", "updated_at": "2020-04-16T17:41:11Z", "closed_at": "2020-04-16T17:41:11Z", "author_association": "MEMBER", "pull_request": null, "body": "I'm sure this used to work, but right now repositories are fetched without their topics.\r\n\r\nhttps://developer.github.com/v3/repos/ says you need to send a custom `Accept` header of `application/vnd.github.mercy-preview+json` to get topics.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 518739697, "node_id": "MDU6SXNzdWU1MTg3Mzk2OTc=", "number": 30, "title": "`followers` fails because `transform_user` is called twice", "user": {"value": 21148, "label": "jacobian"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-11-06T20:44:52Z", "updated_at": "2019-11-09T20:15:28Z", "closed_at": "2019-11-09T19:55:52Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Trying to run `twitter-to-sqlite followers` errors out:\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/bin/twitter-to-sqlite\", line 10, in \r\n sys.exit(cli())\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py\", line 764, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py\", line 717, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py\", line 1137, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py\", line 956, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py\", line 555, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/cli.py\", line 130, in followers\r\n go(bar.update)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/cli.py\", line 116, in go\r\n utils.save_users(db, [profile])\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/utils.py\", line 302, in save_users\r\n transform_user(user)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/utils.py\", line 181, in transform_user\r\n user[\"created_at\"] = parser.parse(user[\"created_at\"])\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py\", line 1374, in parse\r\n return DEFAULTPARSER.parse(timestr, **kwargs)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py\", line 646, in parse\r\n res, skipped_tokens = self._parse(timestr, **kwargs)\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py\", line 725, in _parse\r\n l = _timelex.split(timestr) # Splits the timestr into tokens\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py\", line 207, in split\r\n return list(cls(s))\r\n File \"/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py\", line 76, in __init__\r\n '{itype}'.format(itype=instream.__class__.__name__))\r\nTypeError: Parser must be a string or character stream, not datetime\r\n```\r\n\r\nThis appears to be because https://github.com/dogsheep/twitter-to-sqlite/blob/master/twitter_to_sqlite/cli.py#L111 calls `transform_user`, and then https://github.com/dogsheep/twitter-to-sqlite/blob/master/twitter_to_sqlite/cli.py#L116 calls `transform_user` again, which fails because the user is already transformed.\r\n\r\nI was able to work around this by commenting out https://github.com/dogsheep/twitter-to-sqlite/blob/master/twitter_to_sqlite/cli.py#L116. \r\n\r\nShall I work up a patch for that, or is there a better approach?", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 603618244, "node_id": "MDU6SXNzdWU2MDM2MTgyNDQ=", "number": 30, "title": "Issues milestone column is the wrong type", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-21T00:24:34Z", "updated_at": "2020-04-21T00:45:23Z", "closed_at": "2020-04-21T00:36:22Z", "author_association": "MEMBER", "pull_request": null, "body": "https://github-to-sqlite.dogsheep.net/github/issues?milestone=2857392\r\n\r\n![2A4C1185-2434-4F29-9EA0-3246E2F03F77](https://user-images.githubusercontent.com/9599/79811760-b7e08b00-832b-11ea-9ad7-684a6ae097a6.jpeg)\r\n\r\nIt is TEXT when it should be an INTEGER - which is why the foreign key label is not correctly displayed.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 520508502, "node_id": "MDU6SXNzdWU1MjA1MDg1MDI=", "number": 31, "title": "\"friends\" command (similar to \"followers\")", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-11-09T20:20:20Z", "updated_at": "2022-09-20T05:05:03Z", "closed_at": "2020-02-07T07:03:28Z", "author_association": "MEMBER", "pull_request": null, "body": "Current list of commands:\r\n```\r\n followers Save followers for specified user (defaults to...\r\n followers-ids Populate followers table with IDs of account followers\r\n friends-ids Populate followers table with IDs of account friends\r\n```\r\nObvious omission here is `friends`, which would be powered by `https://api.twitter.com/1.1/friends/list.json`: https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-friends-list", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 462423839, "node_id": "MDU6SXNzdWU0NjI0MjM4Mzk=", "number": 33, "title": "index_foreign_keys / index-foreign-keys utilities", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-06-30T16:42:03Z", "updated_at": "2019-06-30T23:54:11Z", "closed_at": "2019-06-30T23:50:55Z", "author_association": "OWNER", "pull_request": null, "body": "Sometimes it's good to have indices on all columns that are foreign keys, to allow for efficient reverse lookups.\r\n\r\nThis would be a useful utility:\r\n\r\n $ sqlite-utils index-foreign-keys database.db\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/33/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 585266763, "node_id": "MDU6SXNzdWU1ODUyNjY3NjM=", "number": 34, "title": "IndexError running user-timeline command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-03-20T18:54:08Z", "updated_at": "2020-03-20T19:20:52Z", "closed_at": "2020-03-20T19:20:37Z", "author_association": "MEMBER", "pull_request": null, "body": "```\r\n$ twitter-to-sqlite user-timeline data.db --screen_name Allen_Joines\r\nTraceback (most recent call last):\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite\", line 11, in \r\n load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')()\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 764, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 717, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 1137, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 956, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py\", line 555, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py\", line 256, in user_timeline\r\n utils.save_tweets(db, chunk)\r\n File \"/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py\", line 289, in save_tweets\r\n db[\"users\"].upsert(user, pk=\"id\", alter=True)\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1128, in upsert\r\n conversions=conversions,\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1157, in upsert_all\r\n upsert=True,\r\n File \"/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py\", line 1096, in insert_all\r\n row = list(self.rows_where(\"rowid = ?\", [self.last_rowid]))[0]\r\nIndexError: list index out of range\r\n```", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 462430920, "node_id": "MDU6SXNzdWU0NjI0MzA5MjA=", "number": 35, "title": "table.update(...) method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-06-30T18:06:15Z", "updated_at": "2019-07-28T15:43:52Z", "closed_at": "2019-07-28T15:43:52Z", "author_association": "OWNER", "pull_request": null, "body": "Spun off from #23 - this method will allow a user to update a specific row.\r\n\r\nCurrently the only way to do that it is to call `.upsert({full record})` with the primary key field matching an existing record - but this does not support partial updates.\r\n```python\r\ndb[\"events\"].update(3, {\"name\": \"Renamed\"})\r\n```\r\nThis method only works on an existing table, so there's no need for a `pk=\"id\"` specifier - it can detect the primary key by looking at the table.\r\n\r\nIf the primary key is compound the first argument can be a tuple:\r\n```python\r\ndb[\"events_venues\"].update((3, 2), {\"custom_label\": \"Label\"})\r\n```\r\nThe method can be called without the second dictionary argument. Doing this selects the row specified by the primary key (throwing an error if it does not exist) and remembers it so that chained operations can be carried out - see proposal in https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507055345\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/35/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 610842926, "node_id": "MDU6SXNzdWU2MTA4NDI5MjY=", "number": 36, "title": "Add view for better display of dependent repos", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-01T16:33:44Z", "updated_at": "2020-05-02T16:50:31Z", "closed_at": "2020-05-02T16:30:11Z", "author_association": "MEMBER", "pull_request": null, "body": "```sql\r\nselect\r\n repos.full_name as repo,\r\n 'https://github.com/' || repos2.full_name as dependent,\r\n repos2.created_at as dependent_repo_created,\r\n repos2.updated_at as dependent_repo_updated,\r\n repos2.stargazers_count as dependent_repo_stars,\r\n repos2.watchers_count as dependent_repo_watchers\r\nfrom\r\n dependents\r\n join repos as repos2 on dependents.dependent = repos2.id\r\n join repos on dependents.repo = repos.id\r\norder by\r\n repos2.created_at desc\r\n```\r\nhttps://dogsheep.simonwillison.net/github?sql=select%0D%0A++repos.full_name+as+repo%2C%0D%0A++%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos2.full_name+as+dependent%2C%0D%0A++repos2.created_at+as+dependent_repo_created%2C%0D%0A++repos2.updated_at+as+dependent_repo_updated%2C%0D%0A++repos2.stargazers_count+as+dependent_repo_stars%2C%0D%0A++repos2.watchers_count+as+dependent_repo_watchers%0D%0Afrom%0D%0A++dependents%0D%0A++join+repos+as+repos2+on+dependents.dependent+%3D+repos2.id%0D%0A++join+repos+on+dependents.repo+%3D+repos.id%0D%0Aorder+by%0D%0A++repos2.created_at+desc", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 268453968, "node_id": "MDU6SXNzdWUyNjg0NTM5Njg=", "number": 37, "title": "Ability to serialize massive JSON without blocking event loop", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-10-25T15:58:03Z", "updated_at": "2020-05-30T17:29:20Z", "closed_at": "2020-05-30T17:29:20Z", "author_association": "OWNER", "pull_request": null, "body": "We run the risk of someone attempting a select statement that returns thousands of rows and hence takes several seconds just to JSON encode the response, effectively blocking the event loop and pausing all other traffic.\r\n\r\nThe Twisted community have a solution for this, can we adapt that in some way? http://as.ynchrono.us/2010/06/asynchronous-json_18.html?m=1", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/37/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 467862459, "node_id": "MDExOlB1bGxSZXF1ZXN0Mjk3NDEyNDY0", "number": 38, "title": "table.update() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-07-14T17:03:49Z", "updated_at": "2019-07-28T15:43:51Z", "closed_at": "2019-07-28T15:43:51Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/38", "body": "Refs #35\r\n\r\nStill to do:\r\n\r\n- [x] Unit tests\r\n- [x] Switch to using `.get()`\r\n- [x] Better exceptions, plus unit tests for what happens if pk does not exist\r\n- [x] Documentation\r\n- [x] Ensure compound primary keys work properly\r\n- [x] `alter=True` support", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/38/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1827427757, "node_id": "PR_kwDOD079W85WtUKG", "number": 38, "title": "photos-to-sql not found?", "user": {"value": 319473, "label": "coldclimate"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-07-29T09:59:42Z", "updated_at": "2023-07-29T10:01:27Z", "closed_at": "2023-07-29T10:01:23Z", "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/dogsheep-photos/pulls/38", "body": "I wonder if `photos-to-sql` is an old name for `dogsheep-photos`, because I can't find it anywhere.\r\n\r\nI can't actually get this command to work (`sqlite3.OperationalError: no such table: attached.ZGENERICASSET` thrown) but I don't think that's related", "repo": {"value": 256834907, "label": "dogsheep-photos"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/dogsheep-photos/issues/38/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 637899539, "node_id": "MDU6SXNzdWU2Mzc4OTk1Mzk=", "number": 40, "title": "Demo deploy is broken", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-06-12T17:20:17Z", "updated_at": "2020-06-12T18:06:48Z", "closed_at": "2020-06-12T18:06:48Z", "author_association": "MEMBER", "pull_request": null, "body": "https://github.com/dogsheep/github-to-sqlite/runs/766180404?check_suite_focus=true\r\n\r\n```\r\nThe following NEW packages will be installed:\r\n sqlite3\r\n0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.\r\nNeed to get 752 kB of archives.\r\nAfter this operation, 2482 kB of additional disk space will be used.\r\nIgn:1 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3\r\nErr:1 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3\r\n 404 Not Found [IP: 52.177.174.250 80]\r\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/s/sqlite3/sqlite3_3.22.0-1ubuntu0.3_amd64.deb 404 Not Found [IP: 52.177.174.250 80]\r\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\r\n##[error]Process completed with exit code 100.\r\n```", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 471628483, "node_id": "MDU6SXNzdWU0NzE2Mjg0ODM=", "number": 44, "title": "Utilities for building lookup tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-07-23T10:59:58Z", "updated_at": "2019-07-23T13:07:01Z", "closed_at": "2019-07-23T13:07:01Z", "author_association": "OWNER", "pull_request": null, "body": "While building https://github.com/dogsheep/healthkit-to-sqlite I found a need for a neat mechanism for easily building lookup tables - tables where each unique value in a column is replaced by a foreign key to a separate table.\r\n\r\ncsvs-to-sqlite currently creates those with its \"extract\" mechanism - but that's written as custom code against Pandas. I'd like to eventually replace Pandas with sqlite-utils there.\r\n\r\nSee also #42 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/44/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 471818939, "node_id": "MDU6SXNzdWU0NzE4MTg5Mzk=", "number": 48, "title": "Jupyter notebook demo of the library, launchable on Binder", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-07-23T17:05:05Z", "updated_at": "2022-01-26T02:08:46Z", "closed_at": "2022-01-26T02:08:39Z", "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/48/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 681228542, "node_id": "MDExOlB1bGxSZXF1ZXN0NDY5NjUxNzMy", "number": 48, "title": "Add pull requests", "user": {"value": 755825, "label": "adamjonas"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-18T17:58:44Z", "updated_at": "2020-11-29T23:51:09Z", "closed_at": "2020-11-29T23:51:09Z", "author_association": "CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/48", "body": "ref #46 \r\n\r\nIssues don't have merge information on them, which means that PRs need to be pulled separately.\r\n\r\nDid my best to mimic the API of issues.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 272694136, "node_id": "MDU6SXNzdWUyNzI2OTQxMzY=", "number": 50, "title": "Unit tests against application itself", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 2, "created_at": "2017-11-09T19:31:49Z", "updated_at": "2017-11-11T22:23:22Z", "closed_at": "2017-11-11T22:23:22Z", "author_association": "OWNER", "pull_request": null, "body": "Use Sanic\u2019s testing mechanism. Test should create a temporary SQLite database file on disk by executing sql that is stored in the test themselves.\r\n\r\nFor the moment we can just test the JSON API more thoroughly and just sanity check that the HTML output doesn\u2019t throw any errors.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/50/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 698791218, "node_id": "MDU6SXNzdWU2OTg3OTEyMTg=", "number": 50, "title": "favorites --stop_after=N stops after min(N, 200)", "user": {"value": 370930, "label": "mikepqr"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-11T03:38:14Z", "updated_at": "2020-09-13T05:11:14Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "For any number greater than 200, `favorites --stop_after` stops after getting 200 tweets, e.g.\r\n```\r\n$ twitter-to-sqlite favorites tweets.db --stop_after=300\r\nImporting favorites [####################################] 199\r\n$\r\n```\r\nI don't _think_ this is a limitation of the API (if you omit `--stop_after` you get some very large number, possibly all of them), so I _think_ this is a bug.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 273026602, "node_id": "MDU6SXNzdWUyNzMwMjY2MDI=", "number": 52, "title": "Solution for temporarily uploading DB so it can be built by docker", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-10T18:55:25Z", "updated_at": "2017-12-10T03:02:57Z", "closed_at": "2017-12-10T03:02:57Z", "author_association": "OWNER", "pull_request": null, "body": "For the `datasette publish` command I ideally need a way of uploading the specified DB to somewhere temporary on the internet so that when the Dockerfile is built by the final hosting location it can download that database as part of the build process.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/52/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 273127117, "node_id": "MDU6SXNzdWUyNzMxMjcxMTc=", "number": 55, "title": "Ship first version to PyPI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 2, "created_at": "2017-11-11T07:38:48Z", "updated_at": "2017-11-13T21:19:43Z", "closed_at": "2017-11-13T21:19:43Z", "author_association": "OWNER", "pull_request": null, "body": "Just before doing this, update the Dockerfile template to `pip install datasette` https://github.com/simonw/datasette/blob/65e350ca2a4845c25752a62c16ba58cfe2c14b9b/datasette/utils.py#L125", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/55/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 779211940, "node_id": "MDExOlB1bGxSZXF1ZXN0NTQ5MjA0MDYz", "number": 55, "title": "Fix archive imports", "user": {"value": 21148, "label": "jacobian"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-05T15:54:48Z", "updated_at": "2021-08-20T00:02:49Z", "closed_at": "2021-08-20T00:02:49Z", "author_association": "CONTRIBUTOR", "pull_request": "dogsheep/twitter-to-sqlite/pulls/55", "body": "This fixes the issues discussed in #54", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 487847945, "node_id": "MDExOlB1bGxSZXF1ZXN0MzEzMDA3NDgz", "number": 56, "title": "Escape the table name in populate_fts and search.", "user": {"value": 49260, "label": "amjith"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-09-01T06:29:05Z", "updated_at": "2019-09-02T17:23:21Z", "closed_at": "2019-09-02T17:23:21Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/56", "body": "The table names weren't escaped using double quotes in the populate_fts method. \r\n\r\nReproducible case: \r\n```\r\n>>> import sqlite_utils\r\n>>> db = sqlite_utils.Database(\"abc.db\")\r\n>>> db[\"http://example.com\"].insert_all([\r\n... {\"id\": 1, \"age\": 4, \"name\": \"Cleo\"},\r\n... {\"id\": 2, \"age\": 2, \"name\": \"Pancakes\"}\r\n... ], pk=\"id\")\r\n\r\n>>> db[\"http://example.com\"].enable_fts([\"name\"])\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\n db[\"http://example.com\"].enable_fts([\"name\"])\r\n File \"/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/sqlite_utils/db.py\", l\r\nine 705, in enable_fts\r\n self.populate_fts(columns)\r\n File \"/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/sqlite_utils/db.py\", l\r\nine 715, in populate_fts\r\n self.db.conn.executescript(sql)\r\nsqlite3.OperationalError: unrecognized token: \":\"\r\n>>> \r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/56/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 769150394, "node_id": "MDU6SXNzdWU3NjkxNTAzOTQ=", "number": 58, "title": "Readme HTML has broken internal links", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-12-16T17:58:11Z", "updated_at": "2020-12-16T19:20:14Z", "closed_at": "2020-12-16T19:20:14Z", "author_association": "MEMBER", "pull_request": null, "body": "From https://github.com/simonw/datasette.io/issues/46\r\n```html\r\n
  • Filtering tables
    • \r\n...\r\n

    Filtering tables

    \r\n```\r\nSo this is a bug in GitHub's API, but we need to work around it.", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 491219910, "node_id": "MDU6SXNzdWU0OTEyMTk5MTA=", "number": 61, "title": "importing CSV to SQLite as library", "user": {"value": 17739, "label": "witeshadow"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-09-09T17:12:40Z", "updated_at": "2019-11-04T16:25:01Z", "closed_at": "2019-11-04T16:25:01Z", "author_association": "NONE", "pull_request": null, "body": "CSV can be imported to SQLite when used CLI, but I don't see documentation for when using as library. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/61/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1088816961, "node_id": "I_kwDODEm0Qs5A5gdB", "number": 62, "title": "KeyError: 'created_at' for private accounts?", "user": {"value": 6764957, "label": "swyxio"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-12-26T17:51:51Z", "updated_at": "2022-03-12T02:36:32Z", "closed_at": "2022-02-24T18:10:18Z", "author_association": "NONE", "pull_request": null, "body": "hey Simon!\r\n\r\ni was running `twitter-to-sqlite user-timeline twitter.db` for [my private alt](https://twitter.com/swyxio) and ran into this error:\r\n\r\n
    \r\n\r\n\r\n![image](https://user-images.githubusercontent.com/6764957/147416165-46b69c30-100a-406f-8534-8612b75547ae.png)\r\n\r\n\r\n\r\n\r\n\r\n```bash\r\nTraceback (most recent call last):\r\n File \"/Users/swyx/Work/datasette/env/bin/twitter-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/cli.py\", line 291, in user_timeline\r\n profile = utils.get_profile(db, session, **kwargs)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py\", line 133, in get_profile\r\n save_users(db, [profile])\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py\", line 453, in save_users\r\n transform_user(user)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py\", line 285, in transform_user\r\n user[\"created_at\"] = parser.parse(user[\"created_at\"])\r\nKeyError: 'created_at'\r\n```\r\n\r\n
    \r\n\r\n\r\nthis looks awfully like #37 but it can't be, because i'm authed into my account and obviously i have perms to read my own account. wonder if there's any diagnostic methods i should apply here? just filing an issue for others to find while i investigate.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/62/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 923270900, "node_id": "MDExOlB1bGxSZXF1ZXN0NjcyMDUzODEx", "number": 65, "title": "basic support for events", "user": {"value": 231498, "label": "khimaros"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-06-17T00:51:30Z", "updated_at": "2022-10-03T22:35:03Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/github-to-sqlite/pulls/65", "body": "a quick first pass at implementing the feature requested in https://github.com/dogsheep/github-to-sqlite/issues/64\r\n\r\ntesting instructions:\r\n\r\n```\r\n$ github-to-sqlite events events.db user/khimaros\r\n```\r\n\r\nif the specified user is the authenticated user, it will also include private events.\r\n\r\ncaveat: pagination appears to be broken (i don't see `next` in the response JSON from GitHub)", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 273192789, "node_id": "MDU6SXNzdWUyNzMxOTI3ODk=", "number": 67, "title": "Command that builds a local docker container", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 2, "created_at": "2017-11-12T02:13:29Z", "updated_at": "2017-11-13T16:17:52Z", "closed_at": "2017-11-13T16:17:52Z", "author_association": "OWNER", "pull_request": null, "body": "Be nice to indicate that this isn't just for Now. Shouldn't be too hard either.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/67/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 529376481, "node_id": "MDExOlB1bGxSZXF1ZXN0MzQ2MjY0OTI2", "number": 67, "title": "Run tests against 3.5 too", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-11-27T14:20:35Z", "updated_at": "2019-12-31T01:29:44Z", "closed_at": "2019-12-31T01:29:43Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/67", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/67/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 539204432, "node_id": "MDU6SXNzdWU1MzkyMDQ0MzI=", "number": 70, "title": "Implement ON DELETE and ON UPDATE actions for foreign keys", "user": {"value": 26292069, "label": "LucasElArruda"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-12-17T17:19:10Z", "updated_at": "2020-02-27T04:18:53Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Hi! I did not find any mention on the library about ON DELETE and ON UPDATE actions for foreign keys. Are those expected to be implemented? If not, it would be a nice thing to include!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/70/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 273296178, "node_id": "MDU6SXNzdWUyNzMyOTYxNzg=", "number": 73, "title": "_nocache=1 query string option for use with sort-by-random", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-13T02:57:10Z", "updated_at": "2018-05-28T17:25:15Z", "closed_at": "2018-05-28T17:25:15Z", "author_association": "OWNER", "pull_request": null, "body": "The one place where we wouldn\u2019t want cdching is if we have something which uses sort by random to return random items. We can offer a _nocache=1 querystring argument to support this.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/73/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 557825032, "node_id": "MDU6SXNzdWU1NTc4MjUwMzI=", "number": 77, "title": "Ability to insert data that is transformed by a SQL function", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-01-30T23:45:55Z", "updated_at": "2022-02-05T00:04:25Z", "closed_at": "2020-01-31T00:24:32Z", "author_association": "OWNER", "pull_request": null, "body": "I want to be able to run the equivalent of this SQL insert:\r\n```python\r\n# Convert to \"Well Known Text\" format\r\nwkt = shape(geojson['geometry']).wkt\r\n# Insert and commit the record\r\nconn.execute(\"INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))\", (\r\n \"Wales\", wkt\r\n))\r\nconn.commit()\r\n```\r\nFrom the Datasette SpatiaLite docs: https://datasette.readthedocs.io/en/stable/spatialite.html\r\n\r\nTo do this, I need a way of telling `sqlite-utils` that a specific column should be wrapped in `GeomFromText(?, 4326)`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/77/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1570375808, "node_id": "I_kwDODFdgUs5dmgiA", "number": 79, "title": "Deploy demo job is failing due to rate limit", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-02-03T20:05:01Z", "updated_at": "2023-12-08T14:50:15Z", "closed_at": null, "author_association": "MEMBER", "pull_request": null, "body": "https://github.com/dogsheep/github-to-sqlite/actions/runs/4080058087/jobs/7032116511", "repo": {"value": 207052882, "label": "github-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 273569477, "node_id": "MDU6SXNzdWUyNzM1Njk0Nzc=", "number": 80, "title": "Deploy final versions of fivethirtyeight and parlgov datasets (with view pagination)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2857392, "label": "Ship first public release"}, "comments": 2, "created_at": "2017-11-13T20:37:46Z", "updated_at": "2017-11-13T22:09:46Z", "closed_at": "2017-11-13T22:09:46Z", "author_association": "OWNER", "pull_request": null, "body": "Final versions should be deployed using the first released version of datasette.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/80/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 273595473, "node_id": "MDExOlB1bGxSZXF1ZXN0MTUyMzYwNzQw", "number": 81, "title": ":fire: Removes DS_Store", "user": {"value": 50527, "label": "jefftriplett"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-13T22:07:52Z", "updated_at": "2017-11-14T02:24:54Z", "closed_at": "2017-11-13T22:16:55Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/datasette/pulls/81", "body": "", "repo": {"value": 107914493, "label": "datasette"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/81/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 565837965, "node_id": "MDU6SXNzdWU1NjU4Mzc5NjU=", "number": 87, "title": "Should detect collections.OrderedDict as a regular dictionary", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-02-16T02:06:34Z", "updated_at": "2020-02-16T02:20:59Z", "closed_at": "2020-02-16T02:20:59Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\n File \"...python3.7/site-packages/sqlite_utils/db.py\", line 292, in create_table\r\n column_type=COLUMN_TYPE_MAPPING[column_type],\r\nKeyError: \r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/87/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 589801352, "node_id": "MDExOlB1bGxSZXF1ZXN0Mzk1MjU4Njg3", "number": 96, "title": "Add type conversion for Panda's Timestamp", "user": {"value": 32605365, "label": "b0b5h4rp13"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-03-29T14:13:09Z", "updated_at": "2020-03-31T04:40:49Z", "closed_at": "2020-03-31T04:40:48Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/96", "body": "Add type conversion for Panda's Timestamp, if Panda library is present in system\r\n(thanks for this project, I was about to do the same thing from scratch)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/96/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 598640234, "node_id": "MDU6SXNzdWU1OTg2NDAyMzQ=", "number": 99, "title": ".upsert_all() should maybe error if dictionaries passed to it do not have the same keys", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-04-13T03:02:25Z", "updated_at": "2020-04-13T03:05:20Z", "closed_at": "2020-04-13T03:05:04Z", "author_association": "OWNER", "pull_request": null, "body": "While investigating #98 I stumbled across this:\r\n```\r\n def test_upsert_compound_primary_key(fresh_db):\r\n table = fresh_db[\"table\"]\r\n table.upsert_all(\r\n [\r\n {\"species\": \"dog\", \"id\": 1, \"name\": \"Cleo\", \"age\": 4},\r\n {\"species\": \"cat\", \"id\": 1, \"name\": \"Catbag\"},\r\n ],\r\n pk=(\"species\", \"id\"),\r\n )\r\n table.upsert_all(\r\n [\r\n {\"species\": \"dog\", \"id\": 1, \"age\": 5},\r\n {\"species\": \"dog\", \"id\": 2, \"name\": \"New Dog\", \"age\": 1},\r\n ],\r\n pk=(\"species\", \"id\"),\r\n )\r\n> assert [\r\n {\"species\": \"dog\", \"id\": 1, \"name\": \"Cleo\", \"age\": 5},\r\n {\"species\": \"cat\", \"id\": 1, \"name\": \"Catbag\", \"age\": None},\r\n {\"species\": \"dog\", \"id\": 2, \"name\": \"New Dog\", \"age\": 1},\r\n ] == list(table.rows)\r\nE AssertionError: assert [{'age': 5, '...cies': 'dog'}] == [{'age': 5, '...cies': 'dog'}]\r\nE At index 0 diff: {'species': 'dog', 'id': 1, 'name': 'Cleo', 'age': 5} != {'species': 'dog', 'id': 1, 'name': None, 'age': 5}\r\nE Full diff:\r\nE - [{'age': 5, 'id': 1, 'name': 'Cleo', 'species': 'dog'},\r\nE ? ^^^ --\r\nE + [{'age': 5, 'id': 1, 'name': None, 'species': 'dog'},\r\nE ? ^^^\r\nE {'age': None, 'id': 1, 'name': 'Catbag', 'species': 'cat'},\r\nE {'age': 1, 'id': 2, 'name': 'New Dog', 'species': 'dog'}]\r\n```\r\nIf you run `.upsert_all()` with multiple dictionaries it doesn't quite have the effect you might expect.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/99/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 274160723, "node_id": "MDU6SXNzdWUyNzQxNjA3MjM=", "number": 100, "title": "TemplateAssertionError: no filter named 'tojson'", "user": {"value": 13304454, "label": "coisnepe"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-15T13:43:41Z", "updated_at": "2017-11-16T09:25:10Z", "closed_at": "2017-11-16T00:14:13Z", "author_association": "NONE", "pull_request": null, "body": "A 500 error is raised upon clicking on the name of a table on the homepage, say _http://0.0.0.0:8001/_ to _http://0.0.0.0:8001/test_check-c1f4771/users_ The API part seems to function as intended, though...\r\n\r\n```\r\n2017-11-15 14:33:57 - (sanic)[ERROR]: Traceback (most recent call last):\r\n File \"/usr/local/lib/python3.5/dist-packages/sanic/app.py\", line 503, in handle_request\r\n response = await response\r\n File \"/usr/local/lib/python3.5/dist-packages/datasette/app.py\", line 155, in get\r\n return await self.view_get(request, name, hash, **kwargs)\r\n File \"/usr/local/lib/python3.5/dist-packages/datasette/app.py\", line 219, in view_get\r\n **context,\r\n File \"/usr/local/lib/python3.5/dist-packages/sanic_jinja2/__init__.py\", line 84, in render\r\n return html(self.render_string(template, request, **context))\r\n File \"/usr/local/lib/python3.5/dist-packages/sanic_jinja2/__init__.py\", line 81, in render_string\r\n return self.env.get_template(template).render(**context)\r\n File \"/usr/lib/python3/dist-packages/jinja2/environment.py\", line 812, in get_template\r\n return self._load_template(name, self.make_globals(globals))\r\n File \"/usr/lib/python3/dist-packages/jinja2/environment.py\", line 786, in _load_template\r\n template = self.loader.load(self, name, globals)\r\n File \"/usr/lib/python3/dist-packages/jinja2/loaders.py\", line 125, in load\r\n code = environment.compile(source, name, filename)\r\n File \"/usr/lib/python3/dist-packages/jinja2/environment.py\", line 565, in compile\r\n self.handle_exception(exc_info, source_hint=source_hint)\r\n File \"/usr/lib/python3/dist-packages/jinja2/environment.py\", line 754, in handle_exception\r\n reraise(exc_type, exc_value, tb)\r\n File \"/usr/lib/python3/dist-packages/jinja2/_compat.py\", line 37, in reraise\r\n raise value.with_traceback(tb)\r\n File \"/usr/local/lib/python3.5/dist-packages/datasette/templates/table.html\", line 29, in template\r\n 
    params = {{ query.params|tojson(4) }}
    \r\n File \"/usr/lib/python3/dist-packages/jinja2/environment.py\", line 515, in _generate\r\n return generate(source, self, name, filename, defer_init=defer_init)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 62, in generate\r\n generator.visit(node)\r\n File \"/usr/lib/python3/dist-packages/jinja2/visitor.py\", line 38, in visit\r\n return f(node, *args, **kwargs)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 849, in visit_Template\r\n self.blockvisit(block.body, block_frame)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 492, in blockvisit\r\n self.visit(node, frame)\r\n File \"/usr/lib/python3/dist-packages/jinja2/visitor.py\", line 38, in visit\r\n return f(node, *args, **kwargs)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 1172, in visit_If\r\n self.blockvisit(node.body, if_frame)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 492, in blockvisit\r\n self.visit(node, frame)\r\n File \"/usr/lib/python3/dist-packages/jinja2/visitor.py\", line 38, in visit\r\n return f(node, *args, **kwargs)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 1353, in visit_Output\r\n self.visit(argument, frame)\r\n File \"/usr/lib/python3/dist-packages/jinja2/visitor.py\", line 38, in visit\r\n return f(node, *args, **kwargs)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 1565, in visit_Filter\r\n self.fail('no filter named %r' % node.name, node.lineno)\r\n File \"/usr/lib/python3/dist-packages/jinja2/compiler.py\", line 427, in fail\r\n raise TemplateAssertionError(msg, lineno, self.name, self.filename)\r\njinja2.exceptions.TemplateAssertionError: no filter named 'tojson'\r\n\r\n2017-11-15 14:33:57 - (network)[INFO][127.0.0.1:41316]: GET http://0.0.0.0:8001/test_check-c1f4771/users 500 144\r\n2017-11-15 14:33:57 - (network)[INFO][127.0.0.1:41316]: GET http://0.0.0.0:8001/favicon.ico 200 0\r\n```", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/100/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611222968, "node_id": "MDU6SXNzdWU2MTEyMjI5Njg=", "number": 107, "title": "sqlite-utils create-view CLI command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-02T16:15:13Z", "updated_at": "2020-05-03T15:36:58Z", "closed_at": "2020-05-03T15:36:37Z", "author_association": "OWNER", "pull_request": null, "body": "Can go with #27 - `sqlite-utils create-table`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/107/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611326701, "node_id": "MDU6SXNzdWU2MTEzMjY3MDE=", "number": 108, "title": "Documentation unit tests for CLI commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-03T03:58:42Z", "updated_at": "2020-05-03T04:13:57Z", "closed_at": "2020-05-03T04:13:57Z", "author_association": "OWNER", "pull_request": null, "body": "Have a test that ensures all CLI commands are documented.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/108/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 274578142, "node_id": "MDU6SXNzdWUyNzQ1NzgxNDI=", "number": 110, "title": "Add --load-extension option to datasette for loading extra SQLite extensions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-16T16:26:19Z", "updated_at": "2017-11-16T18:38:30Z", "closed_at": "2017-11-16T16:58:50Z", "author_association": "OWNER", "pull_request": null, "body": "This would allow users with extra SQLite extensions installed (like spatialite) to load them at runtime.\r\n\r\nInspired by this comment: https://github.com/simonw/datasette/issues/46#issuecomment-344810525", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/110/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 615477131, "node_id": "MDU6SXNzdWU2MTU0NzcxMzE=", "number": 111, "title": "sqlite-utils drop-table and drop-view commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-10T21:10:42Z", "updated_at": "2020-05-11T01:58:36Z", "closed_at": "2020-05-11T00:44:26Z", "author_association": "OWNER", "pull_request": null, "body": "Would be useful to be able to drop views and tables from the CLI.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/111/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 621286870, "node_id": "MDU6SXNzdWU2MjEyODY4NzA=", "number": 113, "title": "Syntactic sugar for ATTACH DATABASE", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-19T21:10:00Z", "updated_at": "2021-02-19T05:09:12Z", "closed_at": "2021-02-19T04:56:36Z", "author_association": "OWNER", "pull_request": null, "body": "https://www.sqlite.org/lang_attach.html\r\n\r\nMaybe something like this:\r\n```python\r\ndb.attach(\"other_db\", \"other_db.db\")\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/113/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 644122661, "node_id": "MDU6SXNzdWU2NDQxMjI2NjE=", "number": 116, "title": "Documentation for table.pks introspection property", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-06-23T20:27:24Z", "updated_at": "2020-06-23T21:21:33Z", "closed_at": "2020-06-23T21:03:14Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L535-L540", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/116/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 275135393, "node_id": "MDU6SXNzdWUyNzUxMzUzOTM=", "number": 125, "title": "Plot rows on a map with Leaflet and Leaflet.markercluster", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-19T06:05:05Z", "updated_at": "2018-04-26T15:14:31Z", "closed_at": "2018-04-26T15:14:31Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/Leaflet/Leaflet.markercluster would allow us to paginate-load in an enormous set of rows with latitude/longitude points, e.g. https://australian-dunnies.now.sh/\r\n\r\nHere's a demo of it loading 50,000 markers: https://leaflet.github.io/Leaflet.markercluster/example/marker-clustering-realworld.50000.html - and it looks like it's easy to support progress bars for if we were iteratively loading 1,000 markers at a time using datasette pagination.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/125/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665819048, "node_id": "MDU6SXNzdWU2NjU4MTkwNDg=", "number": 126, "title": "Ability to insert binary data on the CLI using JSON", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-07-26T16:54:14Z", "updated_at": "2020-07-27T04:00:33Z", "closed_at": "2020-07-27T03:59:45Z", "author_association": "OWNER", "pull_request": null, "body": "> I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says \"these columns are base64 encoded, store the decoded data in a BLOB\".\r\n>\r\n> That would solve inserting binary data using JSON too.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/126/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 275135719, "node_id": "MDU6SXNzdWUyNzUxMzU3MTk=", "number": 127, "title": "Filtered tables should show count of all matching rows, if fast enough", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 2919870, "label": "Foreign key edition"}, "comments": 2, "created_at": "2017-11-19T06:13:29Z", "updated_at": "2017-11-24T22:02:01Z", "closed_at": "2017-11-24T22:02:01Z", "author_association": "OWNER", "pull_request": null, "body": "Relates to #86. If you are viewing a filtered page e.g. https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9/bob-ross%2Felements-by-episode?CLOUDS=1 we should show the count of matching rows.\r\n\r\nSince this could be an expensive operation, we will run it with a strict time limit (maybe 50ms). If the time limit is exceeded we will display \"many\" instead, perhaps? Maybe even link to a count(*) query that would get the full 1000ms time limit which the user can click on if they like (that could even Ajax-in the result).", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/127/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 275164558, "node_id": "MDU6SXNzdWUyNzUxNjQ1NTg=", "number": 129, "title": "Hide FTS-created tables by default on the database index page", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-19T14:50:42Z", "updated_at": "2017-11-22T20:22:02Z", "closed_at": "2017-11-22T20:19:04Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite databases that use FTS include a number of automatically generated tables, e.g.:\r\n\r\nhttps://sf-trees-search.now.sh/sf-trees-search-a899b92\r\n\r\n\"sf-trees-search_and_sf-trees-search\"\r\n\r\nOf these, only the `Street_Tree_List` table is actually relevant to the user.\r\n\r\nWe can detect which tables are FTS tables by first finding the virtual tables:\r\n\r\n sqlite> .headers on\r\n sqlite> select * from sqlite_master where rootpage = 0;\r\n type|name|tbl_name|rootpage|sql\r\n table|Search|Search|0|CREATE VIRTUAL TABLE \"Street_Tree_List_fts\" USING FTS4 (\"qAddress\", \"qCaretaker\", \"qSpecies\")\r\n\r\nThen parsing the above to figure out which ones are USING FTS? - then assume that any table which starts with that `Street_Tree_List_fts` prefix was created to support search:\r\n\r\n sqlite> select * from sqlite_master where type='table' and tbl_name like 'Street_Tree_List_fts%';\r\n type|name|tbl_name|rootpage|sql\r\n table|Search_content|Search_content|10355|CREATE TABLE 'Street_Tree_List_fts_content'(docid INTEGER PRIMARY KEY, 'c0qAddress', 'c1qCaretaker', 'c2qSpecies')\r\n table|Search_segments|Search_segments|10356|CREATE TABLE 'Street_Tree_List_fts_segments'(blockid INTEGER PRIMARY KEY, block BLOB)\r\n table|Search_segdir|Search_segdir|10357|CREATE TABLE 'Street_Tree_List_fts_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx))\r\n table|Search_docsize|Search_docsize|10359|CREATE TABLE 'Street_Tree_List_fts_docsize'(docid INTEGER PRIMARY KEY, size BLOB)\r\n table|Search_stat|Search_stat|10360|CREATE TABLE 'Street_Tree_List_fts_stat'(id INTEGER PRIMARY KEY, value BLOB)\r\n\r\nWe won't hide these completely - instead, we'll default the database index view to not showing them with a message that says \"5 hidden tables\" and support ?_hidden=1 to display them.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/129/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 668308777, "node_id": "MDU6SXNzdWU2NjgzMDg3Nzc=", "number": 129, "title": "\"insert-files --sqlar\" for creating SQLite archives", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-07-30T02:28:29Z", "updated_at": "2020-07-30T22:41:01Z", "closed_at": "2020-07-30T22:40:55Z", "author_association": "OWNER", "pull_request": null, "body": "A `--sqlar` option could cause `insert-files` to behave in the same way as SQLite's own sqlar mechanism.\r\n\r\nhttps://www.sqlite.org/sqlar.html and https://sqlite.org/sqlar/doc/trunk/README.md", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/129/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 684118950, "node_id": "MDU6SXNzdWU2ODQxMTg5NTA=", "number": 138, "title": "extracts= doesn't configure foreign keys", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-23T05:21:15Z", "updated_at": "2020-09-24T22:47:01Z", "closed_at": "2020-09-24T22:46:52Z", "author_association": "OWNER", "pull_request": null, "body": "In using `extracts=` for `shapefiles-to-sqlite` in https://github.com/simonw/shapefile-to-sqlite/issues/9 I've run into a couple of pretty serious flaws:\r\n\r\n- The columns in the original table are still `TEXT` even when the foreign key they are supposed to reference is an `INTEGER` - which means Datasette foreign key features don't actually work\r\n- Those foreign key relationships aren't setup automatically - creating them is left as an exercise for the developer", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/138/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 275493851, "node_id": "MDU6SXNzdWUyNzU0OTM4NTE=", "number": 139, "title": "Build a visualization plugin for Vega", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-20T20:47:41Z", "updated_at": "2018-07-10T17:48:18Z", "closed_at": "2018-07-10T17:48:18Z", "author_association": "OWNER", "pull_request": null, "body": "https://vega.github.io/vega/examples/population-pyramid/ for example looks pretty easy to hook up to Datasette.\r\n\r\nDepends on #14 ", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/139/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 275755475, "node_id": "MDU6SXNzdWUyNzU3NTU0NzU=", "number": 140, "title": "Heatmap visualization plugin", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-21T15:34:23Z", "updated_at": "2019-05-13T18:33:51Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Could use https://github.com/scottbedard/svelte-heatmap", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/140/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 688395275, "node_id": "MDU6SXNzdWU2ODgzOTUyNzU=", "number": 144, "title": "Run some tests against numpy", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-28T22:53:00Z", "updated_at": "2020-08-28T22:57:05Z", "closed_at": "2020-08-28T22:57:04Z", "author_association": "OWNER", "pull_request": null, "body": "Accidentally removed in #143:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/d7d3f962861ef32c5ead8f514c8756f5b6f7c4a0/.travis.yml#L18-L19", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/144/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688659182, "node_id": "MDU6SXNzdWU2ODg2NTkxODI=", "number": 145, "title": "Bug when first record contains fewer columns than subsequent records", "user": {"value": 96218, "label": "simonwiles"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-30T05:44:44Z", "updated_at": "2020-09-08T23:21:23Z", "closed_at": "2020-09-08T23:21:23Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "`insert_all()` selects the maximum batch size based on the number of fields in the first record. If the first record has fewer fields than subsequent records (and `alter=True` is passed), this can result in SQL statements with more than the maximum permitted number of host parameters. This situation is perhaps unlikely to occur, but could happen if the first record had, say, 10 columns, such that `batch_size` (based on `SQLITE_MAX_VARIABLE_NUMBER = 999`) would be 99. If the next 98 rows had 11 columns, the resulting SQL statement for the first batch would have `10 * 1 + 11 * 98 = 1088` host parameters (and subsequent batches, if the data were consistent from thereon out, would have `99 * 11 = 1089`).\r\n\r\nI suspect that this bug is masked somewhat by the fact that while:\r\n> [`SQLITE_MAX_VARIABLE_NUMBER`](https://www.sqlite.org/limits.html#max_variable_number) ... defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0.\r\n\r\nit is common that it is increased at compile time. Debian-based systems, for example, seem to ship with a version of sqlite compiled with `SQLITE_MAX_VARIABLE_NUMBER` set to 250,000, and I believe this is the case for homebrew installations too.\r\n\r\nA test for this issue might look like this:\r\n```python\r\ndef test_columns_not_in_first_record_should_not_cause_batch_to_be_too_large(fresh_db):\r\n # sqlite on homebrew and Debian/Ubuntu etc. is typically compiled with\r\n # SQLITE_MAX_VARIABLE_NUMBER set to 250,000, so we need to exceed this value to\r\n # trigger the error on these systems.\r\n THRESHOLD = 250000\r\n extra_columns = 1 + (THRESHOLD - 1) // 99\r\n records = [\r\n {\"c0\": \"first record\"}, # one column in first record -> batch_size = 100\r\n # fill out the batch with 99 records with enough columns to exceed THRESHOLD\r\n *[\r\n dict([(\"c{}\".format(i), j) for i in range(extra_columns)])\r\n for j in range(99)\r\n ]\r\n ]\r\n try:\r\n fresh_db[\"too_many_columns\"].insert_all(records, alter=True)\r\n except sqlite3.OperationalError:\r\n raise\r\n```\r\n\r\nThe best solution, I think, is simply to process all the records when determining columns, column types, and the batch size. In my tests this doesn't seem to be particularly costly at all, and cuts out a lot of complications (including obviating my implementation of #139 at #142). I'll raise a PR for your consideration.\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/145/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 276455748, "node_id": "MDU6SXNzdWUyNzY0NTU3NDg=", "number": 146, "title": "datasette publish gcloud", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-23T18:55:03Z", "updated_at": "2019-06-24T06:48:20Z", "closed_at": "2019-06-24T06:48:20Z", "author_association": "OWNER", "pull_request": null, "body": "See also #103 \r\n\r\nIt looks like you can start a Google Cloud VM with a \"docker container\" option - and the Google Cloud Registry is easy to push containers to. So it would be feasible to have `datasette publish gcloud ...` automatically build a container, push it to GCR, then start a new VM instance with it:\r\n\r\nhttps://cloud.google.com/container-registry/docs/pushing-and-pulling\r\n", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/146/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 276718605, "node_id": "MDU6SXNzdWUyNzY3MTg2MDU=", "number": 151, "title": "Set up a pattern portfolio", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2017-11-25T02:09:49Z", "updated_at": "2020-07-02T00:13:24Z", "closed_at": "2020-05-03T03:13:16Z", "author_association": "OWNER", "pull_request": null, "body": "https://www.slideshare.net/nataliedowne/practical-maintainable-css/75\r\n\r\nThis will be a single page that demonstrates all of the different CSS styles and classes available to Datasette.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/151/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695376054, "node_id": "MDU6SXNzdWU2OTUzNzYwNTQ=", "number": 152, "title": "Turn on recursive_triggers by default", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-07T20:26:36Z", "updated_at": "2020-09-07T21:17:48Z", "closed_at": "2020-09-07T20:45:14Z", "author_association": "OWNER", "pull_request": null, "body": "https://www.sqlite.org/pragma.html#pragma_recursive_triggers says:\r\n\r\n> Prior to SQLite [version 3.6.18](https://www.sqlite.org/releaselog/3_6_18.html) (2009-09-11), recursive triggers were not supported. The behavior of SQLite was always as if this pragma was set to OFF. Support for recursive triggers was added in version 3.6.18 but was initially turned OFF by default, for compatibility. Recursive triggers may be turned on by default in future versions of SQLite.\r\n\r\nSo I think the fix for the complex issue in #149 is to turn on `recursive_triggers` globally by default for `sqlite-utils`.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499924_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/152/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695441530, "node_id": "MDU6SXNzdWU2OTU0NDE1MzA=", "number": 154, "title": "OperationalError: cannot change into wal mode from within a transaction", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-07T23:42:44Z", "updated_at": "2020-09-07T23:47:10Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I'm getting this error when running:\r\n\r\n sqlite-utils enable-wal beta.db\r\n\r\n`OperationalError: cannot change into wal mode from within a transaction`\r\n\r\nI'm worried that maybe that's because of this new code from #152:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/deb2eb013ff85bbc828ebc244a9654f0d9c3139e/sqlite_utils/db.py#L128-L129", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/154/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 696045581, "node_id": "MDU6SXNzdWU2OTYwNDU1ODE=", "number": 155, "title": "rebuild-fts command and table.rebuild_fts() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-08T17:19:26Z", "updated_at": "2020-09-24T20:35:46Z", "closed_at": "2020-09-08T23:16:10Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite.org/forum/forumpost/fa777fff86\r\n\r\n> Easiest thing would be to run a 'rebuild' to rebuild the FTS index from scratch based on the contents of the content table. i.e.\r\n>\r\n> INSERT INTO licenses_fts(licenses_fts) VALUES('rebuild');\r\n>\r\n> https://www.sqlite.org/fts5.html#the_rebuild_command", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/155/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 697179806, "node_id": "MDU6SXNzdWU2OTcxNzk4MDY=", "number": 157, "title": "sqlite-utils add-foreign-keys command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5896742, "label": "2.19"}, "comments": 2, "created_at": "2020-09-09T21:44:30Z", "updated_at": "2020-09-24T20:34:50Z", "closed_at": "2020-09-20T20:14:30Z", "author_association": "OWNER", "pull_request": null, "body": "Like `add-foreign-key` but can do multiple foreign keys at once. Inspired by https://github.com/simonw/calands-datasette/blob/99de39dd80a906f5c1f16724467b0cd55ba4ef36/build.sh which does this:\r\n\r\n```\r\nsqlite-utils add-foreign-key calands.db units_with_maps ACCESS_TYP\r\nsqlite-utils add-foreign-key calands.db units_with_maps AGNCY_NAME\r\nsqlite-utils add-foreign-key calands.db units_with_maps AGNCY_LEV\r\nsqlite-utils add-foreign-key calands.db units_with_maps AGNCY_TYP\r\nsqlite-utils add-foreign-key calands.db units_with_maps LAYER\r\nsqlite-utils add-foreign-key calands.db units_with_maps MNG_AGENCY\r\nsqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_LEV\r\nsqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_TYP\r\nsqlite-utils add-foreign-key calands.db units_with_maps COUNTY\r\nsqlite-utils add-foreign-key calands.db units_with_maps DES_TP\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/157/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 705995722, "node_id": "MDU6SXNzdWU3MDU5OTU3MjI=", "number": 162, "title": "A decorator for registering custom SQL functions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-22T00:18:32Z", "updated_at": "2020-09-22T00:40:44Z", "closed_at": "2020-09-22T00:32:17Z", "author_association": "OWNER", "pull_request": null, "body": "Syntactic sugar for `db.conn.create_function` - it would work something like this:\r\n\r\n```python\r\ndb = sqlite_utils.Database(\"mydb.db\")\r\n\r\n@db.register_function\r\ndef scramble(text):\r\n chars = list(text)\r\n random.shuffle(chars)\r\n return \"\".join(chars)\r\n```\r\nThe decorator would inspect the function to find its name and arity (number of arguments). Having run the above you could then do:\r\n```python\r\ndb.execute(\"select scramble('hello')\").fetchall()\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/162/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}