\r\n{% for photo in sql(\"with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 100\") %}\r\n
![](\"/-/media/photo/{{)
\r\n{% endfor %}\r\n
![\"github-to-sqlite_\u2013_Service_details_\u2013_Cloud_Run_\u2013_datasette_\u2013_Google_Cloud_Platform\"](\"https://user-images.githubusercontent.com/9599/86545887-92086280-bee6-11ea-8787-b3b85a7bb4b9.png\")
\r\n![\"github-to-sqlite_\u2013_Service_details_\u2013_Cloud_Run_\u2013_datasette_\u2013_Google_Cloud_Platform\"](\"https://user-images.githubusercontent.com/9599/86545917-c11ed400-bee6-11ea-9cf8-957911ca9f6a.png\")
\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651159727, "label": "Demo is failing to deploy"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653960989", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41", "id": 653960989, "node_id": "MDEyOklzc3VlQ29tbWVudDY1Mzk2MDk4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-06T00:32:34Z", "updated_at": "2020-07-06T00:32:34Z", "author_association": "MEMBER", "body": "Same error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651159727, "label": "Demo is failing to deploy"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653947916", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41", "id": 653947916, "node_id": "MDEyOklzc3VlQ29tbWVudDY1Mzk0NzkxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-05T22:40:47Z", "updated_at": "2020-07-05T22:40:47Z", "author_association": "MEMBER", "body": "Might be that it's not got enough RAM. I'll try deploying to a larger instance.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651159727, "label": "Demo is failing to deploy"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645599881", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47", "id": 645599881, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NTU5OTg4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-17T20:13:48Z", "updated_at": "2020-06-17T20:13:48Z", "author_association": "MEMBER", "body": "I've now figured out how to compile specific SQLite versions to help replicate this problem: https://github.com/simonw/til/blob/master/sqlite/ld-preload.md\r\n\r\nNext step: replicate the problem!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 639542974, "label": "Fall back to FTS4 if FTS5 is not available"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47", "id": 645512127, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NTUxMjEyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-17T17:24:22Z", "updated_at": "2020-06-17T17:24:22Z", "author_association": "MEMBER", "body": "That means your version of SQLite is old enough that it doesn't support the FTS5 extension.\r\n\r\nCould you share what operating system you're running, and what the output is that you get from running this?\r\n\r\n python -c 'import sqlite3; print(sqlite3.connect(\":memory:\").execute(\"select sqlite_version()\").fetchone()[0])'\r\n\r\nI can teach this tool to fall back on FTS4 if FTS5 isn't available.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 639542974, "label": "Fall back to FTS4 if FTS5 is not available"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643414646", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40", "id": 643414646, "node_id": "MDEyOklzc3VlQ29tbWVudDY0MzQxNDY0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-12T18:06:48Z", "updated_at": "2020-06-12T18:06:48Z", "author_association": "MEMBER", "body": "That fixed it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 637899539, "label": "Demo deploy is broken"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643393506", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40", "id": 643393506, "node_id": "MDEyOklzc3VlQ29tbWVudDY0MzM5MzUwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-12T17:21:14Z", "updated_at": "2020-06-12T17:21:14Z", "author_association": "MEMBER", "body": "I only install SQLite for this:\r\n\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/c0d54e0260468be38152293df5abd775c068495d/.github/workflows/deploy-demo.yml#L77-L78\r\n\r\nI'm going to remove the need to install sqlite3 by making this possible with sqlite-utils: https://github.com/simonw/sqlite-utils/issues/115", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 637899539, "label": "Demo deploy is broken"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633704127", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 633704127, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMzcwNDEyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-25T20:14:22Z", "updated_at": "2020-05-25T20:14:22Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/dogsheep-photos/blob/0.4.1/README.md#serving-photos-locally-with-datasette-media", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633644225", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 633644225, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMzY0NDIyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-25T16:30:44Z", "updated_at": "2020-05-25T16:30:44Z", "author_association": "MEMBER", "body": "I'll add docs on using `datasette-json-html` too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633643921", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 633643921, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMzY0MzkyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-25T16:29:44Z", "updated_at": "2020-05-25T16:29:44Z", "author_association": "MEMBER", "body": "https://github.com/dogsheep/dogsheep-photos/blob/dc43fa8653cb9c7238a36f52239b91d1ec916d5c/README.md#serving-photos-locally-with-datasette-media", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 613006393, "label": "Ability to serve thumbnailed Apple Photo from its place on disk"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633629944", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20", "id": 633629944, "node_id": "MDEyOklzc3VlQ29tbWVudDYzMzYyOTk0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-25T15:47:42Z", "updated_at": "2020-05-25T15:47:42Z", "author_association": "MEMBER", "body": "I'll add a proper section to the README, but for the moment here's how I do this.\r\n\r\nFirst, install `datasette` and the `datasette-media` plugin.\r\n\r\nCreate a `metadata.yaml` file with the following content:\r\n\r\n```yaml\r\nplugins:\r\n datasette-media:\r\n photo:\r\n sql: |-\r\n select path as filepath, 200 as resize_height from apple_photos where uuid = :key\r\n photo-big:\r\n sql: |-\r\n select path as filepath, 1024 as resize_height from apple_photos where uuid = :key\r\n```\r\nNow run `datasette -m metadata.yaml photos.db` - thumbnails will be served at http://127.0.0.1:8001/-/media/photo/F4469918-13F3-43D8-9EC1-734C0E6B60AD and larger sizes of the image at http://127.0.0.1:8001/-/media/photo-big/A8B02C7D-365E-448B-9510-69F80C26304D\r\n\r\nI also made myself two custom pages, one showing recent images and one showing random images.\r\n\r\nTo do this, install the `datasette-template-sql` plugin and then create a `templates/pages` directory and add these files:\r\n\r\n`recent-photos.html`\r\n```html\r\n![\"psi_copy__select_json_object__img_src____https___photos_simonwillison_net_i______photos_sha256___________photos_ext______w_400___as_photo__group_concat_strip_null_chars_groups_content_string________as_words__assets_uuid_0__assets_uuid_1__to\"](\"https://user-images.githubusercontent.com/9599/81037264-f1021a80-8e56-11ea-9924-6f9f55a0fb4b.png\")
\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612287234, "label": "Import machine-learning detected labels (dog, llama etc) from Apple Photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623857417", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16", "id": 623857417, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzg1NzQxNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-05T05:01:47Z", "updated_at": "2020-05-05T05:01:47Z", "author_association": "MEMBER", "body": "Even that didn't work - it didn't copy across the rowid values. I'm pretty sure that's what's wrong here:\r\n```\r\nsqlite3 /Users/simon/Pictures/Photos\\ Library.photoslibrary/database/search/psi.sqlite 'select rowid, uuid_0, uuid_1 from assets limit 10' \r\n1619605|-9205353363298198838|4814875488794983828\r\n1641378|-9205348195631362269|390804289838822030\r\n1634974|-9205331524553603243|-3834026796261633148\r\n1619083|-9205326176986145401|7563404215614709654\r\n22131|-9205315724827218763|8370531509591906734\r\n1645633|-9205247376092758131|-1311540150497601346\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612287234, "label": "Import machine-learning detected labels (dog, llama etc) from Apple Photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855885", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16", "id": 623855885, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzg1NTg4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-05T04:54:39Z", "updated_at": "2020-05-05T04:54:53Z", "author_association": "MEMBER", "body": "Trying this import mechanism instead:\r\n`sqlite3 /Users/simon/Pictures/Photos\\ Library.photoslibrary/database/search/psi.sqlite .dump | grep -v 'CREATE INDEX' | grep -v 'CREATE TRIGGER' | grep -v 'CREATE VIRTUAL TABLE' | sqlite3 search.db`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612287234, "label": "Import machine-learning detected labels (dog, llama etc) from Apple Photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855841", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16", "id": 623855841, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzg1NTg0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-05T04:54:28Z", "updated_at": "2020-05-05T04:54:28Z", "author_association": "MEMBER", "body": "Things were not matching up for me correctly:\r\n\r\n![\"search__select_json_object__img_src____https___photos_simonwillison_net_i______photos_sha256___________photos_ext______w_400___as_photo__groups_content_string__assets_uuid_0__assets_uuid_1__to_uuid_assets_uuid_0__assets_uuid_1__as_uuid__pho\"](\"https://user-images.githubusercontent.com/9599/81035923-ca8db080-8e51-11ea-95a7-6ee60bae7502.png\")
\r\n\r\nI think that's because my import script didn't correctly import the existing `rowid` values.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612287234, "label": "Import machine-learning detected labels (dog, llama etc) from Apple Photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623846880", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16", "id": 623846880, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzg0Njg4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-05T04:06:08Z", "updated_at": "2020-05-05T04:06:08Z", "author_association": "MEMBER", "body": "This function seems to convert them into UUIDs that match my photos:\r\n```python\r\ndef to_uuid(uuid_0, uuid_1):\r\n b = uuid_0.to_bytes(8, 'little', signed=True) + uuid_1.to_bytes(8, 'little', signed=True)\r\n return str(uuid.UUID(bytes=b)).upper()\r\n```", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612287234, "label": "Import machine-learning detected labels (dog, llama etc) from Apple Photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623811131", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16", "id": 623811131, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzgxMTEzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-05T03:16:18Z", "updated_at": "2020-05-05T03:16:18Z", "author_association": "MEMBER", "body": "Here's how to convert two integers unto a UUID using Java. Not sure if it's the solution I need though (or how to do the same thing in Python):\r\n\r\nhttps://repl.it/repls/EuphoricSomberClasslibrary\r\n\r\n![\"Repl_it_-_EuphoricSomberClasslibrary\"](\"https://user-images.githubusercontent.com/9599/81032267-0d488c00-8e44-11ea-9be7-680eaccd1611.png\")
\r\n\r\n```java\r\nimport java.util.UUID;\r\n\r\nclass Main {\r\n public static void main(String[] args) {\r\n java.util.UUID uuid = new java.util.UUID(\r\n 2544182952487526660L,\r\n -3640314103732024685L\r\n );\r\n System.out.println(\r\n uuid\r\n );\r\n }\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612287234, "label": "Import machine-learning detected labels (dog, llama etc) from Apple Photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623807568", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16", "id": 623807568, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzgwNzU2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-05T02:56:06Z", "updated_at": "2020-05-05T02:56:06Z", "author_association": "MEMBER", "body": "I'm pretty sure this is what I'm after. The `groups` table has what looks like identified labels in the rows with category = 2025:\r\n\r\n![\"words__groups__2_528_rows_where_where_category___2025\"](\"https://user-images.githubusercontent.com/9599/81031361-e0df4080-8e40-11ea-9060-6d850aa52140.png\")
\r\n\r\nThen there's a `ga` table that maps groups to assets:\r\n\r\n![\"words__ga__633_653_rows\"](\"https://user-images.githubusercontent.com/9599/81031387-f48aa700-8e40-11ea-9a3d-da23903be928.png\")
\r\n\r\nAnd an `assets` table which looks like it has one row for every one of my photos:\r\n\r\n![\"words__assets__40_419_rows\"](\"https://user-images.githubusercontent.com/9599/81031402-04a28680-8e41-11ea-8047-e9199d068563.png\")
\r\n\r\nOne major challenge: these UUIDs are split into two integer numbers, `uuid_0` and `uuid_1` - but the main photos database uses regular UUIDs like this:\r\n\r\n\r\n\r\nI need to figure out how to match up these two different UUID representations. I asked on Twitter if anyone has any ideas: https://twitter.com/simonw/status/1257500689019703296", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612287234, "label": "Import machine-learning detected labels (dog, llama etc) from Apple Photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806687", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16", "id": 623806687, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzgwNjY4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-05T02:51:16Z", "updated_at": "2020-05-05T02:51:16Z", "author_association": "MEMBER", "body": "Running datasette against it directly doesn't work:\r\n```\r\nsimon@Simons-MacBook-Pro search % datasette psi.sqlite\r\nServe! files=('psi.sqlite',) (immutables=()) on port 8001\r\nUsage: datasette serve [OPTIONS] [FILES]...\r\n\r\nError: Connection to psi.sqlite failed check: no such tokenizer: PSITokenizer\r\n```\r\nInstead, I created a new SQLite database with a copy of some of the key tables, like this:\r\n```\r\nsqlite-utils rows psi.sqlite groups | sqlite-utils insert /tmp/search.db groups -\r\nsqlite-utils rows psi.sqlite assets | sqlite-utils insert /tmp/search.db assets -\r\nsqlite-utils rows psi.sqlite ga | sqlite-utils insert /tmp/search.db ga -\r\nsqlite-utils rows psi.sqlite collections | sqlite-utils insert /tmp/search.db collections -\r\nsqlite-utils rows psi.sqlite gc | sqlite-utils insert /tmp/search.db gc -\r\nsqlite-utils rows psi.sqlite lookup | sqlite-utils insert /tmp/search.db lookup -\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612287234, "label": "Import machine-learning detected labels (dog, llama etc) from Apple Photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806533", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16", "id": 623806533, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzgwNjUzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-05T02:50:16Z", "updated_at": "2020-05-05T02:50:16Z", "author_association": "MEMBER", "body": "I figured there must be a separate database that Photos uses to store the text of the identified labels.\r\n\r\nI used \"Open Files and Ports\" in Activity Monitor against the Photos app to try and spot candidates... and found `/Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite` - a 53MB SQLite database file.\r\n\r\n![\"Item-0_and_Item-0_and_Item-0_and_Item-0\"](\"https://user-images.githubusercontent.com/9599/81031213-61ea0800-8e40-11ea-8237-cfce4a5128e0.png\")
\r\n\r\nHere's the schema of that file:\r\n```\r\n$ sqlite3 psi.sqlite .schema\r\nCREATE TABLE word_embedding(word TEXT, extended_word TEXT, score DOUBLE);\r\nCREATE INDEX word_embedding_index ON word_embedding(word);\r\nCREATE VIRTUAL TABLE word_embedding_prefix USING fts5(extended_word)\r\n/* word_embedding_prefix(extended_word) */;\r\nCREATE TABLE IF NOT EXISTS 'word_embedding_prefix_data'(id INTEGER PRIMARY KEY, block BLOB);\r\nCREATE TABLE IF NOT EXISTS 'word_embedding_prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;\r\nCREATE TABLE IF NOT EXISTS 'word_embedding_prefix_content'(id INTEGER PRIMARY KEY, c0);\r\nCREATE TABLE IF NOT EXISTS 'word_embedding_prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB);\r\nCREATE TABLE IF NOT EXISTS 'word_embedding_prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID;\r\nCREATE TABLE groups(category INT2, owning_groupid INT, content_string TEXT, normalized_string TEXT, lookup_identifier TEXT, token_ranges_0 INT8, token_ranges_1 INT8, UNIQUE(category, owning_groupid, content_string, lookup_identifier, token_ranges_0, token_ranges_1));\r\nCREATE TABLE assets(uuid_0 INT, uuid_1 INT, creationDate INT, UNIQUE(uuid_0, uuid_1));\r\nCREATE TABLE ga(groupid INT, assetid INT, PRIMARY KEY(groupid, assetid));\r\nCREATE TABLE collections(uuid_0 INT, uuid_1 INT, startDate INT, endDate INT, title TEXT, subtitle TEXT, keyAssetUUID_0 INT, keyAssetUUID_1 INT, typeAndNumberOfAssets INT32, sortDate DOUBLE, UNIQUE(uuid_0, uuid_1));\r\nCREATE TABLE gc(groupid INT, collectionid INT, PRIMARY KEY(groupid, collectionid));\r\nCREATE VIRTUAL TABLE prefix USING fts5(content='groups', normalized_string, category UNINDEXED, tokenize = 'PSITokenizer');\r\nCREATE TABLE IF NOT EXISTS 'prefix_data'(id INTEGER PRIMARY KEY, block BLOB);\r\nCREATE TABLE IF NOT EXISTS 'prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;\r\nCREATE TABLE IF NOT EXISTS 'prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB);\r\nCREATE TABLE IF NOT EXISTS 'prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID;\r\nCREATE TABLE lookup(identifier TEXT PRIMARY KEY, category INT2);\r\nCREATE TRIGGER trigger_groups_insert AFTER INSERT ON groups BEGIN INSERT INTO prefix(rowid, normalized_string, category) VALUES (new.rowid, new.normalized_string, new.category); END;\r\nCREATE TRIGGER trigger_groups_delete AFTER DELETE ON groups BEGIN INSERT INTO prefix(prefix, rowid, normalized_string, category) VALUES('delete', old.rowid, old.normalized_string, old.category); END;\r\nCREATE INDEX group_pk ON groups(category, content_string, normalized_string, lookup_identifier);\r\nCREATE INDEX asset_pk ON assets(uuid_0, uuid_1);\r\nCREATE INDEX ga_assetid ON ga(assetid, groupid);\r\nCREATE INDEX collection_pk ON collections(uuid_0, uuid_1);\r\nCREATE INDEX gc_collectionid ON gc(collectionid);\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612287234, "label": "Import machine-learning detected labels (dog, llama etc) from Apple Photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806085", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16", "id": 623806085, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzgwNjA4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-05T02:47:18Z", "updated_at": "2020-05-05T02:47:18Z", "author_association": "MEMBER", "body": "In https://github.com/RhetTbull/osxphotos/issues/121#issuecomment-623249263 Rhet Turnbull spotted a table called `ZSCENEIDENTIFIER` which looked like it might have the right data, but the columns in it aren't particularly helpful:\r\n```\r\nZ_PK,Z_ENT,Z_OPT,ZSCENEIDENTIFIER,ZASSETATTRIBUTES,ZCONFIDENCE\r\n8,49,1,731,5,0.11834716796875\r\n9,49,1,684,6,0.0233648251742125\r\n10,49,1,1702,1,0.026153564453125\r\n```\r\nI love the look of those confidence scores, but what do the numbers mean?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612287234, "label": "Import machine-learning detected labels (dog, llama etc) from Apple Photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623805823", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16", "id": 623805823, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzgwNTgyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-05T02:45:56Z", "updated_at": "2020-05-05T02:45:56Z", "author_association": "MEMBER", "body": "I filed an issue with `osxphotos` about this here: https://github.com/RhetTbull/osxphotos/issues/121", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612287234, "label": "Import machine-learning detected labels (dog, llama etc) from Apple Photos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623739934", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15", "id": 623739934, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzczOTkzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-04T22:24:26Z", "updated_at": "2020-05-04T22:24:26Z", "author_association": "MEMBER", "body": "Twitter thread with some examples of photos that are coming up from queries against these scores: https://twitter.com/simonw/status/1257434670750408705", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612151767, "label": "Expose scores from ZCOMPUTEDASSETATTRIBUTES"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623730934", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15", "id": 623730934, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzczMDkzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-04T22:00:38Z", "updated_at": "2020-05-04T22:00:48Z", "author_association": "MEMBER", "body": "Here's the query to create the new table:\r\n```sql\r\ncreate table apple_photos_scores as select\r\n ZGENERICASSET.ZUUID,\r\n ZGENERICASSET.ZOVERALLAESTHETICSCORE,\r\n ZGENERICASSET.ZCURATIONSCORE,\r\n ZGENERICASSET.ZPROMOTIONSCORE,\r\n ZGENERICASSET.ZHIGHLIGHTVISIBILITYSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZBEHAVIORALSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZFAILURESCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZHARMONIOUSCOLORSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZIMMERSIVENESSSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZINTERACTIONSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZINTERESTINGSUBJECTSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZINTRUSIVEOBJECTPRESENCESCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZLIVELYCOLORSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZLOWLIGHT,\r\n ZCOMPUTEDASSETATTRIBUTES.ZNOISESCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCAMERATILTSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCOMPOSITIONSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTLIGHTINGSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPATTERNSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPERSPECTIVESCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPOSTPROCESSINGSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTREFLECTIONSSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTSYMMETRYSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZSHARPLYFOCUSEDSUBJECTSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZTASTEFULLYBLURREDSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZWELLCHOSENSUBJECTSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZWELLFRAMEDSUBJECTSCORE,\r\n ZCOMPUTEDASSETATTRIBUTES.ZWELLTIMEDSHOTSCORE\r\nfrom\r\n attached.ZGENERICASSET\r\n join attached.ZCOMPUTEDASSETATTRIBUTES on\r\n attached.ZGENERICASSET.Z_PK = attached.ZCOMPUTEDASSETATTRIBUTES.Z_PK;\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612151767, "label": "Expose scores from ZCOMPUTEDASSETATTRIBUTES"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723687", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15", "id": 623723687, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzcyMzY4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-04T21:43:06Z", "updated_at": "2020-05-04T21:43:06Z", "author_association": "MEMBER", "body": "It looks like I can map the photos I'm importing to these tables using the `ZUUID` column on `ZGENERICASSET` to get a `Z_PK` which then maps to the rows in `ZGENERICASSET`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612151767, "label": "Expose scores from ZCOMPUTEDASSETATTRIBUTES"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723026", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15", "id": 623723026, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzcyMzAyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-04T21:41:30Z", "updated_at": "2020-05-04T21:41:30Z", "author_association": "MEMBER", "body": "I'm going to put these in a table called `apple_photos_scores` - I'll also pull in the following columns from the `ZGENERICASSET` table:\r\n\r\n* `ZOVERALLAESTHETICSCORE`\r\n* `ZCURATIONSCORE`\r\n* `ZHIGHLIGHTVISIBILITYSCORE`\r\n* `ZPROMOTIONSCORE`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612151767, "label": "Expose scores from ZCOMPUTEDASSETATTRIBUTES"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623232984", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623232984, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzIzMjk4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-04T02:41:32Z", "updated_at": "2020-05-04T02:41:32Z", "author_association": "MEMBER", "body": "Needs documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199750", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623199750, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5OTc1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T23:17:58Z", "updated_at": "2020-05-03T23:17:58Z", "author_association": "MEMBER", "body": "Reading this source code is really useful for figuring out how to store a photo in a DB table: https://github.com/RhetTbull/osxphotos/blob/7444b6d173918a3ad2a07aefce5ecf054786c787/osxphotos/photoinfo.py", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199701", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623199701, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5OTcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T23:17:38Z", "updated_at": "2020-05-03T23:17:38Z", "author_association": "MEMBER", "body": "Record burst_uuid as a column:\r\n```\r\n(Pdb) with_bursts[0]._info[\"burstUUID\"]\r\n'703FAA23-57BF-40B4-8A33-D9CEB143391B'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199214", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623199214, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5OTIxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T23:14:08Z", "updated_at": "2020-05-03T23:14:08Z", "author_association": "MEMBER", "body": "Albums have UUIDs:\r\n```\r\n(Pdb) photo.album_info[0].__dict__\r\n{'_uuid': '17816791-ABF3-447B-942C-9FA8065EEBBA', '_db': osxphotos.PhotosDB(dbfile='/Users/simon/Pictures/Photos Library.photoslibrary/database/photos.db'), '_title': 'Geotaggable Photos geotagged'}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198986", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623198986, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5ODk4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T23:12:31Z", "updated_at": "2020-05-03T23:12:46Z", "author_association": "MEMBER", "body": "To get the taken date in UTC:\r\n```\r\nfrom datetime import timezone\r\n(Pdb) photo.date.astimezone(timezone.utc).isoformat()\r\n'2018-02-13T20:21:31.620000+00:00'\r\n(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(\".\")\r\n['2018-02-13T20:21:31', '620000+00:00']\r\n(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(\".\")[0]\r\n'2018-02-13T20:21:31'\r\n(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(\".\")[0] + \"+00:00\"\r\n'2018-02-13T20:21:31+00:00'\r\n\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198653", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623198653, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5ODY1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T23:09:57Z", "updated_at": "2020-05-03T23:09:57Z", "author_association": "MEMBER", "body": "For locations: I'll add `place_x` columns for all of these:\r\n```\r\n(Pdb) photo.place.address._asdict()\r\n{'street': None, 'sub_locality': None, 'city': 'Loreto', 'sub_administrative_area': 'Loreto', 'state_province': 'BCS', 'postal_code': None, 'country': 'Mexico', 'iso_country_code': 'MX'}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623195197", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623195197, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5NTE5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T22:44:33Z", "updated_at": "2020-05-03T22:44:33Z", "author_association": "MEMBER", "body": "Command will be this:\r\n\r\n $ photos-to-sqlite apple-photos photos.db\r\n\r\nThis will populate a `apple_photos` table with the data imported by the `osxphotos` library, plus the calculated sha256.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623193947", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1", "id": 623193947, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzE5Mzk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T22:36:17Z", "updated_at": "2020-05-03T22:36:17Z", "author_association": "MEMBER", "body": "I'm going to use [osxphotos](https://github.com/RhetTbull/osxphotos) for this.\r\n\r\nSince I've already got code to upload photos and insert them into a table based on their `sha256` hash, my first go at this will be to import data using the tool and foreign-key it to the `sha256` hash in the existing table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533300, "label": "Import photo metadata from Apple Photos into SQLite"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038378", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38", "id": 623038378, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzAzODM3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-03T01:21:13Z", "updated_at": "2020-05-03T01:21:13Z", "author_association": "MEMBER", "body": "No this is really useful feedback! I'm so close to this project that I miss what's not obvious to people dropping in for the first time.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 611284481, "label": "[Feature Request] Support Repo Name in Search \ud83e\udd7a"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623027889", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38", "id": 623027889, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzAyNzg4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T23:15:11Z", "updated_at": "2020-05-02T23:15:11Z", "author_association": "MEMBER", "body": "This is one of the use-cases for the `repos_starred` view: it allows you to easily run this kid of query without having to construct the SQL by hand. Here's a demo:\r\n\r\nhttps://github-to-sqlite.dogsheep.net/github/repos_starred?name__contains=twitter\r\n\r\nMy philosophy here is to keep the raw tables (like `stars`) as normalized as possible, then use SQL views which expose the data in a form that's easier to query.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 611284481, "label": "[Feature Request] Support Repo Name in Search \ud83e\udd7a"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623010272", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4", "id": 623010272, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzAxMDI3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T20:39:14Z", "updated_at": "2020-05-02T20:39:14Z", "author_association": "MEMBER", "body": "Graph of cumulative stars for Datasette over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29#g.mark=line&g.x_column=yyyymmdd&g.x_type=temporal&g.y_column=cumulative_count&g.y_type=quantitative\r\n\r\n![\"github__select_yyyymmdd__sum_n__over___order_by_yyyymmdd_rows_unbounded_preceding___as_cumulative_count_from___select_substr_starred_at__0__11__as_yyyymmdd__count____as_n_from_stars_where_repo___107914493_group_by_yyyymmdd__\"](\"https://user-images.githubusercontent.com/9599/80891642-31587180-8c7a-11ea-9169-60eb3cb6d3e0.png\")
\r\n\r\nStars per day (as a label bar chart, so very wide):\r\n\r\nhttps://github-to-sqlite.dogsheep.net/github?sql=%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++#g.mark=bar&g.x_column=yyyymmdd&g.x_type=ordinal&g.y_column=n&g.y_type=quantitative\r\n\r\n![\"github__select_substr_starred_at__0__11__as_yyyymmdd__count____as_n_from_stars_where_repo___107914493_group_by_yyyymmdd\"](\"https://user-images.githubusercontent.com/9599/80891656-4e8d4000-8c7a-11ea-9ff2-02c9cf4cb5d5.png\")
\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 493670730, "label": "Command to fetch stargazers for one or more repos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623007441", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4", "id": 623007441, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzAwNzQ0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T20:13:37Z", "updated_at": "2020-05-02T20:13:37Z", "author_association": "MEMBER", "body": "Datasette cumulative stars over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 493670730, "label": "Command to fetch stargazers for one or more repos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623006154", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4", "id": 623006154, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzAwNjE1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T20:01:39Z", "updated_at": "2020-05-02T20:01:54Z", "author_association": "MEMBER", "body": "Needs tests and documentation. I shipped it early to check that the live demo works.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 493670730, "label": "Command to fetch stargazers for one or more repos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-623006004", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8", "id": 623006004, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzAwNjAwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T20:00:26Z", "updated_at": "2020-05-02T20:00:26Z", "author_association": "MEMBER", "body": "I'm abandoning this in favour of a new implementation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 516763727, "label": "stargazers command, refs #4"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623004836", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4", "id": 623004836, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzAwNDgzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T19:49:51Z", "updated_at": "2020-05-02T19:49:51Z", "author_association": "MEMBER", "body": "Alternative pattern:\r\n```\r\nsqlite-utils releases.db 'select full_name from repos' --csv --no-headers \\\r\n | tr -d '\\r' \\\r\n | xargs github-to-sqlite stargazers stars.db\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 493670730, "label": "Command to fetch stargazers for one or more repos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623000814", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4", "id": 623000814, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMzAwMDgxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T19:15:23Z", "updated_at": "2020-05-02T19:15:23Z", "author_association": "MEMBER", "body": "I'm not going to do the `--sql` bit just yet. I have patterns for working around this for other commands which are working fine:\r\n\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/d00a53061556dc403c166b443d141c4e1adbd64a/.github/workflows/deploy-demo.yml#L53-L70", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 493670730, "label": "Command to fetch stargazers for one or more repos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 622998813, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T18:58:17Z", "updated_at": "2020-05-02T18:58:17Z", "author_association": "MEMBER", "body": "Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622997410", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33", "id": 622997410, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk5NzQxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T18:46:10Z", "updated_at": "2020-05-02T18:46:10Z", "author_association": "MEMBER", "body": "Documented here: https://github.com/dogsheep/github-to-sqlite/blob/10fb34de41aaa35681f08b5991540d65bfcf2e2e/README.md#authentication", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 609950090, "label": "Fall back to authentication via ENV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-622990947", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4", "id": 622990947, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk5MDk0Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T17:54:16Z", "updated_at": "2020-05-02T17:54:16Z", "author_association": "MEMBER", "body": "I could add that window function query as a view, but only if the detected version of SQLite supports window functions.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 493670730, "label": "Command to fetch stargazers for one or more repos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 622989874, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T17:46:14Z", "updated_at": "2020-05-02T17:46:14Z", "author_association": "MEMBER", "body": "Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622982667", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35", "id": 622982667, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk4MjY2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T16:52:53Z", "updated_at": "2020-05-02T16:52:53Z", "author_association": "MEMBER", "body": "Easiest option: use `db.index_foreign_keys()`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#adding-indexes-for-all-foreign-keys", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610511450, "label": "Create index on issue_comments(user) and other foreign keys"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622982346", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36", "id": 622982346, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk4MjM0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T16:50:31Z", "updated_at": "2020-05-02T16:50:31Z", "author_association": "MEMBER", "body": "Demo: https://github-to-sqlite.dogsheep.net/github/dependent_repos", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610842926, "label": "Add view for better display of dependent repos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622980203", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10", "id": 622980203, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk4MDIwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T16:34:29Z", "updated_at": "2020-05-02T16:34:29Z", "author_association": "MEMBER", "body": "Fixed definition:\r\n```sql\r\nselect\r\n stars.starred_at,\r\n starring_user.login as starred_by,\r\n repos.*\r\nfrom\r\n repos\r\n join stars on repos.id = stars.repo\r\n join users as starring_user on stars.user = starring_user.id\r\n join users on repos.owner = users.id\r\norder by\r\n starred_at desc;\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 516967682, "label": "Add this repos_starred view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622978173", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37", "id": 622978173, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjk3ODE3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-02T16:19:31Z", "updated_at": "2020-05-02T16:19:47Z", "author_association": "MEMBER", "body": "I can use the new `.create_view(..., replace=True)` parameter in `sqlite-utils` 2.7.2 for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610843136, "label": "Mechanism for creating views if they don't yet exist"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461948", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37", "id": 622461948, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTk0OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:36:42Z", "updated_at": "2020-05-01T16:36:42Z", "author_association": "MEMBER", "body": "It should only create views if the underlying tables exist.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610843136, "label": "Mechanism for creating views if they don't yet exist"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461537", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37", "id": 622461537, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTUzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:35:40Z", "updated_at": "2020-05-01T16:35:40Z", "author_association": "MEMBER", "body": "This will check if the view exists and has the exact same matching definition as the one we want. If it doesn't, we will drop it (if it exists) and recreate it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610843136, "label": "Mechanism for creating views if they don't yet exist"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12", "id": 622461223, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:34:52Z", "updated_at": "2020-05-01T16:34:52Z", "author_association": "MEMBER", "body": "Blocked on #37", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520756546, "label": "Add this view for seeing new releases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622461122", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10", "id": 622461122, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTEyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:34:39Z", "updated_at": "2020-05-01T16:34:39Z", "author_association": "MEMBER", "body": "Blocked on #37", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 516967682, "label": "Add this repos_starred view"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622461025", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36", "id": 622461025, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjQ2MTAyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T16:34:24Z", "updated_at": "2020-05-01T16:34:24Z", "author_association": "MEMBER", "body": "Blocked on #37", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610842926, "label": "Add view for better display of dependent repos"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622214262", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35", "id": 622214262, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjIxNDI2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T02:10:32Z", "updated_at": "2020-05-01T02:11:19Z", "author_association": "MEMBER", "body": "This sped that query up even more - down to 4ms.\r\n```sql\r\ncreate index issue_comments_issue on issue_comments(issue);\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610511450, "label": "Create index on issue_comments(user) and other foreign keys"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622213950", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35", "id": 622213950, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjIxMzk1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T02:09:04Z", "updated_at": "2020-05-01T02:09:04Z", "author_association": "MEMBER", "body": "It sped up this query a lot - 2.5s down to 300ms:\r\n```sql\r\nselect\r\n repos.full_name,\r\n json_object(\r\n 'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number,\r\n 'label', '#' || issues.number\r\n ) as issue,\r\n issues.title,\r\n users.login,\r\n users.id,\r\n issues.state,\r\n issues.locked,\r\n issues.assignee,\r\n issues.milestone,\r\n issues.comments,\r\n issues.created_at,\r\n issues.updated_at,\r\n issues.closed_at,\r\n issues.author_association,\r\n issues.pull_request,\r\n issues.repo,\r\n issues.type\r\nfrom\r\n issues\r\n join repos on repos.id = issues.repo\r\n join users on issues.user = users.id\r\nwhere\r\n issues.state = 'open'\r\n and issues.user not in (9599, 27856297)\r\n and not exists (\r\n select\r\n id\r\n from\r\n issue_comments\r\n where\r\n issue_comments.user = 9599\r\n and issues.id = issue_comments.issue\r\n )\r\norder by\r\n issues.updated_at desc;\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610511450, "label": "Create index on issue_comments(user) and other foreign keys"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622171097", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33", "id": 622171097, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjE3MTA5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T23:22:45Z", "updated_at": "2020-04-30T23:23:57Z", "author_association": "MEMBER", "body": "The `auth.json` mechanism this uses is standard across all of the other Dogsheep tools - it's actually designed so you can have one `auth.json` with a bunch of different credentials for different tools:\r\n\r\n```json\r\n{\r\n \"goodreads_personal_token\": \"...\",\r\n \"goodreads_user_id\": \"...\",\r\n \"github_personal_token\": \"...\",\r\n \"pocket_consumer_key\": \"...\",\r\n \"pocket_username\": \"...\",\r\n \"pocket_access_token\": \"...\"\r\n}\r\n```\r\nBut... `github-to-sqlite` does feel like it deserves a special case here, since it's such a good fit for running inside of GitHub Actions - which even provide a `GITHUB_TOKEN` for you to use!\r\n\r\nSo I don't think it will harm the family of tools too much if this has an environment variable alternative to the `-a` file.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 609950090, "label": "Fall back to authentication via ENV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622169728", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33", "id": 622169728, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjE2OTcyOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T23:18:51Z", "updated_at": "2020-04-30T23:18:51Z", "author_association": "MEMBER", "body": "Sure, that sounds fine to me.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 609950090, "label": "Fall back to authentication via ENV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622162835", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34", "id": 622162835, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjE2MjgzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T22:59:26Z", "updated_at": "2020-04-30T22:59:26Z", "author_association": "MEMBER", "body": "Documentation: https://github.com/dogsheep/github-to-sqlite/blob/c9f48404481882e8b3af06f35e4801a80ac79ed6/README.md#scraping-dependents-for-a-repository", "reactions": "{\"total_count\": 2, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 2, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610408908, "label": "Command for retrieving dependents for a repo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622135654", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34", "id": 622135654, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjEzNTY1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T21:53:44Z", "updated_at": "2020-04-30T21:56:06Z", "author_association": "MEMBER", "body": "I think this is the neatest scraping pattern:\r\n```python\r\n[a[\"href\"].lstrip(\"/\") for a in soup.select(\"a[data-hovercard-type=repository]\")]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610408908, "label": "Command for retrieving dependents for a repo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622136585", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34", "id": 622136585, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjEzNjU4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T21:55:51Z", "updated_at": "2020-04-30T21:55:51Z", "author_association": "MEMBER", "body": "And to find the \"Next\" pagination link:\r\n```python\r\nsoup.select(\".paginate-container\")[0].find(\"a\", text=\"Next\")\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610408908, "label": "Command for retrieving dependents for a repo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133775", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34", "id": 622133775, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjEzMzc3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T21:49:27Z", "updated_at": "2020-04-30T21:49:27Z", "author_association": "MEMBER", "body": "Proposed command:\r\n\r\n github-to-sqlite scrape-dependents github.db simonw/datasette\r\n\r\nI'll pull full details of the scraped repos from the regular API. I'll also record when they were \"first seen\" by the command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610408908, "label": "Command for retrieving dependents for a repo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133422", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34", "id": 622133422, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjEzMzQyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T21:48:39Z", "updated_at": "2020-04-30T21:48:39Z", "author_association": "MEMBER", "body": "It looks like the only option is to scrape them. I'll do that and then replace with an API as soon as one becomes available.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610408908, "label": "Command for retrieving dependents for a repo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133298", "issue_url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34", "id": 622133298, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjEzMzI5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-30T21:48:24Z", "updated_at": "2020-04-30T21:48:24Z", "author_association": "MEMBER", "body": "Unfortunately it's not available through any GitHub API - I managed to figure out how to get dependencies, but I need dependents. https://github.com/simonw/til/blob/master/github/dependencies-graphql-api.md", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610408908, "label": "Command for retrieving dependents for a repo"}, "performed_via_github_app": null}