{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932007", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615932007, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMjAwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:27:55Z", "updated_at": "2020-04-18T19:27:55Z", "author_association": "MEMBER", "body": "Research thread: https://twitter.com/simonw/status/1249049694984011776\r\n> I want to build some software that lets people store their own data in their own S3 bucket, but if possible I'd like not to have to teach people the incantations needed to get their bucket setup and minimum-permission credentials figures out\r\n\r\nhttps://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/ looks useful", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615932204", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615932204, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMjIwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:29:22Z", "updated_at": "2020-04-18T19:34:44Z", "author_association": "MEMBER", "body": "I'm going to call my bucket `dogsheep-photos-simon`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615933273", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615933273, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzMzI3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:37:33Z", "updated_at": "2020-04-18T19:37:33Z", "author_association": "MEMBER", "body": "https://console.aws.amazon.com/s3/bucket/create?region=us-west-1\r\n\r\n\r\n\r\nI created it with no public read-write access. I plan to use signed URLs via a transforming proxy to access images for display on the web.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615935577", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615935577, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzNTU3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T19:54:59Z", "updated_at": "2020-04-18T19:55:30Z", "author_association": "MEMBER", "body": "Creating IAM groups called `dogsheep-photos-simon-read-write` and `dogsheep-photos-simon-read`: https://console.aws.amazon.com/iam/home#/groups - I created them with no attached policies.\r\n\r\nNow I can attach an \"inline policy\" to each one. For the read-write group I go here:\r\n\r\nhttps://console.aws.amazon.com/iam/home#/groups/dogsheep-photos-simon-read-write\r\n\r\n\r\n\r\nExample policies are here: https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html\r\n\r\nFor the read-write one I went with:\r\n```json\r\n{\r\n \"Version\": \"2012-10-17\",\r\n \"Statement\": [\r\n {\r\n \"Effect\": \"Allow\",\r\n \"Action\": \"s3:*\",\r\n \"Resource\": [\r\n \"arn:aws:s3:::dogsheep-photos-simon/*\"\r\n ]\r\n }\r\n ]\r\n}\r\n```\r\nFor the read-only policy I'm going to guess that this is appropriate:\r\n\r\n```json\r\n{\r\n \"Version\": \"2012-10-17\",\r\n \"Statement\": [\r\n {\r\n \"Effect\": \"Allow\",\r\n \"Action\": [\r\n \"s3:GetObject*\",\r\n \"s3:ListBucket\"\r\n ],\r\n \"Resource\": [\r\n \"arn:aws:s3:::dogsheep-photos-simon/*\"\r\n ]\r\n }\r\n ]\r\n}\r\n```\r\nI tried the policy simulator to test this out: https://policysim.aws.amazon.com/home/index.jsp?#groups/dogsheep-photos-simon-read - this worked:\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615936880", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615936880, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTkzNjg4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:04:31Z", "updated_at": "2020-04-18T20:04:31Z", "author_association": "MEMBER", "body": "Next step: create two IAM users, one for each of those groups.\r\n\r\nhttps://console.aws.amazon.com/iam/home#/users$new?step=details\r\n\r\n\r\n\r\n\r\n\r\nI copied the keys into a secure note in 1password.\r\n\r\nCouldn't get into Transmit with them though! https://library.panic.com/transmit/transmit5/iam-roles/ may help.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615941746", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615941746, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0MTc0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:29:36Z", "updated_at": "2020-04-18T20:29:36Z", "author_association": "MEMBER", "body": "I'm going to create another user just for Transmit, with full S3 access.\r\n\r\nname: `dogsheep-photos-simon-s3-all-access`\r\n\r\nRather than creating a group for that user, I'm trying the \"Attach existing policies directly\" option:\r\n\r\n\r\n\r\nThat user DID work with Transmit. I uploaded a test HEIC image. I used Transmit to copy a signed URL for it.\r\n\r\n```\r\n~ $ curl -i 'https://dogsheep-photos-simon.s3.us-west-1.amazonaws.com/IMG_7195.HEIC?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAWXFXAI...' | head -n 100\r\n % Total % Received % Xferd Average Speed Time Time Time Current\r\n Dload Upload Total Spent Left Speed\r\n 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0HTTP/1.1 200 OK\r\nx-amz-id-2: gBOCYqZfbNAnv0R/uJ++qm2NbW5SgD4TapgF9RQjzzeDIThcCz/BkKU+YoxlG4NJHlcmMgAHyh4=\r\nx-amz-request-id: C2FE7FCC3BD53A84\r\nDate: Sat, 18 Apr 2020 20:28:54 GMT\r\nLast-Modified: Sat, 18 Apr 2020 20:13:49 GMT\r\nETag: \"fe3e081239a123ef745517878c53b854\"\r\nAccept-Ranges: bytes\r\nContent-Type: image/heic\r\nContent-Length: 1913097\r\nServer: AmazonS3\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615942116", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615942116, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0MjExNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:30:56Z", "updated_at": "2020-04-18T20:30:56Z", "author_association": "MEMBER", "body": "Next step: attempt a programmatic upload using the `dogsheep-photos-simon-read-write` credentials from a Jupyter notebook.\r\n\r\nAlso attempt a programmatic bucket listing and read using `dogsheep-photos-simon-read` credentials.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615944806", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615944806, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NDgwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:41:39Z", "updated_at": "2020-04-18T20:41:39Z", "author_association": "MEMBER", "body": "This worked!\r\n\r\n\r\n\r\nAnd this worked:\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615945056", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615945056, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NTA1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:42:41Z", "updated_at": "2020-04-18T20:42:41Z", "author_association": "MEMBER", "body": "But... `list_objects` failed for both of my keys (read and write):\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615946537", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615946537, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NjUzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:48:13Z", "updated_at": "2020-04-18T20:48:13Z", "author_association": "MEMBER", "body": "How about generating a signed URL?\r\n```python\r\nread_client.generate_presigned_url(\r\n \"get_object\",\r\n Params={\r\n \"Bucket\": \"dogsheep-photos-simon\",\r\n \"Key\": \"this_is_fine.jpg\",\r\n },\r\n ExpiresIn=600\r\n)\r\n```\r\nGave me https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398\r\n\r\nWhich does this:\r\n\r\n```\r\n~ $ curl -i 'https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'\r\nHTTP/1.1 307 Temporary Redirect\r\nx-amz-bucket-region: us-west-1\r\nx-amz-request-id: E78CD859AEE21D33\r\nx-amz-id-2: 648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=\r\nLocation: https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398\r\nContent-Type: application/xml\r\nTransfer-Encoding: chunked\r\nDate: Sat, 18 Apr 2020 20:47:21 GMT\r\nServer: AmazonS3\r\n\r\n\r\nTemporaryRedirectPlease re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.dogsheep-photos-simon.s3-us-west-1.amazonaws.comdogsheep-photos-simonE78CD859AEE21D33648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=~ $ \r\n```\r\nSo it redirects to another URL... which returns this:\r\n```\r\n~ $ curl -i 'https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'\r\nHTTP/1.1 200 OK\r\nx-amz-id-2: XafOl6mswj3yz0GJC9+Ptot1ll5sROVwqsMc10CUUfgpaUANTdIx2GhnONb5d1GVFJ6wlS2j3UY=\r\nx-amz-request-id: 258387C180411AFE\r\nDate: Sat, 18 Apr 2020 20:47:52 GMT\r\nLast-Modified: Sat, 18 Apr 2020 20:37:35 GMT\r\nETag: \"ee04081c3182a44a1c6944e94012e977\"\r\nAccept-Ranges: bytes\r\nContent-Type: binary/octet-stream\r\nContent-Length: 53072\r\nServer: AmazonS3\r\n\r\n????JFIF??C\r\n```\r\nSo that worked! It did come back with `Content-Type: binary/octet-stream` though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947229", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615947229, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NzIyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:51:26Z", "updated_at": "2020-04-18T20:51:26Z", "author_association": "MEMBER", "body": "Running the upload again like this resulted in the correct content-type:\r\n```python\r\nclient.upload_file(\r\n \"/Users/simonw/Desktop/this_is_fine.jpg\",\r\n \"dogsheep-photos-simon\",\r\n \"this_is_fine.jpg\",\r\n ExtraArgs={\r\n \"ContentType\": \"image/jpeg\"\r\n }\r\n)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947370", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615947370, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0NzM3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:52:13Z", "updated_at": "2020-04-18T20:52:13Z", "author_association": "MEMBER", "body": "This is great! I now have a key that can upload photos, and a separate key that can download photos OR generate signed URLs to access those photos.\r\n\r\nNext step: a script that starts uploading my photos.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615948102", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615948102, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk0ODEwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T20:56:59Z", "updated_at": "2020-04-18T20:56:59Z", "author_association": "MEMBER", "body": "I'm going to start with this:\r\n\r\n`photos-to-sqlite upload photos.db ~/path/to/directory`\r\n\r\nThis will scan the provided directory (and all sub-directories) for image files. It will then:\r\n\r\n* Calculate a sha256 of the contents of that file\r\n* Upload the file to a key that's `sha256.jpg` or `.heic`\r\n* Upload a `sha256.json` file with the original path to the image\r\n* Add that image to a `uploads` table in `photos.db`\r\n\r\nStretch goal: grab the EXIF data and include that in the `.json` upload AND the `uploads` database table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615957385", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4", "id": 615957385, "node_id": "MDEyOklzc3VlQ29tbWVudDYxNTk1NzM4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-18T21:56:16Z", "updated_at": "2020-04-18T21:58:11Z", "author_association": "MEMBER", "body": "Got this working! I'll do EXIF in a separate ticket #3.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 602533539, "label": "Upload all my photos to a secure S3 bucket"}, "performed_via_github_app": null}