issue_comments

311 rows where author_association = "MEMBER" sorted by updated_at descending

View and edit SQL

Suggested facets: reactions, created_at (date), updated_at (date)

issue

user

author_association

  • MEMBER · 311
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
645599881 https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645599881 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47 MDEyOklzc3VlQ29tbWVudDY0NTU5OTg4MQ== simonw 9599 2020-06-17T20:13:48Z 2020-06-17T20:13:48Z MEMBER

I've now figured out how to compile specific SQLite versions to help replicate this problem: https://github.com/simonw/til/blob/master/sqlite/ld-preload.md

Next step: replicate the problem!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fall back to FTS4 if FTS5 is not available 639542974
645512127 https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47 MDEyOklzc3VlQ29tbWVudDY0NTUxMjEyNw== simonw 9599 2020-06-17T17:24:22Z 2020-06-17T17:24:22Z MEMBER

That means your version of SQLite is old enough that it doesn't support the FTS5 extension.

Could you share what operating system you're running, and what the output is that you get from running this?

python -c 'import sqlite3; print(sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0])'

I can teach this tool to fall back on FTS4 if FTS5 isn't available.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fall back to FTS4 if FTS5 is not available 639542974
643414646 https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643414646 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40 MDEyOklzc3VlQ29tbWVudDY0MzQxNDY0Ng== simonw 9599 2020-06-12T18:06:48Z 2020-06-12T18:06:48Z MEMBER

That fixed it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Demo deploy is broken 637899539
643393506 https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643393506 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40 MDEyOklzc3VlQ29tbWVudDY0MzM5MzUwNg== simonw 9599 2020-06-12T17:21:14Z 2020-06-12T17:21:14Z MEMBER

I only install SQLite for this:

https://github.com/dogsheep/github-to-sqlite/blob/c0d54e0260468be38152293df5abd775c068495d/.github/workflows/deploy-demo.yml#L77-L78

I'm going to remove the need to install sqlite3 by making this possible with sqlite-utils: https://github.com/simonw/sqlite-utils/issues/115

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Demo deploy is broken 637899539
633704127 https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633704127 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 MDEyOklzc3VlQ29tbWVudDYzMzcwNDEyNw== simonw 9599 2020-05-25T20:14:22Z 2020-05-25T20:14:22Z MEMBER

https://github.com/dogsheep/dogsheep-photos/blob/0.4.1/README.md#serving-photos-locally-with-datasette-media

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to serve thumbnailed Apple Photo from its place on disk 613006393
633644225 https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633644225 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 MDEyOklzc3VlQ29tbWVudDYzMzY0NDIyNQ== simonw 9599 2020-05-25T16:30:44Z 2020-05-25T16:30:44Z MEMBER

I'll add docs on using datasette-json-html too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to serve thumbnailed Apple Photo from its place on disk 613006393
633643921 https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633643921 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 MDEyOklzc3VlQ29tbWVudDYzMzY0MzkyMQ== simonw 9599 2020-05-25T16:29:44Z 2020-05-25T16:29:44Z MEMBER

https://github.com/dogsheep/dogsheep-photos/blob/dc43fa8653cb9c7238a36f52239b91d1ec916d5c/README.md#serving-photos-locally-with-datasette-media

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to serve thumbnailed Apple Photo from its place on disk 613006393
633629944 https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633629944 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 MDEyOklzc3VlQ29tbWVudDYzMzYyOTk0NA== simonw 9599 2020-05-25T15:47:42Z 2020-05-25T15:47:42Z MEMBER

I'll add a proper section to the README, but for the moment here's how I do this.

First, install datasette and the datasette-media plugin.

Create a metadata.yaml file with the following content:

plugins:
  datasette-media:
    photo:
      sql: |-
        select path as filepath, 200 as resize_height from apple_photos where uuid = :key
    photo-big:
      sql: |-
        select path as filepath, 1024 as resize_height from apple_photos where uuid = :key

Now run datasette -m metadata.yaml photos.db - thumbnails will be served at http://127.0.0.1:8001/-/media/photo/F4469918-13F3-43D8-9EC1-734C0E6B60AD and larger sizes of the image at http://127.0.0.1:8001/-/media/photo-big/A8B02C7D-365E-448B-9510-69F80C26304D

I also made myself two custom pages, one showing recent images and one showing random images.

To do this, install the datasette-template-sql plugin and then create a templates/pages directory and add these files:

recent-photos.html

<h1>Recent photos</h1>

<div>
{% for photo in sql("select * from apple_photos order by date desc limit 100") %}
    <img src="/-/media/photo/{{ photo['uuid'] }}">
{% endfor %}
</div>

random-photos.html

<h1>Random photos</h1>

<div>
{% for photo in sql("with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 100") %}
    <img src="/-/media/photo/{{ photo['uuid'] }}">
{% endfor %}
</div>

Now run datasette -m metadata.yaml photos.db --template-dir=templates/

Visit http://127.0.0.1:8001/random-photos to see some random photos or http://127.0.0.1:8002/recent-photos for recent photos.

This is using this mechanism: https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to serve thumbnailed Apple Photo from its place on disk 613006393
633626741 https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633626741 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 MDEyOklzc3VlQ29tbWVudDYzMzYyNjc0MQ== simonw 9599 2020-05-25T15:38:55Z 2020-05-25T15:38:55Z MEMBER

Sure, I should absolutely document this!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to serve thumbnailed Apple Photo from its place on disk 613006393
631253852 https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253852 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25 MDEyOklzc3VlQ29tbWVudDYzMTI1Mzg1Mg== simonw 9599 2020-05-20T05:56:17Z 2020-05-21T22:26:16Z MEMBER

I have a deploy-demo.sh script now:

#!/bin/bash
if [ -f public.db ]; then
  rm public.db
fi
pipenv run dogsheep-photos create-subset photos.db public.db \
    "select sha256 from apple_photos where albums like '%Public%'"
pipenv run sqlite-utils create-view public.db photos_on_a_map \
    "select
      date,
      latitude,
      longitude,
      apple_photos.sha256,
      uploads.ext,
      json_object(
        'title',
        'Taken on ' || date,
        'image',
        'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=400',
        'link',
        'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=1200'
      ) as popup
    from
      apple_photos
      join uploads on apple_photos.sha256 = uploads.sha256
    where
      latitude is not null
    order by
      date desc" \
    --replace
pipenv run datasette publish now public.db --project dogsheep-photos \
    --about=dogsheep/dogsheep-photos \
    --about_url="https://github.com/dogsheep/dogsheep-photos" \
    --install=datasette-json-html \
    --install=datasette-pretty-json \
    --install=datasette-cluster-map>=0.10 \
    --title "Dogsheep Photos demo"
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create a public demo 621332242
631251707 https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631251707 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25 MDEyOklzc3VlQ29tbWVudDYzMTI1MTcwNw== simonw 9599 2020-05-20T05:49:27Z 2020-05-21T15:58:42Z MEMBER

Renaming this demo to dogsheep-photos.dogsheep.net

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create a public demo 621332242
631127454 https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631127454 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25 MDEyOklzc3VlQ29tbWVudDYzMTEyNzQ1NA== simonw 9599 2020-05-19T22:48:00Z 2020-05-21T15:58:32Z MEMBER

I built #23 to help with this.

$ dogsheep-photos create-subset photos.db public.db \
    "select sha256 from apple_photos where albums like '%Public%'"

And publish with Vercel:

$ datasette publish now public.db --project dogsheep-photos \
    --about=dogsheep/dogsheep-photos \
    --about_url="https://github.com/dogsheep/dogsheep-photos" \
    --install=datasette-json-html \
    --install=datasette-cluster-map
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create a public demo 621332242
631255206 https://github.com/dogsheep/dogsheep-photos/issues/24#issuecomment-631255206 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/24 MDEyOklzc3VlQ29tbWVudDYzMTI1NTIwNg== simonw 9599 2020-05-20T06:00:25Z 2020-05-20T06:00:25Z MEMBER

This needs documentation.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Configurable URL for images 621323348
631253248 https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253248 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25 MDEyOklzc3VlQ29tbWVudDYzMTI1MzI0OA== simonw 9599 2020-05-20T05:54:18Z 2020-05-20T05:54:18Z MEMBER

https://dogsheep-photos.dogsheep.net/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create a public demo 621332242
631253136 https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253136 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25 MDEyOklzc3VlQ29tbWVudDYzMTI1MzEzNg== simonw 9599 2020-05-20T05:53:58Z 2020-05-20T05:53:58Z MEMBER

Updated deploy command:

datasette publish now public.db --project dogsheep-photos \
    --about=dogsheep/dogsheep-photos \
    --about_url="https://github.com/dogsheep/dogsheep-photos" \
    --install=datasette-json-html \
    --install=datasette-cluster-map \
    --title "Dogsheep Photos demo"
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create a public demo 621332242
631229485 https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229485 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 MDEyOklzc3VlQ29tbWVudDYzMTIyOTQ4NQ== simonw 9599 2020-05-20T04:31:02Z 2020-05-20T04:31:02Z MEMBER

https://pypi.org/project/dogsheep-photos/ is live.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename project to dogsheep-photos 621444763
631229409 https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229409 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 MDEyOklzc3VlQ29tbWVudDYzMTIyOTQwOQ== simonw 9599 2020-05-20T04:30:40Z 2020-05-20T04:30:40Z MEMBER

https://pypi.org/project/photos-to-sqlite/ now links to dogsheep-photos.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename project to dogsheep-photos 621444763
631227245 https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227245 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 MDEyOklzc3VlQ29tbWVudDYzMTIyNzI0NQ== simonw 9599 2020-05-20T04:21:38Z 2020-05-20T04:21:38Z MEMBER

I'm going to release 0.4 now.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename project to dogsheep-photos 621444763
631227020 https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227020 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 MDEyOklzc3VlQ29tbWVudDYzMTIyNzAyMA== simonw 9599 2020-05-20T04:20:48Z 2020-05-20T04:21:16Z MEMBER

Next time I push a release it will create dogsheep-photos on PyPI.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename project to dogsheep-photos 621444763
631227105 https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227105 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 MDEyOklzc3VlQ29tbWVudDYzMTIyNzEwNQ== simonw 9599 2020-05-20T04:21:06Z 2020-05-20T04:21:06Z MEMBER

Then I just need to push a final photos-to-sqlite release that updates the README to tell people about the name change.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename project to dogsheep-photos 621444763
631226953 https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226953 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 MDEyOklzc3VlQ29tbWVudDYzMTIyNjk1Mw== simonw 9599 2020-05-20T04:20:34Z 2020-05-20T04:20:34Z MEMBER

Huh, it looks like Circle CI picked up the name change automatically. https://app.circleci.com/pipelines/github/dogsheep/dogsheep-photos

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename project to dogsheep-photos 621444763
631226572 https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226572 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 MDEyOklzc3VlQ29tbWVudDYzMTIyNjU3Mg== simonw 9599 2020-05-20T04:18:52Z 2020-05-20T04:18:52Z MEMBER

Need to reconfigure Circle CI.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename project to dogsheep-photos 621444763
631226481 https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226481 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26 MDEyOklzc3VlQ29tbWVudDYzMTIyNjQ4MQ== simonw 9599 2020-05-20T04:18:29Z 2020-05-20T04:18:29Z MEMBER

I just renamed the repository.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename project to dogsheep-photos 621444763
631120771 https://github.com/dogsheep/dogsheep-photos/issues/23#issuecomment-631120771 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/23 MDEyOklzc3VlQ29tbWVudDYzMTEyMDc3MQ== simonw 9599 2020-05-19T22:32:48Z 2020-05-19T22:32:48Z MEMBER

Documentation: https://github.com/dogsheep/photos-to-sqlite/blob/e2fab012551eed05278040b5d57e7373a1b9a0bf/README.md#creating-a-subset-database

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
create-subset command for creating a publishable subset of a photos database 621280529
626941278 https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-626941278 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22 MDEyOklzc3VlQ29tbWVudDYyNjk0MTI3OA== simonw 9599 2020-05-11T20:25:58Z 2020-05-11T20:25:58Z MEMBER

Interesting - do you know if there's anything the exiftool process handles that ExifReader doesn't?

I'm actually just going to extract a subset of the EXIF data at first - since the original photo files will always be available I don't feel the need to get everything out for the first step.

My plan is to use EXIF to help support photo collections that aren't in Apple Photos - I'm going to build a database table keyed by the sha256 of each photo that extracts the camera make, lens, a few settings (ISO, aperture etc) and the GPS lat/lon.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Try out ExifReader 615626118
626395781 https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395781 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21 MDEyOklzc3VlQ29tbWVudDYyNjM5NTc4MQ== simonw 9599 2020-05-10T21:57:09Z 2020-05-10T21:57:09Z MEMBER

Yes, I just recreated my virtual environment from scratch and the error went away.

The problem occurred when I ran pip install datasette-bplist in the same virtual environment - https://github.com/simonw/datasette-bplist/blob/master/setup.py depends on bpylist which is incompatible with bpylist2.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
bpylist.archiver.CircularReference: archive has a cycle with uid(13) 615474990
626395209 https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395209 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21 MDEyOklzc3VlQ29tbWVudDYyNjM5NTIwOQ== simonw 9599 2020-05-10T21:52:42Z 2020-05-10T21:52:42Z MEMBER

Aha! It looks like I accidentally installed the old bplist into the same environment:

$ pip freeze | grep bpylist
bpylist==0.1.4
bpylist2==3.0.0
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
bpylist.archiver.CircularReference: archive has a cycle with uid(13) 615474990
626395103 https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395103 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21 MDEyOklzc3VlQ29tbWVudDYyNjM5NTEwMw== simonw 9599 2020-05-10T21:51:36Z 2020-05-10T21:51:36Z MEMBER

@RhetTbull I tried that workaround and it turns out I'm getting this error on ALL of my photos now!

It's weird: a few day ago this wasn't happening. Now it's happening to everything. I'm not sure what I might have changed.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
bpylist.archiver.CircularReference: archive has a cycle with uid(13) 615474990
626394989 https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626394989 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21 MDEyOklzc3VlQ29tbWVudDYyNjM5NDk4OQ== simonw 9599 2020-05-10T21:50:36Z 2020-05-10T21:50:36Z MEMBER

https://github.com/Marketcircle/bpylist/pull/2 looks relevant here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
bpylist.archiver.CircularReference: archive has a cycle with uid(13) 615474990
626388837 https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388837 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21 MDEyOklzc3VlQ29tbWVudDYyNjM4ODgzNw== simonw 9599 2020-05-10T20:59:32Z 2020-05-10T20:59:32Z MEMBER

So it appears it's possible for photo.place to raise that exception. A workaround could be to catch that and treat those photos as not having a place.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
bpylist.archiver.CircularReference: archive has a cycle with uid(13) 615474990
626388764 https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388764 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21 MDEyOklzc3VlQ29tbWVudDYyNjM4ODc2NA== simonw 9599 2020-05-10T20:58:52Z 2020-05-10T20:58:52Z MEMBER

More from the debugger:

> /Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py(614)place()
-> self._place = PlaceInfo5(self._info["reverse_geolocation"])

And:

> /Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/utils.py(91)osxphoto_to_row()
-> place = photo.place
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
bpylist.archiver.CircularReference: archive has a cycle with uid(13) 615474990
625947133 https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-625947133 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 MDEyOklzc3VlQ29tbWVudDYyNTk0NzEzMw== simonw 9599 2020-05-08T18:13:06Z 2020-05-08T18:13:06Z MEMBER

datasette-media will be able to handle this once I implement https://github.com/simonw/datasette-media/issues/3

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to serve thumbnailed Apple Photo from its place on disk 613006393
624408738 https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408738 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 MDEyOklzc3VlQ29tbWVudDYyNDQwODczOA== simonw 9599 2020-05-06T02:21:05Z 2020-05-06T02:21:32Z MEMBER

Here's rendering code from my hacked-together not-yet-released S3 image proxy:

from starlette.responses import Response
from PIL import Image, ExifTags
import pyheif

for ORIENTATION_TAG in ExifTags.TAGS.keys():
    if ExifTags.TAGS[ORIENTATION_TAG] == "Orientation":
        break
    ...
    # Load it into Pillow
    if ext == "heic":
        heic = pyheif.read_heif(image_response.content)
        image = Image.frombytes(mode=heic.mode, size=heic.size, data=heic.data)
    else:
        image = Image.open(io.BytesIO(image_response.content))

    # Does EXIF tell us to rotate it?
    try:
        exif = dict(image._getexif().items())
        if exif[ORIENTATION_TAG] == 3:
            image = image.rotate(180, expand=True)
        elif exif[ORIENTATION_TAG] == 6:
            image = image.rotate(270, expand=True)
        elif exif[ORIENTATION_TAG] == 8:
            image = image.rotate(90, expand=True)
    except (AttributeError, KeyError, IndexError):
        pass

    # Resize based on ?w= and ?h=, if set
    width, height = image.size
    w = request.query_params.get("w")
    h = request.query_params.get("h")
    if w is not None or h is not None:
        if h is None:
            # Set h based on w
            w = int(w)
            h = int((float(height) / width) * w)
        elif w is None:
            h = int(h)
            # Set w based on h
            w = int((float(width) / height) * h)
        w = int(w)
        h = int(h)
        image.thumbnail((w, h))

    # ?bw= converts to black and white
    if request.query_params.get("bw"):
        image = image.convert("L")

    # ?q= sets the quality - defaults to 75
    quality = 75
    q = request.query_params.get("q")
    if q and q.isdigit() and 1 <= int(q) <= 100:
        quality = int(q)

    # Output as JPEG or PNG
    output_image = io.BytesIO()
    image_type = "JPEG"
    kwargs = {"quality": quality}
    if image.format == "PNG":
        image_type = "PNG"
        kwargs = {}

    image.save(output_image, image_type, **kwargs)
    return Response(
        output_image.getvalue(),
        media_type="image/jpeg",
        headers={"cache-control": "s-maxage={}, public".format(365 * 24 * 60 * 60)},
    )
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to serve thumbnailed Apple Photo from its place on disk 613006393
624408370 https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408370 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 MDEyOklzc3VlQ29tbWVudDYyNDQwODM3MA== simonw 9599 2020-05-06T02:19:27Z 2020-05-06T02:19:27Z MEMBER

The plugin can be generalized: it can be configured to know how to take the URL path, look it up in ANY table (via a custom SQL query) to get a path on disk and then serve that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to serve thumbnailed Apple Photo from its place on disk 613006393
624408220 https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408220 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20 MDEyOklzc3VlQ29tbWVudDYyNDQwODIyMA== simonw 9599 2020-05-06T02:18:47Z 2020-05-06T02:18:47Z MEMBER

The apple_photos table has an indexed uuid column and a path column which stores the full path to that photo file on disk.

I can write a custom Datasette plugin which takes the uuid from the URL, looks up the path, then serves up a thumbnail of the jpeg or heic image file.

I'll prototype this is a one-off plugin first, then package it on PyPI for other people to install.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to serve thumbnailed Apple Photo from its place on disk 613006393
624406285 https://github.com/dogsheep/dogsheep-photos/issues/19#issuecomment-624406285 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/19 MDEyOklzc3VlQ29tbWVudDYyNDQwNjI4NQ== simonw 9599 2020-05-06T02:10:03Z 2020-05-06T02:10:03Z MEMBER

Most annoying part of this is the difficulty of actually showing a photo.

Maybe I need to run a local proxy that I can link to? A custom Datasette plugin perhaps?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
apple-photos command should work even if upload has not run 613002220
624364557 https://github.com/dogsheep/dogsheep-photos/issues/18#issuecomment-624364557 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18 MDEyOklzc3VlQ29tbWVudDYyNDM2NDU1Nw== simonw 9599 2020-05-05T23:49:18Z 2020-05-05T23:49:18Z MEMBER

Label is macos-latest

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Switch CI solution to GitHub Actions with a macOS runner 612860758
624278714 https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624278714 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17 MDEyOklzc3VlQ29tbWVudDYyNDI3ODcxNA== simonw 9599 2020-05-05T20:07:19Z 2020-05-05T20:07:19Z MEMBER

From https://hynek.me/articles/conditional-python-dependencies/ I think this will look like:

setup(
    # ...
    install_requires=[
        # ...
        "osxphotos>=0.28.13 ; sys_platform=='darwin'",
    ]
)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Only install osxphotos if running on macOS 612860531
624278090 https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624278090 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17 MDEyOklzc3VlQ29tbWVudDYyNDI3ODA5MA== simonw 9599 2020-05-05T20:06:01Z 2020-05-05T20:06:01Z MEMBER

https://www.python.org/dev/peps/pep-0508/#environment-markers I think I want sys_platform of darwin.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Only install osxphotos if running on macOS 612860531
623865250 https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623865250 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 MDEyOklzc3VlQ29tbWVudDYyMzg2NTI1MA== simonw 9599 2020-05-05T05:38:16Z 2020-05-05T05:38:16Z MEMBER

It looks like groups.content_string often has a null byte in it. I should clean this up as part of the import.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234
623863902 https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623863902 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 MDEyOklzc3VlQ29tbWVudDYyMzg2MzkwMg== simonw 9599 2020-05-05T05:31:53Z 2020-05-05T05:31:53Z MEMBER

Yes! Turning those rowid values into id with this script did the job:

import sqlite3
import sqlite_utils

conn = sqlite3.connect(
    "/Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite"
)


def all_rows(table):
    result = conn.execute("select rowid as id, * from {}".format(table))
    cols = [c[0] for c in result.description]
    for row in result.fetchall():
        yield dict(zip(cols, row))


if __name__ == "__main__":
    db = sqlite_utils.Database("psi_copy.db")
    for table in ("assets", "collections", "ga", "gc", "groups"):
        db[table].upsert_all(all_rows(table), pk="id", alter=True)

Then I ran this query:

select 
  json_object('img_src', 'https://photos.simonwillison.net/i/' || photos.sha256 || '.' || photos.ext || '?w=400') as photo,
   group_concat(strip_null_chars(groups.content_string), ' ') as words, assets.uuid_0, assets.uuid_1, to_uuid(assets.uuid_0, assets.uuid_1) as uuid
from assets join ga on assets.id = ga.assetid
join groups on ga.groupid = groups.id
join photos on photos.uuid = to_uuid(assets.uuid_0, assets.uuid_1)
where groups.category = 2024
group by assets.id
order by random() limit 10

And got these results!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234
623857417 https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623857417 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 MDEyOklzc3VlQ29tbWVudDYyMzg1NzQxNw== simonw 9599 2020-05-05T05:01:47Z 2020-05-05T05:01:47Z MEMBER

Even that didn't work - it didn't copy across the rowid values. I'm pretty sure that's what's wrong here:

sqlite3 /Users/simon/Pictures/Photos\ Library.photoslibrary/database/search/psi.sqlite 'select rowid, uuid_0, uuid_1 from assets limit 10'                                              
1619605|-9205353363298198838|4814875488794983828
1641378|-9205348195631362269|390804289838822030
1634974|-9205331524553603243|-3834026796261633148
1619083|-9205326176986145401|7563404215614709654
22131|-9205315724827218763|8370531509591906734
1645633|-9205247376092758131|-1311540150497601346
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234
623855885 https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855885 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 MDEyOklzc3VlQ29tbWVudDYyMzg1NTg4NQ== simonw 9599 2020-05-05T04:54:39Z 2020-05-05T04:54:53Z MEMBER

Trying this import mechanism instead:
sqlite3 /Users/simon/Pictures/Photos\ Library.photoslibrary/database/search/psi.sqlite .dump | grep -v 'CREATE INDEX' | grep -v 'CREATE TRIGGER' | grep -v 'CREATE VIRTUAL TABLE' | sqlite3 search.db

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234
623855841 https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855841 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 MDEyOklzc3VlQ29tbWVudDYyMzg1NTg0MQ== simonw 9599 2020-05-05T04:54:28Z 2020-05-05T04:54:28Z MEMBER

Things were not matching up for me correctly:

I think that's because my import script didn't correctly import the existing rowid values.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234
623846880 https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623846880 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 MDEyOklzc3VlQ29tbWVudDYyMzg0Njg4MA== simonw 9599 2020-05-05T04:06:08Z 2020-05-05T04:06:08Z MEMBER

This function seems to convert them into UUIDs that match my photos:

def to_uuid(uuid_0, uuid_1):
    b = uuid_0.to_bytes(8, 'little', signed=True) + uuid_1.to_bytes(8, 'little', signed=True)
    return str(uuid.UUID(bytes=b)).upper()
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234
623811131 https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623811131 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 MDEyOklzc3VlQ29tbWVudDYyMzgxMTEzMQ== simonw 9599 2020-05-05T03:16:18Z 2020-05-05T03:16:18Z MEMBER

Here's how to convert two integers unto a UUID using Java. Not sure if it's the solution I need though (or how to do the same thing in Python):

https://repl.it/repls/EuphoricSomberClasslibrary

import java.util.UUID;

class Main {
  public static void main(String[] args) {
    java.util.UUID uuid = new java.util.UUID(
      2544182952487526660L,
      -3640314103732024685L
    );
    System.out.println(
      uuid
    );
  }
}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234
623807568 https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623807568 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 MDEyOklzc3VlQ29tbWVudDYyMzgwNzU2OA== simonw 9599 2020-05-05T02:56:06Z 2020-05-05T02:56:06Z MEMBER

I'm pretty sure this is what I'm after. The groups table has what looks like identified labels in the rows with category = 2025:

Then there's a ga table that maps groups to assets:

And an assets table which looks like it has one row for every one of my photos:

One major challenge: these UUIDs are split into two integer numbers, uuid_0 and uuid_1 - but the main photos database uses regular UUIDs like this:

I need to figure out how to match up these two different UUID representations. I asked on Twitter if anyone has any ideas: https://twitter.com/simonw/status/1257500689019703296

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234
623806687 https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806687 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 MDEyOklzc3VlQ29tbWVudDYyMzgwNjY4Nw== simonw 9599 2020-05-05T02:51:16Z 2020-05-05T02:51:16Z MEMBER

Running datasette against it directly doesn't work:

simon@Simons-MacBook-Pro search % datasette psi.sqlite
Serve! files=('psi.sqlite',) (immutables=()) on port 8001
Usage: datasette serve [OPTIONS] [FILES]...

Error: Connection to psi.sqlite failed check: no such tokenizer: PSITokenizer

Instead, I created a new SQLite database with a copy of some of the key tables, like this:

sqlite-utils rows psi.sqlite groups | sqlite-utils insert /tmp/search.db groups -
sqlite-utils rows psi.sqlite assets | sqlite-utils insert /tmp/search.db assets -
sqlite-utils rows psi.sqlite ga | sqlite-utils insert /tmp/search.db ga -
sqlite-utils rows psi.sqlite collections | sqlite-utils insert /tmp/search.db collections -
sqlite-utils rows psi.sqlite gc | sqlite-utils insert /tmp/search.db gc -
sqlite-utils rows psi.sqlite lookup | sqlite-utils insert /tmp/search.db lookup -
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234
623806533 https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806533 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 MDEyOklzc3VlQ29tbWVudDYyMzgwNjUzMw== simonw 9599 2020-05-05T02:50:16Z 2020-05-05T02:50:16Z MEMBER

I figured there must be a separate database that Photos uses to store the text of the identified labels.

I used "Open Files and Ports" in Activity Monitor against the Photos app to try and spot candidates... and found /Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite - a 53MB SQLite database file.

Here's the schema of that file:

$ sqlite3 psi.sqlite .schema
CREATE TABLE word_embedding(word TEXT, extended_word TEXT, score DOUBLE);
CREATE INDEX word_embedding_index ON word_embedding(word);
CREATE VIRTUAL TABLE word_embedding_prefix USING fts5(extended_word)
/* word_embedding_prefix(extended_word) */;
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_content'(id INTEGER PRIMARY KEY, c0);
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID;
CREATE TABLE groups(category INT2, owning_groupid INT, content_string TEXT, normalized_string TEXT, lookup_identifier TEXT, token_ranges_0 INT8, token_ranges_1 INT8, UNIQUE(category, owning_groupid, content_string, lookup_identifier, token_ranges_0, token_ranges_1));
CREATE TABLE assets(uuid_0 INT, uuid_1 INT, creationDate INT, UNIQUE(uuid_0, uuid_1));
CREATE TABLE ga(groupid INT, assetid INT, PRIMARY KEY(groupid, assetid));
CREATE TABLE collections(uuid_0 INT, uuid_1 INT, startDate INT, endDate INT, title TEXT, subtitle TEXT, keyAssetUUID_0 INT, keyAssetUUID_1 INT, typeAndNumberOfAssets INT32, sortDate DOUBLE, UNIQUE(uuid_0, uuid_1));
CREATE TABLE gc(groupid INT, collectionid INT, PRIMARY KEY(groupid, collectionid));
CREATE VIRTUAL TABLE prefix USING fts5(content='groups', normalized_string, category UNINDEXED, tokenize = 'PSITokenizer');
CREATE TABLE IF NOT EXISTS 'prefix_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE IF NOT EXISTS 'prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE IF NOT EXISTS 'prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID;
CREATE TABLE lookup(identifier TEXT PRIMARY KEY, category INT2);
CREATE TRIGGER trigger_groups_insert AFTER INSERT ON groups BEGIN INSERT INTO prefix(rowid, normalized_string, category) VALUES (new.rowid, new.normalized_string, new.category); END;
CREATE TRIGGER trigger_groups_delete AFTER DELETE ON groups BEGIN INSERT INTO prefix(prefix, rowid, normalized_string, category) VALUES('delete', old.rowid, old.normalized_string, old.category); END;
CREATE INDEX group_pk ON groups(category, content_string, normalized_string, lookup_identifier);
CREATE INDEX asset_pk ON assets(uuid_0, uuid_1);
CREATE INDEX ga_assetid ON ga(assetid, groupid);
CREATE INDEX collection_pk ON collections(uuid_0, uuid_1);
CREATE INDEX gc_collectionid ON gc(collectionid);
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234
623806085 https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806085 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 MDEyOklzc3VlQ29tbWVudDYyMzgwNjA4NQ== simonw 9599 2020-05-05T02:47:18Z 2020-05-05T02:47:18Z MEMBER

In https://github.com/RhetTbull/osxphotos/issues/121#issuecomment-623249263 Rhet Turnbull spotted a table called ZSCENEIDENTIFIER which looked like it might have the right data, but the columns in it aren't particularly helpful:

Z_PK,Z_ENT,Z_OPT,ZSCENEIDENTIFIER,ZASSETATTRIBUTES,ZCONFIDENCE
8,49,1,731,5,0.11834716796875
9,49,1,684,6,0.0233648251742125
10,49,1,1702,1,0.026153564453125

I love the look of those confidence scores, but what do the numbers mean?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234
623805823 https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623805823 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16 MDEyOklzc3VlQ29tbWVudDYyMzgwNTgyMw== simonw 9599 2020-05-05T02:45:56Z 2020-05-05T02:45:56Z MEMBER

I filed an issue with osxphotos about this here: https://github.com/RhetTbull/osxphotos/issues/121

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import machine-learning detected labels (dog, llama etc) from Apple Photos 612287234
623739934 https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623739934 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15 MDEyOklzc3VlQ29tbWVudDYyMzczOTkzNA== simonw 9599 2020-05-04T22:24:26Z 2020-05-04T22:24:26Z MEMBER

Twitter thread with some examples of photos that are coming up from queries against these scores: https://twitter.com/simonw/status/1257434670750408705

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Expose scores from ZCOMPUTEDASSETATTRIBUTES 612151767
623730934 https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623730934 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15 MDEyOklzc3VlQ29tbWVudDYyMzczMDkzNA== simonw 9599 2020-05-04T22:00:38Z 2020-05-04T22:00:48Z MEMBER

Here's the query to create the new table:

create table apple_photos_scores as select
    ZGENERICASSET.ZUUID,
    ZGENERICASSET.ZOVERALLAESTHETICSCORE,
    ZGENERICASSET.ZCURATIONSCORE,
    ZGENERICASSET.ZPROMOTIONSCORE,
    ZGENERICASSET.ZHIGHLIGHTVISIBILITYSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZBEHAVIORALSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZFAILURESCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZHARMONIOUSCOLORSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZIMMERSIVENESSSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZINTERACTIONSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZINTERESTINGSUBJECTSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZINTRUSIVEOBJECTPRESENCESCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZLIVELYCOLORSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZLOWLIGHT,
    ZCOMPUTEDASSETATTRIBUTES.ZNOISESCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCAMERATILTSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCOMPOSITIONSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTLIGHTINGSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPATTERNSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPERSPECTIVESCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPOSTPROCESSINGSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTREFLECTIONSSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTSYMMETRYSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZSHARPLYFOCUSEDSUBJECTSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZTASTEFULLYBLURREDSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZWELLCHOSENSUBJECTSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZWELLFRAMEDSUBJECTSCORE,
    ZCOMPUTEDASSETATTRIBUTES.ZWELLTIMEDSHOTSCORE
from
    attached.ZGENERICASSET
      join attached.ZCOMPUTEDASSETATTRIBUTES on
          attached.ZGENERICASSET.Z_PK = attached.ZCOMPUTEDASSETATTRIBUTES.Z_PK;
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Expose scores from ZCOMPUTEDASSETATTRIBUTES 612151767
623723687 https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723687 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15 MDEyOklzc3VlQ29tbWVudDYyMzcyMzY4Nw== simonw 9599 2020-05-04T21:43:06Z 2020-05-04T21:43:06Z MEMBER

It looks like I can map the photos I'm importing to these tables using the ZUUID column on ZGENERICASSET to get a Z_PK which then maps to the rows in ZGENERICASSET.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Expose scores from ZCOMPUTEDASSETATTRIBUTES 612151767
623723026 https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723026 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15 MDEyOklzc3VlQ29tbWVudDYyMzcyMzAyNg== simonw 9599 2020-05-04T21:41:30Z 2020-05-04T21:41:30Z MEMBER

I'm going to put these in a table called apple_photos_scores - I'll also pull in the following columns from the ZGENERICASSET table:

  • ZOVERALLAESTHETICSCORE
  • ZCURATIONSCORE
  • ZHIGHLIGHTVISIBILITYSCORE
  • ZPROMOTIONSCORE
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Expose scores from ZCOMPUTEDASSETATTRIBUTES 612151767
623232984 https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623232984 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 MDEyOklzc3VlQ29tbWVudDYyMzIzMjk4NA== simonw 9599 2020-05-04T02:41:32Z 2020-05-04T02:41:32Z MEMBER

Needs documentation.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import photo metadata from Apple Photos into SQLite 602533300
623199750 https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199750 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 MDEyOklzc3VlQ29tbWVudDYyMzE5OTc1MA== simonw 9599 2020-05-03T23:17:58Z 2020-05-03T23:17:58Z MEMBER

Reading this source code is really useful for figuring out how to store a photo in a DB table: https://github.com/RhetTbull/osxphotos/blob/7444b6d173918a3ad2a07aefce5ecf054786c787/osxphotos/photoinfo.py

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import photo metadata from Apple Photos into SQLite 602533300
623199701 https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199701 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 MDEyOklzc3VlQ29tbWVudDYyMzE5OTcwMQ== simonw 9599 2020-05-03T23:17:38Z 2020-05-03T23:17:38Z MEMBER

Record burst_uuid as a column:

(Pdb) with_bursts[0]._info["burstUUID"]
'703FAA23-57BF-40B4-8A33-D9CEB143391B'
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import photo metadata from Apple Photos into SQLite 602533300
623199214 https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199214 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 MDEyOklzc3VlQ29tbWVudDYyMzE5OTIxNA== simonw 9599 2020-05-03T23:14:08Z 2020-05-03T23:14:08Z MEMBER

Albums have UUIDs:

(Pdb) photo.album_info[0].__dict__
{'_uuid': '17816791-ABF3-447B-942C-9FA8065EEBBA', '_db': osxphotos.PhotosDB(dbfile='/Users/simon/Pictures/Photos Library.photoslibrary/database/photos.db'), '_title': 'Geotaggable Photos geotagged'}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import photo metadata from Apple Photos into SQLite 602533300
623198986 https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198986 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 MDEyOklzc3VlQ29tbWVudDYyMzE5ODk4Ng== simonw 9599 2020-05-03T23:12:31Z 2020-05-03T23:12:46Z MEMBER

To get the taken date in UTC:

from datetime import timezone
(Pdb) photo.date.astimezone(timezone.utc).isoformat()
'2018-02-13T20:21:31.620000+00:00'
(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(".")
['2018-02-13T20:21:31', '620000+00:00']
(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(".")[0]
'2018-02-13T20:21:31'
(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(".")[0] + "+00:00"
'2018-02-13T20:21:31+00:00'
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import photo metadata from Apple Photos into SQLite 602533300
623198653 https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198653 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 MDEyOklzc3VlQ29tbWVudDYyMzE5ODY1Mw== simonw 9599 2020-05-03T23:09:57Z 2020-05-03T23:09:57Z MEMBER

For locations: I'll add place_x columns for all of these:

(Pdb) photo.place.address._asdict()
{'street': None, 'sub_locality': None, 'city': 'Loreto', 'sub_administrative_area': 'Loreto', 'state_province': 'BCS', 'postal_code': None, 'country': 'Mexico', 'iso_country_code': 'MX'}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import photo metadata from Apple Photos into SQLite 602533300
623195197 https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623195197 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 MDEyOklzc3VlQ29tbWVudDYyMzE5NTE5Nw== simonw 9599 2020-05-03T22:44:33Z 2020-05-03T22:44:33Z MEMBER

Command will be this:

$ photos-to-sqlite apple-photos photos.db

This will populate a apple_photos table with the data imported by the osxphotos library, plus the calculated sha256.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import photo metadata from Apple Photos into SQLite 602533300
623193947 https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623193947 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1 MDEyOklzc3VlQ29tbWVudDYyMzE5Mzk0Nw== simonw 9599 2020-05-03T22:36:17Z 2020-05-03T22:36:17Z MEMBER

I'm going to use osxphotos for this.

Since I've already got code to upload photos and insert them into a table based on their sha256 hash, my first go at this will be to import data using the tool and foreign-key it to the sha256 hash in the existing table.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Import photo metadata from Apple Photos into SQLite 602533300
623038378 https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038378 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38 MDEyOklzc3VlQ29tbWVudDYyMzAzODM3OA== simonw 9599 2020-05-03T01:21:13Z 2020-05-03T01:21:13Z MEMBER

No this is really useful feedback! I'm so close to this project that I miss what's not obvious to people dropping in for the first time.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
[Feature Request] Support Repo Name in Search 🥺 611284481
623027889 https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623027889 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38 MDEyOklzc3VlQ29tbWVudDYyMzAyNzg4OQ== simonw 9599 2020-05-02T23:15:11Z 2020-05-02T23:15:11Z MEMBER

This is one of the use-cases for the repos_starred view: it allows you to easily run this kid of query without having to construct the SQL by hand. Here's a demo:

https://github-to-sqlite.dogsheep.net/github/repos_starred?name__contains=twitter

My philosophy here is to keep the raw tables (like stars) as normalized as possible, then use SQL views which expose the data in a form that's easier to query.

{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
[Feature Request] Support Repo Name in Search 🥺 611284481
623010272 https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623010272 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 MDEyOklzc3VlQ29tbWVudDYyMzAxMDI3Mg== simonw 9599 2020-05-02T20:39:14Z 2020-05-02T20:39:14Z MEMBER

Graph of cumulative stars for Datasette over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29#g.mark=line&g.x_column=yyyymmdd&g.x_type=temporal&g.y_column=cumulative_count&g.y_type=quantitative

Stars per day (as a label bar chart, so very wide):

https://github-to-sqlite.dogsheep.net/github?sql=%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++#g.mark=bar&g.x_column=yyyymmdd&g.x_type=ordinal&g.y_column=n&g.y_type=quantitative

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command to fetch stargazers for one or more repos 493670730
623007441 https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623007441 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 MDEyOklzc3VlQ29tbWVudDYyMzAwNzQ0MQ== simonw 9599 2020-05-02T20:13:37Z 2020-05-02T20:13:37Z MEMBER

Datasette cumulative stars over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command to fetch stargazers for one or more repos 493670730
623006154 https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623006154 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 MDEyOklzc3VlQ29tbWVudDYyMzAwNjE1NA== simonw 9599 2020-05-02T20:01:39Z 2020-05-02T20:01:54Z MEMBER

Needs tests and documentation. I shipped it early to check that the live demo works.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command to fetch stargazers for one or more repos 493670730
623006004 https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-623006004 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8 MDEyOklzc3VlQ29tbWVudDYyMzAwNjAwNA== simonw 9599 2020-05-02T20:00:26Z 2020-05-02T20:00:26Z MEMBER

I'm abandoning this in favour of a new implementation.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
stargazers command, refs #4 516763727
623004836 https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623004836 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 MDEyOklzc3VlQ29tbWVudDYyMzAwNDgzNg== simonw 9599 2020-05-02T19:49:51Z 2020-05-02T19:49:51Z MEMBER

Alternative pattern:

sqlite-utils releases.db 'select full_name from repos' --csv --no-headers \
  | tr -d '\r' \
  | xargs github-to-sqlite stargazers stars.db
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command to fetch stargazers for one or more repos 493670730
623000814 https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623000814 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 MDEyOklzc3VlQ29tbWVudDYyMzAwMDgxNA== simonw 9599 2020-05-02T19:15:23Z 2020-05-02T19:15:23Z MEMBER

I'm not going to do the --sql bit just yet. I have patterns for working around this for other commands which are working fine:

https://github.com/dogsheep/github-to-sqlite/blob/d00a53061556dc403c166b443d141c4e1adbd64a/.github/workflows/deploy-demo.yml#L53-L70

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command to fetch stargazers for one or more repos 493670730
622998813 https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw== simonw 9599 2020-05-02T18:58:17Z 2020-05-02T18:58:17Z MEMBER

Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add this view for seeing new releases 520756546
622997410 https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622997410 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33 MDEyOklzc3VlQ29tbWVudDYyMjk5NzQxMA== simonw 9599 2020-05-02T18:46:10Z 2020-05-02T18:46:10Z MEMBER

Documented here: https://github.com/dogsheep/github-to-sqlite/blob/10fb34de41aaa35681f08b5991540d65bfcf2e2e/README.md#authentication

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fall back to authentication via ENV 609950090
622990947 https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-622990947 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4 MDEyOklzc3VlQ29tbWVudDYyMjk5MDk0Nw== simonw 9599 2020-05-02T17:54:16Z 2020-05-02T17:54:16Z MEMBER

I could add that window function query as a view, but only if the detected version of SQLite supports window functions.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command to fetch stargazers for one or more repos 493670730
622989874 https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA== simonw 9599 2020-05-02T17:46:14Z 2020-05-02T17:46:14Z MEMBER

Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add this view for seeing new releases 520756546
622982667 https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622982667 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35 MDEyOklzc3VlQ29tbWVudDYyMjk4MjY2Nw== simonw 9599 2020-05-02T16:52:53Z 2020-05-02T16:52:53Z MEMBER

Easiest option: use db.index_foreign_keys(): https://sqlite-utils.readthedocs.io/en/stable/python-api.html#adding-indexes-for-all-foreign-keys

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create index on issue_comments(user) and other foreign keys 610511450
622982346 https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622982346 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36 MDEyOklzc3VlQ29tbWVudDYyMjk4MjM0Ng== simonw 9599 2020-05-02T16:50:31Z 2020-05-02T16:50:31Z MEMBER

Demo: https://github-to-sqlite.dogsheep.net/github/dependent_repos

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add view for better display of dependent repos 610842926
622980203 https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622980203 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10 MDEyOklzc3VlQ29tbWVudDYyMjk4MDIwMw== simonw 9599 2020-05-02T16:34:29Z 2020-05-02T16:34:29Z MEMBER

Fixed definition:

select
  stars.starred_at,
  starring_user.login as starred_by,
  repos.*
from
  repos
  join stars on repos.id = stars.repo
  join users as starring_user on stars.user = starring_user.id
  join users on repos.owner = users.id
order by
  starred_at desc;
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add this repos_starred view 516967682
622978173 https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622978173 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37 MDEyOklzc3VlQ29tbWVudDYyMjk3ODE3Mw== simonw 9599 2020-05-02T16:19:31Z 2020-05-02T16:19:47Z MEMBER

I can use the new .create_view(..., replace=True) parameter in sqlite-utils 2.7.2 for this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for creating views if they don't yet exist 610843136
622461948 https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461948 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37 MDEyOklzc3VlQ29tbWVudDYyMjQ2MTk0OA== simonw 9599 2020-05-01T16:36:42Z 2020-05-01T16:36:42Z MEMBER

It should only create views if the underlying tables exist.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for creating views if they don't yet exist 610843136
622461537 https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461537 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37 MDEyOklzc3VlQ29tbWVudDYyMjQ2MTUzNw== simonw 9599 2020-05-01T16:35:40Z 2020-05-01T16:35:40Z MEMBER

This will check if the view exists and has the exact same matching definition as the one we want. If it doesn't, we will drop it (if it exists) and recreate it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for creating views if they don't yet exist 610843136
622461223 https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12 MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw== simonw 9599 2020-05-01T16:34:52Z 2020-05-01T16:34:52Z MEMBER

Blocked on #37

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add this view for seeing new releases 520756546
622461122 https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622461122 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10 MDEyOklzc3VlQ29tbWVudDYyMjQ2MTEyMg== simonw 9599 2020-05-01T16:34:39Z 2020-05-01T16:34:39Z MEMBER

Blocked on #37

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add this repos_starred view 516967682
622461025 https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622461025 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36 MDEyOklzc3VlQ29tbWVudDYyMjQ2MTAyNQ== simonw 9599 2020-05-01T16:34:24Z 2020-05-01T16:34:24Z MEMBER

Blocked on #37

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add view for better display of dependent repos 610842926
622214262 https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622214262 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35 MDEyOklzc3VlQ29tbWVudDYyMjIxNDI2Mg== simonw 9599 2020-05-01T02:10:32Z 2020-05-01T02:11:19Z MEMBER

This sped that query up even more - down to 4ms.

create index issue_comments_issue on issue_comments(issue);
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create index on issue_comments(user) and other foreign keys 610511450
622213950 https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622213950 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35 MDEyOklzc3VlQ29tbWVudDYyMjIxMzk1MA== simonw 9599 2020-05-01T02:09:04Z 2020-05-01T02:09:04Z MEMBER

It sped up this query a lot - 2.5s down to 300ms:

select
  repos.full_name,
  json_object(
    'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number,
    'label', '#' || issues.number
  ) as issue,
  issues.title,
  users.login,
  users.id,
  issues.state,
  issues.locked,
  issues.assignee,
  issues.milestone,
  issues.comments,
  issues.created_at,
  issues.updated_at,
  issues.closed_at,
  issues.author_association,
  issues.pull_request,
  issues.repo,
  issues.type
from
  issues
  join repos on repos.id = issues.repo
  join users on issues.user = users.id
where
  issues.state = 'open'
  and issues.user not in (9599, 27856297)
  and not exists (
    select
      id
    from
      issue_comments
    where
      issue_comments.user = 9599
      and issues.id = issue_comments.issue
  )
order by
  issues.updated_at desc;
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create index on issue_comments(user) and other foreign keys 610511450
622171097 https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622171097 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33 MDEyOklzc3VlQ29tbWVudDYyMjE3MTA5Nw== simonw 9599 2020-04-30T23:22:45Z 2020-04-30T23:23:57Z MEMBER

The auth.json mechanism this uses is standard across all of the other Dogsheep tools - it's actually designed so you can have one auth.json with a bunch of different credentials for different tools:

{
    "goodreads_personal_token": "...",
    "goodreads_user_id": "...",
    "github_personal_token": "...",
    "pocket_consumer_key": "...",
    "pocket_username": "...",
    "pocket_access_token": "..."
}

But... github-to-sqlite does feel like it deserves a special case here, since it's such a good fit for running inside of GitHub Actions - which even provide a GITHUB_TOKEN for you to use!

So I don't think it will harm the family of tools too much if this has an environment variable alternative to the -a file.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fall back to authentication via ENV 609950090
622169728 https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622169728 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33 MDEyOklzc3VlQ29tbWVudDYyMjE2OTcyOA== simonw 9599 2020-04-30T23:18:51Z 2020-04-30T23:18:51Z MEMBER

Sure, that sounds fine to me.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fall back to authentication via ENV 609950090
622162835 https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622162835 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 MDEyOklzc3VlQ29tbWVudDYyMjE2MjgzNQ== simonw 9599 2020-04-30T22:59:26Z 2020-04-30T22:59:26Z MEMBER

Documentation: https://github.com/dogsheep/github-to-sqlite/blob/c9f48404481882e8b3af06f35e4801a80ac79ed6/README.md#scraping-dependents-for-a-repository

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command for retrieving dependents for a repo 610408908
622135654 https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622135654 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 MDEyOklzc3VlQ29tbWVudDYyMjEzNTY1NA== simonw 9599 2020-04-30T21:53:44Z 2020-04-30T21:56:06Z MEMBER

I think this is the neatest scraping pattern:

[a["href"].lstrip("/") for a in soup.select("a[data-hovercard-type=repository]")]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command for retrieving dependents for a repo 610408908
622136585 https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622136585 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 MDEyOklzc3VlQ29tbWVudDYyMjEzNjU4NQ== simonw 9599 2020-04-30T21:55:51Z 2020-04-30T21:55:51Z MEMBER

And to find the "Next" pagination link:

soup.select(".paginate-container")[0].find("a", text="Next")
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command for retrieving dependents for a repo 610408908
622133775 https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133775 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 MDEyOklzc3VlQ29tbWVudDYyMjEzMzc3NQ== simonw 9599 2020-04-30T21:49:27Z 2020-04-30T21:49:27Z MEMBER

Proposed command:

github-to-sqlite scrape-dependents github.db simonw/datasette

I'll pull full details of the scraped repos from the regular API. I'll also record when they were "first seen" by the command.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command for retrieving dependents for a repo 610408908
622133422 https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133422 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 MDEyOklzc3VlQ29tbWVudDYyMjEzMzQyMg== simonw 9599 2020-04-30T21:48:39Z 2020-04-30T21:48:39Z MEMBER

It looks like the only option is to scrape them. I'll do that and then replace with an API as soon as one becomes available.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command for retrieving dependents for a repo 610408908
622133298 https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133298 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34 MDEyOklzc3VlQ29tbWVudDYyMjEzMzI5OA== simonw 9599 2020-04-30T21:48:24Z 2020-04-30T21:48:24Z MEMBER

Unfortunately it's not available through any GitHub API - I managed to figure out how to get dependencies, but I need dependents. https://github.com/simonw/til/blob/master/github/dependencies-graphql-api.md

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Command for retrieving dependents for a repo 610408908
620774507 https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620774507 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14 MDEyOklzc3VlQ29tbWVudDYyMDc3NDUwNw== simonw 9599 2020-04-28T18:19:06Z 2020-04-28T18:19:06Z MEMBER

The default timeout is a bit aggressive and sometimes failed for me if my resizing proxy took too long to fetch and resize the image.

client.annotate_image(..., timeout=3.0) may be worth trying.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Annotate photos using the Google Cloud Vision API 608512747
620771067 https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620771067 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14 MDEyOklzc3VlQ29tbWVudDYyMDc3MTA2Nw== simonw 9599 2020-04-28T18:12:34Z 2020-04-28T18:15:38Z MEMBER

Python library docs: https://googleapis.dev/python/vision/latest/index.html

I'm creating a new project for this called simonwillison-photos: https://console.cloud.google.com/projectcreate

https://console.cloud.google.com/home/dashboard?project=simonwillison-photos

Then I enabled the Vision API. The direct link to https://console.cloud.google.com/flows/enableapi?apiid=vision-json.googleapis.com which they provided in the docs didn't work - it gave me a "You don't have sufficient permissions to use the requested API" error - but starting at the "Enable APIs" page and searching for it worked fine.

I created a new service account as an "owner" of that project: https://console.cloud.google.com/apis/credentials/serviceaccountkey (and complained about it on Twitter and through their feedback form)

pip install google-cloud-vision

from google.cloud import vision
client = vision.ImageAnnotatorClient.from_service_account_file("simonwillison-photos-18c570b301fe.json")
# Photo of a lemur
response = client.annotate_image(
    {
        "image": {
            "source": {
                "image_uri": "https://photos.simonwillison.net/i/1b3414ee9ade67ce04ade9042e6d4b433d1e523c9a16af17f490e2c0a619755b.jpeg"
            }
        },
        "features": [
            {"type": vision.enums.Feature.Type.IMAGE_PROPERTIES},
            {"type": vision.enums.Feature.Type.OBJECT_LOCALIZATION},
            {"type": vision.enums.Feature.Type.LABEL_DETECTION},
        ],
    }
)
response

Output is:

label_annotations {
  mid: "/m/09686"
  description: "Vertebrate"
  score: 0.9851104021072388
  topicality: 0.9851104021072388
}
label_annotations {
  mid: "/m/04rky"
  description: "Mammal"
  score: 0.975814163684845
  topicality: 0.975814163684845
}
label_annotations {
  mid: "/m/01280g"
  description: "Wildlife"
  score: 0.8973650336265564
  topicality: 0.8973650336265564
}
label_annotations {
  mid: "/m/02f9pk"
  description: "Lemur"
  score: 0.8270352482795715
  topicality: 0.8270352482795715
}
label_annotations {
  mid: "/m/0fbf1m"
  description: "Terrestrial animal"
  score: 0.7443860769271851
  topicality: 0.7443860769271851
}
label_annotations {
  mid: "/m/06z_nw"
  description: "Tail"
  score: 0.6934166550636292
  topicality: 0.6934166550636292
}
label_annotations {
  mid: "/m/0b5gs"
  description: "Branch"
  score: 0.6203985214233398
  topicality: 0.6203985214233398
}
label_annotations {
  mid: "/m/05s2s"
  description: "Plant"
  score: 0.585474967956543
  topicality: 0.585474967956543
}
label_annotations {
  mid: "/m/089v3"
  description: "Zoo"
  score: 0.5488107800483704
  topicality: 0.5488107800483704
}
label_annotations {
  mid: "/m/02tcwp"
  description: "Trunk"
  score: 0.5200017690658569
  topicality: 0.5200017690658569
}
image_properties_annotation {
  dominant_colors {
    colors {
      color {
        red: 172.0
        green: 146.0
        blue: 116.0
      }
      score: 0.24523821473121643
      pixel_fraction: 0.027533333748579025
    }
    colors {
      color {
        red: 54.0
        green: 50.0
        blue: 42.0
      }
      score: 0.10449723154306412
      pixel_fraction: 0.12893334031105042
    }
    colors {
      color {
        red: 141.0
        green: 121.0
        blue: 97.0
      }
      score: 0.1391485631465912
      pixel_fraction: 0.039133332669734955
    }
    colors {
      color {
        red: 28.0
        green: 25.0
        blue: 20.0
      }
      score: 0.08589499443769455
      pixel_fraction: 0.11506666988134384
    }
    colors {
      color {
        red: 87.0
        green: 82.0
        blue: 74.0
      }
      score: 0.0845794677734375
      pixel_fraction: 0.16113333404064178
    }
    colors {
      color {
        red: 121.0
        green: 117.0
        blue: 108.0
      }
      score: 0.05901569500565529
      pixel_fraction: 0.13379999995231628
    }
    colors {
      color {
        red: 94.0
        green: 83.0
        blue: 66.0
      }
      score: 0.049011144787073135
      pixel_fraction: 0.03946666792035103
    }
    colors {
      color {
        red: 155.0
        green: 117.0
        blue: 90.0
      }
      score: 0.04164913296699524
      pixel_fraction: 0.0023333332501351833
    }
    colors {
      color {
        red: 178.0
        green: 143.0
        blue: 102.0
      }
      score: 0.02993861958384514
      pixel_fraction: 0.0012666666880249977
    }
    colors {
      color {
        red: 61.0
        green: 51.0
        blue: 35.0
      }
      score: 0.027391711249947548
      pixel_fraction: 0.01953333243727684
    }
  }
}
crop_hints_annotation {
  crop_hints {
    bounding_poly {
      vertices {
        x: 2073
      }
      vertices {
        x: 4008
      }
      vertices {
        x: 4008
        y: 3455
      }
      vertices {
        x: 2073
        y: 3455
      }
    }
    confidence: 0.65625
    importance_fraction: 0.746666669845581
  }
}
localized_object_annotations {
  mid: "/m/0jbk"
  name: "Animal"
  score: 0.7008256912231445
  bounding_poly {
    normalized_vertices {
      x: 0.0390297956764698
      y: 0.26235100626945496
    }
    normalized_vertices {
      x: 0.8466796875
      y: 0.26235100626945496
    }
    normalized_vertices {
      x: 0.8466796875
      y: 0.9386426210403442
    }
    normalized_vertices {
      x: 0.0390297956764698
      y: 0.9386426210403442
    }
  }
}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Annotate photos using the Google Cloud Vision API 608512747
620772190 https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620772190 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14 MDEyOklzc3VlQ29tbWVudDYyMDc3MjE5MA== simonw 9599 2020-04-28T18:14:43Z 2020-04-28T18:14:43Z MEMBER

Database schema for this will require some thought. Just dumping the output into a JSON column isn't going to be flexible enough - I want to be able to FTS against labels and OCR text, and potentially query against other characteristics too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Annotate photos using the Google Cloud Vision API 608512747
620771698 https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620771698 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14 MDEyOklzc3VlQ29tbWVudDYyMDc3MTY5OA== simonw 9599 2020-04-28T18:13:48Z 2020-04-28T18:13:48Z MEMBER

For face detection:

    {"type": vision.enums.Feature.Type.Type.FACE_DETECTION}

For OCR:

    {"type": vision.enums.Feature.Type.DOCUMENT_TEXT_DETECTION}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Annotate photos using the Google Cloud Vision API 608512747
620769348 https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620769348 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14 MDEyOklzc3VlQ29tbWVudDYyMDc2OTM0OA== simonw 9599 2020-04-28T18:09:21Z 2020-04-28T18:09:21Z MEMBER

Pricing is pretty good: free for first 1,000 calls per month, then $1.50 per thousand after that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Annotate photos using the Google Cloud Vision API 608512747
620309185 https://github.com/dogsheep/dogsheep-photos/issues/13#issuecomment-620309185 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13 MDEyOklzc3VlQ29tbWVudDYyMDMwOTE4NQ== simonw 9599 2020-04-28T00:39:45Z 2020-04-28T00:39:45Z MEMBER

I'm going to leave this until I have the mechanism for associating a live photo video with the photo.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Also upload movie files 607888367

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 261.266ms · About: github-to-sqlite