issues

Table actions

51 rows where author_association = "MEMBER", state = "open" and user = 9599 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: comments, created_at (date), updated_at (date)

type 1

state 1

  • open · 51 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1570375808 I_kwDODFdgUs5dmgiA 79 Deploy demo job is failing due to rate limit simonw 9599 open 0     2 2023-02-03T20:05:01Z 2023-12-08T14:50:15Z   MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1888477283 I_kwDOC8SPRc5wj-Bj 38 Run `rebuild_fts` after building the index simonw 9599 open 0     0 2023-09-08T23:17:45Z 2023-09-08T23:17:45Z   MEMBER  

In: - https://github.com/simonw/datasette.io/issues/152#issuecomment-1712323347

This turned out to be the fix:

bash dogsheep-beta index dogsheep-index.db templates/dogsheep-beta.yml sqlite-utils rebuild-fts dogsheep-index.db

 dogsheep-beta 197431109 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/38/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1617602868 I_kwDOJHON9s5gaqk0 6 Character encoding problem simonw 9599 open 0     2 2023-03-09T16:44:34Z 2023-04-14T15:22:09Z   MEMBER  

I ran against a recent note with this in it:

Or just "Actions ⚙️ "

And got back:

Actions ⚙️

Pasting that into https://ftfy.vercel.app/?s=Actions+%E2%80%9A%C3%B6%C3%B4%C3%94%E2%88%8F%C3%A8+ gives this:

python s = 'Actions â\x80\x9aöôÃ\x94â\x88\x8fè' s = s.encode('latin-1') s = s.decode('utf-8') s = s.encode('macroman') s = s.decode('utf-8') print(s)

apple-notes-to-sqlite 611552758 issue     
{
    "url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1618130434 I_kwDOJHON9s5gcrYC 11 Implement a SQL view to make it easier to query files in a nested folder simonw 9599 open 0     3 2023-03-09T23:19:28Z 2023-03-09T23:24:01Z   MEMBER  

Working with nested data in SQL is tricky, can I make it easier with a view or canned query?

 apple-notes-to-sqlite 611552758 issue     
{
    "url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1617938730 I_kwDOJHON9s5gb8kq 9 Default to just storing plaintext, store HTML if `--html` is passed simonw 9599 open 0     0 2023-03-09T20:19:06Z 2023-03-09T20:19:06Z   MEMBER  

The full body version of the notes can get HUGE, due to embedded images. It turns out for my own purposes I'm usually happy with just the plaintext version.

I'm tempted to say you don't get HTML unless you pass a --html option.

 apple-notes-to-sqlite 611552758 issue     
{
    "url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/9/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1616429236 I_kwDOJHON9s5gWMC0 4 Support incremental updates simonw 9599 open 0     2 2023-03-09T05:14:00Z 2023-03-09T18:20:56Z   MEMBER  

Running this script can take several hours against a large notes database.

Would be neat if it could run against just the notes that have been modified since it last ran. Could pull the max updated date and then keep on looping until it finds one modified before then.

Problem is I don't actually know what order it iterates over the notes in.

 apple-notes-to-sqlite 611552758 issue     
{
    "url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1616440856 I_kwDOJHON9s5gWO4Y 5 Configure full text search simonw 9599 open 0     0 2023-03-09T05:20:46Z 2023-03-09T05:20:46Z   MEMBER  

FTS would be useful.

Maybe even extract the plain text from the notes to make that index easier to create, rather than creating it against the HTML. Can use the plaintext property for that.

 apple-notes-to-sqlite 611552758 issue     
{
    "url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/5/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
703246031 MDU6SXNzdWU3MDMyNDYwMzE= 51 github-to-sqlite should handle rate limits better simonw 9599 open 0     4 2020-09-17T04:01:50Z 2022-10-14T16:34:07Z   MEMBER  

From #50 - right now it will crash with an error of it hits the rate limit. Since the rate limit information (including reset time) is available in the headers it could automatically sleep and try again instead.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
727848625 MDU6SXNzdWU3Mjc4NDg2MjU= 12 Some workout columns should be float, not text simonw 9599 open 0     4 2020-10-23T02:47:02Z 2022-06-23T04:35:02Z   MEMBER   healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1071071397 I_kwDODFdgUs4_10Cl 69 View that combines issues and issue comments simonw 9599 open 0     1 2021-12-04T00:34:33Z 2021-12-04T00:34:52Z   MEMBER  

I want to see a reverse chronologically ordered interface onto both issues and comments - essentially a unified log of comments and issues opened across one or multiple projects.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
602533481 MDU6SXNzdWU2MDI1MzM0ODE= 3 Import EXIF data into SQLite - lens used, ISO, aperture etc simonw 9599 open 0   Apple Photos online and securely browsable 5324096 2 2020-04-18T19:24:31Z 2021-10-05T12:38:24Z   MEMBER  
dogsheep-photos 256834907 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
724759588 MDU6SXNzdWU3MjQ3NTk1ODg= 29 Add search highlighting snippets simonw 9599 open 0     5 2020-10-19T16:00:48Z 2021-08-26T20:23:11Z   MEMBER   dogsheep-beta 197431109 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29/reactions",
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 1,
    "rocket": 0,
    "eyes": 0
}
    
602585497 MDU6SXNzdWU2MDI1ODU0OTc= 7 Integrate image content hashing simonw 9599 open 0     2 2020-04-19T00:36:58Z 2021-08-26T02:01:01Z   MEMBER  

To spot duplicate images (where the file content differs such that the sha256 is no longer a match) it would be useful to calculate and store perceptual hashes of some sort.

 dogsheep-photos 256834907 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7/reactions",
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 1,
    "rocket": 0,
    "eyes": 0
}
    
975166271 MDU6SXNzdWU5NzUxNjYyNzE= 20 Add index on workout_points.date simonw 9599 open 0     2 2021-08-20T01:08:04Z 2021-08-20T01:12:48Z   MEMBER  

Sorting that by date makes sense for seeing most recent points, and my DB has 2.5m points in so it's an expensive sort!

 healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
952189173 MDU6SXNzdWU5NTIxODkxNzM= 3 Use HN algolia endpoint to retrieve trees simonw 9599 open 0     3 2021-07-25T03:35:27Z 2021-07-25T18:41:17Z   MEMBER  

The trees command currently has to make a request for every single comment. Algolia have an endpoint that bundles the entire thread together into a single request.

https://hn.algolia.com/api/v1/items/ID

Here's an example that loads quickly, with about 50 comments: https://hn.algolia.com/api/v1/items/27941108

It doesn't appear to use pagination at all - if a thread is big then the response is big.

I ran this search to find some stories with more than 1000 comments: https://hn.algolia.com/api/v1/search?tags=story&numericFilters=num_comments%3E=1000

Here's one: https://news.ycombinator.com/item?id=25015967 with 4759 comments. Hitting the API takes 41s and returns 3.7 MB of JSON! wget 'https://hn.algolia.com/api/v1/items/25015967' 0.03s user 0.04s system 0% cpu 41.368 total /tmp % ls -lah 25015967 -rw-r--r-- 1 simon wheel 3.7M Jul 24 20:31 25015967

 hacker-news-to-sqlite 248903544 issue     
{
    "url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
952179830 MDU6SXNzdWU5NTIxNzk4MzA= 2 Command for fetching Hacker News threads from the search API simonw 9599 open 0     4 2021-07-25T02:00:45Z 2021-07-25T03:12:57Z   MEMBER  

I want to be able to fetch every item for a domain, e.g. https://news.ycombinator.com/from?site=simonwillison.net

 hacker-news-to-sqlite 248903544 issue     
{
    "url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
897212458 MDU6SXNzdWU4OTcyMTI0NTg= 63 Ability to fetch commits from branches other than the default simonw 9599 open 0     0 2021-05-20T17:58:08Z 2021-05-20T17:58:08Z   MEMBER  

This tool is currently almost entirely ignorant of the concept of branches. One example: you can't retrieve commits from any branch other than the default (usually main).

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/63/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
496415321 MDU6SXNzdWU0OTY0MTUzMjE= 1 Figure out some interesting example SQL queries simonw 9599 open 0     9 2019-09-20T15:28:07Z 2021-05-03T03:46:23Z   MEMBER  

My knowledge of genetics has left me short here. I'd love to be able to provide some interesting example SELECT queries - maybe one that spots if you are likely to have red hair?

 genome-to-sqlite 209590345 issue     
{
    "url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
836923194 MDU6SXNzdWU4MzY5MjMxOTQ= 32 JSON API for search results simonw 9599 open 0     0 2021-03-20T22:21:36Z 2021-03-20T22:21:36Z   MEMBER   dogsheep-beta 197431109 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/32/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
821841046 MDU6SXNzdWU4MjE4NDEwNDY= 6 Upgrade to latest sqlite-utils simonw 9599 open 0     1 2021-03-04T07:21:54Z 2021-03-04T07:22:51Z   MEMBER  

This is pinned to v1 at the moment.

 google-takeout-to-sqlite 206649770 issue     
{
    "url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
753000405 MDU6SXNzdWU3NTMwMDA0MDU= 53 Command for fetching file contents simonw 9599 open 0     1 2020-11-29T20:31:04Z 2020-11-30T00:36:09Z   MEMBER  

Something like this:

github-to-sqlite files github.db simonw/datasette

This would fetch all files from the main branch into a files table.

Additional options could handle things like pulling files from a branch or tag, or just pulling files that match a specific glob or that exist in a specific directory.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/53/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
703218756 MDU6SXNzdWU3MDMyMTg3NTY= 50 Commands for making authenticated API calls simonw 9599 open 0     7 2020-09-17T02:39:07Z 2020-10-19T05:01:29Z   MEMBER  

Similar to twitter-to-sqlite fetch, see https://github.com/dogsheep/twitter-to-sqlite/issues/51

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
718938889 MDU6SXNzdWU3MTg5Mzg4ODk= 5 Figure out how to display images from <en-media> tags inline in Datasette simonw 9599 open 0     6 2020-10-11T22:17:03Z 2020-10-16T20:16:28Z   MEMBER  

Relates to #1. Evernote XML looks like this:

```xml

<en-note>

This note includes two images.

The Python logo
<en-media hash="61098c2c541de7f0a907c301dd6542da" type="image/svg+xml" width="125"/>
The Evernote logo
<en-media hash="91bd26175acac0b2ffdb6efac199f8ca" type="image/svg+xml" width="125"/>

</en-note> ``` That hash is the md5 we use to store resources. It should be possible to turn these into embedded image tags, especially if done in conjunction with the https://github.com/simonw/datasette-media plugin.

 evernote-to-sqlite 303218369 issue     
{
    "url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
718934942 MDU6SXNzdWU3MTg5MzQ5NDI= 1 Documentation on how to use this with Datasette simonw 9599 open 0     1 2020-10-11T21:56:27Z 2020-10-11T22:14:00Z   MEMBER  

In particular how to use datasette-render-images to see the images.

 evernote-to-sqlite 303218369 issue     
{
    "url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/1/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
709789634 MDU6SXNzdWU3MDk3ODk2MzQ= 27 Sort order is not persisted by facet filter links simonw 9599 open 0     0 2020-09-27T18:22:07Z 2020-09-27T18:22:07Z   MEMBER  

A link to /-/beta?category=1&timestamp__date=2018-08-01&q=swedish should be to /-/beta?category=1&timestamp__date=2018-08-01&q=swedish&sort=newest

 dogsheep-beta 197431109 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/27/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
705215230 MDU6SXNzdWU3MDUyMTUyMzA= 26 Pagination simonw 9599 open 0     7 2020-09-21T00:14:37Z 2020-09-21T02:55:54Z   MEMBER  

Useful for #16 (timeline view) since you can now filter to just the items on a specific day - but if there are more than 50 items you can't see them all.

 dogsheep-beta 197431109 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
694493566 MDU6SXNzdWU2OTQ0OTM1NjY= 16 Timeline view simonw 9599 open 0     3 2020-09-06T19:13:58Z 2020-09-21T02:42:29Z   MEMBER  

Ability to browse (and facet) by date.

 dogsheep-beta 197431109 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
694136490 MDU6SXNzdWU2OTQxMzY0OTA= 15 Add a bunch of config examples simonw 9599 open 0     1 2020-09-05T17:58:43Z 2020-09-18T23:17:39Z   MEMBER  

I can bring these over from my personal Dogsheep.

 dogsheep-beta 197431109 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/15/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
703218448 MDU6SXNzdWU3MDMyMTg0NDg= 51 Documentation for twitter-to-sqlite fetch simonw 9599 open 0     0 2020-09-17T02:38:10Z 2020-09-17T02:38:10Z   MEMBER  

It's mentioned in passing in the README but it deserves its own section: $ twitter-to-sqlite fetch \ "https://api.twitter.com/1.1/account/verify_credentials.json" \ | grep '"id"' | head -n 1

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/51/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
703216044 MDU6SXNzdWU3MDMyMTYwNDQ= 49 Feature: gists and starred gists simonw 9599 open 0     0 2020-09-17T02:30:52Z 2020-09-17T02:30:52Z   MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/49/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
695556681 MDU6SXNzdWU2OTU1NTY2ODE= 19 Figure out incremental re-indexing simonw 9599 open 0     2 2020-09-08T05:23:31Z 2020-09-08T05:27:07Z   MEMBER  

As tables get bigger reindexing everything on a schedule (essentially recreating the entire index from scratch) will start to become a performance bottleneck.

 dogsheep-beta 197431109 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
695553522 MDU6SXNzdWU2OTU1NTM1MjI= 18 Deleted records stay in the search index simonw 9599 open 0     2 2020-09-08T05:14:23Z 2020-09-08T05:15:51Z   MEMBER  

Here's why: https://github.com/dogsheep/dogsheep-beta/blob/24f7898d41a39218058f174c75ba62f7c0fcfff6/dogsheep_beta/utils.py#L44-L53

That should probably do DELETE FROM index1.search_index WHERE [table] = ? first.

 dogsheep-beta 197431109 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
692202408 MDU6SXNzdWU2OTIyMDI0MDg= 12 Idea: maps and GeoJSON support simonw 9599 open 0     0 2020-09-03T18:47:10Z 2020-09-04T01:45:03Z   MEMBER  

It would be cool if the display_sql could return a column populated with GeoJSON which would the automatically be displayed on a map in the results (or maybe default JS would look for a class="geojson" element output by the display template) - ala https://github.com/simonw/datasette-leaflet-geojson

Then I could render workout routes on a map, or Swarm checkin points.

 dogsheep-beta 197431109 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/12/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
689850810 MDU6SXNzdWU2ODk4NTA4MTA= 6 Set up a demo instance simonw 9599 open 0     0 2020-09-01T06:20:24Z 2020-09-01T06:20:24Z   MEMBER  

Once I've got the Datasette plugin to a state where it's worth building a demo: #3

I can use data from my public https://github-to-sqlite.dogsheep.net/ demo plus the Pocket data subset I use for the demo in https://github.com/dogsheep/pocket-to-sqlite/issues/5 - I could pull in the https://dogsheep-photos.dogsheep.net/ photos data too.

 dogsheep-beta 197431109 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/6/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
503243784 MDU6SXNzdWU1MDMyNDM3ODQ= 3 Extract images into separate tables simonw 9599 open 0     1 2019-10-07T05:43:01Z 2020-09-01T06:17:45Z   MEMBER  

As already done with authors. Slightly harder because images do not have a universally unique ID. Also need to figure out what to do about there being columns for both image and images.

 pocket-to-sqlite 213286752 issue     
{
    "url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
689848827 MDU6SXNzdWU2ODk4NDg4Mjc= 6 ISO timestamps simonw 9599 open 0     0 2020-09-01T06:16:42Z 2020-09-01T06:16:42Z   MEMBER  

The time_added, time_updated and time_read columns currently store data like this:

September 19, 2019 - 00:30:30 UTC

Should use ISO instead, e.g. 2020-07-26T01:05:24+00:00

 pocket-to-sqlite 213286752 issue     
{
    "url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/6/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
673602857 MDU6SXNzdWU2NzM2MDI4NTc= 9 Define a view that displays photos correctly simonw 9599 open 0     0 2020-08-05T14:53:39Z 2020-08-05T14:53:39Z   MEMBER  

The photos table stores data like this:

id | createdAt | source | prefix | suffix | width | height | visibility | created ▲ | user -- | -- | -- | -- | -- | -- | -- | -- | -- | -- 5e12c9708506bc000840262a | January 06, 2020 - 05:45:20 UTC | Swarm for iOS 1 | https://fastly.4sqi.net/img/general/ | /15889193_AXxGk4I1nbzUZuyYqObgbXdJNyEHiwj6AUDq0tPZWtw.jpg | 1920 | 1440 | public | 2020-01-06T05:45:20 | 15889193

The photo URL can be derived from those pieces - define a SQL view which does that (using datasette-json-html to display the pictures)

 swarm-to-sqlite 205429375 issue     
{
    "url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/9/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
621486115 MDU6SXNzdWU2MjE0ODYxMTU= 27 photos_with_apple_metadata view should include labels simonw 9599 open 0     0 2020-05-20T06:06:17Z 2020-05-20T06:06:17Z   MEMBER  

https://dogsheep-photos.dogsheep.net/public/photos_with_apple_metadata?place_city=New+Orleans&_facet=place_city&_facet_array=albums&_facet_array=persons

Here's one way to add that: sql select rowid, photo, ( select json_group_array( json_object( 'label', normalized_string, 'href', '/photos/labelled?_hide_sql=1&label=' || normalized_string ) ) from labels where labels.uuid = photos_with_apple_metadata.uuid ) as labels, date,

 dogsheep-photos 256834907 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/27/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
621323348 MDU6SXNzdWU2MjEzMjMzNDg= 24 Configurable URL for images simonw 9599 open 0     1 2020-05-19T22:25:56Z 2020-05-20T06:00:29Z   MEMBER   dogsheep-photos 256834907 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/24/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
615626118 MDU6SXNzdWU2MTU2MjYxMTg= 22 Try out ExifReader simonw 9599 open 0     4 2020-05-11T06:32:13Z 2020-05-14T05:59:53Z   MEMBER  

https://pypi.org/project/ExifReader/

New fork that should be able to handle EXIF in HEIC files.

Forked here: https://github.com/ianare/exif-py/issues/102#issuecomment-626376522

Refs #3

dogsheep-photos 256834907 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
612860758 MDU6SXNzdWU2MTI4NjA3NTg= 18 Switch CI solution to GitHub Actions with a macOS runner simonw 9599 open 0     1 2020-05-05T20:03:50Z 2020-05-05T23:49:18Z   MEMBER  

Refs #17.

 dogsheep-photos 256834907 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
612287234 MDU6SXNzdWU2MTIyODcyMzQ= 16 Import machine-learning detected labels (dog, llama etc) from Apple Photos simonw 9599 open 0     13 2020-05-05T02:45:43Z 2020-05-05T05:38:16Z   MEMBER  

Follow-on from #1. Apple Photos runs some very sophisticated machine learning on-device to figure out if photos are of dogs, llamas and so on. I really want to extract those labels out into my own database.

 dogsheep-photos 256834907 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16/reactions",
    "total_count": 2,
    "+1": 0,
    "-1": 0,
    "laugh": 1,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
602533300 MDU6SXNzdWU2MDI1MzMzMDA= 1 Import photo metadata from Apple Photos into SQLite simonw 9599 open 0   Apple Photos online and securely browsable 5324096 8 2020-04-18T19:23:26Z 2020-05-04T02:41:40Z   MEMBER  

Faces, albums, locations, that kind of thing.

 dogsheep-photos 256834907 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
608512747 MDU6SXNzdWU2MDg1MTI3NDc= 14 Annotate photos using the Google Cloud Vision API simonw 9599 open 0     5 2020-04-28T18:09:03Z 2020-04-28T18:19:06Z   MEMBER  

It can detect faces, run OCR, do image labeling (it knows what a lemur is!) and do object localization where it identifies objects and returns bounding polygons for them.

 dogsheep-photos 256834907 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14/reactions",
    "total_count": 3,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 1,
    "rocket": 0,
    "eyes": 0
}
    
607888367 MDU6SXNzdWU2MDc4ODgzNjc= 13 Also upload movie files simonw 9599 open 0     2 2020-04-27T22:11:25Z 2020-04-28T00:39:45Z   MEMBER  

The upload command currently only handles static images:

https://github.com/dogsheep/photos-to-sqlite/blob/d939455af00e07866686457ee2fcb9b2d1b7194e/photos_to_sqlite/utils.py#L26-L33

Need to cover movies taken by my phone and DSLR too.

 dogsheep-photos 256834907 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
606033104 MDU6SXNzdWU2MDYwMzMxMDQ= 12 If less than 500MB, show size in MB not GB simonw 9599 open 0     1 2020-04-24T04:35:01Z 2020-04-24T04:35:25Z   MEMBER  

Just saw this: Uploading 0.05 GB

 dogsheep-photos 256834907 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/12/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
602619330 MDU6SXNzdWU2MDI2MTkzMzA= 45 Use raise_for_status() everywhere simonw 9599 open 0     1 2020-04-19T04:38:28Z 2020-04-19T04:39:22Z   MEMBER  

I keep seeing errors which I think are caused by authentication or rate limit problems but which appear to be unexpected JSON responses - presumably because they are actually an error message.

Recent example: https://github.com/simonw/jsk-fellows-on-twitter/runs/598892575

Using response.raise_for_status() everywhere will make these errors less confusing.

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
530491074 MDU6SXNzdWU1MzA0OTEwNzQ= 14 Command for importing events simonw 9599 open 0     3 2019-11-29T21:28:58Z 2020-04-14T19:38:34Z   MEMBER   github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
599776345 MDU6SXNzdWU1OTk3NzYzNDU= 24 Feature idea: github-to-sqlite everything ... simonw 9599 open 0     0 2020-04-14T18:34:00Z 2020-04-14T18:34:00Z   MEMBER  

At the moment if you want to pull all your repos, issues, issues comments etc you have to do it with a sequence of separate commands.

Consider adding a everything or all command which fetches everything that the tool knows how to fetch, and is designed to be run on a cron in a way that fetches just new stuff each time.

 github-to-sqlite 207052882 issue     
{
    "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/24/reactions",
    "total_count": 7,
    "+1": 7,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
505673645 MDU6SXNzdWU1MDU2NzM2NDU= 16 Do a better job with archived direct message threads simonw 9599 open 0     0 2019-10-11T06:55:21Z 2019-10-11T06:55:27Z   MEMBER   twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/16/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
488874815 MDU6SXNzdWU0ODg4NzQ4MTU= 5 Write tests that simulate the Twitter API simonw 9599 open 0     1 2019-09-03T23:55:35Z 2019-09-03T23:56:28Z   MEMBER  

I can use betamax for this: https://pypi.org/project/betamax/

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/5/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);