Tweet by @{{ display.screen_name }} ({{ display.user_name }}, {{ ""{:,}"".format(display.followers_count or 0) }} followers) on {{ display.created_at }}
{{ display.full_text }}{% if display.media_urls and json.loads(display.media_urls) %} {% for url in json.loads(display.media_urls) %}
Q3 2018 Reflection & Development
... ``` Yeah that is not valid XML!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978743426,xml.etree.ElementTree.ParseError: not well-formed (invalid token), https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307019,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40,606307019,MDEyOklzc3VlQ29tbWVudDYwNjMwNzAxOQ==,9599,simonw,2020-03-30T23:34:27Z,2020-03-30T23:34:27Z,MEMBER,"The count properties available for a user are: * followers_count * friends_count * listed_count * favourites_count * statuses_count May as well track history for all of them? Should be pretty cheap to store.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590669793,Feature: record history of follower counts, https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790370485,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790370485,MDEyOklzc3VlQ29tbWVudDc5MDM3MDQ4NQ==,9599,simonw,2021-03-04T06:57:25Z,2021-03-04T06:57:48Z,MEMBER,"The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count: https://github.com/dogsheep/google-takeout-to-sqlite/blob/a3de045eba0fae4b309da21aa3119102b0efc576/google_takeout_to_sqlite/utils.py#L66-L67 I'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import, https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622171097,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622171097,MDEyOklzc3VlQ29tbWVudDYyMjE3MTA5Nw==,9599,simonw,2020-04-30T23:22:45Z,2020-04-30T23:23:57Z,MEMBER,"The `auth.json` mechanism this uses is standard across all of the other Dogsheep tools - it's actually designed so you can have one `auth.json` with a bunch of different credentials for different tools: ```json { ""goodreads_personal_token"": ""..."", ""goodreads_user_id"": ""..."", ""github_personal_token"": ""..."", ""pocket_consumer_key"": ""..."", ""pocket_username"": ""..."", ""pocket_access_token"": ""..."" } ``` But... `github-to-sqlite` does feel like it deserves a special case here, since it's such a good fit for running inside of GitHub Actions - which even provide a `GITHUB_TOKEN` for you to use! So I don't think it will harm the family of tools too much if this has an environment variable alternative to the `-a` file.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,Fall back to authentication via ENV, https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408220,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,624408220,MDEyOklzc3VlQ29tbWVudDYyNDQwODIyMA==,9599,simonw,2020-05-06T02:18:47Z,2020-05-06T02:18:47Z,MEMBER,"The `apple_photos` table has an indexed `uuid` column and a `path` column which stores the full path to that photo file on disk. I can write a custom Datasette plugin which takes the `uuid` from the URL, looks up the path, then serves up a thumbnail of the jpeg or heic image file. I'll prototype this is a one-off plugin first, then package it on PyPI for other people to install.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk, https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542832952,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19,542832952,MDEyOklzc3VlQ29tbWVudDU0MjgzMjk1Mg==,9599,simonw,2019-10-16T18:30:11Z,2019-10-16T18:30:11Z,MEMBER,"The `--since` option will derive the `since_id` from the max ID in the `timeline_tweets` table: $ twitter-to-sqlite home-timeline --since The `--since_id=xxx` option lets you specify that ID directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",506087267,since_id support for home-timeline, https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606824992,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39,606824992,MDEyOklzc3VlQ29tbWVudDYwNjgyNDk5Mg==,9599,simonw,2020-03-31T19:24:23Z,2020-03-31T19:24:23Z,MEMBER,"The `--since` option is actually used by four commands: * `user-timeline` * `home-timeline` * `mentions-timeline` * `search` All of them use the same `fetch_timeline()` utility function under the hood. I should move the logic that looks up the last `since_id` into that shared function. Question: should I have a table for each of those four methods or a single table that is used by them all? I'm leaning towards four separate tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590666760,--since feature can be confused by retweets, https://github.com/dogsheep/healthkit-to-sqlite/issues/10#issuecomment-550806302,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10,550806302,MDEyOklzc3VlQ29tbWVudDU1MDgwNjMwMg==,9599,simonw,2019-11-07T05:33:31Z,2019-11-07T05:33:31Z,MEMBER,"The XML now includes references to these new files:  ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519038979,Failed to import workout points, https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785086,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6,706785086,MDEyOklzc3VlQ29tbWVudDcwNjc4NTA4Ng==,9599,simonw,2020-10-11T23:28:50Z,2020-10-11T23:28:50Z,MEMBER,"The XML for the OCR stuff is a bit weird. Currently I'm doing this to it: https://github.com/dogsheep/evernote-to-sqlite/blob/c33d7b043a45eb3e88676e5fa3ce31755199d9f8/evernote_to_sqlite/utils.py#L70-L78 This can produce some odd results, for example: > Sure 'Sure, 'Sure. Sure, Sure. sure sure. sure ? If you If Yau [you live jive In m 1n an area devoid of natural wonders, wanders, wonders ? wonders wonders. your mind will be blown, blown' blown. blown ? -e i ? ,1 IL it ? at ? KY ? fl ft bat at Which came from this image:  The XML for that is: ```xmlTaken in {{ display.place_city }} on {{ display.date }}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results, https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329884,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,902329884,IC_kwDODEm0Qs41yHYc,9599,simonw,2021-08-20T00:01:05Z,2021-08-20T00:01:05Z,MEMBER,Maybe Click changed something which meant that this broke things when it didn't used to?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both", https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554162,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45,660554162,MDEyOklzc3VlQ29tbWVudDY2MDU1NDE2Mg==,9599,simonw,2020-07-18T22:56:58Z,2020-07-18T22:56:58Z,MEMBER,"Manually fixing the database: ``` $ wget 'https://github-to-sqlite.dogsheep.net/github.db' --2020-07-18 15:52:33-- https://github-to-sqlite.dogsheep.net/github.db Resolving github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)... 172.217.5.115 Connecting to github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)|172.217.5.115|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 14626816 (14M) [application/octet-stream] Saving to: ‘github.db’ github.db 100%[============================================================================>] 13.95M 1.22MB/s in 18s 2020-07-18 15:52:53 (773 KB/s) - ‘github.db’ saved [14626816/14626816] $ sqlite3 github.db SQLite version 3.28.0 2019-04-15 14:49:49 Enter "".help"" for usage hints. sqlite> drop table tags; sqlite> ^D $ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils dogsheep/healthkit-to-sqlite dogsheep/swarm-to-sqlite dogsheep/twitter-to-sqlite dogsheep/inaturalist-to-sqlite dogsheep/google-takeout-to-sqlite dogsheep/github-to-sqlite dogsheep/genome-to-sqlite dogsheep/pocket-to-sqlite dogsheep/hacker-news-to-sqlite dogsheep/dogsheep-photos $ sqlite-utils tables github.db --counts [{""table"": ""users"", ""count"": 4048}, {""table"": ""repos"", ""count"": 210}, ... {""table"": ""stars"", ""count"": 4140}, {""table"": ""tags"", ""count"": 188}] $ sqlite-utils rows github.db tags [{""repo"": 107914493, ""name"": ""0.45"", ""sha"": ""f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb""}, {""repo"": 107914493, ""name"": ""0.45a5"", ""sha"": ""676bb64c877d73f8ff496cef4632f5a8a5a9283c""}, ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660429601,Fix the demo - it breaks because of the tags table change, https://github.com/dogsheep/twitter-to-sqlite/pull/33#issuecomment-601871930,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/33,601871930,MDEyOklzc3VlQ29tbWVudDYwMTg3MTkzMA==,9599,simonw,2020-03-20T19:21:41Z,2020-03-20T19:21:41Z,MEMBER,Made obsolete by https://github.com/dogsheep/twitter-to-sqlite/commit/d3887d1535c3edd59d3ebf6aea4495bacd82e59b,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",561469252,Upgrade to sqlite-utils 2.2.1, https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790380839,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790380839,MDEyOklzc3VlQ29tbWVudDc5MDM4MDgzOQ==,9599,simonw,2021-03-04T07:17:05Z,2021-03-04T07:17:05Z,MEMBER,"Looks like you're doing this: ```python elif message.get_content_type() == ""text/plain"": body = message.get_payload(decode=True) ``` So presumably that decodes to a unicode string? I imagine the reason the column is a `BLOB` for me is that `sqlite-utils` determines the column type based on the first batch of items - https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1927-L1928 - and I got unlucky and had something in my first batch that wasn't a unicode string. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import, https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543271000,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10,543271000,MDEyOklzc3VlQ29tbWVudDU0MzI3MTAwMA==,9599,simonw,2019-10-17T17:05:59Z,2019-10-17T17:05:59Z,MEMBER,Looks like tqdm already does a TTY check here: https://github.com/tqdm/tqdm/blob/89b73bdc30c099c5b53725806e7edf3a121c9b3a/tqdm/std.py#L889-L890,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",492297930,Rethink progress bars for various commands, https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-669241341,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8,669241341,MDEyOklzc3VlQ29tbWVudDY2OTI0MTM0MQ==,9599,simonw,2020-08-05T14:55:14Z,2020-08-05T14:55:14Z,MEMBER,"Looks like there's a column that's not consistently there, so the table got created without it. Easiest fix is to add `alter=True` to this line: https://github.com/dogsheep/swarm-to-sqlite/blob/f4a82633da927cde672c9d9af92930bfca2e3ddf/swarm_to_sqlite/utils.py#L94 That will cause `sqlite-utils` to notice if there's a missing column and add it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648245071,Error thrown: table photos has no column named hasSticker, https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551646,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694551646,MDEyOklzc3VlQ29tbWVudDY5NDU1MTY0Ng==,9599,simonw,2020-09-17T23:22:48Z,2020-09-17T23:22:48Z,MEMBER,"Looks like its happening in a Jinja fragment template for one of the results: ``` /Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/__init__.py(169)process_results() -> output = compiled.render({**result, **{""json"": json}}) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/asyncsupport.py(71)render() -> return original_render(self, *args, **kwargs) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(1090)render() -> self.environment.handle_exception() /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(832)handle_exception() -> reraise(*rewrite_traceback_stack(source=source)) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/_compat.py(28)reraise() -> raise value.with_traceback(tb) (5)top-level template code() > /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads() -> raise TypeError(f'the JSON object must be str, bytes or bytearray, ' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'", https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426663,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426663,MDEyOklzc3VlQ29tbWVudDc0ODQyNjY2Mw==,9599,simonw,2020-12-19T06:14:06Z,2020-12-19T06:14:06Z,MEMBER,Looks like I already do that here: https://github.com/dogsheep/dogsheep-beta/blob/9ba4401017ac24ffa3bc1db38e0910ea49de7616/dogsheep_beta/__init__.py#L141-L146,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error", https://github.com/dogsheep/dogsheep-photos/issues/18#issuecomment-624364557,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18,624364557,MDEyOklzc3VlQ29tbWVudDYyNDM2NDU1Nw==,9599,simonw,2020-05-05T23:49:18Z,2020-05-05T23:49:18Z,MEMBER,Label is `macos-latest`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612860758,Switch CI solution to GitHub Actions with a macOS runner, https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221621529,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11,1221621529,IC_kwDODLZ_YM5I0HcZ,9599,simonw,2022-08-21T21:10:15Z,2022-08-21T21:11:26Z,MEMBER,"Just saw that's what's implemented here already! - #7","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1345452427,"-a option is used for ""--auth"" and for ""--all""", https://github.com/dogsheep/pocket-to-sqlite/issues/2#issuecomment-605338322,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2,605338322,MDEyOklzc3VlQ29tbWVudDYwNTMzODMyMg==,9599,simonw,2020-03-27T22:18:02Z,2020-03-27T22:18:02Z,MEMBER,Just needs documentation now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503234169,Track and use the 'since' value, https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543273540,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,543273540,MDEyOklzc3VlQ29tbWVudDU0MzI3MzU0MA==,9599,simonw,2019-10-17T17:12:51Z,2019-10-17T17:12:51Z,MEMBER,"Just importing tweets here isn't enough - how are we supposed to know which tweets were imported by which search? So I think the right thing to do here is to also create a `search_runs` table, which records each individual run of this tool (with a timestamp and the search terms used). Then have a `search_runs_tweets` m2m table which shows which Tweets were found by that search.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856967,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695856967,MDEyOklzc3VlQ29tbWVudDY5NTg1Njk2Nw==,9599,simonw,2020-09-21T00:26:59Z,2020-09-21T00:26:59Z,MEMBER,It's a shame Datasette doesn't currently have an easy way to implement sorted-by-rank keyset-paginated using a TableView or QueryView. I'll have to do this using the custom SQL query constructed in the plugin: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L8-L43,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination, https://github.com/dogsheep/github-to-sqlite/issues/79#issuecomment-1498167714,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79,1498167714,IC_kwDODFdgUs5ZTDmi,9599,simonw,2023-04-05T21:12:55Z,2023-04-05T21:12:55Z,MEMBER,"It's a rate limiting problem: https://github.com/dogsheep/github-to-sqlite/actions/runs/4622674009/jobs/8175633155 ``` File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 407, in commits utils.save_commits(db, commits, repo_full[""id""]) File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 635, in save_commits for commit in commits: File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 417, in fetch_commits for commits in paginate(url, headers): File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 478, in paginate raise GitHubError.from_response(response) github_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403) Error: Process completed with exit code 1. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1570375808,Deploy demo job is failing due to rate limit, https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693794700,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693794700,MDEyOklzc3VlQ29tbWVudDY5Mzc5NDcwMA==,9599,simonw,2020-09-17T04:02:39Z,2020-09-17T04:02:39Z,MEMBER,It would be useful if you could pass an `--accept` option to this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls, https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549096321,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3,549096321,MDEyOklzc3VlQ29tbWVudDU0OTA5NjMyMQ==,9599,simonw,2019-11-03T01:27:55Z,2019-11-03T01:28:17Z,MEMBER,"It would be neat if this could support `--since`, with that argument automatically finding the maximum tweet ID from a previous search that used the same exact arguments (using the `search_runs` table).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488833975,Command for running a search and saving tweets for that search, https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622213950,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622213950,MDEyOklzc3VlQ29tbWVudDYyMjIxMzk1MA==,9599,simonw,2020-05-01T02:09:04Z,2020-05-01T02:09:04Z,MEMBER,"It sped up this query a lot - 2.5s down to 300ms: ```sql select repos.full_name, json_object( 'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number, 'label', '#' || issues.number ) as issue, issues.title, users.login, users.id, issues.state, issues.locked, issues.assignee, issues.milestone, issues.comments, issues.created_at, issues.updated_at, issues.closed_at, issues.author_association, issues.pull_request, issues.repo, issues.type from issues join repos on repos.id = issues.repo join users on issues.user = users.id where issues.state = 'open' and issues.user not in (9599, 27856297) and not exists ( select id from issue_comments where issue_comments.user = 9599 and issues.id = issue_comments.issue ) order by issues.updated_at desc; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450,Create index on issue_comments(user) and other foreign keys, https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461948,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622461948,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTk0OA==,9599,simonw,2020-05-01T16:36:42Z,2020-05-01T16:36:42Z,MEMBER,It should only create views if the underlying tables exist.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136,Mechanism for creating views if they don't yet exist, https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112108,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17,541112108,MDEyOklzc3VlQ29tbWVudDU0MTExMjEwOA==,9599,simonw,2019-10-11T15:30:15Z,2019-10-11T15:30:15Z,MEMBER,It should delete the tables entirely. That way it will work even if the table schema has changed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",505674949,import command should empty all archive-* tables first, https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483604,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46,735483604,MDEyOklzc3VlQ29tbWVudDczNTQ4MzYwNA==,9599,simonw,2020-11-30T00:26:50Z,2020-11-30T00:26:50Z,MEMBER,"It seems like there's a lot missing from that - those aren't particularly interesting given the data that is returned. From the docs at https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#reviews it looks like each review consists of multiple comments, and the comments are where the useful material is - https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#list-comments-for-a-pull-request-review `github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews/503368921/comments --accept 'application/vnd.github.v3+json'` ```json [ { ""id"": 500603838, ""node_id"": ""MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwMzgzOA=="", ""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500603838"", ""pull_request_review_id"": 503368921, ""diff_hunk"": ""@@ -0,0 +1,370 @@\n+[\n+ {\n+ \""url\"": \""https://api.github.com/repos/simonw/datasette/pulls/571\"",\n+ \""id\"": 313384926,\n+ \""node_id\"": \""MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2\"",\n+ \""html_url\"": \""https://github.com/simonw/datasette/pull/571\"",\n+ \""diff_url\"": \""https://github.com/simonw/datasette/pull/571.diff\"",\n+ \""patch_url\"": \""https://github.com/simonw/datasette/pull/571.patch\"",\n+ \""issue_url\"": \""https://api.github.com/repos/simonw/datasette/issues/571\"",\n+ \""number\"": 571,\n+ \""state\"": \""closed\"",\n+ \""locked\"": false,\n+ \""title\"": \""detect_fts now works with alternative table escaping\"",\n+ \""user\"": {\n+ \""login\"": \""simonw\"",\n+ \""id\"": 9599,\n+ \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+ \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+ \""gravatar_id\"": \""\"",\n+ \""url\"": \""https://api.github.com/users/simonw\"",\n+ \""html_url\"": \""https://github.com/simonw\"",\n+ \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+ \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+ \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+ \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+ \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+ \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+ \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+ \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+ \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+ \""type\"": \""User\"",\n+ \""site_admin\"": false\n+ },\n+ \""body\"": \""Fixes #570\"",\n+ \""created_at\"": \""2019-09-03T00:23:39Z\"",\n+ \""updated_at\"": \""2019-09-03T00:32:28Z\"",\n+ \""closed_at\"": \""2019-09-03T00:32:28Z\"",\n+ \""merged_at\"": \""2019-09-03T00:32:28Z\"",\n+ \""merge_commit_sha\"": \""2dc5c8dc259a0606162673d394ba8cc1c6f54428\"",\n+ \""assignee\"": null,\n+ \""assignees\"": [\n+\n+ ],\n+ \""requested_reviewers\"": [\n+\n+ ],\n+ \""requested_teams\"": [\n+\n+ ],\n+ \""labels\"": [\n+\n+ ],\n+ \""milestone\"": null,\n+ \""draft\"": false,\n+ \""commits_url\"": \""https://api.github.com/repos/simonw/datasette/pulls/571/commits\"",\n+ \""review_comments_url\"": \""https://api.github.com/repos/simonw/datasette/pulls/571/comments\"",\n+ \""review_comment_url\"": \""https://api.github.com/repos/simonw/datasette/pulls/comments{/number}\"",\n+ \""comments_url\"": \""https://api.github.com/repos/simonw/datasette/issues/571/comments\"",\n+ \""statuses_url\"": \""https://api.github.com/repos/simonw/datasette/statuses/a85239f69261c10f1a9f90514c8b5d113cb94585\"",\n+ \""head\"": {\n+ \""label\"": \""simonw:detect-fts\"",\n+ \""ref\"": \""detect-fts\"",\n+ \""sha\"": \""a85239f69261c10f1a9f90514c8b5d113cb94585\"",\n+ \""user\"": {\n+ \""login\"": \""simonw\"",\n+ \""id\"": 9599,\n+ \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+ \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+ \""gravatar_id\"": \""\"",\n+ \""url\"": \""https://api.github.com/users/simonw\"",\n+ \""html_url\"": \""https://github.com/simonw\"",\n+ \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+ \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+ \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+ \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+ \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+ \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+ \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+ \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+ \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+ \""type\"": \""User\"",\n+ \""site_admin\"": false\n+ },\n+ \""repo\"": {\n+ \""id\"": 107914493,\n+ \""node_id\"": \""MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=\"",\n+ \""name\"": \""datasette\"",\n+ \""full_name\"": \""simonw/datasette\"",\n+ \""private\"": false,\n+ \""owner\"": {\n+ \""login\"": \""simonw\"",\n+ \""id\"": 9599,\n+ \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+ \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+ \""gravatar_id\"": \""\"",\n+ \""url\"": \""https://api.github.com/users/simonw\"",\n+ \""html_url\"": \""https://github.com/simonw\"",\n+ \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+ \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+ \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+ \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+ \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+ \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+ \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+ \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+ \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+ \""type\"": \""User\"",\n+ \""site_admin\"": false\n+ },\n+ \""html_url\"": \""https://github.com/simonw/datasette\"",\n+ \""description\"": \""An open source multi-tool for exploring and publishing data\"",\n+ \""fork\"": false,\n+ \""url\"": \""https://api.github.com/repos/simonw/datasette\"",\n+ \""forks_url\"": \""https://api.github.com/repos/simonw/datasette/forks\"",\n+ \""keys_url\"": \""https://api.github.com/repos/simonw/datasette/keys{/key_id}\"",\n+ \""collaborators_url\"": \""https://api.github.com/repos/simonw/datasette/collaborators{/collaborator}\"",\n+ \""teams_url\"": \""https://api.github.com/repos/simonw/datasette/teams\"",\n+ \""hooks_url\"": \""https://api.github.com/repos/simonw/datasette/hooks\"",\n+ \""issue_events_url\"": \""https://api.github.com/repos/simonw/datasette/issues/events{/number}\"",\n+ \""events_url\"": \""https://api.github.com/repos/simonw/datasette/events\"",\n+ \""assignees_url\"": \""https://api.github.com/repos/simonw/datasette/assignees{/user}\"",\n+ \""branches_url\"": \""https://api.github.com/repos/simonw/datasette/branches{/branch}\"",\n+ \""tags_url\"": \""https://api.github.com/repos/simonw/datasette/tags\"",\n+ \""blobs_url\"": \""https://api.github.com/repos/simonw/datasette/git/blobs{/sha}\"",\n+ \""git_tags_url\"": \""https://api.github.com/repos/simonw/datasette/git/tags{/sha}\"",\n+ \""git_refs_url\"": \""https://api.github.com/repos/simonw/datasette/git/refs{/sha}\"",\n+ \""trees_url\"": \""https://api.github.com/repos/simonw/datasette/git/trees{/sha}\"",\n+ \""statuses_url\"": \""https://api.github.com/repos/simonw/datasette/statuses/{sha}\"",\n+ \""languages_url\"": \""https://api.github.com/repos/simonw/datasette/languages\"",\n+ \""stargazers_url\"": \""https://api.github.com/repos/simonw/datasette/stargazers\"",\n+ \""contributors_url\"": \""https://api.github.com/repos/simonw/datasette/contributors\"",\n+ \""subscribers_url\"": \""https://api.github.com/repos/simonw/datasette/subscribers\"",\n+ \""subscription_url\"": \""https://api.github.com/repos/simonw/datasette/subscription\"",\n+ \""commits_url\"": \""https://api.github.com/repos/simonw/datasette/commits{/sha}\"",\n+ \""git_commits_url\"": \""https://api.github.com/repos/simonw/datasette/git/commits{/sha}\"",\n+ \""comments_url\"": \""https://api.github.com/repos/simonw/datasette/comments{/number}\"",\n+ \""issue_comment_url\"": \""https://api.github.com/repos/simonw/datasette/issues/comments{/number}\"",\n+ \""contents_url\"": \""https://api.github.com/repos/simonw/datasette/contents/{+path}\"",\n+ \""compare_url\"": \""https://api.github.com/repos/simonw/datasette/compare/{base}...{head}\"",\n+ \""merges_url\"": \""https://api.github.com/repos/simonw/datasette/merges\"",\n+ \""archive_url\"": \""https://api.github.com/repos/simonw/datasette/{archive_format}{/ref}\"",\n+ \""downloads_url\"": \""https://api.github.com/repos/simonw/datasette/downloads\"",\n+ \""issues_url\"": \""https://api.github.com/repos/simonw/datasette/issues{/number}\"",\n+ \""pulls_url\"": \""https://api.github.com/repos/simonw/datasette/pulls{/number}\"",\n+ \""milestones_url\"": \""https://api.github.com/repos/simonw/datasette/milestones{/number}\"",\n+ \""notifications_url\"": \""https://api.github.com/repos/simonw/datasette/notifications{?since,all,participating}\"",\n+ \""labels_url\"": \""https://api.github.com/repos/simonw/datasette/labels{/name}\"",\n+ \""releases_url\"": \""https://api.github.com/repos/simonw/datasette/releases{/id}\"",\n+ \""deployments_url\"": \""https://api.github.com/repos/simonw/datasette/deployments\"",\n+ \""created_at\"": \""2017-10-23T00:39:03Z\"",\n+ \""updated_at\"": \""2020-07-27T20:42:15Z\"",\n+ \""pushed_at\"": \""2020-07-26T01:21:05Z\"",\n+ \""git_url\"": \""git://github.com/simonw/datasette.git\"",\n+ \""ssh_url\"": \""git@github.com:simonw/datasette.git\"",\n+ \""clone_url\"": \""https://github.com/simonw/datasette.git\"",\n+ \""svn_url\"": \""https://github.com/simonw/datasette\"",\n+ \""homepage\"": \""http://datasette.readthedocs.io/\"",\n+ \""size\"": 3487,\n+ \""stargazers_count\"": 3642,\n+ \""watchers_count\"": 3642,\n+ \""language\"": \""Python\"",\n+ \""has_issues\"": true,\n+ \""has_projects\"": false,\n+ \""has_downloads\"": true,\n+ \""has_wiki\"": true,\n+ \""has_pages\"": false,\n+ \""forks_count\"": 206,\n+ \""mirror_url\"": null,\n+ \""archived\"": false,\n+ \""disabled\"": false,\n+ \""open_issues_count\"": 190,\n+ \""license\"": {\n+ \""key\"": \""apache-2.0\"",\n+ \""name\"": \""Apache License 2.0\"",\n+ \""spdx_id\"": \""Apache-2.0\"",\n+ \""url\"": \""https://api.github.com/licenses/apache-2.0\"",\n+ \""node_id\"": \""MDc6TGljZW5zZTI=\""\n+ },\n+ \""forks\"": 206,\n+ \""open_issues\"": 190,\n+ \""watchers\"": 3642,\n+ \""default_branch\"": \""master\""\n+ }\n+ },\n+ \""base\"": {\n+ \""label\"": \""simonw:master\"",\n+ \""ref\"": \""master\"",\n+ \""sha\"": \""f04deebec4f3842f7bd610cd5859de529f77d50e\"",\n+ \""user\"": {\n+ \""login\"": \""simonw\"",\n+ \""id\"": 9599,\n+ \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+ \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+ \""gravatar_id\"": \""\"",\n+ \""url\"": \""https://api.github.com/users/simonw\"",\n+ \""html_url\"": \""https://github.com/simonw\"",\n+ \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+ \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+ \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+ \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+ \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+ \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+ \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+ \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+ \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+ \""type\"": \""User\"",\n+ \""site_admin\"": false\n+ },\n+ \""repo\"": {\n+ \""id\"": 107914493,\n+ \""node_id\"": \""MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=\"",\n+ \""name\"": \""datasette\"",\n+ \""full_name\"": \""simonw/datasette\"",\n+ \""private\"": false,\n+ \""owner\"": {\n+ \""login\"": \""simonw\"",\n+ \""id\"": 9599,\n+ \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+ \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+ \""gravatar_id\"": \""\"",\n+ \""url\"": \""https://api.github.com/users/simonw\"",\n+ \""html_url\"": \""https://github.com/simonw\"",\n+ \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+ \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+ \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+ \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+ \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+ \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+ \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+ \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+ \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+ \""type\"": \""User\"",\n+ \""site_admin\"": false\n+ },\n+ \""html_url\"": \""https://github.com/simonw/datasette\"",\n+ \""description\"": \""An open source multi-tool for exploring and publishing data\"",\n+ \""fork\"": false,\n+ \""url\"": \""https://api.github.com/repos/simonw/datasette\"",\n+ \""forks_url\"": \""https://api.github.com/repos/simonw/datasette/forks\"",\n+ \""keys_url\"": \""https://api.github.com/repos/simonw/datasette/keys{/key_id}\"",\n+ \""collaborators_url\"": \""https://api.github.com/repos/simonw/datasette/collaborators{/collaborator}\"",\n+ \""teams_url\"": \""https://api.github.com/repos/simonw/datasette/teams\"",\n+ \""hooks_url\"": \""https://api.github.com/repos/simonw/datasette/hooks\"",\n+ \""issue_events_url\"": \""https://api.github.com/repos/simonw/datasette/issues/events{/number}\"",\n+ \""events_url\"": \""https://api.github.com/repos/simonw/datasette/events\"",\n+ \""assignees_url\"": \""https://api.github.com/repos/simonw/datasette/assignees{/user}\"",\n+ \""branches_url\"": \""https://api.github.com/repos/simonw/datasette/branches{/branch}\"",\n+ \""tags_url\"": \""https://api.github.com/repos/simonw/datasette/tags\"",\n+ \""blobs_url\"": \""https://api.github.com/repos/simonw/datasette/git/blobs{/sha}\"",\n+ \""git_tags_url\"": \""https://api.github.com/repos/simonw/datasette/git/tags{/sha}\"",\n+ \""git_refs_url\"": \""https://api.github.com/repos/simonw/datasette/git/refs{/sha}\"",\n+ \""trees_url\"": \""https://api.github.com/repos/simonw/datasette/git/trees{/sha}\"",\n+ \""statuses_url\"": \""https://api.github.com/repos/simonw/datasette/statuses/{sha}\"",\n+ \""languages_url\"": \""https://api.github.com/repos/simonw/datasette/languages\"",\n+ \""stargazers_url\"": \""https://api.github.com/repos/simonw/datasette/stargazers\"",\n+ \""contributors_url\"": \""https://api.github.com/repos/simonw/datasette/contributors\"",\n+ \""subscribers_url\"": \""https://api.github.com/repos/simonw/datasette/subscribers\"",\n+ \""subscription_url\"": \""https://api.github.com/repos/simonw/datasette/subscription\"",\n+ \""commits_url\"": \""https://api.github.com/repos/simonw/datasette/commits{/sha}\"",\n+ \""git_commits_url\"": \""https://api.github.com/repos/simonw/datasette/git/commits{/sha}\"",\n+ \""comments_url\"": \""https://api.github.com/repos/simonw/datasette/comments{/number}\"",\n+ \""issue_comment_url\"": \""https://api.github.com/repos/simonw/datasette/issues/comments{/number}\"",\n+ \""contents_url\"": \""https://api.github.com/repos/simonw/datasette/contents/{+path}\"",\n+ \""compare_url\"": \""https://api.github.com/repos/simonw/datasette/compare/{base}...{head}\"",\n+ \""merges_url\"": \""https://api.github.com/repos/simonw/datasette/merges\"",\n+ \""archive_url\"": \""https://api.github.com/repos/simonw/datasette/{archive_format}{/ref}\"",\n+ \""downloads_url\"": \""https://api.github.com/repos/simonw/datasette/downloads\"",\n+ \""issues_url\"": \""https://api.github.com/repos/simonw/datasette/issues{/number}\"",\n+ \""pulls_url\"": \""https://api.github.com/repos/simonw/datasette/pulls{/number}\"",\n+ \""milestones_url\"": \""https://api.github.com/repos/simonw/datasette/milestones{/number}\"",\n+ \""notifications_url\"": \""https://api.github.com/repos/simonw/datasette/notifications{?since,all,participating}\"",\n+ \""labels_url\"": \""https://api.github.com/repos/simonw/datasette/labels{/name}\"",\n+ \""releases_url\"": \""https://api.github.com/repos/simonw/datasette/releases{/id}\"",\n+ \""deployments_url\"": \""https://api.github.com/repos/simonw/datasette/deployments\"",\n+ \""created_at\"": \""2017-10-23T00:39:03Z\"",\n+ \""updated_at\"": \""2020-07-27T20:42:15Z\"",\n+ \""pushed_at\"": \""2020-07-26T01:21:05Z\"",\n+ \""git_url\"": \""git://github.com/simonw/datasette.git\"",\n+ \""ssh_url\"": \""git@github.com:simonw/datasette.git\"",\n+ \""clone_url\"": \""https://github.com/simonw/datasette.git\"",\n+ \""svn_url\"": \""https://github.com/simonw/datasette\"",\n+ \""homepage\"": \""http://datasette.readthedocs.io/\"",\n+ \""size\"": 3487,\n+ \""stargazers_count\"": 3642,\n+ \""watchers_count\"": 3642,\n+ \""language\"": \""Python\"",\n+ \""has_issues\"": true,\n+ \""has_projects\"": false,\n+ \""has_downloads\"": true,\n+ \""has_wiki\"": true,\n+ \""has_pages\"": false,\n+ \""forks_count\"": 206,\n+ \""mirror_url\"": null,\n+ \""archived\"": false,\n+ \""disabled\"": false,\n+ \""open_issues_count\"": 190,\n+ \""license\"": {\n+ \""key\"": \""apache-2.0\"",\n+ \""name\"": \""Apache License 2.0\"",\n+ \""spdx_id\"": \""Apache-2.0\"",\n+ \""url\"": \""https://api.github.com/licenses/apache-2.0\"",\n+ \""node_id\"": \""MDc6TGljZW5zZTI=\""\n+ },\n+ \""forks\"": 206,\n+ \""open_issues\"": 190,\n+ \""watchers\"": 3642,\n+ \""default_branch\"": \""master\""\n+ }\n+ },\n+ \""_links\"": {\n+ \""self\"": {\n+ \""href\"": \""https://api.github.com/repos/simonw/datasette/pulls/571\""\n+ },\n+ \""html\"": {\n+ \""href\"": \""https://github.com/simonw/datasette/pull/571\""\n+ },\n+ \""issue\"": {\n+ \""href\"": \""https://api.github.com/repos/simonw/datasette/issues/571\""\n+ },\n+ \""comments\"": {\n+ \""href\"": \""https://api.github.com/repos/simonw/datasette/issues/571/comments\""\n+ },\n+ \""review_comments\"": {\n+ \""href\"": \""https://api.github.com/repos/simonw/datasette/pulls/571/comments\""\n+ },\n+ \""review_comment\"": {\n+ \""href\"": \""https://api.github.com/repos/simonw/datasette/pulls/comments{/number}\""\n+ },\n+ \""commits\"": {\n+ \""href\"": \""https://api.github.com/repos/simonw/datasette/pulls/571/commits\""\n+ },\n+ \""statuses\"": {\n+ \""href\"": \""https://api.github.com/repos/simonw/datasette/statuses/a85239f69261c10f1a9f90514c8b5d113cb94585\""\n+ }\n+ },\n+ \""author_association\"": \""OWNER\"",\n+ \""active_lock_reason\"": null,\n+ \""merged\"": true,\n+ \""mergeable\"": null,\n+ \""rebaseable\"": null,\n+ \""mergeable_state\"": \""unknown\"",\n+ \""merged_by\"": {"", ""path"": ""tests/pull_requests.json"", ""position"": 342, ""original_position"": 342, ""commit_id"": ""3a0d5c498f9faae4e40aab204cd01b965a4f61f3"", ""user"": { ""login"": ""simonw"", ""id"": 9599, ""node_id"": ""MDQ6VXNlcjk1OTk="", ""avatar_url"": ""https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/simonw"", ""html_url"": ""https://github.com/simonw"", ""followers_url"": ""https://api.github.com/users/simonw/followers"", ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"", ""organizations_url"": ""https://api.github.com/users/simonw/orgs"", ""repos_url"": ""https://api.github.com/users/simonw/repos"", ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/simonw/received_events"", ""type"": ""User"", ""site_admin"": false }, ""body"": ""Running this should create a `merged_by` column on the `pull_requests` table which is a foreign key to the `users` table."", ""created_at"": ""2020-10-06T21:22:47Z"", ""updated_at"": ""2020-10-20T20:56:33Z"", ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500603838"", ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"", ""author_association"": ""MEMBER"", ""_links"": { ""self"": { ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500603838"" }, ""html"": { ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500603838"" }, ""pull_request"": { ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"" } }, ""original_commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c"" }, { ""id"": 500606198, ""node_id"": ""MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwNjE5OA=="", ""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606198"", ""pull_request_review_id"": 503368921, ""diff_hunk"": ""@@ -0,0 +1,124 @@\n+from github_to_sqlite import utils\n+import pytest\n+import pathlib\n+import sqlite_utils\n+from sqlite_utils.db import ForeignKey\n+import json\n+\n+\n+@pytest.fixture\n+def pull_requests():\n+ return json.load(open(pathlib.Path(__file__).parent / \""pull_requests.json\""))\n+\n+\n+@pytest.fixture\n+def db(pull_requests):\n+ db = sqlite_utils.Database(memory=True)\n+ db[\""repos\""].insert(\n+ {\""id\"": 1},\n+ pk=\""id\"",\n+ columns={\""organization\"": int, \""topics\"": str, \""name\"": str, \""description\"": str},\n+ )\n+ utils.save_pull_requests(db, pull_requests, {\""id\"": 1})\n+ return db\n+\n+\n+def test_tables(db):\n+ assert {\""pull_requests\"", \""users\"", \""repos\"", \""milestones\""} == set(\n+ db.table_names()\n+ )\n+ assert {\n+ ForeignKey(\n+ table=\""pull_requests\"", column=\""repo\"", other_table=\""repos\"", other_column=\""id\""\n+ ),\n+ ForeignKey(\n+ table=\""pull_requests\"",\n+ column=\""milestone\"",\n+ other_table=\""milestones\"",\n+ other_column=\""id\"",\n+ ),\n+ ForeignKey(\n+ table=\""pull_requests\"", column=\""assignee\"", other_table=\""users\"", other_column=\""id\""\n+ ),\n+ ForeignKey(\n+ table=\""pull_requests\"", column=\""user\"", other_table=\""users\"", other_column=\""id\""\n+ ),\n+ } == set(db[\""pull_requests\""].foreign_keys)\n+\n+\n+def test_pull_requests(db):\n+ pull_request_rows = list(db[\""pull_requests\""].rows)\n+ assert [\n+ {\n+ 'id': 313384926,\n+ 'node_id': 'MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2',\n+ 'number': 571,\n+ 'state': 'closed',\n+ 'locked': 0,\n+ 'title': 'detect_fts now works with alternative table escaping',\n+ 'user': 9599,\n+ 'body': 'Fixes #570',\n+ 'created_at': '2019-09-03T00:23:39Z',\n+ 'updated_at': '2019-09-03T00:32:28Z',\n+ 'closed_at': '2019-09-03T00:32:28Z',\n+ 'merged_at': '2019-09-03T00:32:28Z',\n+ 'merge_commit_sha': '2dc5c8dc259a0606162673d394ba8cc1c6f54428',\n+ 'assignee': None,\n+ 'milestone': None,\n+ 'draft': 0,\n+ 'head': 'a85239f69261c10f1a9f90514c8b5d113cb94585',\n+ 'base': 'f04deebec4f3842f7bd610cd5859de529f77d50e',\n+ 'author_association': 'OWNER',\n+ 'merged': 1,\n+ 'mergeable': None,\n+ 'rebaseable': None,\n+ 'mergeable_state': 'unknown',\n+ 'merged_by': '{\""login\"": \""simonw\"", \""id\"": 9599, \""node_id\"": \""MDQ6VXNlcjk1OTk=\"", \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"", \""gravatar_id\"": \""\"", \""url\"": \""https://api.github.com/users/simonw\"", \""html_url\"": \""https://github.com/simonw\"", \""followers_url\"": \""https://api.github.com/users/simonw/followers\"", \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"", \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"", \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"", \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"", \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"", \""repos_url\"": \""https://api.github.com/users/simonw/repos\"", \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"", \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"", \""type\"": \""User\"", \""site_admin\"": false}',"", ""path"": ""tests/test_pull_requests.py"", ""position"": null, ""original_position"": 76, ""commit_id"": ""3a0d5c498f9faae4e40aab204cd01b965a4f61f3"", ""user"": { ""login"": ""simonw"", ""id"": 9599, ""node_id"": ""MDQ6VXNlcjk1OTk="", ""avatar_url"": ""https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/simonw"", ""html_url"": ""https://github.com/simonw"", ""followers_url"": ""https://api.github.com/users/simonw/followers"", ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"", ""organizations_url"": ""https://api.github.com/users/simonw/orgs"", ""repos_url"": ""https://api.github.com/users/simonw/repos"", ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/simonw/received_events"", ""type"": ""User"", ""site_admin"": false }, ""body"": ""See above - this should be 9599, an integer reference to the row in the users table."", ""created_at"": ""2020-10-06T21:27:43Z"", ""updated_at"": ""2020-10-20T20:56:33Z"", ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606198"", ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"", ""author_association"": ""MEMBER"", ""_links"": { ""self"": { ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606198"" }, ""html"": { ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606198"" }, ""pull_request"": { ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"" } }, ""original_commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c"" }, { ""id"": 500606665, ""node_id"": ""MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwNjY2NQ=="", ""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606665"", ""pull_request_review_id"": 503368921, ""diff_hunk"": ""@@ -0,0 +1,124 @@\n+from github_to_sqlite import utils\n+import pytest\n+import pathlib\n+import sqlite_utils\n+from sqlite_utils.db import ForeignKey\n+import json\n+\n+\n+@pytest.fixture\n+def pull_requests():\n+ return json.load(open(pathlib.Path(__file__).parent / \""pull_requests.json\""))\n+\n+\n+@pytest.fixture\n+def db(pull_requests):\n+ db = sqlite_utils.Database(memory=True)\n+ db[\""repos\""].insert(\n+ {\""id\"": 1},\n+ pk=\""id\"",\n+ columns={\""organization\"": int, \""topics\"": str, \""name\"": str, \""description\"": str},\n+ )\n+ utils.save_pull_requests(db, pull_requests, {\""id\"": 1})\n+ return db\n+\n+\n+def test_tables(db):\n+ assert {\""pull_requests\"", \""users\"", \""repos\"", \""milestones\""} == set(\n+ db.table_names()\n+ )\n+ assert {\n+ ForeignKey(\n+ table=\""pull_requests\"", column=\""repo\"", other_table=\""repos\"", other_column=\""id\""\n+ ),\n+ ForeignKey(\n+ table=\""pull_requests\"",\n+ column=\""milestone\"",\n+ other_table=\""milestones\"",\n+ other_column=\""id\"",\n+ ),\n+ ForeignKey(\n+ table=\""pull_requests\"", column=\""assignee\"", other_table=\""users\"", other_column=\""id\""\n+ ),\n+ ForeignKey(\n+ table=\""pull_requests\"", column=\""user\"", other_table=\""users\"", other_column=\""id\""\n+ ),\n+ } == set(db[\""pull_requests\""].foreign_keys)\n+\n+\n+def test_pull_requests(db):\n+ pull_request_rows = list(db[\""pull_requests\""].rows)\n+ assert [\n+ {\n+ 'id': 313384926,"", ""path"": ""tests/test_pull_requests.py"", ""position"": null, ""original_position"": 53, ""commit_id"": ""3a0d5c498f9faae4e40aab204cd01b965a4f61f3"", ""user"": { ""login"": ""simonw"", ""id"": 9599, ""node_id"": ""MDQ6VXNlcjk1OTk="", ""avatar_url"": ""https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/simonw"", ""html_url"": ""https://github.com/simonw"", ""followers_url"": ""https://api.github.com/users/simonw/followers"", ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"", ""organizations_url"": ""https://api.github.com/users/simonw/orgs"", ""repos_url"": ""https://api.github.com/users/simonw/repos"", ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/simonw/received_events"", ""type"": ""User"", ""site_admin"": false }, ""body"": ""Minor detail: I use Black for this repo, which requires double quotes - running \""black .\"" in the root directory (with the latest version of Black) should handle this for you."", ""created_at"": ""2020-10-06T21:28:31Z"", ""updated_at"": ""2020-10-20T20:56:33Z"", ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606665"", ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"", ""author_association"": ""MEMBER"", ""_links"": { ""self"": { ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606665"" }, ""html"": { ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606665"" }, ""pull_request"": { ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"" } }, ""original_commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c"" } ] ``` That's a lot more interesting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",664485022,Feature: pull request reviews and comments, https://github.com/dogsheep/healthkit-to-sqlite/issues/5#issuecomment-513625406,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5,513625406,MDEyOklzc3VlQ29tbWVudDUxMzYyNTQwNg==,9599,simonw,2019-07-22T03:20:16Z,2019-07-22T03:20:16Z,MEMBER,"It now renders like this: ``` Importing from HealthKit [#-----------------------------------] 5% 00:01:33 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470691622,Add progress bar, https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785201,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6,706785201,MDEyOklzc3VlQ29tbWVudDcwNjc4NTIwMQ==,9599,simonw,2020-10-11T23:29:39Z,2020-10-11T23:29:39Z,MEMBER,"It looks to me like each of those `Recent photos
{% for photo in sql(""select * from apple_photos order by date desc limit 100"") %}
![]()
{% endfor %}
```
`random-photos.html`
```html
Random photos
{% for photo in sql(""with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 100"") %}
![]()
{% endfor %}
```
Now run `datasette -m metadata.yaml photos.db --template-dir=templates/`
Visit http://127.0.0.1:8001/random-photos to see some random photos or http://127.0.0.1:8002/recent-photos for recent photos.
This is using this mechanism: https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694554584,MDEyOklzc3VlQ29tbWVudDY5NDU1NDU4NA==,9599,simonw,2020-09-17T23:31:25Z,2020-09-17T23:31:25Z,MEMBER,"I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693775622,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693775622,MDEyOklzc3VlQ29tbWVudDY5Mzc3NTYyMg==,9599,simonw,2020-09-17T02:48:34Z,2020-09-17T02:48:34Z,MEMBER,I'd like a `--paginate` option that does the same thing as https://github.com/simonw/paginate-json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711074031,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711074031,MDEyOklzc3VlQ29tbWVudDcxMTA3NDAzMQ==,9599,simonw,2020-10-17T20:14:01Z,2020-10-17T20:14:01Z,MEMBER,I'd be happy to teach the tool to look for `export.xml` or `eksport.xml` - and then expand that list to other languages.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543266947,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25,543266947,MDEyOklzc3VlQ29tbWVudDU0MzI2Njk0Nw==,9599,simonw,2019-10-17T16:56:06Z,2019-10-17T16:56:06Z,MEMBER,I wrote a test that proves that this is a problem. Should be an easy fix though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",508578780,Ensure migrations don't accidentally create foreign key twice,
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660536265,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43,660536265,MDEyOklzc3VlQ29tbWVudDY2MDUzNjI2NQ==,9599,simonw,2020-07-18T20:15:12Z,2020-07-18T20:15:12Z,MEMBER,"I want to create a SQL query which shows me all of my repositories that have commits that are NOT in the most recent release.
The releases table doesn't have enough information for this because it doesn't tell you the commit hash associated with each release, just the tag: https://github-to-sqlite.dogsheep.net/github/releases","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660355904,github-to-sqlite tags command for fetching tags,
https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011972,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40,607011972,MDEyOklzc3VlQ29tbWVudDYwNzAxMTk3Mg==,9599,simonw,2020-04-01T03:49:02Z,2020-04-01T03:50:01Z,MEMBER,"I want the datetime value to look like `2020-04-01T03:34:58+00:00` (the format returned by the Twitter API which I am storing in other tables at the moment).
```
>>> datetime.utcnow().isoformat().split('.')[0] + '+00:00'
'2020-04-01T03:49:52+00:00'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",590669793,Feature: record history of follower counts,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462570187,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462570187,IC_kwDOJHON9s5XLQzL,9599,simonw,2023-03-09T18:30:24Z,2023-03-09T18:30:24Z,MEMBER,"I used ChatGPT to write this:
```
osascript -e 'tell application ""Notes""
set allFolders to folders
repeat with aFolder in allFolders
set folderId to id of aFolder
set folderName to name of aFolder
set folderContainer to container of aFolder
set folderContainerName to name of folderContainer
log ""Folder ID: "" & folderId
log ""Folder Name: "" & folderName
log ""Folder Container: "" & folderContainerName
log "" ""
--check for nested folders
if count of folders of aFolder > 0 then
set nestedFolders to folders of aFolder
repeat with aNestedFolder in nestedFolders
set nestedFolderId to id of aNestedFolder
set nestedFolderName to name of aNestedFolder
set nestedFolderContainer to container of aNestedFolder
set nestedFolderContainerName to name of nestedFolderContainer
log "" Nested Folder ID: "" & nestedFolderId
log "" Nested Folder Name: "" & nestedFolderName
log "" Nested Folder Container: "" & nestedFolderContainerName
log "" ""
end repeat
end if
end repeat
end tell
'
```
Which for my account output this:
```
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113
Folder Name: Blog posts
Folder Container: iCloud
Nested Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
Nested Folder Name: Nested inside blog posts
Nested Folder Container: Blog posts
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698
Folder Name: JSK
Folder Container: iCloud
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
Folder Name: Nested inside blog posts
Folder Container: Blog posts
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526
Folder Name: New Folder
Folder Container: iCloud
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839
Folder Name: New Folder 1
Folder Container: iCloud
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2
Folder Name: Notes
Folder Container: iCloud
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059
Folder Name: Quick Notes
Folder Container: iCloud
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283
Folder Name: UK Christmas 2022
Folder Container: iCloud
```
So I think the correct approach here is to run code at the start to list all of the folders (no need to do fancy recursion though, just a flat list with the parent containers is enough) and create a model of that hierarchy in SQLite.
Then when I import notes I can foreign key reference them back to their containing folder.
I'm tempted to use `rowid` for the foreign keys because the official IDs are pretty long.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906646452,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13,906646452,IC_kwDOEhK-wc42ClO0,9599,simonw,2021-08-26T18:34:34Z,2021-08-26T18:35:20Z,MEMBER,"I tried this ampersand fix: https://regex101.com/r/ojU2H9/1
```python
# https://regex101.com/r/ojU2H9/1
_invalid_ampersand_re = re.compile(r'&(?![a-z0-9]+;)')
def fix_bad_xml(xml):
# More fixes for things like '&' not as part of an entity
return _invalid_ampersand_re.sub('&', xml)
```
Even with that I'm still getting total garbage in the `TemporaryRedirect{}
"""""".strip()
EN_MEDIA_SCRIPT = """"""
Array.from(document.querySelectorAll('en-media')).forEach(el => {
let hash = el.getAttribute('hash');
let type = el.getAttribute('type');
let path = `/evernote/resources_data/${hash}.json?_shape=array`;
fetch(path).then(r => r.json()).then(rows => {
let b64 = rows[0].data.encoded;
let data = `data:${type};base64,${b64}`;
el.innerHTML = `"") return jinja2.Markup(TEMPLATE.format(trimmed)) @hookimpl def extra_body_script(): return EN_MEDIA_SCRIPT ``` It works! It does however demonstrate that Evernote's ""clip this webpage"" feature means there is a LOT of weird HTML that can get into a note. It looks like they've filtered out the scripts but I wouldn't bet on it - they certainly don't filter out many of the inline styles. So running Bleach is almost certainly a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from
This note includes two images.