releases

• Initial working version. apple-notes-to-sqlite notes.db dumps your notes out to a SQLite database. #2
• apple-notes-to-sqlite --dump outputs them to standard output as newline-delimited JSON. #3

• Optimize the FTS table and vacuum the database. #4
• First non-alpha release

• Now depends on sqlite-utils >= 3.0. #30
• The user's original search term is now passed to the display_sql SQL query as the :q parameter. #29

• Fix for error if search query contains hyphens such as github-to-sqlite. #31

• Fixed bug where searches consisting only of whitespace caused a server error. #33

• First working version

• Now creates an index on the timestamp column. #1

• Use Porter stemming by default, allow that to be over-ridden with the new --tokenize= option. #2

• New category column plus default categories table. #7

• Working faceted search interface at /-/beta #3
• is_public column for filtering between public and private data. #11
• New received category. #10

• Fixed support for advanced FTS query syntax. #13

• Incomplete preview of the /-/beta search interface. #3 #9

• Added received category. #10
• New is_public column. #11

• Custom template displays can now render a point on a map. #12

• Rerun the search query with FTS quoting if a SQLite FTS error occurs. This means searching for e.g. "#dogfest" will no longer throw an error, while still supporting advanced SQLite FTS syntax. #14
• Maximum width 100% on images in results
• Don't display facet headers if there are no results for that facet.

• Option to sort by relevance / oldest / newest. #21
• Renamed table column to type. #17
• New -d option for passing one or more databases to index. Without this all databases in the config file will be indexed.

• Fixed bug with sort orders displayed for the timeline view. #22

• Renamed table column to type. #17
• New -d option for passing one or more databases to index. Without this all databases in the config file will be indexed.

• New template_debug configuration setting. #25
• Templates now autoescape by default, use {{ display.foo|safe }} if you know you want HTML not to be escaped.
• Fixed bug where the wrong template fragment could be used for a row. #24
• Sort order is now persisted when you execute a new search. #23

• Facet by date. #16
• Now depends on Datasette 0.50.2
• Uses datasette.client and passes through authentication cookies. #28

• Added by date, as part of developing a timeline view. #16

• First alpha release

• Only upload photos not already in S3. #9

• Upload photos to S3 in a thread pool #11
• New --dry-run option to upload command
• New photos-to-sqlite apple-photos command for importing Apple Photos metadata #1
• Apple Photos quality scores are imported into apple_photos_scores table #15
• Machine learning labels imported from Apple Photos into labels table #16

• Renamed this project to dogsheep-photos. #26
• apple-photos command now works even if you haven't yet run upload. #19
• --image-url-prefix and --image-url-suffix options to apple-photos. #24
• New create-subset command. #23

• Documentation now explains how to use this with the datasette-media, datasette-json-html and datasette-template-sql plugins. #20

• Initial release, with support for importing ENEX files.

• Notes now use an ID derived from a hash of their contents. #3
• Date columns are now in ISO format, and indexd. #4
• Notes and resources table now have full-text search configured. #4
• Improved how OCR is stored in the database, only storing the most confident matches. #6

• Correctly handle entities in XHTML notes. #9
• Fixed a bug importing ENEX files on Windows. Thanks, dskrad. #11

• Explicitly open file using utf-8 encoding - thanks, RiverZhou. #10
• Fix import error if note has no "updated" element - thanks, Matej Korosec. #8

• Fixed error when encountering <en-note> documents that were not well-formed XML. #13

First release

• Fix bug in authentication handling code

• Added the github-to-sqlite starred command for retrieving starred repos, #1

• license is now extracted from the repos table into a separate licenses table with a foreign key, #2

• Added github-to-sqlite repos command, #3

• New command: github-to-sqlite issue-comments for importing comments on issues - #7
• github-to-sqlite issues now accepts optional --issue=1 argument
• Fixed bug inserting users into already-created table with wrong columns - #6

• New releases command for fetching releases for a repo, #11
• Repository topics are now fetched by the repos command
• github-to-sqlite repos now accepts multiple usernames
• Command now works without --auth file (using anonymous API calls), #9

• New github-to-sqlite commits my.db dogsheep/github-to-sqlite command. #17

• Full-text search is configured for more tables. #19
• Release assets are now pulled out into a separate assets table. #15
• Now depends on sqlite-utils 2.x. #20
• Commit raw authors are now used to populate a new raw_authors able. #18
• New live demo at https://github-to-sqlite.dogsheep.net/ #13
• GitHub API errors are now raised as Pyton exceptions. #21
• Fixed bug running commits against repos with no commits. #22

• Fixed bug where repository topics were not being correctly fetched #26
• Live demo at https://github-to-sqlite.dogsheep.net/ now uses datasette-render-markdown and pulls in issue comments #25

• New github-to-sqlite contributors command for fetching contributors to one or more repositories. #28
• The live demo now includes contributors, and pulls data from simonw/datasette and simonw/sqlite-utils in addition to the Dogsheep repositories.
• The organization column in the repos table is now a foreign key to users as opposed to a big piece of JSON. #27

This release includes some backwards-incompatible schema changes:

• The milestone and assignee columns on the issues table are now integers that are foreign keys to the milestones and users tables - previously they could be text columns instead. #30
• milestones now has an integer foreign key in the creator column, and a new repo column that is an integer foreign key to the repos table. #29
• The repo column in the issues table is now an integer foreign key to repos. Previously it was a string of the format dogsheep/github-to-sqlite. #31

• New github-to-sqlite scrape-dependents command for importing the dependent repositories for a repo. #34

• New command: github-to-sqlite stargazers (docs) for fetching all users who have starred the specified repositories. #4
• Added several views: dependent_repos, repos_starred, recent_releases. #10 #12 #36
• Added indexes on all foreign key relationships. #35
• GitHub API token can now be read from GITHUB_TOKEN environment variable. #33

• New github-to-sqlite repos my.db -r simonw/datasette option for importing just specific repositories. #42

• New github-to-sqlite tags github.db simonw/datasette command for importing all tags for a repository. #43

• New github-to-sqlite emojis command for fetching emoji supported by GitHub. Demo. #47
• Added table of contents to README.
• Switched to GitHub Actions for CI.
• Switched default branch to main

• New github-to-sqlite get /gists command for making authenticated API calls against any API path. #50

• github-to-sqlite repos command now takes options --readme and --readme-html, which write the README or rendered HTML README into the readme or readme_html columns, respectively. #52
• New --accept option for github-to-sqlite get. #50

• New command: github-to-sqlite workflows, for fetching the YAML configuration used by GitHub Actions workflows and using it to populate new workflows, jobs and steps tables. #54
• New command: github-to-sqlite pull-requests - thanks, @adamjonas! #48
• github-to-sqlite issues --issue option can now be used multiple times.

• README now links to example database tables for each command. #56
• Added documentation for github-to-sqlite releases.

• --readme now stores null if the README file does not exist, rather than throwing an error. #57
• README HTML is rewritten to fix broken internal links, e.g. for a table of contents. #58

• Minor documentation and inline help improvements.

• user and trees commands.

Compatibility with sqlite-utils 1.x

First non-alpha release. Now configures full-text search for the users and items tables. #1

Added documentation on Browsing your data with Datasette.

Progress display now shows the total as well as the number currently in the queue. % hacker-news-to-sqlite trees hn.db 26440397 done: 1, in queue: 0, total: 1 done: 2, in queue: 28, total: 30 done: 3, in queue: 27, total: 30

Fixed a bug where duplicate records could crash the import.

• Tool now displays a progress bar during import - you can disable it with --silent #5
• You can pass a path to a decompressed XML file instead of a zip file, using --xml
• Records table is now broken up into different tables for each type of recorded data #6

Uses less RAM - see #7

Fix for #9 - Too many SQL variables bug

• Fixed workout latitude/longitude points import for iOS 13 - #10

• Upgraded to latest sqlite-utils

• Fixed a bug where export files generated in languages other than English failed to import. #11
• CI now uses GitHub Actions

• Fixed bug where activity_summary table could fail to be built with an error. #19

• Upgraded to sqlite-utils 2.x
• First non-alpha release

• Tweaked dependencies for compatibility with new PyPI resolver.

Initial release

• Defaults to only retrieving updated items, unless you use --all. #2
• Now displays a progress bar for initial data import and --all.
• Data is now fetched 500 items at a time using API pagination. #1
• Now depends on sqlite-utils 2.x.
• auth.json now uses pocket_ prefix on each stored key. #4

• Tweaked dependencies for compatibility with new PyPI resolver.

• -a is now the shortcut for --auth - previously it was duplicated as a shortcut for --all. Thanks, Thomas Lockney. #7
• pocket-to-sqlite auth command now updates the existing auth.json file rather than over-writing it. #10

• Progress bar now includes a count of the total number of items to fetch.
• Now packaged using pyproject.toml. #12

First usable release.

• Added --since option, closes #3

• Upgraded to sqlite-utils 2.x

• Fixed bug with very old checkins that were missing their source - thanks, @mfa! #6

• Fix for "table photos has no column named hasSticker" bug, thanks @mattiaborsoi. #10
• Switched CI to GitHub Actions.

• Fixed bug where tool could crash with an error about missing columns. #11

• Fixed an error in the checkins_detail view. #15

• favorites command now populates favorited_by table - #14
• favorites --stop_after option - #20
• Store unescaped full_text of Tweet - #21

• Added --since_id and --since to user-timeline command, refs #20
• --since and --since_id options for home-timeline, closes #19
• import command now works on files and directories, closes #22

• Fix bugs running home-timeline --since from scratch. If tables were missing, script would throw an error.

• The source column for a tweet is now a foreign key to a new sources table - #12
• New migrations system to upgrade existing databases to handle the new source column extraction - #23
• Experimental implementation of new twitter-to-sqlite search tweets.db search-term command, which runs a search and saves the tweets from that search - #3
• Fixed bug where sometimes a user record for the authenticated user was not persisted to the users table

• New mentions-timeline command (#26)

• search command gained --since_id and --since options, for retrieving tweets since the last time the search was run
• search command is now documented. Closes #3.

• Import command no longer fails on empty files - #29
• Fixed bug in followers command - #30
• following table now has indexes - #28

New friends command for importing accounts that a specific user follows.

• Upgraded to latest sqlite-utils (currently 2.4.2)
• This fixed a bug with user-timeline - #34

• twitter-to-sqlite user-timeline command now accepts --sql and --attach and --ids options, and can also accept multiple screen names or user IDs as command arguments. #35

• The twitter-to-sqlite friends and twitter-to-sqlite followers commands now accept the --sql and --attach options. #36
• Improved the display of the progress bar for the user-timeline command. #38

Full text search for tweets table

• Reworked how --since works to fix a bug spotted in #39. The most recently seen tweet ID for various commands is now stored in a new since_ids table.
• New feature: the friends, followers and listed counts for each user are now stored in a count_history table and updated any time we spot that the counts in that user's profile have changed. #40

• Buf fix: since_id table no longer records a record for None, None. #41

• New command: twitter-to-sqlite lists username fetches lists owned by the specified users. #43
• Handle tweets with a blank source. #44
• Fixed error with twitter-to-sqlite user-timeline --sql=... --ids. #42
• Better error messages for non-existing users. #37

• Fixed bug that occurred when the since_ids table had not yet been created. #46

• Added changelog badge