releases

• The insert-files command can now read from standard input: cat dog.jpg | sqlite-utils insert-files dogs.db pics - --name=dog.jpg. (#127)
• You can now specify a full-text search tokenizer using the new tokenize= parameter to enable_fts(). This means you can enable Porter stemming on a table by running db["articles"].enable_fts(["headline", "body"], tokenize="porter"). (#130)
• You can also set a custom tokenizer using the sqlite-utils enable-fts CLI command, via the new --tokenize option.

• memoryview and uuid.UUID objects are now supported. memoryview objects will be stored using BLOB and uuid.UUID objects will be stored using TEXT. (#128)

The theme of this release is better tools for working with binary data. The new insert-files command can be used to insert binary files directly into a database table, and other commands have been improved with better support for BLOB columns.

• sqlite-utils insert-files my.db gifs *.gif can now insert the contents of files into a specified table. The columns in the table can be customized to include different pieces of metadata derived from the files. See Inserting binary data from files. (#122)
• --raw option to sqlite-utils query - for outputting just a single raw column value - see Returning raw data from a query, such as binary content. (#123)
• JSON output now encodes BLOB values as special base64 obects - see Running queries and returning JSON. (#125)
• The same format of JSON base64 objects can now be used to insert binary data - see Inserting JSON data. (#126)
• The sqlite-utils query command can now accept named parameters, e.g. sqlite-utils :memory: "select :num * :num2" -p num 5 -p num2 6 - see Running queries and returning JSON. (#124)

• Added table of contents to the documentation

• Added changelog badge

• New github-to-sqlite tags github.db simonw/datasette command for importing all tags for a repository. #43

• New github-to-sqlite repos my.db -r simonw/datasette option for importing just specific repositories. #42

• New --truncate option to sqlite-utils insert, and truncate=True argument to .insert_all(). Thanks, Thomas Sibley. (#118)
• The sqlite-utils query command now runs updates in a transaction. Thanks, Thomas Sibley. (#120)

Magic parameters for canned queries, a log out feature, improved plugin documentation and four new plugin hooks.

Magic parameters for canned queries

Canned queries now support Magic parameters, which can be used to insert or select automatically generated values. For example:

insert into logs
  (user_id, timestamp)
values
  (:_actor_id, :_now_datetime_utc)

This inserts the currently authenticated actor ID and the current datetime. (#842)

Log out

The ds_actor cookie can be used by plugins (or by Datasette's --root mechanism) to authenticate users. The new /-/logout page provides a way to clear that cookie.

A "Log out" button now shows in the global navigation provided the user is authenticated using the ds_actor cookie. (#840)

Better plugin documentation

The plugin documentation has been re-arranged into four sections, including a brand new section on testing plugins. (#687)

• Plugins introduces Datasette's plugin system and describes how to install and configure plugins.
• Writing plugins describes how to author plugins, from simple one-off plugins to packaged plugins that can be published to PyPI. It also describes how to start a plugin using the new datasette-plugin cookiecutter template.
• Plugin hooks is a full list of detailed documentation for every Datasette plugin hook.
• Testing plugins describes how to write tests for Datasette plugins, using pytest and HTTPX.

New plugin hooks

• register_magic_parameters(datasette) can be used to define new types of magic canned query parameters.
• startup(datasette) can run custom code when Datasette first starts up. datasette-init is a new plugin that uses this hook to create database tables and views on startup if they have not yet been created. (#834)
• canned_queries(datasette, database, actor) lets plugins provide additional canned queries beyond those defined in Datasette's metadata. See datasette-saved-queries for an example of this hook in action. (#852)
• forbidden(datasette, request, message) is a hook for customizing how Datasette responds to 403 forbidden errors. (#812)

Smaller changes

• Cascading view permissons - so if a user has view-table they can view the table page even if they do not have view-database or view-instance. (#832)
• CSRF protection no longer applies to Authentication: Bearer token requests or requests without cookies. (#835)
• datasette.add_message() now works inside plugins. (#864)
• Workaround for "Too many open files" error in test runs. (#846)
• Respect existing scope["actor"] if already set by ASGI middleware. (#854)
• New process for shipping Alpha and beta releases. (#807)
• {{ csrftoken() }} now works when plugins render a template using datasette.render_template(..., request=request). (#863)
• Datasette now creates a single Request object and uses it throughout the lifetime of the current HTTP request. (#870)

• Added documentation for the table.pks introspection property. #116

• The sqlite-utils command now supports UPDATE/INSERT/DELETE in addition to SELECT. #115

Authentication and permissions, writable canned queries, flash messages, new plugin hooks and more. Full release notes. See also Datasette 0.44: the annotated release notes.

The main focus of this release is a major upgrade to the register_output_renderer(datasette) plugin hook, which allows plugins to provide new output formats for Datasette such as datasette-atom and datasette-ics.

• Redesign of register_output_renderer(datasette) to provide more context to the render callback and support an optional "can_render" callback that controls if a suggested link to the output format is provided. (#581, #770)
• Visually distinguish float and integer columns - useful for figuring out why order-by-column might be returning unexpected results. (#729)
• The Request object, which is passed to several plugin hooks, is now documented. (#706)
• New metadata.json option for setting a custom default page size for specific tables and views, see Setting a custom page size. (#751)
• Canned queries can now be configured with a default URL fragment hash, useful when working with plugins such as datasette-vega, see Setting a default fragment. (#706)
• Fixed a bug in datasette publish when running on operating systems where the /tmp directory lives in a different volume, using a backport of the Python 3.8 shutil.copytree() function. (#744)
• Every plugin hook is now covered by the unit tests, and a new unit test checks that each plugin hook has at least one corresponding test. (#771, #773)

• Documentation now explains how to use this with the datasette-media, datasette-json-html and datasette-template-sql plugins. #20

• Renamed this project to dogsheep-photos. #26
• apple-photos command now works even if you haven't yet run upload. #19
• --image-url-prefix and --image-url-suffix options to apple-photos. #24
• New create-subset command. #23

• Added custom project links to the PyPI listing.

• New sqlite-utils drop-table command, see Dropping tables. (#111)
• New sqlite-utils drop-view command, see Dropping views.
• Python decimal.Decimal objects are now stored as FLOAT. (#110)

A small release which provides improved internal methods for use in plugins, along with documentation. See #685.

• Added documentation for db.execute(), see await db.execute(sql, ...).
• Renamed db.execute_against_connection_in_thread() to db.execute_fn() and made it a documented method, see await db.execute_fn(fn).
• New results.first() and results.single_value() methods, plus documentation for the Results class - see Results.

You can now create custom pages within your Datasette instance using a custom template file. For example, adding a template file called templates/pages/about.html will result in a new page being served at /about on your instance. See the custom pages documentation for full details, including how to return custom HTTP headers, redirects and status codes. (#648)

Configuration directory mode (#731) allows you to define a custom Datasette instance as a directory. So instead of running the following:

$ datasette one.db two.db \
  --metadata.json \
  --template-dir=templates/ \
  --plugins-dir=plugins \
  --static css:css

You can instead arrange your files in a single directory called my-project and run this:

$ datasette my-project/

Also in this release:

• New NOT LIKE table filter: ?colname__notlike=expression. (#750)
• Datasette now has a pattern portfolio at /-/patterns - e.g. https://latest.datasette.io/-/patterns. This is a page that shows every Datasette user interface component in one place, to aid core development and people building custom CSS themes. (#151)
• SQLite PRAGMA functions such as pragma_table_info(tablename) are now allowed in Datasette SQL queries. (#761)
• Datasette pages now consistently return a content-type of text/html; charset=utf-8". (#752)
• Datasette now handles an ASGI raw_path value of None, which should allow compatibilty with the Mangum adapter for running ASGI apps on AWS Lambda. Thanks, Colin Dellow. (#719)
• Installation documentation now covers how to Install using pipx. (#756)
• Improved the documentation for Full-text search.

• Upload photos to S3 in a thread pool #11
• New --dry-run option to upload command
• New photos-to-sqlite apple-photos command for importing Apple Photos metadata #1
• Apple Photos quality scores are imported into apple_photos_scores table #15
• Machine learning labels imported from Apple Photos into labels table #16

• New sqlite-utils create-table command, see Creating tables. (#27)
• New sqlite-utils create-view command, see Creating views. (#107)

• New command: github-to-sqlite stargazers (docs) for fetching all users who have starred the specified repositories. #4
• Added several views: dependent_repos, repos_starred, recent_releases. #10 #12 #36
• Added indexes on all foreign key relationships. #35
• GitHub API token can now be read from GITHUB_TOKEN environment variable. #33

• db.create_view(...) now has additional parameters ignore=True or replace=True, see Creating views. (#106)

• New sqlite-utils views my.db command for listing views in a database, see Listing views. (#105)
• sqlite-utils tables (and views) has a new --schema option which outputs the table/view schema, see Listing tables. (#104)
• Nested structures containing invalid JSON values (e.g. Python bytestrings) are now serialized using repr() instead of throwing an error. (#102)

• New github-to-sqlite scrape-dependents command for importing the dependent repositories for a repo. #34

• Fixed bug that occurred when the since_ids table had not yet been created. #46

• Only upload photos not already in S3. #9

• Datasette Metadata can now be provided as a YAML file as an optional alternative to JSON. See Using YAML for metadata. (#713)
• Removed support for datasette publish now, which used the the now-retired Zeit Now v1 hosting platform. A new plugin, datasette-publish-now, can be installed to publish data to Zeit (now Vercel) Now v2. (#710)
• Fixed a bug where the extra_template_vars(request, view_name) plugin hook was not receiving the correct view_name. (#716)
• Variables added to the template context by the extra_template_vars() plugin hook are now shown in the ?_context=1 debugging mode (see template_debug). (#693)
• Fixed a bug where the “templates considered” HTML comment was no longer being displayed. (#689)
• Fixed a datasette publish bug where --plugin-secret would over-ride plugin configuration in the provided metadata.json file. (#724)
• Added a new CSS class for customizing the canned query page. (#727)

This release includes some backwards-incompatible schema changes:

• The milestone and assignee columns on the issues table are now integers that are foreign keys to the milestones and users tables - previously they could be text columns instead. #30
• milestones now has an integer foreign key in the creator column, and a new repo column that is an integer foreign key to the repos table. #29
• The repo column in the issues table is now an integer foreign key to repos. Previously it was a string of the format dogsheep/github-to-sqlite. #31

• First alpha release

• New github-to-sqlite contributors command for fetching contributors to one or more repositories. #28
• The live demo now includes contributors, and pulls data from simonw/datasette and simonw/sqlite-utils in addition to the Dogsheep repositories.
• The organization column in the repos table is now a foreign key to users as opposed to a big piece of JSON. #27

• New columns= argument for the .insert(), .insert_all(), .upsert() and .upsert_all() methods, for over-riding the auto-detected types for columns and specifying additional columns that should be added when the table is created. See Custom column order and column types. (#100)

• New command: twitter-to-sqlite lists username fetches lists owned by the specified users. #43
• Handle tweets with a blank source. #44
• Fixed error with twitter-to-sqlite user-timeline --sql=... --ids. #42
• Better error messages for non-existing users. #37

• Fixed bug where repository topics were not being correctly fetched #26
• Live demo at https://github-to-sqlite.dogsheep.net/ now uses datasette-render-markdown and pulls in issue comments #25

• New table.rows_where(..., order_by="age desc") argument, see Listing rows. (#76)

• Panda’s Timestamp is now stored as a SQLite TEXT column. Thanks, b0b5h4rp13! (#96)
• table.last_pk is now only available for inserts or upserts of a single record. (#98)
• New Database(filepath, recreate=True) parameter for deleting and recreating the database. (#97)

• Buf fix: since_id table no longer records a record for None, None. #41

• Reworked how --since works to fix a bug spotted in #39. The most recently seen tweet ID for various commands is now stored in a new since_ids table.
• New feature: the friends, followers and listed counts for each user are now stored in a count_history table and updated any time we spot that the counts in that user's profile have changed. #40

• Fixed bug with very old checkins that were missing their source - thanks, @mfa! #6

• Upgraded to sqlite-utils 2.x

• Upgraded to latest sqlite-utils

• Defaults to only retrieving updated items, unless you use --all. #2
• Now displays a progress bar for initial data import and --all.
• Data is now fetched 500 items at a time using API pagination. #1
• Now depends on sqlite-utils 2.x.
• auth.json now uses pocket_ prefix on each stored key. #4

• New base_url configuration setting for serving up the correct links while running Datasette under a different URL prefix. (#394)
• New metadata settings "sort" and "sort_desc" for setting the default sort order for a table. See Setting a default sort order. (#702)
• Sort direction arrow now displays by default on the primary key. This means you only have to click once (not twice) to sort in reverse order. (#677)
• New await Request(scope, receive).post_vars() method for accessing POST form variables. (#700)
• Plugin hooks documentation now links to example uses of each plugin. (#709)

• Upgraded to sqlite-utils 2.x
• First non-alpha release

• Full-text search is configured for more tables. #19
• Release assets are now pulled out into a separate assets table. #15
• Now depends on sqlite-utils 2.x. #20
• Commit raw authors are now used to populate a new raw_authors able. #18
• New live demo at https://github-to-sqlite.dogsheep.net/ #13
• GitHub API errors are now raised as Pyton exceptions. #21
• Fixed bug running commits against repos with no commits. #22

• Fixed bug where columns with only null values were not correctly created. (#95)

• Column type suggestion code is no longer confused by null values. (#94)

Added documentation on Browsing your data with Datasette.

First non-alpha release. Now configures full-text search for the users and items tables. #1

Compatibility with sqlite-utils 1.x

• user and trees commands.

• The twitter-to-sqlite friends and twitter-to-sqlite followers commands now accept the --sql and --attach options. #36
• Improved the display of the progress bar for the user-timeline command. #38

• twitter-to-sqlite user-timeline command now accepts --sql and --attach and --ids options, and can also accept multiple screen names or user IDs as command arguments. #35

• Upgraded to latest sqlite-utils (currently 2.4.2)
• This fixed a bug with user-timeline - #34

• table.column_dicts now works with all column types - previously it would throw errors on types other than TEXT, BLOB, INTEGER or FLOAT. (#92)
• Documentation for NotFoundError thrown by table.get(pk) - see Retrieving a specific record.

• New github-to-sqlite commits my.db dogsheep/github-to-sqlite command. #17

• The Docker build of Datasette now uses SQLite 3.31.1, upgraded from 3.26. (#695)
• datasette publish cloudrun now accepts an optional --memory=2Gi flag for setting the Cloud Run allocated memory to a value other than the default (256Mi). (#694)
• Fixed bug where templates that shipped with plugins were sometimes not being correctly loaded. (#697)

• Don’t attempt to count table rows to display on the index page for databases > 100MB. (#688)
• Print exceptions if they occur in the write thread rather than silently swallowing them.
• Handle the possibility of scope["path"] being a string rather than bytes
• Better documentation for the extra_template_vars(template, database, table, view_name, request, datasette) plugin hook.

• table.enable_fts() now works with columns that contain spaces. (#90)

• table.disable_fts() can now be used to remove FTS tables and triggers that were created using table.enable_fts(...). (#88)
• The sqlite-utils disable-fts command can be used to remove FTS tables and triggers from the command-line. (#88)
• Trying to create table columns with square braces ([ or ]) in the name now raises an error. (#86)
• Subclasses of dict, list and tuple are now detected as needing a JSON column. (#87)

• Plugins now have a supported mechanism for writing to a database, using the new .execute_write() and .execute_write_fn() methods. Documentation. (#682)
• Immutable databases that have had their rows counted using the inspect command now use the calculated count more effectively - thanks, Kevin Keogh. (#666)
• --reload no longer restarts the server if a database file is modified, unless that database was opened immutable mode with -i. (#494)
• New ?_searchmode=raw option turns off escaping for FTS queries in ?_search= allowing full use of SQLite’s FTS5 query syntax. (#676)

• The datasette object passed to plugins now has API documentation: Datasette class. (#576)
• New methods on datasette: .add_database() and .remove_database() - documentation. (#671)
• prepare_connection() plugin hook now takes optional datasette and database arguments - prepare_connection(conn, database, datasette). (#678)
• Added three new plugins and one new conversion tool to the The Datasette Ecosystem.

https://datasette.readthedocs.io/en/latest/changelog.html#v0-36

table.create_index() now works for columns that contain spaces. (#85)

table.exists() is now a method, not a property. This was not a documented part of the API before so I’m considering this a non-breaking change. (#83)

New friends command for importing accounts that a specific user follows.

Fixed a bug where .upsert(..., hash_id="pk") threw an error (#84).

• Added five new plugins and one new conversion tool to the The Datasette Ecosystem.
• The Datasette class has a new render_template() method which can be used by plugins to render templates using Datasette’s pre-configured Jinja templating library.
• You can now execute SQL queries that start with a -- comment - thanks, Jay Graves (#653)

https://datasette.readthedocs.io/en/latest/changelog.html#v0-35

New feature: sqlite_utils.suggest_column_types([records]) returns the suggested column types for a list of records. See Suggesting column types. (#81).

This replaces the undocumented table.detect_column_types() method.

New feature: conversions={...} can be passed to the .insert() family of functions to specify SQL conversions that should be applied to values that are being inserted or updated. See Converting column values using SQL functions. (#77).

• _search= queries are now correctly escaped using a new escape_fts() custom SQL function. This means you can now run searches for strings like park. without seeing errors. (#651)
• Google Cloud Run is no longer in beta, so datasette publish cloudrun has been updated to work even if the user has not installed the gcloud beta components package. Thanks, Katie McLaughlin (#660)
• datasette package now accepts a --port option for specifying which port the resulting Docker container should listen on. (#661)

https://datasette.readthedocs.io/en/stable/changelog.html#v0-34

The .upsert() and .upsert_all() methods now raise a sqlite_utils.db.PrimaryKeyRequired exception if you call them without specifying the primary key column using pk= (#73).

This release changes the behaviour of upsert. It’s a breaking change, hence 2.0.

The upsert command-line utility and the .upsert() and .upsert_all() Python API methods have had their behaviour altered. They used to completely replace the affected records: now, they update the specified values on existing records but leave other columns unaffected.

See Upserting data using the Python API and Upserting data using the CLI for full details.

If you want the old behaviour - where records were completely replaced - you can use $ sqlite-utils insert ... --replace on the command-line and .insert(..., replace=True) and .insert_all(..., replace=True) in the Python API. See Insert-replacing data using the Python API and Insert-replacing data using the CLI for more.

For full background on this change, see issue #66.

• rowid is now included in dropdown menus for filtering tables (#636)
• Columns are now only suggested for faceting if they have at least one value with more than one record (#638)
• Queries with no results now display “0 results” (#637)
• Improved documentation for the --static option (#641)
• asyncio task information is now included on the /-/threads debug page
• Bumped Uvicorn dependency 0.11
• You can now use --port 0 to listen on an available port
• New template_debug setting for debugging templates, e.g. https://latest.datasette.io/fixtures/roadside_attractions?_context=1 (#654)

https://datasette.readthedocs.io/en/latest/changelog.html#v0-33

Datasette now renders templates using Jinja async mode. This makes it easy for plugins to provide custom template functions that perform asynchronous actions, for example the new datasette-template-sql plugin which allows custom templates to directly execute SQL queries and render their results. (#628)

https://datasette.readthedocs.io/en/latest/changelog.html#v0-32

• Fixed a bug where datasette publish heroku applications failed to start (#633)
• Fix for datasette publish with just --source_url - thanks, Stanley Zheng (#572)
• Deployments to Heroku now use Python 3.8.0 (#632)

https://datasette.readthedocs.io/en/latest/changelog.html#v0-31-2

• Deployments created using datasette publish now use python:3.8 base Docker image (#629)

https://datasette.readthedocs.io/en/latest/changelog.html#v0-31-1

This version adds compatibility with Python 3.8 and breaks compatibility with Python 3.5.

Full release notes: https://datasette.readthedocs.io/en/stable/changelog.html#v0-31

• New releases command for fetching releases for a repo, #11
• Repository topics are now fetched by the repos command
• github-to-sqlite repos now accepts multiple usernames
• Command now works without --auth file (using anonymous API calls), #9

• Import command no longer fails on empty files - #29
• Fixed bug in followers command - #30
• following table now has indexes - #28

• Fixed workout latitude/longitude points import for iOS 13 - #10

https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-12-1

https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-12

• search command gained --since_id and --since options, for retrieving tweets since the last time the search was run
• search command is now documented. Closes #3.

https://datasette.readthedocs.io/en/latest/changelog.html#v0-30-2

https://datasette.readthedocs.io/en/stable/changelog.html#v0-30-1

• New mentions-timeline command (#26)

https://datasette.readthedocs.io/en/stable/changelog.html#v0-30

• The source column for a tweet is now a foreign key to a new sources table - #12
• New migrations system to upgrade existing databases to handle the new source column extraction - #23
• Experimental implementation of new twitter-to-sqlite search tweets.db search-term command, which runs a search and saves the tweets from that search - #3
• Fixed bug where sometimes a user record for the authenticated user was not persisted to the users table

• Added --since option, closes #3

• Fix bugs running home-timeline --since from scratch. If tables were missing, script would throw an error.

• Added --since_id and --since to user-timeline command, refs #20
• --since and --since_id options for home-timeline, closes #19
• import command now works on files and directories, closes #22

• favorites command now populates favorited_by table - #14
• favorites --stop_after option - #20
• Store unescaped full_text of Tweet - #21

• New command: github-to-sqlite issue-comments for importing comments on issues - #7
• github-to-sqlite issues now accepts optional --issue=1 argument
• Fixed bug inserting users into already-created table with wrong columns - #6

• New twitter-to-sqlite home-timeline command, for retrieving your timeline of tweets from people you follow - #18
• twitter-to-sqlite import created tables now use the archive_ prefix instead of archive-, for easier querying
• Running twitter-to-sqlite import now deletes existing archive_ tables and recreates them - #17