recent_releases (view)

This release introduces a new sqlite-utils search command for searching tables, see Executing searches. (#192)

The table.search() method has been redesigned, see Searching with table.search(). (#197)

The release includes minor backwards-incompatible changes, hence the version bump to 3.0. Those changes, which should not affect most users, are:

• The -c shortcut option for outputting CSV is no longer available. The full --csv option is required instead.
• The -f shortcut for --fmt has also been removed - use --fmt.
• The table.search() method now defaults to sorting by relevance, not sorting by rowid. (#198)
• The table.search() method now returns a generator over a list of Python dictionaries. It previously returned a list of tuples.

Also in this release:

• The query, tables, rows and search CLI commands now accept a new --tsv option which outputs the results in TSV. (#193)
• A new table.virtual_table_using property reveals if a table is a virtual table, and returns the upper case type of virtual table (e.g. FTS4 or FTS5) if it is. It returns None if the table is not a virtual table. (#196)
• The new table.search_sql() method returns the SQL for searching a table, see Building SQL queries with table.search_sql().
• sqlite-utils rows now accepts multiple optional -c parameters specifying the columns to return. (#200)

Changes since the 3.0a0 alpha release:

• The sqlite-utils search command now defaults to returning every result, unless you add a --limit 20 option.
• The sqlite-utils search -c and table.search(columns=[]) options are now fully respected. (#201)

This release introduces a new sqlite-utils search command for searching tables, see Executing searches. (#192)

The table.search() method has been redesigned, see Searching with table.search(). (#197)

The release includes minor backwards-incompatible changes, hence the version bump to 3.0. Those changes, which should not affect most users, are:

• The -c shortcut option for outputting CSV is no longer available. The full --csv option is required instead.
• The -f shortcut for --fmt has also been removed - use --fmt.
• The table.search() method now defaults to sorting by relevance, not sorting by rowid. (#198)
• The table.search() method now returns a generator over a list of Python dictionaries. It previously returned a list of tuples.

Also in this release:

• The query, tables, rows and search CLI commands now accept a new --tsv option which outputs the results in TSV. (#193)
• A new table.virtual_table_using property reveals if a table is a virtual table, and returns the upper case type of virtual table (e.g. FTS4 or FTS5) if it is. It returns None if the table is not a virtual table. (#196)
• The new table.search_sql() method returns the SQL for searching a table, see Building SQL queries with table.search_sql().
• sqlite-utils rows now accepts multiple optional -c parameters specifying the columns to return. (#200)

• Improvements to the new Binary data documentation page.

A new visual design, plugin hooks for adding navigation options, better handling of binary data, URL building utility methods and better support for running Datasette behind a proxy.

New visual design

Datasette is no longer white and grey with blue and purple links! Natalie Downe has been working on a visual refresh, the first iteration of which is included in this release. (#1056)

<img alt="datasette-0 51" src="https://user-images.githubusercontent.com/9599/97791267-d9011e80-1b8d-11eb-9b1b-42908266f36f.png">

Plugins can now add links within Datasette

A number of existing Datasette plugins add new pages to the Datasette interface, providig tools for things like uploading CSVs, editing table schemas or configuring full-text search.

Plugins like this can now link to themselves from other parts of Datasette interface. The menu_links(datasette, actor) hook (#1064) lets plugins add links to Datasette's new top-right application menu, and the table_actions(datasette, actor, database, table) hook (#1066) adds links to a new "table actions" menu on the table page.

The demo at latest.datasette.io now includes some example plugins. To see the new table actions menu first sign into that demo as root and then visit the facetable table to see the new cog icon menu at the top of the page.

Binary data

SQLite tables can contain binary data in BLOB columns. Datasette now provides links for users to download this data directly from Datasette, and uses those links to make binary data available from CSV exports. See Binary data for more details. (#1036 and #1034).

URL building

The new datasette.urls family of methods can be used to generate URLs to key pages within the Datasette interface, both within custom templates and Datasette plugins. See Building URLs within plugins for more details. (#904)

Running Datasette behind a proxy

The base_url configuration option is designed to help run Datasette on a specific path behind a proxy - for example if you want to run an instance of Datasette at /my-datasette/ within your existing site's URL hierarchy, proxied behind nginx or Apache.

Support for this configuration option has been greatly improved (#1023), and guidelines for using it are now available in a new documentation section on Running Datasette behind a proxy. (#1027)

Smaller changes

• Wide tables shown within Datasette now scroll horizontally (#998). This is achieved using a new <div class="table-wrapper"> element which may impact the implementation of some plugins (for example this change to datasette-cluster-map).
• New debug-menu permission. (#1068)
• Removed --debug option, which didn't do anything. (#814)
• Link: HTTP header pagination. (#1014)
• x button for clearing filters. (#1016)
• Edit SQL button on canned queries, (#1019)
• --load-extension=spatialite shortcut. (#1028)
• scale-in animation for column action menu. (#1039)
• Option to pass a list of templates to .render_template() is now documented. (#1045)
• New datasette.urls.static_plugins() method. (#1033)
• datasette -o option now opens the most relevant page. (#976)
• datasette --cors option now enables access to /database.db downloads. (#1057)
• Database file downloads now implement cascading permissions, so you can download a database if you have view-database-download permission even if you do not have permission to access the Datasette instance. (#1058)
• New documentation on Designing URLs for your plugin. (#1053)

• New load_template(template, request, datasette) plugin hook. (#1042)
• New debug-menu permission. (#1068)

• New colour scheme and improved visual design, courtesy of Natalie Downe. (#1056)
• scale-in animation for column action menu. (#1039)
• Wide tables now scroll horizontally. (#998)
• Option to pass a list of templates to .render_template() is now documented. (#1045)
• New datasette.urls.static_plugins() method. (#1033)
• BLOB column values can now be downloaded directly from the Datasette UI. (#1036)
• .csv exports now link to direct BLOB downloads. (#1034)
• datasette -o option now opens the most relevant page. (#976)
• datasette --cors option now enables access to /database.db downloads. (#1057)
• Database file downloads now implement cascading permissions, so you can download a database if you have view-database-download permission even if you do not have permission to access the Datasette instance. (#1058)
• New documentation on Designing URLs for your plugin. (#1053)
• New navigation menu plus a menu_links(datasette, actor) plugin hook to customize it. (#1064)
• table_actions(datasette, actor, database, table) plugin hook for the new table actions menu. (#1066)

• table.m2m(other_table, records) method now takes any iterable, not just a list or tuple. Thanks, Adam Wolf. (#189)
• sqlite-utils insert now displays a progress bar for CSV or TSV imports. (#173)
• New @db.register_function(deterministic=True) option for registering deterministic SQLite functions in Python 3.8 or higher. (#191)

• Tweaked dependencies for compatibility with new PyPI resolver.

• Tweaked dependencies for compatibility with new PyPI resolver.

• Fixed a bunch of issues relating to the base_url setting. (#1023)
• New datasette.urls URL builder for plugins, see Building URLs within plugins. (#904)
• Removed --debug option, which didn't do anything. (#814)
• Link: HTTP header pagination. (#1014)
• x button for clearing filters. (#1016)
• Edit SQL button on canned queries, (#1019)
• --load-extension=spatialite shortcut. (#1028)

• github-to-sqlite repos command now takes options --readme and --readme-html, which write the README or rendered HTML README into the readme or readme_html columns, respectively. #52
• New --accept option for github-to-sqlite get. #50

• Facet by date. #16
• Now depends on Datasette 0.50.2
• Uses datasette.client and passes through authentication cookies. #28

• Fixed a bug where export files generated in languages other than English failed to import. #11
• CI now uses GitHub Actions

• New --encoding option for processing CSV and TSV files that use a non-utf-8 encoding, for both the insert and update commands. (#182)
• The --load-extension option is now available to many more commands. (#137)
• --load-extension=spatialite can be used to load SpatiaLite from common installation locations, if it is available. (#136)
• Tests now also run against Python 3.9. (#184)
• Passing pk=["id"] now has the same effect as passing pk="id". (#181)

• Fix for "table photos has no column named hasSticker" bug, thanks @mattiaborsoi. #10
• Switched CI to GitHub Actions.

• Notes now use an ID derived from a hash of their contents. #3
• Date columns are now in ISO format, and indexd. #4
• Notes and resources table now have full-text search configured. #4
• Improved how OCR is stored in the database, only storing the most confident matches. #6

• Initial release, with support for importing ENEX files.

• Fixed another bug introduced in 0.50 where column header links on the table page were broken. (#1011)

• Fixed a bug introduced in 0.50 where the export as JSON/CSV links on the table, row and query pages were broken. (#1010)

The key new feature in this release is the column actions menu on the table page (#891). This can be used to sort a column in ascending or descending order, facet data by that column or filter the table to just rows that have a value for that column.

Plugin authors can use the new datasette.client object to make internal HTTP requests from their plugins, allowing them to make use of Datasette's JSON API. (#943)

New Deploying Datasette documentation with guides for deploying Datasette on a Linux server using systemd or to hosting providers that support buildpacks. (#514, #997)

Other improvements in this release:

• Publishing to Google Cloud Run documentation now covers Google Cloud SDK options. Thanks, Geoffrey Hing. (#995)
• New datasette -o option which opens your browser as soon as Datasette starts up. (#970)
• Datasette now sets sqlite3.enable_callback_tracebacks(True) so that errors in custom SQL functions will display tracebacks. (#891)
• Fixed two rendering bugs with column headers in portrait mobile view. (#978, #980)
• New db.table_column_details(table) introspection method for retrieving full details of the columns in a specific table, see Database introspection.
• Fixed a routing bug with custom page wildcard templates. (#996)
• datasette publish heroku now deploys using Python 3.8.6.
• New datasette publish heroku --tar= option. (#969)
• OPTIONS requests against HTML pages no longer return a 500 error. (#1001)
• Datasette now supports Python 3.9.

• Column action menu now shows the column type. (#993)
• Column action sort links now correctly link to the first page of sorted results. (#989)
• Publishing to Google Cloud Run documentation now covers Google Cloud SDK options. Thanks, Geoffrey Hing. (#995)

• New column action menu - table columns now show a cog icon which provides a contextual menu for that column. (#981)
• New datasette -o option which opens your browser as soon as Datasette starts up. (#970
• sqlite3.enable_callback_tracebacks(True) so errors in custom SQL functions will now display tracebacks. (#891)
• Fixed two rendering bugs with column headers in portrait mobile view. (#978, #980)

• table.extract() and sqlite-utils extract now apply much, much faster - one example operation reduced from twelve minutes to just four seconds! (#172)
• sqlite-utils extract no longer shows a progress bar, because it's fast enough not to need one.
• New column_order= option for table.transform() which can be used to alter the order of columns in a table. (#175)
• sqlite-utils transform --column-order= option (with a -o shortcut) for changing column order. (#176)
• The table.transform(drop_foreign_keys=) parameter and the sqlite-utils transform --drop-foreign-key option have changed. They now accept just the name of the column rather than requiring all three of the column, other table and other column. This is technically a backwards-incompatible change but I chose not to bump the major version number because the transform feature is so new. (#177)
• The table .disable_fts(), .rebuild_fts(), .delete(), .delete_where() and .add_missing_columns() methods all now return self, which means they can be chained together with other table operations.

This release introduces two key new capabilities: transform (#114) and extract (#42).

Transform

SQLite's ALTER TABLE has several documented limitations. The table.transform() Python method and sqlite-utils transform CLI command work around these limitations using a pattern where a new table with the desired structure is created, data is copied over to it and the old table is then dropped and replaced by the new one.

You can use these tools to drop columns, change column types, rename columns, add and remove NOT NULL and defaults, remove foreign key constraints and more. See the transforming tables (CLI) and transforming tables (Python library) documentation for full details of how to use them.

Extract

Sometimes a database table - especially one imported from a CSV file - will contain duplicate data. A Trees table may include a Species column with only a few dozen unique values, when the table itself contains thousands of rows.

The table.extract() method and sqlite-utils extract commands can extract a column - or multiple columns - out into a separate lookup table, and set up a foreign key relationship from the original table.

The Python library extract() documentation describes how extraction works in detail, and Extracting columns into a separate table in the CLI documentation includes a detailed example.

Other changes

• The @db.register_function decorator can be used to quickly register Python functions as custom SQL functions, see Registering custom SQL functions. (#162)
• The table.rows_where() method now accepts an optional select= argument for specifying which columns should be selected, see Listing rows.

• Added by date, as part of developing a timeline view. #16

• New sqlite-utils add-foreign-keys command for Adding multiple foreign keys at once. (#157)
• New table.enable_fts(..., replace=True) argument for replacing an existing FTS table with a new configuration. (#160)
• New table.add_foreign_key(..., ignore=True) argument for ignoring a foreign key if it already exists. (#112)

• New template_debug configuration setting. #25
• Templates now autoescape by default, use {{ display.foo|safe }} if you know you want HTML not to be escaped.
• Fixed bug where the wrong template fragment could be used for a row. #24
• Sort order is now persisted when you execute a new search. #23

• Fixed bug with sort orders displayed for the timeline view. #22

• Option to sort by relevance / oldest / newest. #21
• Renamed table column to type. #17
• New -d option for passing one or more databases to index. Without this all databases in the config file will be indexed.

• New github-to-sqlite get /gists command for making authenticated API calls against any API path. #50

• Fixed a bug with writable canned queries that use magic parameters but accept no non-magic arguments. (#967)

• Writable canned queries now expose a JSON API, see JSON API for writable canned queries. (#880)
• New mechanism for defining page templates with custom path parameters - a template file called pages/about/{slug}.html will be used to render any requests to /about/something. See Path parameters for pages. (#944)
• register_output_renderer() render functions can now return a Response. (#953)
• New --upgrade option for datasette install. (#945)
• New datasette --pdb option. (#962)
• datasette --get exit code now reflects the internal HTTP status code. (#947)
• New raise_404() template function for returning 404 errors. (#964)
• datasette publish heroku now deploys using Python 3.8.5
• Upgraded CodeMirror to 5.57.0. (#948)
• Upgraded code style to Black 20.8b1. (#958)
• Fixed bug where selected facets were not correctly persisted in hidden form fields on the table page. (#963)
• Renamed the default error template from 500.html to error.html.
• Custom error pages are now documented, see Custom error pages. (#965)

• Upgraded CodeMirror to 5.57.0. (#948)
• Upgraded code style to Black 20.8b1. (#958)
• New datasette --pdb option. (#962)
• datasette --get exit code now reflects the internal HTTP status code. (#947)
• Fixed bug where selected facets were not correctly persisted in hidden form fields on the table page. (#963)
• New mechanism for defining page templates with custom path parameters. (#944)

• Renamed table column to type. #17
• New -d option for passing one or more databases to index. Without this all databases in the config file will be indexed.

• table.rebuild_fts() method for rebuilding a FTS index, see Rebuilding a full-text search table. (#155)
• sqlite-utils rebuild-fts data.db command for rebuilding FTS indexes across all tables, or just specific tables. (#155)
• table.optimize() method no longer deletes junk rows from the *_fts_docsize table. This was added in 2.17 but it turns out running table.rebuild_fts() is a better solution to this problem.
• Fixed a bug where rows with additional columns that are inserted after the first batch of records could cause an error due to breaking SQLite's maximum number of parameters. Thanks, Simon Wiles. (#145)

This release handles a bug where replacing rows in FTS tables could result in growing numbers of unneccessary rows in the associated *_fts_docsize table. (#149)

• PRAGMA recursive_triggers=on by default for all connections. You can turn it off with Database(recursive_triggers=False). (#152)
• table.optimize() method now deletes unnecessary rows from the *_fts_docsize table. (#153)
• New tracer method for tracking underlying SQL queries, see Tracing queries. (#150)
• Neater indentation for schema SQL. (#148)
• Documentation for sqlite_utils.AlterError exception thrown by in add_foreign_keys().

• Rerun the search query with FTS quoting if a SQLite FTS error occurs. This means searching for e.g. "#dogfest" will no longer throw an error, while still supporting advanced SQLite FTS syntax. #14
• Maximum width 100% on images in results
• Don't display facet headers if there are no results for that facet.

• Custom template displays can now render a point on a map. #12

• Fixed support for advanced FTS query syntax. #13

• Working faceted search interface at /-/beta #3
• is_public column for filtering between public and private data. #11
• New received category. #10

• Added received category. #10
• New is_public column. #11

• Incomplete preview of the /-/beta search interface. #3 #9

• New category column plus default categories table. #7

• Use Porter stemming by default, allow that to be over-ridden with the new --tokenize= option. #2

• Optimize the FTS table and vacuum the database. #4
• First non-alpha release

• Now creates an index on the timestamp column. #1

• First working version

• register_output_renderer() render functions can now return a Response. (#953)
• New --upgrade option for datasette install. (#945)
• datasette publish heroku now deploys using Python 3.8.5

• insert_all(..., alter=True) now works for columns introduced after the first 100 records. Thanks, Simon Wiles! (#139)
• Continuous Integration is now powered by GitHub Actions. (#143)

• --load-extension option for sqlite-utils query for loading SQLite extensions. (#134)
• New sqlite_utils.utils.find_spatialite() function for finding SpatiaLite in common locations. (#135)

• New github-to-sqlite emojis command for fetching emoji supported by GitHub. Demo. #47
• Added table of contents to README.
• Switched to GitHub Actions for CI.
• Switched default branch to main

• Datasette documentation now lives at docs.datasette.io.
• db.is_mutable property is now documented and tested, see Database introspection.
• The extra_template_vars, extra_css_urls, extra_js_urls and extra_body_script plugin hooks now all accept the same arguments. See extra_template_vars(template, database, table, columns, view_name, request, datasette) for details. (#939)
• Those hooks now accept a new columns argument detailing the table columns that will be rendered on that page. (#938)
• Fixed bug where plugins calling db.execute_write_fn() could hang Datasette if the connection failed. (#935)
• Fixed bug with the ?_nl=on output option and binary data. (#914)

• The datasette --get command-line mechanism now ensures any plugins using the startup() hook are correctly executed. (#934)

• Fixed an issue with the Docker image published to Docker Hub. (#931)

• Now available as a sdist package on PyPI in addition to a wheel. (#133)

• Fixed a bug where the sdist distribution of Datasette was not correctly including the template files. (#930)

• Datasette now has a GitHub discussions forum for conversations about the project that go beyond just bug reports and issues.
• Datasette can now be installed on macOS using Homebrew! Run brew install simonw/datasette/datasette. See Using Homebrew. (#335)
• Two new commands: datasette install name-of-plugin and datasette uninstall name-of-plugin. These are equivalent to pip install and pip uninstall but automatically run in the same virtual environment as Datasette, so users don't have to figure out where that virtual environment is - useful for installations created using Homebrew or pipx. See Installing plugins. (#925)
• A new command-line option, datasette --get, accepts a path to a URL within the Datasette instance. It will run that request through Datasette (without starting a web server) and print out the repsonse. See datasette --get for an example. (#926)

• New db.enable_wal() and db.disable_wal() methods for enabling and disabling Write-Ahead Logging for a database file - see WAL mode in the Python API documentation.
• Also sqlite-utils enable-wal file.db and sqlite-utils disable-wal file.db commands for doing the same thing on the command-line, see WAL mode (CLI). (#132)

Warning: This release contains a security fix related to authenticated writable canned queries. If you are using this feature you should upgrade as soon as possible.

• Security fix: CSRF tokens were incorrectly included in read-only canned query forms, which could allow them to be leaked to a sophisticated attacker. See issue 918 for details.
• Datasette now supports GraphQL via the new datasette-graphql plugin - see GraphQL in Datasette with the new datasette-graphql plugin.
• Principle git branch has been renamed from master to main. (#849)
• New debugging tool: /-/allow-debug tool (demo here) helps test allow blocks against actors, as described in Defining permissions with "allow" blocks. (#908)
• New logo for the documentation, and a new project tagline: "An open source multi-tool for exploring and publishing data".
• Whitespace in column values is now respected on display, using white-space: pre-wrap. (#896)
• New await request.post_body() method for accessing the raw POST body, see Request object. (#897)
• Database file downloads now include a content-length HTTP header, enabling download progress bars. (#905)
• File downloads now also correctly set the suggested file name using a content-disposition HTTP header. (#909)
• tests are now excluded from the Datasette package properly - thanks, abeyerpath. (#456)
• The Datasette package published to PyPI now includes sdist as well as bdist_wheel.
• Better titles for canned query pages. (#887)
• Now only loads Python files from a directory passed using the --plugins-dir option - thanks, Amjith Ramanujam. (#890)
• New documentation section on Publishing to Vercel.

• Documentation improvements

• The insert-files command can now read from standard input: cat dog.jpg | sqlite-utils insert-files dogs.db pics - --name=dog.jpg. (#127)
• You can now specify a full-text search tokenizer using the new tokenize= parameter to enable_fts(). This means you can enable Porter stemming on a table by running db["articles"].enable_fts(["headline", "body"], tokenize="porter"). (#130)
• You can also set a custom tokenizer using the sqlite-utils enable-fts CLI command, via the new --tokenize option.

• memoryview and uuid.UUID objects are now supported. memoryview objects will be stored using BLOB and uuid.UUID objects will be stored using TEXT. (#128)

The theme of this release is better tools for working with binary data. The new insert-files command can be used to insert binary files directly into a database table, and other commands have been improved with better support for BLOB columns.

• sqlite-utils insert-files my.db gifs *.gif can now insert the contents of files into a specified table. The columns in the table can be customized to include different pieces of metadata derived from the files. See Inserting binary data from files. (#122)
• --raw option to sqlite-utils query - for outputting just a single raw column value - see Returning raw data from a query, such as binary content. (#123)
• JSON output now encodes BLOB values as special base64 obects - see Running queries and returning JSON. (#125)
• The same format of JSON base64 objects can now be used to insert binary data - see Inserting JSON data. (#126)
• The sqlite-utils query command can now accept named parameters, e.g. sqlite-utils :memory: "select :num * :num2" -p num 5 -p num2 6 - see Running queries and returning JSON. (#124)

• Added table of contents to the documentation

• Added changelog badge

• New github-to-sqlite tags github.db simonw/datasette command for importing all tags for a repository. #43

• New github-to-sqlite repos my.db -r simonw/datasette option for importing just specific repositories. #42

• New --truncate option to sqlite-utils insert, and truncate=True argument to .insert_all(). Thanks, Thomas Sibley. (#118)
• The sqlite-utils query command now runs updates in a transaction. Thanks, Thomas Sibley. (#120)

Magic parameters for canned queries, a log out feature, improved plugin documentation and four new plugin hooks.

Magic parameters for canned queries

Canned queries now support Magic parameters, which can be used to insert or select automatically generated values. For example: sql insert into logs (user_id, timestamp) values (:_actor_id, :_now_datetime_utc) This inserts the currently authenticated actor ID and the current datetime. (#842)

Log out

The ds_actor cookie can be used by plugins (or by Datasette's --root mechanism) to authenticate users. The new /-/logout page provides a way to clear that cookie.

A "Log out" button now shows in the global navigation provided the user is authenticated using the ds_actor cookie. (#840)

Better plugin documentation

The plugin documentation has been re-arranged into four sections, including a brand new section on testing plugins. (#687)

• Plugins introduces Datasette's plugin system and describes how to install and configure plugins.
• Writing plugins describes how to author plugins, from simple one-off plugins to packaged plugins that can be published to PyPI. It also describes how to start a plugin using the new datasette-plugin cookiecutter template.
• Plugin hooks is a full list of detailed documentation for every Datasette plugin hook.
• Testing plugins describes how to write tests for Datasette plugins, using pytest and HTTPX.

New plugin hooks

• register_magic_parameters(datasette) can be used to define new types of magic canned query parameters.
• startup(datasette) can run custom code when Datasette first starts up. datasette-init is a new plugin that uses this hook to create database tables and views on startup if they have not yet been created. (#834)
• canned_queries(datasette, database, actor) lets plugins provide additional canned queries beyond those defined in Datasette's metadata. See datasette-saved-queries for an example of this hook in action. (#852)
• forbidden(datasette, request, message) is a hook for customizing how Datasette responds to 403 forbidden errors. (#812)

Smaller changes

• Cascading view permissons - so if a user has view-table they can view the table page even if they do not have view-database or view-instance. (#832)
• CSRF protection no longer applies to Authentication: Bearer token requests or requests without cookies. (#835)
• datasette.add_message() now works inside plugins. (#864)
• Workaround for "Too many open files" error in test runs. (#846)
• Respect existing scope["actor"] if already set by ASGI middleware. (#854)
• New process for shipping Alpha and beta releases. (#807)
• {{ csrftoken() }} now works when plugins render a template using datasette.render_template(..., request=request). (#863)
• Datasette now creates a single Request object and uses it throughout the lifetime of the current HTTP request. (#870)

• Added documentation for the table.pks introspection property. #116

• The sqlite-utils command now supports UPDATE/INSERT/DELETE in addition to SELECT. #115

Authentication and permissions, writable canned queries, flash messages, new plugin hooks and more. Full release notes. See also Datasette 0.44: the annotated release notes.

The main focus of this release is a major upgrade to the register_output_renderer(datasette) plugin hook, which allows plugins to provide new output formats for Datasette such as datasette-atom and datasette-ics.

• Redesign of register_output_renderer(datasette) to provide more context to the render callback and support an optional "can_render" callback that controls if a suggested link to the output format is provided. (#581, #770)
• Visually distinguish float and integer columns - useful for figuring out why order-by-column might be returning unexpected results. (#729)
• The Request object, which is passed to several plugin hooks, is now documented. (#706)
• New metadata.json option for setting a custom default page size for specific tables and views, see Setting a custom page size. (#751)
• Canned queries can now be configured with a default URL fragment hash, useful when working with plugins such as datasette-vega, see Setting a default fragment. (#706)
• Fixed a bug in datasette publish when running on operating systems where the /tmp directory lives in a different volume, using a backport of the Python 3.8 shutil.copytree() function. (#744)
• Every plugin hook is now covered by the unit tests, and a new unit test checks that each plugin hook has at least one corresponding test. (#771, #773)

• Documentation now explains how to use this with the datasette-media, datasette-json-html and datasette-template-sql plugins. #20

• Renamed this project to dogsheep-photos. #26
• apple-photos command now works even if you haven't yet run upload. #19
• --image-url-prefix and --image-url-suffix options to apple-photos. #24
• New create-subset command. #23

• Added custom project links to the PyPI listing.

• New sqlite-utils drop-table command, see Dropping tables. (#111)
• New sqlite-utils drop-view command, see Dropping views.
• Python decimal.Decimal objects are now stored as FLOAT. (#110)

A small release which provides improved internal methods for use in plugins, along with documentation. See #685.

• Added documentation for db.execute(), see await db.execute(sql, ...).
• Renamed db.execute_against_connection_in_thread() to db.execute_fn() and made it a documented method, see await db.execute_fn(fn).
• New results.first() and results.single_value() methods, plus documentation for the Results class - see Results.

You can now create custom pages within your Datasette instance using a custom template file. For example, adding a template file called templates/pages/about.html will result in a new page being served at /about on your instance. See the custom pages documentation for full details, including how to return custom HTTP headers, redirects and status codes. (#648)

Configuration directory mode (#731) allows you to define a custom Datasette instance as a directory. So instead of running the following:

$ datasette one.db two.db \
  --metadata.json \
  --template-dir=templates/ \
  --plugins-dir=plugins \
  --static css:css

You can instead arrange your files in a single directory called my-project and run this:

$ datasette my-project/

Also in this release:

• New NOT LIKE table filter: ?colname__notlike=expression. (#750)
• Datasette now has a pattern portfolio at /-/patterns - e.g. https://latest.datasette.io/-/patterns. This is a page that shows every Datasette user interface component in one place, to aid core development and people building custom CSS themes. (#151)
• SQLite PRAGMA functions such as pragma_table_info(tablename) are now allowed in Datasette SQL queries. (#761)
• Datasette pages now consistently return a content-type of text/html; charset=utf-8". (#752)
• Datasette now handles an ASGI raw_path value of None, which should allow compatibilty with the Mangum adapter for running ASGI apps on AWS Lambda. Thanks, Colin Dellow. (#719)
• Installation documentation now covers how to Install using pipx. (#756)
• Improved the documentation for Full-text search.

• Upload photos to S3 in a thread pool #11
• New --dry-run option to upload command
• New photos-to-sqlite apple-photos command for importing Apple Photos metadata #1
• Apple Photos quality scores are imported into apple_photos_scores table #15
• Machine learning labels imported from Apple Photos into labels table #16

• New sqlite-utils create-table command, see Creating tables. (#27)
• New sqlite-utils create-view command, see Creating views. (#107)

• New command: github-to-sqlite stargazers (docs) for fetching all users who have starred the specified repositories. #4
• Added several views: dependent_repos, repos_starred, recent_releases. #10 #12 #36
• Added indexes on all foreign key relationships. #35
• GitHub API token can now be read from GITHUB_TOKEN environment variable. #33

• db.create_view(...) now has additional parameters ignore=True or replace=True, see Creating views. (#106)

• New sqlite-utils views my.db command for listing views in a database, see Listing views. (#105)
• sqlite-utils tables (and views) has a new --schema option which outputs the table/view schema, see Listing tables. (#104)
• Nested structures containing invalid JSON values (e.g. Python bytestrings) are now serialized using repr() instead of throwing an error. (#102)

• New github-to-sqlite scrape-dependents command for importing the dependent repositories for a repo. #34

• New columns= argument for the .insert(), .insert_all(), .upsert() and .upsert_all() methods, for over-riding the auto-detected types for columns and specifying additional columns that should be added when the table is created. See Custom column order and column types. (#100)

• Fixed bug that occurred when the since_ids table had not yet been created. #46

• Only upload photos not already in S3. #9

• First alpha release

• Datasette Metadata can now be provided as a YAML file as an optional alternative to JSON. See Using YAML for metadata. (#713)
• Removed support for datasette publish now, which used the the now-retired Zeit Now v1 hosting platform. A new plugin, datasette-publish-now, can be installed to publish data to Zeit (now Vercel) Now v2. (#710)
• Fixed a bug where the extra_template_vars(request, view_name) plugin hook was not receiving the correct view_name. (#716)
• Variables added to the template context by the extra_template_vars() plugin hook are now shown in the ?_context=1 debugging mode (see template_debug). (#693)
• Fixed a bug where the “templates considered” HTML comment was no longer being displayed. (#689)
• Fixed a datasette publish bug where --plugin-secret would over-ride plugin configuration in the provided metadata.json file. (#724)
• Added a new CSS class for customizing the canned query page. (#727)

This release includes some backwards-incompatible schema changes:

• The milestone and assignee columns on the issues table are now integers that are foreign keys to the milestones and users tables - previously they could be text columns instead. #30
• milestones now has an integer foreign key in the creator column, and a new repo column that is an integer foreign key to the repos table. #29
• The repo column in the issues table is now an integer foreign key to repos. Previously it was a string of the format dogsheep/github-to-sqlite. #31

• New github-to-sqlite contributors command for fetching contributors to one or more repositories. #28
• The live demo now includes contributors, and pulls data from simonw/datasette and simonw/sqlite-utils in addition to the Dogsheep repositories.
• The organization column in the repos table is now a foreign key to users as opposed to a big piece of JSON. #27

• New command: twitter-to-sqlite lists username fetches lists owned by the specified users. #43
• Handle tweets with a blank source. #44
• Fixed error with twitter-to-sqlite user-timeline --sql=... --ids. #42
• Better error messages for non-existing users. #37

• Fixed bug where repository topics were not being correctly fetched #26
• Live demo at https://github-to-sqlite.dogsheep.net/ now uses datasette-render-markdown and pulls in issue comments #25