recent_releases (view)

• table.extract() and sqlite-utils extract now apply much, much faster - one example operation reduced from twelve minutes to just four seconds! (#172)
• sqlite-utils extract no longer shows a progress bar, because it's fast enough not to need one.
• New column_order= option for table.transform() which can be used to alter the order of columns in a table. (#175)
• sqlite-utils transform --column-order= option (with a -o shortcut) for changing column order. (#176)
• The table.transform(drop_foreign_keys=) parameter and the sqlite-utils transform --drop-foreign-key option have changed. They now accept just the name of the column rather than requiring all three of the column, other table and other column. This is technically a backwards-incompatible change but I chose not to bump the major version number because the transform feature is so new. (#177)
• The table .disable_fts(), .rebuild_fts(), .delete(), .delete_where() and .add_missing_columns() methods all now return self, which means they can be chained together with other table operations.

This release introduces two key new capabilities: transform (#114) and extract (#42).

Transform

SQLite's ALTER TABLE has several documented limitations. The table.transform() Python method and sqlite-utils transform CLI command work around these limitations using a pattern where a new table with the desired structure is created, data is copied over to it and the old table is then dropped and replaced by the new one.

You can use these tools to drop columns, change column types, rename columns, add and remove NOT NULL and defaults, remove foreign key constraints and more. See the transforming tables (CLI) and transforming tables (Python library) documentation for full details of how to use them.

Extract

Sometimes a database table - especially one imported from a CSV file - will contain duplicate data. A Trees table may include a Species column with only a few dozen unique values, when the table itself contains thousands of rows.

The table.extract() method and sqlite-utils extract commands can extract a column - or multiple columns - out into a separate lookup table, and set up a foreign key relationship from the original table.

The Python library extract() documentation describes how extraction works in detail, and Extracting columns into a separate table in the CLI documentation includes a detailed example.

Other changes

• The @db.register_function decorator can be used to quickly register Python functions as custom SQL functions, see Registering custom SQL functions. (#162)
• The table.rows_where() method now accepts an optional select= argument for specifying which columns should be selected, see Listing rows.

• Added by date, as part of developing a timeline view. #16

• New sqlite-utils add-foreign-keys command for Adding multiple foreign keys at once. (#157)
• New table.enable_fts(..., replace=True) argument for replacing an existing FTS table with a new configuration. (#160)
• New table.add_foreign_key(..., ignore=True) argument for ignoring a foreign key if it already exists. (#112)

• New template_debug configuration setting. #25
• Templates now autoescape by default, use {{ display.foo|safe }} if you know you want HTML not to be escaped.
• Fixed bug where the wrong template fragment could be used for a row. #24
• Sort order is now persisted when you execute a new search. #23

• Fixed bug with sort orders displayed for the timeline view. #22

• Option to sort by relevance / oldest / newest. #21
• Renamed table column to type. #17
• New -d option for passing one or more databases to index. Without this all databases in the config file will be indexed.

• New github-to-sqlite get /gists command for making authenticated API calls against any API path. #50

• Fixed a bug with writable canned queries that use magic parameters but accept no non-magic arguments. (#967)

• Writable canned queries now expose a JSON API, see JSON API for writable canned queries. (#880)
• New mechanism for defining page templates with custom path parameters - a template file called pages/about/{slug}.html will be used to render any requests to /about/something. See Path parameters for pages. (#944)
• register_output_renderer() render functions can now return a Response. (#953)
• New --upgrade option for datasette install. (#945)
• New datasette --pdb option. (#962)
• datasette --get exit code now reflects the internal HTTP status code. (#947)
• New raise_404() template function for returning 404 errors. (#964)
• datasette publish heroku now deploys using Python 3.8.5
• Upgraded CodeMirror to 5.57.0. (#948)
• Upgraded code style to Black 20.8b1. (#958)
• Fixed bug where selected facets were not correctly persisted in hidden form fields on the table page. (#963)
• Renamed the default error template from 500.html to error.html.
• Custom error pages are now documented, see Custom error pages. (#965)

• Upgraded CodeMirror to 5.57.0. (#948)
• Upgraded code style to Black 20.8b1. (#958)
• New datasette --pdb option. (#962)
• datasette --get exit code now reflects the internal HTTP status code. (#947)
• Fixed bug where selected facets were not correctly persisted in hidden form fields on the table page. (#963)
• New mechanism for defining page templates with custom path parameters. (#944)

• Renamed table column to type. #17
• New -d option for passing one or more databases to index. Without this all databases in the config file will be indexed.

• table.rebuild_fts() method for rebuilding a FTS index, see Rebuilding a full-text search table. (#155)
• sqlite-utils rebuild-fts data.db command for rebuilding FTS indexes across all tables, or just specific tables. (#155)
• table.optimize() method no longer deletes junk rows from the *_fts_docsize table. This was added in 2.17 but it turns out running table.rebuild_fts() is a better solution to this problem.
• Fixed a bug where rows with additional columns that are inserted after the first batch of records could cause an error due to breaking SQLite's maximum number of parameters. Thanks, Simon Wiles. (#145)

This release handles a bug where replacing rows in FTS tables could result in growing numbers of unneccessary rows in the associated *_fts_docsize table. (#149)

• PRAGMA recursive_triggers=on by default for all connections. You can turn it off with Database(recursive_triggers=False). (#152)
• table.optimize() method now deletes unnecessary rows from the *_fts_docsize table. (#153)
• New tracer method for tracking underlying SQL queries, see Tracing queries. (#150)
• Neater indentation for schema SQL. (#148)
• Documentation for sqlite_utils.AlterError exception thrown by in add_foreign_keys().

• Rerun the search query with FTS quoting if a SQLite FTS error occurs. This means searching for e.g. "#dogfest" will no longer throw an error, while still supporting advanced SQLite FTS syntax. #14
• Maximum width 100% on images in results
• Don't display facet headers if there are no results for that facet.

• Custom template displays can now render a point on a map. #12

• Fixed support for advanced FTS query syntax. #13

• Working faceted search interface at /-/beta #3
• is_public column for filtering between public and private data. #11
• New received category. #10

• Added received category. #10
• New is_public column. #11

• Incomplete preview of the /-/beta search interface. #3 #9

• New category column plus default categories table. #7

• Use Porter stemming by default, allow that to be over-ridden with the new --tokenize= option. #2

• Optimize the FTS table and vacuum the database. #4
• First non-alpha release

• Now creates an index on the timestamp column. #1

• First working version

• register_output_renderer() render functions can now return a Response. (#953)
• New --upgrade option for datasette install. (#945)
• datasette publish heroku now deploys using Python 3.8.5

• insert_all(..., alter=True) now works for columns introduced after the first 100 records. Thanks, Simon Wiles! (#139)
• Continuous Integration is now powered by GitHub Actions. (#143)

• --load-extension option for sqlite-utils query for loading SQLite extensions. (#134)
• New sqlite_utils.utils.find_spatialite() function for finding SpatiaLite in common locations. (#135)

• New github-to-sqlite emojis command for fetching emoji supported by GitHub. Demo. #47
• Added table of contents to README.
• Switched to GitHub Actions for CI.
• Switched default branch to main

• Datasette documentation now lives at docs.datasette.io.
• db.is_mutable property is now documented and tested, see Database introspection.
• The extra_template_vars, extra_css_urls, extra_js_urls and extra_body_script plugin hooks now all accept the same arguments. See extra_template_vars(template, database, table, columns, view_name, request, datasette) for details. (#939)
• Those hooks now accept a new columns argument detailing the table columns that will be rendered on that page. (#938)
• Fixed bug where plugins calling db.execute_write_fn() could hang Datasette if the connection failed. (#935)
• Fixed bug with the ?_nl=on output option and binary data. (#914)

• The datasette --get command-line mechanism now ensures any plugins using the startup() hook are correctly executed. (#934)

• Fixed an issue with the Docker image published to Docker Hub. (#931)

• Now available as a sdist package on PyPI in addition to a wheel. (#133)

• Fixed a bug where the sdist distribution of Datasette was not correctly including the template files. (#930)

• Datasette now has a GitHub discussions forum for conversations about the project that go beyond just bug reports and issues.
• Datasette can now be installed on macOS using Homebrew! Run brew install simonw/datasette/datasette. See Using Homebrew. (#335)
• Two new commands: datasette install name-of-plugin and datasette uninstall name-of-plugin. These are equivalent to pip install and pip uninstall but automatically run in the same virtual environment as Datasette, so users don't have to figure out where that virtual environment is - useful for installations created using Homebrew or pipx. See Installing plugins. (#925)
• A new command-line option, datasette --get, accepts a path to a URL within the Datasette instance. It will run that request through Datasette (without starting a web server) and print out the repsonse. See datasette --get for an example. (#926)

• New db.enable_wal() and db.disable_wal() methods for enabling and disabling Write-Ahead Logging for a database file - see WAL mode in the Python API documentation.
• Also sqlite-utils enable-wal file.db and sqlite-utils disable-wal file.db commands for doing the same thing on the command-line, see WAL mode (CLI). (#132)

Warning: This release contains a security fix related to authenticated writable canned queries. If you are using this feature you should upgrade as soon as possible.

• Security fix: CSRF tokens were incorrectly included in read-only canned query forms, which could allow them to be leaked to a sophisticated attacker. See issue 918 for details.
• Datasette now supports GraphQL via the new datasette-graphql plugin - see GraphQL in Datasette with the new datasette-graphql plugin.
• Principle git branch has been renamed from master to main. (#849)
• New debugging tool: /-/allow-debug tool (demo here) helps test allow blocks against actors, as described in Defining permissions with "allow" blocks. (#908)
• New logo for the documentation, and a new project tagline: "An open source multi-tool for exploring and publishing data".
• Whitespace in column values is now respected on display, using white-space: pre-wrap. (#896)
• New await request.post_body() method for accessing the raw POST body, see Request object. (#897)
• Database file downloads now include a content-length HTTP header, enabling download progress bars. (#905)
• File downloads now also correctly set the suggested file name using a content-disposition HTTP header. (#909)
• tests are now excluded from the Datasette package properly - thanks, abeyerpath. (#456)
• The Datasette package published to PyPI now includes sdist as well as bdist_wheel.
• Better titles for canned query pages. (#887)
• Now only loads Python files from a directory passed using the --plugins-dir option - thanks, Amjith Ramanujam. (#890)
• New documentation section on Publishing to Vercel.

• Documentation improvements

• The insert-files command can now read from standard input: cat dog.jpg | sqlite-utils insert-files dogs.db pics - --name=dog.jpg. (#127)
• You can now specify a full-text search tokenizer using the new tokenize= parameter to enable_fts(). This means you can enable Porter stemming on a table by running db["articles"].enable_fts(["headline", "body"], tokenize="porter"). (#130)
• You can also set a custom tokenizer using the sqlite-utils enable-fts CLI command, via the new --tokenize option.

• memoryview and uuid.UUID objects are now supported. memoryview objects will be stored using BLOB and uuid.UUID objects will be stored using TEXT. (#128)

The theme of this release is better tools for working with binary data. The new insert-files command can be used to insert binary files directly into a database table, and other commands have been improved with better support for BLOB columns.

• sqlite-utils insert-files my.db gifs *.gif can now insert the contents of files into a specified table. The columns in the table can be customized to include different pieces of metadata derived from the files. See Inserting binary data from files. (#122)
• --raw option to sqlite-utils query - for outputting just a single raw column value - see Returning raw data from a query, such as binary content. (#123)
• JSON output now encodes BLOB values as special base64 obects - see Running queries and returning JSON. (#125)
• The same format of JSON base64 objects can now be used to insert binary data - see Inserting JSON data. (#126)
• The sqlite-utils query command can now accept named parameters, e.g. sqlite-utils :memory: "select :num * :num2" -p num 5 -p num2 6 - see Running queries and returning JSON. (#124)

• Added table of contents to the documentation

• Added changelog badge

• New github-to-sqlite tags github.db simonw/datasette command for importing all tags for a repository. #43

• New github-to-sqlite repos my.db -r simonw/datasette option for importing just specific repositories. #42

• New --truncate option to sqlite-utils insert, and truncate=True argument to .insert_all(). Thanks, Thomas Sibley. (#118)
• The sqlite-utils query command now runs updates in a transaction. Thanks, Thomas Sibley. (#120)

Magic parameters for canned queries, a log out feature, improved plugin documentation and four new plugin hooks.

Magic parameters for canned queries

Canned queries now support Magic parameters, which can be used to insert or select automatically generated values. For example: sql insert into logs (user_id, timestamp) values (:_actor_id, :_now_datetime_utc) This inserts the currently authenticated actor ID and the current datetime. (#842)

Log out

The ds_actor cookie can be used by plugins (or by Datasette's --root mechanism) to authenticate users. The new /-/logout page provides a way to clear that cookie.

A "Log out" button now shows in the global navigation provided the user is authenticated using the ds_actor cookie. (#840)

Better plugin documentation

The plugin documentation has been re-arranged into four sections, including a brand new section on testing plugins. (#687)

• Plugins introduces Datasette's plugin system and describes how to install and configure plugins.
• Writing plugins describes how to author plugins, from simple one-off plugins to packaged plugins that can be published to PyPI. It also describes how to start a plugin using the new datasette-plugin cookiecutter template.
• Plugin hooks is a full list of detailed documentation for every Datasette plugin hook.
• Testing plugins describes how to write tests for Datasette plugins, using pytest and HTTPX.

New plugin hooks

• register_magic_parameters(datasette) can be used to define new types of magic canned query parameters.
• startup(datasette) can run custom code when Datasette first starts up. datasette-init is a new plugin that uses this hook to create database tables and views on startup if they have not yet been created. (#834)
• canned_queries(datasette, database, actor) lets plugins provide additional canned queries beyond those defined in Datasette's metadata. See datasette-saved-queries for an example of this hook in action. (#852)
• forbidden(datasette, request, message) is a hook for customizing how Datasette responds to 403 forbidden errors. (#812)

Smaller changes

• Cascading view permissons - so if a user has view-table they can view the table page even if they do not have view-database or view-instance. (#832)
• CSRF protection no longer applies to Authentication: Bearer token requests or requests without cookies. (#835)
• datasette.add_message() now works inside plugins. (#864)
• Workaround for "Too many open files" error in test runs. (#846)
• Respect existing scope["actor"] if already set by ASGI middleware. (#854)
• New process for shipping Alpha and beta releases. (#807)
• {{ csrftoken() }} now works when plugins render a template using datasette.render_template(..., request=request). (#863)
• Datasette now creates a single Request object and uses it throughout the lifetime of the current HTTP request. (#870)

• Added documentation for the table.pks introspection property. #116

• The sqlite-utils command now supports UPDATE/INSERT/DELETE in addition to SELECT. #115

Authentication and permissions, writable canned queries, flash messages, new plugin hooks and more. Full release notes. See also Datasette 0.44: the annotated release notes.

The main focus of this release is a major upgrade to the register_output_renderer(datasette) plugin hook, which allows plugins to provide new output formats for Datasette such as datasette-atom and datasette-ics.

• Redesign of register_output_renderer(datasette) to provide more context to the render callback and support an optional "can_render" callback that controls if a suggested link to the output format is provided. (#581, #770)
• Visually distinguish float and integer columns - useful for figuring out why order-by-column might be returning unexpected results. (#729)
• The Request object, which is passed to several plugin hooks, is now documented. (#706)
• New metadata.json option for setting a custom default page size for specific tables and views, see Setting a custom page size. (#751)
• Canned queries can now be configured with a default URL fragment hash, useful when working with plugins such as datasette-vega, see Setting a default fragment. (#706)
• Fixed a bug in datasette publish when running on operating systems where the /tmp directory lives in a different volume, using a backport of the Python 3.8 shutil.copytree() function. (#744)
• Every plugin hook is now covered by the unit tests, and a new unit test checks that each plugin hook has at least one corresponding test. (#771, #773)

• Documentation now explains how to use this with the datasette-media, datasette-json-html and datasette-template-sql plugins. #20

• Renamed this project to dogsheep-photos. #26
• apple-photos command now works even if you haven't yet run upload. #19
• --image-url-prefix and --image-url-suffix options to apple-photos. #24
• New create-subset command. #23

• Added custom project links to the PyPI listing.

• New sqlite-utils drop-table command, see Dropping tables. (#111)
• New sqlite-utils drop-view command, see Dropping views.
• Python decimal.Decimal objects are now stored as FLOAT. (#110)

A small release which provides improved internal methods for use in plugins, along with documentation. See #685.

• Added documentation for db.execute(), see await db.execute(sql, ...).
• Renamed db.execute_against_connection_in_thread() to db.execute_fn() and made it a documented method, see await db.execute_fn(fn).
• New results.first() and results.single_value() methods, plus documentation for the Results class - see Results.

You can now create custom pages within your Datasette instance using a custom template file. For example, adding a template file called templates/pages/about.html will result in a new page being served at /about on your instance. See the custom pages documentation for full details, including how to return custom HTTP headers, redirects and status codes. (#648)

Configuration directory mode (#731) allows you to define a custom Datasette instance as a directory. So instead of running the following:

$ datasette one.db two.db \
  --metadata.json \
  --template-dir=templates/ \
  --plugins-dir=plugins \
  --static css:css

You can instead arrange your files in a single directory called my-project and run this:

$ datasette my-project/

Also in this release:

• New NOT LIKE table filter: ?colname__notlike=expression. (#750)
• Datasette now has a pattern portfolio at /-/patterns - e.g. https://latest.datasette.io/-/patterns. This is a page that shows every Datasette user interface component in one place, to aid core development and people building custom CSS themes. (#151)
• SQLite PRAGMA functions such as pragma_table_info(tablename) are now allowed in Datasette SQL queries. (#761)
• Datasette pages now consistently return a content-type of text/html; charset=utf-8". (#752)
• Datasette now handles an ASGI raw_path value of None, which should allow compatibilty with the Mangum adapter for running ASGI apps on AWS Lambda. Thanks, Colin Dellow. (#719)
• Installation documentation now covers how to Install using pipx. (#756)
• Improved the documentation for Full-text search.

• Upload photos to S3 in a thread pool #11
• New --dry-run option to upload command
• New photos-to-sqlite apple-photos command for importing Apple Photos metadata #1
• Apple Photos quality scores are imported into apple_photos_scores table #15
• Machine learning labels imported from Apple Photos into labels table #16

• New sqlite-utils create-table command, see Creating tables. (#27)
• New sqlite-utils create-view command, see Creating views. (#107)

• New command: github-to-sqlite stargazers (docs) for fetching all users who have starred the specified repositories. #4
• Added several views: dependent_repos, repos_starred, recent_releases. #10 #12 #36
• Added indexes on all foreign key relationships. #35
• GitHub API token can now be read from GITHUB_TOKEN environment variable. #33

• db.create_view(...) now has additional parameters ignore=True or replace=True, see Creating views. (#106)

• New sqlite-utils views my.db command for listing views in a database, see Listing views. (#105)
• sqlite-utils tables (and views) has a new --schema option which outputs the table/view schema, see Listing tables. (#104)
• Nested structures containing invalid JSON values (e.g. Python bytestrings) are now serialized using repr() instead of throwing an error. (#102)

• New github-to-sqlite scrape-dependents command for importing the dependent repositories for a repo. #34

• New columns= argument for the .insert(), .insert_all(), .upsert() and .upsert_all() methods, for over-riding the auto-detected types for columns and specifying additional columns that should be added when the table is created. See Custom column order and column types. (#100)

• Fixed bug that occurred when the since_ids table had not yet been created. #46

• Only upload photos not already in S3. #9

• First alpha release

• Datasette Metadata can now be provided as a YAML file as an optional alternative to JSON. See Using YAML for metadata. (#713)
• Removed support for datasette publish now, which used the the now-retired Zeit Now v1 hosting platform. A new plugin, datasette-publish-now, can be installed to publish data to Zeit (now Vercel) Now v2. (#710)
• Fixed a bug where the extra_template_vars(request, view_name) plugin hook was not receiving the correct view_name. (#716)
• Variables added to the template context by the extra_template_vars() plugin hook are now shown in the ?_context=1 debugging mode (see template_debug). (#693)
• Fixed a bug where the “templates considered” HTML comment was no longer being displayed. (#689)
• Fixed a datasette publish bug where --plugin-secret would over-ride plugin configuration in the provided metadata.json file. (#724)
• Added a new CSS class for customizing the canned query page. (#727)

This release includes some backwards-incompatible schema changes:

• The milestone and assignee columns on the issues table are now integers that are foreign keys to the milestones and users tables - previously they could be text columns instead. #30
• milestones now has an integer foreign key in the creator column, and a new repo column that is an integer foreign key to the repos table. #29
• The repo column in the issues table is now an integer foreign key to repos. Previously it was a string of the format dogsheep/github-to-sqlite. #31

• New github-to-sqlite contributors command for fetching contributors to one or more repositories. #28
• The live demo now includes contributors, and pulls data from simonw/datasette and simonw/sqlite-utils in addition to the Dogsheep repositories.
• The organization column in the repos table is now a foreign key to users as opposed to a big piece of JSON. #27

• New command: twitter-to-sqlite lists username fetches lists owned by the specified users. #43
• Handle tweets with a blank source. #44
• Fixed error with twitter-to-sqlite user-timeline --sql=... --ids. #42
• Better error messages for non-existing users. #37

• Fixed bug where repository topics were not being correctly fetched #26
• Live demo at https://github-to-sqlite.dogsheep.net/ now uses datasette-render-markdown and pulls in issue comments #25

• New table.rows_where(..., order_by="age desc") argument, see Listing rows. (#76)

• Panda’s Timestamp is now stored as a SQLite TEXT column. Thanks, b0b5h4rp13! (#96)
• table.last_pk is now only available for inserts or upserts of a single record. (#98)
• New Database(filepath, recreate=True) parameter for deleting and recreating the database. (#97)

• Buf fix: since_id table no longer records a record for None, None. #41

• Reworked how --since works to fix a bug spotted in #39. The most recently seen tweet ID for various commands is now stored in a new since_ids table.
• New feature: the friends, followers and listed counts for each user are now stored in a count_history table and updated any time we spot that the counts in that user's profile have changed. #40

• Fixed bug with very old checkins that were missing their source - thanks, @mfa! #6

• Upgraded to sqlite-utils 2.x

• Upgraded to latest sqlite-utils

• Defaults to only retrieving updated items, unless you use --all. #2
• Now displays a progress bar for initial data import and --all.
• Data is now fetched 500 items at a time using API pagination. #1
• Now depends on sqlite-utils 2.x.
• auth.json now uses pocket_ prefix on each stored key. #4

• New base_url configuration setting for serving up the correct links while running Datasette under a different URL prefix. (#394)
• New metadata settings "sort" and "sort_desc" for setting the default sort order for a table. See Setting a default sort order. (#702)
• Sort direction arrow now displays by default on the primary key. This means you only have to click once (not twice) to sort in reverse order. (#677)
• New await Request(scope, receive).post_vars() method for accessing POST form variables. (#700)
• Plugin hooks documentation now links to example uses of each plugin. (#709)

• Upgraded to sqlite-utils 2.x
• First non-alpha release

• Full-text search is configured for more tables. #19
• Release assets are now pulled out into a separate assets table. #15
• Now depends on sqlite-utils 2.x. #20
• Commit raw authors are now used to populate a new raw_authors able. #18
• New live demo at https://github-to-sqlite.dogsheep.net/ #13
• GitHub API errors are now raised as Pyton exceptions. #21
• Fixed bug running commits against repos with no commits. #22

• Fixed bug where columns with only null values were not correctly created. (#95)

• Column type suggestion code is no longer confused by null values. (#94)

Added documentation on Browsing your data with Datasette.

First non-alpha release. Now configures full-text search for the users and items tables. #1

Compatibility with sqlite-utils 1.x

• user and trees commands.

• The twitter-to-sqlite friends and twitter-to-sqlite followers commands now accept the --sql and --attach options. #36
• Improved the display of the progress bar for the user-timeline command. #38

• twitter-to-sqlite user-timeline command now accepts --sql and --attach and --ids options, and can also accept multiple screen names or user IDs as command arguments. #35

• Upgraded to latest sqlite-utils (currently 2.4.2)
• This fixed a bug with user-timeline - #34

• table.column_dicts now works with all column types - previously it would throw errors on types other than TEXT, BLOB, INTEGER or FLOAT. (#92)
• Documentation for NotFoundError thrown by table.get(pk) - see Retrieving a specific record.

• table.enable_fts() now works with columns that contain spaces. (#90)