issues

We currently show a useful warning (from #331) when the user tries to open a spatialite database without first loading the module:

https://github.com/simonw/datasette/blob/c692cd291111050483a32bea1ee08e994a0b781b/datasette/app.py#L547-L554

This code is part of .inspect() which is going away - see #462 - so I need to find somewhere else for it to live.

This is missing from https://datasette.readthedocs.io/en/0.27.1/facets.html right now

The README is quite out of date now. It includes out-dated copies of help files, promotes the old Zeit Now integration and duplicates a lot of material from the docs.

So much new stuff!

https://github.com/simonw/datasette/compare/0.27...master

If the file changes during download it will probably result in a corrupt download. Safer not to allow downloads at all of mutable databases.

Added in https://github.com/simonw/datasette/commit/bf6b0f918de4aeee7c1036ac975ce2fb23237da7 without docs.

It doesn't work any more, and it's not a particularly useful feature - I've hardly used it since I added it.

See #419: if Datasette is going to work against mutable SQLite files it can no longer assume that the .inspect() method will have cached the correct schema for all tables in all attached databases.

So everywhere in the code at the moment that relies on .inspect() data needs to be modified to use live introspection of the schema instead.

From a comment later on: here are the uses I need to fix as a checklist:

• [x] table_exists()
• [x] info["file"] in .execute()
• [x] resolve_db_name()
• [x] .database_url(database)
• [x] DatabaseDownload file path
• [x] sortable_columns_for_table() uses it to find the columns in a table
• [x] expandable_columns() uses it to find foreign keys
• [x] expand_foreign_keys() uses it to find foreign keys
• [x] display_columns_and_rows() uses it to find primary keys and foreign keys... but also has access to a cursor.description which it uses to list the columns
• [x] TableView.data uses it to lookup columns and primary keys and the table_rows_count (used if the thing isn't a view) and probably a few more things, this method is huge!
• [x] RowView.data uses it for primary keys
• [x] foreign_key_tables() uses it for foreign keys
• [x] DatabaseView list of tables
• [x] IndexView

Split from #419. Mainly useful to see what is connected as mutable v.s. immutable.

Also helps fill the gap left by /-/inspect until #465

This will help clarify that you need a nowv1 account use it.

Follow on from #418 - the Datasette documentation needs an entire section (probably a new page) describing exactly how the hash-in-URL caching mechanism works.

One of the original ideas behind Datasette was that serving immutable data makes everything way easier. Two examples: You don't have to worry about SQLite concurrency and you can bundle the database inside a Docker container and deploy it to immutable hosting. See The interesting ideas in Datasette for more on this.

I'm beginning to see a much stronger case for being able to serve mutable data as well.

SQLite is actually perfectly capable of handling reads against a database that is also being written to, even if the writes are coming from another process. https://www.sqlite.org/wal.htm

There are all kinds of interesting use-cases which Datasette is currently unsuitable for due to its insistence on immutable databases. Some examples:

• Continually run Datasette against a SQLite database updated by another process, e.g. Firefox bookmarks
• Projects where a cron runs every X minutes and writes new entries gathered from other sources to SQLite
• Tail a log file, write those log updates to a SQLite file, view recent log entries in Datasette

This is also relevant to #417, Datasette Library.

The cute performance hack where a hash of the DB contents is included in the URL makes a lot less sense when serving files that frequently change. It's also difficult to explain to people.

It should be optional and default to "off".

Needed for #417 and #419

Split from #419.

This is the last part of #419 - with the move to supporting mutable databases by default, the inspect-data mechanism currently in use no-longer makes much sense.

The one optimization I think it's worth keeping for databases opened in immutable mode is the cached table counts. I think datasette inspect should cut down to only counting the rows in the tables - the other things done by inspect (figuring out columns, foreign key relationships, FTS etc) should all be fast enough that they can be reliably performed at runtime even against large databases.

If performing them at run-time has performance issues, I would rather cache those results internally within Datasette after they are first calculated than continue to support them in the datasette inspect command - to keep things simpler.

Glitch is by far the easiest way to start trying out Datasette. Add a section to https://datasette.readthedocs.io/en/latest/getting_started.html

Needed for #419 - since we can now start up Datasette with a whole bunch of large connected databases that are mutable we can no longer guarantee a quick count of rows across all of the tables.

New proposed homepage tweaks:

Split out from #460.

If a database is mutable, calculating row counts gets expensive. I'm only going to calculate row counts for the index page if it has less than X tables (both hidden and non-hidden) AND each table can be counted in less than 10ms.

If any count takes longer than 10ms I'll cancel the counting entirely. We currently show an inaccurate count if this happens, which is just confusing.

At the moment the number of views is given on a datasette "homepage", but not links to any views themselves

Minor bug in master right now.

https://csvconf.now.sh/

Features like faceting, foreign key expansions and now the inspect-less index view mean Datasette can end up executing a surprisingly large number of SQL queries to render a single page.

Past experience with projects like tikbar have shown that being able to see what actually went into rendering a page can be critical for optimizing performance and generally understanding how everything works.

Support a tracing mode (probably via a ?_trace=1 querystring) which adds information about what is actually going on to both the HTML and the JSON.

Now have deprecated the mechanism we were using for this - running now alias without any parameters - in favour of something new:

https://zeit.co/blog/automatic-aliasing