issue_comments

I think the fiddliest part of the implementation here is code that takes the existing columns_dict of the table and the incoming columns= and drop= and rename= parameters and produces the columns dictionary for the new table, ready to be fed to .create_table().

This logic probably also needs to return a structure that can be used to build the INSERT INTO ... SELECT ... FROM query.

A test that confirms that this mechanism can turn a rowid into a non-rowid table would be good too.

Does it make sense to support the pk= argument for changing the primary key?

If the user requests a primary key that doesn't make sense I think an integrity error will be raised when the SQL is being executed, which should hopefully cancel the transaction and raise an error. Need to check that this is what happens.

Since I have a column_order=None argument already, maybe I can ignore the order of the columns in that first argument and use that instead?

The columns= argument is optional - so you can do just a rename operation like so: table.transform(rename={"age": "dog_age"})

If you want to both change the type of a column AND rename it in the same operation, how would you do that? I think like this: python table.transform({"age": int}, rename={"age": "dog_age"}) So any rename logic is applied at the end, after the type transformation or re-ordering logic.

To expand on what that first argument - the columns argument - does. Say you have a table like this: id integer name text age text Any columns omitted from the columns= argument are left alone - they have to be explicitly dropped using drop= if you want to drop them.

Any new columns are added (at the end of the table):

table.tranform({"size": float})

Any columns that have their type changed will have their type changed:

table.tranform({"age": int})

Should I also re-order columns if the order doesn't match? I think so. Open question as to what happens to columns that aren't mentioned at all in the dictionary though - what order should they go in?

I may need to do something special for rowid tables to ensure that the rowid values in the transformed table match those from the old table.

For FTS tables associated with the table that is being transformed, should I automatically drop the old FTS table and recreate it against the new one or will it just magically continue to work after the table is renamed?

I'm going to sketch out a prototype of this new API design in that branch.

I'm rethinking the API design now. Maybe it could look like this:

To change the type of the author_id column from text to int: python books.transform({"author_id": int}) This would leave the existing columns alone, but would change the type of this column.

To rename author_id to author_identifier: python books.transform(rename={"author_id": "author_identifier"}) To drop a column: python books.transform(drop=["author_id"]) Since the parameters all operate on columns they don't need to be called drop_column and rename_column.

I've decided to call this table.transform() - I was over-thinking whether people would remember that .transform() actually transforms the table, but that's what documentation is for.

I did a bunch of initial work for this in #427.

Given dbstat isn't as widely available as I thought I'm going to let people who want to use dbstat run their own select * from dbstat queries rather than bake support directly into Datasette.

The experience of exploring dbstat will improve if I land support for running facets against arbitrary custom SQL queries, which is half-done in that facets now execute against wrapped subqueries as-of ea66c45df96479ef66a89caa71fff1a97a862646

https://github.com/simonw/datasette/blob/ea66c45df96479ef66a89caa71fff1a97a862646/datasette/facets.py#L192-L200

Looks like the pysqlite3-binary package doesn't support dbstat either.

Easiest way to get this may be to run create view dbstat_view as select * from dbstat on databases that support it.

... made harder to work on because I apparently don't have the DBSTAT_VTAB module on macOS.

Or, do this: sql SELECT 1 FROM dbstat limit 1; And see if it returns a "table does not exist" error.

https://www.sqlite.org/dbstat.html

The DBSTAT virtual table is an eponymous virtual table, meaning that is not necessary to run CREATE VIRTUAL TABLE to create an instance of the dbstat virtual table before using it.

How to detect it? Looks like it's visible in SQLite compile time options: https://latest.datasette.io/-/versions

"compile_options": [ "COMPILER=gcc-8.3.0", "ENABLE_COLUMN_METADATA", "ENABLE_DBSTAT_VTAB",

I'm currently using PostgREST to serve OpenAPI APIs off Postgresql databases. I would like to try out datasette once this becomes available on Postgres.

The Python standard library has a module for this: https://docs.python.org/3/library/webbrowser.html

I like this. It could work with the --root option too and automatically sign you in as the root user.

Actually for the tie-breaker it should be something like https://latest.datasette.io/fixtures?sql=select+pk%2C+created%2C+planet_int%2C+on_earth%2C+state%2C+city_id%2C+neighborhood%2C+tags%2C+complex_array%2C+distinct_some_null+from+facetable+where+%28created+%3E+%3Ap1+or+%28created+%3D+%3Ap1+and+%28%28pk+%3E+%3Ap0%29%29%29%29+order+by+created%2C+pk+limit+11&p0=10&p1=2019-01-16+08%3A00%3A00 sql where ( created > :p1 or ( created = :p1 and ((pk > :p0)) ) ) But with rowid and timestamp in place of pk and created.

If previous page ended at 2018-02-11T16:32:53+00:00: sql select search_index.rowid, search_index.type, search_index.key, search_index.title, search_index.category, search_index.timestamp, search_index.search_1 from search_index where date("timestamp") = '2018-02-11' and timestamp < '2018-02-11T16:32:53+00:00' order by search_index.timestamp desc, rowid limit 41

Fun twist: assuming timestamp is always stored as UTC, I need the interface to be timezone aware so I can see e.g. everything from 4th July 2020 in the San Francisco timezone definition of 4th July 2020.

Datasette's implementation is complex because it has to support compound primary keys: https://github.com/simonw/datasette/blob/a258339a935d8d29a95940ef1db01e98bb85ae63/datasette/utils/init.py#L88-L114 - but that's not something that's needed for dogsheep-beta.

It's a shame Datasette doesn't currently have an easy way to implement sorted-by-rank keyset-paginated using a TableView or QueryView. I'll have to do this using the custom SQL query constructed in the plugin: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/init.py#L8-L43

I'm going to try for keyset pagination sorted by relevance just as a learning exercise.

It feels a bit weird to implement keyset pagination against results sorted by rank because the ranks could change substantially if the search index gets updated while the user is paginating.

I may just ignore that though. If you want reliable pagination you can get it by sorting by date. Maybe it doesn't even make sense to offer pagination if you sort by relevance?

Should I do this with offset/limit or should I do proper keyset pagination?

I think keyset because then it will work well for the full search interface with no filters or search string.