issue_comments

3,301 rows sorted by updated_at descending

View and edit SQL

Suggested facets: reactions, created_at (date), updated_at (date)

issue

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
656363548 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-656363548 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NjM2MzU0OA== simonw 9599 2020-07-09T21:37:28Z 2020-07-09T21:37:28Z OWNER

I'm going to add a second method .transform_table_sql(...) - which returns the SQL that would have been executed but does NOT execute it.

Advanced callers can use this to include their own additional steps in the same transaction - e.g. recreating views or triggers.

More importantly it gives me a useful hook for writing some unit tests against the generated SQL.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740
655898722 https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655898722 https://api.github.com/repos/simonw/sqlite-utils/issues/121 MDEyOklzc3VlQ29tbWVudDY1NTg5ODcyMg== tsibley 79913 2020-07-09T04:53:08Z 2020-07-09T04:53:08Z CONTRIBUTOR

Yep, I agree that makes more sense for backwards compat and more casual use cases. I think it should be possible for the Database/Queryable methods to DTRT based on seeing if it's within a context-manager-managed transaction.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improved (and better documented) support for transactions 652961907
655786374 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655786374 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTc4NjM3NA== simonw 9599 2020-07-08T22:16:54Z 2020-07-08T22:16:54Z OWNER

According to https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes the hardest bits to consider are how to deal with existing foreign key relationships, triggers and views.

I'm OK leaving views as an exercise for the caller - many of these transformations may not need any view changes at all.

Foreign key relationships are important: it should handle these automatically as effectively as possible.

Likewise trigger changes: need to think about what this means.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740
655785396 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655785396 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTc4NTM5Ng== simonw 9599 2020-07-08T22:14:10Z 2020-07-08T22:14:10Z OWNER

Work in progress: not quite right yet, I need smarter logic for how renamed columns are reflected in the generated INSERT INTO ... SELECT ... query:

    def transform_table(
        self,
        columns=None,
        rename=None,
        change_type=None,
        pk=None,
        foreign_keys=None,
        column_order=None,
        not_null=None,
        defaults=None,
        hash_id=None,
        extracts=None,
    ):
        assert self.exists(), "Cannot transform a table that doesn't exist yet"
        columns = columns or self.columns_dict
        if rename is not None or change_type is not None:
            columns = {rename.get(key, key): change_type.get(key, value) for key, value in columns.items()}
        new_table_name = "{}_new_{}".format(self.name, os.urandom(6).hex())
        previous_columns = set(self.columns_dict.keys())
        with self.db.conn:
            columns = {name: value for (name, value) in columns.items()}
            new_table = self.db.create_table(
                new_table_name,
                columns,
                pk=pk,
                foreign_keys=foreign_keys,
                column_order=column_order,
                not_null=not_null,
                defaults=defaults,
                hash_id=hash_id,
                extracts=extracts,
            )
            # Copy across data - but only for columns that exist in both
            new_columns = set(columns.keys())
            columns_to_copy = new_columns.intersection(previous_columns)
            copy_sql = "INSERT INTO [{new_table}] ({new_cols}) SELECT {old_cols} FROM [{old_table}]".format(
                new_table=new_table_name,
                old_table=self.name,
                old_cols=", ".join("[{}]".format(col) for col in columns_to_copy),
                new_cols=", ".join("[{}]".format(rename.get(col, col)) for col in columns_to_copy),
            )
            self.db.conn.execute(copy_sql)
            # Drop the old table
            self.db.conn.execute("DROP TABLE [{}]".format(self.name))
            # Rename the new one
            self.db.conn.execute(
                "ALTER TABLE [{}] RENAME TO [{}]".format(new_table_name, self.name)
            )
        return self
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740
655783875 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655783875 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTc4Mzg3NQ== simonw 9599 2020-07-08T22:09:51Z 2020-07-08T22:10:16Z OWNER

I can have a convenient change_type={...} parameter for changing column types too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740
655782477 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655782477 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTc4MjQ3Nw== simonw 9599 2020-07-08T22:06:23Z 2020-07-08T22:06:23Z OWNER

Thinking about the method signature:

    def transform_table(
        self,
        columns,
        pk=None,
        foreign_keys=None,
        column_order=None,
        not_null=None,
        defaults=None,
        hash_id=None,
        extracts=None,
    ):

This requires the caller to provide the exact set of columns for the new table.

It would be useful if this was optional - if you could omit the columns and have it automatically use the previous columns. This would let you change things like the primary key or the column order using the other arguments.

Even better: allow column renaming using an optional rename={...} argument:

db["dogs"].transform_table(rename={"name": "dog_name"})
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740
655778058 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655778058 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTc3ODA1OA== simonw 9599 2020-07-08T21:54:30Z 2020-07-08T21:54:30Z OWNER

Don't forget this step:

If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740
655677909 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677909 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTY3NzkwOQ== simonw 9599 2020-07-08T18:16:39Z 2020-07-08T18:16:39Z OWNER

Since neither the term "transform" or "migrate" are used in the codebase at the moment, I think I'll go with .transform_table() - that leaves the term "migrate" available for any future database migrations system (similar to Django's).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740
655677396 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677396 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTY3NzM5Ng== simonw 9599 2020-07-08T18:15:39Z 2020-07-08T18:15:39Z OWNER

Alternative possible names:
- .transform_table()
- .migrate()
- .transform()

I'm torn between .migrate_table() and .transform_table().

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740
655677099 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677099 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTY3NzA5OQ== simonw 9599 2020-07-08T18:15:02Z 2020-07-08T18:15:02Z OWNER

I'm not so keen on that chained API - it's pretty complicated.

Here's an idea for a much simpler interface. Essentially it lets you say "take table X and migrate its contents to a new table with this structure - then atomically rename the tables to switch them":

db["mytable"].migrate_table({"id": int, "name": str"}, pk="id")

The migrate_table() method would take the same exact signature as the table.create() method: https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L615-L625

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740
655674910 https://github.com/simonw/sqlite-utils/issues/119#issuecomment-655674910 https://api.github.com/repos/simonw/sqlite-utils/issues/119 MDEyOklzc3VlQ29tbWVudDY1NTY3NDkxMA== simonw 9599 2020-07-08T18:10:18Z 2020-07-08T18:10:18Z OWNER

This will work similar to how .add_foreign_keys() works: turn on writable_schema and rewrite the sql for that table in the sqlite_master table.

Here's that code today - it could be adapted to include removal of foreign keys that we no longer want:

https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L391-L401

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to remove a foreign key 652700770
655673896 https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655673896 https://api.github.com/repos/simonw/sqlite-utils/issues/121 MDEyOklzc3VlQ29tbWVudDY1NTY3Mzg5Ng== simonw 9599 2020-07-08T18:08:11Z 2020-07-08T18:08:11Z OWNER

I'm with you on most of this. Completely agreed that the CLI should do everything in a transaction.

The one thing I'm not keen on is forcing calling code to explicitly start a transaction, for a couple of reasons:

  1. It will break all of the existing code out there
  2. It doesn't match to how I most commonly use this library - as an interactive tool in a Jupyter notebook, where I'm generally working against a brand new scratch database and any errors don't actually matter

So... how about this: IF you wrap your code in a with db: block then the .insert() and suchlike methods expect you to manage transactions yourself. But if you don't use the context manager they behave like they do at the moment (or maybe a bit more sensibly).

That way existing code works as it does today, lazy people like me can call .insert() without thinking about transactions, but people writing actual production code (as opposed to Jupyter hacks) have a sensible way to take control of the transactions themselves.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improved (and better documented) support for transactions 652961907
655653292 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655653292 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTY1MzI5Mg== simonw 9599 2020-07-08T17:26:02Z 2020-07-08T17:26:02Z OWNER

Awesome, thank you very much.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316
655652679 https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655652679 https://api.github.com/repos/simonw/sqlite-utils/issues/121 MDEyOklzc3VlQ29tbWVudDY1NTY1MjY3OQ== tsibley 79913 2020-07-08T17:24:46Z 2020-07-08T17:24:46Z CONTRIBUTOR

Better transaction handling would be really great. Some of my thoughts on implementing better transaction discipline are in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728.

My preferences:

  • Each CLI command should operate in a single transaction so that either the whole thing succeeds or the whole thing is rolled back. This avoids partially completed operations when an error occurs part way through processing. Partially completed operations are typically much harder to recovery from gracefully and may cause inconsistent data states.

  • The Python API should be transaction-agnostic and rely on the caller to coordinate transactions. Only the caller knows how individual insert, create, update, etc operations/methods should be bundled conceptually into transactions. When the caller is the CLI, for example, that bundling would be at the CLI command-level. Other callers might want to break up operations into multiple transactions. Transactions are usually most useful when controlled at the application-level (like logging configuration) instead of the library level. The library needs to provide an API that's conducive to transaction use, though.

  • The Python API should provide a context manager to provide consistent transactions handling with more useful defaults than Python's sqlite3 module. The latter issues implicit BEGIN statements by default for most DML (INSERT, UPDATE, DELETE, … but not SELECT, I believe), but not DDL (CREATE TABLE, DROP TABLE, CREATE VIEW, …). Notably, the sqlite3 module doesn't issue the implicit BEGIN until the first DML statement. It does not issue it when entering the with conn block, like other DBAPI2-compatible modules do. The with conn block for sqlite3 only arranges to commit or rollback an existing transaction when exiting. Including DDL and SELECTs in transactions is important for operation consistency, though. There are several existing bugs.python.org tickets about this and future changes are in the works, but sqlite-utils can provide its own API sooner. sqlite-utils's Database class could itself be a context manager (built on the sqlite3 connection context manager) which additionally issues an explicit BEGIN when entering. This would then let Python API callers do something like:

db = sqlite_utils.Database(path)

with db: # ← BEGIN issued here by Database.__enter__
    db.insert(…)
    db.create_view(…)
# ← COMMIT/ROLLBACK issue here by sqlite3.connection.__exit__
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improved (and better documented) support for transactions 652961907
655643078 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655643078 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTY0MzA3OA== tsibley 79913 2020-07-08T17:05:59Z 2020-07-08T17:05:59Z CONTRIBUTOR

The only thing missing from this PR is updates to the documentation.

Ah, yes, thanks for this reminder! I've repushed with doc bits added.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316
655290625 https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655290625 https://api.github.com/repos/simonw/sqlite-utils/issues/114 MDEyOklzc3VlQ29tbWVudDY1NTI5MDYyNQ== simonw 9599 2020-07-08T05:15:45Z 2020-07-08T05:15:45Z OWNER

Ideally this would all happen in a single transaction, such that other processes talking to the database would not see any inconsistent state while the table copy was taking place. Need to confirm that this is possible. Also refs transactions thoughts in #121.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.transform_table() method for advanced alter table 621989740
655289686 https://github.com/simonw/sqlite-utils/pull/120#issuecomment-655289686 https://api.github.com/repos/simonw/sqlite-utils/issues/120 MDEyOklzc3VlQ29tbWVudDY1NTI4OTY4Ng== simonw 9599 2020-07-08T05:13:11Z 2020-07-08T05:13:11Z OWNER

This is an excellent fix, thanks!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix query command's support for DML 652816158
655286864 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655286864 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTI4Njg2NA== simonw 9599 2020-07-08T05:05:27Z 2020-07-08T05:05:36Z OWNER

The only thing missing from this PR is updates to the documentation. Those need to go in two places:

Here's an example of a previous commit that includes updates to both CLI and API documentation: https://github.com/simonw/sqlite-utils/commit/f9473ace14878212c1fa968b7bd2f51e4f064dba#diff-e3e2a9bfd88566b05001b02a3f51d286

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316
655284168 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284168 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTI4NDE2OA== simonw 9599 2020-07-08T04:58:00Z 2020-07-08T04:58:00Z OWNER

Oops didn't mean to click "close" there.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316
655284054 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284054 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTI4NDA1NA== simonw 9599 2020-07-08T04:57:38Z 2020-07-08T04:57:38Z OWNER

Thoughts on transactions would be much appreciated in #121

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316
655283393 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTI4MzM5Mw== simonw 9599 2020-07-08T04:55:18Z 2020-07-08T04:55:18Z OWNER

This is a really good idea - and thank you for the detailed discussion in the pull request.

I'm keen to discuss how transactions can work better. I tend to use this pattern in my own code:

with db.conn:
    db["table"].insert(...)

But it's not documented and I've not though very hard about it!

I like having inserts that handle 10,000+ rows commit on every chunk so I can watch their progress from another process, but the library should absolutely support people who want to commit all of the rows in a single transaction - or combine changes with DML.

Lots to discuss here. I'll start a new issue.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316
655239728 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTIzOTcyOA== tsibley 79913 2020-07-08T02:16:42Z 2020-07-08T02:16:42Z CONTRIBUTOR

I fixed my original oops by moving the DELETE FROM $table out of the chunking loop and repushed. I think this change can be considered in isolation from issues around transactions, which I discuss next.

I wanted to make the DELETE + INSERT happen all in the same transaction so it was robust, but that was more complicated than I expected. The transaction handling in the Database/Table classes isn't systematic, and this poses big hurdles to making Table.insert_all (or other operations) consistent and robust in the face of errors.

For example, I wanted to do this (whitespace ignored in diff, so indentation change not highlighted):

diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index d6b9ecf..4107ceb 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -1028,6 +1028,11 @@ class Table(Queryable):
         batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))
         self.last_rowid = None
         self.last_pk = None
+        with self.db.conn:
+            # Explicit BEGIN is necessary because Python's sqlite3 doesn't
+            # issue implicit BEGINs for DDL, only DML.  We mix DDL and DML
+            # below and might execute DDL first, e.g. for table creation.
+            self.db.conn.execute("BEGIN")
             if truncate and self.exists():
                 self.db.conn.execute("DELETE FROM [{}];".format(self.name))
             for chunk in chunks(itertools.chain([first_record], records), batch_size):
@@ -1038,7 +1043,11 @@ class Table(Queryable):
                         # Use the first batch to derive the table names
                         column_types = suggest_column_types(chunk)
                         column_types.update(columns or {})
-                    self.create(
+                        # Not self.create() because that is wrapped in its own
+                        # transaction and Python's sqlite3 doesn't support
+                        # nested transactions.
+                        self.db.create_table(
+                            self.name,
                             column_types,
                             pk,
                             foreign_keys,
@@ -1139,7 +1148,6 @@ class Table(Queryable):
                     flat_values = list(itertools.chain(*values))
                     queries_and_params = [(sql, flat_values)]

-            with self.db.conn:
                 for query, params in queries_and_params:
                     try:
                         result = self.db.conn.execute(query, params)

but that fails in tests because other methods call insert/upsert/insert_all/upsert_all in the middle of their transactions, so the BEGIN statement throws an error (no nested transactions allowed).

Stepping back, it would be nice to make the transaction handling systematic and predictable. One way to do this is to make the sqlite_utils/db.py code generally not begin or commit any transactions, and require the caller to do that instead. This lets the caller mix and match the Python API calls into transactions as appropriate (which is impossible for the API methods themselves to fully determine). Then, make sqlite_utils/cli.py begin and commit a transaction in each @cli.command function, making each command robust and consistent in the face of errors. The big change here, and why I didn't just submit a patch, is that it dramatically changes the Python API to require callers to begin a transaction rather than just immediately calling methods.

There is also the caveat that for each transaction, an explicit BEGIN is also necessary so that DDL as well as DML (as well as SELECTs) are consistent and rolled back on error. There are several bugs.python.org discussions around this particular problem of DDL and some plans to make it better and consistent with DBAPI2, eventually. In the meantime, the sqlite-utils Database class could be a context manager which supports the incantations necessary to do proper transactions. This would still be a Python API change for callers but wouldn't expose them to the weirdness of the sqlite3's default transaction handling.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316
655052451 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655052451 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTA1MjQ1MQ== tsibley 79913 2020-07-07T18:45:23Z 2020-07-07T18:45:23Z CONTRIBUTOR

Ah, I see the problem. The truncate is inside a loop I didn't realize was there.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316
655018966 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655018966 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTAxODk2Ng== tsibley 79913 2020-07-07T17:41:06Z 2020-07-07T17:41:06Z CONTRIBUTOR

Hmm, while tests pass, this may not work as intended on larger datasets. Looking into it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316
654424704 https://github.com/simonw/datasette/issues/784#issuecomment-654424704 https://api.github.com/repos/simonw/datasette/issues/784 MDEyOklzc3VlQ29tbWVudDY1NDQyNDcwNA== simonw 9599 2020-07-06T19:31:53Z 2020-07-06T19:31:53Z OWNER

Documentation: https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to sign in to Datasette as a root account 628003707
653966670 https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653966670 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 MDEyOklzc3VlQ29tbWVudDY1Mzk2NjY3MA== simonw 9599 2020-07-06T01:07:02Z 2020-07-06T01:07:02Z MEMBER

OK that fix worked.https://github.com/dogsheep/github-to-sqlite/runs/839764768?check_suite_focus=true

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Demo is failing to deploy 651159727
653962708 https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962708 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 MDEyOklzc3VlQ29tbWVudDY1Mzk2MjcwOA== simonw 9599 2020-07-06T00:43:10Z 2020-07-06T00:43:10Z MEMBER

I bet it's datasette-search-all.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Demo is failing to deploy 651159727
653962669 https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962669 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 MDEyOklzc3VlQ29tbWVudDY1Mzk2MjY2OQ== simonw 9599 2020-07-06T00:42:57Z 2020-07-06T00:42:57Z MEMBER

https://github-to-sqlite.dogsheep.net/-/plugins

[
    {
        "name": "datasette-json-html",
        "static": false,
        "templates": false,
        "version": "0.6",
        "hooks": [
            "prepare_connection",
            "render_cell"
        ]
    },
    {
        "name": "datasette-render-markdown",
        "static": false,
        "templates": false,
        "version": "1.1.2",
        "hooks": [
            "extra_template_vars",
            "render_cell"
        ]
    },
    {
        "name": "datasette-pretty-json",
        "static": false,
        "templates": false,
        "version": "0.2",
        "hooks": [
            "render_cell"
        ]
    },
    {
        "name": "datasette-search-all",
        "static": false,
        "templates": true,
        "version": "0.2.1",
        "hooks": [
            "asgi_wrapper",
            "extra_template_vars"
        ]
    },
    {
        "name": "datasette-vega",
        "static": true,
        "templates": false,
        "version": "0.6.2",
        "hooks": [
            "extra_css_urls",
            "extra_js_urls"
        ]
    }
]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Demo is failing to deploy 651159727
653962530 https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962530 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 MDEyOklzc3VlQ29tbWVudDY1Mzk2MjUzMA== simonw 9599 2020-07-06T00:42:13Z 2020-07-06T00:42:13Z MEMBER

So it looks like it's the ASGI lifespan change I made in https://github.com/simonw/datasette/commit/16f592247a2a0e140ada487e9972645406dcae69 - It must be incompatible with one of the plugins.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Demo is failing to deploy 651159727
653962418 https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962418 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 MDEyOklzc3VlQ29tbWVudDY1Mzk2MjQxOA== simonw 9599 2020-07-06T00:41:38Z 2020-07-06T00:41:38Z MEMBER

https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/logs?project=datasette-222320 has some clues.


{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Demo is failing to deploy 651159727
653960989 https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653960989 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 MDEyOklzc3VlQ29tbWVudDY1Mzk2MDk4OQ== simonw 9599 2020-07-06T00:32:34Z 2020-07-06T00:32:34Z MEMBER

Same error.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Demo is failing to deploy 651159727
653947916 https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653947916 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41 MDEyOklzc3VlQ29tbWVudDY1Mzk0NzkxNg== simonw 9599 2020-07-05T22:40:47Z 2020-07-05T22:40:47Z MEMBER

Might be that it's not got enough RAM. I'll try deploying to a larger instance.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Demo is failing to deploy 651159727
653314465 https://github.com/simonw/datasette/pull/890#issuecomment-653314465 https://api.github.com/repos/simonw/datasette/issues/890 MDEyOklzc3VlQ29tbWVudDY1MzMxNDQ2NQ== simonw 9599 2020-07-03T03:07:41Z 2020-07-03T03:07:41Z OWNER

This is an excellent fix. Thanks!

Not sure why codecov is complaining. I'm going to merge it as-is.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Load only python files from plugins-dir. 650305298
653309545 https://github.com/simonw/datasette/pull/890#issuecomment-653309545 https://api.github.com/repos/simonw/datasette/issues/890 MDEyOklzc3VlQ29tbWVudDY1MzMwOTU0NQ== codecov[bot] 22429695 2020-07-03T02:52:25Z 2020-07-03T03:03:00Z NONE

Codecov Report

Merging #890 into master will decrease coverage by 0.01%.
The diff coverage is 80.00%.

@@            Coverage Diff             @@
##           master     #890      +/-   ##
==========================================
- Coverage   83.42%   83.40%   -0.02%     
==========================================
  Files          27       27              
  Lines        3632     3634       +2     
==========================================
+ Hits         3030     3031       +1     
- Misses        602      603       +1     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/app.py</td> <td>95.99% <80.00%> (-0.17%)</td> <td>:arrow_down:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 57879dc...745af3b. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Load only python files from plugins-dir. 650305298
643711117 https://github.com/simonw/datasette/pull/848#issuecomment-643711117 https://api.github.com/repos/simonw/datasette/issues/848 MDEyOklzc3VlQ29tbWVudDY0MzcxMTExNw== codecov[bot] 22429695 2020-06-14T03:05:55Z 2020-07-03T02:44:09Z NONE

Codecov Report

Merging #848 into master will decrease coverage by 0.60%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master     #848      +/-   ##
==========================================
- Coverage   83.42%   82.82%   -0.61%     
==========================================
  Files          27       26       -1     
  Lines        3632     3540      -92     
==========================================
- Hits         3030     2932      -98     
- Misses        602      608       +6     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/cli.py</td> <td>71.34% <0.00%> (-0.89%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/views/special.py</td> <td>77.77% <0.00%> (-3.40%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/app.py</td> <td>94.58% <0.00%> (-1.58%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/utils/asgi.py</td> <td>90.90% <0.00%> (-0.42%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/utils/__init__.py</td> <td>93.84% <0.00%> (-0.09%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/plugins.py</td> <td>82.35% <0.00%> (ø)</td> <td></td> </tr> <tr> <td>datasette/hookspecs.py</td> <td>100.00% <0.00%> (ø)</td> <td></td> </tr> <tr> <td>datasette/default_permissions.py</td> <td>100.00% <0.00%> (ø)</td> <td></td> </tr> <tr> <td>datasette/default_magic_parameters.py</td> <td></td> <td></td> </tr> <tr> <td>datasette/views/base.py</td> <td>93.40% <0.00%> (+<0.01%)</td> <td>:arrow_up:</td> </tr> <tr> <td>... and 2 more</td> <td></td> <td></td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 57879dc...0d100d1. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Reload support for config_dir mode. 638270441
653002499 https://github.com/simonw/datasette/issues/889#issuecomment-653002499 https://api.github.com/repos/simonw/datasette/issues/889 MDEyOklzc3VlQ29tbWVudDY1MzAwMjQ5OQ== amjith 49260 2020-07-02T13:22:13Z 2020-07-02T13:22:13Z CONTRIBUTOR

I was able to narrow this down to the fact that lifespan protocol is turned on.

I see the workaround you've used here: https://github.com/simonw/datasette-debug-asgi/commit/72d568d32a3159c763ce908c0b269736935c6987

If so, maybe it's time to update some of the asg_wrapper plugins.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
asgi_wrapper plugin hook is crashing at startup 649907676
652990131 https://github.com/simonw/datasette/issues/889#issuecomment-652990131 https://api.github.com/repos/simonw/datasette/issues/889 MDEyOklzc3VlQ29tbWVudDY1Mjk5MDEzMQ== amjith 49260 2020-07-02T12:58:11Z 2020-07-02T13:00:18Z CONTRIBUTOR

FWIW, this error does NOT happen in datasette 0.45a4.

It only started on 0.45a5

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
asgi_wrapper plugin hook is crashing at startup 649907676
652732460 https://github.com/simonw/datasette/issues/886#issuecomment-652732460 https://api.github.com/repos/simonw/datasette/issues/886 MDEyOklzc3VlQ29tbWVudDY1MjczMjQ2MA== simonw 9599 2020-07-02T01:52:02Z 2020-07-02T01:52:02Z OWNER

In investigating this I'm not convinced 500 errors are being correctly raised by errors in canned writable queries.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Reconsider how _actor_X magic parameter deals with missing values 649429772
652731459 https://github.com/simonw/datasette/issues/886#issuecomment-652731459 https://api.github.com/repos/simonw/datasette/issues/886 MDEyOklzc3VlQ29tbWVudDY1MjczMTQ1OQ== simonw 9599 2020-07-02T01:48:08Z 2020-07-02T01:48:08Z OWNER

A common error with this (and other) magic parameters is for the database query to result in the following:

You did not supply a value for binding 3.

This is a pretty crufty error. I'm inclined to say that ANY missing or invalid magic parameter should be treated as a None value instead.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Reconsider how _actor_X magic parameter deals with missing values 649429772
652711822 https://github.com/simonw/datasette/issues/887#issuecomment-652711822 https://api.github.com/repos/simonw/datasette/issues/887 MDEyOklzc3VlQ29tbWVudDY1MjcxMTgyMg== simonw 9599 2020-07-02T00:31:33Z 2020-07-02T00:31:33Z OWNER

If a canned query has a title defined that will be used instead: https://latest.datasette.io/fixtures/neighborhood_search

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Canned query page should show the name of the canned query 649437530
652711562 https://github.com/simonw/datasette/issues/887#issuecomment-652711562 https://api.github.com/repos/simonw/datasette/issues/887 MDEyOklzc3VlQ29tbWVudDY1MjcxMTU2Mg== simonw 9599 2020-07-02T00:30:43Z 2020-07-02T00:30:43Z OWNER

Demo has updated: https://latest.datasette.io/fixtures/magic_parameters

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Canned query page should show the name of the canned query 649437530
652710178 https://github.com/simonw/datasette/pull/883#issuecomment-652710178 https://api.github.com/repos/simonw/datasette/issues/883 MDEyOklzc3VlQ29tbWVudDY1MjcxMDE3OA== simonw 9599 2020-07-02T00:25:44Z 2020-07-02T00:25:44Z OWNER

This is a great idea.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Skip counting hidden tables 648749062
652709199 https://github.com/simonw/datasette/issues/887#issuecomment-652709199 https://api.github.com/repos/simonw/datasette/issues/887 MDEyOklzc3VlQ29tbWVudDY1MjcwOTE5OQ== simonw 9599 2020-07-02T00:21:54Z 2020-07-02T00:21:54Z OWNER

Example in the live demo: https://latest.datasette.io/fixtures/magic_parameters

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Canned query page should show the name of the canned query 649437530
652681996 https://github.com/simonw/datasette/issues/885#issuecomment-652681996 https://api.github.com/repos/simonw/datasette/issues/885 MDEyOklzc3VlQ29tbWVudDY1MjY4MTk5Ng== simonw 9599 2020-07-01T22:44:47Z 2020-07-01T22:44:47Z OWNER

https://simonwillison.net/2020/Jul/1/datasette-045/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Blog entry about the release 649373451
652663177 https://github.com/simonw/datasette/issues/882#issuecomment-652663177 https://api.github.com/repos/simonw/datasette/issues/882 MDEyOklzc3VlQ29tbWVudDY1MjY2MzE3Nw== simonw 9599 2020-07-01T21:48:08Z 2020-07-01T21:48:08Z OWNER

https://datasette.readthedocs.io/en/latest/changelog.html#v0-45

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Release notes for 0.45 648673556
652646487 https://github.com/simonw/datasette/issues/880#issuecomment-652646487 https://api.github.com/repos/simonw/datasette/issues/880 MDEyOklzc3VlQ29tbWVudDY1MjY0NjQ4Nw== simonw 9599 2020-07-01T21:05:48Z 2020-07-01T21:05:48Z OWNER

I've been testing the WIP using this in the console:

fetch('/data/add_name.json', {
  method: 'POST',
  body: 'name=XXXfetch',
  credentials: 'omit',
  headers: {'Content-Type': 'application/x-www-form-urlencoded'}
})
.then(response => console.log(response))

Against a canned query configured like this:

databases:
  data:
    queries:
      add_name:
        sql: insert into names (name) values (:name)
        write: true

I haven't got it to work yet. Latest error is this one:

INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
Traceback (most recent call last):
  File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 975, in route_path
    await response.asgi_send(send)
AttributeError: 'tuple' object has no attribute 'asgi_send'
INFO:     127.0.0.1:49938 - "POST /data/add_name.json HTTP/1.1" 500 Internal Server Error

It looks like I'm going to have to rethink how the BaseView code around tables, formats and hashes is structured in order to fix this. That's a big refactoring! I'm moving this to a new milestone for Datasette 0.46.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
POST to /db/canned-query.json should be supported 648637666
652604569 https://github.com/simonw/datasette/issues/882#issuecomment-652604569 https://api.github.com/repos/simonw/datasette/issues/882 MDEyOklzc3VlQ29tbWVudDY1MjYwNDU2OQ== simonw 9599 2020-07-01T19:27:17Z 2020-07-01T19:27:17Z OWNER

Don't forget to update the news in the README.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Release notes for 0.45 648673556
652597975 https://github.com/simonw/datasette/issues/877#issuecomment-652597975 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MjU5Nzk3NQ== simonw 9599 2020-07-01T19:12:15Z 2020-07-01T19:12:15Z OWNER

The latest release of https://github.com/simonw/datasette-auth-tokens (0.2) now supports SQL configuration of tokens.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
652520496 https://github.com/simonw/datasette/issues/877#issuecomment-652520496 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MjUyMDQ5Ng== simonw 9599 2020-07-01T16:26:52Z 2020-07-01T16:26:52Z OWNER

Tokens get verified by plugins. So far there's only one: https://github.com/simonw/datasette-auth-tokens - which has you hard-coding plugins in a configuration file. I have a issue there to add support for database-backed tokens too: https://github.com/simonw/datasette-auth-tokens/issues/1

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
652394742 https://github.com/simonw/datasette/pull/883#issuecomment-652394742 https://api.github.com/repos/simonw/datasette/issues/883 MDEyOklzc3VlQ29tbWVudDY1MjM5NDc0Mg== abdusco 3243482 2020-07-01T12:41:13Z 2020-07-01T12:41:13Z CONTRIBUTOR

Well tests need to be updated.

I need to get tests working on Windows.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Skip counting hidden tables 648749062
652311990 https://github.com/simonw/datasette/pull/883#issuecomment-652311990 https://api.github.com/repos/simonw/datasette/issues/883 MDEyOklzc3VlQ29tbWVudDY1MjMxMTk5MA== codecov[bot] 22429695 2020-07-01T09:40:40Z 2020-07-01T09:40:40Z NONE

Codecov Report

Merging #883 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #883   +/-   ##
=======================================
  Coverage   83.42%   83.42%           
=======================================
  Files          27       27           
  Lines        3632     3632           
=======================================
  Hits         3030     3030           
  Misses        602      602           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 676bb64...251884f. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Skip counting hidden tables 648749062
652297139 https://github.com/simonw/datasette/pull/883#issuecomment-652297139 https://api.github.com/repos/simonw/datasette/issues/883 MDEyOklzc3VlQ29tbWVudDY1MjI5NzEzOQ== abdusco 3243482 2020-07-01T09:11:29Z 2020-07-01T09:11:29Z CONTRIBUTOR

Turns out we should include hidden tables in the result dict, or we're breaking tests. I've committed a refactor https://github.com/simonw/datasette/pull/883/commits/4f06e1bf6fbe4b73be770b87f610bf7c0e6e3ea7

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Skip counting hidden tables 648749062
652255960 https://github.com/simonw/datasette/issues/877#issuecomment-652255960 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MjI1NTk2MA== abdusco 3243482 2020-07-01T07:52:25Z 2020-07-01T08:10:00Z CONTRIBUTOR

I am calling the API from another origin, so injecting CSRF token into templates wouldn't work.

EDIT:

I'll try the new version, it sounds promising

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
652261382 https://github.com/simonw/datasette/issues/877#issuecomment-652261382 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MjI2MTM4Mg== abdusco 3243482 2020-07-01T08:03:17Z 2020-07-01T08:03:23Z CONTRIBUTOR

Bearer tokens sound interesting. Where do tokens come from? An auth provider of my choosing? How do they get verified?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
652182990 https://github.com/simonw/datasette/issues/877#issuecomment-652182990 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MjE4Mjk5MA== simonw 9599 2020-07-01T04:29:38Z 2020-07-01T04:42:59Z OWNER

Have you tried the method described here? https://datasette.readthedocs.io/en/latest/internals.html#csrf-protection - I'm happy to bulk out that section of the documentation if that doesn't help solve your problem.

I just closed #835 which should make CSRF protection easier to work with - it won't interfere with requests without cookies or requests with Authentication: Bearer token tokens. See also https://github.com/simonw/asgi-csrf/issues/11

You can try out pip install datasette==0.45a5 to get those features. Hopefully releasing a full 0.45 tomorrow.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
652166115 https://github.com/simonw/datasette/issues/877#issuecomment-652166115 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MjE2NjExNQ== abdusco 3243482 2020-07-01T03:28:07Z 2020-07-01T03:28:07Z CONTRIBUTOR

Does this mean custom routes get to expose endpoints accepting POST requests? I've tried earlier to add some POST endpoints, but requests were being rejected by Datasette due to CSRF

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
652165709 https://github.com/simonw/datasette/issues/812#issuecomment-652165709 https://api.github.com/repos/simonw/datasette/issues/812 MDEyOklzc3VlQ29tbWVudDY1MjE2NTcwOQ== simonw 9599 2020-07-01T03:26:35Z 2020-07-01T03:26:35Z OWNER

This case may not be covered without extra work:
https://github.com/simonw/datasette/blob/3ec5b1abf6afa2d22a3378092809a1a8c0249d26/datasette/views/database.py#L122-L123

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to customize what happens when a view permission fails 634112607
652163450 https://github.com/simonw/datasette/issues/812#issuecomment-652163450 https://api.github.com/repos/simonw/datasette/issues/812 MDEyOklzc3VlQ29tbWVudDY1MjE2MzQ1MA== simonw 9599 2020-07-01T03:18:51Z 2020-07-01T03:20:28Z OWNER

This can be a plugin hook:

@hookspec
def forbidden(datasette, request, message, send):
    "Custom response for a 403 forbidden error"

If the hook returns a Response object, it will be returned to the user. Plugins are likely to want to return a redirect response.

Maybe the hook can instead use the send argument to respond to the request and return True which means "I've responded to this"?

I'm going to leave send off for the moment - I can add that in the future if it turns out it would have been a good idea.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to customize what happens when a view permission fails 634112607
652162722 https://github.com/simonw/datasette/issues/880#issuecomment-652162722 https://api.github.com/repos/simonw/datasette/issues/880 MDEyOklzc3VlQ29tbWVudDY1MjE2MjcyMg== simonw 9599 2020-07-01T03:16:07Z 2020-07-01T03:16:07Z OWNER

The response from this will never be a 302 - it will always be a 200 if the response worked or a 400 for bad parameters or a 500 for errors. The body returned will always be in JSON format.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
POST to /db/canned-query.json should be supported 648637666
652160909 https://github.com/simonw/datasette/issues/859#issuecomment-652160909 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY1MjE2MDkwOQ== abdusco 3243482 2020-07-01T03:09:32Z 2020-07-01T03:10:21Z CONTRIBUTOR

I've just realized Datasette tries to count hidden tables too. There are 5 visible tables, 25 hidden tables, which I haven't realize earlier to consider their effect. I've turned off counting for hidden tables to see if it has any effect.

What's the point of counting FTS tables?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
652159398 https://github.com/simonw/datasette/issues/835#issuecomment-652159398 https://api.github.com/repos/simonw/datasette/issues/835 MDEyOklzc3VlQ29tbWVudDY1MjE1OTM5OA== simonw 9599 2020-07-01T03:03:51Z 2020-07-01T03:03:51Z OWNER

I'm going to add some tests for this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for skipping CSRF checks on API posts 637363686
652106227 https://github.com/simonw/datasette/issues/876#issuecomment-652106227 https://api.github.com/repos/simonw/datasette/issues/876 MDEyOklzc3VlQ29tbWVudDY1MjEwNjIyNw== simonw 9599 2020-06-30T23:49:55Z 2020-06-30T23:50:04Z OWNER

Done: https://latest.datasette.io/-/patterns

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add log out link to the pattern portfolio 647879783
652105722 https://github.com/simonw/datasette/issues/879#issuecomment-652105722 https://api.github.com/repos/simonw/datasette/issues/879 MDEyOklzc3VlQ29tbWVudDY1MjEwNTcyMg== simonw 9599 2020-06-30T23:48:06Z 2020-06-30T23:48:06Z OWNER

Updated documentation: https://datasette.readthedocs.io/en/latest/pages.html

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page documentation still talks about hashes in URLs 648569227
652103895 https://github.com/simonw/datasette/issues/832#issuecomment-652103895 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MjEwMzg5NQ== simonw 9599 2020-06-30T23:41:22Z 2020-06-30T23:41:22Z OWNER

I don't think this needs any additional documentation - the new behaviour matches how the permissions are documented here: https://datasette.readthedocs.io/en/0.44/authentication.html#built-in-permissions

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651999516 https://github.com/simonw/datasette/issues/832#issuecomment-651999516 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MTk5OTUxNg== simonw 9599 2020-06-30T19:33:49Z 2020-06-30T21:34:59Z OWNER

Tests needed for this:

  • If a user has view table but NOT view database / view instance, can they view the table page?
  • If a user has view canned query but NOT view database / view instance, can they view the canned query page?
  • If a user has view database but NOT view instance, can they view the database page?
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651995453 https://github.com/simonw/datasette/issues/832#issuecomment-651995453 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MTk5NTQ1Mw== simonw 9599 2020-06-30T19:25:13Z 2020-06-30T19:25:26Z OWNER

I'm going to put the new check_permissions() method on BaseView as well. If I want that method to be available to plugins I can do so by turning that BaseView class into a documented API that plugins are encouraged to use themselves.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651994978 https://github.com/simonw/datasette/issues/832#issuecomment-651994978 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MTk5NDk3OA== simonw 9599 2020-06-30T19:24:12Z 2020-06-30T19:24:12Z OWNER

Hah... but check_permissionis a method onBaseView`. Here are the various permission methods at the moment:

https://github.com/simonw/datasette/blob/6c2634583627bfab750c115cb13850252821d637/datasette/default_permissions.py#L5-L14

And on BaseView:

https://github.com/simonw/datasette/blob/a8a5f813722f72703a7aae41135ccc40635cc02f/datasette/views/base.py#L65-L70

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651993977 https://github.com/simonw/datasette/issues/832#issuecomment-651993977 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MTk5Mzk3Nw== simonw 9599 2020-06-30T19:22:06Z 2020-06-30T19:22:06Z OWNER

permission_allowed is already the name of the pugin hook. It's actually a bit confusing that it's also the name of a method on datasette..

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651993537 https://github.com/simonw/datasette/issues/832#issuecomment-651993537 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MTk5MzUzNw== simonw 9599 2020-06-30T19:21:15Z 2020-06-30T19:21:15Z OWNER

I could rename permission_allowed() to check_permission() and have a complementary check_permissions() method.

This is a breaking change but we're pre-1.0 so I think that's OK. I could even set up a temporary permission_allowed() alias which prints a deprecation warning to the console, then remove that at 1.0.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651992737 https://github.com/simonw/datasette/issues/832#issuecomment-651992737 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MTk5MjczNw== simonw 9599 2020-06-30T19:19:33Z 2020-06-30T19:20:02Z OWNER

I already have this method on Datasette:

async def permission_allowed(self, actor, action, resource=None, default=False):

What would be a good method name that complements that and indicates "check a list of permissions in order"? Should it even run against the request or should you have to hand it request.actor?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651984989 https://github.com/simonw/datasette/issues/877#issuecomment-651984989 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MTk4NDk4OQ== simonw 9599 2020-06-30T19:03:25Z 2020-06-30T19:03:25Z OWNER

Relevant: #835

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
651984355 https://github.com/simonw/datasette/issues/877#issuecomment-651984355 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MTk4NDM1NQ== simonw 9599 2020-06-30T19:02:15Z 2020-06-30T19:02:15Z OWNER

https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#login-csrf

Login CSRF can be mitigated by creating pre-sessions (sessions before a user is authenticated) and including tokens in login form.

Sounds like regular CSRF protection to me.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
650340914 https://github.com/simonw/datasette/pull/868#issuecomment-650340914 https://api.github.com/repos/simonw/datasette/issues/868 MDEyOklzc3VlQ29tbWVudDY1MDM0MDkxNA== codecov[bot] 22429695 2020-06-26T18:53:02Z 2020-06-30T03:51:22Z NONE

Codecov Report

Merging #868 into master will increase coverage by 0.11%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #868      +/-   ##
==========================================
+ Coverage   83.31%   83.42%   +0.11%     
==========================================
  Files          27       27              
  Lines        3595     3614      +19     
==========================================
+ Hits         2995     3015      +20     
+ Misses        600      599       -1     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/utils/__init__.py</td> <td>93.93% <0.00%> (+0.05%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/app.py</td> <td>96.47% <0.00%> (+0.19%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/views/special.py</td> <td>81.17% <0.00%> (+3.39%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8a5f81...ef837b3. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
initial windows ci setup 646448486
651302221 https://github.com/simonw/datasette/issues/805#issuecomment-651302221 https://api.github.com/repos/simonw/datasette/issues/805 MDEyOklzc3VlQ29tbWVudDY1MTMwMjIyMQ== simonw 9599 2020-06-29T19:02:45Z 2020-06-29T19:05:26Z OWNER

No I prefer the idea that logged out users can still perform some writes, in a not-likely-to-attract-abuse way.

So a root-user-can-configure-polls, logged-out-users-can-vote-in-them demo would be good.

Or... crazy idea: a collaborative drawing program? A grid of cells of emoji, anyone can add an emoji to a cell. Would involve a bit of JavaScript. I could use https://github.com/joeattardi/emoji-button for this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries live demo on Glitch 632724154
651301202 https://github.com/simonw/datasette/issues/805#issuecomment-651301202 https://api.github.com/repos/simonw/datasette/issues/805 MDEyOklzc3VlQ29tbWVudDY1MTMwMTIwMg== simonw 9599 2020-06-29T19:00:37Z 2020-06-29T19:00:37Z OWNER

How about a blog? Pre-configured canned queries that are only available to "root", plus datasette-template-sql and default templates for the index page and blog entry pages.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries live demo on Glitch 632724154
651293559 https://github.com/simonw/datasette/issues/875#issuecomment-651293559 https://api.github.com/repos/simonw/datasette/issues/875 MDEyOklzc3VlQ29tbWVudDY1MTI5MzU1OQ== simonw 9599 2020-06-29T18:43:50Z 2020-06-29T18:43:50Z OWNER

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Logged in as: XXX - logout" navigation item 647103735
651203178 https://github.com/simonw/datasette/issues/873#issuecomment-651203178 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MTIwMzE3OA== simonw 9599 2020-06-29T15:44:38Z 2020-06-29T15:44:54Z OWNER

I'm having real trouble figuring out how to gain access to the port that was used to start the server. I'm treating this as a very low priority - it only affects the exact -p 0 --root combination which isn't going to affect many people at all.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
651193594 https://github.com/simonw/datasette/issues/873#issuecomment-651193594 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MTE5MzU5NA== simonw 9599 2020-06-29T15:27:46Z 2020-06-29T15:27:46Z OWNER

Uninstalling datasette-debug-asgi caused the server to startup correctly again.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
651193131 https://github.com/simonw/datasette/issues/873#issuecomment-651193131 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MTE5MzEzMQ== simonw 9599 2020-06-29T15:27:00Z 2020-06-29T15:27:00Z OWNER

Aha! Yes it's not being called, and the reason is this: https://github.com/encode/starlette/issues/486

Short version: by default an exception raised during that phase is silently swallowed! You can avoid the swallowing by adding lifespan="on" to the call to uvicorn.run().

When I did that here:

uvicorn.run(ds.app(), host=host, port=port, log_level="info", lifespan="on")

The server failed to start with this error:

INFO:     Started server process [68849]
INFO:     Waiting for application startup.
ERROR:    Exception in 'lifespan' protocol
Traceback (most recent call last):
  File ".../uvicorn/lifespan/on.py", line 48, in main
    await app(scope, self.receive, self.send)
  File ".../uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File ".../datasette_debug_asgi.py", line 9, in wrapped_app
    if scope["path"] == "/-/asgi-scope":
KeyError: 'path'
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650910137 https://github.com/simonw/datasette/issues/873#issuecomment-650910137 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkxMDEzNw== simonw 9599 2020-06-29T05:16:32Z 2020-06-29T05:16:32Z OWNER

I'm not convinced that function is ever actually being called - I added a print() statement to it and it's not executing. I don't think the tests cover it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650909476 https://github.com/simonw/datasette/issues/873#issuecomment-650909476 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwOTQ3Ng== simonw 9599 2020-06-29T05:14:08Z 2020-06-29T05:14:08Z OWNER

I already have a AsgiLifespan class:
https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L896-L905

It runs this function: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L890-L894

Could that startup function also output the --root login URL, if needed?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650909136 https://github.com/simonw/datasette/issues/873#issuecomment-650909136 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwOTEzNg== simonw 9599 2020-06-29T05:12:58Z 2020-06-29T05:12:58Z OWNER

On startup Datasette currently outputs:

INFO:     Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.

So the ASGI lifespan protocol is almost certainly the right way to solve this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650908854 https://github.com/simonw/datasette/issues/873#issuecomment-650908854 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwODg1NA== simonw 9599 2020-06-29T05:12:04Z 2020-06-29T05:12:04Z OWNER

Can I detect the port the server is running on from within the regular Datasette ASGI code? If so I could use that ability and maybe output the magic --root link a second after the server starts up somehow.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650908534 https://github.com/simonw/datasette/issues/873#issuecomment-650908534 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwODUzNA== simonw 9599 2020-06-29T05:11:06Z 2020-06-29T05:11:06Z OWNER

Uvicorn's lifespan stuff isn't easy to figure out, but this test suite holds some clues: https://github.com/encode/uvicorn/blob/master/tests/test_lifespan.py

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650907323 https://github.com/simonw/datasette/issues/873#issuecomment-650907323 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwNzMyMw== simonw 9599 2020-06-29T05:07:16Z 2020-06-29T05:07:16Z OWNER

This line is interesting: is this a hook I can attach to somehow?

        await self.lifespan.startup()

From https://github.com/encode/uvicorn/blob/a75fe1381f6b1f78901691c71894f3cf487b5d30/uvicorn/main.py#L475

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650906533 https://github.com/simonw/datasette/issues/873#issuecomment-650906533 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwNjUzMw== simonw 9599 2020-06-29T05:04:44Z 2020-06-29T05:04:44Z OWNER

The challenge is... can we run our own custom code after that line has executed that has access to server and can hence access server.servers[0].sockets[0].getsockname()[1] to find the port?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650906318 https://github.com/simonw/datasette/issues/873#issuecomment-650906318 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwNjMxOA== simonw 9599 2020-06-29T05:04:04Z 2020-06-29T05:04:12Z OWNER

Within uvicorn it does this:

            if port == 0:
                port = server.sockets[0].getsockname()[1]

That server variable is later stashed here:

self.servers = [server]

Where self is the instance of class Server - which is the class that Uvicorn instantiates and calls .run() on when we do uvicorn.run() here: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/cli.py#L409

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650905399 https://github.com/simonw/datasette/issues/873#issuecomment-650905399 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwNTM5OQ== simonw 9599 2020-06-29T05:01:03Z 2020-06-29T05:01:03Z OWNER

This is a bit tricky to fix. This change to uvicorn is relevant: https://github.com/encode/uvicorn/commit/a75fe1381f6b1f78901691c71894f3cf487b5d30

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650899265 https://github.com/simonw/datasette/issues/875#issuecomment-650899265 https://api.github.com/repos/simonw/datasette/issues/875 MDEyOklzc3VlQ29tbWVudDY1MDg5OTI2NQ== simonw 9599 2020-06-29T04:34:32Z 2020-06-29T04:34:32Z OWNER

From https://github.com/simonw/datasette/issues/840#issuecomment-643454625

Another problem: what to display in the "you are logged in as", since we don't dictate an actor design.

I'm going to use a includes template for this that can easily be over-ridden by administrators or by plugins.

The default will look for the first available of the following keys:

* display
* name
* username
* login
* id
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Logged in as: XXX - logout" navigation item 647103735
650898808 https://github.com/simonw/datasette/issues/875#issuecomment-650898808 https://api.github.com/repos/simonw/datasette/issues/875 MDEyOklzc3VlQ29tbWVudDY1MDg5ODgwOA== simonw 9599 2020-06-29T04:32:31Z 2020-06-29T04:33:30Z OWNER

I could borrow the implementation for this from datasette-auth-github
https://github.com/simonw/datasette-auth-github/blob/182298b034ecb647971b65057d1d3e7b7fbbb482/datasette_auth_github/templates/base.html

{% extends "default:base.html" %}

{% block extra_head %}
<style type="text/css">
.hd .logout {
    float: right;
    text-align: right;
    padding-left: 1em;
}
</style>
{% endblock %}

{% block nav %}
    {{ super() }}
    {% if auth and auth.username %}
        <p class="logout">
            <strong>{{ auth.username }}</strong> &middot; <a href="/-/logout">Log out</a>
        </p>
    {% endif %}
{% endblock %}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Logged in as: XXX - logout" navigation item 647103735
650895874 https://github.com/simonw/datasette/issues/840#issuecomment-650895874 https://api.github.com/repos/simonw/datasette/issues/840 MDEyOklzc3VlQ29tbWVudDY1MDg5NTg3NA== simonw 9599 2020-06-29T04:18:59Z 2020-06-29T04:19:11Z OWNER

Now just need the "Logged in as: XXX <logout>" navigation item.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Log out mechanism for clearing ds_actor cookie 637966833
650891502 https://github.com/simonw/datasette/issues/840#issuecomment-650891502 https://api.github.com/repos/simonw/datasette/issues/840 MDEyOklzc3VlQ29tbWVudDY1MDg5MTUwMg== simonw 9599 2020-06-29T03:58:08Z 2020-06-29T03:58:08Z OWNER

Step one: a "logout" page at /-/logout - which shows you a single CSRF-protected "logout" button if you do a GET against it and logs you out if you do a POST against it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Log out mechanism for clearing ds_actor cookie 637966833
650891257 https://github.com/simonw/datasette/issues/805#issuecomment-650891257 https://api.github.com/repos/simonw/datasette/issues/805 MDEyOklzc3VlQ29tbWVudDY1MDg5MTI1Nw== simonw 9599 2020-06-29T03:56:48Z 2020-06-29T03:56:48Z OWNER

Using datasette-glitch and the new https://github.com/simonw/datasette-write - currently running on datasette==0.45a4 - works on Glitch. The console shows a login link which gives you a cookie which allows you access to the /-/write interface.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries live demo on Glitch 632724154
650847013 https://github.com/simonw/datasette/issues/864#issuecomment-650847013 https://api.github.com/repos/simonw/datasette/issues/864 MDEyOklzc3VlQ29tbWVudDY1MDg0NzAxMw== simonw 9599 2020-06-29T00:41:55Z 2020-06-29T00:41:55Z OWNER

To test this I'll need a plugin test that renders a custom template. Here's an example I can imitate: https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/tests/test_plugins.py#L588-L596

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.add_message() doesn't work inside plugins 644309017
650846625 https://github.com/simonw/datasette/issues/864#issuecomment-650846625 https://api.github.com/repos/simonw/datasette/issues/864 MDEyOklzc3VlQ29tbWVudDY1MDg0NjYyNQ== simonw 9599 2020-06-29T00:39:47Z 2020-06-29T00:39:47Z OWNER

I think the fix is to move the "show_messages" variable to here:

https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/app.py#L735-L748

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.add_message() doesn't work inside plugins 644309017
650846473 https://github.com/simonw/datasette/issues/864#issuecomment-650846473 https://api.github.com/repos/simonw/datasette/issues/864 MDEyOklzc3VlQ29tbWVudDY1MDg0NjQ3Mw== simonw 9599 2020-06-29T00:39:04Z 2020-06-29T00:39:04Z OWNER

Re-opening: plugins may get to set messages but they don't display them, even if they render a template that extends base.html. For example, this code in a plugin:

        return Response.html(
            await datasette.render_template(
                "write.html",
                {"databases": databases, "sql": request.args.get("sql") or ""},
                request=request,
            )
        )

This won't display messages. The reason is that the messages are made available to the template context in the BaseView.render() method here:
https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/views/base.py#L87-L95

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.add_message() doesn't work inside plugins 644309017
650842514 https://github.com/simonw/datasette/issues/864#issuecomment-650842514 https://api.github.com/repos/simonw/datasette/issues/864 MDEyOklzc3VlQ29tbWVudDY1MDg0MjUxNA== simonw 9599 2020-06-29T00:12:59Z 2020-06-29T00:12:59Z OWNER

I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with register_routes()) as part of Datasette 0.45 but it would be OK if it slipped to a later release.
https://github.com/simonw/datasette/issues/870#issuecomment-650842381

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.add_message() doesn't work inside plugins 644309017
650842381 https://github.com/simonw/datasette/issues/870#issuecomment-650842381 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDg0MjM4MQ== simonw 9599 2020-06-29T00:12:07Z 2020-06-29T00:12:07Z OWNER

I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with register_routes()) as part of Datasette 0.45 but it would be OK if it slipped to a later release.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558
650838972 https://github.com/simonw/datasette/issues/870#issuecomment-650838972 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDgzODk3Mg== simonw 9599 2020-06-28T23:46:40Z 2020-06-28T23:46:40Z OWNER

I'm going to create the single Request() instance in the DatasetteRouter class - at the beginning of the route_path method: https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L905-L925

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558
650838691 https://github.com/simonw/datasette/issues/870#issuecomment-650838691 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDgzODY5MQ== simonw 9599 2020-06-28T23:44:12Z 2020-06-28T23:44:25Z OWNER

This code is interesting:

https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L948-L955

I want to change the signature of that return await view(new_scope, receive, send) method to instead take (request, send) - so I can have a single shared request object that's created just once per HTTP request.

The problem is the scope modification: I have code that modifies the scope, but how should that impact a shared Request instance? Should its .scope be replaced with alternative scopes as it travels through the codebase?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 869.647ms · About: github-to-sqlite