Taken in {{ display.place_city }} on {{ display.date }}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results, https://github.com/simonw/datasette/pull/952#issuecomment-686061028,https://api.github.com/repos/simonw/datasette/issues/952,686061028,MDEyOklzc3VlQ29tbWVudDY4NjA2MTAyOA==,27856297,dependabot-preview[bot],2020-09-02T22:26:14Z,2020-09-02T22:26:14Z,CONTRIBUTOR,"Looks like black is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687245650,"Update black requirement from ~=19.10b0 to >=19.10,<21.0", https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685970384,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685970384,MDEyOklzc3VlQ29tbWVudDY4NTk3MDM4NA==,9599,simonw,2020-09-02T20:11:41Z,2020-09-02T20:11:59Z,MEMBER,"Default categories: - 1 = created - 2 = saved","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""", https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966707,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685966707,MDEyOklzc3VlQ29tbWVudDY4NTk2NjcwNw==,9599,simonw,2020-09-02T20:04:08Z,2020-09-02T20:04:08Z,MEMBER,I'll make `category` a foreign key to a `categories` table so Datasette can automatically show the `name` column.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""", https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966361,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685966361,MDEyOklzc3VlQ29tbWVudDY4NTk2NjM2MQ==,9599,simonw,2020-09-02T20:03:29Z,2020-09-02T20:03:41Z,MEMBER,"I'm going to implement the first version of this as an indexed integer `category` column which has 1 for ""about me"" and 2 for ""liked by me"" - and space for other category numerals in the future, albeit a row can only belong to one category. I'll think about a full tagging system separately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""", https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685965516,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685965516,MDEyOklzc3VlQ29tbWVudDY4NTk2NTUxNg==,9599,simonw,2020-09-02T20:01:54Z,2020-09-02T20:01:54Z,MEMBER,"Relevant post: https://sqlite.org/forum/forumpost/9f06fedaa5 - drh says: > Indexes are one-to-one. There is one entry in the index for each row in the table. > > You are asking for an index that is many-to-one - multiple index entries for each table row. > > A Full-Text Index is basically a many-to-one index. So if all of your array entries really are words, you could probably get this to work using a Full-Text Index.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""", https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685962280,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685962280,MDEyOklzc3VlQ29tbWVudDY4NTk2MjI4MA==,9599,simonw,2020-09-02T19:55:26Z,2020-09-02T19:59:58Z,MEMBER,"Relevant: https://charlesleifer.com/blog/a-tour-of-tagging-schemas-many-to-many-bitmaps-and-more/ SQLite supports bitwise operators Binary AND (&) and Binary OR (|) - I could try those. Not sure how they interact with indexes though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""", https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-685961809,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3,685961809,MDEyOklzc3VlQ29tbWVudDY4NTk2MTgwOQ==,9599,simonw,2020-09-02T19:54:24Z,2020-09-02T19:54:24Z,MEMBER,"This should implement search highlighting too, as seen on https://til.simonwillison.net/til/search?q=cloud` tags for the cases that I care about so they're already expecting white-space to be pre wrapped in some way.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",658476055,Use white-space: pre-wrap on ALL table cell contents,
https://github.com/simonw/datasette/issues/896#issuecomment-659734703,https://api.github.com/repos/simonw/datasette/issues/896,659734703,MDEyOklzc3VlQ29tbWVudDY1OTczNDcwMw==,9599,simonw,2020-07-16T23:34:57Z,2020-07-16T23:34:57Z,OWNER,"I'm worried about how this will interact with some of the plugins:
* https://github.com/simonw/datasette-json-html
* https://github.com/simonw/datasette-pretty-json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",658476055,Use white-space: pre-wrap on ALL table cell contents,
https://github.com/simonw/datasette/issues/896#issuecomment-659615034,https://api.github.com/repos/simonw/datasette/issues/896,659615034,MDEyOklzc3VlQ29tbWVudDY1OTYxNTAzNA==,9599,simonw,2020-07-16T19:14:07Z,2020-07-16T19:14:07Z,OWNER,"Demo: https://srccon-2020.datasette.io/srccon?sql=select+id%2C+day%2C+time%2C+event_name%2C+event_description%2C+facilitators+from+sessions+order+by+event_dtstart+limit+101
I really like this:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",658476055,Use white-space: pre-wrap on ALL table cell contents,
https://github.com/simonw/datasette/issues/896#issuecomment-659610687,https://api.github.com/repos/simonw/datasette/issues/896,659610687,MDEyOklzc3VlQ29tbWVudDY1OTYxMDY4Nw==,9599,simonw,2020-07-16T19:05:43Z,2020-07-16T19:05:43Z,OWNER,I'm going to give this a go - if it turns out to be a bad idea I can revert it back out again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",658476055,Use white-space: pre-wrap on ALL table cell contents,
https://github.com/simonw/datasette/issues/895#issuecomment-659085528,https://api.github.com/repos/simonw/datasette/issues/895,659085528,MDEyOklzc3VlQ29tbWVudDY1OTA4NTUyOA==,9599,simonw,2020-07-16T00:32:47Z,2020-07-16T00:32:47Z,OWNER,"This was added in https://github.com/simonw/datasette/commit/504196341c49840270bd75ea1a1871ef386ba7ea - here's the relevant code (which only applies on the table page, not the query page):
https://github.com/simonw/datasette/blob/d6e03b04302a0852e7133dc030eab50177c37be7/datasette/views/table.py#L196-L204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657747959,SQL query output should show numeric values in a different colour,
https://github.com/simonw/datasette/issues/892#issuecomment-657268433,https://api.github.com/repos/simonw/datasette/issues/892,657268433,MDEyOklzc3VlQ29tbWVudDY1NzI2ODQzMw==,9599,simonw,2020-07-12T20:02:17Z,2020-07-12T20:02:35Z,OWNER,"Fixed https://datasette.readthedocs.io/en/latest/
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",655465863,"""latest"" in new documentation navbar is invisible",
https://github.com/simonw/datasette/issues/892#issuecomment-657268051,https://api.github.com/repos/simonw/datasette/issues/892,657268051,MDEyOklzc3VlQ29tbWVudDY1NzI2ODA1MQ==,9599,simonw,2020-07-12T19:58:24Z,2020-07-12T19:58:24Z,OWNER,"```css
.wy-side-nav-search > div.version {
margin-top: -.4045em;
margin-bottom: .809em;
font-weight: normal;
color: rgba(255,255,255,0.3);
}
```
Fix can go here: https://github.com/simonw/datasette/blob/ee0ef016523a765b6ef6eaa43cad9ad568f78ae4/docs/_static/css/custom.css#L1-L3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",655465863,"""latest"" in new documentation navbar is invisible",
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-656363548,https://api.github.com/repos/simonw/sqlite-utils/issues/114,656363548,MDEyOklzc3VlQ29tbWVudDY1NjM2MzU0OA==,9599,simonw,2020-07-09T21:37:28Z,2020-07-09T21:37:28Z,OWNER,"I'm going to add a second method `.transform_table_sql(...)` - which returns the SQL that would have been executed but does NOT execute it.
Advanced callers can use this to include their own additional steps in the same transaction - e.g. recreating views or triggers.
More importantly it gives me a useful hook for writing some unit tests against the generated SQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655898722,https://api.github.com/repos/simonw/sqlite-utils/issues/121,655898722,MDEyOklzc3VlQ29tbWVudDY1NTg5ODcyMg==,79913,tsibley,2020-07-09T04:53:08Z,2020-07-09T04:53:08Z,CONTRIBUTOR,"Yep, I agree that makes more sense for backwards compat and more casual use cases. I think it should be possible for the Database/Queryable methods to DTRT based on seeing if it's within a context-manager-managed transaction.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652961907,Improved (and better documented) support for transactions,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655786374,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655786374,MDEyOklzc3VlQ29tbWVudDY1NTc4NjM3NA==,9599,simonw,2020-07-08T22:16:54Z,2020-07-08T22:16:54Z,OWNER,"According to https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes the hardest bits to consider are how to deal with existing foreign key relationships, triggers and views.
I'm OK leaving views as an exercise for the caller - many of these transformations may not need any view changes at all.
Foreign key relationships are important: it should handle these automatically as effectively as possible.
Likewise trigger changes: need to think about what this means.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655785396,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655785396,MDEyOklzc3VlQ29tbWVudDY1NTc4NTM5Ng==,9599,simonw,2020-07-08T22:14:10Z,2020-07-08T22:14:10Z,OWNER,"Work in progress: not quite right yet, I need smarter logic for how renamed columns are reflected in the generated `INSERT INTO ... SELECT ...` query:
```python
def transform_table(
self,
columns=None,
rename=None,
change_type=None,
pk=None,
foreign_keys=None,
column_order=None,
not_null=None,
defaults=None,
hash_id=None,
extracts=None,
):
assert self.exists(), ""Cannot transform a table that doesn't exist yet""
columns = columns or self.columns_dict
if rename is not None or change_type is not None:
columns = {rename.get(key, key): change_type.get(key, value) for key, value in columns.items()}
new_table_name = ""{}_new_{}"".format(self.name, os.urandom(6).hex())
previous_columns = set(self.columns_dict.keys())
with self.db.conn:
columns = {name: value for (name, value) in columns.items()}
new_table = self.db.create_table(
new_table_name,
columns,
pk=pk,
foreign_keys=foreign_keys,
column_order=column_order,
not_null=not_null,
defaults=defaults,
hash_id=hash_id,
extracts=extracts,
)
# Copy across data - but only for columns that exist in both
new_columns = set(columns.keys())
columns_to_copy = new_columns.intersection(previous_columns)
copy_sql = ""INSERT INTO [{new_table}] ({new_cols}) SELECT {old_cols} FROM [{old_table}]"".format(
new_table=new_table_name,
old_table=self.name,
old_cols="", "".join(""[{}]"".format(col) for col in columns_to_copy),
new_cols="", "".join(""[{}]"".format(rename.get(col, col)) for col in columns_to_copy),
)
self.db.conn.execute(copy_sql)
# Drop the old table
self.db.conn.execute(""DROP TABLE [{}]"".format(self.name))
# Rename the new one
self.db.conn.execute(
""ALTER TABLE [{}] RENAME TO [{}]"".format(new_table_name, self.name)
)
return self
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655783875,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655783875,MDEyOklzc3VlQ29tbWVudDY1NTc4Mzg3NQ==,9599,simonw,2020-07-08T22:09:51Z,2020-07-08T22:10:16Z,OWNER,I can have a convenient `change_type={...}` parameter for changing column types too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655782477,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655782477,MDEyOklzc3VlQ29tbWVudDY1NTc4MjQ3Nw==,9599,simonw,2020-07-08T22:06:23Z,2020-07-08T22:06:23Z,OWNER,"Thinking about the method signature:
```python
def transform_table(
self,
columns,
pk=None,
foreign_keys=None,
column_order=None,
not_null=None,
defaults=None,
hash_id=None,
extracts=None,
):
```
This requires the caller to provide the exact set of columns for the new table.
It would be useful if this was optional - if you could omit the columns and have it automatically use the previous columns. This would let you change things like the primary key or the column order using the other arguments.
Even better: allow column renaming using an optional `rename={...}` argument:
```python
db[""dogs""].transform_table(rename={""name"": ""dog_name""})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655778058,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655778058,MDEyOklzc3VlQ29tbWVudDY1NTc3ODA1OA==,9599,simonw,2020-07-08T21:54:30Z,2020-07-08T21:54:30Z,OWNER,"Don't forget this step:
> If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677909,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655677909,MDEyOklzc3VlQ29tbWVudDY1NTY3NzkwOQ==,9599,simonw,2020-07-08T18:16:39Z,2020-07-08T18:16:39Z,OWNER,"Since neither the term ""transform"" or ""migrate"" are used in the codebase at the moment, I think I'll go with `.transform_table()` - that leaves the term ""migrate"" available for any future database migrations system (similar to Django's).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677396,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655677396,MDEyOklzc3VlQ29tbWVudDY1NTY3NzM5Ng==,9599,simonw,2020-07-08T18:15:39Z,2020-07-08T18:15:39Z,OWNER,"Alternative possible names:
- `.transform_table()`
- `.migrate()`
- `.transform()`
I'm torn between `.migrate_table()` and `.transform_table()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677099,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655677099,MDEyOklzc3VlQ29tbWVudDY1NTY3NzA5OQ==,9599,simonw,2020-07-08T18:15:02Z,2020-07-08T18:15:02Z,OWNER,"I'm not so keen on that chained API - it's pretty complicated.
Here's an idea for a much simpler interface. Essentially it lets you say ""take table X and migrate its contents to a new table with this structure - then atomically rename the tables to switch them"":
```python
db[""mytable""].migrate_table({""id"": int, ""name"": str""}, pk=""id"")
```
The `migrate_table()` method would take the same exact signature as the `table.create()` method: https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L615-L625","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/119#issuecomment-655674910,https://api.github.com/repos/simonw/sqlite-utils/issues/119,655674910,MDEyOklzc3VlQ29tbWVudDY1NTY3NDkxMA==,9599,simonw,2020-07-08T18:10:18Z,2020-07-08T18:10:18Z,OWNER,"This will work similar to how `.add_foreign_keys()` works: turn on `writable_schema` and rewrite the `sql` for that table in the `sqlite_master` table.
Here's that code today - it could be adapted to include removal of foreign keys that we no longer want:
https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L391-L401","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652700770,Ability to remove a foreign key,
https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655673896,https://api.github.com/repos/simonw/sqlite-utils/issues/121,655673896,MDEyOklzc3VlQ29tbWVudDY1NTY3Mzg5Ng==,9599,simonw,2020-07-08T18:08:11Z,2020-07-08T18:08:11Z,OWNER,"I'm with you on most of this. Completely agreed that the CLI should do everything in a transaction.
The one thing I'm not keen on is forcing calling code to explicitly start a transaction, for a couple of reasons:
1. It will break all of the existing code out there
2. It doesn't match to how I most commonly use this library - as an interactive tool in a Jupyter notebook, where I'm generally working against a brand new scratch database and any errors don't actually matter
So... how about this: IF you wrap your code in a `with db:` block then the `.insert()` and suchlike methods expect you to manage transactions yourself. But if you don't use the context manager they behave like they do at the moment (or maybe a bit more sensibly).
That way existing code works as it does today, lazy people like me can call `.insert()` without thinking about transactions, but people writing actual production code (as opposed to Jupyter hacks) have a sensible way to take control of the transactions themselves.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652961907,Improved (and better documented) support for transactions,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655653292,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655653292,MDEyOklzc3VlQ29tbWVudDY1NTY1MzI5Mg==,9599,simonw,2020-07-08T17:26:02Z,2020-07-08T17:26:02Z,OWNER,"Awesome, thank you very much.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655652679,https://api.github.com/repos/simonw/sqlite-utils/issues/121,655652679,MDEyOklzc3VlQ29tbWVudDY1NTY1MjY3OQ==,79913,tsibley,2020-07-08T17:24:46Z,2020-07-08T17:24:46Z,CONTRIBUTOR,"Better transaction handling would be really great. Some of my thoughts on implementing better transaction discipline are in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728.
My preferences:
- Each CLI command should operate in a single transaction so that either the whole thing succeeds or the whole thing is rolled back. This avoids partially completed operations when an error occurs part way through processing. Partially completed operations are typically much harder to recovery from gracefully and may cause inconsistent data states.
- The Python API should be transaction-agnostic and rely on the caller to coordinate transactions. Only the caller knows how individual insert, create, update, etc operations/methods should be bundled conceptually into transactions. When the caller is the CLI, for example, that bundling would be at the CLI command-level. Other callers might want to break up operations into multiple transactions. Transactions are usually most useful when controlled at the application-level (like logging configuration) instead of the library level. The library needs to provide an API that's conducive to transaction use, though.
- The Python API should provide a context manager to provide consistent transactions handling with more useful defaults than Python's `sqlite3` module. The latter issues implicit `BEGIN` statements by default for most DML (`INSERT`, `UPDATE`, `DELETE`, … but not `SELECT`, I believe), but **not** DDL (`CREATE TABLE`, `DROP TABLE`, `CREATE VIEW`, …). Notably, the `sqlite3` module doesn't issue the implicit `BEGIN` until the first DML statement. It _does not_ issue it when entering the `with conn` block, like other DBAPI2-compatible modules do. The `with conn` block for `sqlite3` only arranges to commit or rollback an existing transaction when exiting. Including DDL and `SELECT`s in transactions is important for operation consistency, though. There are several existing bugs.python.org tickets about this and future changes are in the works, but sqlite-utils can provide its own API sooner. sqlite-utils's `Database` class could itself be a context manager (built on the `sqlite3` connection context manager) which additionally issues an explicit `BEGIN` when entering. This would then let Python API callers do something like:
```python
db = sqlite_utils.Database(path)
with db: # ← BEGIN issued here by Database.__enter__
db.insert(…)
db.create_view(…)
# ← COMMIT/ROLLBACK issue here by sqlite3.connection.__exit__
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652961907,Improved (and better documented) support for transactions,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655643078,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655643078,MDEyOklzc3VlQ29tbWVudDY1NTY0MzA3OA==,79913,tsibley,2020-07-08T17:05:59Z,2020-07-08T17:05:59Z,CONTRIBUTOR,"> The only thing missing from this PR is updates to the documentation.
Ah, yes, thanks for this reminder! I've repushed with doc bits added.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655290625,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655290625,MDEyOklzc3VlQ29tbWVudDY1NTI5MDYyNQ==,9599,simonw,2020-07-08T05:15:45Z,2020-07-08T05:15:45Z,OWNER,"Ideally this would all happen in a single transaction, such that other processes talking to the database would not see any inconsistent state while the table copy was taking place. Need to confirm that this is possible. Also refs transactions thoughts in #121.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/pull/120#issuecomment-655289686,https://api.github.com/repos/simonw/sqlite-utils/issues/120,655289686,MDEyOklzc3VlQ29tbWVudDY1NTI4OTY4Ng==,9599,simonw,2020-07-08T05:13:11Z,2020-07-08T05:13:11Z,OWNER,"This is an excellent fix, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652816158,Fix query command's support for DML,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655286864,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655286864,MDEyOklzc3VlQ29tbWVudDY1NTI4Njg2NA==,9599,simonw,2020-07-08T05:05:27Z,2020-07-08T05:05:36Z,OWNER,"The only thing missing from this PR is updates to the documentation. Those need to go in two places:
- In the Python API docs. I suggest adding a note to this section about bulk inserts: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/python-api.rst#bulk-inserts
- In the CLI docs, in this section: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/cli.rst#inserting-json-data
Here's an example of a previous commit that includes updates to both CLI and API documentation: https://github.com/simonw/sqlite-utils/commit/f9473ace14878212c1fa968b7bd2f51e4f064dba#diff-e3e2a9bfd88566b05001b02a3f51d286","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284168,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655284168,MDEyOklzc3VlQ29tbWVudDY1NTI4NDE2OA==,9599,simonw,2020-07-08T04:58:00Z,2020-07-08T04:58:00Z,OWNER,"Oops didn't mean to click ""close"" there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284054,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655284054,MDEyOklzc3VlQ29tbWVudDY1NTI4NDA1NA==,9599,simonw,2020-07-08T04:57:38Z,2020-07-08T04:57:38Z,OWNER,Thoughts on transactions would be much appreciated in #121 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655283393,MDEyOklzc3VlQ29tbWVudDY1NTI4MzM5Mw==,9599,simonw,2020-07-08T04:55:18Z,2020-07-08T04:55:18Z,OWNER,"This is a really good idea - and thank you for the detailed discussion in the pull request.
I'm keen to discuss how transactions can work better. I tend to use this pattern in my own code:
with db.conn:
db[""table""].insert(...)
But it's not documented and I've not though very hard about it!
I like having inserts that handle 10,000+ rows commit on every chunk so I can watch their progress from another process, but the library should absolutely support people who want to commit all of the rows in a single transaction - or combine changes with DML.
Lots to discuss here. I'll start a new issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655239728,MDEyOklzc3VlQ29tbWVudDY1NTIzOTcyOA==,79913,tsibley,2020-07-08T02:16:42Z,2020-07-08T02:16:42Z,CONTRIBUTOR,"I fixed my original oops by moving the `DELETE FROM $table` out of the chunking loop and repushed. I think this change can be considered in isolation from issues around transactions, which I discuss next.
I wanted to make the DELETE + INSERT happen all in the same transaction so it was robust, but that was more complicated than I expected. The transaction handling in the Database/Table classes isn't systematic, and this poses big hurdles to making `Table.insert_all` (or other operations) consistent and robust in the face of errors.
For example, I wanted to do this (whitespace ignored in diff, so indentation change not highlighted):
```diff
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index d6b9ecf..4107ceb 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -1028,6 +1028,11 @@ class Table(Queryable):
batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))
self.last_rowid = None
self.last_pk = None
+ with self.db.conn:
+ # Explicit BEGIN is necessary because Python's sqlite3 doesn't
+ # issue implicit BEGINs for DDL, only DML. We mix DDL and DML
+ # below and might execute DDL first, e.g. for table creation.
+ self.db.conn.execute(""BEGIN"")
if truncate and self.exists():
self.db.conn.execute(""DELETE FROM [{}];"".format(self.name))
for chunk in chunks(itertools.chain([first_record], records), batch_size):
@@ -1038,7 +1043,11 @@ class Table(Queryable):
# Use the first batch to derive the table names
column_types = suggest_column_types(chunk)
column_types.update(columns or {})
- self.create(
+ # Not self.create() because that is wrapped in its own
+ # transaction and Python's sqlite3 doesn't support
+ # nested transactions.
+ self.db.create_table(
+ self.name,
column_types,
pk,
foreign_keys,
@@ -1139,7 +1148,6 @@ class Table(Queryable):
flat_values = list(itertools.chain(*values))
queries_and_params = [(sql, flat_values)]
- with self.db.conn:
for query, params in queries_and_params:
try:
result = self.db.conn.execute(query, params)
```
but that fails in tests because other methods call `insert/upsert/insert_all/upsert_all` in the middle of their transactions, so the BEGIN statement throws an error (no nested transactions allowed).
Stepping back, it would be nice to make the transaction handling systematic and predictable. One way to do this is to make the `sqlite_utils/db.py` code generally not begin or commit any transactions, and require the caller to do that instead. This lets the caller mix and match the Python API calls into transactions as appropriate (which is impossible for the API methods themselves to fully determine). Then, make `sqlite_utils/cli.py` begin and commit a transaction in each `@cli.command` function, making each command robust and consistent in the face of errors. The big change here, and why I didn't just submit a patch, is that it dramatically changes the Python API to _require_ callers to begin a transaction rather than just immediately calling methods.
There is also the caveat that for each transaction, an explicit `BEGIN` is also necessary so that DDL as well as DML (as well as `SELECT`s) are consistent and rolled back on error. There are several bugs.python.org discussions around this particular problem of DDL and some plans to make it better and consistent with DBAPI2, eventually. In the meantime, the sqlite-utils Database class could be a context manager which supports the incantations necessary to do proper transactions. This would still be a Python API change for callers but wouldn't expose them to the weirdness of the sqlite3's default transaction handling.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655052451,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655052451,MDEyOklzc3VlQ29tbWVudDY1NTA1MjQ1MQ==,79913,tsibley,2020-07-07T18:45:23Z,2020-07-07T18:45:23Z,CONTRIBUTOR,"Ah, I see the problem. The truncate is inside a loop I didn't realize was there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655018966,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655018966,MDEyOklzc3VlQ29tbWVudDY1NTAxODk2Ng==,79913,tsibley,2020-07-07T17:41:06Z,2020-07-07T17:41:06Z,CONTRIBUTOR,"Hmm, while tests pass, this may not work as intended on larger datasets. Looking into it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/datasette/issues/784#issuecomment-654424704,https://api.github.com/repos/simonw/datasette/issues/784,654424704,MDEyOklzc3VlQ29tbWVudDY1NDQyNDcwNA==,9599,simonw,2020-07-06T19:31:53Z,2020-07-06T19:31:53Z,OWNER,Documentation: https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628003707,Ability to sign in to Datasette as a root account,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653966670,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653966670,MDEyOklzc3VlQ29tbWVudDY1Mzk2NjY3MA==,9599,simonw,2020-07-06T01:07:02Z,2020-07-06T01:07:02Z,MEMBER,OK that fix worked.https://github.com/dogsheep/github-to-sqlite/runs/839764768?check_suite_focus=true,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962708,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653962708,MDEyOklzc3VlQ29tbWVudDY1Mzk2MjcwOA==,9599,simonw,2020-07-06T00:43:10Z,2020-07-06T00:43:10Z,MEMBER,I bet it's datasette-search-all.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962669,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653962669,MDEyOklzc3VlQ29tbWVudDY1Mzk2MjY2OQ==,9599,simonw,2020-07-06T00:42:57Z,2020-07-06T00:42:57Z,MEMBER,"https://github-to-sqlite.dogsheep.net/-/plugins
```json
[
{
""name"": ""datasette-json-html"",
""static"": false,
""templates"": false,
""version"": ""0.6"",
""hooks"": [
""prepare_connection"",
""render_cell""
]
},
{
""name"": ""datasette-render-markdown"",
""static"": false,
""templates"": false,
""version"": ""1.1.2"",
""hooks"": [
""extra_template_vars"",
""render_cell""
]
},
{
""name"": ""datasette-pretty-json"",
""static"": false,
""templates"": false,
""version"": ""0.2"",
""hooks"": [
""render_cell""
]
},
{
""name"": ""datasette-search-all"",
""static"": false,
""templates"": true,
""version"": ""0.2.1"",
""hooks"": [
""asgi_wrapper"",
""extra_template_vars""
]
},
{
""name"": ""datasette-vega"",
""static"": true,
""templates"": false,
""version"": ""0.6.2"",
""hooks"": [
""extra_css_urls"",
""extra_js_urls""
]
}
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962530,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653962530,MDEyOklzc3VlQ29tbWVudDY1Mzk2MjUzMA==,9599,simonw,2020-07-06T00:42:13Z,2020-07-06T00:42:13Z,MEMBER,So it looks like it's the ASGI lifespan change I made in https://github.com/simonw/datasette/commit/16f592247a2a0e140ada487e9972645406dcae69 - It must be incompatible with one of the plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962418,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653962418,MDEyOklzc3VlQ29tbWVudDY1Mzk2MjQxOA==,9599,simonw,2020-07-06T00:41:38Z,2020-07-06T00:41:38Z,MEMBER,"https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/logs?project=datasette-222320 has some clues.
![]()
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653960989,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653960989,MDEyOklzc3VlQ29tbWVudDY1Mzk2MDk4OQ==,9599,simonw,2020-07-06T00:32:34Z,2020-07-06T00:32:34Z,MEMBER,Same error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653947916,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653947916,MDEyOklzc3VlQ29tbWVudDY1Mzk0NzkxNg==,9599,simonw,2020-07-05T22:40:47Z,2020-07-05T22:40:47Z,MEMBER,Might be that it's not got enough RAM. I'll try deploying to a larger instance.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/simonw/datasette/pull/890#issuecomment-653314465,https://api.github.com/repos/simonw/datasette/issues/890,653314465,MDEyOklzc3VlQ29tbWVudDY1MzMxNDQ2NQ==,9599,simonw,2020-07-03T03:07:41Z,2020-07-03T03:07:41Z,OWNER,"This is an excellent fix. Thanks!
Not sure why codecov is complaining. I'm going to merge it as-is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",650305298,Load only python files from plugins-dir.,
https://github.com/simonw/datasette/pull/890#issuecomment-653309545,https://api.github.com/repos/simonw/datasette/issues/890,653309545,MDEyOklzc3VlQ29tbWVudDY1MzMwOTU0NQ==,22429695,codecov[bot],2020-07-03T02:52:25Z,2020-07-03T03:03:00Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=h1) Report
> Merging [#890](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/57879dc8b346a435804a9e45ffaacbf2a0228bc6&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `80.00%`.
[](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #890 +/- ##
==========================================
- Coverage 83.42% 83.40% -0.02%
==========================================
Files 27 27
Lines 3632 3634 +2
==========================================
+ Hits 3030 3031 +1
- Misses 602 603 +1
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/890/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.99% <80.00%> (-0.17%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=footer). Last update [57879dc...745af3b](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",650305298,Load only python files from plugins-dir.,
https://github.com/simonw/datasette/pull/848#issuecomment-643711117,https://api.github.com/repos/simonw/datasette/issues/848,643711117,MDEyOklzc3VlQ29tbWVudDY0MzcxMTExNw==,22429695,codecov[bot],2020-06-14T03:05:55Z,2020-07-03T02:44:09Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=h1) Report
> Merging [#848](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/57879dc8b346a435804a9e45ffaacbf2a0228bc6&el=desc) will **decrease** coverage by `0.60%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #848 +/- ##
==========================================
- Coverage 83.42% 82.82% -0.61%
==========================================
Files 27 26 -1
Lines 3632 3540 -92
==========================================
- Hits 3030 2932 -98
- Misses 602 608 +6
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `71.34% <0.00%> (-0.89%)` | :arrow_down: |
| [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `77.77% <0.00%> (-3.40%)` | :arrow_down: |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.58% <0.00%> (-1.58%)` | :arrow_down: |
| [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `90.90% <0.00%> (-0.42%)` | :arrow_down: |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.84% <0.00%> (-0.09%)` | :arrow_down: |
| [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <0.00%> (ø)` | |
| [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <0.00%> (ø)` | |
| [datasette/default\_permissions.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `100.00% <0.00%> (ø)` | |
| [datasette/default\_magic\_parameters.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWFnaWNfcGFyYW1ldGVycy5weQ==) | | |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.40% <0.00%> (+<0.01%)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=footer). Last update [57879dc...0d100d1](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638270441,Reload support for config_dir mode.,
https://github.com/simonw/datasette/issues/889#issuecomment-653002499,https://api.github.com/repos/simonw/datasette/issues/889,653002499,MDEyOklzc3VlQ29tbWVudDY1MzAwMjQ5OQ==,49260,amjith,2020-07-02T13:22:13Z,2020-07-02T13:22:13Z,CONTRIBUTOR,"I was able to narrow this down to the fact that lifespan protocol is turned on.
I see the workaround you've used here: https://github.com/simonw/datasette-debug-asgi/commit/72d568d32a3159c763ce908c0b269736935c6987
If so, maybe it's time to update some of the asg_wrapper [plugins](https://datasette.readthedocs.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649907676,asgi_wrapper plugin hook is crashing at startup,
https://github.com/simonw/datasette/issues/889#issuecomment-652990131,https://api.github.com/repos/simonw/datasette/issues/889,652990131,MDEyOklzc3VlQ29tbWVudDY1Mjk5MDEzMQ==,49260,amjith,2020-07-02T12:58:11Z,2020-07-02T13:00:18Z,CONTRIBUTOR,"FWIW, this error does NOT happen in datasette 0.45a4.
It only started on 0.45a5","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649907676,asgi_wrapper plugin hook is crashing at startup,
https://github.com/simonw/datasette/issues/886#issuecomment-652732460,https://api.github.com/repos/simonw/datasette/issues/886,652732460,MDEyOklzc3VlQ29tbWVudDY1MjczMjQ2MA==,9599,simonw,2020-07-02T01:52:02Z,2020-07-02T01:52:02Z,OWNER,In investigating this I'm not convinced 500 errors are being correctly raised by errors in canned writable queries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649429772,Reconsider how _actor_X magic parameter deals with missing values,
https://github.com/simonw/datasette/issues/886#issuecomment-652731459,https://api.github.com/repos/simonw/datasette/issues/886,652731459,MDEyOklzc3VlQ29tbWVudDY1MjczMTQ1OQ==,9599,simonw,2020-07-02T01:48:08Z,2020-07-02T01:48:08Z,OWNER,"A common error with this (and other) magic parameters is for the database query to result in the following:
You did not supply a value for binding 3.
This is a pretty crufty error. I'm inclined to say that ANY missing or invalid magic parameter should be treated as a `None` value instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649429772,Reconsider how _actor_X magic parameter deals with missing values,
https://github.com/simonw/datasette/issues/887#issuecomment-652711822,https://api.github.com/repos/simonw/datasette/issues/887,652711822,MDEyOklzc3VlQ29tbWVudDY1MjcxMTgyMg==,9599,simonw,2020-07-02T00:31:33Z,2020-07-02T00:31:33Z,OWNER,"If a canned query has a title defined that will be used instead: https://latest.datasette.io/fixtures/neighborhood_search
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649437530,Canned query page should show the name of the canned query,
https://github.com/simonw/datasette/issues/887#issuecomment-652711562,https://api.github.com/repos/simonw/datasette/issues/887,652711562,MDEyOklzc3VlQ29tbWVudDY1MjcxMTU2Mg==,9599,simonw,2020-07-02T00:30:43Z,2020-07-02T00:30:43Z,OWNER,"Demo has updated: https://latest.datasette.io/fixtures/magic_parameters
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649437530,Canned query page should show the name of the canned query,
https://github.com/simonw/datasette/pull/883#issuecomment-652710178,https://api.github.com/repos/simonw/datasette/issues/883,652710178,MDEyOklzc3VlQ29tbWVudDY1MjcxMDE3OA==,9599,simonw,2020-07-02T00:25:44Z,2020-07-02T00:25:44Z,OWNER,This is a great idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648749062,Skip counting hidden tables,
https://github.com/simonw/datasette/issues/887#issuecomment-652709199,https://api.github.com/repos/simonw/datasette/issues/887,652709199,MDEyOklzc3VlQ29tbWVudDY1MjcwOTE5OQ==,9599,simonw,2020-07-02T00:21:54Z,2020-07-02T00:21:54Z,OWNER,"Example in the live demo: https://latest.datasette.io/fixtures/magic_parameters
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649437530,Canned query page should show the name of the canned query,
https://github.com/simonw/datasette/issues/885#issuecomment-652681996,https://api.github.com/repos/simonw/datasette/issues/885,652681996,MDEyOklzc3VlQ29tbWVudDY1MjY4MTk5Ng==,9599,simonw,2020-07-01T22:44:47Z,2020-07-01T22:44:47Z,OWNER,https://simonwillison.net/2020/Jul/1/datasette-045/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649373451,Blog entry about the release,
https://github.com/simonw/datasette/issues/882#issuecomment-652663177,https://api.github.com/repos/simonw/datasette/issues/882,652663177,MDEyOklzc3VlQ29tbWVudDY1MjY2MzE3Nw==,9599,simonw,2020-07-01T21:48:08Z,2020-07-01T21:48:08Z,OWNER,https://datasette.readthedocs.io/en/latest/changelog.html#v0-45,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648673556,Release notes for 0.45,
https://github.com/simonw/datasette/issues/880#issuecomment-652646487,https://api.github.com/repos/simonw/datasette/issues/880,652646487,MDEyOklzc3VlQ29tbWVudDY1MjY0NjQ4Nw==,9599,simonw,2020-07-01T21:05:48Z,2020-07-01T21:05:48Z,OWNER,"I've been testing the WIP using this in the console:
```javascript
fetch('/data/add_name.json', {
method: 'POST',
body: 'name=XXXfetch',
credentials: 'omit',
headers: {'Content-Type': 'application/x-www-form-urlencoded'}
})
.then(response => console.log(response))
```
Against a canned query configured like this:
```yaml
databases:
data:
queries:
add_name:
sql: insert into names (name) values (:name)
write: true
```
I haven't got it to work yet. Latest error is this one:
```
INFO: Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
Traceback (most recent call last):
File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 975, in route_path
await response.asgi_send(send)
AttributeError: 'tuple' object has no attribute 'asgi_send'
INFO: 127.0.0.1:49938 - ""POST /data/add_name.json HTTP/1.1"" 500 Internal Server Error
```
It looks like I'm going to have to rethink how the `BaseView` code around tables, formats and hashes is structured in order to fix this. That's a big refactoring! I'm moving this to a new milestone for Datasette 0.46.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/882#issuecomment-652604569,https://api.github.com/repos/simonw/datasette/issues/882,652604569,MDEyOklzc3VlQ29tbWVudDY1MjYwNDU2OQ==,9599,simonw,2020-07-01T19:27:17Z,2020-07-01T19:27:17Z,OWNER,Don't forget to update the news in the README.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648673556,Release notes for 0.45,
https://github.com/simonw/datasette/issues/877#issuecomment-652597975,https://api.github.com/repos/simonw/datasette/issues/877,652597975,MDEyOklzc3VlQ29tbWVudDY1MjU5Nzk3NQ==,9599,simonw,2020-07-01T19:12:15Z,2020-07-01T19:12:15Z,OWNER,The latest release of https://github.com/simonw/datasette-auth-tokens (0.2) now supports SQL configuration of tokens.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-652520496,https://api.github.com/repos/simonw/datasette/issues/877,652520496,MDEyOklzc3VlQ29tbWVudDY1MjUyMDQ5Ng==,9599,simonw,2020-07-01T16:26:52Z,2020-07-01T16:26:52Z,OWNER,Tokens get verified by plugins. So far there's only one: https://github.com/simonw/datasette-auth-tokens - which has you hard-coding plugins in a configuration file. I have a issue there to add support for database-backed tokens too: https://github.com/simonw/datasette-auth-tokens/issues/1,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/pull/883#issuecomment-652394742,https://api.github.com/repos/simonw/datasette/issues/883,652394742,MDEyOklzc3VlQ29tbWVudDY1MjM5NDc0Mg==,3243482,abdusco,2020-07-01T12:41:13Z,2020-07-01T12:41:13Z,CONTRIBUTOR,"Well tests need to be updated.
I need to get tests working on Windows.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648749062,Skip counting hidden tables,
https://github.com/simonw/datasette/pull/883#issuecomment-652311990,https://api.github.com/repos/simonw/datasette/issues/883,652311990,MDEyOklzc3VlQ29tbWVudDY1MjMxMTk5MA==,22429695,codecov[bot],2020-07-01T09:40:40Z,2020-07-01T09:40:40Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=h1) Report
> Merging [#883](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/676bb64c877d73f8ff496cef4632f5a8a5a9283c&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #883 +/- ##
=======================================
Coverage 83.42% 83.42%
=======================================
Files 27 27
Lines 3632 3632
=======================================
Hits 3030 3030
Misses 602 602
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=footer). Last update [676bb64...251884f](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648749062,Skip counting hidden tables,
https://github.com/simonw/datasette/pull/883#issuecomment-652297139,https://api.github.com/repos/simonw/datasette/issues/883,652297139,MDEyOklzc3VlQ29tbWVudDY1MjI5NzEzOQ==,3243482,abdusco,2020-07-01T09:11:29Z,2020-07-01T09:11:29Z,CONTRIBUTOR,"Turns out we should include hidden tables in the result dict, or we're breaking tests. I've committed a refactor https://github.com/simonw/datasette/pull/883/commits/4f06e1bf6fbe4b73be770b87f610bf7c0e6e3ea7","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648749062,Skip counting hidden tables,
https://github.com/simonw/datasette/issues/877#issuecomment-652255960,https://api.github.com/repos/simonw/datasette/issues/877,652255960,MDEyOklzc3VlQ29tbWVudDY1MjI1NTk2MA==,3243482,abdusco,2020-07-01T07:52:25Z,2020-07-01T08:10:00Z,CONTRIBUTOR,"I am calling the API from another origin, so injecting CSRF token into templates wouldn't work.
EDIT:
I'll try the new version, it sounds promising","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-652261382,https://api.github.com/repos/simonw/datasette/issues/877,652261382,MDEyOklzc3VlQ29tbWVudDY1MjI2MTM4Mg==,3243482,abdusco,2020-07-01T08:03:17Z,2020-07-01T08:03:23Z,CONTRIBUTOR,Bearer tokens sound interesting. Where do tokens come from? An auth provider of my choosing? How do they get verified?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-652182990,https://api.github.com/repos/simonw/datasette/issues/877,652182990,MDEyOklzc3VlQ29tbWVudDY1MjE4Mjk5MA==,9599,simonw,2020-07-01T04:29:38Z,2020-07-01T04:42:59Z,OWNER,"Have you tried the method described here? https://datasette.readthedocs.io/en/latest/internals.html#csrf-protection - I'm happy to bulk out that section of the documentation if that doesn't help solve your problem.
I just closed #835 which should make CSRF protection easier to work with - it won't interfere with requests without cookies or requests with `Authentication: Bearer token` tokens. See also https://github.com/simonw/asgi-csrf/issues/11
You can try out `pip install datasette==0.45a5` to get those features. Hopefully releasing a full 0.45 tomorrow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-652166115,https://api.github.com/repos/simonw/datasette/issues/877,652166115,MDEyOklzc3VlQ29tbWVudDY1MjE2NjExNQ==,3243482,abdusco,2020-07-01T03:28:07Z,2020-07-01T03:28:07Z,CONTRIBUTOR,"Does this mean custom routes get to expose endpoints accepting POST requests? I've tried earlier to add some POST endpoints, but requests were being rejected by Datasette due to CSRF","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/812#issuecomment-652165709,https://api.github.com/repos/simonw/datasette/issues/812,652165709,MDEyOklzc3VlQ29tbWVudDY1MjE2NTcwOQ==,9599,simonw,2020-07-01T03:26:35Z,2020-07-01T03:26:35Z,OWNER,"This case may not be covered without extra work:
https://github.com/simonw/datasette/blob/3ec5b1abf6afa2d22a3378092809a1a8c0249d26/datasette/views/database.py#L122-L123","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634112607,Ability to customize what happens when a view permission fails,
https://github.com/simonw/datasette/issues/812#issuecomment-652163450,https://api.github.com/repos/simonw/datasette/issues/812,652163450,MDEyOklzc3VlQ29tbWVudDY1MjE2MzQ1MA==,9599,simonw,2020-07-01T03:18:51Z,2020-07-01T03:20:28Z,OWNER,"This can be a plugin hook:
```python
@hookspec
def forbidden(datasette, request, message, send):
""Custom response for a 403 forbidden error""
```
If the hook returns a `Response` object, it will be returned to the user. Plugins are likely to want to return a redirect response.
Maybe the hook can instead use the `send` argument to respond to the request and return `True` which means ""I've responded to this""?
I'm going to leave `send` off for the moment - I can add that in the future if it turns out it would have been a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634112607,Ability to customize what happens when a view permission fails,
https://github.com/simonw/datasette/issues/880#issuecomment-652162722,https://api.github.com/repos/simonw/datasette/issues/880,652162722,MDEyOklzc3VlQ29tbWVudDY1MjE2MjcyMg==,9599,simonw,2020-07-01T03:16:07Z,2020-07-01T03:16:07Z,OWNER,The response from this will never be a 302 - it will always be a 200 if the response worked or a 400 for bad parameters or a 500 for errors. The body returned will always be in JSON format.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/859#issuecomment-652160909,https://api.github.com/repos/simonw/datasette/issues/859,652160909,MDEyOklzc3VlQ29tbWVudDY1MjE2MDkwOQ==,3243482,abdusco,2020-07-01T03:09:32Z,2020-07-01T03:10:21Z,CONTRIBUTOR,"I've just realized Datasette tries to count hidden tables too. There are 5 visible tables, 25 hidden tables, which I haven't realize earlier to consider their effect. I've turned off counting for hidden tables to see if it has any effect.
What's the point of counting FTS tables?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/835#issuecomment-652159398,https://api.github.com/repos/simonw/datasette/issues/835,652159398,MDEyOklzc3VlQ29tbWVudDY1MjE1OTM5OA==,9599,simonw,2020-07-01T03:03:51Z,2020-07-01T03:03:51Z,OWNER,I'm going to add some tests for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/876#issuecomment-652106227,https://api.github.com/repos/simonw/datasette/issues/876,652106227,MDEyOklzc3VlQ29tbWVudDY1MjEwNjIyNw==,9599,simonw,2020-06-30T23:49:55Z,2020-06-30T23:50:04Z,OWNER,"Done: https://latest.datasette.io/-/patterns
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647879783,Add log out link to the pattern portfolio,
https://github.com/simonw/datasette/issues/879#issuecomment-652105722,https://api.github.com/repos/simonw/datasette/issues/879,652105722,MDEyOklzc3VlQ29tbWVudDY1MjEwNTcyMg==,9599,simonw,2020-06-30T23:48:06Z,2020-06-30T23:48:06Z,OWNER,Updated documentation: https://datasette.readthedocs.io/en/latest/pages.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648569227,Database page documentation still talks about hashes in URLs,
https://github.com/simonw/datasette/issues/832#issuecomment-652103895,https://api.github.com/repos/simonw/datasette/issues/832,652103895,MDEyOklzc3VlQ29tbWVudDY1MjEwMzg5NQ==,9599,simonw,2020-06-30T23:41:22Z,2020-06-30T23:41:22Z,OWNER,I don't think this needs any additional documentation - the new behaviour matches how the permissions are documented here: https://datasette.readthedocs.io/en/0.44/authentication.html#built-in-permissions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651999516,https://api.github.com/repos/simonw/datasette/issues/832,651999516,MDEyOklzc3VlQ29tbWVudDY1MTk5OTUxNg==,9599,simonw,2020-06-30T19:33:49Z,2020-06-30T21:34:59Z,OWNER,"Tests needed for this:
- If a user has view table but NOT view database / view instance, can they view the table page?
- If a user has view canned query but NOT view database / view instance, can they view the canned query page?
- If a user has view database but NOT view instance, can they view the database page?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651995453,https://api.github.com/repos/simonw/datasette/issues/832,651995453,MDEyOklzc3VlQ29tbWVudDY1MTk5NTQ1Mw==,9599,simonw,2020-06-30T19:25:13Z,2020-06-30T19:25:26Z,OWNER,I'm going to put the new `check_permissions()` method on `BaseView` as well. If I want that method to be available to plugins I can do so by turning that `BaseView` class into a documented API that plugins are encouraged to use themselves.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651994978,https://api.github.com/repos/simonw/datasette/issues/832,651994978,MDEyOklzc3VlQ29tbWVudDY1MTk5NDk3OA==,9599,simonw,2020-06-30T19:24:12Z,2020-06-30T19:24:12Z,OWNER,"Hah... but check_permission` is a method on `BaseView`. Here are the various permission methods at the moment:
https://github.com/simonw/datasette/blob/6c2634583627bfab750c115cb13850252821d637/datasette/default_permissions.py#L5-L14
And on BaseView:
https://github.com/simonw/datasette/blob/a8a5f813722f72703a7aae41135ccc40635cc02f/datasette/views/base.py#L65-L70","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651993977,https://api.github.com/repos/simonw/datasette/issues/832,651993977,MDEyOklzc3VlQ29tbWVudDY1MTk5Mzk3Nw==,9599,simonw,2020-06-30T19:22:06Z,2020-06-30T19:22:06Z,OWNER,`permission_allowed` is already the name of the pugin hook. It's actually a bit confusing that it's also the name of a method on `datasette.`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651993537,https://api.github.com/repos/simonw/datasette/issues/832,651993537,MDEyOklzc3VlQ29tbWVudDY1MTk5MzUzNw==,9599,simonw,2020-06-30T19:21:15Z,2020-06-30T19:21:15Z,OWNER,"I could rename `permission_allowed()` to `check_permission()` and have a complementary `check_permissions()` method.
This is a breaking change but we're pre-1.0 so I think that's OK. I could even set up a temporary `permission_allowed()` alias which prints a deprecation warning to the console, then remove that at 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651992737,https://api.github.com/repos/simonw/datasette/issues/832,651992737,MDEyOklzc3VlQ29tbWVudDY1MTk5MjczNw==,9599,simonw,2020-06-30T19:19:33Z,2020-06-30T19:20:02Z,OWNER,"I already have this method on Datasette:
```python
async def permission_allowed(self, actor, action, resource=None, default=False):
```
What would be a good method name that complements that and indicates ""check a list of permissions in order""? Should it even run against the request or should you have to hand it `request.actor`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/877#issuecomment-651984989,https://api.github.com/repos/simonw/datasette/issues/877,651984989,MDEyOklzc3VlQ29tbWVudDY1MTk4NDk4OQ==,9599,simonw,2020-06-30T19:03:25Z,2020-06-30T19:03:25Z,OWNER,Relevant: #835,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-651984355,https://api.github.com/repos/simonw/datasette/issues/877,651984355,MDEyOklzc3VlQ29tbWVudDY1MTk4NDM1NQ==,9599,simonw,2020-06-30T19:02:15Z,2020-06-30T19:02:15Z,OWNER,"https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#login-csrf
> Login CSRF can be mitigated by creating pre-sessions (sessions before a user is authenticated) and including tokens in login form.
Sounds like regular CSRF protection to me.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/805#issuecomment-651302221,https://api.github.com/repos/simonw/datasette/issues/805,651302221,MDEyOklzc3VlQ29tbWVudDY1MTMwMjIyMQ==,9599,simonw,2020-06-29T19:02:45Z,2020-06-29T19:05:26Z,OWNER,"No I prefer the idea that logged out users can still perform some writes, in a not-likely-to-attract-abuse way.
So a root-user-can-configure-polls, logged-out-users-can-vote-in-them demo would be good.
Or... crazy idea: a collaborative drawing program? A grid of cells of emoji, anyone can add an emoji to a cell. Would involve a bit of JavaScript. I could use https://github.com/joeattardi/emoji-button for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/805#issuecomment-651301202,https://api.github.com/repos/simonw/datasette/issues/805,651301202,MDEyOklzc3VlQ29tbWVudDY1MTMwMTIwMg==,9599,simonw,2020-06-29T19:00:37Z,2020-06-29T19:00:37Z,OWNER,"How about a blog? Pre-configured canned queries that are only available to `""root""`, plus datasette-template-sql and default templates for the index page and blog entry pages.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/875#issuecomment-651293559,https://api.github.com/repos/simonw/datasette/issues/875,651293559,MDEyOklzc3VlQ29tbWVudDY1MTI5MzU1OQ==,9599,simonw,2020-06-29T18:43:50Z,2020-06-29T18:43:50Z,OWNER,"![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647103735,"""Logged in as: XXX - logout"" navigation item",
https://github.com/simonw/datasette/issues/873#issuecomment-651203178,https://api.github.com/repos/simonw/datasette/issues/873,651203178,MDEyOklzc3VlQ29tbWVudDY1MTIwMzE3OA==,9599,simonw,2020-06-29T15:44:38Z,2020-06-29T15:44:54Z,OWNER,I'm having real trouble figuring out how to gain access to the port that was used to start the server. I'm treating this as a very low priority - it only affects the exact `-p 0 --root` combination which isn't going to affect many people at all.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-651193594,https://api.github.com/repos/simonw/datasette/issues/873,651193594,MDEyOklzc3VlQ29tbWVudDY1MTE5MzU5NA==,9599,simonw,2020-06-29T15:27:46Z,2020-06-29T15:27:46Z,OWNER,Uninstalling `datasette-debug-asgi` caused the server to startup correctly again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-651193131,https://api.github.com/repos/simonw/datasette/issues/873,651193131,MDEyOklzc3VlQ29tbWVudDY1MTE5MzEzMQ==,9599,simonw,2020-06-29T15:27:00Z,2020-06-29T15:27:00Z,OWNER,"Aha! Yes it's not being called, and the reason is this: https://github.com/encode/starlette/issues/486
Short version: by default an exception raised during that phase is silently swallowed! You can avoid the swallowing by adding `lifespan=""on""` to the call to `uvicorn.run()`.
When I did that here:
`uvicorn.run(ds.app(), host=host, port=port, log_level=""info"", lifespan=""on"")`
The server failed to start with this error:
```
INFO: Started server process [68849]
INFO: Waiting for application startup.
ERROR: Exception in 'lifespan' protocol
Traceback (most recent call last):
File "".../uvicorn/lifespan/on.py"", line 48, in main
await app(scope, self.receive, self.send)
File "".../uvicorn/middleware/proxy_headers.py"", line 45, in __call__
return await self.app(scope, receive, send)
File "".../datasette_debug_asgi.py"", line 9, in wrapped_app
if scope[""path""] == ""/-/asgi-scope"":
KeyError: 'path'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650910137,https://api.github.com/repos/simonw/datasette/issues/873,650910137,MDEyOklzc3VlQ29tbWVudDY1MDkxMDEzNw==,9599,simonw,2020-06-29T05:16:32Z,2020-06-29T05:16:32Z,OWNER,I'm not convinced that function is ever actually being called - I added a `print()` statement to it and it's not executing. I don't think the tests cover it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650909476,https://api.github.com/repos/simonw/datasette/issues/873,650909476,MDEyOklzc3VlQ29tbWVudDY1MDkwOTQ3Ng==,9599,simonw,2020-06-29T05:14:08Z,2020-06-29T05:14:08Z,OWNER,"I already have a `AsgiLifespan` class:
https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L896-L905
It runs this function: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L890-L894
Could that startup function also output the `--root` login URL, if needed?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650909136,https://api.github.com/repos/simonw/datasette/issues/873,650909136,MDEyOklzc3VlQ29tbWVudDY1MDkwOTEzNg==,9599,simonw,2020-06-29T05:12:58Z,2020-06-29T05:12:58Z,OWNER,"On startup Datasette currently outputs:
```
INFO: Waiting for application startup.
INFO: ASGI 'lifespan' protocol appears unsupported.
INFO: Application startup complete.
```
So the ASGI lifespan protocol is almost certainly the right way to solve this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650908854,https://api.github.com/repos/simonw/datasette/issues/873,650908854,MDEyOklzc3VlQ29tbWVudDY1MDkwODg1NA==,9599,simonw,2020-06-29T05:12:04Z,2020-06-29T05:12:04Z,OWNER,Can I detect the port the server is running on from within the regular Datasette ASGI code? If so I could use that ability and maybe output the magic `--root` link a second after the server starts up somehow.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650908534,https://api.github.com/repos/simonw/datasette/issues/873,650908534,MDEyOklzc3VlQ29tbWVudDY1MDkwODUzNA==,9599,simonw,2020-06-29T05:11:06Z,2020-06-29T05:11:06Z,OWNER,"Uvicorn's lifespan stuff isn't easy to figure out, but this test suite holds some clues: https://github.com/encode/uvicorn/blob/master/tests/test_lifespan.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650907323,https://api.github.com/repos/simonw/datasette/issues/873,650907323,MDEyOklzc3VlQ29tbWVudDY1MDkwNzMyMw==,9599,simonw,2020-06-29T05:07:16Z,2020-06-29T05:07:16Z,OWNER,"This line is interesting: is this a hook I can attach to somehow?
```python
await self.lifespan.startup()
```
From https://github.com/encode/uvicorn/blob/a75fe1381f6b1f78901691c71894f3cf487b5d30/uvicorn/main.py#L475","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650906533,https://api.github.com/repos/simonw/datasette/issues/873,650906533,MDEyOklzc3VlQ29tbWVudDY1MDkwNjUzMw==,9599,simonw,2020-06-29T05:04:44Z,2020-06-29T05:04:44Z,OWNER,The challenge is... can we run our own custom code after that line has executed that has access to `server` and can hence access `server.servers[0].sockets[0].getsockname()[1]` to find the port?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650906318,https://api.github.com/repos/simonw/datasette/issues/873,650906318,MDEyOklzc3VlQ29tbWVudDY1MDkwNjMxOA==,9599,simonw,2020-06-29T05:04:04Z,2020-06-29T05:04:12Z,OWNER,"Within uvicorn it does this:
```python
if port == 0:
port = server.sockets[0].getsockname()[1]
```
That `server` variable is later stashed here:
```
self.servers = [server]
```
Where `self` is the instance of `class Server` - which is the class that Uvicorn instantiates and calls `.run()` on when we do `uvicorn.run()` here: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/cli.py#L409","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650905399,https://api.github.com/repos/simonw/datasette/issues/873,650905399,MDEyOklzc3VlQ29tbWVudDY1MDkwNTM5OQ==,9599,simonw,2020-06-29T05:01:03Z,2020-06-29T05:01:03Z,OWNER,This is a bit tricky to fix. This change to uvicorn is relevant: https://github.com/encode/uvicorn/commit/a75fe1381f6b1f78901691c71894f3cf487b5d30,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/875#issuecomment-650899265,https://api.github.com/repos/simonw/datasette/issues/875,650899265,MDEyOklzc3VlQ29tbWVudDY1MDg5OTI2NQ==,9599,simonw,2020-06-29T04:34:32Z,2020-06-29T04:34:32Z,OWNER,"From https://github.com/simonw/datasette/issues/840#issuecomment-643454625
> Another problem: what to display in the ""you are logged in as"", since we don't dictate an actor design.
>
> I'm going to use a includes template for this that can easily be over-ridden by administrators or by plugins.
>
> The default will look for the first available of the following keys:
>
> * display
> * name
> * username
> * login
> * id","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647103735,"""Logged in as: XXX - logout"" navigation item",
https://github.com/simonw/datasette/issues/875#issuecomment-650898808,https://api.github.com/repos/simonw/datasette/issues/875,650898808,MDEyOklzc3VlQ29tbWVudDY1MDg5ODgwOA==,9599,simonw,2020-06-29T04:32:31Z,2020-06-29T04:33:30Z,OWNER,"I could borrow the implementation for this from `datasette-auth-github`
https://github.com/simonw/datasette-auth-github/blob/182298b034ecb647971b65057d1d3e7b7fbbb482/datasette_auth_github/templates/base.html
```html+jinja
{% extends ""default:base.html"" %}
{% block extra_head %}
{% endblock %}
{% block nav %}
{{ super() }}
{% if auth and auth.username %}
{{ auth.username }} · Log out
{% endif %}
{% endblock %}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647103735,"""Logged in as: XXX - logout"" navigation item",
https://github.com/simonw/datasette/issues/840#issuecomment-650895874,https://api.github.com/repos/simonw/datasette/issues/840,650895874,MDEyOklzc3VlQ29tbWVudDY1MDg5NTg3NA==,9599,simonw,2020-06-29T04:18:59Z,2020-06-29T04:19:11Z,OWNER,"Now just need the ""Logged in as: XXX <logout>"" navigation item.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637966833,Log out mechanism for clearing ds_actor cookie,
https://github.com/simonw/datasette/issues/840#issuecomment-650891502,https://api.github.com/repos/simonw/datasette/issues/840,650891502,MDEyOklzc3VlQ29tbWVudDY1MDg5MTUwMg==,9599,simonw,2020-06-29T03:58:08Z,2020-06-29T03:58:08Z,OWNER,"Step one: a ""logout"" page at `/-/logout` - which shows you a single CSRF-protected ""logout"" button if you do a GET against it and logs you out if you do a POST against it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637966833,Log out mechanism for clearing ds_actor cookie,
https://github.com/simonw/datasette/issues/805#issuecomment-650891257,https://api.github.com/repos/simonw/datasette/issues/805,650891257,MDEyOklzc3VlQ29tbWVudDY1MDg5MTI1Nw==,9599,simonw,2020-06-29T03:56:48Z,2020-06-29T03:56:48Z,OWNER,Using `datasette-glitch` and the new https://github.com/simonw/datasette-write - currently running on `datasette==0.45a4` - works on Glitch. The console shows a login link which gives you a cookie which allows you access to the `/-/write` interface.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/864#issuecomment-650847013,https://api.github.com/repos/simonw/datasette/issues/864,650847013,MDEyOklzc3VlQ29tbWVudDY1MDg0NzAxMw==,9599,simonw,2020-06-29T00:41:55Z,2020-06-29T00:41:55Z,OWNER,To test this I'll need a plugin test that renders a custom template. Here's an example I can imitate: https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/tests/test_plugins.py#L588-L596,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/864#issuecomment-650846625,https://api.github.com/repos/simonw/datasette/issues/864,650846625,MDEyOklzc3VlQ29tbWVudDY1MDg0NjYyNQ==,9599,simonw,2020-06-29T00:39:47Z,2020-06-29T00:39:47Z,OWNER,"I think the fix is to move the `""show_messages""` variable to here:
https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/app.py#L735-L748","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/864#issuecomment-650846473,https://api.github.com/repos/simonw/datasette/issues/864,650846473,MDEyOklzc3VlQ29tbWVudDY1MDg0NjQ3Mw==,9599,simonw,2020-06-29T00:39:04Z,2020-06-29T00:39:04Z,OWNER,"Re-opening: plugins may get to set messages but they don't display them, even if they render a template that extends `base.html`. For example, this code in a plugin:
```python
return Response.html(
await datasette.render_template(
""write.html"",
{""databases"": databases, ""sql"": request.args.get(""sql"") or """"},
request=request,
)
)
```
This won't display messages. The reason is that the messages are made available to the template context in the `BaseView.render()` method here:
https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/views/base.py#L87-L95","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/864#issuecomment-650842514,https://api.github.com/repos/simonw/datasette/issues/864,650842514,MDEyOklzc3VlQ29tbWVudDY1MDg0MjUxNA==,9599,simonw,2020-06-29T00:12:59Z,2020-06-29T00:12:59Z,OWNER,"> I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with `register_routes()`) as part of Datasette 0.45 but it would be OK if it slipped to a later release.
https://github.com/simonw/datasette/issues/870#issuecomment-650842381","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/870#issuecomment-650842381,https://api.github.com/repos/simonw/datasette/issues/870,650842381,MDEyOklzc3VlQ29tbWVudDY1MDg0MjM4MQ==,9599,simonw,2020-06-29T00:12:07Z,2020-06-29T00:12:07Z,OWNER,I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with `register_routes()`) as part of Datasette 0.45 but it would be OK if it slipped to a later release.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650838972,https://api.github.com/repos/simonw/datasette/issues/870,650838972,MDEyOklzc3VlQ29tbWVudDY1MDgzODk3Mg==,9599,simonw,2020-06-28T23:46:40Z,2020-06-28T23:46:40Z,OWNER,I'm going to create the single `Request()` instance in the `DatasetteRouter` class - at the beginning of the `route_path` method: https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L905-L925,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650838691,https://api.github.com/repos/simonw/datasette/issues/870,650838691,MDEyOklzc3VlQ29tbWVudDY1MDgzODY5MQ==,9599,simonw,2020-06-28T23:44:12Z,2020-06-28T23:44:25Z,OWNER,"This code is interesting:
https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L948-L955
I want to change the signature of that `return await view(new_scope, receive, send)` method to instead take `(request, send)` - so I can have a single shared request object that's created just once per HTTP request.
The problem is the scope modification: I have code that modifies the scope, but how should that impact a shared `Request` instance? Should its `.scope` be replaced with alternative scopes as it travels through the codebase?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650834666,https://api.github.com/repos/simonw/datasette/issues/870,650834666,MDEyOklzc3VlQ29tbWVudDY1MDgzNDY2Ng==,9599,simonw,2020-06-28T23:07:19Z,2020-06-28T23:07:19Z,OWNER,So now the problem is simpler: I need to get `BaseView` to a state where it can accept a shared `request` object and it can be used in conjunction with `register_routes()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650834251,https://api.github.com/repos/simonw/datasette/issues/870,650834251,MDEyOklzc3VlQ29tbWVudDY1MDgzNDI1MQ==,9599,simonw,2020-06-28T23:03:28Z,2020-06-28T23:03:28Z,OWNER,"I'm going to ditch that `AsgiView` class too, by combining it into `BaseView`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650820068,https://api.github.com/repos/simonw/datasette/issues/870,650820068,MDEyOklzc3VlQ29tbWVudDY1MDgyMDA2OA==,9599,simonw,2020-06-28T20:52:09Z,2020-06-28T20:53:00Z,OWNER,"Maybe I could add a `as_request_view` method as an alternative to `as_asgi`:
https://github.com/simonw/datasette/blob/a8bcafc1775c8a8655b365ae22a3d64f6361c74a/datasette/utils/asgi.py#L150-L174
Or I could teach the `Router` to spot the `dispatch_request` method and call it directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/847#issuecomment-650819895,https://api.github.com/repos/simonw/datasette/issues/847,650819895,MDEyOklzc3VlQ29tbWVudDY1MDgxOTg5NQ==,9599,simonw,2020-06-28T20:50:21Z,2020-06-28T20:50:21Z,OWNER,I'm happy enough with https://codecov.io/gh/simonw/datasette that I'm not going to spend any more time on this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638259643,Take advantage of .coverage being a SQLite database,
https://github.com/simonw/datasette/issues/870#issuecomment-650818309,https://api.github.com/repos/simonw/datasette/issues/870,650818309,MDEyOklzc3VlQ29tbWVudDY1MDgxODMwOQ==,9599,simonw,2020-06-28T20:36:28Z,2020-06-28T20:36:52Z,OWNER,"Since `AsgiRouter` is only used as the super-class of the `DatasetteRouter` class maybe I should get rid of `AsgiRouter` entirely - no point in having a Datasette-specific subclass of it if the parent class isn't ever used by anything else.
I could also rename it to just `Router` which is a nicer name than `DatasetteRouter`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650818086,https://api.github.com/repos/simonw/datasette/issues/870,650818086,MDEyOklzc3VlQ29tbWVudDY1MDgxODA4Ng==,9599,simonw,2020-06-28T20:34:33Z,2020-06-28T20:34:33Z,OWNER,"The key to all of this may be the `DatasetteRouter` class. It deals with `scope` right now but if it internally dealt with `request` that could be enough to fix #864 by adding logic needed by the `.add_message()` mechanism.
https://github.com/simonw/datasette/blob/0991ea75cc7b265389aa8362414a305ba532d31a/datasette/app.py#L904-L938","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650815278,https://api.github.com/repos/simonw/datasette/issues/870,650815278,MDEyOklzc3VlQ29tbWVudDY1MDgxNTI3OA==,9599,simonw,2020-06-28T20:09:07Z,2020-06-28T20:11:21Z,OWNER,"There's a lot of complex logic in the `DataView` class, which handles conditionally returning content as `.json` or as HTML or as `.csv`.
That view subclasses `AsgiView` which is itself request-aware, so maybe I don't need to reconsider how those classes work - just figure out how to hook them up with `register_routes`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/871#issuecomment-650812444,https://api.github.com/repos/simonw/datasette/issues/871,650812444,MDEyOklzc3VlQ29tbWVudDY1MDgxMjQ0NA==,9599,simonw,2020-06-28T19:43:27Z,2020-06-28T19:43:27Z,OWNER,"Currently:
> `_timestamp_epoch`
>
> The number of seconds since the Unix epoch.
>
> `_timestamp_date_utc`
>
> The date in UTC, e.g. `2020-06-01`
>
> `_timestamp_datetime_utc`
>
> The ISO 8601 datetime in UTC, e.g. `2020-06-24T18:01:07Z`
I'm going to rename them to:
- `_now_epoch`
- `_now_date_utc`
- `_now_datetime_utc`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646840273,Rename the _timestamp magic parameters to _now,
https://github.com/simonw/datasette/issues/834#issuecomment-650811919,https://api.github.com/repos/simonw/datasette/issues/834,650811919,MDEyOklzc3VlQ29tbWVudDY1MDgxMTkxOQ==,9599,simonw,2020-06-28T19:38:50Z,2020-06-28T19:38:50Z,OWNER,"I have two plugins in progress that use this hook now:
- https://github.com/simonw/datasette-init creates tables and views on startup
- https://github.com/simonw/datasette-glitch outputs the login-as-root secret link on Glitch","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/805#issuecomment-650784162,https://api.github.com/repos/simonw/datasette/issues/805,650784162,MDEyOklzc3VlQ29tbWVudDY1MDc4NDE2Mg==,9599,simonw,2020-06-28T15:48:32Z,2020-06-28T15:48:32Z,OWNER,https://github.com/simonw/datasette-glitch is my new plugin that outputs the root login link on Glitch when the server starts.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/834#issuecomment-643657067,https://api.github.com/repos/simonw/datasette/issues/834,643657067,MDEyOklzc3VlQ29tbWVudDY0MzY1NzA2Nw==,9599,simonw,2020-06-13T17:59:42Z,2020-06-28T04:01:52Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#startup-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/842#issuecomment-650684635,https://api.github.com/repos/simonw/datasette/issues/842,650684635,MDEyOklzc3VlQ29tbWVudDY1MDY4NDYzNQ==,9599,simonw,2020-06-28T03:30:31Z,2020-06-28T03:30:31Z,OWNER,Live demo: https://latest.datasette.io/fixtures/magic_parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/805#issuecomment-650681496,https://api.github.com/repos/simonw/datasette/issues/805,650681496,MDEyOklzc3VlQ29tbWVudDY1MDY4MTQ5Ng==,9599,simonw,2020-06-28T03:11:51Z,2020-06-28T03:11:51Z,OWNER,I can use magic parameters from #842 in this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/842#issuecomment-650679100,https://api.github.com/repos/simonw/datasette/issues/842,650679100,MDEyOklzc3VlQ29tbWVudDY1MDY3OTEwMA==,9599,simonw,2020-06-28T03:00:44Z,2020-06-28T03:00:44Z,OWNER,I'm going to add some canned queries to the `metadata.json` used by the live demo that illustrate this feature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650678951,https://api.github.com/repos/simonw/datasette/issues/842,650678951,MDEyOklzc3VlQ29tbWVudDY1MDY3ODk1MQ==,9599,simonw,2020-06-28T02:59:52Z,2020-06-28T02:59:52Z,OWNER,"Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#magic-parameters
Plugin hook documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#plugin-hook-register-magic-parameters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/pull/869#issuecomment-650600176,https://api.github.com/repos/simonw/datasette/issues/869,650600176,MDEyOklzc3VlQ29tbWVudDY1MDYwMDE3Ng==,22429695,codecov[bot],2020-06-27T18:41:31Z,2020-06-28T02:54:21Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=h1) Report
> Merging [#869](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c&el=desc) will **increase** coverage by `0.23%`.
> The diff coverage is `90.62%`.
[](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #869 +/- ##
==========================================
+ Coverage 82.99% 83.23% +0.23%
==========================================
Files 26 27 +1
Lines 3547 3609 +62
==========================================
+ Hits 2944 3004 +60
- Misses 603 605 +2
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | |
| [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.45% <86.36%> (-1.88%)` | :arrow_down: |
| [datasette/default\_magic\_parameters.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWFnaWNfcGFyYW1ldGVycy5weQ==) | `91.17% <91.17%> (ø)` | |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.07% <100.00%> (+0.81%)` | :arrow_up: |
| [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.87% <100.00%> (+0.02%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=footer). Last update [1bb33da...9e693a7](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646734280,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650648434,https://api.github.com/repos/simonw/datasette/issues/842,650648434,MDEyOklzc3VlQ29tbWVudDY1MDY0ODQzNA==,9599,simonw,2020-06-27T23:27:35Z,2020-06-27T23:37:38Z,OWNER,I'm going to rename `_request_X` to `_header_X` as that better reflects what it now does.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/pull/868#issuecomment-650600606,https://api.github.com/repos/simonw/datasette/issues/868,650600606,MDEyOklzc3VlQ29tbWVudDY1MDYwMDYwNg==,9599,simonw,2020-06-27T18:44:28Z,2020-06-27T18:44:28Z,OWNER,"This is really exciting! Thanks so much for looking into this.
I'm interested in moving CI for this repo over to GitHub Actions, so I'd be fine with you getting this to work as an Action rather than through Travis. If you can get it working in Travis though I'll happily land that and figure out how to convert that to GitHub Actions later on.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646448486,initial windows ci setup,
https://github.com/simonw/datasette/issues/835#issuecomment-650598710,https://api.github.com/repos/simonw/datasette/issues/835,650598710,MDEyOklzc3VlQ29tbWVudDY1MDU5ODcxMA==,9599,simonw,2020-06-27T18:32:22Z,2020-06-27T18:32:22Z,OWNER,"Skipping CSRF on `Authorization: Bearer xxx` headers also makes sense for JWT applications, which tend to send JWTs using that form of header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/842#issuecomment-650593122,https://api.github.com/repos/simonw/datasette/issues/842,650593122,MDEyOklzc3VlQ29tbWVudDY1MDU5MzEyMg==,9599,simonw,2020-06-27T18:03:02Z,2020-06-27T18:03:10Z,OWNER,"> Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter `_cookie_ds_actor` to be resolved and returned as JSON data that the attacker can see.
This is an open security hole in https://github.com/simonw/datasette/commit/94c1315f0030fd58ce46a9294052c5c9d9d181c7 - it's useful for testing, but I need to remove it before I land that branch.
https://github.com/simonw/datasette/blob/94c1315f0030fd58ce46a9294052c5c9d9d181c7/datasette/views/database.py#L231-L237
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650458857,https://api.github.com/repos/simonw/datasette/issues/842,650458857,MDEyOklzc3VlQ29tbWVudDY1MDQ1ODg1Nw==,9599,simonw,2020-06-27T00:11:04Z,2020-06-27T00:11:04Z,OWNER,Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter `_cookie_ds_actor` to be resolved and returned as JSON data that the attacker can see.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650455793,https://api.github.com/repos/simonw/datasette/issues/842,650455793,MDEyOklzc3VlQ29tbWVudDY1MDQ1NTc5Mw==,9599,simonw,2020-06-26T23:57:30Z,2020-06-27T00:00:16Z,OWNER,"Maybe I should ship a default `_scope_headers_...` parameter instead, which reads from a dictionary of `scope[""headers""]` - https://asgi-scope.now.sh/ shows what those look like.
```
{'client': ('148.64.98.14', 0),
'headers': [[b'host', b'asgi-scope.now.sh'],
[b'x-forwarded-for', b'148.64.98.14'],
[b'x-vercel-id', b'sw72x-1593215573008-024e4e603806'],
[b'x-forwarded-host', b'asgi-scope.now.sh'],
[b'accept',
b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
b'webp,*/*;q=0.8'],
[b'x-real-ip', b'148.64.98.14'],
[b'x-vercel-deployment-url', b'asgi-scope-9eyeojbek.now.sh'],
[b'upgrade-insecure-requests', b'1'],
[b'x-vercel-trace', b'sfo1'],
[b'x-forwarded-proto', b'https'],
[b'accept-language', b'en-US,en;q=0.5'],
[b'user-agent',
b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko'
b'/20100101 Firefox/77.0'],
[b'x-vercel-forwarded-for', b'148.64.98.14'],
[b'accept-encoding', b'gzip, deflate, br'],
[b'dnt', b'1'],
[b'te', b'trailers']],
'http_version': '1.1',
'method': 'GET',
'path': '/',
'query_string': b'',
'raw_path': b'/',
'root_path': '',
'scheme': 'https',
'server': ('asgi-scope.now.sh', 80),
'type': 'http'}
```
I'm going to have `_request_X` actually mean ""find the first value for X in `scope[""headers""`]"" - with underscores converted to hyphens.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650455353,https://api.github.com/repos/simonw/datasette/issues/842,650455353,MDEyOklzc3VlQ29tbWVudDY1MDQ1NTM1Mw==,9599,simonw,2020-06-26T23:55:40Z,2020-06-26T23:55:40Z,OWNER,"`_request_ip` is actually quite hard to implement - should it take into account things like the `x-forwarded-for` header?
It probably should - but that means it now needs a bunch of extra configuration to tell it which of those headers can be trusted in the current environment.
As such I think I'll leave that for a plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/867#issuecomment-649931714,https://api.github.com/repos/simonw/datasette/issues/867,649931714,MDEyOklzc3VlQ29tbWVudDY0OTkzMTcxNA==,9599,simonw,2020-06-26T03:12:51Z,2020-06-26T03:12:51Z,OWNER,"Here's the relevant code:
https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/datasette/app.py#L1057-L1070
And the relevant test code:
https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/test_plugins.py#L567-L573
https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/plugins/my_plugin.py#L162-L196","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",645975649,register_routes() should support non-async view functions too,
https://github.com/simonw/datasette/issues/842#issuecomment-649014757,https://api.github.com/repos/simonw/datasette/issues/842,649014757,MDEyOklzc3VlQ29tbWVudDY0OTAxNDc1Nw==,9599,simonw,2020-06-24T19:15:46Z,2020-06-24T19:31:52Z,OWNER,I'm building this documentation-first - here's the documentation so far: https://github.com/simonw/datasette/blob/6fc8bd9c473f4a25e0a076f24c7e5a9b2f353bb8/docs/sql_queries.rst#magic-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-646271834,https://api.github.com/repos/simonw/datasette/issues/842,646271834,MDEyOklzc3VlQ29tbWVudDY0NjI3MTgzNA==,9599,simonw,2020-06-18T19:49:41Z,2020-06-24T18:49:22Z,OWNER,"But then what kind of magic parameters might plugins want to add?
Here's a crazy idea: `_scrapedcontent_url` - it would look for the `url` column on the data being inserted, scrape the content from it and insert that. This does suggest that the magic resolving function `scrapedcontent()` would need to optionally be sent the full row dictionary being inserted too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-646270702,https://api.github.com/repos/simonw/datasette/issues/842,646270702,MDEyOklzc3VlQ29tbWVudDY0NjI3MDcwMg==,9599,simonw,2020-06-18T19:47:19Z,2020-06-24T18:48:48Z,OWNER,"Brainstorming more potential magic parameters:
* `_actor_id`
* `_actor_name`
* `_request_ip`
* `_request_user_agent`
* `_cookie_cookiename`
* `_signedcookie_cookiename` - reading signed cookies would be cool, not sure how to specify namespace though, maybe always use the same one? Or have the namespace come last, `_signedcookie_cookiename_mynamespace`. Might not need special signed cookie support since `actor` is already usually from a signed cookie.
* `_timestamp_unix` (not happy with these names yet)
* `_timestamp_localtime`
* `_timestamp_datetime`
* `_timestamp_utc`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-649000075,https://api.github.com/repos/simonw/datasette/issues/842,649000075,MDEyOklzc3VlQ29tbWVudDY0OTAwMDA3NQ==,9599,simonw,2020-06-24T18:46:36Z,2020-06-24T18:47:37Z,OWNER,"Another magic parameter that would be useful would be `_random`. Consider https://github.com/simonw/datasette-auth-tokens/issues/1 for example - I'd like to be able to provide a writable canned query which can create new authentication tokens in the database, but ideally it would automatically populate a secure random secret for each one.
Maybe `_random_chars_128` to create a 128 character long random string (using `os.urandom(64).hex()`).
This would be the first example of a magic parameter where part of the parameter name is used to configure the resulting value. Maybe neater to separate that with a different character? Unfortunately `_random_chars:128` wouldn't work because these parameters are used in a SQLite query where `:` has special meaning: `insert into blah (secret) values (:_random_chars:128)` wouldn't make sense.
Actually this is already supported by the proposed design - `_random_chars_128` would become `random(""chars_128"")` so the `random()` function could split off the 128 itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/865#issuecomment-648998264,https://api.github.com/repos/simonw/datasette/issues/865,648998264,MDEyOklzc3VlQ29tbWVudDY0ODk5ODI2NA==,9599,simonw,2020-06-24T18:43:02Z,2020-06-24T18:43:02Z,OWNER,Thanks for the bug report. Yes I think #838 may be the same issue. Will investigate.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/858#issuecomment-648997857,https://api.github.com/repos/simonw/datasette/issues/858,648997857,MDEyOklzc3VlQ29tbWVudDY0ODk5Nzg1Nw==,9599,simonw,2020-06-24T18:42:10Z,2020-06-24T18:42:10Z,OWNER,I really need to get myself a Windows 10 development environment working so I can dig into this kind of bug properly. I have a gaming PC lying around that I could re-task for that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642388564,publish heroku does not work on Windows 10,
https://github.com/simonw/datasette/pull/866#issuecomment-648818707,https://api.github.com/repos/simonw/datasette/issues/866,648818707,MDEyOklzc3VlQ29tbWVudDY0ODgxODcwNw==,22429695,codecov[bot],2020-06-24T13:26:14Z,2020-06-24T13:26:14Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=h1) Report
> Merging [#866](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #866 +/- ##
=======================================
Coverage 82.99% 82.99%
=======================================
Files 26 26
Lines 3547 3547
=======================================
Hits 2944 2944
Misses 603 603
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=footer). Last update [1a5b7d3...fb64dda](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644610729,"Update pytest-asyncio requirement from <0.13,>=0.10 to >=0.10,<0.15",
https://github.com/simonw/datasette/issues/838#issuecomment-648800356,https://api.github.com/repos/simonw/datasette/issues/838,648800356,MDEyOklzc3VlQ29tbWVudDY0ODgwMDM1Ng==,6739646,tballison,2020-06-24T12:51:48Z,2020-06-24T12:51:48Z,NONE,">But also want to say thanks for a great tool
+1!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/865#issuecomment-648799963,https://api.github.com/repos/simonw/datasette/issues/865,648799963,MDEyOklzc3VlQ29tbWVudDY0ODc5OTk2Mw==,6739646,tballison,2020-06-24T12:51:01Z,2020-06-24T12:51:01Z,NONE,This seems to be a duplicate of: https://github.com/simonw/datasette/issues/838,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/859#issuecomment-648669523,https://api.github.com/repos/simonw/datasette/issues/859,648669523,MDEyOklzc3VlQ29tbWVudDY0ODY2OTUyMw==,3243482,abdusco,2020-06-24T08:13:23Z,2020-06-24T10:30:36Z,CONTRIBUTOR,"I tried setting `cache_size_kb=0` then `cache_size_kb=100000`, still getting this behavior. I even changed `Database::table_counts` and lowered time limit to 1
```py
table_count = (
await self.execute(
""select count(*) from [{}]"".format(table),
custom_time_limit=1,
)
).rows[0][0]
counts[table] = table_count
```
I feel like 10 seconds is a magic number, like a processing timeout and datasette gives up and returns the page.
Index page loads instantly, table page, query page, as well. But when I return to database page after some time, it loads in 10s.
EDIT:
It's always like 10 + 0.3s, like 10s wait and timeout then 300ms to render the page","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/864#issuecomment-648580556,https://api.github.com/repos/simonw/datasette/issues/864,648580556,MDEyOklzc3VlQ29tbWVudDY0ODU4MDU1Ng==,9599,simonw,2020-06-24T04:40:49Z,2020-06-24T04:40:49Z,OWNER,The ideal fix here would be to rework my `BaseView` subclass mechanism to work with `register_routes()` so that those views don't have any special privileges above plugin-provided views.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/864#issuecomment-648580236,https://api.github.com/repos/simonw/datasette/issues/864,648580236,MDEyOklzc3VlQ29tbWVudDY0ODU4MDIzNg==,9599,simonw,2020-06-24T04:39:39Z,2020-06-24T04:39:39Z,OWNER,"Urgh, fixing this is going to be a bit of a pain.
Here's where I added that custom `dispatch_request()` method - it was to implement flash messaging in #790: https://github.com/simonw/datasette/blame/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/views/base.py#L85
If I want this to be made available to `register_routes()` views as well, I'm going to have to move the logic somewhere else. In particular I need to make sure that the `request` object is created once and used throughout the whole request cycle.
Currently `register_routes()` view functions get their own separate request object which is created here:
https://github.com/simonw/datasette/blob/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/app.py#L1057-L1068
So I'm going to have to refactor this quite a bit to get that shared request object which can be passed both to `register_routes` views and to my various `BaseView` subclasses.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648442511,https://api.github.com/repos/simonw/sqlite-utils/issues/117,648442511,MDEyOklzc3VlQ29tbWVudDY0ODQ0MjUxMQ==,9599,simonw,2020-06-23T21:39:41Z,2020-06-23T21:39:41Z,OWNER,"So there are two sides to supporting this:
- Being able to sensibly introspect composite foreign keys
- Being able to define composite foreign keys when creating a table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644161221,Support for compound (composite) foreign keys,
https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648440634,https://api.github.com/repos/simonw/sqlite-utils/issues/117,648440634,MDEyOklzc3VlQ29tbWVudDY0ODQ0MDYzNA==,9599,simonw,2020-06-23T21:35:16Z,2020-06-23T21:35:16Z,OWNER,Relevant discussion: https://github.com/simonw/sqlite-generate/issues/8#issuecomment-648438056,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644161221,Support for compound (composite) foreign keys,
https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648440525,https://api.github.com/repos/simonw/sqlite-utils/issues/117,648440525,MDEyOklzc3VlQ29tbWVudDY0ODQ0MDUyNQ==,9599,simonw,2020-06-23T21:35:01Z,2020-06-23T21:35:01Z,OWNER,"Here's what's missing:
```
In [11]: db.conn.execute('PRAGMA foreign_key_list(song)').fetchall()
Out[11]:
[(0,
0,
'album',
'songartist',
'albumartist',
'NO ACTION',
'NO ACTION',
'NONE'),
(0, 1, 'album', 'songalbum', 'albumname', 'NO ACTION', 'NO ACTION', 'NONE')]
```
Compare with this code here:
https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/sqlite_utils/db.py#L563-L579
The first two columns returned by `PRAGMA foreign_key_list(table)` are `id` and `seq` - these show when two foreign key records are part of the same compound foreign key. `sqlite-utils` entirely ignores those at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644161221,Support for compound (composite) foreign keys,
https://github.com/simonw/sqlite-utils/issues/116#issuecomment-648434885,https://api.github.com/repos/simonw/sqlite-utils/issues/116,648434885,MDEyOklzc3VlQ29tbWVudDY0ODQzNDg4NQ==,9599,simonw,2020-06-23T21:21:33Z,2020-06-23T21:21:33Z,OWNER,New docs: https://github.com/simonw/sqlite-utils/blob/2.10.1/docs/python-api.rst#introspection,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644122661,Documentation for table.pks introspection property,
https://github.com/simonw/sqlite-utils/issues/116#issuecomment-648403834,https://api.github.com/repos/simonw/sqlite-utils/issues/116,648403834,MDEyOklzc3VlQ29tbWVudDY0ODQwMzgzNA==,9599,simonw,2020-06-23T20:36:29Z,2020-06-23T20:36:29Z,OWNER,Should go in this section https://sqlite-utils.readthedocs.io/en/stable/python-api.html#introspection - under `.columns_dict` and before `.foreign_keys`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644122661,Documentation for table.pks introspection property,
https://github.com/simonw/datasette/issues/694#issuecomment-648296323,https://api.github.com/repos/simonw/datasette/issues/694,648296323,MDEyOklzc3VlQ29tbWVudDY0ODI5NjMyMw==,3903726,kwladyka,2020-06-23T17:10:51Z,2020-06-23T17:10:51Z,NONE,"@simonw
Did you find the reason? I had similar situation and I check this on millions ways. I am sure app doesn't consume such memory.
I was trying the app with:
`docker run --rm -it -p 80:80 -m 128M foo`
I was watching app with `docker stats`. Even limited memory by `CMD [""java"", ""-Xms60M"", ""-Xmx60M"", ""-jar"", ""api.jar""]`.
Checked memory usage by app in code and print bash commands. The app definitely doesn't use this memory. Also doesn't write files.
Only one solution is to change memory to 512M.
It is definitely something wrong with `cloud run`.
I even did special app for testing this. It looks like when I cross very small amount of code / memory / app size in random when, then memory needs grow +hundreds. Nothing make sense here. Especially it works everywhere expect cloud run.
Please let me know if you discovered something more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",576582604,datasette publish cloudrun --memory option,
https://github.com/simonw/datasette/issues/859#issuecomment-648234787,https://api.github.com/repos/simonw/datasette/issues/859,648234787,MDEyOklzc3VlQ29tbWVudDY0ODIzNDc4Nw==,9599,simonw,2020-06-23T15:22:51Z,2020-06-23T15:22:51Z,OWNER,"I wonder if this is a SQLite caching issue then?
Datasette has a configuration option for this but I haven't spent much time experimenting with it so I don't know how much of an impact it can have: https://datasette.readthedocs.io/en/stable/config.html#cache-size-kb","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-648232645,https://api.github.com/repos/simonw/datasette/issues/859,648232645,MDEyOklzc3VlQ29tbWVudDY0ODIzMjY0NQ==,3243482,abdusco,2020-06-23T15:19:53Z,2020-06-23T15:19:53Z,CONTRIBUTOR,"The issue seems to appear sporadically, like when I return to database page after a while, during which some records have been added to the database.
I've just visited database, page first visit took ~10s, consecutive visits took 0.3s.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-648163272,https://api.github.com/repos/simonw/datasette/issues/859,648163272,MDEyOklzc3VlQ29tbWVudDY0ODE2MzI3Mg==,9599,simonw,2020-06-23T13:52:23Z,2020-06-23T13:52:23Z,OWNER,"I'm chunking inserts at 100 at a time right now: https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L1030
I think the performance is more down to using Faker to create the test data - generating millions of entirely fake, randomized records takes a fair bit of time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647925594,https://api.github.com/repos/simonw/datasette/issues/859,647925594,MDEyOklzc3VlQ29tbWVudDY0NzkyNTU5NA==,3243482,abdusco,2020-06-23T05:55:21Z,2020-06-23T06:28:29Z,CONTRIBUTOR,"Hmm, not seeing the problem now.
I've removed the commented out sections in `database.py` and restarted the process. Database page now loads in <250ms.
I have couple of workers that check some pages regularly and scrape new content and save to the DB. Could it be that datasette tries to recount tables every time database size changes? Normally it keeps a count cache, but as DB gets updated so often (new content every 5 min or so) it's practically recounting every time I go to the database page?
EDIT:
It turns out it doesn't hold cache with mutable databases.
I'll update the issue with more findings and a better way to reproduce the problem if I encounter it again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647936117,https://api.github.com/repos/simonw/datasette/issues/859,647936117,MDEyOklzc3VlQ29tbWVudDY0NzkzNjExNw==,3243482,abdusco,2020-06-23T06:25:17Z,2020-06-23T06:25:17Z,CONTRIBUTOR,">
>
> ```
> sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50
> ```
>
> Looks like that will take 35 minutes to run (it's not a particularly fast tool).
Try chunking write operations into batches every 1000 records or so.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647935300,https://api.github.com/repos/simonw/datasette/issues/859,647935300,MDEyOklzc3VlQ29tbWVudDY0NzkzNTMwMA==,3243482,abdusco,2020-06-23T06:23:01Z,2020-06-23T06:23:01Z,CONTRIBUTOR,"> You said ""200k+, 50+ rows in a couple of tables"" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.
Ah that was a typo, I meant 50k.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647923666,https://api.github.com/repos/simonw/datasette/issues/859,647923666,MDEyOklzc3VlQ29tbWVudDY0NzkyMzY2Ng==,3243482,abdusco,2020-06-23T05:49:31Z,2020-06-23T05:49:31Z,CONTRIBUTOR,"I think I should mention that having FTS on all tables mean I have 5 visible, 25 hidden (FTS) tables displayed on database page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647894903,https://api.github.com/repos/simonw/datasette/issues/859,647894903,MDEyOklzc3VlQ29tbWVudDY0Nzg5NDkwMw==,9599,simonw,2020-06-23T04:07:59Z,2020-06-23T04:07:59Z,OWNER,"Just to check: are you seeing the problem on this page: https://latest.datasette.io/fixtures (the database page) - or this page (the table page): https://latest.datasette.io/fixtures/compound_three_primary_keys
If it's the table page then the problem may well be #862.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/596#issuecomment-647893140,https://api.github.com/repos/simonw/datasette/issues/596,647893140,MDEyOklzc3VlQ29tbWVudDY0Nzg5MzE0MA==,9599,simonw,2020-06-23T03:59:51Z,2020-06-23T03:59:51Z,OWNER,Related: #862 - a time limit on the total time spent considering suggested facets for a table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",507454958,Handle really wide tables better,
https://github.com/simonw/datasette/issues/862#issuecomment-647892930,https://api.github.com/repos/simonw/datasette/issues/862,647892930,MDEyOklzc3VlQ29tbWVudDY0Nzg5MjkzMA==,9599,simonw,2020-06-23T03:58:48Z,2020-06-23T03:58:48Z,OWNER,Should this be controlled be a separate configuration setting? I'm inclined to say no - I think instead I'll set the limit to be 10 * whatever `facet_suggest_time_limit_ms` is.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",643510821,Set an upper limit on total facet suggestion time for a page,
https://github.com/simonw/datasette/issues/859#issuecomment-647890619,https://api.github.com/repos/simonw/datasette/issues/859,647890619,MDEyOklzc3VlQ29tbWVudDY0Nzg5MDYxOQ==,9599,simonw,2020-06-23T03:48:21Z,2020-06-23T03:48:21Z,OWNER," sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50
Looks like that will take 35 minutes to run (it's not a particularly fast tool).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647890378,https://api.github.com/repos/simonw/datasette/issues/859,647890378,MDEyOklzc3VlQ29tbWVudDY0Nzg5MDM3OA==,9599,simonw,2020-06-23T03:47:19Z,2020-06-23T03:47:19Z,OWNER,"I generated a 600MB database using [sqlite-generate](https://github.com/simonw/sqlite-generate) just now - with 100 tables at 100,00 rows and 3 tables at 1,000,000 rows - and performance of the database page was fine, 250ms.
Those tables only had 4 columns each though.
You said ""200k+, 50+ rows in a couple of tables"" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/861#issuecomment-647889674,https://api.github.com/repos/simonw/datasette/issues/861,647889674,MDEyOklzc3VlQ29tbWVudDY0Nzg4OTY3NA==,9599,simonw,2020-06-23T03:44:17Z,2020-06-23T03:44:17Z,OWNER,https://github.com/simonw/sqlite-generate is now ready to be used - see also https://pypi.org/project/sqlite-generate/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642652808,Script to generate larger SQLite test files,
https://github.com/simonw/datasette/issues/861#issuecomment-647822757,https://api.github.com/repos/simonw/datasette/issues/861,647822757,MDEyOklzc3VlQ29tbWVudDY0NzgyMjc1Nw==,9599,simonw,2020-06-22T23:40:43Z,2020-06-22T23:40:43Z,OWNER,"I started building that tool here: https://github.com/simonw/sqlite-generate
(I built a new cookiecutter template for that too, https://github.com/simonw/click-app)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642652808,Script to generate larger SQLite test files,
https://github.com/simonw/datasette/issues/838#issuecomment-647803394,https://api.github.com/repos/simonw/datasette/issues/838,647803394,MDEyOklzc3VlQ29tbWVudDY0NzgwMzM5NA==,6289012,ChristopherWilks,2020-06-22T22:36:34Z,2020-06-22T22:36:34Z,NONE,"I also am seeing the same issue with an Apache setup (same even w/o `ProxyPassReverse`, though I typically use it as @tsibley stated).
But also want to say thanks for a great tool (this issue not withstanding)!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/861#issuecomment-647266979,https://api.github.com/repos/simonw/datasette/issues/861,647266979,MDEyOklzc3VlQ29tbWVudDY0NzI2Njk3OQ==,9599,simonw,2020-06-22T04:26:25Z,2020-06-22T04:26:25Z,OWNER,I think this is a separate Click utility. I'm going to call it `sqlite-generate`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642652808,Script to generate larger SQLite test files,
https://github.com/simonw/datasette/issues/687#issuecomment-647258199,https://api.github.com/repos/simonw/datasette/issues/687,647258199,MDEyOklzc3VlQ29tbWVudDY0NzI1ODE5OQ==,9599,simonw,2020-06-22T03:55:20Z,2020-06-22T03:55:20Z,OWNER,https://datasette.readthedocs.io/en/latest/testing_plugins.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",572896293,Expand plugins documentation to multiple pages,
https://github.com/simonw/datasette/issues/687#issuecomment-647237091,https://api.github.com/repos/simonw/datasette/issues/687,647237091,MDEyOklzc3VlQ29tbWVudDY0NzIzNzA5MQ==,9599,simonw,2020-06-22T02:44:10Z,2020-06-22T02:44:10Z,OWNER,"Now split into four pages:
- https://datasette.readthedocs.io/en/latest/plugins.html
- https://datasette.readthedocs.io/en/latest/writing_plugins.html
- https://datasette.readthedocs.io/en/latest/plugin_hooks.html
- https://datasette.readthedocs.io/en/latest/internals.html
Still need to add the ""Testing plugins"" page, then I can close this issue.
I should also do #855, documenting the new `datasette-plugin` cookiecutter template. That can go in `writing_plugins.rst`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",572896293,Expand plugins documentation to multiple pages,
https://github.com/simonw/datasette/issues/687#issuecomment-647203845,https://api.github.com/repos/simonw/datasette/issues/687,647203845,MDEyOklzc3VlQ29tbWVudDY0NzIwMzg0NQ==,9599,simonw,2020-06-22T00:32:42Z,2020-06-22T00:32:42Z,OWNER,"Maybe add this to the plugins.rst page near the top:
```
.. toctree::
:caption: See also
:maxdepth: 1
plugin_hooks
internals
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",572896293,Expand plugins documentation to multiple pages,
https://github.com/simonw/datasette/issues/859#issuecomment-647194131,https://api.github.com/repos/simonw/datasette/issues/859,647194131,MDEyOklzc3VlQ29tbWVudDY0NzE5NDEzMQ==,3243482,abdusco,2020-06-21T23:15:54Z,2020-06-21T23:26:09Z,CONTRIBUTOR,"I'm not sure if table counts are to blame. There shouldn't be a ~3 orders of magnitude difference.
```fish
user@klein /a/w/scrapyard (master)> set sql ""select count(*) from table_1; select count(*) from table_2; select count(*) from table_3;""
user@klein /a/w/scrapyard (master)> time sqlite3 scrapyard.db ""$sql""
187489
46492
2229
________________________________________________________
Executed in 25.57 millis fish external
usr time 3.55 millis 0.00 micros 3.55 millis
sys time 22.42 millis 1123.00 micros 21.30 millis
```
but not letting datasette count the tables definitely helps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/687#issuecomment-647190177,https://api.github.com/repos/simonw/datasette/issues/687,647190177,MDEyOklzc3VlQ29tbWVudDY0NzE5MDE3Nw==,9599,simonw,2020-06-21T22:32:36Z,2020-06-21T22:32:36Z,OWNER,I'm going to break out the plugin hooks first in a single commit to make for a cleaner commit history (since that way git can hopefully detect that the content moved).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",572896293,Expand plugins documentation to multiple pages,
https://github.com/simonw/datasette/issues/687#issuecomment-647190144,https://api.github.com/repos/simonw/datasette/issues/687,647190144,MDEyOklzc3VlQ29tbWVudDY0NzE5MDE0NA==,9599,simonw,2020-06-21T22:32:13Z,2020-06-21T22:32:13Z,OWNER,"So the new plan is NOT to have a `plugins/` folder, but instead have several top-level pages:
- Plugins (exists)
- Writing plugins
- Plugin hooks
- Testing plugins
- Internals for plugins (exists)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",572896293,Expand plugins documentation to multiple pages,
https://github.com/simonw/datasette/issues/859#issuecomment-647189948,https://api.github.com/repos/simonw/datasette/issues/859,647189948,MDEyOklzc3VlQ29tbWVudDY0NzE4OTk0OA==,9599,simonw,2020-06-21T22:30:12Z,2020-06-21T22:30:43Z,OWNER,"I'll write a little script which generates a 300MB SQLite file with a bunch of tables with lots of randomly generated rows in to help test this.
Having a tool like that which can generate larger databases with different gnarly performance characteristics will be useful for other performance work too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647189666,https://api.github.com/repos/simonw/datasette/issues/859,647189666,MDEyOklzc3VlQ29tbWVudDY0NzE4OTY2Ng==,9599,simonw,2020-06-21T22:26:55Z,2020-06-21T22:26:55Z,OWNER,"This makes a lot of sense. I implemented the mechanism for the index page because I have my own instance of Datasette that was running slow, but it had a dozen database files attached to it. I've not run into this with a single giant database file but it absolutely makes sense that the same optimization would be necessary for the database page there too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/860#issuecomment-647189535,https://api.github.com/repos/simonw/datasette/issues/860,647189535,MDEyOklzc3VlQ29tbWVudDY0NzE4OTUzNQ==,9599,simonw,2020-06-21T22:25:16Z,2020-06-21T22:25:27Z,OWNER,"This is also relevant to #639, and may mean I can close that ticket in place of this one. I'm going to get this at least to a proof-of-concept stage first though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/357#issuecomment-647189045,https://api.github.com/repos/simonw/datasette/issues/357,647189045,MDEyOklzc3VlQ29tbWVudDY0NzE4OTA0NQ==,9599,simonw,2020-06-21T22:19:58Z,2020-06-21T22:19:58Z,OWNER,"I'm going to take this in a different direction.
I'm not happy with how `metadata.(json|yaml)` keeps growing new features. Rather than having a single plugin hook for all of `metadata.json` I'm going to split out the feature that shows actual real metadata for tables and databases - `source`, `license` etc - into its own plugin-powered mechanism.
So I'm going to close this ticket and spin up a new one for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",348043884,Plugin hook for loading metadata.json,
https://github.com/simonw/datasette/issues/859#issuecomment-647135713,https://api.github.com/repos/simonw/datasette/issues/859,647135713,MDEyOklzc3VlQ29tbWVudDY0NzEzNTcxMw==,3243482,abdusco,2020-06-21T14:30:02Z,2020-06-21T14:30:02Z,CONTRIBUTOR,"Oops, the same method is called from both index and database pages. But removing select count queries speed up the page load quite a bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/687#issuecomment-646938984,https://api.github.com/repos/simonw/datasette/issues/687,646938984,MDEyOklzc3VlQ29tbWVudDY0NjkzODk4NA==,9599,simonw,2020-06-20T04:22:25Z,2020-06-20T04:23:02Z,OWNER,"I think I want the ""Plugin hooks"" page to be top-level, parallel to ""Plugins"" and ""Internals for Plugins"". It's the page of documentation refer to most often so I don't want to have to click down a hierarchy from the side navigation to find it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",572896293,Expand plugins documentation to multiple pages,
https://github.com/simonw/datasette/issues/687#issuecomment-646930455,https://api.github.com/repos/simonw/datasette/issues/687,646930455,MDEyOklzc3VlQ29tbWVudDY0NjkzMDQ1NQ==,9599,simonw,2020-06-20T03:22:21Z,2020-06-20T03:22:21Z,OWNER,The tutorial can start by showing how to use the new cookiecutter template from #642.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",572896293,Expand plugins documentation to multiple pages,
https://github.com/simonw/datasette/issues/855#issuecomment-646930365,https://api.github.com/repos/simonw/datasette/issues/855,646930365,MDEyOklzc3VlQ29tbWVudDY0NjkzMDM2NQ==,9599,simonw,2020-06-20T03:21:48Z,2020-06-20T03:21:48Z,OWNER,"Maybe I should also refactor the plugin documentation, as contemplated in #687.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642127307,Add instructions for using cookiecutter plugin template to plugin docs,
https://github.com/simonw/datasette/issues/642#issuecomment-646930160,https://api.github.com/repos/simonw/datasette/issues/642,646930160,MDEyOklzc3VlQ29tbWVudDY0NjkzMDE2MA==,9599,simonw,2020-06-20T03:20:25Z,2020-06-20T03:20:25Z,OWNER,Shipped this today! https://github.com/simonw/datasette-plugin is a cookiecutter template for creating new plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",529429214,Provide a cookiecutter template for creating new plugins,
https://github.com/simonw/datasette/issues/642#issuecomment-646930059,https://api.github.com/repos/simonw/datasette/issues/642,646930059,MDEyOklzc3VlQ29tbWVudDY0NjkzMDA1OQ==,9599,simonw,2020-06-20T03:19:57Z,2020-06-20T03:19:57Z,OWNER,"@psychemedia sorry I missed your comment before.
Niche Museums is definitely the best example of custom templates at the moment: https://github.com/simonw/museums/tree/master/templates
I want to comprehensively document the variables made available to custom templates before shipping Datasette 1.0 - just filed that as #857.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",529429214,Provide a cookiecutter template for creating new plugins,
https://github.com/simonw/datasette/issues/855#issuecomment-646928638,https://api.github.com/repos/simonw/datasette/issues/855,646928638,MDEyOklzc3VlQ29tbWVudDY0NjkyODYzOA==,9599,simonw,2020-06-20T03:09:41Z,2020-06-20T03:09:41Z,OWNER,I've shipped the cookiecutter template and used it to build https://github.com/simonw/datasette-saved-queries - it's ready to add to the official documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642127307,Add instructions for using cookiecutter plugin template to plugin docs,
https://github.com/simonw/datasette/issues/852#issuecomment-646905073,https://api.github.com/repos/simonw/datasette/issues/852,646905073,MDEyOklzc3VlQ29tbWVudDY0NjkwNTA3Mw==,9599,simonw,2020-06-20T00:21:34Z,2020-06-20T00:22:28Z,OWNER,New repo: https://github.com/simonw/datasette-saved-queries - which I created using the new cookiecutter template at https://github.com/simonw/datasette-plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",640917326,canned_queries() plugin hook,
https://github.com/simonw/datasette/issues/852#issuecomment-646760805,https://api.github.com/repos/simonw/datasette/issues/852,646760805,MDEyOklzc3VlQ29tbWVudDY0Njc2MDgwNQ==,9599,simonw,2020-06-19T17:07:45Z,2020-06-19T17:07:45Z,OWNER,"Plugin idea: `datasette-saved-queries` - it uses the `startup` hook to initialize a `saved_queries` table, then uses the `canned_queries` hook to add a writable canned query for saving records to that table.
Then it returns any queries from that table as additional canned queries.
Bonus idea: it could write the user's actor_id to a column if they are signed in, and provide a link to see ""just my saved queries"" in that case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",640917326,canned_queries() plugin hook,
https://github.com/simonw/datasette/issues/849#issuecomment-646686493,https://api.github.com/repos/simonw/datasette/issues/849,646686493,MDEyOklzc3VlQ29tbWVudDY0NjY4NjQ5Mw==,9599,simonw,2020-06-19T15:04:51Z,2020-06-19T15:04:51Z,OWNER,https://twitter.com/jaffathecake/status/1273983493006077952 concerns what happens to open pull requests - they will automatically close when you remove `master` unless you repoint them to `main` first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/852#issuecomment-646396772,https://api.github.com/repos/simonw/datasette/issues/852,646396772,MDEyOklzc3VlQ29tbWVudDY0NjM5Njc3Mg==,9599,simonw,2020-06-19T02:16:47Z,2020-06-19T02:16:47Z,OWNER,I'll close this once I've built a plugin against it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",640917326,canned_queries() plugin hook,
https://github.com/simonw/datasette/issues/852#issuecomment-646396690,https://api.github.com/repos/simonw/datasette/issues/852,646396690,MDEyOklzc3VlQ29tbWVudDY0NjM5NjY5MA==,9599,simonw,2020-06-19T02:16:24Z,2020-06-19T02:16:24Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/plugins.html#canned-queries-datasette-database-actor,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",640917326,canned_queries() plugin hook,
https://github.com/simonw/datasette/issues/852#issuecomment-646396499,https://api.github.com/repos/simonw/datasette/issues/852,646396499,MDEyOklzc3VlQ29tbWVudDY0NjM5NjQ5OQ==,9599,simonw,2020-06-19T02:15:49Z,2020-06-19T02:15:58Z,OWNER,"Released an alpha preview in https://github.com/simonw/datasette/releases/tag/0.45a1
Wrote about this here: https://simonwillison.net/2020/Jun/19/datasette-alphas/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",640917326,canned_queries() plugin hook,
https://github.com/simonw/datasette/issues/852#issuecomment-646350530,https://api.github.com/repos/simonw/datasette/issues/852,646350530,MDEyOklzc3VlQ29tbWVudDY0NjM1MDUzMA==,9599,simonw,2020-06-18T23:13:57Z,2020-06-18T23:14:11Z,OWNER,"```python
@hookspec
def canned_queries(datasette, database, actor):
""Return a dictionary of canned query definitions or an awaitable function that returns them""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",640917326,canned_queries() plugin hook,
https://github.com/simonw/datasette/issues/852#issuecomment-646329456,https://api.github.com/repos/simonw/datasette/issues/852,646329456,MDEyOklzc3VlQ29tbWVudDY0NjMyOTQ1Ng==,9599,simonw,2020-06-18T22:07:09Z,2020-06-18T22:07:37Z,OWNER,"It would be neat if the queries returned by this hook could be restricted to specific users. I think I can do that by returning an ""allow"" block as part of the query.
But... what if we allow users to save private queries and we might have thousands of users each with hundreds of saved queries?
For that case it would be good if the plugin hook could take an optional `actor` parameter.
This would also allow us to dynamically generate a canned query for ""return the bookmarks belonging to this actor"" or similar!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",640917326,canned_queries() plugin hook,
https://github.com/simonw/datasette/issues/807#issuecomment-646320237,https://api.github.com/repos/simonw/datasette/issues/807,646320237,MDEyOklzc3VlQ29tbWVudDY0NjMyMDIzNw==,9599,simonw,2020-06-18T21:41:16Z,2020-06-18T21:41:16Z,OWNER,"https://pypi.org/project/datasette/0.45a0/ is the release on PyPI.
And in a fresh virtual environment:
```
$ pip install datasette==0.45a0
...
$ datasette --version
datasette, version 0.45a0
```
But running `pip install datasette` still gets 0.44.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646319315,https://api.github.com/repos/simonw/datasette/issues/807,646319315,MDEyOklzc3VlQ29tbWVudDY0NjMxOTMxNQ==,9599,simonw,2020-06-18T21:38:56Z,2020-06-18T21:38:56Z,OWNER,"This worked!
https://pypi.org/project/datasette/#history
![]()
https://github.com/simonw/datasette/releases/tag/0.45a0 is my manually created GitHub prerelease.
https://datasette.readthedocs.io/en/latest/changelog.html#a0-2020-06-18 has the release notes.
A shame Read The Docs doesn't seem to build the docs for these releases -it's not showing the tag in the releases pane here:
![]()
Also the new tag isn't an option in the Build menu on https://readthedocs.org/projects/datasette/builds/
Not a big problem though since the ""latest"" tag on Read The Docs will still carry the in-development documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/835#issuecomment-646308467,https://api.github.com/repos/simonw/datasette/issues/835,646308467,MDEyOklzc3VlQ29tbWVudDY0NjMwODQ2Nw==,9599,simonw,2020-06-18T21:12:50Z,2020-06-18T21:12:50Z,OWNER,"Problem there is Login CSRF attacks: https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#login-csrf - I still want to perform CSRF checks on login forms, even though the user may not yet have any cookies.
Maybe I can turn off CSRF checks for cookie-free requests but allow login forms to specifically opt back in to CSRF protection?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/835#issuecomment-646307083,https://api.github.com/repos/simonw/datasette/issues/835,646307083,MDEyOklzc3VlQ29tbWVudDY0NjMwNzA4Mw==,9599,simonw,2020-06-18T21:09:35Z,2020-06-18T21:09:35Z,OWNER,So maybe one really easy fix here is to disable CSRF checks entirely for any request that doesn't have any cookies? Also suggested here: https://twitter.com/mrkurt/status/1273682965168603137,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/807#issuecomment-646303240,https://api.github.com/repos/simonw/datasette/issues/807,646303240,MDEyOklzc3VlQ29tbWVudDY0NjMwMzI0MA==,9599,simonw,2020-06-18T21:00:41Z,2020-06-18T21:00:41Z,OWNER,New documentation about the alpha/beta releases: https://datasette.readthedocs.io/en/latest/contributing.html#contributing-alpha-beta,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646302909,https://api.github.com/repos/simonw/datasette/issues/807,646302909,MDEyOklzc3VlQ29tbWVudDY0NjMwMjkwOQ==,9599,simonw,2020-06-18T21:00:02Z,2020-06-18T21:00:02Z,OWNER,Alpha release is running through Travis now: https://travis-ci.org/github/simonw/datasette/builds/699864168,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646293670,https://api.github.com/repos/simonw/datasette/issues/807,646293670,MDEyOklzc3VlQ29tbWVudDY0NjI5MzY3MA==,9599,simonw,2020-06-18T20:38:50Z,2020-06-18T20:38:50Z,OWNER,"https://pypi.org/project/datasette-render-images/#history worked:
![]()
I'm now confident enough that I'll make these changes and ship an alpha of Datasette itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646293029,https://api.github.com/repos/simonw/datasette/issues/807,646293029,MDEyOklzc3VlQ29tbWVudDY0NjI5MzAyOQ==,9599,simonw,2020-06-18T20:37:28Z,2020-06-18T20:37:46Z,OWNER,"Here's the Read The Docs documentation on versioned releases: https://docs.readthedocs.io/en/stable/versions.html
It looks like they do the right thing:
> We in fact are parsing your tag names against the rules given by PEP 440. This spec allows “normal” version numbers like 1.4.2 as well as pre-releases. An alpha version or a release candidate are examples of pre-releases and they look like this: 2.0a1.
>
> We only consider non pre-releases for the stable version of your documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646292578,https://api.github.com/repos/simonw/datasette/issues/807,646292578,MDEyOklzc3VlQ29tbWVudDY0NjI5MjU3OA==,9599,simonw,2020-06-18T20:36:22Z,2020-06-18T20:36:22Z,OWNER,"https://travis-ci.com/github/simonw/datasette-render-images/builds/172118541 demonstrates that the alpha/beta conditional is working as intended:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646291309,https://api.github.com/repos/simonw/datasette/issues/807,646291309,MDEyOklzc3VlQ29tbWVudDY0NjI5MTMwOQ==,9599,simonw,2020-06-18T20:33:31Z,2020-06-18T20:33:31Z,OWNER,"One more experiment: I'm going to ship `datasette-render-images` 0.2 and see if that works correctly - including printing out the new debug section I put in the Travis config here: https://github.com/simonw/datasette-render-images/blob/6b5f22dab75ca364f671f5597556d2665a251bd8/.travis.yml#L35-L39 - which should demonstrate if my conditional for pushing to Docker Hub will work or not.
In the alpha releasing run on Travis that echo statement did NOT execute: https://travis-ci.com/github/simonw/datasette-render-images/builds/172116625","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646290171,https://api.github.com/repos/simonw/datasette/issues/807,646290171,MDEyOklzc3VlQ29tbWVudDY0NjI5MDE3MQ==,9599,simonw,2020-06-18T20:30:48Z,2020-06-18T20:30:48Z,OWNER,"OK, I just shipped 0.2a0 of `datasette-render-images` - https://pypi.org/project/datasette-render-images/ has no indication of that:
![]()
But this page does: https://pypi.org/project/datasette-render-images/#history
![]()
And https://pypi.org/project/datasette-render-images/0.2a0/ exists.
In a fresh virtual environment `pip install datasette-render-images` gets 0.1.
`pip install datasette-render-images==0.2a0` gets 0.2a0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/835#issuecomment-646288146,https://api.github.com/repos/simonw/datasette/issues/835,646288146,MDEyOklzc3VlQ29tbWVudDY0NjI4ODE0Ng==,9599,simonw,2020-06-18T20:26:22Z,2020-06-18T20:26:31Z,OWNER,"Useful tip from Carlton Gibson: https://twitter.com/carltongibson/status/1273680590672453632
> DRF makes ALL views CSRF exempt and then enforces CSRF if you're using Session auth only.
>
> View: https://github.com/encode/django-rest-framework/blob/e18e40d6ae42457f60ca9c68054ad40d15ba8433/rest_framework/views.py#L144
> Auth: https://github.com/encode/django-rest-framework/blob/e18e40d6ae42457f60ca9c68054ad40d15ba8433/rest_framework/authentication.py#L130","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/807#issuecomment-646280134,https://api.github.com/repos/simonw/datasette/issues/807,646280134,MDEyOklzc3VlQ29tbWVudDY0NjI4MDEzNA==,9599,simonw,2020-06-18T20:08:15Z,2020-06-18T20:08:15Z,OWNER,https://github.com/simonw/datasette-render-images uses Travis and is low-risk for trying this out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646279428,https://api.github.com/repos/simonw/datasette/issues/807,646279428,MDEyOklzc3VlQ29tbWVudDY0NjI3OTQyOA==,9599,simonw,2020-06-18T20:06:43Z,2020-06-18T20:06:43Z,OWNER,I'm going to try this on a separate repository so I don't accidentally publish a Datasette release I didn't mean to publish!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646279280,https://api.github.com/repos/simonw/datasette/issues/807,646279280,MDEyOklzc3VlQ29tbWVudDY0NjI3OTI4MA==,9599,simonw,2020-06-18T20:06:24Z,2020-06-18T20:06:24Z,OWNER,"So maybe this condition is right?
if: (tag IS present) AND NOT (tag =~ [ab])","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646278801,https://api.github.com/repos/simonw/datasette/issues/807,646278801,MDEyOklzc3VlQ29tbWVudDY0NjI3ODgwMQ==,9599,simonw,2020-06-18T20:05:18Z,2020-06-18T20:05:18Z,OWNER,"Travis conditions documentation: https://docs.travis-ci.com/user/conditions-v1
These look useful:
```
branch =~ /^(one|two)-three$/
(tag =~ ^v) AND (branch = master)
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646277680,https://api.github.com/repos/simonw/datasette/issues/807,646277680,MDEyOklzc3VlQ29tbWVudDY0NjI3NzY4MA==,9599,simonw,2020-06-18T20:02:42Z,2020-06-18T20:02:42Z,OWNER,"So I think if I push a tag of `0.45a0` everything might just work - Travis will build it, push the build to PyPI, PyPI won't treat it as a stable release.
Except... I don't want to push alphas as Docker images - so I need to fix this code:
https://github.com/simonw/datasette/blob/6151c25a5a8d566c109af296244b9267c536bd9a/.travis.yml#L34-L43","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646277155,https://api.github.com/repos/simonw/datasette/issues/807,646277155,MDEyOklzc3VlQ29tbWVudDY0NjI3NzE1NQ==,9599,simonw,2020-06-18T20:01:31Z,2020-06-18T20:01:31Z,OWNER,"I thought I might have to update a regex (my CircleCI configs won't match on `a0`, [example](https://github.com/simonw/datasette-publish-now/blob/420f349b278857f62183d8e9835d64f116758be7/.circleci/config.yml#L22)) but it turns out Travis is currently configured to treat ALL tags as potential releases:
https://github.com/simonw/datasette/blob/6151c25a5a8d566c109af296244b9267c536bd9a/.travis.yml#L21-L35","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646276150,https://api.github.com/repos/simonw/datasette/issues/807,646276150,MDEyOklzc3VlQ29tbWVudDY0NjI3NjE1MA==,9599,simonw,2020-06-18T19:59:17Z,2020-06-18T19:59:17Z,OWNER,"Relevant PEP: https://www.python.org/dev/peps/pep-0440/
Django's implementation dates back 8 years: https://github.com/django/django/commit/40f0ecc56a23d35c2849f8e79276f6d8931412d1
From the PEP:
> Implicit pre-release number
>
> Pre releases allow omitting the numeral in which case it is implicitly assumed to be 0. The normal form for this is to include the 0 explicitly. This allows versions such as 1.2a which is normalized to 1.2a0.
I'm going to habitually include the 0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/807#issuecomment-646273035,https://api.github.com/repos/simonw/datasette/issues/807,646273035,MDEyOklzc3VlQ29tbWVudDY0NjI3MzAzNQ==,9599,simonw,2020-06-18T19:52:28Z,2020-06-18T19:52:28Z,OWNER,"I'd like this soon, because I want to start experimenting with things like #852 and #842 without shipping those plugin hooks in a full stable release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/842#issuecomment-646272627,https://api.github.com/repos/simonw/datasette/issues/842,646272627,MDEyOklzc3VlQ29tbWVudDY0NjI3MjYyNw==,9599,simonw,2020-06-18T19:51:32Z,2020-06-18T19:51:32Z,OWNER,I'd be OK with the first version of this not including a plugin hook.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-646264051,https://api.github.com/repos/simonw/datasette/issues/842,646264051,MDEyOklzc3VlQ29tbWVudDY0NjI2NDA1MQ==,9599,simonw,2020-06-18T19:32:13Z,2020-06-18T19:32:37Z,OWNER,"If every magic parameter has a prefix and suffix, like `_request_ip` and `_actor_id`, then plugins could register a function for a prefix. Register a function to `_actor` and `actor(""id"")`will be called for `_actor_id`.
But does it make sense for every magic parameter to be of form `_a_b`? I think so.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-646246062,https://api.github.com/repos/simonw/datasette/issues/842,646246062,MDEyOklzc3VlQ29tbWVudDY0NjI0NjA2Mg==,9599,simonw,2020-06-18T18:54:41Z,2020-06-18T18:54:41Z,OWNER,"The `_actor_id` param makes this a bit trickier, because we can't just say ""if you see an unknown parameter called X call this function"" - our magic parameter logic isn't adding single parameters, it might add a whole family of them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-646242172,https://api.github.com/repos/simonw/datasette/issues/842,646242172,MDEyOklzc3VlQ29tbWVudDY0NjI0MjE3Mg==,9599,simonw,2020-06-18T18:46:06Z,2020-06-18T18:53:31Z,OWNER,"Yes that can work - and using `__missing__` (new in Python 3) is nicer because then the regular dictionary gets checked first:
```python
import sqlite3
conn = sqlite3.connect("":memory:"")
class Magic(dict):
def __missing__(self, key):
return key.upper()
conn.execute(""select :name"", Magic()).fetchall()
```
Outputs:
```
[('NAME',)]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-646238702,https://api.github.com/repos/simonw/datasette/issues/842,646238702,MDEyOklzc3VlQ29tbWVudDY0NjIzODcwMg==,9599,simonw,2020-06-18T18:39:07Z,2020-06-18T18:39:07Z,OWNER,"It would be nice if Datasette didn't have to do any additional work to find e.g. `_request_ip` if that parameter turned out not to be used by the query.
Could I do this with a custom class that implements `__getitem__()` and then gets passed as SQLite arguments?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/820#issuecomment-646218809,https://api.github.com/repos/simonw/datasette/issues/820,646218809,MDEyOklzc3VlQ29tbWVudDY0NjIxODgwOQ==,9599,simonw,2020-06-18T17:58:02Z,2020-06-18T17:58:02Z,OWNER,I had the same idea again ten days later: #852.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635049296,Idea: Plugin hook for registering canned queries,
https://github.com/simonw/datasette/issues/835#issuecomment-646217766,https://api.github.com/repos/simonw/datasette/issues/835,646217766,MDEyOklzc3VlQ29tbWVudDY0NjIxNzc2Ng==,9599,simonw,2020-06-18T17:55:54Z,2020-06-18T17:56:04Z,OWNER,Idea: a mechanism where the `asgi_csrf()` can take an optional `should_protect()` callback function which gets called with the `scope` and decides if the current request should be protected or not. It can then look at headers and paths and suchlike and make its own decisions. Datasette could then provide a `should_protect()` callback which can interact with plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/835#issuecomment-646216934,https://api.github.com/repos/simonw/datasette/issues/835,646216934,MDEyOklzc3VlQ29tbWVudDY0NjIxNjkzNA==,9599,simonw,2020-06-18T17:54:14Z,2020-06-18T17:54:14Z,OWNER,"> if you did Origin based CSRF checks, then could the absence of an Origin header be used?
https://twitter.com/cnorthwood/status/1273674392757829632","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/835#issuecomment-646214158,https://api.github.com/repos/simonw/datasette/issues/835,646214158,MDEyOklzc3VlQ29tbWVudDY0NjIxNDE1OA==,9599,simonw,2020-06-18T17:48:45Z,2020-06-18T17:48:45Z,OWNER,"I wonder if it's safe to generically say ""Don't do CSRF protection on any request that includes a `Authorization: Bearer...` header - because it's not possible for a regular browser to send that header since the format is different from the header used in browser-based HTTP basic auth?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/835#issuecomment-646209520,https://api.github.com/repos/simonw/datasette/issues/835,646209520,MDEyOklzc3VlQ29tbWVudDY0NjIwOTUyMA==,9599,simonw,2020-06-18T17:39:30Z,2020-06-18T17:40:53Z,OWNER,"`datasette-auth-tokens` could switch to using `asgi_wrapper` instead of `actor_from_request` - then it could add a `scope[""skip_csrf""] = True` scope property to indicate that CSRF should not be protected.
Since `asgi_wrapper` wraps the CSRF protection middleware changes made to the `scope` by an `asgi_wrapper` will be visible to the CSRF middleware:
https://github.com/simonw/datasette/blob/d2aef9f7ef30fa20b1450cd181cf803f44fb4e21/datasette/app.py#L877-L888","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/835#issuecomment-646204308,https://api.github.com/repos/simonw/datasette/issues/835,646204308,MDEyOklzc3VlQ29tbWVudDY0NjIwNDMwOA==,9599,simonw,2020-06-18T17:32:41Z,2020-06-18T17:32:41Z,OWNER,The only way I can think of for a view to opt-out of CSRF protection is for them to be able to reconfigure the `asgi-csrf` middleware to skip specific URL patterns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/835#issuecomment-646175055,https://api.github.com/repos/simonw/datasette/issues/835,646175055,MDEyOklzc3VlQ29tbWVudDY0NjE3NTA1NQ==,9599,simonw,2020-06-18T17:00:45Z,2020-06-18T17:00:45Z,OWNER,Here's the Rails pattern for this: https://gist.github.com/maxivak/a25957942b6c21a41acd,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/835#issuecomment-646172200,https://api.github.com/repos/simonw/datasette/issues/835,646172200,MDEyOklzc3VlQ29tbWVudDY0NjE3MjIwMA==,9599,simonw,2020-06-18T16:57:45Z,2020-06-18T16:57:45Z,OWNER,"I think there are a couple of steps to this one.
The nature of CSRF is that it's about hijacking existing authentication credentials. If your Datasette site runs without any authentication plugins at all CSRF protection isn't actually useful.
Some POST endpoints should be able to opt-out of CSRF protection entirely. A writable canned query that accepts anonymous poll submissions for example might determine that CSRF is not needed.
If a plugin adds `Authorization: Bearer xxx` token support that plugin should also be able to specify that CSRF protection can be skipped. https://github.com/simonw/datasette-auth-tokens could do this.
This means I need two new mechanisms:
- A way for wrapped views to indicate ""actually don't CSRF protect me"". I'm not sure how feasible this is without a major redesign, since the decision to return a 403 forbidden status is made before the wrapped function has even been called.
- A way for authentication plugins like `datasette-auth-tokens` to say ""CSRF protection is not needed for this request"". This is a bit tricky too, since right now the `actor_from_request` hook doesn't have a channel for information other than returning the actor dictionary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/835#issuecomment-646151706,https://api.github.com/repos/simonw/datasette/issues/835,646151706,MDEyOklzc3VlQ29tbWVudDY0NjE1MTcwNg==,9599,simonw,2020-06-18T16:36:23Z,2020-06-18T16:36:23Z,OWNER,Tweeted about this here: https://twitter.com/simonw/status/1273655053170077701,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/853#issuecomment-646140022,https://api.github.com/repos/simonw/datasette/issues/853,646140022,MDEyOklzc3VlQ29tbWVudDY0NjE0MDAyMg==,9599,simonw,2020-06-18T16:21:53Z,2020-06-18T16:21:53Z,OWNER,"I have a test that demonstrates this working, but also demonstrates that the CSRF protection from #798 makes this really tricky to work with. I'd like to improve that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",640943441,Ensure register_routes() works for POST,
https://github.com/simonw/datasette/issues/852#issuecomment-645785830,https://api.github.com/repos/simonw/datasette/issues/852,645785830,MDEyOklzc3VlQ29tbWVudDY0NTc4NTgzMA==,9599,simonw,2020-06-18T05:37:00Z,2020-06-18T05:37:00Z,OWNER,"The easiest way to do this would be with a new plugin hook:
def canned_queries(datasette, database):
""""""Return a list of canned query definitions
or an awaitable function that returns them""
Another approach would be to make the whole of `metadata.json` customizable by plugins.
I think I like the dedicated `canned_queries` option better. I'm not happy with the way metadata keeps growing - see #493 - so adding a dedicated hook would be more future proof against other changes I might make to the metadata mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",640917326,canned_queries() plugin hook,
https://github.com/simonw/datasette/issues/852#issuecomment-645781482,https://api.github.com/repos/simonw/datasette/issues/852,645781482,MDEyOklzc3VlQ29tbWVudDY0NTc4MTQ4Mg==,9599,simonw,2020-06-18T05:24:55Z,2020-06-18T05:25:00Z,OWNER,Question about this on Twitter: https://twitter.com/amjithr/status/1273440766862352384,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",640917326,canned_queries() plugin hook,
https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645599881,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47,645599881,MDEyOklzc3VlQ29tbWVudDY0NTU5OTg4MQ==,9599,simonw,2020-06-17T20:13:48Z,2020-06-17T20:13:48Z,MEMBER,"I've now figured out how to compile specific SQLite versions to help replicate this problem: https://github.com/simonw/til/blob/master/sqlite/ld-preload.md
Next step: replicate the problem!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639542974,Fall back to FTS4 if FTS5 is not available,
https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645515103,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47,645515103,MDEyOklzc3VlQ29tbWVudDY0NTUxNTEwMw==,73579,hpk42,2020-06-17T17:30:01Z,2020-06-17T17:30:01Z,NONE,"It's the one with python3.7::
>>> sqlite3.sqlite_version
'3.11.0'
On Wed, Jun 17, 2020 at 10:24 -0700, Simon Willison wrote:
> That means your version of SQLite is old enough that it doesn't support the FTS5 extension.
>
> Could you share what operating system you're running, and what the output is that you get from running this?
>
> python -c 'import sqlite3; print(sqlite3.connect("":memory:"").execute(""select sqlite_version()"").fetchone()[0])'
>
> I can teach this tool to fall back on FTS4 if FTS5 isn't available.
>
> --
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub:
> https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639542974,Fall back to FTS4 if FTS5 is not available,
https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47,645512127,MDEyOklzc3VlQ29tbWVudDY0NTUxMjEyNw==,9599,simonw,2020-06-17T17:24:22Z,2020-06-17T17:24:22Z,MEMBER,"That means your version of SQLite is old enough that it doesn't support the FTS5 extension.
Could you share what operating system you're running, and what the output is that you get from running this?
python -c 'import sqlite3; print(sqlite3.connect("":memory:"").execute(""select sqlite_version()"").fetchone()[0])'
I can teach this tool to fall back on FTS4 if FTS5 isn't available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639542974,Fall back to FTS4 if FTS5 is not available,
https://github.com/simonw/datasette/issues/851#issuecomment-645293374,https://api.github.com/repos/simonw/datasette/issues/851,645293374,MDEyOklzc3VlQ29tbWVudDY0NTI5MzM3NA==,3243482,abdusco,2020-06-17T10:32:02Z,2020-06-17T10:32:28Z,CONTRIBUTOR,"Welp, I'm an idiot.
Turns out I had a sneaky comma `,` after `sql` key:
```
... (:name, :url),
```
which tells sqlite to expect another `values(...)` list.
Correcting the SQL solved the issue.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",640330278,Having trouble getting writable canned queries to work,
https://github.com/simonw/datasette/issues/850#issuecomment-645068128,https://api.github.com/repos/simonw/datasette/issues/850,645068128,MDEyOklzc3VlQ29tbWVudDY0NTA2ODEyOA==,9599,simonw,2020-06-16T23:52:16Z,2020-06-16T23:52:16Z,OWNER,https://aws.amazon.com/blogs/compute/announcing-http-apis-for-amazon-api-gateway/ looks very important here: AWS HTTP APIs were introduced in December 2019 and appear to be a third of the price of API Gateway.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/236#issuecomment-645067611,https://api.github.com/repos/simonw/datasette/issues/236,645067611,MDEyOklzc3VlQ29tbWVudDY0NTA2NzYxMQ==,9599,simonw,2020-06-16T23:50:12Z,2020-06-16T23:50:59Z,OWNER,"As for your other questions:
> 1. I assume the goal is to have a CORS-friendly HTTPS endpoint that hosts the datasette service + user's db.
Yes, exactly. I know this will limit the size of database that can be deployed (since Lambda has a 50MB total package limit as far as I can tell) but there are plenty of interesting databases that are small enough to fit there.
The new EFS support for Lambda means that theoretically the size of database is now unlimited, which is really interesting. That's what got me inspired to take a look at a proof of concept in #850.
> 2. If that's the goal, I think Lambda alone is insufficient. Lambda provides the compute fabric, but not the HTTP routing. You'd also need to add Application Load Balancer or API Gateway to provide an HTTP endpoint that routes to the lambda function.
>
> Do you have a preference between ALB or API GW? ALB has better economics at scale, but has a minimum monthly cost. API GW has worse per-request economics, but scales to zero when no requests are happening.
I personally like scale-to-zero because many of my projects are likely to receive very little traffic. So API GW first, and maybe ALB as an option later on for people operating at scale?
> 3. Does Datasette have any native components, or is it all pure python? If it has native bits, they'll likely need to be recompiled to work on Amazon Linux 2.
As you've found, the only native component is uvloop which is only needed if uvicorn is being used to serve requests.
> 4. There are a few disparate services that need to be wired together to expose a Python service securely to the web. If I was doing this outside of the datasette publish system, I'd use an AWS CloudFormation template. Even within datasette, I think it still makes sense to use a CloudFormation template and just have the publish plugin invoke it (via the standard `aws` cli) with user-specified parameters. Does that sound reasonable to you?
For the eventual ""datasette publish lambda"" command I want whatever results in the smallest amount of inconvenience for users. I've been trying out Amazon SAM in #850 and it requires users to run Docker on their machines, which is a pretty huge barrier to entry! I don't have much experience with CloudFormation but it's probably a better bet, especially if you can ""pip install"" the dependencies needed to deploy with it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/simonw/datasette/issues/236#issuecomment-645066486,https://api.github.com/repos/simonw/datasette/issues/236,645066486,MDEyOklzc3VlQ29tbWVudDY0NTA2NjQ4Ng==,9599,simonw,2020-06-16T23:45:45Z,2020-06-16T23:45:45Z,OWNER,"Hi Colin,
Sorry I didn't see this sooner! I've just started digging into this myself, to try and play with the new EFS Lambda support: #850.
Yes, uvloop is only needed because of uvicorn. I have a branch here that removes that dependency just for trying out Lambda: https://github.com/simonw/datasette/tree/no-uvicorn - so you can run `pip install https://github.com/simonw/datasette/archive/no-uvicorn.zip` to get that.
I'm going to try out your `datasette-lambda` project next - really excited to see how far you've got with it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/simonw/datasette/issues/850#issuecomment-645064332,https://api.github.com/repos/simonw/datasette/issues/850,645064332,MDEyOklzc3VlQ29tbWVudDY0NTA2NDMzMg==,9599,simonw,2020-06-16T23:37:34Z,2020-06-16T23:37:34Z,OWNER,Just realized Colin Dellow reported an issue with Datasette and Mangum back in April - #719 - and has in fact been working on https://github.com/code402/datasette-lambda for a while!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645063386,https://api.github.com/repos/simonw/datasette/issues/850,645063386,MDEyOklzc3VlQ29tbWVudDY0NTA2MzM4Ng==,9599,simonw,2020-06-16T23:34:07Z,2020-06-16T23:34:07Z,OWNER,"Tried `sam local invoke`:
```
simon@Simons-MacBook-Pro datasette-proof-of-concept % sam local invoke
Invoking app.lambda_handler (python3.8)
Fetching lambci/lambda:python3.8 Docker container image......
Mounting /private/tmp/datasette-proof-of-concept/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: 7c04480b-5d42-168e-dec0-4e8bf34fa596 Version: $LATEST
[INFO] 2020-06-16T23:33:27.24Z 7c04480b-5d42-168e-dec0-4e8bf34fa596 Waiting for application startup.
[INFO] 2020-06-16T23:33:27.24Z 7c04480b-5d42-168e-dec0-4e8bf34fa596 LifespanCycleState.STARTUP: 'lifespan.startup.complete' event received from application.
[INFO] 2020-06-16T23:33:27.24Z 7c04480b-5d42-168e-dec0-4e8bf34fa596 Application startup complete.
[INFO] 2020-06-16T23:33:27.24Z 7c04480b-5d42-168e-dec0-4e8bf34fa596 Waiting for application shutdown.
[INFO] 2020-06-16T23:33:27.24Z 7c04480b-5d42-168e-dec0-4e8bf34fa596 LifespanCycleState.SHUTDOWN: 'lifespan.shutdown.complete' event received from application.
[ERROR] KeyError: 'requestContext'
Traceback (most recent call last):
File ""/var/task/mangum/adapter.py"", line 110, in __call__
return self.handler(event, context)
File ""/var/task/mangum/adapter.py"", line 130, in handler
if ""eventType"" in event[""requestContext""]:
END RequestId: 7c04480b-5d42-168e-dec0-4e8bf34fa596
REPORT RequestId: 7c04480b-5d42-168e-dec0-4e8bf34fa596 Init Duration: 1120.76 ms Duration: 7.08 ms Billed Duration: 100 ms Memory Size: 128 MBMax Memory Used: 47 MB
{""errorType"":""KeyError"",""errorMessage"":""'requestContext'"",""stackTrace"":["" File \""/var/task/mangum/adapter.py\"", line 110, in __call__\n return self.handler(event, context)\n"","" File \""/var/task/mangum/adapter.py\"", line 130, in handler\n if \""eventType\"" in event[\""requestContext\""]:\n""]}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645062266,https://api.github.com/repos/simonw/datasette/issues/850,645062266,MDEyOklzc3VlQ29tbWVudDY0NTA2MjI2Ng==,9599,simonw,2020-06-16T23:30:12Z,2020-06-16T23:33:12Z,OWNER,"OK, changed `requirements.txt` to this:
```
https://github.com/simonw/datasette/archive/no-uvicorn.zip
mangum
```
No `sam build --use-container` runs without errors. Ran `sam deploy` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645063058,https://api.github.com/repos/simonw/datasette/issues/850,645063058,MDEyOklzc3VlQ29tbWVudDY0NTA2MzA1OA==,9599,simonw,2020-06-16T23:32:57Z,2020-06-16T23:32:57Z,OWNER,https://q7lymja3sj.execute-api.us-east-1.amazonaws.com/Prod/hello/ is now giving me a 500 internal server error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645061088,https://api.github.com/repos/simonw/datasette/issues/850,645061088,MDEyOklzc3VlQ29tbWVudDY0NTA2MTA4OA==,9599,simonw,2020-06-16T23:25:41Z,2020-06-16T23:25:41Z,OWNER,"Someone else ran into this problem: https://github.com/iwpnd/fastapi-aws-lambda-example/issues/1
So I need to be able to pip install MOST of Datasette, but skip `uvicorn`. Tricky. I'll try installing a custom fork?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645060598,https://api.github.com/repos/simonw/datasette/issues/850,645060598,MDEyOklzc3VlQ29tbWVudDY0NTA2MDU5OA==,9599,simonw,2020-06-16T23:24:01Z,2020-06-16T23:24:01Z,OWNER,"I changed `requirements.txt` to this:
```
datasette
mangum
```
And `app.py` to this:
```python
from datasette.app import Datasette
from mangum import Mangum
datasette = Datasette([], memory=True)
lambda_handler = Mangum(datasette.app())
```
But then when I ran `sam build --use-container` I got this:
```
simon@Simons-MacBook-Pro datasette-proof-of-concept % sam build --use-container
Starting Build inside a container
Building function 'HelloWorldFunction'
Fetching lambci/lambda:build-python3.8 Docker container image......
Mounting /private/tmp/datasette-proof-of-concept/hello_world as /tmp/samcli/source:ro,delegated inside runtime container
Build Failed
Running PythonPipBuilder:ResolveDependencies
Error: PythonPipBuilder:ResolveDependencies - {uvloop==0.14.0(wheel)}
```
`uvloop` isn't actually necessary for this project, since it's used by `uvicorn` which isn't needed if Lambda is serving ASGI traffic directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645059663,https://api.github.com/repos/simonw/datasette/issues/850,645059663,MDEyOklzc3VlQ29tbWVudDY0NTA1OTY2Mw==,9599,simonw,2020-06-16T23:20:46Z,2020-06-16T23:20:46Z,OWNER,"I added an exclamation mark to hello world and ran `sam deploy` again. https://q7lymja3sj.execute-api.us-east-1.amazonaws.com/Prod/hello/ still shows the old message.
Running `sam build --use-container` first and then `sam deploy` did the right thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645058947,https://api.github.com/repos/simonw/datasette/issues/850,645058947,MDEyOklzc3VlQ29tbWVudDY0NTA1ODk0Nw==,9599,simonw,2020-06-16T23:18:18Z,2020-06-16T23:18:18Z,OWNER,"https://q7lymja3sj.execute-api.us-east-1.amazonaws.com/Prod/hello/
That's a pretty ugly URL. I'm not sure how to get rid of the `/Prod/` prefix on it. Might have to use the `base_url` setting to get something working: https://datasette.readthedocs.io/en/stable/config.html#base-url ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645058617,https://api.github.com/repos/simonw/datasette/issues/850,645058617,MDEyOklzc3VlQ29tbWVudDY0NTA1ODYxNw==,9599,simonw,2020-06-16T23:17:09Z,2020-06-16T23:17:09Z,OWNER,"OK, `sam deploy --guided` now works!
```
simon@Simons-MacBook-Pro datasette-proof-of-concept % sam deploy --guided
Configuring SAM deploy
======================
Looking for samconfig.toml : Not found
Setting default arguments for 'sam deploy'
=========================================
Stack Name [sam-app]: datasette-proof-of-concept
AWS Region [us-east-1]:
#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
Confirm changes before deploy [y/N]:
#SAM needs permission to be able to create roles to connect to the resources in your template
Allow SAM CLI IAM role creation [Y/n]:
HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
Save arguments to samconfig.toml [Y/n]:
Looking for resources needed for deployment: Not found.
Creating the required resources...
Successfully created!
Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-1ksajo4h62s07
A different default S3 bucket can be set in samconfig.toml
Saved arguments to config file
Running 'sam deploy' for future deployments will use the parameters saved above.
The above parameters can be changed by modifying samconfig.toml
Learn more about samconfig.toml syntax at
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html
Deploying with following values
===============================
Stack name : datasette-proof-of-concept
Region : us-east-1
Confirm changeset : False
Deployment s3 bucket : aws-sam-cli-managed-default-samclisourcebucket-1ksajo4h62s07
Capabilities : [""CAPABILITY_IAM""]
Parameter overrides : {}
Initiating deployment
=====================
Uploading to datasette-proof-of-concept/0c208b5656a7aeb6186d49bebc595237 535344 / 535344.0 (100.00%)
HelloWorldFunction may not have authorization defined.
Uploading to datasette-proof-of-concept/14bd9ce3e21f9c88634d13c0c9b377e4.template 1147 / 1147.0 (100.00%)
Waiting for changeset to be created..
CloudFormation stack changeset
---------------------------------------------------------------------------------------------------------------------------------------------------------
Operation LogicalResourceId ResourceType
---------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add HelloWorldFunctionHelloWorldPermissionProd AWS::Lambda::Permission
+ Add HelloWorldFunctionRole AWS::IAM::Role
+ Add HelloWorldFunction AWS::Lambda::Function
+ Add ServerlessRestApiDeployment47fc2d5f9d AWS::ApiGateway::Deployment
+ Add ServerlessRestApiProdStage AWS::ApiGateway::Stage
+ Add ServerlessRestApi AWS::ApiGateway::RestApi
---------------------------------------------------------------------------------------------------------------------------------------------------------
Changeset created successfully. arn:aws:cloudformation:us-east-1:462092780466:changeSet/samcli-deploy1592349262/d685f2de-87c1-4b8e-b13a-67b94f8fc928
2020-06-16 16:14:29 - Waiting for stack create/update to complete
CloudFormation events from changeset
---------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus ResourceType LogicalResourceId ResourceStatusReason
---------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS AWS::IAM::Role HelloWorldFunctionRole -
CREATE_IN_PROGRESS AWS::IAM::Role HelloWorldFunctionRole Resource creation Initiated
CREATE_COMPLETE AWS::IAM::Role HelloWorldFunctionRole -
CREATE_IN_PROGRESS AWS::Lambda::Function HelloWorldFunction Resource creation Initiated
CREATE_IN_PROGRESS AWS::Lambda::Function HelloWorldFunction -
CREATE_COMPLETE AWS::Lambda::Function HelloWorldFunction -
CREATE_IN_PROGRESS AWS::ApiGateway::RestApi ServerlessRestApi Resource creation Initiated
CREATE_IN_PROGRESS AWS::ApiGateway::RestApi ServerlessRestApi -
CREATE_COMPLETE AWS::ApiGateway::RestApi ServerlessRestApi -
CREATE_IN_PROGRESS AWS::Lambda::Permission HelloWorldFunctionHelloWorldPermissi -
onProd
CREATE_IN_PROGRESS AWS::ApiGateway::Deployment ServerlessRestApiDeployment47fc2d5f9 -
d
CREATE_COMPLETE AWS::ApiGateway::Deployment ServerlessRestApiDeployment47fc2d5f9 -
d
CREATE_IN_PROGRESS AWS::ApiGateway::Deployment ServerlessRestApiDeployment47fc2d5f9 Resource creation Initiated
d
CREATE_IN_PROGRESS AWS::Lambda::Permission HelloWorldFunctionHelloWorldPermissi Resource creation Initiated
onProd
CREATE_IN_PROGRESS AWS::ApiGateway::Stage ServerlessRestApiProdStage -
CREATE_COMPLETE AWS::ApiGateway::Stage ServerlessRestApiProdStage -
CREATE_IN_PROGRESS AWS::ApiGateway::Stage ServerlessRestApiProdStage Resource creation Initiated
CREATE_COMPLETE AWS::Lambda::Permission HelloWorldFunctionHelloWorldPermissi -
onProd
CREATE_COMPLETE AWS::CloudFormation::Stack datasette-proof-of-concept -
---------------------------------------------------------------------------------------------------------------------------------------------------------
CloudFormation outputs from deployed stack
---------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs
---------------------------------------------------------------------------------------------------------------------------------------------------------
Key HelloWorldFunctionIamRole
Description Implicit IAM Role created for Hello World function
Value arn:aws:iam::462092780466:role/datasette-proof-of-concept-HelloWorldFunctionRole-8MIDNIV5ECA6
Key HelloWorldApi
Description API Gateway endpoint URL for Prod stage for Hello World function
Value https://q7lymja3sj.execute-api.us-east-1.amazonaws.com/Prod/hello/
Key HelloWorldFunction
Description Hello World Lambda Function ARN
Value arn:aws:lambda:us-east-1:462092780466:function:datasette-proof-of-concept-HelloWorldFunction-QTF78ZEUDCB
---------------------------------------------------------------------------------------------------------------------------------------------------------
Successfully created/updated stack - datasette-proof-of-concept in us-east-1
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645056636,https://api.github.com/repos/simonw/datasette/issues/850,645056636,MDEyOklzc3VlQ29tbWVudDY0NTA1NjYzNg==,9599,simonw,2020-06-16T23:10:22Z,2020-06-16T23:10:22Z,OWNER,"Clicking that button generated me an access key ID / access key secret pair. Dropping those into `~/.aws/credentials` using this format:
```
[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645055200,https://api.github.com/repos/simonw/datasette/issues/850,645055200,MDEyOklzc3VlQ29tbWVudDY0NTA1NTIwMA==,9599,simonw,2020-06-16T23:05:48Z,2020-06-16T23:05:48Z,OWNER,"Logged in as `simon-administrator` I'm using https://console.aws.amazon.com/iam/home?region=us-east-2#/security_credentials to create credentials:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645054206,https://api.github.com/repos/simonw/datasette/issues/850,645054206,MDEyOklzc3VlQ29tbWVudDY0NTA1NDIwNg==,9599,simonw,2020-06-16T23:02:54Z,2020-06-16T23:04:59Z,OWNER,"I think I need to sign in to the AWS console with this new `simon-administrator` account and create IAM credentials for it.
... for which I needed my root ""account ID"" - a 12 digit number - to use on the IAM login form.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645053923,https://api.github.com/repos/simonw/datasette/issues/850,645053923,MDEyOklzc3VlQ29tbWVudDY0NTA1MzkyMw==,9599,simonw,2020-06-16T23:01:49Z,2020-06-16T23:01:49Z,OWNER,"I used https://console.aws.amazon.com/billing/home?#/account and activated ""IAM user/role access to billing information"" - what a puzzling first step!
I created a new user with AWS console access (which means access to the web UI) called `simon-administrator` and set a password. I created an `Administrators` group with `AdministratorAccess`.
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645051972,https://api.github.com/repos/simonw/datasette/issues/850,645051972,MDEyOklzc3VlQ29tbWVudDY0NTA1MTk3Mg==,9599,simonw,2020-06-16T22:55:04Z,2020-06-16T22:55:04Z,OWNER,"```
simon@Simons-MacBook-Pro datasette-proof-of-concept % sam deploy --guided
Configuring SAM deploy
======================
Looking for samconfig.toml : Not found
Setting default arguments for 'sam deploy'
=========================================
Stack Name [sam-app]: datasette-proof-of-concept
AWS Region [us-east-1]:
#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
Confirm changes before deploy [y/N]: y
#SAM needs permission to be able to create roles to connect to the resources in your template
Allow SAM CLI IAM role creation [Y/n]: y
HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
Save arguments to samconfig.toml [Y/n]: y
Error: Failed to create managed resources: Unable to locate credentials
```
I need to get my AWS credentials sorted. I'm going to follow https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html and https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-set-up-credentials.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645051370,https://api.github.com/repos/simonw/datasette/issues/850,645051370,MDEyOklzc3VlQ29tbWVudDY0NTA1MTM3MA==,9599,simonw,2020-06-16T22:53:05Z,2020-06-16T22:53:05Z,OWNER,"```
simon@Simons-MacBook-Pro datasette-proof-of-concept % sam local invoke
Invoking app.lambda_handler (python3.8)
Fetching lambci/lambda:python3.8 Docker container image....................................................................................................................................................................................................................................
Mounting /private/tmp/datasette-proof-of-concept/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: 4616ab43-6882-1627-e5e3-5a29730d52f9 Version: $LATEST
END RequestId: 4616ab43-6882-1627-e5e3-5a29730d52f9
REPORT RequestId: 4616ab43-6882-1627-e5e3-5a29730d52f9 Init Duration: 140.84 ms Duration: 2.49 ms Billed Duration: 100 ms Memory Size: 128 MBMax Memory Used: 25 MB
{""statusCode"":200,""body"":""{\""message\"": \""hello world\""}""}
simon@Simons-MacBook-Pro datasette-proof-of-concept % sam local invoke
Invoking app.lambda_handler (python3.8)
Fetching lambci/lambda:python3.8 Docker container image......
Mounting /private/tmp/datasette-proof-of-concept/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: 3189df2f-e9c0-1be4-b9ac-f329c5fcd067 Version: $LATEST
END RequestId: 3189df2f-e9c0-1be4-b9ac-f329c5fcd067
REPORT RequestId: 3189df2f-e9c0-1be4-b9ac-f329c5fcd067 Init Duration: 87.22 ms Duration: 2.34 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 25 MB
{""statusCode"":200,""body"":""{\""message\"": \""hello world\""}""}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645050948,https://api.github.com/repos/simonw/datasette/issues/850,645050948,MDEyOklzc3VlQ29tbWVudDY0NTA1MDk0OA==,9599,simonw,2020-06-16T22:51:30Z,2020-06-16T22:52:30Z,OWNER,"```
simon@Simons-MacBook-Pro datasette-proof-of-concept % sam build --use-container
Starting Build inside a container
Building function 'HelloWorldFunction'
Fetching lambci/lambda:build-python3.8 Docker container image..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Mounting /private/tmp/datasette-proof-of-concept/hello_world as /tmp/samcli/source:ro,delegated inside runtime container
Build Succeeded
Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645048062,https://api.github.com/repos/simonw/datasette/issues/850,645048062,MDEyOklzc3VlQ29tbWVudDY0NTA0ODA2Mg==,9599,simonw,2020-06-16T22:41:33Z,2020-06-16T22:41:33Z,OWNER,"```
simon@Simons-MacBook-Pro /tmp % sam init
SAM CLI now collects telemetry to better understand customer needs.
You can OPT OUT and disable telemetry collection by setting the
environment variable SAM_CLI_TELEMETRY=0 in your shell.
Thanks for your help!
Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html
Which template source would you like to use?
1 - AWS Quick Start Templates
2 - Custom Template Location
Choice: 1
Which runtime would you like to use?
1 - nodejs12.x
2 - python3.8
3 - ruby2.7
4 - go1.x
5 - java11
6 - dotnetcore3.1
7 - nodejs10.x
8 - python3.7
9 - python3.6
10 - python2.7
11 - ruby2.5
12 - java8
13 - dotnetcore2.1
Runtime: 2
Project name [sam-app]: datasette-proof-of-concept
Cloning app templates from https://github.com/awslabs/aws-sam-cli-app-templates.git
AWS quick start application templates:
1 - Hello World Example
2 - EventBridge Hello World
3 - EventBridge App from scratch (100+ Event Schemas)
4 - Step Functions Sample App (Stock Trader)
Template selection: 1
-----------------------
Generating application:
-----------------------
Name: datasette-proof-of-concept
Runtime: python3.8
Dependency Manager: pip
Application Template: hello-world
Output Directory: .
Next steps can be found in the README file at ./datasette-proof-of-concept/README.md
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645047703,https://api.github.com/repos/simonw/datasette/issues/850,645047703,MDEyOklzc3VlQ29tbWVudDY0NTA0NzcwMw==,9599,simonw,2020-06-16T22:40:19Z,2020-06-16T22:40:19Z,OWNER,"Installed SAM:
```
brew tap aws/tap
brew install aws-sam-cli
sam --version
SAM CLI, version 0.52.0
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645045055,https://api.github.com/repos/simonw/datasette/issues/850,645045055,MDEyOklzc3VlQ29tbWVudDY0NTA0NTA1NQ==,9599,simonw,2020-06-16T22:31:49Z,2020-06-16T22:31:49Z,OWNER,It looks like SAM - AWS Serverless Application Model - is the currently recommended way to deploy Python apps to Lambda from the command-line: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-hello-world.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645042625,https://api.github.com/repos/simonw/datasette/issues/850,645042625,MDEyOklzc3VlQ29tbWVudDY0NTA0MjYyNQ==,9599,simonw,2020-06-16T22:24:26Z,2020-06-16T22:24:26Z,OWNER,"From https://mangum.io/adapter/
> The AWS Lambda handler `event` and `context` arguments are made available to an ASGI application in the ASGI connection scope.
>
> ```
> scope['aws.event']
> scope['aws.context']
> ```
I can use https://github.com/simonw/datasette-debug-asgi to see that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645041663,https://api.github.com/repos/simonw/datasette/issues/850,645041663,MDEyOklzc3VlQ29tbWVudDY0NTA0MTY2Mw==,9599,simonw,2020-06-16T22:21:44Z,2020-06-16T22:21:44Z,OWNER,"https://github.com/jordaneremieff/mangum looks like the best way to run an ASGI app on Lambda at the moment.
```python
from mangum import Mangum
async def app(scope, receive, send):
await send(
{
""type"": ""http.response.start"",
""status"": 200,
""headers"": [[b""content-type"", b""text/plain; charset=utf-8""]],
}
)
await send({""type"": ""http.response.body"", ""body"": b""Hello, world!""})
handler = Mangum(app)
``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645032643,https://api.github.com/repos/simonw/datasette/issues/850,645032643,MDEyOklzc3VlQ29tbWVudDY0NTAzMjY0Mw==,9599,simonw,2020-06-16T21:57:10Z,2020-06-16T21:57:10Z,OWNER,https://docs.aws.amazon.com/efs/latest/ug/wt1-getting-started.html is an EFS walk-through using the AWS CLI tool instead of clicking around in their web interface.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645031225,https://api.github.com/repos/simonw/datasette/issues/850,645031225,MDEyOklzc3VlQ29tbWVudDY0NTAzMTIyNQ==,9599,simonw,2020-06-16T21:53:25Z,2020-06-16T21:53:25Z,OWNER,"Easier solution to this might be to have two functions - a ""read-only"" one which is allowed to scale as much as it likes, and a ""write-only"" one which can write to the database files but is limited to running a maximum of one Lambda instance. https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/850#issuecomment-645030262,https://api.github.com/repos/simonw/datasette/issues/850,645030262,MDEyOklzc3VlQ29tbWVudDY0NTAzMDI2Mg==,9599,simonw,2020-06-16T21:51:01Z,2020-06-16T21:51:39Z,OWNER,"File locking is interesting here. https://docs.aws.amazon.com/lambda/latest/dg/services-efs.html
> Amazon EFS supports [file locking](https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html#consistency) to prevent corruption if multiple functions try to write to the same file system at the same time. Locking in Amazon EFS follows the NFS v4.1 protocol for advisory locking, and enables your applications to use both whole file and byte range locks.
SQLite can apparently work on NFS v4.1. I think I'd rather set things up so there's only ever one writer - so a Datasette instance could scale reads by running lots more lambda functions but only one function ever writes to a file at a time. Not sure if that's feasible with Lambda though - maybe by adding some additional shared state mechanism like Redis?","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",639993467,Proof of concept for Datasette on AWS Lambda with EFS,
https://github.com/simonw/datasette/issues/690#issuecomment-644987083,https://api.github.com/repos/simonw/datasette/issues/690,644987083,MDEyOklzc3VlQ29tbWVudDY0NDk4NzA4Mw==,9599,simonw,2020-06-16T20:11:35Z,2020-06-16T20:11:35Z,OWNER,"Twitter conversation about drop-down menu solutions that are accessible, fast loading and use minimal JavaScript: https://twitter.com/simonw/status/1272974294545395712
I _really_ like the approach taken by GitHub Primer, which builds on top of HTML `` `` tags: https://primer.style/css/components/dropdown","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/issues/849#issuecomment-644584075,https://api.github.com/repos/simonw/datasette/issues/849,644584075,MDEyOklzc3VlQ29tbWVudDY0NDU4NDA3NQ==,9599,simonw,2020-06-16T07:24:08Z,2020-06-16T07:24:08Z,OWNER,This guide is fantastic - I'll be following it closely: https://github.com/chancancode/branch-rename/blob/main/README.md - in particular the Action to mirror master and main for a while.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/849#issuecomment-644384787,https://api.github.com/repos/simonw/datasette/issues/849,644384787,MDEyOklzc3VlQ29tbWVudDY0NDM4NDc4Nw==,9599,simonw,2020-06-15T20:56:07Z,2020-06-15T20:56:19Z,OWNER,"The big question is how this impacts existing CI configuration. `datasette-psutil` is configured to use Circle CI, what happens if I push a new commit?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/849#issuecomment-644384417,https://api.github.com/repos/simonw/datasette/issues/849,644384417,MDEyOklzc3VlQ29tbWVudDY0NDM4NDQxNw==,9599,simonw,2020-06-15T20:55:23Z,2020-06-15T20:55:23Z,OWNER,"I'm doing https://github.com/simonw/datasette-psutil first.
In my local checkout:
```
git branch -m master main
git push -u origin main
```
(Thanks, https://www.hanselman.com/blog/EasilyRenameYourGitDefaultBranchFromMasterToMain.aspx)
Then in https://github.com/simonw/datasette-psutil/settings/branches I changed the default branch to `main`.
![]()
Links to these docs: https://help.github.com/en/github/administering-a-repository/setting-the-default-branch
That worked! https://github.com/simonw/datasette-psutil
One catch, which I think will impact my most widely used repos the most (like datasette) - linking to a specific file now looks like this:
https://github.com/simonw/datasette-psutil/blob/main/datasette_psutil/__init__.py
The old https://github.com/simonw/datasette-psutil/blob/master/datasette_psutil/__init__.py link is presumably frozen in time?
I've definitely got links spread around the web to my ""most recent version of this code"" that would use the `master` reference, which would need to be updated to `main` instead. Most of those are probably in the Datasette docs and on my blog though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/849#issuecomment-644322234,https://api.github.com/repos/simonw/datasette/issues/849,644322234,MDEyOklzc3VlQ29tbWVudDY0NDMyMjIzNA==,9599,simonw,2020-06-15T19:06:16Z,2020-06-15T19:06:16Z,OWNER,I'll make this change on a few of my other repos first to make sure I haven't missed any tricky edge-cases.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/691#issuecomment-643709037,https://api.github.com/repos/simonw/datasette/issues/691,643709037,MDEyOklzc3VlQ29tbWVudDY0MzcwOTAzNw==,49260,amjith,2020-06-14T02:35:16Z,2020-06-14T02:35:16Z,CONTRIBUTOR,"The server should reload in the `config_dir` mode.
Ref: #848","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",574021194,--reload sould reload server if code in --plugins-dir changes,
https://github.com/simonw/datasette/issues/847#issuecomment-643704730,https://api.github.com/repos/simonw/datasette/issues/847,643704730,MDEyOklzc3VlQ29tbWVudDY0MzcwNDczMA==,9599,simonw,2020-06-14T01:28:34Z,2020-06-14T01:28:34Z,OWNER,"Here's the plugin that adds those custom SQLite functions:
```python
from datasette import hookimpl
from coverage.numbits import register_sqlite_functions
@hookimpl
def prepare_connection(conn):
register_sqlite_functions(conn)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638259643,Take advantage of .coverage being a SQLite database,
https://github.com/simonw/datasette/issues/847#issuecomment-643704565,https://api.github.com/repos/simonw/datasette/issues/847,643704565,MDEyOklzc3VlQ29tbWVudDY0MzcwNDU2NQ==,9599,simonw,2020-06-14T01:26:56Z,2020-06-14T01:26:56Z,OWNER,"On closer inspection, I don't know if there's that much useful stuff you can do with the data from `.coverage` on its own.
Consider the following query against a `.coverage` run against Datasette itself:
```sql
select file_id, context_id, numbits_to_nums(numbits) from line_bits
```
![]()
It looks like this tells me which lines of which files were executed during the test run. But... without the actual source code, I don't think I can calculate the coverage percentage for each file. I don't want to count comment lines or whitespace as untested for example, and I don't know how many lines were in the file.
If I'm right that it's not possible to calculate percentage coverage from just the `.coverage` data then I'll need to do something a bit more involved - maybe parsing the `coverage.xml` report and loading that into my own schema?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638259643,Take advantage of .coverage being a SQLite database,
https://github.com/simonw/datasette/issues/847#issuecomment-643702715,https://api.github.com/repos/simonw/datasette/issues/847,643702715,MDEyOklzc3VlQ29tbWVudDY0MzcwMjcxNQ==,9599,simonw,2020-06-14T01:03:30Z,2020-06-14T01:03:40Z,OWNER,Filed a related issue with some ideas against `coveragepy` here: https://github.com/nedbat/coveragepy/issues/999,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638259643,Take advantage of .coverage being a SQLite database,
https://github.com/simonw/datasette/issues/846#issuecomment-643699583,https://api.github.com/repos/simonw/datasette/issues/846,643699583,MDEyOklzc3VlQ29tbWVudDY0MzY5OTU4Mw==,9599,simonw,2020-06-14T00:26:31Z,2020-06-14T00:26:31Z,OWNER,"That seems to have fixed the problem, at least for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638241779,"""Too many open files"" error running tests",
https://github.com/simonw/datasette/issues/846#issuecomment-643699063,https://api.github.com/repos/simonw/datasette/issues/846,643699063,MDEyOklzc3VlQ29tbWVudDY0MzY5OTA2Mw==,9599,simonw,2020-06-14T00:22:32Z,2020-06-14T00:22:32Z,OWNER,"Idea: `num_sql_threads` (described as ""Number of threads in the thread pool for executing SQLite queries"") defaults to 3 - can I knock that down to 1 in the tests and open less connections as a result?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638241779,"""Too many open files"" error running tests",
https://github.com/simonw/datasette/issues/846#issuecomment-643698790,https://api.github.com/repos/simonw/datasette/issues/846,643698790,MDEyOklzc3VlQ29tbWVudDY0MzY5ODc5MA==,9599,simonw,2020-06-14T00:20:42Z,2020-06-14T00:20:42Z,OWNER,"Released a new plugin, `datasette-psutil`, as a side-effect of this investigation: https://github.com/simonw/datasette-psutil","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638241779,"""Too many open files"" error running tests",
https://github.com/simonw/datasette/issues/846#issuecomment-643685669,https://api.github.com/repos/simonw/datasette/issues/846,643685669,MDEyOklzc3VlQ29tbWVudDY0MzY4NTY2OQ==,9599,simonw,2020-06-13T22:24:22Z,2020-06-13T22:24:22Z,OWNER,"I tried this experiment:
```python
import sqlite3, psutil
def show_things():
conn = sqlite3.connect(""fixtures.db"")
tables = [r[0] for r in conn.execute(""select * from sqlite_master"").fetchall()]
return tables
print(psutil.Process().open_files())
print(show_things())
print(psutil.Process().open_files())
```
To see if the connection would be automatically released when the `conn` variable was garbage collected at the end of the function... and it was correctly released - the two calls to `open_files()` showed that the file did not remain open.
Likewise:
```
In [11]: conn = sqlite3.connect(""fixtures.db"")
In [12]: psutil.Process().open_files()
Out[12]:
[popenfile(path='/Users/simon/.ipython/profile_default/history.sqlite', fd=4),
popenfile(path='/Users/simon/.ipython/profile_default/history.sqlite', fd=5),
popenfile(path='/Users/simon/Dropbox/Development/datasette/fixtures.db', fd=12)]
In [13]: del conn
In [14]: psutil.Process().open_files()
Out[14]:
[popenfile(path='/Users/simon/.ipython/profile_default/history.sqlite', fd=4),
popenfile(path='/Users/simon/.ipython/profile_default/history.sqlite', fd=5)]
```
So presumably there's something about the way my pytest fixtures work that's causing the many different `Datasette()` instances and their underlying SQLite connections that I create not to be cleaned up later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638241779,"""Too many open files"" error running tests",
https://github.com/simonw/datasette/issues/846#issuecomment-643685333,https://api.github.com/repos/simonw/datasette/issues/846,643685333,MDEyOklzc3VlQ29tbWVudDY0MzY4NTMzMw==,9599,simonw,2020-06-13T22:19:38Z,2020-06-13T22:19:38Z,OWNER,That's 91 open files but only 29 unique filenames.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638241779,"""Too many open files"" error running tests",
https://github.com/simonw/datasette/issues/846#issuecomment-643685207,https://api.github.com/repos/simonw/datasette/issues/846,643685207,MDEyOklzc3VlQ29tbWVudDY0MzY4NTIwNw==,9599,simonw,2020-06-13T22:18:01Z,2020-06-13T22:18:01Z,OWNER,"This shows currently open files (after `pip install psutil`):
```
import psutil
psutil.Process().open_files()
```
I ran it inside `pytest -x --pdb` and got this:
```
> /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.7/site-packages/jinja2/utils.py(154)open_if_exists()
-> return open(filename, mode)
(Pdb) import psutil
(Pdb) psutil.Process().open_files()
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp9uhx5d8x/fixtures.db', fd=10),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyfw44ica/fixtures.dot.db', fd=11),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyrg6g48b/fixtures.db', fd=12),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=13),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=14),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=15),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=16),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=17),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=18),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=19),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpng4lg84_/fixtures.db', fd=20),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp9uhx5d8x/fixtures.db', fd=21),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp9uhx5d8x/fixtures.db', fd=22),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp9uhx5d8x/fixtures.db', fd=23),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmph11oalw_/fixtures.db', fd=24),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyfw44ica/fixtures.dot.db', fd=25),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyfw44ica/fixtures.dot.db', fd=26),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyfw44ica/fixtures.dot.db', fd=27),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpiorb2bo9/fixtures.db', fd=28),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyrg6g48b/fixtures.db', fd=29),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyrg6g48b/fixtures.db', fd=30),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyrg6g48b/fixtures.db', fd=31),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprvyj5udv/fixtures.db', fd=32),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpng4lg84_/fixtures.db', fd=33),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpng4lg84_/fixtures.db', fd=34),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpng4lg84_/fixtures.db', fd=35),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpb_l6gmq0/fixtures.db', fd=36),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmph11oalw_/extra database.db', fd=40),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/fixtures.db', fd=41),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpiorb2bo9/fixtures.db', fd=42),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpiorb2bo9/fixtures.db', fd=43),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpiorb2bo9/fixtures.db', fd=44),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmph11oalw_/fixtures.db', fd=45),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmph11oalw_/fixtures.db', fd=52),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/fixtures.db', fd=53),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprvyj5udv/fixtures.db', fd=54),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprvyj5udv/fixtures.db', fd=55),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprvyj5udv/fixtures.db', fd=56),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoveuwqn6/fixtures.db', fd=57),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpb_l6gmq0/fixtures.db', fd=61),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpb_l6gmq0/fixtures.db', fd=62),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpb_l6gmq0/fixtures.db', fd=63),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp_j4h9mrn/fixtures.db', fd=64),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/fixtures.db', fd=65),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/fixtures.db', fd=66),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/extra database.db', fd=67),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/extra database.db', fd=68),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/fixtures.db', fd=69),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/extra database.db', fd=70)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpub3eodj1/fixtures.db', fd=71)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/fixtures.db', fd=72)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo.db', fd=73)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo.db', fd=74)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/fixtures.db', fd=75)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/fixtures.db', fd=76)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo.db', fd=77)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo-bar.db', fd=78)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo-bar.db', fd=79)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo-bar.db', fd=80)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-4/config-dir0/immutable.db', fd=81),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoveuwqn6/fixtures.db', fd=82),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoveuwqn6/fixtures.db', fd=83),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoveuwqn6/fixtures.db', fd=84),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp44w5d5wo/fixtures.db', fd=85),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp_j4h9mrn/fixtures.db', fd=86),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp_j4h9mrn/fixtures.db', fd=87),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp_j4h9mrn/fixtures.db', fd=88),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpvu7h14uy/fixtures.db', fd=89),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-4/config-dir0/demo.db', fd=119),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-4/config-dir0/demo.db', fd=120),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-4/config-dir0/demo.db', fd=121),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0xcnrjag/fixtures.db', fd=122),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpub3eodj1/fixtures.db', fd=123),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpub3eodj1/fixtures.db', fd=124),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpub3eodj1/fixtures.db', fd=125),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfz8go8rk/fixtures.db', fd=126),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp44w5d5wo/fixtures.db', fd=127),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp44w5d5wo/fixtures.db', fd=128),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp44w5d5wo/fixtures.db', fd=129),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/fixtures.db', fd=130)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpvu7h14uy/fixtures.db', fd=131),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpvu7h14uy/fixtures.db', fd=132),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpvo3cobk9/fixtures.db', fd=133),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp2t9txyir/fixtures.db', fd=134),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfz8go8rk/fixtures.db', fd=135),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfz8go8rk/fixtures.db', fd=136),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp7h3skv8b/fixtures.db', fd=137),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/fixtures.db', fd=138),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/fixtures.db', fd=139),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/fixtures.db', fd=140),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/extra database.db', fd=141),
```
So yeah, that's too many open files!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638241779,"""Too many open files"" error running tests",
https://github.com/simonw/datasette/issues/841#issuecomment-643681747,https://api.github.com/repos/simonw/datasette/issues/841,643681747,MDEyOklzc3VlQ29tbWVudDY0MzY4MTc0Nw==,9599,simonw,2020-06-13T21:38:46Z,2020-06-13T21:38:46Z,OWNER,Closing this because I've researched feasibility. I may start a milestone in the future to help me get to 100%.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/pull/844#issuecomment-643681517,https://api.github.com/repos/simonw/datasette/issues/844,643681517,MDEyOklzc3VlQ29tbWVudDY0MzY4MTUxNw==,9599,simonw,2020-06-13T21:36:15Z,2020-06-13T21:36:15Z,OWNER,"OK, this works now: https://codecov.io/gh/simonw/datasette/tree/1210d9f41841bdca450f85a2342cdb0ff339c1b4","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638230433,Action to run tests and upload coverage report,
https://github.com/simonw/datasette/issues/843#issuecomment-643676314,https://api.github.com/repos/simonw/datasette/issues/843,643676314,MDEyOklzc3VlQ29tbWVudDY0MzY3NjMxNA==,9599,simonw,2020-06-13T20:47:37Z,2020-06-13T20:47:37Z,OWNER,I can use this action: https://github.com/codecov/codecov-action,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638229448,Configure codecov.io,
https://github.com/simonw/datasette/issues/843#issuecomment-643676069,https://api.github.com/repos/simonw/datasette/issues/843,643676069,MDEyOklzc3VlQ29tbWVudDY0MzY3NjA2OQ==,9599,simonw,2020-06-13T20:45:29Z,2020-06-13T20:45:29Z,OWNER,I set up https://codecov.io/gh/simonw/datasette/settings and added a `CODECOV_TOKEN` to the GitHub Actions secrets for this repo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638229448,Configure codecov.io,
https://github.com/simonw/datasette/issues/842#issuecomment-643663005,https://api.github.com/repos/simonw/datasette/issues/842,643663005,MDEyOklzc3VlQ29tbWVudDY0MzY2MzAwNQ==,9599,simonw,2020-06-13T18:51:57Z,2020-06-13T18:51:57Z,OWNER,"Two potential designs:
- `_actor_id`, `_request_ip`, `_now_timestamp` - so special reserved parameters
- a SQL function: `update blah set up = special('ip')`
I fee the first would be easier to implement.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/841#issuecomment-643661125,https://api.github.com/repos/simonw/datasette/issues/841,643661125,MDEyOklzc3VlQ29tbWVudDY0MzY2MTEyNQ==,9599,simonw,2020-06-13T18:35:30Z,2020-06-13T18:36:50Z,OWNER,"I ran export CODECOV_TOKEN=""f7935cad..."", then ran this:
```
datasette $ bash <(curl -s https://codecov.io/bash)
_____ _
/ ____| | |
| | ___ __| | ___ ___ _____ __
| | / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| | __/ (_| (_) \ V /
\_____\___/ \__,_|\___|\___\___/ \_/
Bash-20200602-f809a24
x> No CI provider detected.
Testing inside Docker? http://docs.codecov.io/docs/testing-with-docker
Testing with Tox? https://docs.codecov.io/docs/python#section-testing-with-tox
project root: .
--> token set from env
Yaml not found, that's ok! Learn more at http://docs.codecov.io/docs/codecov-yaml
==> Running gcov in . (disable via -X gcov)
==> Searching for coverage reports in:
+ .
-> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
+ ./coverage.xml bytes=139174
==> Appending adjustments
https://docs.codecov.io/docs/fixing-reports
-> No adjustments found
==> Gzipping contents
==> Uploading reports
url: https://codecov.io
query: branch=master&commit=0e49842e227a0f1f69d48108c87d17fe0379e548&build=&build_url=&name=&tag=&slug=simonw%2Fdatasette&service=&flags=&pr=&job=
-> Pinging Codecov
https://codecov.io/upload/v4?package=bash-20200602-f809a24&token=secret&branch=master&commit=0e49842e227a0f1f69d48108c87d17fe0379e548&build=&build_url=&name=&tag=&slug=simonw%2Fdatasette&service=&flags=&pr=&job=
-> Uploading
-> View reports at https://codecov.io/github/simonw/datasette/commit/0e49842e227a0f1f69d48108c87d17fe0379e548
```
But https://codecov.io/github/simonw/datasette/commit/0e49842e227a0f1f69d48108c87d17fe0379e548 is a 404, so it doesn't seem to have worked?
UPDATE: It works now, took about 30 seconds before the report showed up at that URL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/841#issuecomment-643660757,https://api.github.com/repos/simonw/datasette/issues/841,643660757,MDEyOklzc3VlQ29tbWVudDY0MzY2MDc1Nw==,9599,simonw,2020-06-13T18:32:20Z,2020-06-13T18:32:20Z,OWNER,"Looking at options for publishing coverage reports:
* https://github.com/codecov/codecov-action
* https://github.com/coveralls-clients/coveralls-python
I'm going to try https://codecov.io/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/841#issuecomment-643660427,https://api.github.com/repos/simonw/datasette/issues/841,643660427,MDEyOklzc3VlQ29tbWVudDY0MzY2MDQyNw==,9599,simonw,2020-06-13T18:29:30Z,2020-06-13T18:29:36Z,OWNER,"This one looks easy enough to fix:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/841#issuecomment-643658036,https://api.github.com/repos/simonw/datasette/issues/841,643658036,MDEyOklzc3VlQ29tbWVudDY0MzY1ODAzNg==,9599,simonw,2020-06-13T18:08:13Z,2020-06-13T18:08:13Z,OWNER,"From digging through that report it looks like the majority stuff that isn't fully covered is corner-cases... which are the kind of things I really do want the tests to catch.
I'm not entirely ready to commit to 100%, but I'm going to start digging through and seeing how close I can get. If I can get to 98% (I'm on 91% already) I may as well push all the way to 100.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/841#issuecomment-643657287,https://api.github.com/repos/simonw/datasette/issues/841,643657287,MDEyOklzc3VlQ29tbWVudDY0MzY1NzI4Nw==,9599,simonw,2020-06-13T18:01:39Z,2020-06-13T18:01:39Z,OWNER,Added `--cov-report html` and got this report: https://static.simonwillison.net/static/2020/htmlcov-issue-841/index.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/841#issuecomment-643656053,https://api.github.com/repos/simonw/datasette/issues/841,643656053,MDEyOklzc3VlQ29tbWVudDY0MzY1NjA1Mw==,9599,simonw,2020-06-13T17:50:34Z,2020-06-13T17:50:34Z,OWNER,"Added a `.coveragerc` file:
```
[run]
omit = datasette/_version.py, datasette/utils/shutil_backport.py
```
And ran again: `pytest --cov=datasette --cov-config=.coveragerc`
```
Name Stmts Miss Cover
------------------------------------------------------
datasette/__init__.py 3 0 100%
datasette/__main__.py 3 3 0%
datasette/actor_auth_cookie.py 19 3 84%
datasette/app.py 499 27 95%
datasette/cli.py 157 45 71%
datasette/database.py 233 17 93%
datasette/default_permissions.py 39 0 100%
datasette/facets.py 209 24 89%
datasette/filters.py 122 7 94%
datasette/hookspecs.py 19 0 100%
datasette/inspect.py 37 23 38%
datasette/plugins.py 34 6 82%
datasette/publish/__init__.py 0 0 100%
datasette/publish/cloudrun.py 55 2 96%
datasette/publish/common.py 19 1 95%
datasette/publish/heroku.py 95 13 86%
datasette/renderer.py 63 4 94%
datasette/sql_functions.py 4 0 100%
datasette/tracer.py 85 16 81%
datasette/utils/__init__.py 503 31 94%
datasette/utils/asgi.py 253 25 90%
datasette/version.py 4 0 100%
datasette/views/__init__.py 0 0 100%
datasette/views/base.py 288 19 93%
datasette/views/database.py 120 2 98%
datasette/views/index.py 57 2 96%
datasette/views/special.py 72 16 78%
datasette/views/table.py 418 18 96%
------------------------------------------------------
TOTAL 3410 304 91%
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/841#issuecomment-643655108,https://api.github.com/repos/simonw/datasette/issues/841,643655108,MDEyOklzc3VlQ29tbWVudDY0MzY1NTEwOA==,9599,simonw,2020-06-13T17:43:15Z,2020-06-13T17:43:15Z,OWNER,"Using https://pypi.org/project/pytest-cov/ and running `pytest --cov=datasette`:
```
---------- coverage: platform darwin, python 3.7.7-final-0 -----------
Name Stmts Miss Cover
--------------------------------------------------------
datasette/__init__.py 3 0 100%
datasette/__main__.py 3 3 0%
datasette/_version.py 277 152 45%
datasette/actor_auth_cookie.py 19 3 84%
datasette/app.py 499 27 95%
datasette/cli.py 157 45 71%
datasette/database.py 233 17 93%
datasette/default_permissions.py 39 0 100%
datasette/facets.py 209 24 89%
datasette/filters.py 122 7 94%
datasette/hookspecs.py 19 0 100%
datasette/inspect.py 37 23 38%
datasette/plugins.py 34 6 82%
datasette/publish/__init__.py 0 0 100%
datasette/publish/cloudrun.py 55 2 96%
datasette/publish/common.py 19 1 95%
datasette/publish/heroku.py 95 13 86%
datasette/renderer.py 63 4 94%
datasette/sql_functions.py 4 0 100%
datasette/tracer.py 85 16 81%
datasette/utils/__init__.py 503 31 94%
datasette/utils/asgi.py 253 25 90%
datasette/utils/shutil_backport.py 44 40 9%
datasette/version.py 4 0 100%
datasette/views/__init__.py 0 0 100%
datasette/views/base.py 288 19 93%
datasette/views/database.py 120 2 98%
datasette/views/index.py 57 2 96%
datasette/views/special.py 72 16 78%
datasette/views/table.py 418 18 96%
--------------------------------------------------------
TOTAL 3731 496 87%
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/834#issuecomment-643648359,https://api.github.com/repos/simonw/datasette/issues/834,643648359,MDEyOklzc3VlQ29tbWVudDY0MzY0ODM1OQ==,9599,simonw,2020-06-13T16:47:29Z,2020-06-13T16:47:29Z,OWNER,"Implementing this is proving surprisingly tricky, because of the need to be able to optionally `await` the returned value. It's a bit of a fiddle to get this to work within unit tests because they run in non-async functions - due to this cunning `async_to_sync` usage in the test client:
https://github.com/simonw/datasette/blob/b906030235efbdff536405d66078f4868ce0d3bd/tests/fixtures.py#L115-L133
I could switch to using `async def test_*` functions decorated with `@pytest.mark.asyncio` but I'd rather not re-engineer the entire test suite just for this one feature, so I'll try to find another way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/841#issuecomment-643576372,https://api.github.com/repos/simonw/datasette/issues/841,643576372,MDEyOklzc3VlQ29tbWVudDY0MzU3NjM3Mg==,9599,simonw,2020-06-13T06:08:34Z,2020-06-13T06:08:34Z,OWNER,Starlette achieves this. https://github.com/encode/starlette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/834#issuecomment-643510240,https://api.github.com/repos/simonw/datasette/issues/834,643510240,MDEyOklzc3VlQ29tbWVudDY0MzUxMDI0MA==,9599,simonw,2020-06-12T22:40:26Z,2020-06-12T22:40:26Z,OWNER,Another use-case: plugins that need their own database with the correct tables. They can write to the database on startup to create their tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/834#issuecomment-643509358,https://api.github.com/repos/simonw/datasette/issues/834,643509358,MDEyOklzc3VlQ29tbWVudDY0MzUwOTM1OA==,9599,simonw,2020-06-12T22:36:37Z,2020-06-12T22:36:37Z,OWNER,This should be able to optionally return an async function which is then awaited.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/805#issuecomment-643501428,https://api.github.com/repos/simonw/datasette/issues/805,643501428,MDEyOklzc3VlQ29tbWVudDY0MzUwMTQyOA==,9599,simonw,2020-06-12T22:06:08Z,2020-06-12T22:06:08Z,OWNER,"This needs the `startup` hook, see https://github.com/simonw/datasette/issues/834#issuecomment-643501064","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/834#issuecomment-643501064,https://api.github.com/repos/simonw/datasette/issues/834,643501064,MDEyOklzc3VlQ29tbWVudDY0MzUwMTA2NA==,9599,simonw,2020-06-12T22:04:43Z,2020-06-12T22:04:43Z,OWNER,Another use-case for this: I want to use the `--root` option on Glitch but it gives me a 127.0.0.1 URL. Glitch has a `PROJECT_DOMAIN` environment variable which tells me the URL. A `datasette-glitch` plugin could use a `startup` hook to output the correct login URL.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/840#issuecomment-643454625,https://api.github.com/repos/simonw/datasette/issues/840,643454625,MDEyOklzc3VlQ29tbWVudDY0MzQ1NDYyNQ==,9599,simonw,2020-06-12T19:47:38Z,2020-06-12T19:47:53Z,OWNER,"Another problem: what to display in the ""you are logged in as"", since we don't dictate an actor design.
I'm going to use a includes template for this that can easily be over-ridden by administrators or by plugins.
The default will look for the first available of the following keys:
- display
- name
- username
- login
- id","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637966833,Log out mechanism for clearing ds_actor cookie,
https://github.com/simonw/datasette/issues/840#issuecomment-643453128,https://api.github.com/repos/simonw/datasette/issues/840,643453128,MDEyOklzc3VlQ29tbWVudDY0MzQ1MzEyOA==,9599,simonw,2020-06-12T19:43:15Z,2020-06-12T19:43:15Z,OWNER,"I don't like how this often involves a logout link that can be maliciously activated.
I'm going to use a CSRF protected form button styled to look like a link instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637966833,Log out mechanism for clearing ds_actor cookie,
https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643414646,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40,643414646,MDEyOklzc3VlQ29tbWVudDY0MzQxNDY0Ng==,9599,simonw,2020-06-12T18:06:48Z,2020-06-12T18:06:48Z,MEMBER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637899539,Demo deploy is broken,
https://github.com/simonw/sqlite-utils/issues/115#issuecomment-643406939,https://api.github.com/repos/simonw/sqlite-utils/issues/115,643406939,MDEyOklzc3VlQ29tbWVudDY0MzQwNjkzOQ==,9599,simonw,2020-06-12T17:51:11Z,2020-06-12T17:51:11Z,OWNER,https://github.com/simonw/sqlite-utils/blob/03ee97d2258254581bea72842518904fc1cbe60f/tests/test_cli.py#L1112-L1128,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637889964,Ability to execute insert/update statements with the CLI,
https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643393506,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40,643393506,MDEyOklzc3VlQ29tbWVudDY0MzM5MzUwNg==,9599,simonw,2020-06-12T17:21:14Z,2020-06-12T17:21:14Z,MEMBER,"I only install SQLite for this:
https://github.com/dogsheep/github-to-sqlite/blob/c0d54e0260468be38152293df5abd775c068495d/.github/workflows/deploy-demo.yml#L77-L78
I'm going to remove the need to install sqlite3 by making this possible with sqlite-utils: https://github.com/simonw/sqlite-utils/issues/115","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637899539,Demo deploy is broken,
https://github.com/simonw/datasette/issues/838#issuecomment-643083451,https://api.github.com/repos/simonw/datasette/issues/838,643083451,MDEyOklzc3VlQ29tbWVudDY0MzA4MzQ1MQ==,79913,tsibley,2020-06-12T06:04:14Z,2020-06-12T06:04:14Z,NONE,"Hmm, I haven't tried removing `ProxyPassReverse`, but it doesn't touch the HTML, which is the issue I'm seeing. You can read the [documentation here](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse). `ProxyPassReverse` is a standard directive when proxying with Apache. I've used it dozens of times with other applications.
Looking a little more at the code, I think the issue here is that the behaviour of `base_url` makes sense when Datasette is _mounted_ at a path within a larger application, but not when HTTP requests are being _proxied_ to it.
In a _mount_ situation, it is perfectly fine to construct URLs reusing the domain and path from the request. In a _proxy_ situation, it never is, as the domain and path in the request are not the domain and path that the non-proxy client actually needs to use. That is, links which include the Apache → Datasette request origin, `localhost:8001`, instead of the browser → Apache request origin, `example.com`, will be broken.
The tests you pointed to also reflect this in two ways:
1. They strip a leading `http://localhost`, allowing such URLs in the facet links to pass, but inclusion of that in a proxy situation would mean the URL is broken.
2. The test client emits direct ASGI events instead of actual proxied HTTP requests. The headers of these ASGI events don't reflect the way an HTTP proxy works; instead they pass through the original request path which contains `base_url`. This works because Datasette responds to requests equivalently at either `/…` or `/{base_url}/…`, which makes some sense in a _mount_ situation but is unconventional (albeit workable) for a proxied app.
Apps that support being proxied automatically support being mounted, but apps that only support being mounted don't automatically support being proxied.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/806#issuecomment-643010591,https://api.github.com/repos/simonw/datasette/issues/806,643010591,MDEyOklzc3VlQ29tbWVudDY0MzAxMDU5MQ==,9599,simonw,2020-06-12T01:13:06Z,2020-06-12T01:13:06Z,OWNER,Tests are passing again: https://github.com/simonw/datasette/commit/9ae0d483ead93c0832142e5dc85959ae3c8f73ea,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/806#issuecomment-643000948,https://api.github.com/repos/simonw/datasette/issues/806,643000948,MDEyOklzc3VlQ29tbWVudDY0MzAwMDk0OA==,9599,simonw,2020-06-12T00:34:21Z,2020-06-12T00:34:21Z,OWNER,I'm going to add https://github.com/simonw/datasette-auth-tokens and https://github.com/simonw/datasette-permissions-sql to the documentation and release notes in a few places.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/806#issuecomment-642998097,https://api.github.com/repos/simonw/datasette/issues/806,642998097,MDEyOklzc3VlQ29tbWVudDY0Mjk5ODA5Nw==,9599,simonw,2020-06-12T00:26:00Z,2020-06-12T00:26:00Z,OWNER,"OK, I'm ready to ship. Last check of the release notes, then I'll update the news section in the README and release 0.44!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/838#issuecomment-642993277,https://api.github.com/repos/simonw/datasette/issues/838,642993277,MDEyOklzc3VlQ29tbWVudDY0Mjk5MzI3Nw==,9599,simonw,2020-06-12T00:18:26Z,2020-06-12T00:18:50Z,OWNER,"Have you tried this without the `ProxyPassReverse` directive? I'm worried that might be confusing Datasette.
This is the test I used to ensure this feature works - it scrapes all of the links on a bunch of different pages. Could it be missing something here?
https://github.com/simonw/datasette/blob/647c5ff0f3e8140f40d7f41f0874ce4e1f4df65c/tests/test_html.py#L1233-L1274
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/824#issuecomment-642991513,https://api.github.com/repos/simonw/datasette/issues/824,642991513,MDEyOklzc3VlQ29tbWVudDY0Mjk5MTUxMw==,9599,simonw,2020-06-12T00:11:50Z,2020-06-12T00:11:50Z,OWNER,Done: https://github.com/simonw/datasette-auth-tokens and https://pypi.org/project/datasette-auth-tokens/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635108074,Example authentication plugin,
https://github.com/simonw/datasette/issues/833#issuecomment-642958225,https://api.github.com/repos/simonw/datasette/issues/833,642958225,MDEyOklzc3VlQ29tbWVudDY0Mjk1ODIyNQ==,9599,simonw,2020-06-11T22:15:32Z,2020-06-11T22:15:32Z,OWNER,https://github.com/simonw/datasette/blob/29c5ff493ad7918b8fc44ea7920b41530e56dd5d/tests/test_permissions.py#L327-L348,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637253789,/-/metadata and so on should respect view-instance permission,
https://github.com/simonw/datasette/issues/824#issuecomment-642953605,https://api.github.com/repos/simonw/datasette/issues/824,642953605,MDEyOklzc3VlQ29tbWVudDY0Mjk1MzYwNQ==,9599,simonw,2020-06-11T22:02:32Z,2020-06-11T22:02:32Z,OWNER,`datasette-auth-tokens` can be the name. I can get a simple initial version of it running pretty quickly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635108074,Example authentication plugin,
https://github.com/simonw/datasette/issues/824#issuecomment-642952962,https://api.github.com/repos/simonw/datasette/issues/824,642952962,MDEyOklzc3VlQ29tbWVudDY0Mjk1Mjk2Mg==,9599,simonw,2020-06-11T22:01:58Z,2020-06-11T22:01:58Z,OWNER,"Alternative idea: a plugin that handles Bearer token authentication. Uses `metadata.json` with secret plugin values to map an incoming token to an actor dictionary, which can then be mapped to permissions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635108074,Example authentication plugin,
https://github.com/simonw/datasette/issues/824#issuecomment-642951150,https://api.github.com/repos/simonw/datasette/issues/824,642951150,MDEyOklzc3VlQ29tbWVudDY0Mjk1MTE1MA==,9599,simonw,2020-06-11T22:00:17Z,2020-06-11T22:00:17Z,OWNER,"I got this working: https://github.com/simonw/datasette-auth-github/pull/64
Just one problem: it uses the existing `ds_actor` cookie, which means it doesn't actually exercise the `actor_from_request` plugin!
It does use `register_routes` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635108074,Example authentication plugin,
https://github.com/simonw/datasette/issues/220#issuecomment-642944645,https://api.github.com/repos/simonw/datasette/issues/220,642944645,MDEyOklzc3VlQ29tbWVudDY0Mjk0NDY0NQ==,9599,simonw,2020-06-11T21:49:55Z,2020-06-11T21:49:55Z,OWNER,"I'm OK with not implementing this - I've got used to the existing mechanism, and it doesn't frustrate me enough to work on this more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314847571,Investigate syntactic sugar for plugins,
https://github.com/simonw/datasette/issues/832#issuecomment-642907021,https://api.github.com/repos/simonw/datasette/issues/832,642907021,MDEyOklzc3VlQ29tbWVudDY0MjkwNzAyMQ==,9599,simonw,2020-06-11T20:20:35Z,2020-06-11T20:20:35Z,OWNER,"I think the new `.check_permissions()` should be a documented utility that is available to plugins.
Maybe a method on `datasette`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-642906681,https://api.github.com/repos/simonw/datasette/issues/832,642906681,MDEyOklzc3VlQ29tbWVudDY0MjkwNjY4MQ==,9599,simonw,2020-06-11T20:19:47Z,2020-06-11T20:20:02Z,OWNER,"So for the following:
```
await self.check_permissions(request, [
(""view-table"", (database, table)),
(""view-database"", database),
""view-instance"",
])
```
The logic is: if the first test returns `True`, you get access. If it returns `False` you are denied. If it says `None` then move on to the next check in the list and repeat.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/833#issuecomment-642905424,https://api.github.com/repos/simonw/datasette/issues/833,642905424,MDEyOklzc3VlQ29tbWVudDY0MjkwNTQyNA==,9599,simonw,2020-06-11T20:16:41Z,2020-06-11T20:16:41Z,OWNER,I'll add a new test in `test_permissions.py` which locks down an instance and then loops through paths as the anonymous user making sure they aren't accessible.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637253789,/-/metadata and so on should respect view-instance permission,
https://github.com/simonw/datasette/issues/833#issuecomment-642902208,https://api.github.com/repos/simonw/datasette/issues/833,642902208,MDEyOklzc3VlQ29tbWVudDY0MjkwMjIwOA==,9599,simonw,2020-06-11T20:08:57Z,2020-06-11T20:08:57Z,OWNER,"I'm tempted to add a `view-instance` check before routing any URLs, but that wouldn't be compatible with the idea in #832 that having `view-table` should be enough to view a table even if you don't pass `view-instance`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637253789,/-/metadata and so on should respect view-instance permission,
https://github.com/simonw/datasette/issues/833#issuecomment-642874724,https://api.github.com/repos/simonw/datasette/issues/833,642874724,MDEyOklzc3VlQ29tbWVudDY0Mjg3NDcyNA==,9599,simonw,2020-06-11T19:07:49Z,2020-06-11T19:07:49Z,OWNER,A live demo running the `datasette-auth-github` plugin will help demonstrate this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637253789,/-/metadata and so on should respect view-instance permission,
https://github.com/simonw/datasette/issues/801#issuecomment-642870553,https://api.github.com/repos/simonw/datasette/issues/801,642870553,MDEyOklzc3VlQ29tbWVudDY0Mjg3MDU1Mw==,9599,simonw,2020-06-11T18:58:49Z,2020-06-11T18:58:49Z,OWNER,I've implemented this in a plugin instead: https://github.com/simonw/datasette-permissions-sql,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631932926,allow_by_query setting for configuring permissions with a SQL statement,
https://github.com/simonw/datasette/issues/832#issuecomment-642795966,https://api.github.com/repos/simonw/datasette/issues/832,642795966,MDEyOklzc3VlQ29tbWVudDY0Mjc5NTk2Ng==,9599,simonw,2020-06-11T16:37:21Z,2020-06-11T16:37:21Z,OWNER,"How would I document this? Probably in another section on https://datasette.readthedocs.io/en/latest/authentication.html#permissions
But I'd also need to add documentation to the individual views stating what permissions are checked and in what order. I could do that on this page: https://datasette.readthedocs.io/en/latest/pages.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/pull/809#issuecomment-642772344,https://api.github.com/repos/simonw/datasette/issues/809,642772344,MDEyOklzc3VlQ29tbWVudDY0Mjc3MjM0NA==,9599,simonw,2020-06-11T16:01:15Z,2020-06-11T16:01:15Z,OWNER,"```
datasette package fixtures.db --secret woot --branch master
Sending build context to Docker daemon 260.6kB
Step 1/9 : FROM python:3.8
3.8: Pulling from library/python
e9afc4f90ab0: Downloading [=======> ] 7.195MB/50.39MB
989e6b19a265: Downloading [============================> ] 4.475MB/7.812MB
af14b6c2f878: Downloading [===========================> ] 5.422MB/9.996MB
5573c4b30949: Waiting
11a88e764313: Waiting
ee776f0e36af: Waiting
513c90a1afc3: Waiting
df9b9e95bdb9: Waiting
86c9edb54464: Waiting
...
datasette package fixtures.db --secret woot --branch master
docker run -p 8001:8001 a155798bd842
```
This works too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632919570,Publish secrets,
https://github.com/simonw/datasette/pull/809#issuecomment-642754589,https://api.github.com/repos/simonw/datasette/issues/809,642754589,MDEyOklzc3VlQ29tbWVudDY0Mjc1NDU4OQ==,9599,simonw,2020-06-11T15:45:25Z,2020-06-11T15:45:25Z,OWNER," datasette publish cloudrun fixtures.db --service datasette-publish-secret --branch=master
https://datasette-publish-secret-j7hipcg4aq-uw.a.run.app/-/messages","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632919570,Publish secrets,
https://github.com/simonw/datasette/pull/809#issuecomment-642750790,https://api.github.com/repos/simonw/datasette/issues/809,642750790,MDEyOklzc3VlQ29tbWVudDY0Mjc1MDc5MA==,9599,simonw,2020-06-11T15:42:23Z,2020-06-11T15:42:23Z,OWNER," datasette publish heroku fixtures.db -n datasette-publish-secret --branch=master
https://datasette-publish-secret.herokuapp.com/-/messages - Heroku works.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632919570,Publish secrets,
https://github.com/simonw/datasette/pull/809#issuecomment-642745518,https://api.github.com/repos/simonw/datasette/issues/809,642745518,MDEyOklzc3VlQ29tbWVudDY0Mjc0NTUxOA==,9599,simonw,2020-06-11T15:38:51Z,2020-06-11T15:38:51Z,OWNER,The way to manually test this is to publish a database to each provider and then check that the `/-/messages` debug tool works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632919570,Publish secrets,
https://github.com/simonw/datasette/issues/832#issuecomment-642741930,https://api.github.com/repos/simonw/datasette/issues/832,642741930,MDEyOklzc3VlQ29tbWVudDY0Mjc0MTkzMA==,9599,simonw,2020-06-11T15:35:53Z,2020-06-11T15:36:05Z,OWNER,"May the fix here is to implement a `.check_permissions()` method which passes when the first permission passes?
```python
await self.check_permissions(request, [
(""view-table"", (database, table)),
(""view-database"", database),
""view-instance"",
])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/394#issuecomment-642522285,https://api.github.com/repos/simonw/datasette/issues/394,642522285,MDEyOklzc3VlQ29tbWVudDY0MjUyMjI4NQ==,58298410,LVerneyPEReN,2020-06-11T09:15:19Z,2020-06-11T09:15:19Z,NONE,"Hi @wragge,
This looks great, thanks for the share! I refactored it into a self-contained function, binding on a random available TCP port (multi-user context). I am using subprocess API directly since the `%run` magic was leaving defunct process behind :/
```python
import socket
from signal import SIGINT
from subprocess import Popen, PIPE
from IPython.display import display, HTML
from notebook.notebookapp import list_running_servers
def get_free_tcp_port():
""""""
Get a free TCP port.
""""""
tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcp.bind(('', 0))
_, port = tcp.getsockname()
tcp.close()
return port
def datasette(database):
""""""
Run datasette on an SQLite database.
""""""
# Get current running servers
servers = list_running_servers()
# Get the current base url
base_url = next(servers)['base_url']
# Get a free port
port = get_free_tcp_port()
# Create a base url for Datasette suing the proxy path
proxy_url = f'{base_url}proxy/absolute/{port}/'
# Display a link to Datasette
display(HTML(f'View Datasette (Click on the stop button to close the Datasette server)
'))
# Launch Datasette
with Popen(
[
'python', '-m', 'datasette', '--',
database,
'--port', str(port),
'--config', f'base_url:{proxy_url}'
],
stdout=PIPE,
stderr=PIPE,
bufsize=1,
universal_newlines=True
) as p:
print(p.stdout.readline(), end='')
while True:
try:
line = p.stderr.readline()
if not line:
break
print(line, end='')
exit_code = p.poll()
except KeyboardInterrupt:
p.send_signal(SIGINT)
```
Ideally, I'd like some extra magic to notify users when they are leaving the closing the notebook tab and make them terminate the running datasette processes. I'll be looking for it.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",396212021,base_url configuration setting,
https://github.com/simonw/datasette/issues/818#issuecomment-642420375,https://api.github.com/repos/simonw/datasette/issues/818,642420375,MDEyOklzc3VlQ29tbWVudDY0MjQyMDM3NQ==,9599,simonw,2020-06-11T05:40:07Z,2020-06-11T05:40:07Z,OWNER,https://github.com/simonw/datasette-permissions-sql is now released as a 0.1a here: https://pypi.org/project/datasette-permissions-sql/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/832#issuecomment-642412017,https://api.github.com/repos/simonw/datasette/issues/832,642412017,MDEyOklzc3VlQ29tbWVudDY0MjQxMjAxNw==,9599,simonw,2020-06-11T05:13:59Z,2020-06-11T05:13:59Z,OWNER,"Relevant code:
https://github.com/simonw/datasette/blob/ce4958018ede00fbdadf0c37a99889b6901bfb9b/datasette/views/table.py#L267-L272","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/831#issuecomment-642324847,https://api.github.com/repos/simonw/datasette/issues/831,642324847,MDEyOklzc3VlQ29tbWVudDY0MjMyNDg0Nw==,9599,simonw,2020-06-10T23:50:55Z,2020-06-10T23:50:55Z,OWNER,"Actually I'm not sure about this. If `""allow"": null` means ""no-one can do this"", what's the allow block syntax for ""everyone can do this""?
It could be `""allow"": {}` - but that's not intuitive because normally the allow block shows keys that need to match. `{}` suggests to me that no matches are possible.
So I think I'm going to stick with the current mechanism, which is that `""allow"": null` means ""anyone can do this"" and `""allow"": {}` means ""no-one can do this"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636614868,"It would be more intuitive if ""allow"": none meant ""no-one can do this""",
https://github.com/simonw/datasette/issues/818#issuecomment-642231871,https://api.github.com/repos/simonw/datasette/issues/818,642231871,MDEyOklzc3VlQ29tbWVudDY0MjIzMTg3MQ==,9599,simonw,2020-06-10T20:11:50Z,2020-06-10T20:11:50Z,OWNER,"`datasette-permissions-sql`
```yaml
plugins:
datasette-permissions-sql:
view-instance: |-
select count(*) from users where admin = 1 and id = :id
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/818#issuecomment-642230499,https://api.github.com/repos/simonw/datasette/issues/818,642230499,MDEyOklzc3VlQ29tbWVudDY0MjIzMDQ5OQ==,9599,simonw,2020-06-10T20:08:46Z,2020-06-10T20:09:26Z,OWNER,"What's a simple but useful plugin I could release that exercises this hook?
Ideally one which executes permission checks against the database somehow.
I could do a simplest-possible implementation of the idea in #801 (allow-by-query).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/818#issuecomment-642229899,https://api.github.com/repos/simonw/datasette/issues/818,642229899,MDEyOklzc3VlQ29tbWVudDY0MjIyOTg5OQ==,9599,simonw,2020-06-10T20:07:36Z,2020-06-10T20:07:36Z,OWNER,"New policy in 9f236c4 dictates that this should be in Milestone 0.44 after all:
> * **New plugin hooks** should only be shipped if accompanied by a separate release of a non-demo plugin that uses them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/829#issuecomment-642217520,https://api.github.com/repos/simonw/datasette/issues/829,642217520,MDEyOklzc3VlQ29tbWVudDY0MjIxNzUyMA==,9599,simonw,2020-06-10T19:41:35Z,2020-06-10T19:41:35Z,OWNER,"I didn't bother with the alternative epoch - it only shaves off two or three bytes from the cookie.
Documentation for the new `ds_actor` cookie shape is here: https://datasette.readthedocs.io/en/latest/authentication.html#the-ds-actor-cookie","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636426530,Ability to set ds_actor cookie such that it expires,
https://github.com/simonw/datasette/issues/829#issuecomment-642178604,https://api.github.com/repos/simonw/datasette/issues/829,642178604,MDEyOklzc3VlQ29tbWVudDY0MjE3ODYwNA==,9599,simonw,2020-06-10T18:18:36Z,2020-06-10T18:20:19Z,OWNER,"Even shorter: encode an integer that is the difference between that expiry timestamp and a more recent epoch - June 1st 2020 will do.
```
>>> import datetime, calendar
>>> calendar.timegm(datetime.date(2020, 6, 1).timetuple())
1590969600
>>> import baseconv
>>> baseconv.base62.encode(int(time.time() - 1590969600))
'3XST'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636426530,Ability to set ds_actor cookie such that it expires,
https://github.com/simonw/datasette/issues/829#issuecomment-642176180,https://api.github.com/repos/simonw/datasette/issues/829,642176180,MDEyOklzc3VlQ29tbWVudDY0MjE3NjE4MA==,9599,simonw,2020-06-10T18:14:02Z,2020-06-10T18:14:15Z,OWNER,"And the `e` key can be `null`or missing for ""never expires"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636426530,Ability to set ds_actor cookie such that it expires,
https://github.com/simonw/datasette/issues/829#issuecomment-642175892,https://api.github.com/repos/simonw/datasette/issues/829,642175892,MDEyOklzc3VlQ29tbWVudDY0MjE3NTg5Mg==,9599,simonw,2020-06-10T18:13:26Z,2020-06-10T18:13:26Z,OWNER,"I'm going with `expires_at` - except to keep the cookies shorter the key will be called `e` and the actor will go in `a`, like this:
```json
{
""e"": ""1UuHoo"",
""a"": {""id"": ""root""}
}
```
That `e` value is a base64 encoded expiry integer timestamp (again for a shorter cookie) - using https://pypi.org/project/python-baseconv/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636426530,Ability to set ds_actor cookie such that it expires,
https://github.com/simonw/datasette/issues/829#issuecomment-642174272,https://api.github.com/repos/simonw/datasette/issues/829,642174272,MDEyOklzc3VlQ29tbWVudDY0MjE3NDI3Mg==,9599,simonw,2020-06-10T18:10:13Z,2020-06-10T18:10:13Z,OWNER,"Some options:
- Redesign the `ds_actor` cookie to be `{""expires_at"": 1591811250, ""actor"": ...}` - check if it has expired in that default `actor_from_request` hook
- Let plugins set an additional cookie of some sort
- Expect plugins that care about this to set a cookie with a different name and implement their own `actor_from_request` against that","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636426530,Ability to set ds_actor cookie such that it expires,
https://github.com/simonw/datasette/issues/829#issuecomment-642161210,https://api.github.com/repos/simonw/datasette/issues/829,642161210,MDEyOklzc3VlQ29tbWVudDY0MjE2MTIxMA==,9599,simonw,2020-06-10T17:45:58Z,2020-06-10T17:45:58Z,OWNER,"`itsdangerous` has this ability but you specify the max-age when you call unsign: https://itsdangerous.palletsprojects.com/en/1.1.x/timed/
> s.unsign(string, max_age=5)
> Traceback (most recent call last):
> ...
> itsdangerous.exc.SignatureExpired: Signature age 15 > 5 seconds
I currently only decode the `ds_actor` cookie in one place: https://github.com/simonw/datasette/blob/d828abaddec0dce3ec4b4eeddc3a74384e52cf34/datasette/actor_auth_cookie.py#L5-L12
If plugins want to be able to set their own policies on how long the `ds_actor` cookie should remain valid, how do I know to listen to them when decoding the cookie here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636426530,Ability to set ds_actor cookie such that it expires,
https://github.com/simonw/datasette/issues/394#issuecomment-641908346,https://api.github.com/repos/simonw/datasette/issues/394,641908346,MDEyOklzc3VlQ29tbWVudDY0MTkwODM0Ng==,127565,wragge,2020-06-10T10:22:54Z,2020-06-10T10:22:54Z,CONTRIBUTOR,"There's a working demo here: https://github.com/wragge/datasette-test
And if you want something that's more than just proof-of-concept, here's a notebook which does some harvesting from web archives and then displays the results using Datasette: https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/explore_presentations.ipynb","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",396212021,base_url configuration setting,
https://github.com/simonw/datasette/issues/394#issuecomment-641889565,https://api.github.com/repos/simonw/datasette/issues/394,641889565,MDEyOklzc3VlQ29tbWVudDY0MTg4OTU2NQ==,58298410,LVerneyPEReN,2020-06-10T09:49:34Z,2020-06-10T09:49:34Z,NONE,"Hi,
I came across this issue while looking for a way to spawn Datasette as a SQLite files viewer in JupyterLab. I found https://github.com/simonw/jupyterserverproxy-datasette-demo which seems to be the most up to date proof of concept, but it seems to be failing to list the available db (at least in the Binder demo, https://hub.gke.mybinder.org/user/simonw-jupyters--datasette-demo-uw4dmlnn/datasette/, I only have `:memory`).
Does anyone tried to improve on this proof of concept to have a Datasette visualization for SQLite files?
Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",396212021,base_url configuration setting,
https://github.com/simonw/datasette/issues/828#issuecomment-641713087,https://api.github.com/repos/simonw/datasette/issues/828,641713087,MDEyOklzc3VlQ29tbWVudDY0MTcxMzA4Nw==,9599,simonw,2020-06-10T04:28:17Z,2020-06-10T04:28:17Z,OWNER,"Fixed. https://datasette.readthedocs.io/en/latest/changelog.html
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635914822,Horizontal scrollbar on changelog page on mobile,
https://github.com/simonw/datasette/issues/828#issuecomment-641710745,https://api.github.com/repos/simonw/datasette/issues/828,641710745,MDEyOklzc3VlQ29tbWVudDY0MTcxMDc0NQ==,9599,simonw,2020-06-10T04:19:31Z,2020-06-10T04:19:31Z,OWNER,https://docs.readthedocs.io/en/stable/guides/adding-custom-css.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635914822,Horizontal scrollbar on changelog page on mobile,
https://github.com/simonw/datasette/issues/828#issuecomment-641710670,https://api.github.com/repos/simonw/datasette/issues/828,641710670,MDEyOklzc3VlQ29tbWVudDY0MTcxMDY3MA==,9599,simonw,2020-06-10T04:19:17Z,2020-06-10T04:19:17Z,OWNER,"This CSS seems to fix it:
```css
a.external {overflow-wrap: anywhere;}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635914822,Horizontal scrollbar on changelog page on mobile,
https://github.com/simonw/datasette/issues/806#issuecomment-641637696,https://api.github.com/repos/simonw/datasette/issues/806,641637696,MDEyOklzc3VlQ29tbWVudDY0MTYzNzY5Ng==,9599,simonw,2020-06-09T23:46:00Z,2020-06-09T23:46:00Z,OWNER,"The issues that should be referenced from this release are:
#395, #519, #576, #699, #706, #774, #777, #781, #784, #788, #790, #797, #798, #800, #802, #804, #819, #822 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/806#issuecomment-641634749,https://api.github.com/repos/simonw/datasette/issues/806,641634749,MDEyOklzc3VlQ29tbWVudDY0MTYzNDc0OQ==,9599,simonw,2020-06-09T23:34:52Z,2020-06-09T23:34:52Z,OWNER,Preview of the release notes is now available here: https://datasette.readthedocs.io/en/latest/changelog.html#v0-44,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/795#issuecomment-641616185,https://api.github.com/repos/simonw/datasette/issues/795,641616185,MDEyOklzc3VlQ29tbWVudDY0MTYxNjE4NQ==,9599,simonw,2020-06-09T22:33:33Z,2020-06-09T22:33:33Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/internals.html#setting-cookies-with-response-set-cookie,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629541395,response.set_cookie() method,
https://github.com/simonw/datasette/issues/826#issuecomment-641616060,https://api.github.com/repos/simonw/datasette/issues/826,641616060,MDEyOklzc3VlQ29tbWVudDY0MTYxNjA2MA==,9599,simonw,2020-06-09T22:33:12Z,2020-06-09T22:33:12Z,OWNER,https://datasette.readthedocs.io/en/latest/authentication.html#the-ds-actor-cookie,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635519358,Document the ds_actor signed cookie,
https://github.com/simonw/datasette/issues/806#issuecomment-641604210,https://api.github.com/repos/simonw/datasette/issues/806,641604210,MDEyOklzc3VlQ29tbWVudDY0MTYwNDIxMA==,9599,simonw,2020-06-09T21:59:33Z,2020-06-09T22:00:11Z,OWNER,"AWS IAM uses action and resource terminology: https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction_access-management.html - I think that's where I got that language:
> ```json
> {
> ""Version"": ""2012-10-17"",
> ""Statement"": {
> ""Effect"": ""Allow"",
> ""Action"": ""dynamodb:*"",
> ""Resource"": ""arn:aws:dynamodb:us-east-2:123456789012:table/Books""
> }
> }
> ```
I'm going to stick with ""action"" in its current meaning.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/806#issuecomment-641603457,https://api.github.com/repos/simonw/datasette/issues/806,641603457,MDEyOklzc3VlQ29tbWVudDY0MTYwMzQ1Nw==,9599,simonw,2020-06-09T21:57:32Z,2020-06-09T21:57:32Z,OWNER,"operation, procedure, process as alternative words for those menu items?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/806#issuecomment-641602794,https://api.github.com/repos/simonw/datasette/issues/806,641602794,MDEyOklzc3VlQ29tbWVudDY0MTYwMjc5NA==,9599,simonw,2020-06-09T21:55:45Z,2020-06-09T21:55:45Z,OWNER,"Last-minute thought:
Should I worry about calling permissions ""actions"", when I have an idea for a future plugin hook that allows plugins to add something I was going to call ""actions"" to database, table and row pages?
Those actions would take the form of menu item commands that Do Something to the selected object. If I use ""actions"" to mean permission names, will I be able to find a good alternative name for these dynamic menu items?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/804#issuecomment-641538982,https://api.github.com/repos/simonw/datasette/issues/804,641538982,MDEyOklzc3VlQ29tbWVudDY0MTUzODk4Mg==,9599,simonw,2020-06-09T20:01:30Z,2020-06-09T20:01:30Z,OWNER,Now fully documented here: https://datasette.readthedocs.io/en/latest/contributing.html#setting-up-a-development-environment,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632673972,python tests/fixtures.py command has a bug,
https://github.com/simonw/datasette/issues/804#issuecomment-641538799,https://api.github.com/repos/simonw/datasette/issues/804,641538799,MDEyOklzc3VlQ29tbWVudDY0MTUzODc5OQ==,9599,simonw,2020-06-09T20:01:08Z,2020-06-09T20:01:08Z,OWNER," $ python tests/fixtures.py fixtures.db fixtures-metadata.json fixtures-plugins
Test tables written to fixtures.db
- metadata written to fixtures-metadata.json
Wrote plugin: fixtures-plugins/register_output_renderer.py
Wrote plugin: fixtures-plugins/view_name.py
Wrote plugin: fixtures-plugins/my_plugin.py
Wrote plugin: fixtures-plugins/messages_output_renderer.py
Wrote plugin: fixtures-plugins/my_plugin_2.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632673972,python tests/fixtures.py command has a bug,
https://github.com/simonw/datasette/issues/804#issuecomment-641528737,https://api.github.com/repos/simonw/datasette/issues/804,641528737,MDEyOklzc3VlQ29tbWVudDY0MTUyODczNw==,9599,simonw,2020-06-09T19:39:24Z,2020-06-09T19:39:24Z,OWNER,Switched to 0.44 milestone because I don't like shipping releases with known bugs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632673972,python tests/fixtures.py command has a bug,
https://github.com/simonw/datasette/issues/827#issuecomment-641528269,https://api.github.com/repos/simonw/datasette/issues/827,641528269,MDEyOklzc3VlQ29tbWVudDY0MTUyODI2OQ==,9599,simonw,2020-06-09T19:38:30Z,2020-06-09T19:38:30Z,OWNER,https://datasette.readthedocs.io/en/latest/internals.html#csrf-protection,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635696400,Document CSRF protection (for plugins),
https://github.com/simonw/datasette/issues/825#issuecomment-641406944,https://api.github.com/repos/simonw/datasette/issues/825,641406944,MDEyOklzc3VlQ29tbWVudDY0MTQwNjk0NA==,9599,simonw,2020-06-09T16:12:02Z,2020-06-09T17:19:19Z,OWNER,"Alternative design: leave actor alone. Instead specify that allow blocks can look like this:
```json
{
""allow"": {
""unauthenticated"": true
}
}
```
I like this: the above block is very self-documenting.
The `""id"": ""*""` mechanism means there is already precedent for allow keys with special meaning.
**I'm going with this design.**","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635147716,Way to enable a default=False permission for anonymous users,
https://github.com/simonw/datasette/issues/825#issuecomment-641452563,https://api.github.com/repos/simonw/datasette/issues/825,641452563,MDEyOklzc3VlQ29tbWVudDY0MTQ1MjU2Mw==,9599,simonw,2020-06-09T17:08:00Z,2020-06-09T17:08:00Z,OWNER,https://datasette.readthedocs.io/en/latest/authentication.html#defining-permissions-with-allow-blocks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635147716,Way to enable a default=False permission for anonymous users,
https://github.com/simonw/datasette/issues/825#issuecomment-641449725,https://api.github.com/repos/simonw/datasette/issues/825,641449725,MDEyOklzc3VlQ29tbWVudDY0MTQ0OTcyNQ==,9599,simonw,2020-06-09T17:02:31Z,2020-06-09T17:02:31Z,OWNER,Documented at the bottom of this section: https://github.com/simonw/datasette/blob/7633b9ab249b2dce5ee0b4fcf9542c13a1703ef0/docs/authentication.rst#defining-permissions-with-allow-blocks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635147716,Way to enable a default=False permission for anonymous users,
https://github.com/simonw/datasette/issues/825#issuecomment-641412424,https://api.github.com/repos/simonw/datasette/issues/825,641412424,MDEyOklzc3VlQ29tbWVudDY0MTQxMjQyNA==,9599,simonw,2020-06-09T16:22:07Z,2020-06-09T16:22:07Z,OWNER,"When I implement this I should also document default allow vs default deny as a concept, and specify that default next to every documented permission.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635147716,Way to enable a default=False permission for anonymous users,
https://github.com/simonw/datasette/issues/795#issuecomment-641361311,https://api.github.com/repos/simonw/datasette/issues/795,641361311,MDEyOklzc3VlQ29tbWVudDY0MTM2MTMxMQ==,9599,simonw,2020-06-09T15:11:50Z,2020-06-09T15:11:50Z,OWNER,Also: https://github.com/simonw/datasette/blob/dfff34e1987976e72f58ee7b274952840b1f4b71/datasette/views/special.py#L63-L76,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629541395,response.set_cookie() method,
https://github.com/simonw/datasette/issues/826#issuecomment-641360187,https://api.github.com/repos/simonw/datasette/issues/826,641360187,MDEyOklzc3VlQ29tbWVudDY0MTM2MDE4Nw==,9599,simonw,2020-06-09T15:10:00Z,2020-06-09T15:11:24Z,OWNER,Also a good reminder that I need a `set_cookie()` function (#795) so I don't have to mess around with `SimpleCookie` directly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635519358,Document the ds_actor signed cookie,
https://github.com/simonw/datasette/issues/826#issuecomment-641359103,https://api.github.com/repos/simonw/datasette/issues/826,641359103,MDEyOklzc3VlQ29tbWVudDY0MTM1OTEwMw==,9599,simonw,2020-06-09T15:08:07Z,2020-06-09T15:10:33Z,OWNER,"I should probably add a utility function for setting that cookie - right now the only code that does that is here:
https://github.com/simonw/datasette/blob/dfff34e1987976e72f58ee7b274952840b1f4b71/datasette/views/special.py#L63-L76","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635519358,Document the ds_actor signed cookie,
https://github.com/simonw/datasette/issues/812#issuecomment-641353729,https://api.github.com/repos/simonw/datasette/issues/812,641353729,MDEyOklzc3VlQ29tbWVudDY0MTM1MzcyOQ==,9599,simonw,2020-06-09T14:59:25Z,2020-06-09T14:59:25Z,OWNER,I'm going to figure this out by working with https://github.com/simonw/datasette-auth-github/issues/62,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634112607,Ability to customize what happens when a view permission fails,
https://github.com/simonw/datasette/issues/823#issuecomment-641353186,https://api.github.com/repos/simonw/datasette/issues/823,641353186,MDEyOklzc3VlQ29tbWVudDY0MTM1MzE4Ng==,9599,simonw,2020-06-09T14:58:36Z,2020-06-09T14:58:36Z,OWNER,"Docs now say:
> The actor dictionary can be any shape - the design of that data structure is left up to the plugins. A useful convention is to include an `""id""` string, as demonstrated by the ""root"" actor below.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635107393,"Documentation is inconsistent about ""id"" as required field on actor",
https://github.com/simonw/datasette/issues/825#issuecomment-641320947,https://api.github.com/repos/simonw/datasette/issues/825,641320947,MDEyOklzc3VlQ29tbWVudDY0MTMyMDk0Nw==,9599,simonw,2020-06-09T14:06:46Z,2020-06-09T14:06:46Z,OWNER,"I'm torn between `anonymous` and `anon` - because the latter is less typing, and I envisage people writing a lot of code like this:
```python
if actor.get(""anonymous""):
# ...
```
I'm going with `anonymous` because it's that tiny bit clearer than `anon`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635147716,Way to enable a default=False permission for anonymous users,
https://github.com/simonw/datasette/issues/825#issuecomment-641062164,https://api.github.com/repos/simonw/datasette/issues/825,641062164,MDEyOklzc3VlQ29tbWVudDY0MTA2MjE2NA==,9599,simonw,2020-06-09T06:30:24Z,2020-06-09T14:05:33Z,OWNER,"Idea: the anonymous actor could be passed to `actor_matches_allow()` as:
```json
{""anonymous"": true}
```
Then allow blocks like this could be used to allow them:
```json
{
""plugins"": {
""datasette-upload-csvs"": {
""allow"": {
""anonymous"": true
}
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635147716,Way to enable a default=False permission for anonymous users,
https://github.com/simonw/datasette/issues/823#issuecomment-641059221,https://api.github.com/repos/simonw/datasette/issues/823,641059221,MDEyOklzc3VlQ29tbWVudDY0MTA1OTIyMQ==,9599,simonw,2020-06-09T06:23:51Z,2020-06-09T06:24:09Z,OWNER,"I don't like the ""id"" requirement.
I can think of plenty of situations where a unique ID might not be available:
- auth against an external token - an email address or a phone number for example
- auth using encrypted tokens - where decrypting the token tells you exactly what permissions that token should have, like in https://blog.thea.codes/building-a-stateless-api-proxy/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635107393,"Documentation is inconsistent about ""id"" as required field on actor",
https://github.com/simonw/datasette/issues/806#issuecomment-641026726,https://api.github.com/repos/simonw/datasette/issues/806,641026726,MDEyOklzc3VlQ29tbWVudDY0MTAyNjcyNg==,9599,simonw,2020-06-09T04:52:07Z,2020-06-09T04:52:07Z,OWNER,Changelog for this is going to be huge - 96 commits since 0.43 already! https://github.com/simonw/datasette/compare/0.43...master,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/818#issuecomment-641026230,https://api.github.com/repos/simonw/datasette/issues/818,641026230,MDEyOklzc3VlQ29tbWVudDY0MTAyNjIzMA==,9599,simonw,2020-06-09T04:50:24Z,2020-06-09T04:50:24Z,OWNER,I'm dropping this from the 0.44 milestone.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/823#issuecomment-641025760,https://api.github.com/repos/simonw/datasette/issues/823,641025760,MDEyOklzc3VlQ29tbWVudDY0MTAyNTc2MA==,9599,simonw,2020-06-09T04:48:40Z,2020-06-09T04:48:40Z,OWNER,"I should assert that `""id""` exists and is a string in the code that calls the `actor_from_request` hook.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635107393,"Documentation is inconsistent about ""id"" as required field on actor",
https://github.com/simonw/datasette/issues/805#issuecomment-641017851,https://api.github.com/repos/simonw/datasette/issues/805,641017851,MDEyOklzc3VlQ29tbWVudDY0MTAxNzg1MQ==,9599,simonw,2020-06-09T04:17:00Z,2020-06-09T04:17:00Z,OWNER,I can't get Datasette working on Glitch installed from a URL - I'm going to try this on Glitch once I've shipped the 0.44 release in #806.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/805#issuecomment-641017721,https://api.github.com/repos/simonw/datasette/issues/805,641017721,MDEyOklzc3VlQ29tbWVudDY0MTAxNzcyMQ==,9599,simonw,2020-06-09T04:16:28Z,2020-06-09T04:16:28Z,OWNER,"Create `data.db` with:
```
echo '{""emoji"": ""🐯"", ""score"": 0}' | sqlite-utils insert data.db emojis --pk=emoji -
echo '{""emoji"": ""🐺"", ""score"": 0}' | sqlite-utils insert data.db emojis --pk=emoji -
```
Then run Datasette with this `metadata.yaml`:
```yaml
title: Datasette Poll
databases:
data:
queries:
vote:
sql: |-
update emojis set score = score + 1 where emoji = :emoji
write: true
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/797#issuecomment-638301073,https://api.github.com/repos/simonw/datasette/issues/797,638301073,MDEyOklzc3VlQ29tbWVudDYzODMwMTA3Mw==,9599,simonw,2020-06-03T16:14:54Z,2020-06-09T04:00:40Z,OWNER,I want a unit test that exercises this for both writable and regular canned queries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",630120235,"Documentation for new ""params"" setting for canned queries",
https://github.com/simonw/datasette/issues/818#issuecomment-641013524,https://api.github.com/repos/simonw/datasette/issues/818,641013524,MDEyOklzc3VlQ29tbWVudDY0MTAxMzUyNA==,9599,simonw,2020-06-09T03:57:38Z,2020-06-09T04:00:24Z,OWNER,"Problem with that is it's more of a `actor_from_request` opportunity than `permission_allowed`. You could use `actor_from_request` to authenticate API clients from their `Authorization:` header, then use the regular `""allow""` blocks in `metadata.json` to actually assign their permissions.
The most interesting permissions plugin would be one that implements permissions against some kind of database schema, hence allowing admins to edit permissions through writable canned queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/818#issuecomment-641009744,https://api.github.com/repos/simonw/datasette/issues/818,641009744,MDEyOklzc3VlQ29tbWVudDY0MTAwOTc0NA==,9599,simonw,2020-06-09T03:43:18Z,2020-06-09T03:43:18Z,OWNER,`datasette-auth-bearer` perhaps?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/818#issuecomment-641009442,https://api.github.com/repos/simonw/datasette/issues/818,641009442,MDEyOklzc3VlQ29tbWVudDY0MTAwOTQ0Mg==,9599,simonw,2020-06-09T03:41:55Z,2020-06-09T03:41:55Z,OWNER,I want to build a plugin that does `Authorization: Bearer xxx` API key authentication.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/822#issuecomment-641003291,https://api.github.com/repos/simonw/datasette/issues/822,641003291,MDEyOklzc3VlQ29tbWVudDY0MTAwMzI5MQ==,9599,simonw,2020-06-09T03:17:43Z,2020-06-09T03:17:43Z,OWNER,I'm leaning towards `request.url_vars`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635077656,request.url_vars helper property,
https://github.com/simonw/datasette/issues/822#issuecomment-641003237,https://api.github.com/repos/simonw/datasette/issues/822,641003237,MDEyOklzc3VlQ29tbWVudDY0MTAwMzIzNw==,9599,simonw,2020-06-09T03:17:32Z,2020-06-09T03:17:32Z,OWNER,Currently querystring parameters are accessed through `request.args` and POST variables through `request.post_vars()`. Would be good to have a name that was somewhat consistent with those.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635077656,request.url_vars helper property,
https://github.com/simonw/datasette/issues/215#issuecomment-641002504,https://api.github.com/repos/simonw/datasette/issues/215,641002504,MDEyOklzc3VlQ29tbWVudDY0MTAwMjUwNA==,9599,simonw,2020-06-09T03:14:32Z,2020-06-09T03:14:32Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/plugins.html#register-routes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/820#issuecomment-640982533,https://api.github.com/repos/simonw/datasette/issues/820,640982533,MDEyOklzc3VlQ29tbWVudDY0MDk4MjUzMw==,9599,simonw,2020-06-09T02:00:21Z,2020-06-09T02:00:21Z,OWNER,In the case of registering API tokens it would be useful if the plugin could call a writable canned query which knows how to insert a randomly generated value. This could be achieved using a custom SQL function.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635049296,Idea: Plugin hook for registering canned queries,
https://github.com/simonw/datasette/issues/215#issuecomment-640972952,https://api.github.com/repos/simonw/datasette/issues/215,640972952,MDEyOklzc3VlQ29tbWVudDY0MDk3Mjk1Mg==,9599,simonw,2020-06-09T01:24:52Z,2020-06-09T01:25:33Z,OWNER,WIP documentation: https://github.com/simonw/datasette/blob/770dedb21adfc706592e6b5cdf5e751a8720fdf9/docs/plugins.rst#register_routes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640971470,https://api.github.com/repos/simonw/datasette/issues/215,640971470,MDEyOklzc3VlQ29tbWVudDY0MDk3MTQ3MA==,9599,simonw,2020-06-09T01:19:44Z,2020-06-09T01:19:44Z,OWNER,I'll need to add documentation of the `Response` object (and `Response.html()` and `Response.text()` class methods - I should add `Response.json()` too) to the internals page https://datasette.readthedocs.io/en/stable/internals.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640960667,https://api.github.com/repos/simonw/datasette/issues/215,640960667,MDEyOklzc3VlQ29tbWVudDY0MDk2MDY2Nw==,9599,simonw,2020-06-09T00:41:35Z,2020-06-09T00:41:35Z,OWNER,I'm going to implement this one documentation-first in a pull request.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640960553,https://api.github.com/repos/simonw/datasette/issues/215,640960553,MDEyOklzc3VlQ29tbWVudDY0MDk2MDU1Mw==,9599,simonw,2020-06-09T00:41:09Z,2020-06-09T00:41:09Z,OWNER,"I'm going to imitate `register_output_renderer` and `register_facet_classes` - both return a list of things to register.
So I'll do this:
```python
@hookspec
def register_routes():
""Register URL routes. Return a list of (regex, view_function) pairs""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/777#issuecomment-640957423,https://api.github.com/repos/simonw/datasette/issues/777,640957423,MDEyOklzc3VlQ29tbWVudDY0MDk1NzQyMw==,9599,simonw,2020-06-09T00:29:03Z,2020-06-09T00:29:03Z,OWNER,"Here's why:
https://github.com/simonw/datasette/blob/49d6d2f7b0f6cb02e25022e1c9403811f1fa0a7c/datasette/app.py#L1024-L1029
404 errors are rendered by looking for a template from `[""404.html"", ""500.html""]`.
`404.html` doesn't actually ship with Datasette (plugins or custom template directories can provide it). So the `500.html` template is used.
That template extends `base.html`, which expects there to be `base_url` and `app_css_hash` variables. But as you can see in the excerpt above, those variables are not being passed to the template context when the error page is rendered.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626171242,Error pages not correctly loading CSS,
https://github.com/simonw/datasette/issues/777#issuecomment-640955788,https://api.github.com/repos/simonw/datasette/issues/777,640955788,MDEyOklzc3VlQ29tbWVudDY0MDk1NTc4OA==,9599,simonw,2020-06-09T00:23:26Z,2020-06-09T00:23:57Z,OWNER,"Clue: https://latest.datasette.io/404 displays correctly but https://latest.datasette.io/fixtures/404 does not.
That's because `` does the correct thing if you are on the root of the site but not if you are in a sub-directory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626171242,Error pages not correctly loading CSS,
https://github.com/simonw/datasette/issues/813#issuecomment-640951947,https://api.github.com/repos/simonw/datasette/issues/813,640951947,MDEyOklzc3VlQ29tbWVudDY0MDk1MTk0Nw==,9599,simonw,2020-06-09T00:09:56Z,2020-06-09T00:09:56Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/authentication.html#controlling-the-ability-to-execute-arbitrary-sql,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634139848,Mechanism for specifying allow_sql permission in metadata.json,
https://github.com/simonw/datasette/issues/818#issuecomment-640929693,https://api.github.com/repos/simonw/datasette/issues/818,640929693,MDEyOklzc3VlQ29tbWVudDY0MDkyOTY5Mw==,9599,simonw,2020-06-08T22:56:38Z,2020-06-08T22:56:38Z,OWNER,https://datasette.readthedocs.io/en/latest/plugins.html#permission-allowed-datasette-actor-action-resource has a couple of examples now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/777#issuecomment-640925018,https://api.github.com/repos/simonw/datasette/issues/777,640925018,MDEyOklzc3VlQ29tbWVudDY0MDkyNTAxOA==,9599,simonw,2020-06-08T22:41:42Z,2020-06-08T22:41:42Z,OWNER,This is particularly worth fixing now that 403 forbidden pages are much more likely due to #811.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626171242,Error pages not correctly loading CSS,
https://github.com/simonw/datasette/issues/493#issuecomment-640924558,https://api.github.com/repos/simonw/datasette/issues/493,640924558,MDEyOklzc3VlQ29tbWVudDY0MDkyNDU1OA==,9599,simonw,2020-06-08T22:40:01Z,2020-06-08T22:40:01Z,OWNER,I'll also rename `--config` to `--setting`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",449886319,Rename metadata.json to config.json,
https://github.com/simonw/datasette/issues/493#issuecomment-640924482,https://api.github.com/repos/simonw/datasette/issues/493,640924482,MDEyOklzc3VlQ29tbWVudDY0MDkyNDQ4Mg==,9599,simonw,2020-06-08T22:39:45Z,2020-06-08T22:39:45Z,OWNER,"I'm definitely doing this rename, now that `metadata.json` is used for `allow` permissions configuration as well as-of #811.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",449886319,Rename metadata.json to config.json,
https://github.com/simonw/datasette/issues/806#issuecomment-640916991,https://api.github.com/repos/simonw/datasette/issues/806,640916991,MDEyOklzc3VlQ29tbWVudDY0MDkxNjk5MQ==,9599,simonw,2020-06-08T22:18:45Z,2020-06-08T22:18:45Z,OWNER,Reminder for release notes: I removed `--config allow_sql:0` - see https://github.com/simonw/datasette/issues/813#issuecomment-640916807,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/813#issuecomment-640916807,https://api.github.com/repos/simonw/datasette/issues/813,640916807,MDEyOklzc3VlQ29tbWVudDY0MDkxNjgwNw==,9599,simonw,2020-06-08T22:18:09Z,2020-06-08T22:18:09Z,OWNER,"I could retire the `--config allow_sql:0` option entirely, since the new `metadata.json` mechanism can be used to achieve the exact same thing.
I'm going to do that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634139848,Mechanism for specifying allow_sql permission in metadata.json,
https://github.com/simonw/datasette/issues/813#issuecomment-640916290,https://api.github.com/repos/simonw/datasette/issues/813,640916290,MDEyOklzc3VlQ29tbWVudDY0MDkxNjI5MA==,9599,simonw,2020-06-08T22:16:39Z,2020-06-08T22:17:32Z,OWNER,"Naming problem: Datasette already has a config option with this name:
$ datasette serve data.db --config allow_sql:1
https://datasette.readthedocs.io/en/stable/config.html#allow-sql
It's confusing to have two things called `allow_sql` that do slightly different things.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634139848,Mechanism for specifying allow_sql permission in metadata.json,
https://github.com/simonw/datasette/issues/801#issuecomment-640905609,https://api.github.com/repos/simonw/datasette/issues/801,640905609,MDEyOklzc3VlQ29tbWVudDY0MDkwNTYwOQ==,9599,simonw,2020-06-08T21:48:44Z,2020-06-08T21:48:44Z,OWNER,"Dropping this out of Datasette 0.44 again - I have enough other stuff to finish, this can wait.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631932926,allow_by_query setting for configuring permissions with a SQL statement,
https://github.com/simonw/datasette/issues/813#issuecomment-640837908,https://api.github.com/repos/simonw/datasette/issues/813,640837908,MDEyOklzc3VlQ29tbWVudDY0MDgzNzkwOA==,9599,simonw,2020-06-08T19:33:03Z,2020-06-08T19:33:03Z,OWNER,Don't forget to link to the `allow_sql` docs from the warning block here: https://github.com/simonw/datasette/blob/54370853828bdf87ca844fd0fc00900e0e2e659d/docs/authentication.rst#controlling-access-to-specific-tables-and-views,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634139848,Mechanism for specifying allow_sql permission in metadata.json,
https://github.com/simonw/datasette/issues/813#issuecomment-640831842,https://api.github.com/repos/simonw/datasette/issues/813,640831842,MDEyOklzc3VlQ29tbWVudDY0MDgzMTg0Mg==,9599,simonw,2020-06-08T19:27:47Z,2020-06-08T19:27:47Z,OWNER,"This needs to be ready for Datasette 0.44 because without it the ""view-table"" permission is useless - it will protect the https://latest.datasette.io/fixtures/facetable page but will not prevent users from executing https://latest.datasette.io/fixtures?sql=select+*+from+facetable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634139848,Mechanism for specifying allow_sql permission in metadata.json,
https://github.com/simonw/datasette/issues/813#issuecomment-640830088,https://api.github.com/repos/simonw/datasette/issues/813,640830088,MDEyOklzc3VlQ29tbWVudDY0MDgzMDA4OA==,9599,simonw,2020-06-08T19:26:15Z,2020-06-08T19:26:15Z,OWNER,This needs to affect the `?_where=` parameter on table pages as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634139848,Mechanism for specifying allow_sql permission in metadata.json,
https://github.com/simonw/datasette/issues/816#issuecomment-640815550,https://api.github.com/repos/simonw/datasette/issues/816,640815550,MDEyOklzc3VlQ29tbWVudDY0MDgxNTU1MA==,9599,simonw,2020-06-08T19:06:44Z,2020-06-08T19:06:44Z,OWNER,https://github.com/simonw/datasette/blob/c7d145e016522dd6ee229d4d0b3ba79a7a8877c1/docs/plugins.rst#extra_template_varstemplate-database-table-view_name-request-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634783573,Come up with a new example for extra_template_vars plugin,
https://github.com/simonw/datasette/issues/817#issuecomment-640808161,https://api.github.com/repos/simonw/datasette/issues/817,640808161,MDEyOklzc3VlQ29tbWVudDY0MDgwODE2MQ==,9599,simonw,2020-06-08T18:51:42Z,2020-06-08T18:54:37Z,OWNER,I'm also going to rename `resource_identifier` to just `resource`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634844634,Drop resource_type from permission_allowed system,
https://github.com/simonw/datasette/issues/816#issuecomment-640763899,https://api.github.com/repos/simonw/datasette/issues/816,640763899,MDEyOklzc3VlQ29tbWVudDY0MDc2Mzg5OQ==,9599,simonw,2020-06-08T17:21:59Z,2020-06-08T17:21:59Z,OWNER,I'm going to show how to display the current user's user-agent.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634783573,Come up with a new example for extra_template_vars plugin,
https://github.com/simonw/datasette/issues/815#issuecomment-640673405,https://api.github.com/repos/simonw/datasette/issues/815,640673405,MDEyOklzc3VlQ29tbWVudDY0MDY3MzQwNQ==,9599,simonw,2020-06-08T14:41:55Z,2020-06-08T14:41:55Z,OWNER,"I want to be able to display the HTTP path and verb - `GET /fixtures`, `POST /fixtures/myquery` etc. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/815#issuecomment-640673138,https://api.github.com/repos/simonw/datasette/issues/815,640673138,MDEyOklzc3VlQ29tbWVudDY0MDY3MzEzOA==,9599,simonw,2020-06-08T14:41:24Z,2020-06-08T14:41:24Z,OWNER,I could reuse that `get_task_id()` function though (I can move it to utils).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/815#issuecomment-640672540,https://api.github.com/repos/simonw/datasette/issues/815,640672540,MDEyOklzc3VlQ29tbWVudDY0MDY3MjU0MA==,9599,simonw,2020-06-08T14:40:22Z,2020-06-08T14:40:22Z,OWNER,"Here's the current tracer mechanism. Note that it captures a stacktrace (which is expensive) - but only if the tracer system has been enabled for a request.
https://github.com/simonw/datasette/blob/1c063fae9dba70f70244db010d55a18846640f07/datasette/tracer.py#L27-L51
For permissions checks I want to ALWAYS track those calls, not just on requests that have opted in.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/815#issuecomment-640671398,https://api.github.com/repos/simonw/datasette/issues/815,640671398,MDEyOklzc3VlQ29tbWVudDY0MDY3MTM5OA==,9599,simonw,2020-06-08T14:38:20Z,2020-06-08T14:38:20Z,OWNER,But `ds._permission_checks` is also used for unit tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/815#issuecomment-640671241,https://api.github.com/repos/simonw/datasette/issues/815,640671241,MDEyOklzc3VlQ29tbWVudDY0MDY3MTI0MQ==,9599,simonw,2020-06-08T14:38:04Z,2020-06-08T14:38:04Z,OWNER,"Alternative to a correlation ID would be to use the existing `AsgiTracer` / `capture_traces` mechanism. That's probably smarter.
It could even start logging SQL queries to an in-memory deque too, so a debug tool could show you queries executed by other requests!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/815#issuecomment-640656143,https://api.github.com/repos/simonw/datasette/issues/815,640656143,MDEyOklzc3VlQ29tbWVudDY0MDY1NjE0Mw==,9599,simonw,2020-06-08T14:25:48Z,2020-06-08T14:26:45Z,OWNER,Will we need a request correlation ID for this? Multiple asyncio threads can write things to the `ds._permission_checks` deque at the same time.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/814#issuecomment-640638057,https://api.github.com/repos/simonw/datasette/issues/814,640638057,MDEyOklzc3VlQ29tbWVudDY0MDYzODA1Nw==,9599,simonw,2020-06-08T14:11:51Z,2020-06-08T14:12:12Z,OWNER,"The only impact it has at all is on this code here:
https://github.com/simonw/datasette/blob/cc218fa9be55842656d030545c308392e3736053/datasette/views/base.py#L515-L527
That `ds.cache_headers` property looks like it needs rethinking too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634651079,Remove --debug option from datasette serve,
https://github.com/simonw/datasette/issues/811#issuecomment-640362879,https://api.github.com/repos/simonw/datasette/issues/811,640362879,MDEyOklzc3VlQ29tbWVudDY0MDM2Mjg3OQ==,9599,simonw,2020-06-08T04:42:28Z,2020-06-08T13:39:46Z,OWNER,"I'm finding myself repeating this pattern a lot:
```python
for table in table_counts:
allowed = await self.ds.permission_allowed(
request.scope.get(""actor""),
""view-table"",
resource_type=""table"",
resource_identifier=(database, table),
default=True,
)
if not allowed:
continue
private = not await self.ds.permission_allowed(
None,
""view-table"",
resource_type=""table"",
resource_identifier=(database, table),
)
```
I use a similar pattern for lists of databases and lists of queries, and I'll be doing the same thing for lists of SQL views too.
An abstraction around this would be useful.
Idea:
```python
visible, private = await check_visibility(
self.ds, actor, ""view-table"", ""table"", (database, table)
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640367128,https://api.github.com/repos/simonw/datasette/issues/811,640367128,MDEyOklzc3VlQ29tbWVudDY0MDM2NzEyOA==,9599,simonw,2020-06-08T05:00:13Z,2020-06-08T05:00:49Z,OWNER,"Should the padlock show up on tables that are private only because they inherited their privacy from their parent database or even the parent instance?
Interesting question. If an instance is private, I'm not sure it makes sense to show padlocks on absolutely everything.
Likewise, a list of tables shown on the database table with a padlock next to every single table (when the database itself is private) doesn't seem to add any useful information.
I think ""Show 🔒 in header on private database page"" will resolve this for me. I'll always show the padlock in the header of a database/table page even if that privacy is inherited - but I won't do that for padlocks shown in the list of tables or list of databases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640365512,https://api.github.com/repos/simonw/datasette/issues/811,640365512,MDEyOklzc3VlQ29tbWVudDY0MDM2NTUxMg==,9599,simonw,2020-06-08T04:53:49Z,2020-06-08T04:53:49Z,OWNER,"I really like the padlocks. I should include a screenshot in the documentation that illustrates them.
![]()
Maybe I should figure out a way to have the https://latest.datasette.io/ demo illustrate both a logged-in and a logged-out state.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640348785,https://api.github.com/repos/simonw/datasette/issues/811,640348785,MDEyOklzc3VlQ29tbWVudDY0MDM0ODc4NQ==,9599,simonw,2020-06-08T03:51:50Z,2020-06-08T03:51:50Z,OWNER,"New convention: the 🔒 icon is now shown next to resources that are private - that are visible to you now, but would not be visible to the anonymous user.
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640345115,https://api.github.com/repos/simonw/datasette/issues/811,640345115,MDEyOklzc3VlQ29tbWVudDY0MDM0NTExNQ==,9599,simonw,2020-06-08T03:37:33Z,2020-06-08T03:37:33Z,OWNER,Per-table permissions is pretty interesting for large installations though - an organization might have hundreds of CSV files imported into Datasette and then allow users to specify which exact users within that organization are allowed to see which CSV.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640344950,https://api.github.com/repos/simonw/datasette/issues/811,640344950,MDEyOklzc3VlQ29tbWVudDY0MDM0NDk1MA==,9599,simonw,2020-06-08T03:36:49Z,2020-06-08T03:36:49Z,OWNER,"Oh this is a bit awkward - should I be running per-table permission checks for every table that might be shown on the index page?
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/801#issuecomment-640339828,https://api.github.com/repos/simonw/datasette/issues/801,640339828,MDEyOklzc3VlQ29tbWVudDY0MDMzOTgyOA==,9599,simonw,2020-06-08T03:18:47Z,2020-06-08T03:18:47Z,OWNER,"Example. This will only allow users to access the `fixtures` database if the logged-in actor's ID value appears for a record in the `users` table which has `admin` = 1.
```json
{
""databases"": {
""fixtures"": {
""allow_by_query"": ""select * from users where id = :id and admin = 1""
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631932926,allow_by_query setting for configuring permissions with a SQL statement,
https://github.com/simonw/datasette/issues/811#issuecomment-640339674,https://api.github.com/repos/simonw/datasette/issues/811,640339674,MDEyOklzc3VlQ29tbWVudDY0MDMzOTY3NA==,9599,simonw,2020-06-08T03:18:15Z,2020-06-08T03:18:15Z,OWNER,I should take these permissions into account when displaying a list of tables or a list of databases (like I do right now when displaying a list of queries).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/801#issuecomment-640339117,https://api.github.com/repos/simonw/datasette/issues/801,640339117,MDEyOklzc3VlQ29tbWVudDY0MDMzOTExNw==,9599,simonw,2020-06-08T03:16:16Z,2020-06-08T03:16:16Z,OWNER,"I'm going to call this key `""allow_by_query""` - I think I need `allow_sql` for something else (for configuring if users are allowed to execute arbitrary SQL queries).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631932926,allow_by_query setting for configuring permissions with a SQL statement,
https://github.com/simonw/datasette/issues/811#issuecomment-640338347,https://api.github.com/repos/simonw/datasette/issues/811,640338347,MDEyOklzc3VlQ29tbWVudDY0MDMzODM0Nw==,9599,simonw,2020-06-08T03:13:23Z,2020-06-08T03:13:23Z,OWNER,Do row-level permissions even make sense? Might be a good idea to remove those until I have a good use-case for them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640338151,https://api.github.com/repos/simonw/datasette/issues/811,640338151,MDEyOklzc3VlQ29tbWVudDY0MDMzODE1MQ==,9599,simonw,2020-06-08T03:12:41Z,2020-06-08T03:12:41Z,OWNER,"Also need to expand the docs on https://datasette.readthedocs.io/en/latest/authentication.html to explain where you can put `allow` blocks to control access to the instance, database or table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640337951,https://api.github.com/repos/simonw/datasette/issues/811,640337951,MDEyOklzc3VlQ29tbWVudDY0MDMzNzk1MQ==,9599,simonw,2020-06-08T03:11:58Z,2020-06-08T03:11:58Z,OWNER,"I'd like to be able to apply permissions for the ability to run a SQL query - but I'm not sure where the best place for that `""allow""` block to live would be.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640287967,https://api.github.com/repos/simonw/datasette/issues/811,640287967,MDEyOklzc3VlQ29tbWVudDY0MDI4Nzk2Nw==,9599,simonw,2020-06-07T22:16:10Z,2020-06-07T22:16:10Z,OWNER,The tests in test_permissions.py could check the .json variants and assert that permission checks were carried out too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/395#issuecomment-640280741,https://api.github.com/repos/simonw/datasette/issues/395,640280741,MDEyOklzc3VlQ29tbWVudDY0MDI4MDc0MQ==,9599,simonw,2020-06-07T21:12:57Z,2020-06-07T21:12:57Z,OWNER,"This is a pattern I like:
```python
with make_app_client(
template_dir=str(pathlib.Path(__file__).parent / ""test_templates"")
) as client:
response = client.get(""/-/metadata"")
assert response.status == 200
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",396215043,Find a cleaner pattern for fixtures with arguments,
https://github.com/simonw/datasette/issues/801#issuecomment-640277775,https://api.github.com/repos/simonw/datasette/issues/801,640277775,MDEyOklzc3VlQ29tbWVudDY0MDI3Nzc3NQ==,9599,simonw,2020-06-07T20:49:40Z,2020-06-07T20:49:40Z,OWNER,"I'm going to pass the entire actor object as a dictionary of available named query parameters. So if the actor looks like this:
```json
{
""id"": ""simonw"",
""roles"": [""staff"", ""developer""]
}
```
Then the SQL query will be called like this:
```python
conn.execute(sql, {
""id"": ""simonw"",
""roles: '[""staff"", ""developer""]',
})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631932926,allow_by_query setting for configuring permissions with a SQL statement,
https://github.com/simonw/datasette/issues/801#issuecomment-640277557,https://api.github.com/repos/simonw/datasette/issues/801,640277557,MDEyOklzc3VlQ29tbWVudDY0MDI3NzU1Nw==,9599,simonw,2020-06-07T20:48:00Z,2020-06-07T20:48:00Z,OWNER,"Now that I'm expanding permission checks to everything else too (#811), not just canned queries, I think it makes sense to re-prioritize this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631932926,allow_by_query setting for configuring permissions with a SQL statement,
https://github.com/simonw/datasette/issues/811#issuecomment-640274171,https://api.github.com/repos/simonw/datasette/issues/811,640274171,MDEyOklzc3VlQ29tbWVudDY0MDI3NDE3MQ==,9599,simonw,2020-06-07T20:21:14Z,2020-06-07T20:21:14Z,OWNER,"Next step: fix this
```
- # TODO: fix this to use that permission check
- if not actor_matches_allow(
- request.scope.get(""actor"", None), metadata.get(""allow"")
- ):
- return Response(""Permission denied"", status=403)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640273945,https://api.github.com/repos/simonw/datasette/issues/811,640273945,MDEyOklzc3VlQ29tbWVudDY0MDI3Mzk0NQ==,9599,simonw,2020-06-07T20:19:15Z,2020-06-07T20:19:15Z,OWNER,I'm going to add a `test_permissions.py` module that checks for 403 errors against different patterns of the `actors` block at different levels in `metadata.json`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640270178,https://api.github.com/repos/simonw/datasette/issues/811,640270178,MDEyOklzc3VlQ29tbWVudDY0MDI3MDE3OA==,9599,simonw,2020-06-07T19:48:39Z,2020-06-07T19:48:39Z,OWNER,"Testing pattern:
```python
def test_canned_query_with_custom_metadata(app_client):
response = app_client.get(""/fixtures/neighborhood_search?text=town"")
assert_permissions_checked(
app_client.ds,
[
""view-instance"",
(""view-database"", ""database"", ""fixtures""),
(""view-query"", ""query"", (""fixtures"", ""neighborhood_search"")),
],
)
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640248972,https://api.github.com/repos/simonw/datasette/issues/811,640248972,MDEyOklzc3VlQ29tbWVudDY0MDI0ODk3Mg==,9599,simonw,2020-06-07T17:04:22Z,2020-06-07T17:04:22Z,OWNER,I'll need a neat testing pattern for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/810#issuecomment-640248864,https://api.github.com/repos/simonw/datasette/issues/810,640248864,MDEyOklzc3VlQ29tbWVudDY0MDI0ODg2NA==,9599,simonw,2020-06-07T17:03:15Z,2020-06-07T17:03:15Z,OWNER,This is obsoleted by #811.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633066114,Refactor permission check for canned query,
https://github.com/simonw/datasette/issues/811#issuecomment-640248669,https://api.github.com/repos/simonw/datasette/issues/811,640248669,MDEyOklzc3VlQ29tbWVudDY0MDI0ODY2OQ==,9599,simonw,2020-06-07T17:01:44Z,2020-06-07T17:01:44Z,OWNER,"If the allow block at the database level forbids access this needs to cascade down to the table, query and row levels as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/215#issuecomment-640121917,https://api.github.com/repos/simonw/datasette/issues/215,640121917,MDEyOklzc3VlQ29tbWVudDY0MDEyMTkxNw==,9599,simonw,2020-06-06T21:42:58Z,2020-06-07T05:58:36Z,OWNER,"I might use some dependency injection here, with `call_with_supported_arguments()` from https://github.com/simonw/datasette/commit/41a0cd7b6afe0397efbbf27ad822679fc574811a#diff-942305c83055fdc0ff5f4e7d6ab06b29
Maybe a view function can take `request` and optionally also take `datasette`? Or `scope` or `receive` or `send`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/800#issuecomment-640160487,https://api.github.com/repos/simonw/datasette/issues/800,640160487,MDEyOklzc3VlQ29tbWVudDY0MDE2MDQ4Nw==,9599,simonw,2020-06-07T05:34:07Z,2020-06-07T05:34:07Z,OWNER,See #810 for work to finish this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/808#issuecomment-640157216,https://api.github.com/repos/simonw/datasette/issues/808,640157216,MDEyOklzc3VlQ29tbWVudDY0MDE1NzIxNg==,9599,simonw,2020-06-07T04:58:40Z,2020-06-07T04:58:40Z,OWNER,... and I want a unit test which confirms that all permissions are documented.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632918799,Permission check for every view in Datasette (plus docs),
https://github.com/simonw/datasette/issues/808#issuecomment-640152036,https://api.github.com/repos/simonw/datasette/issues/808,640152036,MDEyOklzc3VlQ29tbWVudDY0MDE1MjAzNg==,9599,simonw,2020-06-07T03:38:07Z,2020-06-07T03:38:07Z,OWNER,I'm going to need to add permissions documentation for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632918799,Permission check for every view in Datasette (plus docs),
https://github.com/simonw/datasette/issues/807#issuecomment-640135332,https://api.github.com/repos/simonw/datasette/issues/807,640135332,MDEyOklzc3VlQ29tbWVudDY0MDEzNTMzMg==,9599,simonw,2020-06-07T00:13:51Z,2020-06-07T00:13:51Z,OWNER,"These should not be shipped as the latest version on Docker Hub. They also should not become the ""stable"" release on ReadTheDocs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/800#issuecomment-640123488,https://api.github.com/repos/simonw/datasette/issues/800,640123488,MDEyOklzc3VlQ29tbWVudDY0MDEyMzQ4OA==,9599,simonw,2020-06-06T21:59:14Z,2020-06-06T21:59:14Z,OWNER,I didn't build this quite right: it should be using the permissions plugin hook.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/805#issuecomment-640122664,https://api.github.com/repos/simonw/datasette/issues/805,640122664,MDEyOklzc3VlQ29tbWVudDY0MDEyMjY2NA==,9599,simonw,2020-06-06T21:50:41Z,2020-06-06T21:50:41Z,OWNER,Part of #806 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/215#issuecomment-504881900,https://api.github.com/repos/simonw/datasette/issues/215,504881900,MDEyOklzc3VlQ29tbWVudDUwNDg4MTkwMA==,9599,simonw,2019-06-24T06:51:29Z,2020-06-06T21:47:11Z,OWNER,See also #520 - asgi_wrapper plugin hook.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-398826108,https://api.github.com/repos/simonw/datasette/issues/215,398826108,MDEyOklzc3VlQ29tbWVudDM5ODgyNjEwOA==,9599,simonw,2018-06-20T17:09:18Z,2020-06-06T21:46:51Z,OWNER,This depends on #272 - Datasette ported to ASGI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640122120,https://api.github.com/repos/simonw/datasette/issues/215,640122120,MDEyOklzc3VlQ29tbWVudDY0MDEyMjEyMA==,9599,simonw,2020-06-06T21:45:13Z,2020-06-06T21:45:52Z,OWNER,"Stretch goal: make it easy for plugin views to implement formats, so they can produce HTML by default and .json or .csv etc as alternative outputs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640121036,https://api.github.com/repos/simonw/datasette/issues/215,640121036,MDEyOklzc3VlQ29tbWVudDY0MDEyMTAzNg==,9599,simonw,2020-06-06T21:34:03Z,2020-06-06T21:34:03Z,OWNER,"I'll refactor existing code to register views using the same mechanism that plugins will have access to.
Maybe plugins get to register their routes first? That would allow plugins to do things like entirely take over the / page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640119259,https://api.github.com/repos/simonw/datasette/issues/215,640119259,MDEyOklzc3VlQ29tbWVudDY0MDExOTI1OQ==,9599,simonw,2020-06-06T21:16:46Z,2020-06-06T21:16:46Z,OWNER,"I deprioritised this a while ago because the asgi_wrapper hook allowed me to set up new URL routes: https://datasette.readthedocs.io/en/0.43/plugins.html#asgi-wrapper-datasette
But... those were pretty low level, for example this code here: https://github.com/simonw/datasette-auth-github/blob/6c971064f6f4e6857bade5c6b88842f9cdeca9d9/datasette_auth_github/github_auth.py#L104-L113
Now that Datasette has a documented request object #706 and that object is used by things like the flash messages system (#790) - https://datasette.readthedocs.io/en/latest/internals.html#add-message-request-message-message-type-datasette-info - I find myself wanting to add views which get a request, as opposed to an ASGI scope.
So I'm re-prioritising this, with the main need being a way for plugins to hook up their own view functions that can accept a request and return a response. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640118802,https://api.github.com/repos/simonw/datasette/issues/215,640118802,MDEyOklzc3VlQ29tbWVudDY0MDExODgwMg==,9599,simonw,2020-06-06T21:12:41Z,2020-06-06T21:12:41Z,OWNER,@clausjuhl your use-case there is now covered by custom pages from Datasette 0.41 https://datasette.readthedocs.io/en/stable/changelog.html#v0-41,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/805#issuecomment-640116970,https://api.github.com/repos/simonw/datasette/issues/805,640116970,MDEyOklzc3VlQ29tbWVudDY0MDExNjk3MA==,9599,simonw,2020-06-06T20:55:03Z,2020-06-06T20:55:03Z,OWNER,"Would be useful if I had a plugin that could authenticate users based on a secret environment variable (maybe for a password) - that way I could have an ""admin"" account on the Glitch app that is allowed to setup new polls, while anonymous users can only vote on them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/805#issuecomment-640116842,https://api.github.com/repos/simonw/datasette/issues/805,640116842,MDEyOklzc3VlQ29tbWVudDY0MDExNjg0Mg==,9599,simonw,2020-06-06T20:53:51Z,2020-06-06T20:53:51Z,OWNER,"I'd like to illustrate writable canned queries without the risk of someone abusing and breaking it (or filling it with bad content).
I don't want to have to monitor it, so an application that won't run out of disk space after a few months would be good too.
Maybe a polling app? If I'm only tracking integer numbers of votes it shouldn't ever run out of space.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/791#issuecomment-640116494,https://api.github.com/repos/simonw/datasette/issues/791,640116494,MDEyOklzc3VlQ29tbWVudDY0MDExNjQ5NA==,9599,simonw,2020-06-06T20:50:41Z,2020-06-06T20:50:41Z,OWNER,"I have a better idea: a feed reader! You can insert URLs to feeds, then have a command which fetches the latest entries from them into a separate table. Then implement favorites as a canned query, let you search your favorites, etc.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",628572716,Tutorial: building a something-interesting with writable canned queries,
https://github.com/simonw/datasette/issues/787#issuecomment-640111383,https://api.github.com/repos/simonw/datasette/issues/787,640111383,MDEyOklzc3VlQ29tbWVudDY0MDExMTM4Mw==,9599,simonw,2020-06-06T20:04:20Z,2020-06-06T20:04:20Z,OWNER,"I should let people running the 'publish' command set this explicitly if they want to, so they can re-deploy a published Datasette without invalidating every user's cookies.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628089318,"""datasette publish"" should bake in a random --secret",
https://github.com/simonw/datasette/issues/698#issuecomment-640108942,https://api.github.com/repos/simonw/datasette/issues/698,640108942,MDEyOklzc3VlQ29tbWVudDY0MDEwODk0Mg==,9599,simonw,2020-06-06T19:43:48Z,2020-06-06T19:43:48Z,OWNER,"Landed - documentation is here: https://datasette.readthedocs.io/en/latest/sql_queries.html#writable-canned-queries
See also https://datasette.readthedocs.io/en/latest/authentication.html#permissions-for-canned-queries","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/800#issuecomment-640108835,https://api.github.com/repos/simonw/datasette/issues/800,640108835,MDEyOklzc3VlQ29tbWVudDY0MDEwODgzNQ==,9599,simonw,2020-06-06T19:42:46Z,2020-06-06T19:42:46Z,OWNER,This is implemented and documented: https://datasette.readthedocs.io/en/latest/authentication.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/699#issuecomment-640108763,https://api.github.com/repos/simonw/datasette/issues/699,640108763,MDEyOklzc3VlQ29tbWVudDY0MDEwODc2Mw==,9599,simonw,2020-06-06T19:42:11Z,2020-06-06T19:42:11Z,OWNER,I landed canned query writes. This feature can now be considered complete: https://datasette.readthedocs.io/en/latest/authentication.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-640106668,https://api.github.com/repos/simonw/datasette/issues/699,640106668,MDEyOklzc3VlQ29tbWVudDY0MDEwNjY2OA==,9599,simonw,2020-06-06T19:22:36Z,2020-06-06T19:22:36Z,OWNER,The canned queries feature is gaining permissions support in #800.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/804#issuecomment-640106569,https://api.github.com/repos/simonw/datasette/issues/804,640106569,MDEyOklzc3VlQ29tbWVudDY0MDEwNjU2OQ==,9599,simonw,2020-06-06T19:21:41Z,2020-06-06T19:21:41Z,OWNER,I don't think this is fully documented either. Current partial documentation is on https://datasette.readthedocs.io/en/stable/contributing.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632673972,python tests/fixtures.py command has a bug,
https://github.com/simonw/datasette/issues/804#issuecomment-640106342,https://api.github.com/repos/simonw/datasette/issues/804,640106342,MDEyOklzc3VlQ29tbWVudDY0MDEwNjM0Mg==,9599,simonw,2020-06-06T19:19:33Z,2020-06-06T19:19:33Z,OWNER,I should replace the bodged-together argument passing with Click while I'm fixing this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632673972,python tests/fixtures.py command has a bug,
https://github.com/simonw/datasette/issues/804#issuecomment-640106202,https://api.github.com/repos/simonw/datasette/issues/804,640106202,MDEyOklzc3VlQ29tbWVudDY0MDEwNjIwMg==,9599,simonw,2020-06-06T19:18:23Z,2020-06-06T19:18:43Z,OWNER,"I broke this in #775 https://github.com/simonw/datasette/commit/446e5de65d1b9c6c877e38b0ef13bc9285c465a1
Here's the now-broken code (I removed the `PLUGIN1` and `PLUGIN2` constants): https://github.com/simonw/datasette/blob/9c563d6aed072f14d3d25f58e84659f9caa1a243/tests/fixtures.py#L828-L835","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632673972,python tests/fixtures.py command has a bug,
https://github.com/simonw/datasette/issues/800#issuecomment-640103204,https://api.github.com/repos/simonw/datasette/issues/800,640103204,MDEyOklzc3VlQ29tbWVudDY0MDEwMzIwNA==,9599,simonw,2020-06-06T18:52:56Z,2020-06-06T18:52:56Z,OWNER,"I'm also going to add an indicator to the UI next to queries that you can only execute because you are signed in:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640102200,https://api.github.com/repos/simonw/datasette/issues/800,640102200,MDEyOklzc3VlQ29tbWVudDY0MDEwMjIwMA==,9599,simonw,2020-06-06T18:45:11Z,2020-06-06T18:45:11Z,OWNER,"In the code that's:
https://github.com/simonw/datasette/blob/9c563d6aed072f14d3d25f58e84659f9caa1a243/datasette/views/database.py#L56-L64
And:
https://github.com/simonw/datasette/blob/9c563d6aed072f14d3d25f58e84659f9caa1a243/datasette/views/database.py#L98-L112
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640101762,https://api.github.com/repos/simonw/datasette/issues/800,640101762,MDEyOklzc3VlQ29tbWVudDY0MDEwMTc2Mg==,9599,simonw,2020-06-06T18:41:20Z,2020-06-06T18:41:20Z,OWNER,Now the actual permission checks. I need these in two places: the code that generates the list of available queries on https://latest.datasette.io/fixtures#queries and the query page itself at https://latest.datasette.io/fixtures/pragma_cache_size,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640101625,https://api.github.com/repos/simonw/datasette/issues/800,640101625,MDEyOklzc3VlQ29tbWVudDY0MDEwMTYyNQ==,9599,simonw,2020-06-06T18:40:09Z,2020-06-06T18:40:09Z,OWNER,Documentation for `actor_matches_allow`: https://github.com/simonw/datasette/blob/14f6b4d200f24940a795ddc0825319ab2891bde2/docs/authentication.rst#actor_matches_allow,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640099707,https://api.github.com/repos/simonw/datasette/issues/800,640099707,MDEyOklzc3VlQ29tbWVudDY0MDA5OTcwNw==,9599,simonw,2020-06-06T18:24:54Z,2020-06-06T18:24:54Z,OWNER,Next step: a utility function and tests for matching actors to allow blocks.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640099404,https://api.github.com/repos/simonw/datasette/issues/800,640099404,MDEyOklzc3VlQ29tbWVudDY0MDA5OTQwNA==,9599,simonw,2020-06-06T18:22:10Z,2020-06-06T18:24:26Z,OWNER,Docs here: https://github.com/simonw/datasette/blob/d4c7b85f556230923d37ff327a068ed08aa9b62b/docs/authentication.rst#setting-permissions-for-canned-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640099434,https://api.github.com/repos/simonw/datasette/issues/800,640099434,MDEyOklzc3VlQ29tbWVudDY0MDA5OTQzNA==,9599,simonw,2020-06-06T18:22:29Z,2020-06-06T18:22:29Z,OWNER,I should add the '*' bit to the docs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/786#issuecomment-640099333,https://api.github.com/repos/simonw/datasette/issues/786,640099333,MDEyOklzc3VlQ29tbWVudDY0MDA5OTMzMw==,9599,simonw,2020-06-06T18:21:36Z,2020-06-06T18:21:36Z,OWNER,"This is done but currently lives in a branch, will close this issue when that branch lands: Implemented in this branch: https://github.com/simonw/datasette/blob/30a8132d58a89fed0e034e058b62fab5180fae0f/docs/authentication.rst","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628087971,Documentation page describing Datasette's authentication system,
https://github.com/simonw/datasette/issues/800#issuecomment-640090575,https://api.github.com/repos/simonw/datasette/issues/800,640090575,MDEyOklzc3VlQ29tbWVudDY0MDA5MDU3NQ==,9599,simonw,2020-06-06T17:06:28Z,2020-06-06T17:06:28Z,OWNER,I'm going to implement this documentation-first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640090343,https://api.github.com/repos/simonw/datasette/issues/800,640090343,MDEyOklzc3VlQ29tbWVudDY0MDA5MDM0Mw==,9599,simonw,2020-06-06T17:04:36Z,2020-06-06T17:04:36Z,OWNER,I like this mechanism better than the SQL query one. Constructing SQL queries that return true if a particular string is embedded inside a JSON list in a larger object is decidedly non-trivial.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/802#issuecomment-639895450,https://api.github.com/repos/simonw/datasette/issues/802,639895450,MDEyOklzc3VlQ29tbWVudDYzOTg5NTQ1MA==,9599,simonw,2020-06-05T23:33:52Z,2020-06-05T23:33:52Z,OWNER,"https://github.com/simonw/datasette/blob/033a1bb22c70a955d9fd1d3b4675a0e2e5c8b8cd/datasette/cli.py#L126-L129
But I changed the `.plugins()` method to this:
https://github.com/simonw/datasette/blob/033a1bb22c70a955d9fd1d3b4675a0e2e5c8b8cd/datasette/app.py#L628-L633","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632056825,"""datasette plugins"" command is broken",
https://github.com/simonw/datasette/issues/800#issuecomment-639803719,https://api.github.com/repos/simonw/datasette/issues/800,639803719,MDEyOklzc3VlQ29tbWVudDYzOTgwMzcxOQ==,9599,simonw,2020-06-05T20:40:34Z,2020-06-05T20:40:34Z,OWNER,It's a bit obscure though. I'll try building both and see how they feel in practice.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-639803099,https://api.github.com/repos/simonw/datasette/issues/800,639803099,MDEyOklzc3VlQ29tbWVudDYzOTgwMzA5OQ==,9599,simonw,2020-06-05T20:39:34Z,2020-06-05T20:39:34Z,OWNER,"Maybe #801 (configuring permissions with a SQL query) is enough here - might not need this mechanism at all, since that mechanism covers it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/698#issuecomment-639788562,https://api.github.com/repos/simonw/datasette/issues/698,639788562,MDEyOklzc3VlQ29tbWVudDYzOTc4ODU2Mg==,9599,simonw,2020-06-05T20:27:49Z,2020-06-05T20:27:49Z,OWNER,"There can be a detailed section explaining these different mechanisms on the authentication documentation page.
I imagine they will end up applying to more than just canned queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-639787304,https://api.github.com/repos/simonw/datasette/issues/698,639787304,MDEyOklzc3VlQ29tbWVudDYzOTc4NzMwNA==,9599,simonw,2020-06-05T20:26:57Z,2020-06-05T20:26:57Z,OWNER,"Idea: an `""allow_sql""` key with a SQL query that gets passed the actor JSON as `:actor` and can extract the relevant keys from it and return 1 or 0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-639785878,https://api.github.com/repos/simonw/datasette/issues/698,639785878,MDEyOklzc3VlQ29tbWVudDYzOTc4NTg3OA==,9599,simonw,2020-06-05T20:25:55Z,2020-06-05T20:25:55Z,OWNER,"I'd really like to support SQL query defined permissions too, mainly to set an example for how plugins could do something similar.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-639784651,https://api.github.com/repos/simonw/datasette/issues/698,639784651,MDEyOklzc3VlQ29tbWVudDYzOTc4NDY1MQ==,9599,simonw,2020-06-05T20:25:02Z,2020-06-05T20:25:02Z,OWNER,"Idea: default is anyone can execute a query.
Or you can specify the following:
```json
{
""databases"": {
""my-database"": {
""queries"": {
""add_twitter_handle"": {
""sql"": ""insert into twitter_handles (username) values (:username)"",
""write"": true,
""allow"": {
""id"": [""simon""],
""role"": [""staff""]
}
}
}
}
}
}
```
These get matched against the actor JSON. If any of the fields in any of the keys of `""allow""` match a key on the actor, the query is allowed.
`""id"": ""*""` matches any actor with an `id` key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-639779403,https://api.github.com/repos/simonw/datasette/issues/698,639779403,MDEyOklzc3VlQ29tbWVudDYzOTc3OTQwMw==,9599,simonw,2020-06-05T20:20:12Z,2020-06-05T20:20:12Z,OWNER,CSRF is done. Last step: figure out a smart way to integrate this with permissions and authentication.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/pull/798#issuecomment-639712835,https://api.github.com/repos/simonw/datasette/issues/798,639712835,MDEyOklzc3VlQ29tbWVudDYzOTcxMjgzNQ==,9599,simonw,2020-06-05T18:53:32Z,2020-06-05T18:53:32Z,OWNER,Add unit tests illustrating the `Vary: Cookie` header and I'm done here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631300342,CSRF protection,
https://github.com/simonw/datasette/pull/798#issuecomment-639685550,https://api.github.com/repos/simonw/datasette/issues/798,639685550,MDEyOklzc3VlQ29tbWVudDYzOTY4NTU1MA==,9599,simonw,2020-06-05T18:20:34Z,2020-06-05T18:20:34Z,OWNER,I'm solving the compatibility with caching problem in this ticket: https://github.com/simonw/asgi-csrf/issues/7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631300342,CSRF protection,
https://github.com/simonw/datasette/issues/799#issuecomment-639661014,https://api.github.com/repos/simonw/datasette/issues/799,639661014,MDEyOklzc3VlQ29tbWVudDYzOTY2MTAxNA==,9599,simonw,2020-06-05T17:43:41Z,2020-06-05T17:43:41Z,OWNER,I'm going to rename that `MultiParams` and use it in both places.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631789422,TestResponse needs to handle multiple set-cookie headers,
https://github.com/simonw/datasette/issues/799#issuecomment-639660667,https://api.github.com/repos/simonw/datasette/issues/799,639660667,MDEyOklzc3VlQ29tbWVudDYzOTY2MDY2Nw==,9599,simonw,2020-06-05T17:43:08Z,2020-06-05T17:43:08Z,OWNER,"This really needs a `MultiValueDict` ala Django: https://github.com/django/django/blob/24b82cd201e21060fbc02117dc16d1702877a1f3/django/utils/datastructures.py#L42
Turns out I have one of these in Datasette already - `RequestParameters` from https://github.com/simonw/datasette/commit/81be31322a968d23cf57cee62b58df55433385e3
The name isn't quite right though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631789422,TestResponse needs to handle multiple set-cookie headers,
https://github.com/simonw/datasette/pull/798#issuecomment-639269994,https://api.github.com/repos/simonw/datasette/issues/798,639269994,MDEyOklzc3VlQ29tbWVudDYzOTI2OTk5NA==,9599,simonw,2020-06-05T05:36:35Z,2020-06-05T05:38:25Z,OWNER,"Django docs on CSRF and caching: https://docs.djangoproject.com/en/3.0/ref/csrf/#caching
> If the csrf_token template tag is used by a template (or the get_token function is called some other way), CsrfViewMiddleware will add a cookie and a Vary: Cookie header to the response. This means that the middleware will play well with the cache middleware if it is used as instructed
So the cookie is only set for pages that included a hidden csrftoken form field! This could work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631300342,CSRF protection,
https://github.com/simonw/datasette/pull/798#issuecomment-639269559,https://api.github.com/repos/simonw/datasette/issues/798,639269559,MDEyOklzc3VlQ29tbWVudDYzOTI2OTU1OQ==,9599,simonw,2020-06-05T05:34:56Z,2020-06-05T05:35:23Z,OWNER,"I don't want to set a cookie on a page response that is being cached.
Right now the ASGI middleware will be doing exactly that, which is bad.
But how do I get certainty that when you load a page with a form that will be CSRF protected you have been served the cookie?
Maybe those pages should do something explicit to the request object indicating that the cookie is needed?
That works for Datasette (since it has mutable request objects) but I'm not sure how it would work in the asgi-csrf pure ASGI middleware context.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631300342,CSRF protection,
https://github.com/simonw/datasette/pull/798#issuecomment-639249743,https://api.github.com/repos/simonw/datasette/issues/798,639249743,MDEyOklzc3VlQ29tbWVudDYzOTI0OTc0Mw==,9599,simonw,2020-06-05T04:23:01Z,2020-06-05T04:23:01Z,OWNER,"Needs unit tests.
More importantly: needs very, very careful consideration of how this plays with HTTP caching.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631300342,CSRF protection,
https://github.com/simonw/datasette/issues/684#issuecomment-639053707,https://api.github.com/repos/simonw/datasette/issues/684,639053707,MDEyOklzc3VlQ29tbWVudDYzOTA1MzcwNw==,9599,simonw,2020-06-04T18:56:15Z,2020-06-04T18:56:15Z,OWNER,This documentation is live here: https://datasette.readthedocs.io/en/latest/internals.html#database-introspection,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570301333,Add documentation on Database introspection methods to internals.rst,
https://github.com/simonw/datasette/issues/119#issuecomment-639047315,https://api.github.com/repos/simonw/datasette/issues/119,639047315,MDEyOklzc3VlQ29tbWVudDYzOTA0NzMxNQ==,9599,simonw,2020-06-04T18:46:39Z,2020-06-04T18:46:39Z,OWNER,"The OAuth dance needed for this is a pretty nasty barrier to plugin installation and configuration.
I'm going to focus on making it easy to copy and paste data into sheets instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275082158,"Build an ""export this data to google sheets"" plugin",
https://github.com/simonw/datasette/issues/793#issuecomment-638462052,https://api.github.com/repos/simonw/datasette/issues/793,638462052,MDEyOklzc3VlQ29tbWVudDYzODQ2MjA1Mg==,9599,simonw,2020-06-03T21:07:39Z,2020-06-03T21:07:39Z,OWNER,I need to land and release the fix for signing cookies in https://github.com/simonw/asgi-csrf/issues/2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629524205,CSRF protection for /-/messages tool and writable canned queries,
https://github.com/simonw/datasette/issues/797#issuecomment-638461797,https://api.github.com/repos/simonw/datasette/issues/797,638461797,MDEyOklzc3VlQ29tbWVudDYzODQ2MTc5Nw==,9599,simonw,2020-06-03T21:07:06Z,2020-06-03T21:07:06Z,OWNER,"Docs here (search for ""params""): https://datasette.readthedocs.io/en/latest/sql_queries.html#canned-queries-named-parameters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",630120235,"Documentation for new ""params"" setting for canned queries",
https://github.com/simonw/datasette/issues/797#issuecomment-638289878,https://api.github.com/repos/simonw/datasette/issues/797,638289878,MDEyOklzc3VlQ29tbWVudDYzODI4OTg3OA==,9599,simonw,2020-06-03T15:57:47Z,2020-06-03T15:57:47Z,OWNER,Also mention ability to pre-fill the form for writable canned queries using the querystring.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",630120235,"Documentation for new ""params"" setting for canned queries",
https://github.com/simonw/datasette/issues/698#issuecomment-638266171,https://api.github.com/repos/simonw/datasette/issues/698,638266171,MDEyOklzc3VlQ29tbWVudDYzODI2NjE3MQ==,9599,simonw,2020-06-03T15:18:49Z,2020-06-03T15:18:49Z,OWNER,Landed the work so far from #796! Here's the documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#writable-canned-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/793#issuecomment-638265394,https://api.github.com/repos/simonw/datasette/issues/793,638265394,MDEyOklzc3VlQ29tbWVudDYzODI2NTM5NA==,9599,simonw,2020-06-03T15:17:35Z,2020-06-03T15:17:51Z,OWNER,I need this for writable canned queries in #698 and #796 too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629524205,CSRF protection for /-/messages tool and writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638257697,https://api.github.com/repos/simonw/datasette/issues/796,638257697,MDEyOklzc3VlQ29tbWVudDYzODI1NzY5Nw==,9599,simonw,2020-06-03T15:05:07Z,2020-06-03T15:05:07Z,OWNER,"I'm going to document this, land it and then continue to work on the other pieces - CSRF protection and .json mode - in separate tickets.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638249652,https://api.github.com/repos/simonw/datasette/issues/796,638249652,MDEyOklzc3VlQ29tbWVudDYzODI0OTY1Mg==,9599,simonw,2020-06-03T14:51:29Z,2020-06-03T14:51:51Z,OWNER,"Consider this one:
```
""delete_name"": {
""sql"": ""delete from names where rowid = :rowid"",
""write"": True,
""on_success_message"": ""Name deleted"",
},
```
If the user enters an invalid `rowid` the query will still execute without errors and hence the success message will still be displayed. Can I address this?
Maybe allow an optional `""rowcount_expected"": 1` property? And if that count isn't matched treat the query as an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638241738,https://api.github.com/repos/simonw/datasette/issues/796,638241738,MDEyOklzc3VlQ29tbWVudDYzODI0MTczOA==,9599,simonw,2020-06-03T14:38:45Z,2020-06-03T14:38:45Z,OWNER,Violating a unique constraint will throw an error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638241366,https://api.github.com/repos/simonw/datasette/issues/796,638241366,MDEyOklzc3VlQ29tbWVudDYzODI0MTM2Ng==,9599,simonw,2020-06-03T14:38:03Z,2020-06-03T14:38:31Z,OWNER,"Maybe I need some kind of optional validation mechanism? SQLite lets me insert text or floating point numbers into integer columns right now, which feels wrong.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638240919,https://api.github.com/repos/simonw/datasette/issues/796,638240919,MDEyOklzc3VlQ29tbWVudDYzODI0MDkxOQ==,9599,simonw,2020-06-03T14:37:22Z,2020-06-03T14:37:22Z,OWNER,"I'm having trouble coming up with a canned SQL query that will only error with certain parameters, which I need in order to test out the `on_error_redirect` and `on_error_message` properties.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638238144,https://api.github.com/repos/simonw/datasette/issues/796,638238144,MDEyOklzc3VlQ29tbWVudDYzODIzODE0NA==,9599,simonw,2020-06-03T14:32:54Z,2020-06-03T14:33:11Z,OWNER,I'm going to have `on_success_redirect` and `on_error_redirect` properties for specifying where the redirect should go too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638188196,https://api.github.com/repos/simonw/datasette/issues/796,638188196,MDEyOklzc3VlQ29tbWVudDYzODE4ODE5Ng==,9599,simonw,2020-06-03T13:13:27Z,2020-06-03T14:32:27Z,OWNER,"""Query executed"" is the default message, but it's pretty bland:
![]()
How about letting queries define custom success messages in their metadata configuration?
`""on_success_message""` and `""on_error_message""`
How can the system tell if an ""update"" query was actually successful? Maybe I should expose `.rowcount` somehow, so I can report back on how many rows were updated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638206851,https://api.github.com/repos/simonw/datasette/issues/796,638206851,MDEyOklzc3VlQ29tbWVudDYzODIwNjg1MQ==,9599,simonw,2020-06-03T13:44:29Z,2020-06-03T13:44:29Z,OWNER,"Default message is now ""Query executed, 1 row affected"" as of https://github.com/simonw/datasette/pull/796/commits/f45c44ac8f3a9a2182d76c6bda44a06676499e4b","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638205923,https://api.github.com/repos/simonw/datasette/issues/796,638205923,MDEyOklzc3VlQ29tbWVudDYzODIwNTkyMw==,9599,simonw,2020-06-03T13:43:12Z,2020-06-03T13:43:12Z,OWNER,For `.json` mode (when the URL has a `.json` suffix) I'm going to handle both form-encoded and JSON encoded inputs and return a 200 status code with JSON that tells you if the query executed successfully and how many rows were affected.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/issues/698#issuecomment-638183337,https://api.github.com/repos/simonw/datasette/issues/698,638183337,MDEyOklzc3VlQ29tbWVudDYzODE4MzMzNw==,9599,simonw,2020-06-03T13:05:03Z,2020-06-03T13:05:03Z,OWNER,"One challenge with this feature is that it confuses the messaging about what Datasette does somewhat.
Prior to shipping this, Datasette's core value proposition is as a way to publish read-only data.
That changed a little [in 0.37 in February](https://datasette.readthedocs.io/en/stable/changelog.html#v0-37) when plugins gained the supported ability to execute writes, but there was no way of doing that without a plugin.
With this feature, Datasette becomes a read-write database solution.
I should update the documentation to help explain this. Essentially the message is that Datasette can be used in one of two ""modes"" - it can be used just for sharing/publishing data, or you can use it to collect and manage data, most likely still in collaboration with plugins for things like authentication.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-637934813,https://api.github.com/repos/simonw/datasette/issues/698,637934813,MDEyOklzc3VlQ29tbWVudDYzNzkzNDgxMw==,9599,simonw,2020-06-03T03:45:07Z,2020-06-03T03:45:07Z,OWNER,"Some extra thoughts now that this is mostly working:
- ""Edit this row"" is such an obvious use-case. Could I automatically support row editing where every column except the primary key can be updated?
- It would be useful to be able to link to a query in a way that pre-populates various form fields. The ""edit"" interface could then be a link that pre-populates the form with all of the existing values.
- Can the redirect URL be configured to include values from the form submission? So you could e.g. add a blog post with a unique slug and then redirect to that URL?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-637879242,https://api.github.com/repos/simonw/datasette/issues/698,637879242,MDEyOklzc3VlQ29tbWVudDYzNzg3OTI0Mg==,9599,simonw,2020-06-03T00:10:30Z,2020-06-03T00:10:30Z,OWNER,Started a fresh pull request for this in #796 - the one in #703 got a bit untidy.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/pull/703#issuecomment-637875307,https://api.github.com/repos/simonw/datasette/issues/703,637875307,MDEyOklzc3VlQ29tbWVudDYzNzg3NTMwNw==,9599,simonw,2020-06-02T23:57:35Z,2020-06-02T23:57:35Z,OWNER,This pull request got too messy. I'm going to abandon this and start a new one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585597133,WIP implementation of writable canned queries,
https://github.com/simonw/datasette/issues/790#issuecomment-637843494,https://api.github.com/repos/simonw/datasette/issues/790,637843494,MDEyOklzc3VlQ29tbWVudDYzNzg0MzQ5NA==,9599,simonw,2020-06-02T22:35:22Z,2020-06-02T22:35:22Z,OWNER,Message CSS is now demonstrated on https://latest.datasette.io/-/patterns,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/794#issuecomment-637841942,https://api.github.com/repos/simonw/datasette/issues/794,637841942,MDEyOklzc3VlQ29tbWVudDYzNzg0MTk0Mg==,9599,simonw,2020-06-02T22:30:17Z,2020-06-02T22:30:17Z,OWNER,Another demo: https://til.simonwillison.net/-/plugins,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629535669,Show hooks implemented by each plugin on /-/plugins,
https://github.com/simonw/datasette/issues/794#issuecomment-637832651,https://api.github.com/repos/simonw/datasette/issues/794,637832651,MDEyOklzc3VlQ29tbWVudDYzNzgzMjY1MQ==,9599,simonw,2020-06-02T22:10:34Z,2020-06-02T22:10:34Z,OWNER,"Demo: https://latest.datasette.io/-/plugins?all=1
```json
[
{
""name"": ""datasette.facets"",
""static"": false,
""templates"": false,
""version"": null,
""hooks"": [
""register_facet_classes""
]
},
{
""name"": ""datasette.publish.cloudrun"",
""static"": false,
""templates"": false,
""version"": null,
""hooks"": [
""publish_subcommand""
]
},
{
""name"": ""datasette.actor_auth_cookie"",
""static"": false,
""templates"": false,
""version"": null,
""hooks"": [
""actor_from_request""
]
},
{
""name"": ""datasette.default_permissions"",
""static"": false,
""templates"": false,
""version"": null,
""hooks"": [
""permission_allowed""
]
},
{
""name"": ""datasette.sql_functions"",
""static"": false,
""templates"": false,
""version"": null,
""hooks"": [
""prepare_connection""
]
},
{
""name"": ""datasette.publish.heroku"",
""static"": false,
""templates"": false,
""version"": null,
""hooks"": [
""publish_subcommand""
]
}
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629535669,Show hooks implemented by each plugin on /-/plugins,
https://github.com/simonw/datasette/issues/699#issuecomment-637819025,https://api.github.com/repos/simonw/datasette/issues/699,637819025,MDEyOklzc3VlQ29tbWVudDYzNzgxOTAyNQ==,9599,simonw,2020-06-02T21:34:31Z,2020-06-02T21:34:31Z,OWNER,I can close this issue once I've expanded out this page of documentation https://datasette.readthedocs.io/en/latest/authentication.html - and published at least one plugin and/or feature that takes advantage of this new mechanism.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/793#issuecomment-637813993,https://api.github.com/repos/simonw/datasette/issues/793,637813993,MDEyOklzc3VlQ29tbWVudDYzNzgxMzk5Mw==,9599,simonw,2020-06-02T21:22:50Z,2020-06-02T21:22:50Z,OWNER,"This is a minor security issue with `master` at the moment, but I'll resolve this before I ship the next release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629524205,CSRF protection for /-/messages tool and writable canned queries,
https://github.com/simonw/datasette/issues/790#issuecomment-637813616,https://api.github.com/repos/simonw/datasette/issues/790,637813616,MDEyOklzc3VlQ29tbWVudDYzNzgxMzYxNg==,9599,simonw,2020-06-02T21:22:02Z,2020-06-02T21:22:02Z,OWNER,"Debug tool is live here: https://latest.datasette.io/-/messages
Documentation is here: https://github.com/simonw/datasette/blob/master/docs/internals.rst#add_messagerequest-message-message_typedatasetteinfo","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-637793590,https://api.github.com/repos/simonw/datasette/issues/790,637793590,MDEyOklzc3VlQ29tbWVudDYzNzc5MzU5MA==,9599,simonw,2020-06-02T20:40:02Z,2020-06-02T20:40:02Z,OWNER,"From https://github.com/simonw/datasette/issues/698#issuecomment-621037724
> Concept for displaying a success message:
>
> ![]()
> CSS:
>
> ```css
> .success {
> padding: 1em;
> border: 1px solid green;
> background-color: #c7fbc7;
> }","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-637790860,https://api.github.com/repos/simonw/datasette/issues/790,637790860,MDEyOklzc3VlQ29tbWVudDYzNzc5MDg2MA==,9599,simonw,2020-06-02T20:34:15Z,2020-06-02T20:34:15Z,OWNER,The `/-/messages` debug tool will need CSRF protection or people will be able to add messages using a hidden form on another website.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-637708090,https://api.github.com/repos/simonw/datasette/issues/790,637708090,MDEyOklzc3VlQ29tbWVudDYzNzcwODA5MA==,9599,simonw,2020-06-02T17:52:30Z,2020-06-02T17:52:30Z,OWNER,I need to make sure that any time cookies are set there's no cache-control header (or it is set to private).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-637699337,https://api.github.com/repos/simonw/datasette/issues/790,637699337,MDEyOklzc3VlQ29tbWVudDYzNzY5OTMzNw==,9599,simonw,2020-06-02T17:34:47Z,2020-06-02T17:34:47Z,OWNER,"I'm going to use a output renderer plugin to test this, since then my unit tests can run against custom code that both sets and displays messages.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-637066496,https://api.github.com/repos/simonw/datasette/issues/790,637066496,MDEyOklzc3VlQ29tbWVudDYzNzA2NjQ5Ng==,9599,simonw,2020-06-01T19:48:20Z,2020-06-01T19:48:20Z,OWNER,"I'm going to stash these on the `request` object after all, so the memory used for the messages gets automatically cleaned up at the end of the request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-637009509,https://api.github.com/repos/simonw/datasette/issues/790,637009509,MDEyOklzc3VlQ29tbWVudDYzNzAwOTUwOQ==,9599,simonw,2020-06-01T17:44:55Z,2020-06-01T17:46:18Z,OWNER,"Problem with `datasette.fetch_and_clear_messages(request, response)` is that I want to call it from the template (so we only clear messages if they have been displayed) - but by that point in the code the `Response` object has not yet been created, so it can't have cookie set on it to clear the list of messages.
Solution: call it `datasette.show_messages(request)` and have it update internal state on the `datasette` object such that a later call to `write_messages_to_response(request, response)` knows to clear the cookie.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636978065,https://api.github.com/repos/simonw/datasette/issues/790,636978065,MDEyOklzc3VlQ29tbWVudDYzNjk3ODA2NQ==,9599,simonw,2020-06-01T16:42:59Z,2020-06-01T17:44:12Z,OWNER,"`datasette.add_message(request, message, type=datasette.INFO)`
Then later:
`datasette.write_messages_to_response(request, response)`
Writes the messages as cookies in the response.
`datasette.fetch_and_clear_messages(request, response)`
To display messages and clears them from the response.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/791#issuecomment-636973355,https://api.github.com/repos/simonw/datasette/issues/791,636973355,MDEyOklzc3VlQ29tbWVudDYzNjk3MzM1NQ==,9599,simonw,2020-06-01T16:33:33Z,2020-06-01T16:33:33Z,OWNER,"A fun thing about this tutorial is that it can start with a classic, basic todo list - and then start growing all kinds of outlandish features to help demonstrate various Datasette plugins and approaches.
Your TODOs on a map. URLs in TODOs that have been unfurled. Tag your TODOs and browse them with facets. Vega graphs showing your progress. Etc etc etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628572716,Tutorial: building a something-interesting with writable canned queries,
https://github.com/simonw/datasette/issues/790#issuecomment-636959774,https://api.github.com/repos/simonw/datasette/issues/790,636959774,MDEyOklzc3VlQ29tbWVudDYzNjk1OTc3NA==,9599,simonw,2020-06-01T16:15:33Z,2020-06-01T16:15:33Z,OWNER,It would be neat if this was driven by a method on `datasette` just because that's already the object passed to plugins as a documented API.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636934016,https://api.github.com/repos/simonw/datasette/issues/790,636934016,MDEyOklzc3VlQ29tbWVudDYzNjkzNDAxNg==,9599,simonw,2020-06-01T15:49:26Z,2020-06-01T15:49:26Z,OWNER,"Flask and Django both support ""types"" of message - info, warning etc. I think I should do the same.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636925354,https://api.github.com/repos/simonw/datasette/issues/790,636925354,MDEyOklzc3VlQ29tbWVudDYzNjkyNTM1NA==,9599,simonw,2020-06-01T15:32:02Z,2020-06-01T15:32:02Z,OWNER,"If `scope` had an immutable correlation ID I could use that with a dict somewhere mapping `correlation_id` to a messages object.
The problem then is how do I know to clean up the memory used by that dictionary when the request flows out of the system? I guess the code that updates the cookies in the response could do that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636922104,https://api.github.com/repos/simonw/datasette/issues/790,636922104,MDEyOklzc3VlQ29tbWVudDYzNjkyMjEwNA==,9599,simonw,2020-06-01T15:25:39Z,2020-06-01T15:25:39Z,OWNER,"What if I use a mutable key on `scope` to track messages for the duration of the request? Is that an OK thing to do?
ASGI spec says this: https://asgi.readthedocs.io/en/latest/specs/main.html#middleware
> ### Middleware
>
> It is possible to have ASGI ""middleware"" - code that plays the role of both server and application, taking in a scope and the send/receive awaitables, potentially modifying them, and then calling an inner application.
>
> When middleware is modifying the scope, it should make a copy of the scope object before mutating it and passing it to the inner application, as changes may leak upstream otherwise. In particular, you should not assume that the copy of the scope you pass down to the application is the one that it ends up using, as there may be other middleware in the way; thus, do not keep a reference to it and try to mutate it outside of the initial ASGI constructor callable that receives `scope`. Your one and only chance to add to it is before you hand control to the child application.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636920304,https://api.github.com/repos/simonw/datasette/issues/790,636920304,MDEyOklzc3VlQ29tbWVudDYzNjkyMDMwNA==,9599,simonw,2020-06-01T15:22:15Z,2020-06-01T15:22:15Z,OWNER,"Here's how the Django stuff works: https://github.com/django/django/blob/master/django/contrib/messages/storage/base.py
Notably the messages are mostly dealt with on the request object, with a piece of middleware that reads from the request and modifies the response (to set or clear cookies) right at the end: https://github.com/django/django/blob/master/django/contrib/messages/middleware.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636916107,https://api.github.com/repos/simonw/datasette/issues/790,636916107,MDEyOklzc3VlQ29tbWVudDYzNjkxNjEwNw==,9599,simonw,2020-06-01T15:14:30Z,2020-06-01T15:15:52Z,OWNER,"Alternative: `datasette.add_message(message)` and `datasette.read_and_clear_messages()` - these would need some kind of dark magic to ensure that the message was associated with the current request flowing through the system though, since that `datasette` object is shared my multiple concurrent requests.
Maybe use a request correlation ID that gets added to the scope? This is all getting a bit messy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636915499,https://api.github.com/repos/simonw/datasette/issues/790,636915499,MDEyOklzc3VlQ29tbWVudDYzNjkxNTQ5OQ==,9599,simonw,2020-06-01T15:13:40Z,2020-06-01T15:13:40Z,OWNER,"Maybe two utility functions:
`add_message(request, message)` - adds a Flash message (will be set later)
`read_and_clear_messages(request)` - reads messages and sets them to be cleared
Problem: the `request` object isn't created at the very top of the stack - it's actually created within each view. So maybe I need to move its creation up to the top of the routing stuff so that the code that returns the response can see it?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636912730,https://api.github.com/repos/simonw/datasette/issues/790,636912730,MDEyOklzc3VlQ29tbWVudDYzNjkxMjczMA==,9599,simonw,2020-06-01T15:08:13Z,2020-06-01T15:08:13Z,OWNER,"I'm going to build the first version of this with signed cookies.
I'm inclined to do this all on the request object, since it's the object representing the current request as it flows through the application. I need the ability to remember which messages were set and which need to be cleared, so I need to do that on something that is available for the lifetime of the request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636908972,https://api.github.com/repos/simonw/datasette/issues/790,636908972,MDEyOklzc3VlQ29tbWVudDYzNjkwODk3Mg==,9599,simonw,2020-06-01T15:01:00Z,2020-06-01T15:01:00Z,OWNER,"Setting messages just needs access to the response. Reading messages needs access to both request AND response, since it needs to clear the messages that are being displayed.
That's if the messages are persisted exclusively in cookies - which makes sense for Django since it's designed to run as many different load-balanced processes.
Since Datasette is a single process which can access an on-file database, maybe consider storing the flash messages within Datasette memory itself - a sort of session mechanism?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636906773,https://api.github.com/repos/simonw/datasette/issues/790,636906773,MDEyOklzc3VlQ29tbWVudDYzNjkwNjc3Mw==,9599,simonw,2020-06-01T14:57:02Z,2020-06-01T14:58:14Z,OWNER,"Actually I'm inclined to use cookies now, ala Django: https://docs.djangoproject.com/en/3.0/ref/contrib/messages/
> This class stores the message data in a cookie (signed with a secret hash to prevent manipulation) to persist notifications across requests. Old messages are dropped if the cookie data size would exceed 2048 bytes.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636906581,https://api.github.com/repos/simonw/datasette/issues/790,636906581,MDEyOklzc3VlQ29tbWVudDYzNjkwNjU4MQ==,9599,simonw,2020-06-01T14:56:42Z,2020-06-01T14:56:42Z,OWNER,I can use the new signed values support from #785 to help build this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/698#issuecomment-636617140,https://api.github.com/repos/simonw/datasette/issues/698,636617140,MDEyOklzc3VlQ29tbWVudDYzNjYxNzE0MA==,9599,simonw,2020-06-01T05:14:39Z,2020-06-01T05:14:39Z,OWNER,Here's the new `default_permissions.py` file I can add this permission check to: https://github.com/simonw/datasette/blob/dfdbdf378aba9afb66666f66b78df2f2069d2595/datasette/default_permissions.py#L1-L7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/788#issuecomment-636616638,https://api.github.com/repos/simonw/datasette/issues/788,636616638,MDEyOklzc3VlQ29tbWVudDYzNjYxNjYzOA==,9599,simonw,2020-06-01T05:12:30Z,2020-06-01T05:12:30Z,OWNER,"Looks like this (at the moment):
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628121234, /-/permissions debugging tool,
https://github.com/simonw/datasette/issues/789#issuecomment-636616307,https://api.github.com/repos/simonw/datasette/issues/789,636616307,MDEyOklzc3VlQ29tbWVudDYzNjYxNjMwNw==,9599,simonw,2020-06-01T05:11:03Z,2020-06-01T05:11:03Z,OWNER,Or I could get fancy and implement my own `pm.trace.root.setwriter` function which collects data that can be appended to the `?_trace=1` dump.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628156527,Mechanism for enabling pluggy tracing,
https://github.com/simonw/datasette/issues/789#issuecomment-636616155,https://api.github.com/repos/simonw/datasette/issues/789,636616155,MDEyOklzc3VlQ29tbWVudDYzNjYxNjE1NQ==,9599,simonw,2020-06-01T05:10:27Z,2020-06-01T05:10:27Z,OWNER,Easiest way to do this would be with an environment variable.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628156527,Mechanism for enabling pluggy tracing,
https://github.com/simonw/datasette/issues/786#issuecomment-636614062,https://api.github.com/repos/simonw/datasette/issues/786,636614062,MDEyOklzc3VlQ29tbWVudDYzNjYxNDA2Mg==,9599,simonw,2020-06-01T05:02:18Z,2020-06-01T05:02:18Z,OWNER,The skeleton of this page now exists at https://datasette.readthedocs.io/en/latest/authentication.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628087971,Documentation page describing Datasette's authentication system,
https://github.com/simonw/datasette/issues/788#issuecomment-636598949,https://api.github.com/repos/simonw/datasette/issues/788,636598949,MDEyOklzc3VlQ29tbWVudDYzNjU5ODk0OQ==,9599,simonw,2020-06-01T03:53:00Z,2020-06-01T03:53:00Z,OWNER,I can use a deque with a max length for this: https://docs.python.org/3/library/collections.html#deque-objects,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628121234, /-/permissions debugging tool,
https://github.com/simonw/datasette/issues/699#issuecomment-636576603,https://api.github.com/repos/simonw/datasette/issues/699,636576603,MDEyOklzc3VlQ29tbWVudDYzNjU3NjYwMw==,9599,simonw,2020-06-01T02:13:26Z,2020-06-01T03:13:31Z,OWNER,"Debugging tool idea: `/-/permissions` page which shows you the actor and lets you type in the strings for `action`, `resource_type` and `resource_identifier` - then shows you EVERY plugin hook that would have executed and what it would have said, plus when the chain would have terminated.
Bonus: if you're logged in as the `root` user (or a user that matches some kind of permission check, maybe a check for `permissions_debug`) you get to see a rolling log of the last 30 permission checks and what the results were across the whole of Datasette. This should make figuring out permissions policies a whole lot easier.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636576252,https://api.github.com/repos/simonw/datasette/issues/699,636576252,MDEyOklzc3VlQ29tbWVudDYzNjU3NjI1Mg==,9599,simonw,2020-06-01T02:11:40Z,2020-06-01T02:11:40Z,OWNER,"Plugin idea: `datasette-allow-all` - really simple plugin which just says ""yes"" to every permission check.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/698#issuecomment-636569917,https://api.github.com/repos/simonw/datasette/issues/698,636569917,MDEyOklzc3VlQ29tbWVudDYzNjU2OTkxNw==,9599,simonw,2020-06-01T01:39:44Z,2020-06-01T01:39:44Z,OWNER,"Idea for the authentication piece: I'll have the canned query code execute the following:
```python
if await datasette.permission_allowed(
request.scope.get(""actor""), ""execute_query"", ""canned_query"", query_name, default=True
):
```
Then I'll add a default plugin to Datasette which implements that plugin hook, looks at the Datasette metadata for that query, and says ""No"" if the following (and `request.scope[""actor""]` is empty):
```json
{
""databases"": {
""my-database"": {
""queries"": {
""add_twitter_handle"": {
""sql"": ""insert into twitter_handles (username) values (:username)"",
""write"": true,
""requires_actor"": true
}
}
}
}
}
```
I think I'll support this too:
```json
""allowed_actors"": [""root""]
```
So you can configure queries to only be available to specific `{""id"": xxx}` actors.
This will be the first time the new `permission_allowed` mechanism from #699 will be exercised in Datasette core.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/699#issuecomment-636566616,https://api.github.com/repos/simonw/datasette/issues/699,636566616,MDEyOklzc3VlQ29tbWVudDYzNjU2NjYxNg==,9599,simonw,2020-06-01T01:23:48Z,2020-06-01T01:23:48Z,OWNER,https://latest.datasette.io/-/actor is now live (it returns `null` because there's no current way to sign into the `latest.datasette.io` site - not even with a fake `ds_actor` cookie because there's no way to know what that site's random secret is).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636566433,https://api.github.com/repos/simonw/datasette/issues/699,636566433,MDEyOklzc3VlQ29tbWVudDYzNjU2NjQzMw==,9599,simonw,2020-06-01T01:22:59Z,2020-06-01T01:22:59Z,OWNER,"Some next steps:
- Try out a branch of `datasette-auth-github` that builds on these new plugin hooks
- Build a `datasette-api-tokens` plugin which implements `Authorization: bearer xxx` token support for API access
- Maybe prototype up a `datasette-user-accounts` plugin which supports username/password accounts and allows an admin user to create/delete them
- Do more work on writable canned queries in #698 and see what they look like if they take advantage of the permissions hook (to restrict some to only allowing authenticated users)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636565610,https://api.github.com/repos/simonw/datasette/issues/699,636565610,MDEyOklzc3VlQ29tbWVudDYzNjU2NTYxMA==,9599,simonw,2020-06-01T01:19:45Z,2020-06-01T01:19:45Z,OWNER,I rebased in #783 so all of this is on master now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/784#issuecomment-636565242,https://api.github.com/repos/simonw/datasette/issues/784,636565242,MDEyOklzc3VlQ29tbWVudDYzNjU2NTI0Mg==,9599,simonw,2020-06-01T01:18:20Z,2020-06-01T01:18:20Z,OWNER,I'm considering this done. I'm going to leave it to plugins to implement a web-based sign-in flow for accounts (at least for the moment).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628003707,Ability to sign in to Datasette as a root account,
https://github.com/simonw/datasette/issues/699#issuecomment-636562999,https://api.github.com/repos/simonw/datasette/issues/699,636562999,MDEyOklzc3VlQ29tbWVudDYzNjU2Mjk5OQ==,9599,simonw,2020-06-01T01:09:47Z,2020-06-01T01:09:47Z,OWNER,I should add an entire page to the documentation describing Datasette authentication.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636562658,https://api.github.com/repos/simonw/datasette/issues/699,636562658,MDEyOklzc3VlQ29tbWVudDYzNjU2MjY1OA==,9599,simonw,2020-06-01T01:08:20Z,2020-06-01T01:08:54Z,OWNER,"OK, the implementation in PR #783 is in a good state now - it implements the new plugin hooks with tests and documentation, plus it implements this:
$ datasette . --root
http://127.0.0.1:8001/-/auth-token?token=3ca9ee460a6451142389351d19b147bce27d2a785dfb6b5a74f82211be1ede49
...
That URL, when clicked, will set a cookie for the `{""id"": ""root""}` user. The cookie is respected and used to populate `scope[""actor""]`.
I'm going to merge that pull request and continue working on this stuff on master.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/784#issuecomment-636554258,https://api.github.com/repos/simonw/datasette/issues/784,636554258,MDEyOklzc3VlQ29tbWVudDYzNjU1NDI1OA==,9599,simonw,2020-06-01T00:21:33Z,2020-06-01T00:21:33Z,OWNER,"The URL for this will be:
`/-/auth-token?token=xxx`
The token will be generated by Datasette on startup and will only be valid for a single request, at which point it will be used to set a signed `ds_actor` cookie and then redirect to the homepage.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628003707,Ability to sign in to Datasette as a root account,
https://github.com/simonw/datasette/issues/785#issuecomment-636553736,https://api.github.com/repos/simonw/datasette/issues/785,636553736,MDEyOklzc3VlQ29tbWVudDYzNjU1MzczNg==,9599,simonw,2020-06-01T00:18:40Z,2020-06-01T00:18:40Z,OWNER,That documentation: https://github.com/simonw/datasette/blob/c818de88a9c2683437875f788e325d911c8b767b/docs/config.rst#configuring-the-secret,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636541827,https://api.github.com/repos/simonw/datasette/issues/785,636541827,MDEyOklzc3VlQ29tbWVudDYzNjU0MTgyNw==,9599,simonw,2020-05-31T22:46:34Z,2020-06-01T00:17:35Z,OWNER,This is nearly ready to close. I'm going to add documentation for `--secret` and the `DATASETTE_SECRET` environment variable.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636541929,https://api.github.com/repos/simonw/datasette/issues/785,636541929,MDEyOklzc3VlQ29tbWVudDYzNjU0MTkyOQ==,9599,simonw,2020-05-31T22:47:17Z,2020-05-31T22:47:17Z,OWNER,I'll add a section about secrets to this page: https://datasette.readthedocs.io/en/latest/config.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636541630,https://api.github.com/repos/simonw/datasette/issues/785,636541630,MDEyOklzc3VlQ29tbWVudDYzNjU0MTYzMA==,9599,simonw,2020-05-31T22:45:07Z,2020-05-31T22:45:07Z,OWNER,Documentation for those new methods: https://github.com/simonw/datasette/blob/e28207e76ec3b26b2c396370fd3fb325a60bfd49/docs/internals.rst#signvalue-namespacedefault,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636539295,https://api.github.com/repos/simonw/datasette/issues/785,636539295,MDEyOklzc3VlQ29tbWVudDYzNjUzOTI5NQ==,9599,simonw,2020-05-31T22:24:14Z,2020-05-31T22:28:27Z,OWNER,"I'll add two utility methods to the Datasette class:
- `datasette.sign(value, ""namespace"")` - returns signed string
- `datasette.unsign(signed, ""namespace"")` - returns value OR raises `BadSignature`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636538298,https://api.github.com/repos/simonw/datasette/issues/785,636538298,MDEyOklzc3VlQ29tbWVudDYzNjUzODI5OA==,9599,simonw,2020-05-31T22:14:43Z,2020-05-31T22:15:01Z,OWNER,"... actually no I'll do it using a CLI option that can also be in an environment variable:
https://click.palletsprojects.com/en/7.x/options/#values-from-environment-variables
```python
@click.command()
@click.option('--secret', envvar='DATASETTE_SECRET')
def greet(secret):
...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636537921,https://api.github.com/repos/simonw/datasette/issues/785,636537921,MDEyOklzc3VlQ29tbWVudDYzNjUzNzkyMQ==,9599,simonw,2020-05-31T22:11:29Z,2020-05-31T22:11:29Z,OWNER,First version of cookie signing will use a secret that is either pulled from `DATASETTE_SECRET` environment variable or generated every time the server starts. I'll add a non-environment-variable based secret later.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636537679,https://api.github.com/repos/simonw/datasette/issues/785,636537679,MDEyOklzc3VlQ29tbWVudDYzNjUzNzY3OQ==,9599,simonw,2020-05-31T22:09:23Z,2020-05-31T22:09:23Z,OWNER,"I'm going to use https://github.com/pallets/itsdangerous for this.
Annoyingly they're very close to release v2.0 which adds support for key rotation... but it's not quite out of pre-release yet. I'll go with 1.1.0 for the moment and upgrade to 2.0 as soon as that is out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636515763,https://api.github.com/repos/simonw/datasette/issues/785,636515763,MDEyOklzc3VlQ29tbWVudDYzNjUxNTc2Mw==,9599,simonw,2020-05-31T19:19:03Z,2020-05-31T19:19:13Z,OWNER,Maybe Datasette should have a `--secrets=path/to/secrets.json` command-line option for storing these?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636515671,https://api.github.com/repos/simonw/datasette/issues/785,636515671,MDEyOklzc3VlQ29tbWVudDYzNjUxNTY3MQ==,9599,simonw,2020-05-31T19:18:18Z,2020-05-31T19:18:18Z,OWNER,That `user_state_dir` solution may have been more trouble than it was worth though - I seem to remember it causing issues on some hosting providers.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636515599,https://api.github.com/repos/simonw/datasette/issues/785,636515599,MDEyOklzc3VlQ29tbWVudDYzNjUxNTU5OQ==,9599,simonw,2020-05-31T19:17:43Z,2020-05-31T19:17:43Z,OWNER,"I previously solved this for the `datasette-auth-existing-cookies` plugin as described in this issue: https://github.com/simonw/datasette-auth-existing-cookies/issues/1
> Concrete plan: you have to pass a secret to the class constructor. The Datasette plugin (the code in `__init__.py`) uses the following in order of preference (first things are most preferred):
>
> - A plugin configuration option called `cookie_secret` - which can be protected by this mechanism: https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values
> - A JSON configuration file in the `user_state_dir` file, if it exists
> - If that does not exist, a secret is generated and written to that JSON file
>
> I originally planned to have separate support for an environment variable, but the existence of the [secret configuration values](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values) mechanism means this is already handled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/784#issuecomment-636514974,https://api.github.com/repos/simonw/datasette/issues/784,636514974,MDEyOklzc3VlQ29tbWVudDYzNjUxNDk3NA==,9599,simonw,2020-05-31T19:12:48Z,2020-05-31T19:12:48Z,OWNER,"For the first version of this I'm not going to use passwords at all. I'll implement this:
$ datasette fixtures.db --root
The `--root` option will cause Datasette to output a URL with a one-time-use token in it which, when clicked, will authenticate the user as the root account (by setting a signed cookie).
Signed cookie means Datasette needs a secrets recipe. I'll open a new issue for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628003707,Ability to sign in to Datasette as a root account,
https://github.com/simonw/datasette/issues/784#issuecomment-636510838,https://api.github.com/repos/simonw/datasette/issues/784,636510838,MDEyOklzc3VlQ29tbWVudDYzNjUxMDgzOA==,9599,simonw,2020-05-31T18:39:08Z,2020-05-31T18:39:08Z,OWNER,"I'm calling this the `root` account now, for reasons discussed in these two comments: https://github.com/simonw/datasette/issues/699#issuecomment-636510647","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628003707,Ability to sign in to Datasette as a root account,
https://github.com/simonw/datasette/issues/699#issuecomment-636510761,https://api.github.com/repos/simonw/datasette/issues/699,636510761,MDEyOklzc3VlQ29tbWVudDYzNjUxMDc2MQ==,9599,simonw,2020-05-31T18:38:30Z,2020-05-31T18:38:30Z,OWNER,"I quite like `root` - it supports the idea that best practice is to NOT do things as the root account, but to use a plugin to set up separate accounts for different purposes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636510647,https://api.github.com/repos/simonw/datasette/issues/699,636510647,MDEyOklzc3VlQ29tbWVudDYzNjUxMDY0Nw==,9599,simonw,2020-05-31T18:37:39Z,2020-05-31T18:37:39Z,OWNER,Maybe the default single account should be called something other than `admin`? The problem with `admin` is that it sounds like more of a role - in larger installations one can expect multiple admins. `root` may be better since there's clearly only one root account. Bit of a technical term though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636510398,https://api.github.com/repos/simonw/datasette/issues/699,636510398,MDEyOklzc3VlQ29tbWVudDYzNjUxMDM5OA==,9599,simonw,2020-05-31T18:35:57Z,2020-05-31T18:36:05Z,OWNER,Again I will use exploratory prototyping to inform a decision on the minimum subset design for the `actor` dictionary.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636510303,https://api.github.com/repos/simonw/datasette/issues/699,636510303,MDEyOklzc3VlQ29tbWVudDYzNjUxMDMwMw==,9599,simonw,2020-05-31T18:35:17Z,2020-05-31T18:35:17Z,OWNER,"Keeping the structure of the actor dictionary completely undefined doesn't make sense if Datasette is going to ship with a default authentication mechanism for admin users.
I'm going to define a small set of required keys for the actor dictionary, and enforce them in code.
But which keys? I feel I need a unique key representing the identity of the actor, plus a key that can be displayed in the ""You are logged in as X"" navigation. Maybe these are the same key?
So the single required key could be `id`. Problem is: is that a string or an integer? Some use-cases may call for an integer, which matches to how SQLite auto incrementing primary keys work. `admin` is a string.
Maybe `id` is required, `name` is optional - but if `name` is present then the ""You are logged in as..."" uses that in preference to `id`. `id` has to be a string, and if you want to store integer IDs in your database you need to remember to convert them to a string in your `actor_from_request` implementation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636498913,https://api.github.com/repos/simonw/datasette/issues/699,636498913,MDEyOklzc3VlQ29tbWVudDYzNjQ5ODkxMw==,9599,simonw,2020-05-31T17:04:50Z,2020-05-31T17:06:40Z,OWNER,"This also means some writable canned queries can allow writes from unauthenticated users (for stuff like feedback forms), while others can require an authenticated user - all with core Datasette without any plugins needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636499075,https://api.github.com/repos/simonw/datasette/issues/699,636499075,MDEyOklzc3VlQ29tbWVudDYzNjQ5OTA3NQ==,9599,simonw,2020-05-31T17:06:09Z,2020-05-31T17:06:09Z,OWNER,"I believe that this plugin hook design is flexible enough that role-based permissions could be built on top of it as a separate plugin.
Would be good to check that with a proof of concept though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636498770,https://api.github.com/repos/simonw/datasette/issues/699,636498770,MDEyOklzc3VlQ29tbWVudDYzNjQ5ODc3MA==,9599,simonw,2020-05-31T17:03:38Z,2020-05-31T17:03:38Z,OWNER,"I'm going to draw the line here: default Datasette supports authentication but only for a single user account (""admin""). Plugins can then add support for multiple user accounts, social auth, SSO etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636495124,https://api.github.com/repos/simonw/datasette/issues/699,636495124,MDEyOklzc3VlQ29tbWVudDYzNjQ5NTEyNA==,9599,simonw,2020-05-31T16:36:08Z,2020-05-31T16:36:08Z,OWNER,HTTP Basic auth would be a good default option. No need to build a custom login UI for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636495005,https://api.github.com/repos/simonw/datasette/issues/699,636495005,MDEyOklzc3VlQ29tbWVudDYzNjQ5NTAwNQ==,9599,simonw,2020-05-31T16:35:10Z,2020-05-31T16:35:26Z,OWNER,I think I want to keep full username/password authentication against a database table as a plugin. I'll experiment with Jupyter-style URLs as a starting point.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636494374,https://api.github.com/repos/simonw/datasette/issues/699,636494374,MDEyOklzc3VlQ29tbWVudDYzNjQ5NDM3NA==,9599,simonw,2020-05-31T16:29:48Z,2020-05-31T16:29:48Z,OWNER,"If Datasette were to support authentication out-of-the-box, without plugins (which makes more sense with writable canned queries, #698) what would that look like?
Some options:
- Jupyter notebook style: output a magic URL on the console with a one-time token to authenticate the user as an ""admin""
- Really simple password authentication - via an environment variable perhaps?
- SQL based authentication: I was going to do this as a plugin, but maybe it should be default? A way of configuring a SQL query which can be used to authenticate a user based on their username and password.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636395263,https://api.github.com/repos/simonw/datasette/issues/699,636395263,MDEyOklzc3VlQ29tbWVudDYzNjM5NTI2Mw==,9599,simonw,2020-05-30T22:54:09Z,2020-05-30T22:54:09Z,OWNER,"Idea: add a `/-/actor.json` special page which JSON dumps out the current `request.scope[""actor""]` - so you can easily test how your request has been authenticated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636393204,https://api.github.com/repos/simonw/datasette/issues/699,636393204,MDEyOklzc3VlQ29tbWVudDYzNjM5MzIwNA==,9599,simonw,2020-05-30T22:29:44Z,2020-05-30T22:30:15Z,OWNER,"Robust testing of permissions is really important. I should think about utilities I may be able to add to Datasette's unit testing tools that make it as easy as possible to confirm which permission checks were carried out on a specific HTTP request.
That way I can set a good example that any Datasette plugin which makes permission checks can follow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636392850,https://api.github.com/repos/simonw/datasette/issues/699,636392850,MDEyOklzc3VlQ29tbWVudDYzNjM5Mjg1MA==,9599,simonw,2020-05-30T22:25:19Z,2020-05-30T22:25:19Z,OWNER,The branch is now usable! Next step: write some experimental plugins that exercise some real authentication use-cases with it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636391331,https://api.github.com/repos/simonw/datasette/issues/699,636391331,MDEyOklzc3VlQ29tbWVudDYzNjM5MTMzMQ==,9599,simonw,2020-05-30T22:08:21Z,2020-05-30T22:08:21Z,OWNER,"I'm going to add an awaitable utility method to the Datasette class for checking permissions:
await datasette.permission_allowed(actor, action, resource_type, resource_identifier)
The second two arguments will be optional.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636388288,https://api.github.com/repos/simonw/datasette/issues/699,636388288,MDEyOklzc3VlQ29tbWVudDYzNjM4ODI4OA==,9599,simonw,2020-05-30T21:34:50Z,2020-05-30T21:34:50Z,OWNER,"Debugging permissions is going to be important. Optional tooling that supports the following would be useful:
- Log every check to `permission_allowed` to the console - optionally with tracebacks showing where in the code the check was made
- Log every check to the https://latest.datasette.io/?_trace=1 output
- A tool that shows you exactly what permissions the current authenticated user/entity has
- A tool showing all available permissions
That last one is tricky if permissions are just strings that might be passed to `permission_allowed` - so maybe there needs to be a plugin hook that lets plugins register their permissions, such that they can be introspected later on? A `register_permission_actions()` hook that returns a list of permission action strings (or objects of some sort) perhaps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636379067,https://api.github.com/repos/simonw/datasette/issues/699,636379067,MDEyOklzc3VlQ29tbWVudDYzNjM3OTA2Nw==,9599,simonw,2020-05-30T20:12:47Z,2020-05-30T20:40:42Z,OWNER,"I could bake some permission checks into default Datasette, which are all treated as allow by default but can then be locked down by plugins. Maybe the following:
permission_allowed(request.actor, ""execute-sql"", ""database"", ""name-of-database"")
Checks that current user can execute arbitrary SQL queries against a specific database (or use the `?_where=` feature). Equivalent to current [allow_sql](https://datasette.readthedocs.io/en/0.43/config.html#allow-sql) setting.
permission_allowed(request.actor, ""download-database"", ""database"", ""name-of-database"")
Can the user download the database file? Like [allow_download](https://datasette.readthedocs.io/en/0.43/config.html#allow-download).
Maybe one for [allow_csv_stream](https://datasette.readthedocs.io/en/0.43/config.html#allow-csv-stream) too.
Having a permission check (defaulting to True) on every single ""view"" would be useful:
- view_index
- view_database
- view_table
- view_row
- view_query
- view_special (for `/-/versions` and so on)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636381732,https://api.github.com/repos/simonw/datasette/issues/699,636381732,MDEyOklzc3VlQ29tbWVudDYzNjM4MTczMg==,9599,simonw,2020-05-30T20:32:11Z,2020-05-30T20:39:11Z,OWNER,I started sketching this out in the [authentication](https://github.com/simonw/datasette/tree/authentication) branch. Here's the documentation so far: https://github.com/simonw/datasette/blob/8871c20/docs/plugins.rst#actor_from_requestdatasette-request,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636376209,https://api.github.com/repos/simonw/datasette/issues/699,636376209,MDEyOklzc3VlQ29tbWVudDYzNjM3NjIwOQ==,9599,simonw,2020-05-30T19:53:28Z,2020-05-30T20:09:10Z,OWNER,"I think there are two hooks here:
`actor_from_request(datasette, request)` - returns `None` or a dictionary.
- `datasette` is a Datasette instance - useful for things like reading plugin configuration or executing queries
- `request` is a [Request object](https://datasette.readthedocs.io/en/latest/internals.html#request-object) - which means ASGI scope can be accessed as `request.scope`
A non-None value means the request is authenticated in some way. The shape of that dictionary is entirely undefined.
The second hook is for checking permissions. It can look something like this:
`permission_allowed(actor, action, resource_type, resource_identifier)`
- `actor` = the dictionary that was returned by `actor_from_scope`
- `action` = a string representing the action to be performed, e.g. `edit-schema`
- `resource_type` = a string representing the type of resource being acted on, e.g. `table`
- `resource_identifier` = a string (or maybe tuple?) representing the specific resource, e.g. the table name
I don't know if Datasette should provide default implementations of these hooks. It may be that leaving them completely up to plugins is the way to go.
I think I need to prototype this quickly to start feeling for how well it might work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636376893,https://api.github.com/repos/simonw/datasette/issues/699,636376893,MDEyOklzc3VlQ29tbWVudDYzNjM3Njg5Mw==,9599,simonw,2020-05-30T19:57:54Z,2020-05-30T20:09:05Z,OWNER,"`auth_from_scope(datasette, scope)` needs to be able to return an awaitable which is then awaited - so it can execute database queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636376974,https://api.github.com/repos/simonw/datasette/issues/699,636376974,MDEyOklzc3VlQ29tbWVudDYzNjM3Njk3NA==,9599,simonw,2020-05-30T19:58:40Z,2020-05-30T20:08:59Z,OWNER,"Maybe call that `actor_from_request(datasette, request)` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636378228,https://api.github.com/repos/simonw/datasette/issues/699,636378228,MDEyOklzc3VlQ29tbWVudDYzNjM3ODIyOA==,9599,simonw,2020-05-30T20:07:25Z,2020-05-30T20:07:25Z,OWNER,"I like ""actor"" better than ""entity"" to mean ""the user or API key that is authenticated for this request"".
I'm going to use ""resource"" instead of ""subject"" - updating the design comment again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636378121,https://api.github.com/repos/simonw/datasette/issues/699,636378121,MDEyOklzc3VlQ29tbWVudDYzNjM3ODEyMQ==,9599,simonw,2020-05-30T20:06:47Z,2020-05-30T20:06:47Z,OWNER,"In AWS IAM world the following terminology is used: https://aws.amazon.com/iam/features/manage-permissions/
> Permissions are granted to IAM **entities** (users, groups, and roles) [...]
>
> To assign permissions to a user, group, role, or resource, you create a policy that lets you specify:
>
> * **Actions** – Which AWS service actions you allow. For example, you might allow a user to call the Amazon S3 ListBucket action. Any actions that you don't explicitly allow are denied.
> * **Resources** – Which AWS resources you allow the action on. For example, what Amazon S3 buckets will you allow the user to perform the ListBucket action on? Users cannot access any resources that you do not explicitly grant permissions to.
> * **Effect** – Whether to allow or deny access. Because access is denied by default, you typically write policies where the effect is to allow.
> * **Conditions** – Which conditions must be present for the policy to take effect. For example, you might allow access only to the specific S3 buckets if the user is connecting from a specific IP range or has used multi-factor authentication at login.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636377755,https://api.github.com/repos/simonw/datasette/issues/699,636377755,MDEyOklzc3VlQ29tbWVudDYzNjM3Nzc1NQ==,9599,simonw,2020-05-30T20:04:23Z,2020-05-30T20:04:23Z,OWNER,"My usage of the term `subject` here to mean ""the thing I am checking I have permission to interact with, e.g. a database table"" may be misleading. https://stackoverflow.com/questions/4989063/what-is-the-meaning-and-difference-between-subject-user-and-principal for example shows that JAAS (Java Authentication and Authorization Service) defines subject as ""The purpose of the Subject is to represent the authenticated user"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636377235,https://api.github.com/repos/simonw/datasette/issues/699,636377235,MDEyOklzc3VlQ29tbWVudDYzNjM3NzIzNQ==,9599,simonw,2020-05-30T20:00:42Z,2020-05-30T20:01:35Z,OWNER,I'm changing `auth` to `actor` and updating the above [design comment](https://github.com/simonw/datasette/issues/699#issuecomment-636376209).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/782#issuecomment-636370064,https://api.github.com/repos/simonw/datasette/issues/782,636370064,MDEyOklzc3VlQ29tbWVudDYzNjM3MDA2NA==,9599,simonw,2020-05-30T18:51:19Z,2020-05-30T18:51:19Z,OWNER,"https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_shape=array returns this:
```json
[
{
""pk1"": ""a"",
""pk2"": ""a"",
""pk3"": ""a"",
""content"": ""a-a-a""
},
{
""pk1"": ""a"",
""pk2"": ""a"",
""pk3"": ""b"",
""content"": ""a-a-b""
}
]
```
There's one big problem with this format: it doesn't provide any space for pagination information.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-636369978,https://api.github.com/repos/simonw/datasette/issues/782,636369978,MDEyOklzc3VlQ29tbWVudDYzNjM2OTk3OA==,9599,simonw,2020-05-30T18:50:31Z,2020-05-30T18:50:31Z,OWNER,"Here's the default JSON at the moment: https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2
```json
{
""database"": ""fixtures"",
""table"": ""compound_three_primary_keys"",
""is_view"": false,
""human_description_en"": """",
""rows"": [
[
""a"",
""a"",
""a"",
""a-a-a""
],
[
""a"",
""a"",
""b"",
""a-a-b""
]
],
""truncated"": false,
""filtered_table_rows_count"": 1001,
""expanded_columns"": [],
""expandable_columns"": [],
""columns"": [
""pk1"",
""pk2"",
""pk3"",
""content""
],
""primary_keys"": [
""pk1"",
""pk2"",
""pk3""
],
""units"": {},
""query"": {
""sql"": ""select pk1, pk2, pk3, content from compound_three_primary_keys order by pk1, pk2, pk3 limit 3"",
""params"": {}
},
""facet_results"": {},
""suggested_facets"": [
{
""name"": ""pk1"",
""toggle_url"": ""http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_facet=pk1""
},
{
""name"": ""pk2"",
""toggle_url"": ""http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_facet=pk2""
},
{
""name"": ""pk3"",
""toggle_url"": ""http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_facet=pk3""
}
],
""next"": ""a,a,b"",
""next_url"": ""http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_next=a%2Ca%2Cb"",
""query_ms"": 17.56119728088379,
""source"": ""tests/fixtures.py"",
""source_url"": ""https://github.com/simonw/datasette/blob/master/tests/fixtures.py"",
""license"": ""Apache License 2.0"",
""license_url"": ""https://github.com/simonw/datasette/blob/master/LICENSE""
}
```
There's a lot of stuff in there. This increases the risk that future minor changes might break existing API consumers.
It returns rows as a list of lists of values, and expects you to correlate these with the list of columns. I originally designed it like this because I thought this was a more efficient representation than repeating the column names in a dictionary for every row. With hindsight this was a bad optimization - I _always_ use `?shape=array` because it's more convenient, and gzip encoding of the response means there's no bandwidth saving. Users who want that efficiency should request it using a custom `?_shape=`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/576#issuecomment-636340618,https://api.github.com/repos/simonw/datasette/issues/576,636340618,MDEyOklzc3VlQ29tbWVudDYzNjM0MDYxOA==,9599,simonw,2020-05-30T14:46:04Z,2020-05-30T18:41:41Z,OWNER,"I should also think about the class properties (as opposed to methods) that are setup in the Datasette constructor. Many of these should be private, some should be documented.
- cache_headers
- cors
- databases
- executor
- files
- immutables
- inspect_data
- jinja_env
- max_returned_rows
- page_size
- plugins_dir
- renderers
- sql_time_limit_ms
- sqlite_extensions
- sqlite_functions
- static_mounts
- template_dir
- version_note
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/37#issuecomment-636360861,https://api.github.com/repos/simonw/datasette/issues/37,636360861,MDEyOklzc3VlQ29tbWVudDYzNjM2MDg2MQ==,9599,simonw,2020-05-30T17:29:20Z,2020-05-30T17:29:20Z,OWNER,I'm not going to do this: 2.5 years later I have yet to run into anything that makes me think that JSON serialization performance is worth any extra work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",268453968,Ability to serialize massive JSON without blocking event loop,
https://github.com/simonw/datasette/issues/498#issuecomment-636360574,https://api.github.com/repos/simonw/datasette/issues/498,636360574,MDEyOklzc3VlQ29tbWVudDYzNjM2MDU3NA==,9599,simonw,2020-05-30T17:26:02Z,2020-05-30T17:26:02Z,OWNER,"I released a plugin that implements an early version of this a while ago: https://github.com/simonw/datasette-search-all
I'm still thinking about how a plugin / separate tool could work that builds a single FTS index over the content from multiple tables such that you can run relevance-calculated queries against those multiple tables.
Since that's going to be a plugin / separate tool too, I'm closing this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",451513541,Full text search of all tables at once?,
https://github.com/simonw/datasette/issues/576#issuecomment-636332183,https://api.github.com/repos/simonw/datasette/issues/576,636332183,MDEyOklzc3VlQ29tbWVudDYzNjMzMjE4Mw==,9599,simonw,2020-05-30T13:37:51Z,2020-05-30T13:38:35Z,OWNER,"**\_\_init\_\_**(self, files, immutables=None, cache_headers=True, cors=False, inspect_data=None, metadata=None, sqlite_extensions=None, template_dir=None, plugins_dir=None, static_mounts=None, memory=False, config=None, version_note=None, config_dir=None)
`Initialize self. See help(type(self)) for accurate signature.`
**absolute_url**(self, request, path)
**add_database**(self, name, db)
**app**(self)
`Returns an ASGI app function that serves the whole of [Datasette](http://localhost:8066/datasette.app.html#Datasette)`
**app_css_hash**(self)
**config**(self, key)
**config_dict**(self)
**connected_databases**(self)
**execute**(self, db_name, sql, params=None, truncate=False, custom_time_limit=None, page_size=None, log_sql_errors=True)
**expand_foreign_keys**(self, database, table, column, values)
`Returns dict mapping (column, value) -> label`
**get_canned_queries**(self, database_name)
**get_canned_query**(self, database_name, query_name)
**metadata**(self, key=None, database=None, table=None, fallback=True)
`Looks up metadata, cascading backwards from specified level.\
Returns None if metadata value is not found.`
**plugin_config**(self, plugin_name, database=None, table=None, fallback=True)
`Return config for plugin, falling back from specified database/table`
**plugins**(self, show_all=False)
**prepare_connection**(self, conn, database)
**register_custom_units**(self)
`Register any custom units defined in the metadata.json with Pint`
**register_renderers**(self)
`Register output renderers which output data in custom formats.`
**remove_database**(self, name)
**render_template**(self, templates, context=None, request=None, view_name=None)
**table_metadata**(self, database, table)
`Fetch table-specific metadata.`
**threads**(self)
**update_with_inherited_metadata**(self, metadata)
**versions**(self)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/576#issuecomment-636332083,https://api.github.com/repos/simonw/datasette/issues/576,636332083,MDEyOklzc3VlQ29tbWVudDYzNjMzMjA4Mw==,9599,simonw,2020-05-30T13:36:55Z,2020-05-30T13:36:55Z,OWNER,"Here's the current `Datasette` class as introspected using the webserver run by `pydoc -p 8000`:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/576#issuecomment-636330843,https://api.github.com/repos/simonw/datasette/issues/576,636330843,MDEyOklzc3VlQ29tbWVudDYzNjMzMDg0Mw==,9599,simonw,2020-05-30T13:26:38Z,2020-05-30T13:26:38Z,OWNER,There's a bunch of methods on that class which I could add a `_` prefix to and leave undocumented. `datasette.register_renderers()` should be `datasette._register_renderers()` for example.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/576#issuecomment-636329095,https://api.github.com/repos/simonw/datasette/issues/576,636329095,MDEyOklzc3VlQ29tbWVudDYzNjMyOTA5NQ==,9599,simonw,2020-05-30T13:11:39Z,2020-05-30T13:25:10Z,OWNER,"I need to document the `.databases` property - both how to get access to specific databases and how to access the first/only database.
Idea: `datasette.get_database(name)` method (consistent with existing `.add_database()` and `.remove_database()` methods) - but `name` parameter is optional, returns first database if you omit that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/576#issuecomment-636330438,https://api.github.com/repos/simonw/datasette/issues/576,636330438,MDEyOklzc3VlQ29tbWVudDYzNjMzMDQzOA==,9599,simonw,2020-05-30T13:23:04Z,2020-05-30T13:23:04Z,OWNER,Need to document `datasette.metadata()` - see #780,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/576#issuecomment-586053947,https://api.github.com/repos/simonw/datasette/issues/576,586053947,MDEyOklzc3VlQ29tbWVudDU4NjA1Mzk0Nw==,9599,simonw,2020-02-14T01:29:48Z,2020-05-30T13:22:09Z,OWNER,"OK, I've made a start on this now in 3ffb8f3b98252531d11897fd431711e9b8045ace - still plenty more methods to document. More importantly that class has a LOT of junk methods on that no-one should ever call from a plugin, so I need to decide what to do about those.
https://datasette.readthedocs.io/en/latest/internals.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/519#issuecomment-636329468,https://api.github.com/repos/simonw/datasette/issues/519,636329468,MDEyOklzc3VlQ29tbWVudDYzNjMyOTQ2OA==,9599,simonw,2020-05-30T13:14:52Z,2020-05-30T13:21:03Z,OWNER,"I've made a lot of progress towards this recently:
- Started internals documentation #576
- Improved design of the request object #706
- Redesigned register_output_renderer plugin hook #581","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459590021,Decide what goes into Datasette 1.0,
https://github.com/simonw/datasette/issues/519#issuecomment-636330023,https://api.github.com/repos/simonw/datasette/issues/519,636330023,MDEyOklzc3VlQ29tbWVudDYzNjMzMDAyMw==,9599,simonw,2020-05-30T13:19:24Z,2020-05-30T13:19:24Z,OWNER,"Goals for Datasette 1.0:
- Generally signify confidence in the quality/stability of Datasette. I think I'm there already.
- Plugin authors can have confidence that their plugins will work for the whole 1.x release cycle
- Developers building against Datasette JSON APIs should have confidence in 1.x compatibility
- Template authors and CSS theme authors should have that confidence too
I think I'm very nearly there for the plugin API. The harder ones are JSON APIs and template authors.
For JSON APIs: The default JSON just isn't right. I find myself using `?_shape=array` for almost everything I build.
The template part is harder. I think I need to fully document the template variables for every view - and add protective unit tests that match that documentation.
The CSS theme part is so hard it may not be possible. How do you guarantee stable HTML that won't break with custom CSS when you'll be adding new features during the 1.x release run? The pattern portfolio from #151 should hopefully help a lot there, but I still don't have a complete idea of what this entails or how feasible it is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459590021,Decide what goes into Datasette 1.0,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-636322089,https://api.github.com/repos/simonw/sqlite-utils/issues/114,636322089,MDEyOklzc3VlQ29tbWVudDYzNjMyMjA4OQ==,9599,simonw,2020-05-30T12:08:43Z,2020-05-30T12:08:43Z,OWNER,"Idea: use a chained API to define a complex transition and then execute it all at once. For example:
```python
db[""mytable""].transform().rename(""col1"", ""col_1"") \
.change_type(""col1"", float) \
.execute()
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/datasette/issues/774#issuecomment-636234759,https://api.github.com/repos/simonw/datasette/issues/774,636234759,MDEyOklzc3VlQ29tbWVudDYzNjIzNDc1OQ==,9599,simonw,2020-05-29T23:27:35Z,2020-05-29T23:27:35Z,OWNER,"Oh dear... it looks like `.raw_args` is used in my TIL script, which has been copied by a few people!
https://github.com/search?q=request+raw_args+datasette&type=Code
I'll fix it in mine and file pull requests against other pieces before this code gets released.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/774#issuecomment-636234067,https://api.github.com/repos/simonw/datasette/issues/774,636234067,MDEyOklzc3VlQ29tbWVudDYzNjIzNDA2Nw==,9599,simonw,2020-05-29T23:24:34Z,2020-05-29T23:24:34Z,OWNER,Updated documentation for `RequestParameters`: https://datasette.readthedocs.io/en/latest/internals.html#the-requestparameters-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/774#issuecomment-636229764,https://api.github.com/repos/simonw/datasette/issues/774,636229764,MDEyOklzc3VlQ29tbWVudDYzNjIyOTc2NA==,9599,simonw,2020-05-29T23:05:48Z,2020-05-29T23:05:48Z,OWNER,"I'm going to rebuild `RequestParameters` to no longer subclass `dict`. I'll keep the following methods:
- `__contains__()`
- `__getitem__()` (with the new behaviour)
- `keys()`
- iterating iterates keys
- `__len__`
- `get`
- `getlist`
It won't support writing, so it will effectively be immutable after you have constructed it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/774#issuecomment-636228656,https://api.github.com/repos/simonw/datasette/issues/774,636228656,MDEyOklzc3VlQ29tbWVudDYzNjIyODY1Ng==,9599,simonw,2020-05-29T23:01:22Z,2020-05-29T23:01:22Z,OWNER,"As far as I can tell the only code I've ever written that would break if I made this change is in `russian-ira-facebook-ads`:
https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/e7106710abdd7bdcae035bedd8bdaba75ae56a12/plugins/target.py#L22
https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/b8a22348c6b315ab94ddba69e8117dfdfd9573dc/plugins/regexp.py#L17
That doesn't work against latest Datasette anyway, so I think I can safely make this change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/774#issuecomment-636227927,https://api.github.com/repos/simonw/datasette/issues/774,636227927,MDEyOklzc3VlQ29tbWVudDYzNjIyNzkyNw==,9599,simonw,2020-05-29T22:58:32Z,2020-05-29T22:58:32Z,OWNER,"I think I want `request.args[""key""]` to return the FIRST item for that key or raise a `KeyError` if none are found.
Right now it returns the full list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/774#issuecomment-635702385,https://api.github.com/repos/simonw/datasette/issues/774,635702385,MDEyOklzc3VlQ29tbWVudDYzNTcwMjM4NQ==,9599,simonw,2020-05-29T01:21:15Z,2020-05-29T01:21:15Z,OWNER,"I think `request.args.getlist()` should return a list, not None, if the key does not exist.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/774#issuecomment-635702201,https://api.github.com/repos/simonw/datasette/issues/774,635702201,MDEyOklzc3VlQ29tbWVudDYzNTcwMjIwMQ==,9599,simonw,2020-05-29T01:20:34Z,2020-05-29T01:20:34Z,OWNER,"Or change `request.args` to behave more like `request.raw_args` - mainly to return a single value when you look things up by key.
It's currently defined like this: https://github.com/simonw/datasette/blob/3c1a60589e14849344acd8aa6da0a60b40fbfc60/datasette/utils/__init__.py#L756-L766","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/351#issuecomment-635533807,https://api.github.com/repos/simonw/datasette/issues/351,635533807,MDEyOklzc3VlQ29tbWVudDYzNTUzMzgwNw==,9599,simonw,2020-05-28T18:56:15Z,2020-05-28T18:56:15Z,OWNER,I'm not going to do this. I have a good enough solution now pasting the rendered HTML from the release notes into https://euangoddard.github.io/clipboard2markdown/ to get the Markdown.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",345469355,Automatically create a GitHub release linking to release notes for every tagged release,
https://github.com/simonw/datasette/issues/508#issuecomment-635532216,https://api.github.com/repos/simonw/datasette/issues/508,635532216,MDEyOklzc3VlQ29tbWVudDYzNTUzMjIxNg==,9599,simonw,2020-05-28T18:53:02Z,2020-05-28T18:53:02Z,OWNER,I fixed this a while ago in #702: https://datasette.readthedocs.io/en/stable/metadata.html#setting-a-default-sort-order,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455965174,Ability to set default sort order for a table or view in metadata.json,
https://github.com/simonw/datasette/issues/774#issuecomment-635530876,https://api.github.com/repos/simonw/datasette/issues/774,635530876,MDEyOklzc3VlQ29tbWVudDYzNTUzMDg3Ng==,9599,simonw,2020-05-28T18:50:18Z,2020-05-28T18:50:18Z,OWNER,"How about moving this functionality to the request object itself?
```python
q = request[""q""] # Raises KeyError if missing, otherwise returns first
q = request.get(""q"", ""default"") # Returns first, or optional default or None
facets = request.getlist(""_facet"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/777#issuecomment-635513983,https://api.github.com/repos/simonw/datasette/issues/777,635513983,MDEyOklzc3VlQ29tbWVudDYzNTUxMzk4Mw==,63653929,thisismyfuckingusername,2020-05-28T18:16:49Z,2020-05-28T18:16:49Z,NONE," think, because the given URL of the CSS file doesn't have any complete parameters after query
Try to complete the parameter
``","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626171242,Error pages not correctly loading CSS,
https://github.com/simonw/datasette/issues/781#issuecomment-635494730,https://api.github.com/repos/simonw/datasette/issues/781,635494730,MDEyOklzc3VlQ29tbWVudDYzNTQ5NDczMA==,9599,simonw,2020-05-28T17:39:54Z,2020-05-28T17:39:54Z,OWNER,https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fwww.niche-museums.com%2Fbrowse%2Ffeed.atom validates now!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626663119,request.url and request.scheme should obey force_https_urls config setting,
https://github.com/simonw/datasette/issues/780#issuecomment-635480948,https://api.github.com/repos/simonw/datasette/issues/780,635480948,MDEyOklzc3VlQ29tbWVudDYzNTQ4MDk0OA==,9599,simonw,2020-05-28T17:12:45Z,2020-05-28T17:12:45Z,OWNER,This is a good opportunity to reconsider the design of this function and see if I'm happy with it as-is or if there are some improvements I want to make before adding it to the documented API.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626593402,Internals documentation for datasette.metadata() method,
https://github.com/simonw/datasette/issues/781#issuecomment-635471007,https://api.github.com/repos/simonw/datasette/issues/781,635471007,MDEyOklzc3VlQ29tbWVudDYzNTQ3MTAwNw==,9599,simonw,2020-05-28T16:58:47Z,2020-05-28T16:58:47Z,OWNER,"I'm inclined to do this at the earliest possible moment. I think that's probably in the `DatasetteRouter` class, which should see every `scope` first and be able to apply that setting to it. It already has some special case logic to deal with the `base_url` setting here:
https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/datasette/app.py#L779-L789","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626663119,request.url and request.scheme should obey force_https_urls config setting,
https://github.com/simonw/datasette/issues/781#issuecomment-635468994,https://api.github.com/repos/simonw/datasette/issues/781,635468994,MDEyOklzc3VlQ29tbWVudDYzNTQ2ODk5NA==,9599,simonw,2020-05-28T16:55:35Z,2020-05-28T16:55:35Z,OWNER,"I think the right way to fix this is to modify the `scope[""scheme""]` ASGI key before the Request object is constructed - otherwise that Request class will have to gain knowledge of Datasette's configuration mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626663119,request.url and request.scheme should obey force_https_urls config setting,
https://github.com/simonw/datasette/issues/780#issuecomment-635413437,https://api.github.com/repos/simonw/datasette/issues/780,635413437,MDEyOklzc3VlQ29tbWVudDYzNTQxMzQzNw==,9599,simonw,2020-05-28T15:15:03Z,2020-05-28T15:15:03Z,OWNER,Also: I think I should add a `query=` parameter to help lookup metadata about a specific named canned query.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626593402,Internals documentation for datasette.metadata() method,
https://github.com/simonw/datasette/issues/744#issuecomment-635386935,https://api.github.com/repos/simonw/datasette/issues/744,635386935,MDEyOklzc3VlQ29tbWVudDYzNTM4NjkzNQ==,30607,aborruso,2020-05-28T14:32:53Z,2020-05-28T14:32:53Z,NONE,"Wow, I'm in some way very proud!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-635384739,https://api.github.com/repos/simonw/datasette/issues/744,635384739,MDEyOklzc3VlQ29tbWVudDYzNTM4NDczOQ==,9599,simonw,2020-05-28T14:28:58Z,2020-05-28T14:28:58Z,OWNER,This is now released. `pip install datasette==0.43` should get the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/758#issuecomment-635195322,https://api.github.com/repos/simonw/datasette/issues/758,635195322,MDEyOklzc3VlQ29tbWVudDYzNTE5NTMyMg==,2181410,clausjuhl,2020-05-28T08:23:27Z,2020-05-28T08:23:27Z,NONE,@simonw I would prefer just the 7 character hash. No need to make the urls any longer than they need to be :),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612382643,Question: Access to immutable database-path,
https://github.com/simonw/datasette/issues/706#issuecomment-635131217,https://api.github.com/repos/simonw/datasette/issues/706,635131217,MDEyOklzc3VlQ29tbWVudDYzNTEzMTIxNw==,9599,simonw,2020-05-28T06:20:44Z,2020-05-28T06:20:44Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#canned-queries-default-fragment,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585633142,"Documentation for the ""request"" object",
https://github.com/simonw/datasette/issues/758#issuecomment-635102675,https://api.github.com/repos/simonw/datasette/issues/758,635102675,MDEyOklzc3VlQ29tbWVudDYzNTEwMjY3NQ==,9599,simonw,2020-05-28T05:04:07Z,2020-05-28T05:04:07Z,OWNER,"@clausjuhl do you have any thoughts on what would be most useful for you in these JSON responses? The full `/databasename-hash` path, just the 7 character hash, or something else?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612382643,Question: Access to immutable database-path,
https://github.com/simonw/datasette/issues/751#issuecomment-635101438,https://api.github.com/repos/simonw/datasette/issues/751,635101438,MDEyOklzc3VlQ29tbWVudDYzNTEwMTQzOA==,9599,simonw,2020-05-28T05:00:21Z,2020-05-28T05:00:21Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/metadata.html#setting-a-custom-page-size,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611540797,Ability to set custom default _size on a per-table basis,
https://github.com/simonw/datasette/issues/770#issuecomment-634880090,https://api.github.com/repos/simonw/datasette/issues/770,634880090,MDEyOklzc3VlQ29tbWVudDYzNDg4MDA5MA==,9599,simonw,2020-05-27T19:10:57Z,2020-05-28T04:22:47Z,OWNER,"This `can_render` callback should take the same arguments as the redesigned `render` (previously called `callback`): https://github.com/simonw/datasette/issues/581#issuecomment-634879258
- `datasette` - a Datasette instance
- `columns` - the list of columns
- `rows` - the list of rows (each one a SQLite `Row` object)
- `sql` - the SQL query being executed
- `query_name` - the name of the canned query, if this is one
- `database` - the database name
- `table` - the table or view name
- `request` - the request object (to be documented in #706)
- `view_name` - the name of the view","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625930207,register_output_renderer can_render mechanism,
https://github.com/simonw/datasette/issues/773#issuecomment-635069987,https://api.github.com/repos/simonw/datasette/issues/773,635069987,MDEyOklzc3VlQ29tbWVudDYzNTA2OTk4Nw==,9599,simonw,2020-05-28T03:13:27Z,2020-05-28T03:13:27Z,OWNER,`register_output_renderer` test added in https://github.com/simonw/datasette/commit/52c4387c7d37c867104e3728cc1f4c4d1e100642#diff-56f7d7b4778bac73b6b655c02c8467aaR336,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626001501,All plugin hooks should have unit tests,
https://github.com/simonw/datasette/issues/770#issuecomment-635056520,https://api.github.com/repos/simonw/datasette/issues/770,635056520,MDEyOklzc3VlQ29tbWVudDYzNTA1NjUyMA==,9599,simonw,2020-05-28T02:28:00Z,2020-05-28T02:28:00Z,OWNER,"This should be optionally awaitable, as in #776","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625930207,register_output_renderer can_render mechanism,
https://github.com/simonw/datasette/issues/776#issuecomment-635056357,https://api.github.com/repos/simonw/datasette/issues/776,635056357,MDEyOklzc3VlQ29tbWVudDYzNTA1NjM1Nw==,9599,simonw,2020-05-28T02:27:26Z,2020-05-28T02:27:26Z,OWNER,"Example code from elsewhere:
https://github.com/simonw/datasette/blob/52c4387c7d37c867104e3728cc1f4c4d1e100642/datasette/views/base.py#L416-L420","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626163974,register_output_renderer render callback should be optionally awaitable,
https://github.com/simonw/datasette/issues/581#issuecomment-635055346,https://api.github.com/repos/simonw/datasette/issues/581,635055346,MDEyOklzc3VlQ29tbWVudDYzNTA1NTM0Ng==,9599,simonw,2020-05-28T02:24:14Z,2020-05-28T02:24:14Z,OWNER,Updated documentation is here: https://datasette.readthedocs.io/en/latest/plugins.html#register-output-renderer-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/645#issuecomment-635054690,https://api.github.com/repos/simonw/datasette/issues/645,635054690,MDEyOklzc3VlQ29tbWVudDYzNTA1NDY5MA==,9599,simonw,2020-05-28T02:22:12Z,2020-05-28T02:22:12Z,OWNER,"This is a duplicate of a more recent, more developed issue: #770","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",530653633,Mechanism for register_output_renderer to suggest extension or not,
https://github.com/simonw/datasette/issues/581#issuecomment-634879258,https://api.github.com/repos/simonw/datasette/issues/581,634879258,MDEyOklzc3VlQ29tbWVudDYzNDg3OTI1OA==,9599,simonw,2020-05-27T19:09:22Z,2020-05-28T01:33:45Z,OWNER,"OK, the new design: your callback function can take any of the following arguments:
- `datasette` - a Datasette instance
- `columns` - the list of columns
- `rows` - the list of rows (each one a SQLite `Row` object)
- `sql` - the SQL query being executed
- `query_name` - the name of the canned query, if this is one
- `database` - the database name
- `table` - the table or view name
- `request` - the request object (to be documented in #706)
- `view_name` - the name of the view
We will also continue to support the existing `args` and `data` arguments, but these will be undocumented and will be deprecated in Datasette 1.0. UPDATE: Decided against this, see https://github.com/simonw/datasette/issues/581#issuecomment-634946197","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/775#issuecomment-635023389,https://api.github.com/repos/simonw/datasette/issues/775,635023389,MDEyOklzc3VlQ29tbWVudDYzNTAyMzM4OQ==,9599,simonw,2020-05-28T00:47:32Z,2020-05-28T00:47:32Z,OWNER,"These:
https://github.com/simonw/datasette/blob/4b96857f170e329a73186e703cc0d9ca4e8719cc/tests/fixtures.py#L337-L506","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626131309,Move test plugins into datasette/tests/plugins/ directory,
https://github.com/simonw/datasette/issues/770#issuecomment-634980179,https://api.github.com/repos/simonw/datasette/issues/770,634980179,MDEyOklzc3VlQ29tbWVudDYzNDk4MDE3OQ==,9599,simonw,2020-05-27T22:37:19Z,2020-05-27T22:37:19Z,OWNER,"Can I come up with a better name than `should_suggest`?
It's a check that sees if the current query is supported by the renderer plugin. Some options:
- `can_render`
- `supports_query`
- `is_supported`
I like `can_render`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625930207,register_output_renderer can_render mechanism,
https://github.com/simonw/datasette/issues/581#issuecomment-634978388,https://api.github.com/repos/simonw/datasette/issues/581,634978388,MDEyOklzc3VlQ29tbWVudDYzNDk3ODM4OA==,9599,simonw,2020-05-27T22:32:03Z,2020-05-27T22:32:03Z,OWNER,Request object is now documented: https://datasette.readthedocs.io/en/latest/internals.html#request-object,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/706#issuecomment-634975252,https://api.github.com/repos/simonw/datasette/issues/706,634975252,MDEyOklzc3VlQ29tbWVudDYzNDk3NTI1Mg==,9599,simonw,2020-05-27T22:23:26Z,2020-05-27T22:30:05Z,OWNER,I'm going to leave `.raw_args` in for the moment but deliberately not document it. I'll hope to phase it out entirely at a later date.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585633142,"Documentation for the ""request"" object",
https://github.com/simonw/datasette/issues/706#issuecomment-634974819,https://api.github.com/repos/simonw/datasette/issues/706,634974819,MDEyOklzc3VlQ29tbWVudDYzNDk3NDgxOQ==,9599,simonw,2020-05-27T22:22:20Z,2020-05-27T22:22:20Z,OWNER,"What would a better name be?
- `.simple_args`
- `.kv_args`
- `.pair_args`
- `.dict_args`
- `.args_dict`
I dislike the last two the least.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585633142,"Documentation for the ""request"" object",
https://github.com/simonw/datasette/issues/706#issuecomment-634974088,https://api.github.com/repos/simonw/datasette/issues/706,634974088,MDEyOklzc3VlQ29tbWVudDYzNDk3NDA4OA==,9599,simonw,2020-05-27T22:20:20Z,2020-05-27T22:20:20Z,OWNER,It looks like I inherited `.raw_args` from Sanic - I use it in a few places: https://github.com/search?q=user%3Asimonw+raw_args&type=Code,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585633142,"Documentation for the ""request"" object",
https://github.com/simonw/datasette/issues/706#issuecomment-634973596,https://api.github.com/repos/simonw/datasette/issues/706,634973596,MDEyOklzc3VlQ29tbWVudDYzNDk3MzU5Ng==,9599,simonw,2020-05-27T22:19:02Z,2020-05-27T22:19:02Z,OWNER,"New documentation can be seen here: https://github.com/simonw/datasette/blob/6d7cb02f00010d3cb4b4bac0460d41277652b80e/docs/internals.rst#request-object
It's inspired me to reconsider the name of the `.raw_args` property, which isn't particularly clear.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585633142,"Documentation for the ""request"" object",
https://github.com/simonw/datasette/issues/706#issuecomment-634965148,https://api.github.com/repos/simonw/datasette/issues/706,634965148,MDEyOklzc3VlQ29tbWVudDYzNDk2NTE0OA==,9599,simonw,2020-05-27T21:59:07Z,2020-05-27T21:59:07Z,OWNER,"This is the full current implementation of the request object:
https://github.com/simonw/datasette/blob/9424687e9e94401438896116898a071702b09d40/datasette/utils/asgi.py#L15-L95","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585633142,"Documentation for the ""request"" object",
https://github.com/simonw/datasette/issues/581#issuecomment-634964457,https://api.github.com/repos/simonw/datasette/issues/581,634964457,MDEyOklzc3VlQ29tbWVudDYzNDk2NDQ1Nw==,9599,simonw,2020-05-27T21:57:35Z,2020-05-27T21:57:35Z,OWNER,(I wonder if this would be enough to allow really smart plugins to implement ETag/conditional get),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634964294,https://api.github.com/repos/simonw/datasette/issues/581,634964294,MDEyOklzc3VlQ29tbWVudDYzNDk2NDI5NA==,9599,simonw,2020-05-27T21:57:10Z,2020-05-27T21:57:10Z,OWNER,"Right now a rendering callback returns the following:
```
body - string or bytes, optional
The response body, default empty
content_type - string, optional
The Content-Type header, default text/plain
status_code - integer, optional
The HTTP status code, default 200
```
I'm going to add an optional `headers` dictionary key, too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/758#issuecomment-634951605,https://api.github.com/repos/simonw/datasette/issues/758,634951605,MDEyOklzc3VlQ29tbWVudDYzNDk1MTYwNQ==,9599,simonw,2020-05-27T21:29:19Z,2020-05-27T21:29:19Z,OWNER,"But... https://datasette-hash-urls-j7hipcg4aq-uw.a.run.app/fixtures-bda7daa.json doesn't expose that hash:
```
{
""database"": ""fixtures"",
""size"": 258048,
""tables"": [
{
""name"": ""123_starts_with_digits"",
```
Likewise https://datasette-hash-urls-j7hipcg4aq-uw.a.run.app/fixtures-bda7daa/complex_foreign_keys.json
```
{
""database"": ""fixtures"",
""table"": ""complex_foreign_keys"",
""is_view"": false,
""human_description_en"": """",
""rows"": [
[
""1"",
""1"",
""2"",
""1""
]
],
```
And https://datasette-hash-urls-j7hipcg4aq-uw.a.run.app/fixtures-bda7daa/complex_foreign_keys/1.json
```
{
""database"": ""fixtures"",
""table"": ""complex_foreign_keys"",
""rows"": [
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612382643,Question: Access to immutable database-path,
https://github.com/simonw/datasette/issues/758#issuecomment-634950200,https://api.github.com/repos/simonw/datasette/issues/758,634950200,MDEyOklzc3VlQ29tbWVudDYzNDk1MDIwMA==,9599,simonw,2020-05-27T21:26:37Z,2020-05-27T21:26:37Z,OWNER,"https://latest.datasette.io/.json currently returns:
```
{
""fixtures"": {
""name"": ""fixtures"",
""hash"": ""87b3f2c55dfb81ff1452dd306c2623fa5550b90982cfa32bad404c4d8bbedde2"",
""color"": ""87b3f2"",
""path"": ""/fixtures"",
""tables_and_views_truncated"": [
```
I published `fixtures.db` here like this:
datasette publish cloudrun fixtures.db --service datasette-hash-urls --extra-options '--config hash_urls:1'
https://datasette-hash-urls-j7hipcg4aq-uw.a.run.app/.json
```
{
""fixtures"": {
""name"": ""fixtures"",
""hash"": ""bda7daa889c23f9a8f06e46d7d280dd423c76275e9593c4c1cad7c53b19032fe"",
""color"": ""bda7da"",
""path"": ""/fixtures-bda7daa"",
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612382643,Question: Access to immutable database-path,
https://github.com/simonw/datasette/issues/581#issuecomment-634946319,https://api.github.com/repos/simonw/datasette/issues/581,634946319,MDEyOklzc3VlQ29tbWVudDYzNDk0NjMxOQ==,9599,simonw,2020-05-27T21:18:50Z,2020-05-27T21:18:50Z,OWNER,(I used GitHub code search to find code using this plugin hook: https://github.com/search?q=register_output_renderer&type=Code ),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634946197,https://api.github.com/repos/simonw/datasette/issues/581,634946197,MDEyOklzc3VlQ29tbWVudDYzNDk0NjE5Nw==,9599,simonw,2020-05-27T21:18:30Z,2020-05-27T21:18:30Z,OWNER,"I'm going to break backwards compatibility directly here, without waiting for Datasette 1.0.
The reason is that https://github.com/russss/datasette-geo hasn't been updated in 13 months so is already broken against current Datasette, and the other two plugins using this hook are owned by me so I can upgrade them myself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634944832,https://api.github.com/repos/simonw/datasette/issues/581,634944832,MDEyOklzc3VlQ29tbWVudDYzNDk0NDgzMg==,9599,simonw,2020-05-27T21:15:50Z,2020-05-27T21:16:28Z,OWNER,"It bothers me that `query_name` here means the configured name of the canned query, but `view_name` means the name of the Datasette view class, NOT the name of an associated SQL view. That's in `table`.
Can I come up with clearer names for these?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634943336,https://api.github.com/repos/simonw/datasette/issues/581,634943336,MDEyOklzc3VlQ29tbWVudDYzNDk0MzMzNg==,9599,simonw,2020-05-27T21:13:04Z,2020-05-27T21:13:04Z,OWNER,Since I'm passing `request` I won't pass `scope` - if people want that they can access `request.scope`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-592621235,https://api.github.com/repos/simonw/datasette/issues/581,592621235,MDEyOklzc3VlQ29tbWVudDU5MjYyMTIzNQ==,9599,simonw,2020-02-28T17:24:06Z,2020-05-27T21:12:21Z,OWNER,"Rather than pass a request object (hence promoting that object into part of the documented, stable API) I think I'll pass the ASGI scope - that's already a stable, documented standard.
UPDATE: changed my mind since `request` is used by other plugins too, see #706.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/773#issuecomment-634940522,https://api.github.com/repos/simonw/datasette/issues/773,634940522,MDEyOklzc3VlQ29tbWVudDYzNDk0MDUyMg==,9599,simonw,2020-05-27T21:07:48Z,2020-05-27T21:07:48Z,OWNER,Remove this `xfail` decorator once they are all tested: https://github.com/simonw/datasette/blob/da87e963bff24e47878a5bc2025c8bfc63d4bc93/tests/test_plugins.py#L23-L28,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626001501,All plugin hooks should have unit tests,
https://github.com/simonw/datasette/issues/581#issuecomment-634921101,https://api.github.com/repos/simonw/datasette/issues/581,634921101,MDEyOklzc3VlQ29tbWVudDYzNDkyMTEwMQ==,9599,simonw,2020-05-27T20:27:36Z,2020-05-27T20:27:36Z,OWNER,Actually passing the `request` object would be OK if I document it see https://github.com/simonw/datasette/issues/706,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/771#issuecomment-634916313,https://api.github.com/repos/simonw/datasette/issues/771,634916313,MDEyOklzc3VlQ29tbWVudDYzNDkxNjMxMw==,9599,simonw,2020-05-27T20:17:13Z,2020-05-27T20:17:13Z,OWNER,Closed in da87e963bff24e47878a5bc2025c8bfc63d4bc93,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625980317,Unit test that checks that all plugin hooks have corresponding unit tests,
https://github.com/simonw/datasette/issues/771#issuecomment-634915104,https://api.github.com/repos/simonw/datasette/issues/771,634915104,MDEyOklzc3VlQ29tbWVudDYzNDkxNTEwNA==,9599,simonw,2020-05-27T20:14:32Z,2020-05-27T20:14:32Z,OWNER,"```
$ pytest -k test_plugin_hooks_have_tests -vv
====================================== test session starts ======================================
platform darwin -- Python 3.7.7, pytest-5.2.4, py-1.8.1, pluggy-0.13.1 -- /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/python
cachedir: .pytest_cache
rootdir: /Users/simon/Dropbox/Development/datasette, inifile: pytest.ini
plugins: asyncio-0.10.0
collected 486 items / 475 deselected / 11 selected
tests/test_plugins.py::test_plugin_hooks_have_tests[asgi_wrapper] XPASS [ 9%]
tests/test_plugins.py::test_plugin_hooks_have_tests[extra_body_script] XPASS [ 18%]
tests/test_plugins.py::test_plugin_hooks_have_tests[extra_css_urls] XPASS [ 27%]
tests/test_plugins.py::test_plugin_hooks_have_tests[extra_js_urls] XPASS [ 36%]
tests/test_plugins.py::test_plugin_hooks_have_tests[extra_template_vars] XPASS [ 45%]
tests/test_plugins.py::test_plugin_hooks_have_tests[prepare_connection] XPASS [ 54%]
tests/test_plugins.py::test_plugin_hooks_have_tests[prepare_jinja2_environment] XFAIL [ 63%]
tests/test_plugins.py::test_plugin_hooks_have_tests[publish_subcommand] XFAIL [ 72%]
tests/test_plugins.py::test_plugin_hooks_have_tests[register_facet_classes] XFAIL [ 81%]
tests/test_plugins.py::test_plugin_hooks_have_tests[register_output_renderer] XFAIL [ 90%]
tests/test_plugins.py::test_plugin_hooks_have_tests[render_cell] XPASS [100%]
========================= 475 deselected, 4 xfailed, 7 xpassed in 1.70s =========================","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625980317,Unit test that checks that all plugin hooks have corresponding unit tests,
https://github.com/simonw/datasette/issues/771#issuecomment-634909818,https://api.github.com/repos/simonw/datasette/issues/771,634909818,MDEyOklzc3VlQ29tbWVudDYzNDkwOTgxOA==,9599,simonw,2020-05-27T20:02:52Z,2020-05-27T20:02:52Z,OWNER,Actually I'll land this using `@pytest.mark.xfail`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625980317,Unit test that checks that all plugin hooks have corresponding unit tests,
https://github.com/simonw/datasette/issues/771#issuecomment-634909347,https://api.github.com/repos/simonw/datasette/issues/771,634909347,MDEyOklzc3VlQ29tbWVudDYzNDkwOTM0Nw==,9599,simonw,2020-05-27T20:01:52Z,2020-05-27T20:01:52Z,OWNER,I'll do the work for this in the pull request #772.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625980317,Unit test that checks that all plugin hooks have corresponding unit tests,
https://github.com/simonw/datasette/issues/771#issuecomment-634900776,https://api.github.com/repos/simonw/datasette/issues/771,634900776,MDEyOklzc3VlQ29tbWVudDYzNDkwMDc3Ng==,9599,simonw,2020-05-27T19:44:25Z,2020-05-27T19:44:25Z,OWNER,"This seems to work:
```diff
diff --git a/tests/test_plugins.py b/tests/test_plugins.py
index 8b6a6b4..e9a40aa 100644
--- a/tests/test_plugins.py
+++ b/tests/test_plugins.py
@@ -7,7 +7,7 @@ from .fixtures import (
TestClient as _TestClient,
) # noqa
from datasette.app import Datasette
-from datasette.plugins import get_plugins, DEFAULT_PLUGINS
+from datasette.plugins import get_plugins, DEFAULT_PLUGINS, pm
from datasette.utils import sqlite3
import base64
import json
@@ -20,6 +20,21 @@ import pytest
import urllib
+def test_plugin_hooks_have_tests():
+ ""Every plugin hook should be referenced in this test module""
+ hooks = [name for name in dir(pm.hook) if not name.startswith(""_"")]
+ tests_in_this_module = [t for t in globals().keys() if t.startswith('test_')]
+ untested = []
+ for hook in hooks:
+ ok = False
+ for test in tests_in_this_module:
+ if hook in test:
+ ok = True
+ if not ok:
+ untested.append(hook)
+ assert not untested, 'These plugin hooks are missing tests: {}'.format(untested)
+
+
def test_plugins_dir_plugin_prepare_connection(app_client):
response = app_client.get(
""/fixtures.json?sql=select+convert_units(100%2C+'m'%2C+'ft')""
```
Based on how the documentation unit tests work.
Currently fails with:
AssertionError: These plugin hooks are missing tests:
['prepare_jinja2_environment', 'publish_subcommand', 'register_facet_classes', 'register_output_renderer']
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625980317,Unit test that checks that all plugin hooks have corresponding unit tests,
https://github.com/simonw/datasette/issues/581#issuecomment-634893744,https://api.github.com/repos/simonw/datasette/issues/581,634893744,MDEyOklzc3VlQ29tbWVudDYzNDg5Mzc0NA==,9599,simonw,2020-05-27T19:32:08Z,2020-05-27T19:32:08Z,OWNER,Need to figure out how best to unit test this plugin hook.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634888582,https://api.github.com/repos/simonw/datasette/issues/581,634888582,MDEyOklzc3VlQ29tbWVudDYzNDg4ODU4Mg==,9599,simonw,2020-05-27T19:23:23Z,2020-05-27T19:23:23Z,OWNER,"Here's the function I just wrote for this:
```python
def call_with_supported_arguments(fn, **kwargs):
parameters = inspect.signature(fn).parameters.keys()
call_with = []
for parameter in parameters:
if parameter not in kwargs:
raise TypeError(""{} requires parameters {}"".format(fn, tuple(parameters)))
call_with.append(kwargs[parameter])
return fn(*call_with)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634882770,https://api.github.com/repos/simonw/datasette/issues/581,634882770,MDEyOklzc3VlQ29tbWVudDYzNDg4Mjc3MA==,9599,simonw,2020-05-27T19:16:19Z,2020-05-27T19:16:19Z,OWNER,"```
In [1]: import inspect
In [2]: def foo(view, sql, inspect):
...: pass
...:
In [3]: inspect.signature(foo)
Out[3]:
In [4]: inspect.signature(foo).parameters
Out[4]:
mappingproxy({'view': ,
'sql': ,
'inspect': })
In [5]: inspect.signature(foo).parameters.keys()
Out[5]: odict_keys(['view', 'sql', 'inspect'])
In [6]: set(inspect.signature(foo).parameters.keys())
Out[6]: {'inspect', 'sql', 'view'}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634882112,https://api.github.com/repos/simonw/datasette/issues/581,634882112,MDEyOklzc3VlQ29tbWVudDYzNDg4MjExMg==,9599,simonw,2020-05-27T19:14:55Z,2020-05-27T19:14:55Z,OWNER,"https://docs.python.org/3/library/inspect.html#introspecting-callables-with-the-signature-object
> New in version 3.3.
>
> The Signature object represents the call signature of a callable object and its return annotation. To retrieve a Signature object, use the [`signature()`](https://docs.python.org/3/library/inspect.html#inspect.signature ""inspect.signature"") function.
>
> `inspect.``signature`(*callable*, ***, *follow_wrapped=True*)
>
> Return a [`Signature`](https://docs.python.org/3/library/inspect.html#inspect.Signature ""inspect.Signature"") object for the given `callable`:
>
> ```
> >>> from inspect import signature
> >>> def foo(a, *, b:int, **kwargs):
> ... pass
>
> >>> sig = signature(foo)
>
> >>> str(sig)
> '(a, *, b:int, **kwargs)'
>
> >>> str(sig.parameters['b'])
> 'b:int'
>
> >>> sig.parameters['b'].annotation
>
> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634881287,https://api.github.com/repos/simonw/datasette/issues/581,634881287,MDEyOklzc3VlQ29tbWVudDYzNDg4MTI4Nw==,9599,simonw,2020-05-27T19:13:09Z,2020-05-27T19:13:09Z,OWNER,"I think I need a utility function for ""call this function with this dictionary of arguments, but only pass the arguments which are inspected by the function"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634879734,https://api.github.com/repos/simonw/datasette/issues/581,634879734,MDEyOklzc3VlQ29tbWVudDYzNDg3OTczNA==,9599,simonw,2020-05-27T19:10:17Z,2020-05-27T19:12:36Z,OWNER,The `should_suggest` callback will take the same arguments: https://github.com/simonw/datasette/issues/770#issuecomment-634880090,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/770#issuecomment-634880474,https://api.github.com/repos/simonw/datasette/issues/770,634880474,MDEyOklzc3VlQ29tbWVudDYzNDg4MDQ3NA==,9599,simonw,2020-05-27T19:11:39Z,2020-05-27T19:11:39Z,OWNER,I'm going to rename `callback` to `render` but continue supporting `callback` until Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625930207,register_output_renderer can_render mechanism,
https://github.com/simonw/datasette/issues/581#issuecomment-634865620,https://api.github.com/repos/simonw/datasette/issues/581,634865620,MDEyOklzc3VlQ29tbWVudDYzNDg2NTYyMA==,9599,simonw,2020-05-27T18:44:06Z,2020-05-27T18:44:06Z,OWNER,"The existing render callback takes the following arguments:
> `args` - dictionary
> The GET parameters of the request
>
> `data` - dictionary
> The data to be rendered
>
> `view_name` - string
> The name of the view where the renderer is being called. (`index`, `database`, `table`, and `row` are the most important ones.)
The `data` argument is a bit of a problem, because it tightly couples plugins to a currently undocumented datastructure within Datasette. Here's how `datasette-atom` picks that apart for example: https://github.com/simonw/datasette-atom/blob/095941c23c81b70c4787cdeef873c556b573b5fa/datasette_atom/__init__.py#L15-L66 - it does things like access `data[""query""][""sql""]` to figure out the SQL query that was used.
I'm going to change the design of part of this ticket. I won't break the old `data` value just yet, but I'll mark it to be deprecated by Datasette 1.0.
I think the only plugins using it right now are my `datasette-atom` and `datasette-ics` and @russss's [datasette-geo](https://github.com/russss/datasette-geo/blob/0d24c9fd782eeae8d136ee791143ee8cbf49ebdf/datasette_plugin_geo/__init__.py#L94-L101) so hopefully changing this won't cause any wider damage.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634857975,https://api.github.com/repos/simonw/datasette/issues/581,634857975,MDEyOklzc3VlQ29tbWVudDYzNDg1Nzk3NQ==,9599,simonw,2020-05-27T18:30:29Z,2020-05-27T18:30:29Z,OWNER,I'll use #770 for the `should_suggest` mechanism - this issue is for the extra arguments passed to the rendering callback.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634856748,https://api.github.com/repos/simonw/datasette/issues/581,634856748,MDEyOklzc3VlQ29tbWVudDYzNDg1Njc0OA==,9599,simonw,2020-05-27T18:28:32Z,2020-05-27T18:28:32Z,OWNER,"Here's the code that passes a list of renderers to the template:
https://github.com/simonw/datasette/blob/2d099ad9c657d2cab59de91cdb8bfed2da236ef6/datasette/views/base.py#L411-L423
A renderer is currently defined as a two-key dictionary:
```python
@hookimpl
def register_output_renderer(datasette):
return {
'extension': 'test',
'callback': render_test
}
```
I can add a third key, `""should_suggest""` which is a function that returns `True` or `False` for a given query. If that key is missing it is assumed to return `True`.
One catch: what arguments should be passed to the `should_suggest(...)` function?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634853296,https://api.github.com/repos/simonw/datasette/issues/581,634853296,MDEyOklzc3VlQ29tbWVudDYzNDg1MzI5Ng==,9599,simonw,2020-05-27T18:22:46Z,2020-05-27T18:22:46Z,OWNER,"While I'm doing this, another feature I would like is the ability for renderers to opt-in / opt-out of being displayed as options on the page.
https://www.niche-museums.com/browse/museums for example shows a `atom` link because the `datasette-atom` plugin is installed... but clicking it will give you a 400 error because the correct columns are not present:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634852196,https://api.github.com/repos/simonw/datasette/issues/581,634852196,MDEyOklzc3VlQ29tbWVudDYzNDg1MjE5Ng==,9599,simonw,2020-05-27T18:20:46Z,2020-05-27T18:20:46Z,OWNER,"Here's the code that calls the renderers - this needs to be expanded to check for those extra optional arguments:
https://github.com/simonw/datasette/blob/2d099ad9c657d2cab59de91cdb8bfed2da236ef6/datasette/views/base.py#L387-L398","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/744#issuecomment-634850676,https://api.github.com/repos/simonw/datasette/issues/744,634850676,MDEyOklzc3VlQ29tbWVudDYzNDg1MDY3Ng==,9599,simonw,2020-05-27T18:18:01Z,2020-05-27T18:18:01Z,OWNER,Thanks for helping test this @aborruso!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-634446887,https://api.github.com/repos/simonw/datasette/issues/744,634446887,MDEyOklzc3VlQ29tbWVudDYzNDQ0Njg4Nw==,30607,aborruso,2020-05-27T06:01:28Z,2020-05-27T06:01:28Z,NONE,"Dear @simonw thank you for your time, now IT WORKS!!!
I hope that this edit to datasette code is not for an exceptional case (my PC configuration) and that it will be useful to other users.
Thank you again!!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-634395343,https://api.github.com/repos/simonw/datasette/issues/744,634395343,MDEyOklzc3VlQ29tbWVudDYzNDM5NTM0Mw==,9599,simonw,2020-05-27T02:49:26Z,2020-05-27T02:49:26Z,OWNER,"OK, here's a new branch you can try. Install it like this:
pip install https://github.com/simonw/datasette/archive/shutil-backport.zip
If it works for you I'll merge that branch into master.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-634283355,https://api.github.com/repos/simonw/datasette/issues/744,634283355,MDEyOklzc3VlQ29tbWVudDYzNDI4MzM1NQ==,30607,aborruso,2020-05-26T21:15:34Z,2020-05-26T21:15:34Z,NONE,"> Oh no! It looks like `dirs_exist_ok` is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.
Thank you very much","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-634254943,https://api.github.com/repos/simonw/datasette/issues/744,634254943,MDEyOklzc3VlQ29tbWVudDYzNDI1NDk0Mw==,9599,simonw,2020-05-26T20:15:11Z,2020-05-26T20:15:11Z,OWNER,"Oh no! It looks like `dirs_exist_ok` is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-634254700,https://api.github.com/repos/simonw/datasette/issues/744,634254700,MDEyOklzc3VlQ29tbWVudDYzNDI1NDcwMA==,9599,simonw,2020-05-26T20:14:35Z,2020-05-26T20:14:35Z,OWNER,What version of Python are you running?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633704127,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633704127,MDEyOklzc3VlQ29tbWVudDYzMzcwNDEyNw==,9599,simonw,2020-05-25T20:14:22Z,2020-05-25T20:14:22Z,MEMBER,https://github.com/dogsheep/dogsheep-photos/blob/0.4.1/README.md#serving-photos-locally-with-datasette-media,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633644225,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633644225,MDEyOklzc3VlQ29tbWVudDYzMzY0NDIyNQ==,9599,simonw,2020-05-25T16:30:44Z,2020-05-25T16:30:44Z,MEMBER,I'll add docs on using `datasette-json-html` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633643921,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633643921,MDEyOklzc3VlQ29tbWVudDYzMzY0MzkyMQ==,9599,simonw,2020-05-25T16:29:44Z,2020-05-25T16:29:44Z,MEMBER,https://github.com/dogsheep/dogsheep-photos/blob/dc43fa8653cb9c7238a36f52239b91d1ec916d5c/README.md#serving-photos-locally-with-datasette-media,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633629944,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633629944,MDEyOklzc3VlQ29tbWVudDYzMzYyOTk0NA==,9599,simonw,2020-05-25T15:47:42Z,2020-05-25T15:47:42Z,MEMBER,"I'll add a proper section to the README, but for the moment here's how I do this.
First, install `datasette` and the `datasette-media` plugin.
Create a `metadata.yaml` file with the following content:
```yaml
plugins:
datasette-media:
photo:
sql: |-
select path as filepath, 200 as resize_height from apple_photos where uuid = :key
photo-big:
sql: |-
select path as filepath, 1024 as resize_height from apple_photos where uuid = :key
```
Now run `datasette -m metadata.yaml photos.db` - thumbnails will be served at http://127.0.0.1:8001/-/media/photo/F4469918-13F3-43D8-9EC1-734C0E6B60AD and larger sizes of the image at http://127.0.0.1:8001/-/media/photo-big/A8B02C7D-365E-448B-9510-69F80C26304D
I also made myself two custom pages, one showing recent images and one showing random images.
To do this, install the `datasette-template-sql` plugin and then create a `templates/pages` directory and add these files:
`recent-photos.html`
```html
Recent photos
{% for photo in sql(""select * from apple_photos order by date desc limit 100"") %}
![]()
{% endfor %}
```
`random-photos.html`
```html
Random photos
{% for photo in sql(""with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 100"") %}
![]()
{% endfor %}
```
Now run `datasette -m metadata.yaml photos.db --template-dir=templates/`
Visit http://127.0.0.1:8001/random-photos to see some random photos or http://127.0.0.1:8002/recent-photos for recent photos.
This is using this mechanism: https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633626741,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633626741,MDEyOklzc3VlQ29tbWVudDYzMzYyNjc0MQ==,9599,simonw,2020-05-25T15:38:55Z,2020-05-25T15:38:55Z,MEMBER,"Sure, I should absolutely document this!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633234781,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633234781,MDEyOklzc3VlQ29tbWVudDYzMzIzNDc4MQ==,41439,dmd,2020-05-24T13:56:13Z,2020-05-24T13:56:13Z,NONE,"As that seems to be closed, can you give a hint on how to make this work?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/simonw/datasette/issues/767#issuecomment-632555800,https://api.github.com/repos/simonw/datasette/issues/767,632555800,MDEyOklzc3VlQ29tbWVudDYzMjU1NTgwMA==,2657547,rixx,2020-05-22T08:00:23Z,2020-05-22T08:00:23Z,CONTRIBUTOR,That would be perfect!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",620969465,Allow to specify a URL fragment for canned queries,
https://github.com/simonw/datasette/issues/767#issuecomment-632398475,https://api.github.com/repos/simonw/datasette/issues/767,632398475,MDEyOklzc3VlQ29tbWVudDYzMjM5ODQ3NQ==,9599,simonw,2020-05-21T23:36:00Z,2020-05-21T23:36:00Z,OWNER,"Interesting. So this would take effect exclusively on pages that link to the queries, which I think just means this page: https://latest.datasette.io/fixtures
How about this for the design: there's a new optional key in the metadata configuration for a query called `""fragment""` - used like this:
```json
{
""databases"": {
""fixtures"": {
""queries"": {
""neighborhood_search"": {
""sql"": ""\nselect neighborhood, facet_cities.name, state\nfrom facetable\n join facet_cities\n on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%'\norder by neighborhood;\n"",
""title"": ""Search neighborhoods"",
""description_html"": ""Demonstrating simple like search"",
""name"": ""neighborhood_search"",
""fragment"": ""sql-format""
}
}
}
}
}
```
Because of the `""fragment"": ""sql-format""` bit there the link to that query from https://latest.datasette.io/fixtures would be to `https://latest.datasette.io/fixtures/neighborhood_search#sql-format`.
Does that cover what you need here? Have I missed anything?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",620969465,Allow to specify a URL fragment for canned queries,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253852,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631253852,MDEyOklzc3VlQ29tbWVudDYzMTI1Mzg1Mg==,9599,simonw,2020-05-20T05:56:17Z,2020-05-21T22:26:16Z,MEMBER,"I have a `deploy-demo.sh` script now:
```bash
#!/bin/bash
if [ -f public.db ]; then
rm public.db
fi
pipenv run dogsheep-photos create-subset photos.db public.db \
""select sha256 from apple_photos where albums like '%Public%'""
pipenv run sqlite-utils create-view public.db photos_on_a_map \
""select
date,
latitude,
longitude,
apple_photos.sha256,
uploads.ext,
json_object(
'title',
'Taken on ' || date,
'image',
'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=400',
'link',
'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=1200'
) as popup
from
apple_photos
join uploads on apple_photos.sha256 = uploads.sha256
where
latitude is not null
order by
date desc"" \
--replace
pipenv run datasette publish now public.db --project dogsheep-photos \
--about=dogsheep/dogsheep-photos \
--about_url=""https://github.com/dogsheep/dogsheep-photos"" \
--install=datasette-json-html \
--install=datasette-pretty-json \
--install=datasette-cluster-map>=0.10 \
--title ""Dogsheep Photos demo""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/simonw/datasette/issues/744#issuecomment-632305868,https://api.github.com/repos/simonw/datasette/issues/744,632305868,MDEyOklzc3VlQ29tbWVudDYzMjMwNTg2OA==,30607,aborruso,2020-05-21T19:43:23Z,2020-05-21T19:43:23Z,NONE,"@simonw now I have
```
Traceback (most recent call last):
File ""/home/aborruso/.local/bin/datasette"", line 8, in
sys.exit(cli())
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py"", line 103, in heroku
extra_metadata,
File ""/usr/lib/python3.7/contextlib.py"", line 112, in __enter__
return next(self.gen)
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py"", line 191, in temporary_heroku_directory
os.path.join(tmp.name, ""templates""),
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py"", line 605, in link_or_copy_directory
shutil.copytree(src, dst, copy_function=os.link, dirs_exist_ok=True)
TypeError: copytree() got an unexpected keyword argument 'dirs_exist_ok'
```
Do I must open a new issue?
Thank you","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632255088,https://api.github.com/repos/simonw/datasette/issues/744,632255088,MDEyOklzc3VlQ29tbWVudDYzMjI1NTA4OA==,30607,aborruso,2020-05-21T17:58:51Z,2020-05-21T17:58:51Z,NONE,"Thank you very much!!
I will try and I write you here","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632253073,https://api.github.com/repos/simonw/datasette/issues/744,632253073,MDEyOklzc3VlQ29tbWVudDYzMjI1MzA3Mw==,9599,simonw,2020-05-21T17:54:55Z,2020-05-21T17:54:55Z,OWNER,"I think this is the fix! I just landed it in master.
@aborruso you can install Datasette master to test it out like this:
pip install https://github.com/simonw/datasette/archive/cee671a58f417f827d1735b1abaa40716534ea67.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632249565,https://api.github.com/repos/simonw/datasette/issues/744,632249565,MDEyOklzc3VlQ29tbWVudDYzMjI0OTU2NQ==,30607,aborruso,2020-05-21T17:47:40Z,2020-05-21T17:47:40Z,NONE,"@simonw can I test it know? What I must do to update it?
Thank you","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632237195,https://api.github.com/repos/simonw/datasette/issues/744,632237195,MDEyOklzc3VlQ29tbWVudDYzMjIzNzE5NQ==,9599,simonw,2020-05-21T17:23:39Z,2020-05-21T17:23:48Z,OWNER,Actually maybe the answer here is to use `dirs_exist_ok=True` when calling `shutil.copytree`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632233393,https://api.github.com/repos/simonw/datasette/issues/744,632233393,MDEyOklzc3VlQ29tbWVudDYzMjIzMzM5Mw==,9599,simonw,2020-05-21T17:16:15Z,2020-05-21T17:17:43Z,OWNER,"I just hit this bug myself, or at least a variant of it!
https://github.com/simonw/museums/runs/697063068?check_suite_focus=true
```
2020-05-21T17:14:02.2596471Z Traceback (most recent call last):
2020-05-21T17:14:02.2599146Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py"", line 605, in link_or_copy_directory
2020-05-21T17:14:02.2599861Z shutil.copytree(src, dst, copy_function=os.link)
2020-05-21T17:14:02.2600377Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py"", line 554, in copytree
2020-05-21T17:14:02.2600786Z return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
2020-05-21T17:14:02.2601196Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py"", line 510, in _copytree
2020-05-21T17:14:02.2601580Z raise Error(errors)
2020-05-21T17:14:02.2604565Z shutil.Error: [('/home/runner/work/museums/museums/templates/row-browse-museums.html', '/tmp/tmpq47xi96y/datasette/templates/row-browse-museums.html',
""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/row-browse-museums.html' -> '/tmp/tmpq47xi96y/datasette/templates/row-browse-museums.html'""), ('/home/runner/work/museums/museums/templates/index.html', '/tmp/tmpq47xi96y/datasette/templates/index.html',
""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/index.html' -> '/tmp/tmpq47xi96y/datasette/templates/index.html'""), ('/home/runner/work/museums/museums/templates/query-browse-search.html', '/tmp/tmpq47xi96y/datasette/templates/query-browse-search.html',
""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/query-browse-search.html' -> '/tmp/tmpq47xi96y/datasette/templates/query-browse-search.html'""), ('/home/runner/work/museums/museums/templates/_analytics.html', '/tmp/tmpq47xi96y/datasette/templates/_analytics.html', ""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/_analytics.html' -> '/tmp/tmpq47xi96y/datasette/templates/_analytics.html'""), ('/home/runner/work/museums/museums/templates/_museum_card.html', '/tmp/tmpq47xi96y/datasette/templates/_museum_card.html', ""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/_museum_card.html' -> '/tmp/tmpq47xi96y/datasette/templates/_museum_card.html'""), ('/home/runner/work/museums/museums/templates/pages/about.html', '/tmp/tmpq47xi96y/datasette/templates/pages/about.html', ""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/pages/about.html' -> '/tmp/tmpq47xi96y/datasette/templates/pages/about.html'""), ('/home/runner/work/museums/museums/templates/pages/map.html', '/tmp/tmpq47xi96y/datasette/templates/pages/map.html', ""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/pages/map.html' -> '/tmp/tmpq47xi96y/datasette/templates/pages/map.html'"")]
2020-05-21T17:14:02.2605437Z
2020-05-21T17:14:02.2605797Z During handling of the above exception, another exception occurred:
2020-05-21T17:14:02.2606102Z
2020-05-21T17:14:02.2606423Z Traceback (most recent call last):
2020-05-21T17:14:02.2606817Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette"", line 8, in
2020-05-21T17:14:02.2607189Z sys.exit(cli())
2020-05-21T17:14:02.2607907Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
2020-05-21T17:14:02.2608347Z return self.main(*args, **kwargs)
2020-05-21T17:14:02.2609024Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 782, in main
2020-05-21T17:14:02.2609451Z rv = self.invoke(ctx)
2020-05-21T17:14:02.2610116Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
2020-05-21T17:14:02.2610550Z return _process_result(sub_ctx.command.invoke(sub_ctx))
2020-05-21T17:14:02.2611451Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
2020-05-21T17:14:02.2611989Z return _process_result(sub_ctx.command.invoke(sub_ctx))
2020-05-21T17:14:02.2612682Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
2020-05-21T17:14:02.2613117Z return ctx.invoke(self.callback, **ctx.params)
2020-05-21T17:14:02.2613801Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
2020-05-21T17:14:02.2614235Z return callback(*args, **kwargs)
2020-05-21T17:14:02.2614946Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/cloudrun.py"", line 111, in cloudrun
2020-05-21T17:14:02.2615371Z with temporary_docker_directory(
2020-05-21T17:14:02.2615769Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/contextlib.py"", line 113, in __enter__
2020-05-21T17:14:02.2616135Z return next(self.gen)
2020-05-21T17:14:02.2616957Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py"", line 392, in temporary_docker_directory
2020-05-21T17:14:02.2617403Z link_or_copy_directory(
2020-05-21T17:14:02.2618298Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py"", line 607, in link_or_copy_directory
2020-05-21T17:14:02.2619270Z shutil.copytree(src, dst)
2020-05-21T17:14:02.2619872Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py"", line 554, in copytree
2020-05-21T17:14:02.2622179Z return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
2020-05-21T17:14:02.2622609Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py"", line 455, in _copytree
2020-05-21T17:14:02.2622878Z os.makedirs(dst, exist_ok=dirs_exist_ok)
2020-05-21T17:14:02.2623157Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/os.py"", line 223, in makedirs
2020-05-21T17:14:02.2623405Z mkdir(name, mode)
2020-05-21T17:14:02.2623978Z FileExistsError: [Errno 17] File exists: '/tmp/tmpq47xi96y/datasette/templates'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631251707,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631251707,MDEyOklzc3VlQ29tbWVudDYzMTI1MTcwNw==,9599,simonw,2020-05-20T05:49:27Z,2020-05-21T15:58:42Z,MEMBER,Renaming this demo to `dogsheep-photos.dogsheep.net`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631127454,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631127454,MDEyOklzc3VlQ29tbWVudDYzMTEyNzQ1NA==,9599,simonw,2020-05-19T22:48:00Z,2020-05-21T15:58:32Z,MEMBER,"I built #23 to help with this.
$ dogsheep-photos create-subset photos.db public.db \
""select sha256 from apple_photos where albums like '%Public%'""
And publish with Vercel:
$ datasette publish now public.db --project dogsheep-photos \
--about=dogsheep/dogsheep-photos \
--about_url=""https://github.com/dogsheep/dogsheep-photos"" \
--install=datasette-json-html \
--install=datasette-cluster-map","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/24#issuecomment-631255206,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/24,631255206,MDEyOklzc3VlQ29tbWVudDYzMTI1NTIwNg==,9599,simonw,2020-05-20T06:00:25Z,2020-05-20T06:00:25Z,MEMBER,This needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621323348,Configurable URL for images,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253248,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631253248,MDEyOklzc3VlQ29tbWVudDYzMTI1MzI0OA==,9599,simonw,2020-05-20T05:54:18Z,2020-05-20T05:54:18Z,MEMBER,https://dogsheep-photos.dogsheep.net/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253136,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631253136,MDEyOklzc3VlQ29tbWVudDYzMTI1MzEzNg==,9599,simonw,2020-05-20T05:53:58Z,2020-05-20T05:53:58Z,MEMBER,"Updated deploy command:
```
datasette publish now public.db --project dogsheep-photos \
--about=dogsheep/dogsheep-photos \
--about_url=""https://github.com/dogsheep/dogsheep-photos"" \
--install=datasette-json-html \
--install=datasette-cluster-map \
--title ""Dogsheep Photos demo""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229485,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631229485,MDEyOklzc3VlQ29tbWVudDYzMTIyOTQ4NQ==,9599,simonw,2020-05-20T04:31:02Z,2020-05-20T04:31:02Z,MEMBER,https://pypi.org/project/dogsheep-photos/ is live.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229409,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631229409,MDEyOklzc3VlQ29tbWVudDYzMTIyOTQwOQ==,9599,simonw,2020-05-20T04:30:40Z,2020-05-20T04:30:40Z,MEMBER,https://pypi.org/project/photos-to-sqlite/ now links to dogsheep-photos.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227245,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631227245,MDEyOklzc3VlQ29tbWVudDYzMTIyNzI0NQ==,9599,simonw,2020-05-20T04:21:38Z,2020-05-20T04:21:38Z,MEMBER,I'm going to release 0.4 now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227020,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631227020,MDEyOklzc3VlQ29tbWVudDYzMTIyNzAyMA==,9599,simonw,2020-05-20T04:20:48Z,2020-05-20T04:21:16Z,MEMBER,Next time I push a release it will create `dogsheep-photos` on PyPI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227105,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631227105,MDEyOklzc3VlQ29tbWVudDYzMTIyNzEwNQ==,9599,simonw,2020-05-20T04:21:06Z,2020-05-20T04:21:06Z,MEMBER,Then I just need to push a final photos-to-sqlite release that updates the README to tell people about the name change.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226953,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631226953,MDEyOklzc3VlQ29tbWVudDYzMTIyNjk1Mw==,9599,simonw,2020-05-20T04:20:34Z,2020-05-20T04:20:34Z,MEMBER,"Huh, it looks like Circle CI picked up the name change automatically. https://app.circleci.com/pipelines/github/dogsheep/dogsheep-photos","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226572,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631226572,MDEyOklzc3VlQ29tbWVudDYzMTIyNjU3Mg==,9599,simonw,2020-05-20T04:18:52Z,2020-05-20T04:18:52Z,MEMBER,Need to reconfigure Circle CI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226481,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631226481,MDEyOklzc3VlQ29tbWVudDYzMTIyNjQ4MQ==,9599,simonw,2020-05-20T04:18:29Z,2020-05-20T04:18:29Z,MEMBER,I just renamed the repository.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/23#issuecomment-631120771,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/23,631120771,MDEyOklzc3VlQ29tbWVudDYzMTEyMDc3MQ==,9599,simonw,2020-05-19T22:32:48Z,2020-05-19T22:32:48Z,MEMBER,Documentation: https://github.com/dogsheep/photos-to-sqlite/blob/e2fab012551eed05278040b5d57e7373a1b9a0bf/README.md#creating-a-subset-database,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621280529,create-subset command for creating a publishable subset of a photos database,
https://github.com/simonw/sqlite-utils/issues/113#issuecomment-631084748,https://api.github.com/repos/simonw/sqlite-utils/issues/113,631084748,MDEyOklzc3VlQ29tbWVudDYzMTA4NDc0OA==,9599,simonw,2020-05-19T21:11:22Z,2020-05-19T21:11:22Z,OWNER,"Here's how `twitter-to-sqlite` does it: https://github.com/dogsheep/twitter-to-sqlite/blob/6be0ffcee24504fa2964b0e526842b8cfce7567b/twitter_to_sqlite/utils.py#L506-L511
```python
attach_sql = """"""
ATTACH DATABASE '{}' AS [{}];
"""""".format(
str(pathlib.Path(filepath).resolve()), alias
)
db.conn.execute(attach_sql)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621286870,Syntactic sugar for ATTACH DATABASE,
https://github.com/simonw/datasette/issues/729#issuecomment-630323870,https://api.github.com/repos/simonw/datasette/issues/729,630323870,MDEyOklzc3VlQ29tbWVudDYzMDMyMzg3MA==,9599,simonw,2020-05-18T17:20:02Z,2020-05-18T17:20:02Z,OWNER,"Deployed here: https://covid-19.datasettes.com/covid/economist_excess_deaths - and it helped me spot a bug! The floating point numbers there were being treated as strings:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629050775,https://api.github.com/repos/simonw/datasette/issues/729,629050775,MDEyOklzc3VlQ29tbWVudDYyOTA1MDc3NQ==,9599,simonw,2020-05-15T06:17:12Z,2020-05-15T06:17:12Z,OWNER,"
I don't like how the column headers themselves are no longer black in mobile view.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629043480,https://api.github.com/repos/simonw/datasette/issues/729,629043480,MDEyOklzc3VlQ29tbWVudDYyOTA0MzQ4MA==,9599,simonw,2020-05-15T05:55:44Z,2020-05-15T05:55:44Z,OWNER,Live demo: https://latest.datasette.io/fixtures/sortable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629042437,https://api.github.com/repos/simonw/datasette/issues/729,629042437,MDEyOklzc3VlQ29tbWVudDYyOTA0MjQzNw==,9599,simonw,2020-05-15T05:52:39Z,2020-05-15T05:52:39Z,OWNER,`type-pk` is a more accurate name than `type-link` - since people might assume that `type-link` refers to columns containing a link or a URL or similar.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629033753,https://api.github.com/repos/simonw/datasette/issues/729,629033753,MDEyOklzc3VlQ29tbWVudDYyOTAzMzc1Mw==,9599,simonw,2020-05-15T05:21:42Z,2020-05-15T05:21:51Z,OWNER,I'm going to make all integers AND floats show up as `#666666`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629033619,https://api.github.com/repos/simonw/datasette/issues/729,629033619,MDEyOklzc3VlQ29tbWVudDYyOTAzMzYxOQ==,9599,simonw,2020-05-15T05:21:17Z,2020-05-15T05:21:30Z,OWNER,"Turns out `1.0` and `2` stored in a float column already displays like that:
```
echo '[{""foo"": 1.0, ""bar"": 1}, {""foo"": 2, ""bar"": 2}]' | sqlite-utils insert /tmp/db.db t -
```
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629032619,https://api.github.com/repos/simonw/datasette/issues/729,629032619,MDEyOklzc3VlQ29tbWVudDYyOTAzMjYxOQ==,9599,simonw,2020-05-15T05:17:53Z,2020-05-15T05:18:29Z,OWNER,"I like the gray. The purple is a bit confusing.
I could have both integers and floats show up as gray? The floating point should help show which is which.
Maybe I could default floating point columns with integer values in them to `11.0` just to make it clear that they are floats?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629032425,https://api.github.com/repos/simonw/datasette/issues/729,629032425,MDEyOklzc3VlQ29tbWVudDYyOTAzMjQyNQ==,9599,simonw,2020-05-15T05:17:09Z,2020-05-15T05:17:39Z,OWNER,"Not sure the best way to do this. Here's an example using very subtle colour changes - but I'm worried it's not at all discoverable:
![]()
That's with `#4c0077` for floats and `#666666` for integers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-628405453,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,628405453,MDEyOklzc3VlQ29tbWVudDYyODQwNTQ1Mw==,41546558,RhetTbull,2020-05-14T05:59:53Z,2020-05-14T05:59:53Z,CONTRIBUTOR,"I've added support for the above exif data to [v0.28.17](https://github.com/RhetTbull/osxphotos/releases/tag/v0.28.17) of osxphotos. `PhotoInfo.exif_info` will return an `ExifInfo` [dataclass](https://docs.python.org/3/library/dataclasses.html) object with the following properties:
```python
flash_fired: bool
iso: int
metering_mode: int
sample_rate: int
track_format: int
white_balance: int
aperture: float
bit_rate: float
duration: float
exposure_bias: float
focal_length: float
fps: float
latitude: float
longitude: float
shutter_speed: float
camera_make: str
camera_model: str
codec: str
lens_model: str
```
It's not all the EXIF data available in most files but is the data Photos deems important to save. Of course, you can get all the exif_data
Note: this only works in Photos 5. As best as I can tell, EXIF data is not stored in the database for earlier versions. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118,Try out ExifReader,
https://github.com/simonw/sqlite-utils/issues/112#issuecomment-627036749,https://api.github.com/repos/simonw/sqlite-utils/issues/112,627036749,MDEyOklzc3VlQ29tbWVudDYyNzAzNjc0OQ==,9599,simonw,2020-05-12T00:27:24Z,2020-05-12T00:27:24Z,OWNER,Here's an example of some code that would be cleaner with this mechanism: https://github.com/dogsheep/swarm-to-sqlite/blob/f4a82633da927cde672c9d9af92930bfca2e3ddf/swarm_to_sqlite/utils.py#L120-L143,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616271236,"add_foreign_key(...., ignore=True)",
https://github.com/simonw/sqlite-utils/issues/112#issuecomment-627036475,https://api.github.com/repos/simonw/sqlite-utils/issues/112,627036475,MDEyOklzc3VlQ29tbWVudDYyNzAzNjQ3NQ==,9599,simonw,2020-05-12T00:26:25Z,2020-05-12T00:26:48Z,OWNER,"Question: if you use `ignore=True` should it still raise an error if one of the columns you referenced does not exist?
That's tricky. I'm leaning towards ""that's still an error"".
Are we ignoring an already existing foreign key, or ignoring all errors?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616271236,"add_foreign_key(...., ignore=True)",
https://github.com/simonw/sqlite-utils/issues/112#issuecomment-627035928,https://api.github.com/repos/simonw/sqlite-utils/issues/112,627035928,MDEyOklzc3VlQ29tbWVudDYyNzAzNTkyOA==,9599,simonw,2020-05-12T00:24:16Z,2020-05-12T00:24:51Z,OWNER,"I would add `ignore=True/False` as a parameter here:
https://github.com/simonw/sqlite-utils/blob/af3f81b540923f2cf04c76cfa81b0d811c0084bf/sqlite_utils/db.py#L731-L761","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616271236,"add_foreign_key(...., ignore=True)",
https://github.com/simonw/sqlite-utils/issues/112#issuecomment-627036010,https://api.github.com/repos/simonw/sqlite-utils/issues/112,627036010,MDEyOklzc3VlQ29tbWVudDYyNzAzNjAxMA==,9599,simonw,2020-05-12T00:24:37Z,2020-05-12T00:24:37Z,OWNER,"Here's how it works for `create_view`:
https://github.com/simonw/sqlite-utils/blob/af3f81b540923f2cf04c76cfa81b0d811c0084bf/sqlite_utils/db.py#L327-L336","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616271236,"add_foreign_key(...., ignore=True)",
https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-627007458,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,627007458,MDEyOklzc3VlQ29tbWVudDYyNzAwNzQ1OA==,41546558,RhetTbull,2020-05-11T22:51:52Z,2020-05-11T22:52:26Z,CONTRIBUTOR,"I'm not familiar with `ExifReader`. I wrote my own wrapper around `exiftool` because I wanted a simple way to write EXIF data when exporting photos (e.g. writing out to PersonInImage and keywords to IPTC:Keywords) and the existing python packages like [pyexiftool](https://github.com/smarnach/pyexiftool) didn't do quite what I wanted. If all you're after is the camera and shot info, that's available in `ZEXTENDEDATTRIBUTES` table. I've got an open issue [#11](https://github.com/RhetTbull/osxphotos/issues/11) to add this to osxphotos but it hasn't bubbled to the top of my backlog yet.
osxphotos will give you the location info: `PhotoInfo.location` returns a tuple of (lat, lon) though this info is in ZEXTENDEDATTRIBUTES too (though it might not be correct as I believe Photos creates this table at import and the user might have changed the location of a photo, e.g. if camera didn't have GPS).
```sql
CREATE TABLE ZEXTENDEDATTRIBUTES (
Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER,
Z_OPT INTEGER, ZFLASHFIRED INTEGER,
ZISO INTEGER, ZMETERINGMODE INTEGER,
ZSAMPLERATE INTEGER, ZTRACKFORMAT INTEGER,
ZWHITEBALANCE INTEGER, ZASSET INTEGER,
ZAPERTURE FLOAT, ZBITRATE FLOAT, ZDURATION FLOAT,
ZEXPOSUREBIAS FLOAT, ZFOCALLENGTH FLOAT,
ZFPS FLOAT, ZLATITUDE FLOAT, ZLONGITUDE FLOAT,
ZSHUTTERSPEED FLOAT, ZCAMERAMAKE VARCHAR,
ZCAMERAMODEL VARCHAR, ZCODEC VARCHAR,
ZLENSMODEL VARCHAR
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118,Try out ExifReader,
https://github.com/simonw/datasette/issues/699#issuecomment-626991001,https://api.github.com/repos/simonw/datasette/issues/699,626991001,MDEyOklzc3VlQ29tbWVudDYyNjk5MTAwMQ==,8431341,zeluspudding,2020-05-11T22:06:34Z,2020-05-11T22:06:34Z,NONE,Very nice! Thank you for sharing that :+1: :) Will try it out!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-626945281,https://api.github.com/repos/simonw/datasette/issues/699,626945281,MDEyOklzc3VlQ29tbWVudDYyNjk0NTI4MQ==,9599,simonw,2020-05-11T20:32:33Z,2020-05-11T20:32:33Z,OWNER,"I did have a bit of trouble with this one-off plugin getting it to load in the correct order - since I need authentication to work if EITHER the one-off plugin spots a token or my `datasette-auth-github` plugin authenticates the user.
That's why I want authentication as a core Datasette concept - so plugins like these can easily play together in a predictable manner.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-626943809,https://api.github.com/repos/simonw/datasette/issues/699,626943809,MDEyOklzc3VlQ29tbWVudDYyNjk0MzgwOQ==,9599,simonw,2020-05-11T20:30:07Z,2020-05-11T20:31:18Z,OWNER,"I implemented bearer tokens in a private project of mine as a one-off plugin. I'm going to extract that out into a installable plugin soon. For the moment, my `plugins/token_auth.py` file looks like this:
```python
from datasette import hookimpl
import secrets
class TokenAuth:
def __init__(
self, app, secret, auth,
):
self.app = app
self.secret = secret
self.auth = auth
async def __call__(self, scope, receive, send):
if scope.get(""type"") != ""http"":
return await self.app(scope, receive, send)
authorization = dict(scope.get(""headers"") or {}).get(b""authorization"") or b""""
expected = ""Bearer {}"".format(self.secret).encode(""utf8"")
if secrets.compare_digest(authorization, expected):
scope = dict(scope, auth=self.auth)
return await self.app(scope, receive, send)
@hookimpl(trylast=True)
def asgi_wrapper(datasette):
config = datasette.plugin_config(""token-auth"") or {}
secret = config.get(""secret"")
auth = config.get(""auth"")
def wrap_with_asgi_auth(app):
return TokenAuth(app, secret=secret, auth=auth,)
return wrap_with_asgi_auth
```
Then I have the following in `metadata.json`:
```json
{
""plugins"": {
""token-auth"": {
""auth"": {
""name"": ""token-bot""
},
""secret"": {
""$env"": ""TOKEN_SECRET""
}
}
}
}
```
And a `TOKEN_SECRET` environment variable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-626941278,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,626941278,MDEyOklzc3VlQ29tbWVudDYyNjk0MTI3OA==,9599,simonw,2020-05-11T20:25:58Z,2020-05-11T20:25:58Z,MEMBER,"Interesting - do you know if there's anything the `exiftool` process handles that `ExifReader` doesn't?
I'm actually just going to extract a subset of the EXIF data at first - since the original photo files will always be available I don't feel the need to get everything out for the first step.
My plan is to use EXIF to help support photo collections that aren't in Apple Photos - I'm going to build a database table keyed by the `sha256` of each photo that extracts the camera make, lens, a few settings (ISO, aperture etc) and the GPS lat/lon.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118,Try out ExifReader,
https://github.com/simonw/sqlite-utils/issues/30#issuecomment-626903632,https://api.github.com/repos/simonw/sqlite-utils/issues/30,626903632,MDEyOklzc3VlQ29tbWVudDYyNjkwMzYzMg==,9599,simonw,2020-05-11T19:17:17Z,2020-05-11T19:17:17Z,OWNER,I don't think this is a useful feature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",461215118,Option to open database in read-only mode,
https://github.com/simonw/datasette/issues/765#issuecomment-626874374,https://api.github.com/repos/simonw/datasette/issues/765,626874374,MDEyOklzc3VlQ29tbWVudDYyNjg3NDM3NA==,9599,simonw,2020-05-11T18:25:43Z,2020-05-11T18:25:43Z,OWNER,"Trickiness here is what tag to use. Do we use the tag of the installed copy of Datasette that is running `datasette publish`? That would mean you don't get the latest features in your deployed release.
Could hit an API to figure out the most recent version? Bit odd.
Could we output the version of Datasette that was deployed and tell people they can run `--latest` to force the most recent release?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616087149,publish heroku should default to currently tagged version,
https://github.com/simonw/datasette/issues/764#issuecomment-626813924,https://api.github.com/repos/simonw/datasette/issues/764,626813924,MDEyOklzc3VlQ29tbWVudDYyNjgxMzkyNA==,9599,simonw,2020-05-11T16:36:06Z,2020-05-11T16:36:06Z,OWNER,Made a TIL: https://github.com/simonw/til/blob/master/pypi/project-links.md,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616012427,Add PyPI project urls to setup.py,
https://github.com/simonw/datasette/issues/764#issuecomment-626810129,https://api.github.com/repos/simonw/datasette/issues/764,626810129,MDEyOklzc3VlQ29tbWVudDYyNjgxMDEyOQ==,9599,simonw,2020-05-11T16:28:43Z,2020-05-11T16:28:43Z,OWNER,"For Datasette I'll go with:
* Documentation
* Changelog
* Live demo
* Source code
* Issues
* CI","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616012427,Add PyPI project urls to setup.py,
https://github.com/simonw/datasette/issues/764#issuecomment-626808805,https://api.github.com/repos/simonw/datasette/issues/764,626808805,MDEyOklzc3VlQ29tbWVudDYyNjgwODgwNQ==,9599,simonw,2020-05-11T16:26:17Z,2020-05-11T16:27:27Z,OWNER,"The keys here can be anything: https://packaging.python.org/guides/distributing-packages-using-setuptools/#project-urls
So where do the icons come from? Turns out the PyPI site has special case rules for the icons here: https://github.com/pypa/warehouse/blob/2f00f4a9f208546ff0ebb6a6e61439021ca60a43/warehouse/templates/packaging/detail.html#L16-L60","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616012427,Add PyPI project urls to setup.py,
https://github.com/simonw/datasette/issues/699#issuecomment-626807487,https://api.github.com/repos/simonw/datasette/issues/699,626807487,MDEyOklzc3VlQ29tbWVudDYyNjgwNzQ4Nw==,8431341,zeluspudding,2020-05-11T16:23:57Z,2020-05-11T16:24:59Z,NONE,`Authorization: bearer xxx` auth for API keys is a plus plus for me. Looked into just adding this into your `Flask` logic but learned this project doesn't use flask. Interesting 🤔,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-626667235,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,626667235,MDEyOklzc3VlQ29tbWVudDYyNjY2NzIzNQ==,41546558,RhetTbull,2020-05-11T12:20:34Z,2020-05-11T12:20:34Z,CONTRIBUTOR,"@simonw FYI, osxphotos includes a built in ExifTool class that uses [exiftool](https://exiftool.org/) to read and write exif data. It's not exposed yet in the docs because I really only use it right now in the osphotos command line interface to write tags when exporting. In v0.28.16 (just pushed) I added an ExifTool.as_dict() method which will give you a dict with all the exif tags in a file. For example:
```python
import osxphotos
photos = osxphotos.PhotosDB().photos()
exiftool = osxphotos.exiftool.ExifTool(photos[0].path)
exifdata = exiftool.as_dict()
tags = exifdata[""IPTC:Keywords""]
```
Not as elegant perhaps as a python only implementation because ExifTool has to make subprocess calls to an external tool but exiftool is by far the best tool available for reading and writing EXIF data and it does support HEIC.
As for implementation, ExifTool uses a singleton pattern so the first time you instantiate it, it spawns an IPC to exiftool but then keeps it open and uses the same process for any subsequent calls (even on different files). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118,Try out ExifReader,
https://github.com/simonw/sqlite-utils/issues/111#issuecomment-626431562,https://api.github.com/repos/simonw/sqlite-utils/issues/111,626431562,MDEyOklzc3VlQ29tbWVudDYyNjQzMTU2Mg==,9599,simonw,2020-05-11T01:58:36Z,2020-05-11T01:58:36Z,OWNER,Released in 2.9 https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v2-9,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615477131,sqlite-utils drop-table and drop-view commands,
https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626431484,https://api.github.com/repos/simonw/sqlite-utils/issues/110,626431484,MDEyOklzc3VlQ29tbWVudDYyNjQzMTQ4NA==,9599,simonw,2020-05-11T01:58:20Z,2020-05-11T01:58:20Z,OWNER,Released in 2.9 https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v2-9,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613755043,Support decimal.Decimal type,
https://github.com/simonw/sqlite-utils/issues/111#issuecomment-626417220,https://api.github.com/repos/simonw/sqlite-utils/issues/111,626417220,MDEyOklzc3VlQ29tbWVudDYyNjQxNzIyMA==,9599,simonw,2020-05-11T00:46:04Z,2020-05-11T00:46:04Z,OWNER,"Docs:
* https://sqlite-utils.readthedocs.io/en/latest/cli.html#dropping-tables
* https://sqlite-utils.readthedocs.io/en/latest/cli.html#dropping-views","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615477131,sqlite-utils drop-table and drop-view commands,
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626396379,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626396379,MDEyOklzc3VlQ29tbWVudDYyNjM5NjM3OQ==,41546558,RhetTbull,2020-05-10T22:01:48Z,2020-05-10T22:01:48Z,CONTRIBUTOR,"Frustrates me when package authors create a ""drop in"" replacement with the same import name...this kind of thing has bitten me more than once! Would've been nicer I think for bpylist2 to do ""import bpylist2 as bpylist""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395781,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395781,MDEyOklzc3VlQ29tbWVudDYyNjM5NTc4MQ==,9599,simonw,2020-05-10T21:57:09Z,2020-05-10T21:57:09Z,MEMBER,"Yes, I just recreated my virtual environment from scratch and the error went away.
The problem occurred when I ran `pip install datasette-bplist` in the same virtual environment - https://github.com/simonw/datasette-bplist/blob/master/setup.py depends on `bpylist` which is incompatible with `bpylist2`.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395641,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395641,MDEyOklzc3VlQ29tbWVudDYyNjM5NTY0MQ==,41546558,RhetTbull,2020-05-10T21:55:54Z,2020-05-10T21:55:54Z,CONTRIBUTOR,Did removing old bpylist solve the original problem or do you still have a photo that throws circular reference?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395507,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395507,MDEyOklzc3VlQ29tbWVudDYyNjM5NTUwNw==,41546558,RhetTbull,2020-05-10T21:54:45Z,2020-05-10T21:54:45Z,CONTRIBUTOR,"@simonw does Photos show valid reverse geolocation info? Are you sure you're using [bpylist2](https://github.com/xa4a/bpylist2) and not bpylist? They're both unfortunately imported as ""bpylist"" so if you somehow got the wrong (original bpylist) version installed, it could be the issue. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395209,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395209,MDEyOklzc3VlQ29tbWVudDYyNjM5NTIwOQ==,9599,simonw,2020-05-10T21:52:42Z,2020-05-10T21:52:42Z,MEMBER,"Aha! It looks like I accidentally installed the old bplist into the same environment:
```
$ pip freeze | grep bpylist
bpylist==0.1.4
bpylist2==3.0.0
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395103,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395103,MDEyOklzc3VlQ29tbWVudDYyNjM5NTEwMw==,9599,simonw,2020-05-10T21:51:36Z,2020-05-10T21:51:36Z,MEMBER,"@RhetTbull I tried that workaround and it turns out I'm getting this error on ALL of my photos now!
It's weird: a few day ago this wasn't happening. Now it's happening to everything. I'm not sure what I might have changed. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626390317,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626390317,MDEyOklzc3VlQ29tbWVudDYyNjM5MDMxNw==,41546558,RhetTbull,2020-05-10T21:11:24Z,2020-05-10T21:50:58Z,CONTRIBUTOR,"Ugh....Yeah, I think easiest is to catch the exception and return no place as you suggest. This particular bit of code involves un-archiving a serialized NSKeyedArchiver which uses an object table and it is certainly possible to create a circular reference that way. Because this is happening in the decode, the circular reference must be in the original data. Does Photos show valid reverse geolocation info for the photo in question? If so, Photos may be doing something beyond a simple decode of the binary plist. For now, I'll push a patch to catch the exception.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626394989,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626394989,MDEyOklzc3VlQ29tbWVudDYyNjM5NDk4OQ==,9599,simonw,2020-05-10T21:50:36Z,2020-05-10T21:50:36Z,MEMBER,https://github.com/Marketcircle/bpylist/pull/2 looks relevant here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391307,https://api.github.com/repos/simonw/sqlite-utils/issues/110,626391307,MDEyOklzc3VlQ29tbWVudDYyNjM5MTMwNw==,9599,simonw,2020-05-10T21:19:04Z,2020-05-10T21:19:04Z,OWNER,"I'm going to set it up so that Python `decimal.Decimal` is treated in a FLOAT column, until someone convinces me otherwise!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613755043,Support decimal.Decimal type,
https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391217,https://api.github.com/repos/simonw/sqlite-utils/issues/110,626391217,MDEyOklzc3VlQ29tbWVudDYyNjM5MTIxNw==,9599,simonw,2020-05-10T21:18:28Z,2020-05-10T21:18:28Z,OWNER,"`sqlite-utils` currently treats the SQLite NUMERIC concept as a float:
https://github.com/simonw/sqlite-utils/blob/daf2a245aa4e0b0cf62a94c1232cfb858821803b/tests/test_column_affinity.py#L28-L30","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613755043,Support decimal.Decimal type,
https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391063,https://api.github.com/repos/simonw/sqlite-utils/issues/110,626391063,MDEyOklzc3VlQ29tbWVudDYyNjM5MTA2Mw==,9599,simonw,2020-05-10T21:17:16Z,2020-05-10T21:17:16Z,OWNER,"From https://www.sqlite.org/datatype3.html#type_affinity :
> A column with NUMERIC affinity may contain values using all five storage classes. When text data is inserted into a NUMERIC column, the storage class of the text is converted to INTEGER or REAL (in order of preference) if the text is a well-formed integer or real literal, respectively. If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL. For conversions between TEXT and REAL storage classes, only the first 15 significant decimal digits of the number are preserved. If the TEXT value is not a well-formed integer or real literal, then the value is stored as TEXT.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613755043,Support decimal.Decimal type,
https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626390822,https://api.github.com/repos/simonw/sqlite-utils/issues/110,626390822,MDEyOklzc3VlQ29tbWVudDYyNjM5MDgyMg==,9599,simonw,2020-05-10T21:15:28Z,2020-05-10T21:15:28Z,OWNER,"https://www.sqlite.org/datatype3.html#affinity_name_examples suggests that `DECIMAL(10,5)` should be mapped to the SQLite affinity of `NUMERIC` - which I've not worked with before.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613755043,Support decimal.Decimal type,