` tags: https://primer.style/css/components/dropdown","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/issues/849#issuecomment-644584075,https://api.github.com/repos/simonw/datasette/issues/849,644584075,MDEyOklzc3VlQ29tbWVudDY0NDU4NDA3NQ==,9599,simonw,2020-06-16T07:24:08Z,2020-06-16T07:24:08Z,OWNER,This guide is fantastic - I'll be following it closely: https://github.com/chancancode/branch-rename/blob/main/README.md - in particular the Action to mirror master and main for a while.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/849#issuecomment-644384787,https://api.github.com/repos/simonw/datasette/issues/849,644384787,MDEyOklzc3VlQ29tbWVudDY0NDM4NDc4Nw==,9599,simonw,2020-06-15T20:56:07Z,2020-06-15T20:56:19Z,OWNER,"The big question is how this impacts existing CI configuration. `datasette-psutil` is configured to use Circle CI, what happens if I push a new commit?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/849#issuecomment-644384417,https://api.github.com/repos/simonw/datasette/issues/849,644384417,MDEyOklzc3VlQ29tbWVudDY0NDM4NDQxNw==,9599,simonw,2020-06-15T20:55:23Z,2020-06-15T20:55:23Z,OWNER,"I'm doing https://github.com/simonw/datasette-psutil first.
In my local checkout:
```
git branch -m master main
git push -u origin main
```
(Thanks, https://www.hanselman.com/blog/EasilyRenameYourGitDefaultBranchFromMasterToMain.aspx)
Then in https://github.com/simonw/datasette-psutil/settings/branches I changed the default branch to `main`.
![]()
Links to these docs: https://help.github.com/en/github/administering-a-repository/setting-the-default-branch
That worked! https://github.com/simonw/datasette-psutil
One catch, which I think will impact my most widely used repos the most (like datasette) - linking to a specific file now looks like this:
https://github.com/simonw/datasette-psutil/blob/main/datasette_psutil/__init__.py
The old https://github.com/simonw/datasette-psutil/blob/master/datasette_psutil/__init__.py link is presumably frozen in time?
I've definitely got links spread around the web to my ""most recent version of this code"" that would use the `master` reference, which would need to be updated to `main` instead. Most of those are probably in the Datasette docs and on my blog though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/849#issuecomment-644322234,https://api.github.com/repos/simonw/datasette/issues/849,644322234,MDEyOklzc3VlQ29tbWVudDY0NDMyMjIzNA==,9599,simonw,2020-06-15T19:06:16Z,2020-06-15T19:06:16Z,OWNER,I'll make this change on a few of my other repos first to make sure I haven't missed any tricky edge-cases.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/691#issuecomment-643709037,https://api.github.com/repos/simonw/datasette/issues/691,643709037,MDEyOklzc3VlQ29tbWVudDY0MzcwOTAzNw==,49260,amjith,2020-06-14T02:35:16Z,2020-06-14T02:35:16Z,CONTRIBUTOR,"The server should reload in the `config_dir` mode.
Ref: #848","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",574021194,--reload sould reload server if code in --plugins-dir changes,
https://github.com/simonw/datasette/issues/847#issuecomment-643704730,https://api.github.com/repos/simonw/datasette/issues/847,643704730,MDEyOklzc3VlQ29tbWVudDY0MzcwNDczMA==,9599,simonw,2020-06-14T01:28:34Z,2020-06-14T01:28:34Z,OWNER,"Here's the plugin that adds those custom SQLite functions:
```python
from datasette import hookimpl
from coverage.numbits import register_sqlite_functions
@hookimpl
def prepare_connection(conn):
register_sqlite_functions(conn)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638259643,Take advantage of .coverage being a SQLite database,
https://github.com/simonw/datasette/issues/847#issuecomment-643704565,https://api.github.com/repos/simonw/datasette/issues/847,643704565,MDEyOklzc3VlQ29tbWVudDY0MzcwNDU2NQ==,9599,simonw,2020-06-14T01:26:56Z,2020-06-14T01:26:56Z,OWNER,"On closer inspection, I don't know if there's that much useful stuff you can do with the data from `.coverage` on its own.
Consider the following query against a `.coverage` run against Datasette itself:
```sql
select file_id, context_id, numbits_to_nums(numbits) from line_bits
```
![]()
It looks like this tells me which lines of which files were executed during the test run. But... without the actual source code, I don't think I can calculate the coverage percentage for each file. I don't want to count comment lines or whitespace as untested for example, and I don't know how many lines were in the file.
If I'm right that it's not possible to calculate percentage coverage from just the `.coverage` data then I'll need to do something a bit more involved - maybe parsing the `coverage.xml` report and loading that into my own schema?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638259643,Take advantage of .coverage being a SQLite database,
https://github.com/simonw/datasette/issues/847#issuecomment-643702715,https://api.github.com/repos/simonw/datasette/issues/847,643702715,MDEyOklzc3VlQ29tbWVudDY0MzcwMjcxNQ==,9599,simonw,2020-06-14T01:03:30Z,2020-06-14T01:03:40Z,OWNER,Filed a related issue with some ideas against `coveragepy` here: https://github.com/nedbat/coveragepy/issues/999,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638259643,Take advantage of .coverage being a SQLite database,
https://github.com/simonw/datasette/issues/846#issuecomment-643699583,https://api.github.com/repos/simonw/datasette/issues/846,643699583,MDEyOklzc3VlQ29tbWVudDY0MzY5OTU4Mw==,9599,simonw,2020-06-14T00:26:31Z,2020-06-14T00:26:31Z,OWNER,"That seems to have fixed the problem, at least for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638241779,"""Too many open files"" error running tests",
https://github.com/simonw/datasette/issues/846#issuecomment-643699063,https://api.github.com/repos/simonw/datasette/issues/846,643699063,MDEyOklzc3VlQ29tbWVudDY0MzY5OTA2Mw==,9599,simonw,2020-06-14T00:22:32Z,2020-06-14T00:22:32Z,OWNER,"Idea: `num_sql_threads` (described as ""Number of threads in the thread pool for executing SQLite queries"") defaults to 3 - can I knock that down to 1 in the tests and open less connections as a result?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638241779,"""Too many open files"" error running tests",
https://github.com/simonw/datasette/issues/846#issuecomment-643698790,https://api.github.com/repos/simonw/datasette/issues/846,643698790,MDEyOklzc3VlQ29tbWVudDY0MzY5ODc5MA==,9599,simonw,2020-06-14T00:20:42Z,2020-06-14T00:20:42Z,OWNER,"Released a new plugin, `datasette-psutil`, as a side-effect of this investigation: https://github.com/simonw/datasette-psutil","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638241779,"""Too many open files"" error running tests",
https://github.com/simonw/datasette/issues/846#issuecomment-643685669,https://api.github.com/repos/simonw/datasette/issues/846,643685669,MDEyOklzc3VlQ29tbWVudDY0MzY4NTY2OQ==,9599,simonw,2020-06-13T22:24:22Z,2020-06-13T22:24:22Z,OWNER,"I tried this experiment:
```python
import sqlite3, psutil
def show_things():
conn = sqlite3.connect(""fixtures.db"")
tables = [r[0] for r in conn.execute(""select * from sqlite_master"").fetchall()]
return tables
print(psutil.Process().open_files())
print(show_things())
print(psutil.Process().open_files())
```
To see if the connection would be automatically released when the `conn` variable was garbage collected at the end of the function... and it was correctly released - the two calls to `open_files()` showed that the file did not remain open.
Likewise:
```
In [11]: conn = sqlite3.connect(""fixtures.db"")
In [12]: psutil.Process().open_files()
Out[12]:
[popenfile(path='/Users/simon/.ipython/profile_default/history.sqlite', fd=4),
popenfile(path='/Users/simon/.ipython/profile_default/history.sqlite', fd=5),
popenfile(path='/Users/simon/Dropbox/Development/datasette/fixtures.db', fd=12)]
In [13]: del conn
In [14]: psutil.Process().open_files()
Out[14]:
[popenfile(path='/Users/simon/.ipython/profile_default/history.sqlite', fd=4),
popenfile(path='/Users/simon/.ipython/profile_default/history.sqlite', fd=5)]
```
So presumably there's something about the way my pytest fixtures work that's causing the many different `Datasette()` instances and their underlying SQLite connections that I create not to be cleaned up later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638241779,"""Too many open files"" error running tests",
https://github.com/simonw/datasette/issues/846#issuecomment-643685333,https://api.github.com/repos/simonw/datasette/issues/846,643685333,MDEyOklzc3VlQ29tbWVudDY0MzY4NTMzMw==,9599,simonw,2020-06-13T22:19:38Z,2020-06-13T22:19:38Z,OWNER,That's 91 open files but only 29 unique filenames.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638241779,"""Too many open files"" error running tests",
https://github.com/simonw/datasette/issues/846#issuecomment-643685207,https://api.github.com/repos/simonw/datasette/issues/846,643685207,MDEyOklzc3VlQ29tbWVudDY0MzY4NTIwNw==,9599,simonw,2020-06-13T22:18:01Z,2020-06-13T22:18:01Z,OWNER,"This shows currently open files (after `pip install psutil`):
```
import psutil
psutil.Process().open_files()
```
I ran it inside `pytest -x --pdb` and got this:
```
> /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.7/site-packages/jinja2/utils.py(154)open_if_exists()
-> return open(filename, mode)
(Pdb) import psutil
(Pdb) psutil.Process().open_files()
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp9uhx5d8x/fixtures.db', fd=10),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyfw44ica/fixtures.dot.db', fd=11),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyrg6g48b/fixtures.db', fd=12),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=13),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=14),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=15),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=16),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=17),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=18),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp33kkg62s/fixtures.db', fd=19),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpng4lg84_/fixtures.db', fd=20),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp9uhx5d8x/fixtures.db', fd=21),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp9uhx5d8x/fixtures.db', fd=22),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp9uhx5d8x/fixtures.db', fd=23),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmph11oalw_/fixtures.db', fd=24),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyfw44ica/fixtures.dot.db', fd=25),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyfw44ica/fixtures.dot.db', fd=26),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyfw44ica/fixtures.dot.db', fd=27),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpiorb2bo9/fixtures.db', fd=28),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyrg6g48b/fixtures.db', fd=29),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyrg6g48b/fixtures.db', fd=30),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpyrg6g48b/fixtures.db', fd=31),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprvyj5udv/fixtures.db', fd=32),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpng4lg84_/fixtures.db', fd=33),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpng4lg84_/fixtures.db', fd=34),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpng4lg84_/fixtures.db', fd=35),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpb_l6gmq0/fixtures.db', fd=36),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmph11oalw_/extra database.db', fd=40),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/fixtures.db', fd=41),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpiorb2bo9/fixtures.db', fd=42),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpiorb2bo9/fixtures.db', fd=43),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpiorb2bo9/fixtures.db', fd=44),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmph11oalw_/fixtures.db', fd=45),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmph11oalw_/fixtures.db', fd=52),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/fixtures.db', fd=53),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprvyj5udv/fixtures.db', fd=54),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprvyj5udv/fixtures.db', fd=55),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprvyj5udv/fixtures.db', fd=56),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoveuwqn6/fixtures.db', fd=57),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpb_l6gmq0/fixtures.db', fd=61),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpb_l6gmq0/fixtures.db', fd=62),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpb_l6gmq0/fixtures.db', fd=63),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp_j4h9mrn/fixtures.db', fd=64),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/fixtures.db', fd=65),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/fixtures.db', fd=66),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/extra database.db', fd=67),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/extra database.db', fd=68),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/fixtures.db', fd=69),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf0py4thp/extra database.db', fd=70)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpub3eodj1/fixtures.db', fd=71)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/fixtures.db', fd=72)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo.db', fd=73)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo.db', fd=74)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/fixtures.db', fd=75)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/fixtures.db', fd=76)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo.db', fd=77)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo-bar.db', fd=78)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo-bar.db', fd=79)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwgcnmg4b/foo-bar.db', fd=80)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-4/config-dir0/immutable.db', fd=81),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoveuwqn6/fixtures.db', fd=82),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoveuwqn6/fixtures.db', fd=83),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoveuwqn6/fixtures.db', fd=84),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp44w5d5wo/fixtures.db', fd=85),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp_j4h9mrn/fixtures.db', fd=86),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp_j4h9mrn/fixtures.db', fd=87),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp_j4h9mrn/fixtures.db', fd=88),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpvu7h14uy/fixtures.db', fd=89),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-4/config-dir0/demo.db', fd=119),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-4/config-dir0/demo.db', fd=120),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-4/config-dir0/demo.db', fd=121),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0xcnrjag/fixtures.db', fd=122),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpub3eodj1/fixtures.db', fd=123),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpub3eodj1/fixtures.db', fd=124),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpub3eodj1/fixtures.db', fd=125),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfz8go8rk/fixtures.db', fd=126),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp44w5d5wo/fixtures.db', fd=127),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp44w5d5wo/fixtures.db', fd=128),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp44w5d5wo/fixtures.db', fd=129),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/fixtures.db', fd=130)
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpvu7h14uy/fixtures.db', fd=131),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpvu7h14uy/fixtures.db', fd=132),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpvo3cobk9/fixtures.db', fd=133),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp2t9txyir/fixtures.db', fd=134),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfz8go8rk/fixtures.db', fd=135),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfz8go8rk/fixtures.db', fd=136),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp7h3skv8b/fixtures.db', fd=137),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/fixtures.db', fd=138),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/fixtures.db', fd=139),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/fixtures.db', fd=140),
popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp5j3k1ep_/extra database.db', fd=141),
```
So yeah, that's too many open files!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638241779,"""Too many open files"" error running tests",
https://github.com/simonw/datasette/issues/841#issuecomment-643681747,https://api.github.com/repos/simonw/datasette/issues/841,643681747,MDEyOklzc3VlQ29tbWVudDY0MzY4MTc0Nw==,9599,simonw,2020-06-13T21:38:46Z,2020-06-13T21:38:46Z,OWNER,Closing this because I've researched feasibility. I may start a milestone in the future to help me get to 100%.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/pull/844#issuecomment-643681517,https://api.github.com/repos/simonw/datasette/issues/844,643681517,MDEyOklzc3VlQ29tbWVudDY0MzY4MTUxNw==,9599,simonw,2020-06-13T21:36:15Z,2020-06-13T21:36:15Z,OWNER,"OK, this works now: https://codecov.io/gh/simonw/datasette/tree/1210d9f41841bdca450f85a2342cdb0ff339c1b4","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638230433,Action to run tests and upload coverage report,
https://github.com/simonw/datasette/issues/843#issuecomment-643676314,https://api.github.com/repos/simonw/datasette/issues/843,643676314,MDEyOklzc3VlQ29tbWVudDY0MzY3NjMxNA==,9599,simonw,2020-06-13T20:47:37Z,2020-06-13T20:47:37Z,OWNER,I can use this action: https://github.com/codecov/codecov-action,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638229448,Configure codecov.io,
https://github.com/simonw/datasette/issues/843#issuecomment-643676069,https://api.github.com/repos/simonw/datasette/issues/843,643676069,MDEyOklzc3VlQ29tbWVudDY0MzY3NjA2OQ==,9599,simonw,2020-06-13T20:45:29Z,2020-06-13T20:45:29Z,OWNER,I set up https://codecov.io/gh/simonw/datasette/settings and added a `CODECOV_TOKEN` to the GitHub Actions secrets for this repo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638229448,Configure codecov.io,
https://github.com/simonw/datasette/issues/842#issuecomment-643663005,https://api.github.com/repos/simonw/datasette/issues/842,643663005,MDEyOklzc3VlQ29tbWVudDY0MzY2MzAwNQ==,9599,simonw,2020-06-13T18:51:57Z,2020-06-13T18:51:57Z,OWNER,"Two potential designs:
- `_actor_id`, `_request_ip`, `_now_timestamp` - so special reserved parameters
- a SQL function: `update blah set up = special('ip')`
I fee the first would be easier to implement.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/841#issuecomment-643661125,https://api.github.com/repos/simonw/datasette/issues/841,643661125,MDEyOklzc3VlQ29tbWVudDY0MzY2MTEyNQ==,9599,simonw,2020-06-13T18:35:30Z,2020-06-13T18:36:50Z,OWNER,"I ran export CODECOV_TOKEN=""f7935cad..."", then ran this:
```
datasette $ bash <(curl -s https://codecov.io/bash)
_____ _
/ ____| | |
| | ___ __| | ___ ___ _____ __
| | / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| | __/ (_| (_) \ V /
\_____\___/ \__,_|\___|\___\___/ \_/
Bash-20200602-f809a24
x> No CI provider detected.
Testing inside Docker? http://docs.codecov.io/docs/testing-with-docker
Testing with Tox? https://docs.codecov.io/docs/python#section-testing-with-tox
project root: .
--> token set from env
Yaml not found, that's ok! Learn more at http://docs.codecov.io/docs/codecov-yaml
==> Running gcov in . (disable via -X gcov)
==> Searching for coverage reports in:
+ .
-> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
+ ./coverage.xml bytes=139174
==> Appending adjustments
https://docs.codecov.io/docs/fixing-reports
-> No adjustments found
==> Gzipping contents
==> Uploading reports
url: https://codecov.io
query: branch=master&commit=0e49842e227a0f1f69d48108c87d17fe0379e548&build=&build_url=&name=&tag=&slug=simonw%2Fdatasette&service=&flags=&pr=&job=
-> Pinging Codecov
https://codecov.io/upload/v4?package=bash-20200602-f809a24&token=secret&branch=master&commit=0e49842e227a0f1f69d48108c87d17fe0379e548&build=&build_url=&name=&tag=&slug=simonw%2Fdatasette&service=&flags=&pr=&job=
-> Uploading
-> View reports at https://codecov.io/github/simonw/datasette/commit/0e49842e227a0f1f69d48108c87d17fe0379e548
```
But https://codecov.io/github/simonw/datasette/commit/0e49842e227a0f1f69d48108c87d17fe0379e548 is a 404, so it doesn't seem to have worked?
UPDATE: It works now, took about 30 seconds before the report showed up at that URL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/841#issuecomment-643660757,https://api.github.com/repos/simonw/datasette/issues/841,643660757,MDEyOklzc3VlQ29tbWVudDY0MzY2MDc1Nw==,9599,simonw,2020-06-13T18:32:20Z,2020-06-13T18:32:20Z,OWNER,"Looking at options for publishing coverage reports:
* https://github.com/codecov/codecov-action
* https://github.com/coveralls-clients/coveralls-python
I'm going to try https://codecov.io/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/841#issuecomment-643660427,https://api.github.com/repos/simonw/datasette/issues/841,643660427,MDEyOklzc3VlQ29tbWVudDY0MzY2MDQyNw==,9599,simonw,2020-06-13T18:29:30Z,2020-06-13T18:29:36Z,OWNER,"This one looks easy enough to fix:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/841#issuecomment-643658036,https://api.github.com/repos/simonw/datasette/issues/841,643658036,MDEyOklzc3VlQ29tbWVudDY0MzY1ODAzNg==,9599,simonw,2020-06-13T18:08:13Z,2020-06-13T18:08:13Z,OWNER,"From digging through that report it looks like the majority stuff that isn't fully covered is corner-cases... which are the kind of things I really do want the tests to catch.
I'm not entirely ready to commit to 100%, but I'm going to start digging through and seeing how close I can get. If I can get to 98% (I'm on 91% already) I may as well push all the way to 100.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/841#issuecomment-643657287,https://api.github.com/repos/simonw/datasette/issues/841,643657287,MDEyOklzc3VlQ29tbWVudDY0MzY1NzI4Nw==,9599,simonw,2020-06-13T18:01:39Z,2020-06-13T18:01:39Z,OWNER,Added `--cov-report html` and got this report: https://static.simonwillison.net/static/2020/htmlcov-issue-841/index.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/841#issuecomment-643656053,https://api.github.com/repos/simonw/datasette/issues/841,643656053,MDEyOklzc3VlQ29tbWVudDY0MzY1NjA1Mw==,9599,simonw,2020-06-13T17:50:34Z,2020-06-13T17:50:34Z,OWNER,"Added a `.coveragerc` file:
```
[run]
omit = datasette/_version.py, datasette/utils/shutil_backport.py
```
And ran again: `pytest --cov=datasette --cov-config=.coveragerc`
```
Name Stmts Miss Cover
------------------------------------------------------
datasette/__init__.py 3 0 100%
datasette/__main__.py 3 3 0%
datasette/actor_auth_cookie.py 19 3 84%
datasette/app.py 499 27 95%
datasette/cli.py 157 45 71%
datasette/database.py 233 17 93%
datasette/default_permissions.py 39 0 100%
datasette/facets.py 209 24 89%
datasette/filters.py 122 7 94%
datasette/hookspecs.py 19 0 100%
datasette/inspect.py 37 23 38%
datasette/plugins.py 34 6 82%
datasette/publish/__init__.py 0 0 100%
datasette/publish/cloudrun.py 55 2 96%
datasette/publish/common.py 19 1 95%
datasette/publish/heroku.py 95 13 86%
datasette/renderer.py 63 4 94%
datasette/sql_functions.py 4 0 100%
datasette/tracer.py 85 16 81%
datasette/utils/__init__.py 503 31 94%
datasette/utils/asgi.py 253 25 90%
datasette/version.py 4 0 100%
datasette/views/__init__.py 0 0 100%
datasette/views/base.py 288 19 93%
datasette/views/database.py 120 2 98%
datasette/views/index.py 57 2 96%
datasette/views/special.py 72 16 78%
datasette/views/table.py 418 18 96%
------------------------------------------------------
TOTAL 3410 304 91%
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/841#issuecomment-643655108,https://api.github.com/repos/simonw/datasette/issues/841,643655108,MDEyOklzc3VlQ29tbWVudDY0MzY1NTEwOA==,9599,simonw,2020-06-13T17:43:15Z,2020-06-13T17:43:15Z,OWNER,"Using https://pypi.org/project/pytest-cov/ and running `pytest --cov=datasette`:
```
---------- coverage: platform darwin, python 3.7.7-final-0 -----------
Name Stmts Miss Cover
--------------------------------------------------------
datasette/__init__.py 3 0 100%
datasette/__main__.py 3 3 0%
datasette/_version.py 277 152 45%
datasette/actor_auth_cookie.py 19 3 84%
datasette/app.py 499 27 95%
datasette/cli.py 157 45 71%
datasette/database.py 233 17 93%
datasette/default_permissions.py 39 0 100%
datasette/facets.py 209 24 89%
datasette/filters.py 122 7 94%
datasette/hookspecs.py 19 0 100%
datasette/inspect.py 37 23 38%
datasette/plugins.py 34 6 82%
datasette/publish/__init__.py 0 0 100%
datasette/publish/cloudrun.py 55 2 96%
datasette/publish/common.py 19 1 95%
datasette/publish/heroku.py 95 13 86%
datasette/renderer.py 63 4 94%
datasette/sql_functions.py 4 0 100%
datasette/tracer.py 85 16 81%
datasette/utils/__init__.py 503 31 94%
datasette/utils/asgi.py 253 25 90%
datasette/utils/shutil_backport.py 44 40 9%
datasette/version.py 4 0 100%
datasette/views/__init__.py 0 0 100%
datasette/views/base.py 288 19 93%
datasette/views/database.py 120 2 98%
datasette/views/index.py 57 2 96%
datasette/views/special.py 72 16 78%
datasette/views/table.py 418 18 96%
--------------------------------------------------------
TOTAL 3731 496 87%
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/834#issuecomment-643648359,https://api.github.com/repos/simonw/datasette/issues/834,643648359,MDEyOklzc3VlQ29tbWVudDY0MzY0ODM1OQ==,9599,simonw,2020-06-13T16:47:29Z,2020-06-13T16:47:29Z,OWNER,"Implementing this is proving surprisingly tricky, because of the need to be able to optionally `await` the returned value. It's a bit of a fiddle to get this to work within unit tests because they run in non-async functions - due to this cunning `async_to_sync` usage in the test client:
https://github.com/simonw/datasette/blob/b906030235efbdff536405d66078f4868ce0d3bd/tests/fixtures.py#L115-L133
I could switch to using `async def test_*` functions decorated with `@pytest.mark.asyncio` but I'd rather not re-engineer the entire test suite just for this one feature, so I'll try to find another way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/841#issuecomment-643576372,https://api.github.com/repos/simonw/datasette/issues/841,643576372,MDEyOklzc3VlQ29tbWVudDY0MzU3NjM3Mg==,9599,simonw,2020-06-13T06:08:34Z,2020-06-13T06:08:34Z,OWNER,Starlette achieves this. https://github.com/encode/starlette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638104520,Research feasibility of 100% test coverage,
https://github.com/simonw/datasette/issues/834#issuecomment-643510240,https://api.github.com/repos/simonw/datasette/issues/834,643510240,MDEyOklzc3VlQ29tbWVudDY0MzUxMDI0MA==,9599,simonw,2020-06-12T22:40:26Z,2020-06-12T22:40:26Z,OWNER,Another use-case: plugins that need their own database with the correct tables. They can write to the database on startup to create their tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/834#issuecomment-643509358,https://api.github.com/repos/simonw/datasette/issues/834,643509358,MDEyOklzc3VlQ29tbWVudDY0MzUwOTM1OA==,9599,simonw,2020-06-12T22:36:37Z,2020-06-12T22:36:37Z,OWNER,This should be able to optionally return an async function which is then awaited.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/805#issuecomment-643501428,https://api.github.com/repos/simonw/datasette/issues/805,643501428,MDEyOklzc3VlQ29tbWVudDY0MzUwMTQyOA==,9599,simonw,2020-06-12T22:06:08Z,2020-06-12T22:06:08Z,OWNER,"This needs the `startup` hook, see https://github.com/simonw/datasette/issues/834#issuecomment-643501064","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/834#issuecomment-643501064,https://api.github.com/repos/simonw/datasette/issues/834,643501064,MDEyOklzc3VlQ29tbWVudDY0MzUwMTA2NA==,9599,simonw,2020-06-12T22:04:43Z,2020-06-12T22:04:43Z,OWNER,Another use-case for this: I want to use the `--root` option on Glitch but it gives me a 127.0.0.1 URL. Glitch has a `PROJECT_DOMAIN` environment variable which tells me the URL. A `datasette-glitch` plugin could use a `startup` hook to output the correct login URL.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/840#issuecomment-643454625,https://api.github.com/repos/simonw/datasette/issues/840,643454625,MDEyOklzc3VlQ29tbWVudDY0MzQ1NDYyNQ==,9599,simonw,2020-06-12T19:47:38Z,2020-06-12T19:47:53Z,OWNER,"Another problem: what to display in the ""you are logged in as"", since we don't dictate an actor design.
I'm going to use a includes template for this that can easily be over-ridden by administrators or by plugins.
The default will look for the first available of the following keys:
- display
- name
- username
- login
- id","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637966833,Log out mechanism for clearing ds_actor cookie,
https://github.com/simonw/datasette/issues/840#issuecomment-643453128,https://api.github.com/repos/simonw/datasette/issues/840,643453128,MDEyOklzc3VlQ29tbWVudDY0MzQ1MzEyOA==,9599,simonw,2020-06-12T19:43:15Z,2020-06-12T19:43:15Z,OWNER,"I don't like how this often involves a logout link that can be maliciously activated.
I'm going to use a CSRF protected form button styled to look like a link instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637966833,Log out mechanism for clearing ds_actor cookie,
https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643414646,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40,643414646,MDEyOklzc3VlQ29tbWVudDY0MzQxNDY0Ng==,9599,simonw,2020-06-12T18:06:48Z,2020-06-12T18:06:48Z,MEMBER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637899539,Demo deploy is broken,
https://github.com/simonw/sqlite-utils/issues/115#issuecomment-643406939,https://api.github.com/repos/simonw/sqlite-utils/issues/115,643406939,MDEyOklzc3VlQ29tbWVudDY0MzQwNjkzOQ==,9599,simonw,2020-06-12T17:51:11Z,2020-06-12T17:51:11Z,OWNER,https://github.com/simonw/sqlite-utils/blob/03ee97d2258254581bea72842518904fc1cbe60f/tests/test_cli.py#L1112-L1128,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637889964,Ability to execute insert/update statements with the CLI,
https://github.com/dogsheep/github-to-sqlite/issues/40#issuecomment-643393506,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40,643393506,MDEyOklzc3VlQ29tbWVudDY0MzM5MzUwNg==,9599,simonw,2020-06-12T17:21:14Z,2020-06-12T17:21:14Z,MEMBER,"I only install SQLite for this:
https://github.com/dogsheep/github-to-sqlite/blob/c0d54e0260468be38152293df5abd775c068495d/.github/workflows/deploy-demo.yml#L77-L78
I'm going to remove the need to install sqlite3 by making this possible with sqlite-utils: https://github.com/simonw/sqlite-utils/issues/115","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637899539,Demo deploy is broken,
https://github.com/simonw/datasette/issues/838#issuecomment-643083451,https://api.github.com/repos/simonw/datasette/issues/838,643083451,MDEyOklzc3VlQ29tbWVudDY0MzA4MzQ1MQ==,79913,tsibley,2020-06-12T06:04:14Z,2020-06-12T06:04:14Z,NONE,"Hmm, I haven't tried removing `ProxyPassReverse`, but it doesn't touch the HTML, which is the issue I'm seeing. You can read the [documentation here](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse). `ProxyPassReverse` is a standard directive when proxying with Apache. I've used it dozens of times with other applications.
Looking a little more at the code, I think the issue here is that the behaviour of `base_url` makes sense when Datasette is _mounted_ at a path within a larger application, but not when HTTP requests are being _proxied_ to it.
In a _mount_ situation, it is perfectly fine to construct URLs reusing the domain and path from the request. In a _proxy_ situation, it never is, as the domain and path in the request are not the domain and path that the non-proxy client actually needs to use. That is, links which include the Apache → Datasette request origin, `localhost:8001`, instead of the browser → Apache request origin, `example.com`, will be broken.
The tests you pointed to also reflect this in two ways:
1. They strip a leading `http://localhost`, allowing such URLs in the facet links to pass, but inclusion of that in a proxy situation would mean the URL is broken.
2. The test client emits direct ASGI events instead of actual proxied HTTP requests. The headers of these ASGI events don't reflect the way an HTTP proxy works; instead they pass through the original request path which contains `base_url`. This works because Datasette responds to requests equivalently at either `/…` or `/{base_url}/…`, which makes some sense in a _mount_ situation but is unconventional (albeit workable) for a proxied app.
Apps that support being proxied automatically support being mounted, but apps that only support being mounted don't automatically support being proxied.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/806#issuecomment-643010591,https://api.github.com/repos/simonw/datasette/issues/806,643010591,MDEyOklzc3VlQ29tbWVudDY0MzAxMDU5MQ==,9599,simonw,2020-06-12T01:13:06Z,2020-06-12T01:13:06Z,OWNER,Tests are passing again: https://github.com/simonw/datasette/commit/9ae0d483ead93c0832142e5dc85959ae3c8f73ea,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/806#issuecomment-643000948,https://api.github.com/repos/simonw/datasette/issues/806,643000948,MDEyOklzc3VlQ29tbWVudDY0MzAwMDk0OA==,9599,simonw,2020-06-12T00:34:21Z,2020-06-12T00:34:21Z,OWNER,I'm going to add https://github.com/simonw/datasette-auth-tokens and https://github.com/simonw/datasette-permissions-sql to the documentation and release notes in a few places.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/806#issuecomment-642998097,https://api.github.com/repos/simonw/datasette/issues/806,642998097,MDEyOklzc3VlQ29tbWVudDY0Mjk5ODA5Nw==,9599,simonw,2020-06-12T00:26:00Z,2020-06-12T00:26:00Z,OWNER,"OK, I'm ready to ship. Last check of the release notes, then I'll update the news section in the README and release 0.44!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/838#issuecomment-642993277,https://api.github.com/repos/simonw/datasette/issues/838,642993277,MDEyOklzc3VlQ29tbWVudDY0Mjk5MzI3Nw==,9599,simonw,2020-06-12T00:18:26Z,2020-06-12T00:18:50Z,OWNER,"Have you tried this without the `ProxyPassReverse` directive? I'm worried that might be confusing Datasette.
This is the test I used to ensure this feature works - it scrapes all of the links on a bunch of different pages. Could it be missing something here?
https://github.com/simonw/datasette/blob/647c5ff0f3e8140f40d7f41f0874ce4e1f4df65c/tests/test_html.py#L1233-L1274
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/824#issuecomment-642991513,https://api.github.com/repos/simonw/datasette/issues/824,642991513,MDEyOklzc3VlQ29tbWVudDY0Mjk5MTUxMw==,9599,simonw,2020-06-12T00:11:50Z,2020-06-12T00:11:50Z,OWNER,Done: https://github.com/simonw/datasette-auth-tokens and https://pypi.org/project/datasette-auth-tokens/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635108074,Example authentication plugin,
https://github.com/simonw/datasette/issues/833#issuecomment-642958225,https://api.github.com/repos/simonw/datasette/issues/833,642958225,MDEyOklzc3VlQ29tbWVudDY0Mjk1ODIyNQ==,9599,simonw,2020-06-11T22:15:32Z,2020-06-11T22:15:32Z,OWNER,https://github.com/simonw/datasette/blob/29c5ff493ad7918b8fc44ea7920b41530e56dd5d/tests/test_permissions.py#L327-L348,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637253789,/-/metadata and so on should respect view-instance permission,
https://github.com/simonw/datasette/issues/824#issuecomment-642953605,https://api.github.com/repos/simonw/datasette/issues/824,642953605,MDEyOklzc3VlQ29tbWVudDY0Mjk1MzYwNQ==,9599,simonw,2020-06-11T22:02:32Z,2020-06-11T22:02:32Z,OWNER,`datasette-auth-tokens` can be the name. I can get a simple initial version of it running pretty quickly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635108074,Example authentication plugin,
https://github.com/simonw/datasette/issues/824#issuecomment-642952962,https://api.github.com/repos/simonw/datasette/issues/824,642952962,MDEyOklzc3VlQ29tbWVudDY0Mjk1Mjk2Mg==,9599,simonw,2020-06-11T22:01:58Z,2020-06-11T22:01:58Z,OWNER,"Alternative idea: a plugin that handles Bearer token authentication. Uses `metadata.json` with secret plugin values to map an incoming token to an actor dictionary, which can then be mapped to permissions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635108074,Example authentication plugin,
https://github.com/simonw/datasette/issues/824#issuecomment-642951150,https://api.github.com/repos/simonw/datasette/issues/824,642951150,MDEyOklzc3VlQ29tbWVudDY0Mjk1MTE1MA==,9599,simonw,2020-06-11T22:00:17Z,2020-06-11T22:00:17Z,OWNER,"I got this working: https://github.com/simonw/datasette-auth-github/pull/64
Just one problem: it uses the existing `ds_actor` cookie, which means it doesn't actually exercise the `actor_from_request` plugin!
It does use `register_routes` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635108074,Example authentication plugin,
https://github.com/simonw/datasette/issues/220#issuecomment-642944645,https://api.github.com/repos/simonw/datasette/issues/220,642944645,MDEyOklzc3VlQ29tbWVudDY0Mjk0NDY0NQ==,9599,simonw,2020-06-11T21:49:55Z,2020-06-11T21:49:55Z,OWNER,"I'm OK with not implementing this - I've got used to the existing mechanism, and it doesn't frustrate me enough to work on this more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314847571,Investigate syntactic sugar for plugins,
https://github.com/simonw/datasette/issues/832#issuecomment-642907021,https://api.github.com/repos/simonw/datasette/issues/832,642907021,MDEyOklzc3VlQ29tbWVudDY0MjkwNzAyMQ==,9599,simonw,2020-06-11T20:20:35Z,2020-06-11T20:20:35Z,OWNER,"I think the new `.check_permissions()` should be a documented utility that is available to plugins.
Maybe a method on `datasette`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-642906681,https://api.github.com/repos/simonw/datasette/issues/832,642906681,MDEyOklzc3VlQ29tbWVudDY0MjkwNjY4MQ==,9599,simonw,2020-06-11T20:19:47Z,2020-06-11T20:20:02Z,OWNER,"So for the following:
```
await self.check_permissions(request, [
(""view-table"", (database, table)),
(""view-database"", database),
""view-instance"",
])
```
The logic is: if the first test returns `True`, you get access. If it returns `False` you are denied. If it says `None` then move on to the next check in the list and repeat.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/833#issuecomment-642905424,https://api.github.com/repos/simonw/datasette/issues/833,642905424,MDEyOklzc3VlQ29tbWVudDY0MjkwNTQyNA==,9599,simonw,2020-06-11T20:16:41Z,2020-06-11T20:16:41Z,OWNER,I'll add a new test in `test_permissions.py` which locks down an instance and then loops through paths as the anonymous user making sure they aren't accessible.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637253789,/-/metadata and so on should respect view-instance permission,
https://github.com/simonw/datasette/issues/833#issuecomment-642902208,https://api.github.com/repos/simonw/datasette/issues/833,642902208,MDEyOklzc3VlQ29tbWVudDY0MjkwMjIwOA==,9599,simonw,2020-06-11T20:08:57Z,2020-06-11T20:08:57Z,OWNER,"I'm tempted to add a `view-instance` check before routing any URLs, but that wouldn't be compatible with the idea in #832 that having `view-table` should be enough to view a table even if you don't pass `view-instance`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637253789,/-/metadata and so on should respect view-instance permission,
https://github.com/simonw/datasette/issues/833#issuecomment-642874724,https://api.github.com/repos/simonw/datasette/issues/833,642874724,MDEyOklzc3VlQ29tbWVudDY0Mjg3NDcyNA==,9599,simonw,2020-06-11T19:07:49Z,2020-06-11T19:07:49Z,OWNER,A live demo running the `datasette-auth-github` plugin will help demonstrate this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637253789,/-/metadata and so on should respect view-instance permission,
https://github.com/simonw/datasette/issues/801#issuecomment-642870553,https://api.github.com/repos/simonw/datasette/issues/801,642870553,MDEyOklzc3VlQ29tbWVudDY0Mjg3MDU1Mw==,9599,simonw,2020-06-11T18:58:49Z,2020-06-11T18:58:49Z,OWNER,I've implemented this in a plugin instead: https://github.com/simonw/datasette-permissions-sql,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631932926,allow_by_query setting for configuring permissions with a SQL statement,
https://github.com/simonw/datasette/issues/832#issuecomment-642795966,https://api.github.com/repos/simonw/datasette/issues/832,642795966,MDEyOklzc3VlQ29tbWVudDY0Mjc5NTk2Ng==,9599,simonw,2020-06-11T16:37:21Z,2020-06-11T16:37:21Z,OWNER,"How would I document this? Probably in another section on https://datasette.readthedocs.io/en/latest/authentication.html#permissions
But I'd also need to add documentation to the individual views stating what permissions are checked and in what order. I could do that on this page: https://datasette.readthedocs.io/en/latest/pages.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/pull/809#issuecomment-642772344,https://api.github.com/repos/simonw/datasette/issues/809,642772344,MDEyOklzc3VlQ29tbWVudDY0Mjc3MjM0NA==,9599,simonw,2020-06-11T16:01:15Z,2020-06-11T16:01:15Z,OWNER,"```
datasette package fixtures.db --secret woot --branch master
Sending build context to Docker daemon 260.6kB
Step 1/9 : FROM python:3.8
3.8: Pulling from library/python
e9afc4f90ab0: Downloading [=======> ] 7.195MB/50.39MB
989e6b19a265: Downloading [============================> ] 4.475MB/7.812MB
af14b6c2f878: Downloading [===========================> ] 5.422MB/9.996MB
5573c4b30949: Waiting
11a88e764313: Waiting
ee776f0e36af: Waiting
513c90a1afc3: Waiting
df9b9e95bdb9: Waiting
86c9edb54464: Waiting
...
datasette package fixtures.db --secret woot --branch master
docker run -p 8001:8001 a155798bd842
```
This works too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632919570,Publish secrets,
https://github.com/simonw/datasette/pull/809#issuecomment-642754589,https://api.github.com/repos/simonw/datasette/issues/809,642754589,MDEyOklzc3VlQ29tbWVudDY0Mjc1NDU4OQ==,9599,simonw,2020-06-11T15:45:25Z,2020-06-11T15:45:25Z,OWNER," datasette publish cloudrun fixtures.db --service datasette-publish-secret --branch=master
https://datasette-publish-secret-j7hipcg4aq-uw.a.run.app/-/messages","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632919570,Publish secrets,
https://github.com/simonw/datasette/pull/809#issuecomment-642750790,https://api.github.com/repos/simonw/datasette/issues/809,642750790,MDEyOklzc3VlQ29tbWVudDY0Mjc1MDc5MA==,9599,simonw,2020-06-11T15:42:23Z,2020-06-11T15:42:23Z,OWNER," datasette publish heroku fixtures.db -n datasette-publish-secret --branch=master
https://datasette-publish-secret.herokuapp.com/-/messages - Heroku works.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632919570,Publish secrets,
https://github.com/simonw/datasette/pull/809#issuecomment-642745518,https://api.github.com/repos/simonw/datasette/issues/809,642745518,MDEyOklzc3VlQ29tbWVudDY0Mjc0NTUxOA==,9599,simonw,2020-06-11T15:38:51Z,2020-06-11T15:38:51Z,OWNER,The way to manually test this is to publish a database to each provider and then check that the `/-/messages` debug tool works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632919570,Publish secrets,
https://github.com/simonw/datasette/issues/832#issuecomment-642741930,https://api.github.com/repos/simonw/datasette/issues/832,642741930,MDEyOklzc3VlQ29tbWVudDY0Mjc0MTkzMA==,9599,simonw,2020-06-11T15:35:53Z,2020-06-11T15:36:05Z,OWNER,"May the fix here is to implement a `.check_permissions()` method which passes when the first permission passes?
```python
await self.check_permissions(request, [
(""view-table"", (database, table)),
(""view-database"", database),
""view-instance"",
])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/394#issuecomment-642522285,https://api.github.com/repos/simonw/datasette/issues/394,642522285,MDEyOklzc3VlQ29tbWVudDY0MjUyMjI4NQ==,58298410,LVerneyPEReN,2020-06-11T09:15:19Z,2020-06-11T09:15:19Z,NONE,"Hi @wragge,
This looks great, thanks for the share! I refactored it into a self-contained function, binding on a random available TCP port (multi-user context). I am using subprocess API directly since the `%run` magic was leaving defunct process behind :/
```python
import socket
from signal import SIGINT
from subprocess import Popen, PIPE
from IPython.display import display, HTML
from notebook.notebookapp import list_running_servers
def get_free_tcp_port():
""""""
Get a free TCP port.
""""""
tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcp.bind(('', 0))
_, port = tcp.getsockname()
tcp.close()
return port
def datasette(database):
""""""
Run datasette on an SQLite database.
""""""
# Get current running servers
servers = list_running_servers()
# Get the current base url
base_url = next(servers)['base_url']
# Get a free port
port = get_free_tcp_port()
# Create a base url for Datasette suing the proxy path
proxy_url = f'{base_url}proxy/absolute/{port}/'
# Display a link to Datasette
display(HTML(f'![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635914822,Horizontal scrollbar on changelog page on mobile,
https://github.com/simonw/datasette/issues/828#issuecomment-641710745,https://api.github.com/repos/simonw/datasette/issues/828,641710745,MDEyOklzc3VlQ29tbWVudDY0MTcxMDc0NQ==,9599,simonw,2020-06-10T04:19:31Z,2020-06-10T04:19:31Z,OWNER,https://docs.readthedocs.io/en/stable/guides/adding-custom-css.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635914822,Horizontal scrollbar on changelog page on mobile,
https://github.com/simonw/datasette/issues/828#issuecomment-641710670,https://api.github.com/repos/simonw/datasette/issues/828,641710670,MDEyOklzc3VlQ29tbWVudDY0MTcxMDY3MA==,9599,simonw,2020-06-10T04:19:17Z,2020-06-10T04:19:17Z,OWNER,"This CSS seems to fix it:
```css
a.external {overflow-wrap: anywhere;}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635914822,Horizontal scrollbar on changelog page on mobile,
https://github.com/simonw/datasette/issues/806#issuecomment-641637696,https://api.github.com/repos/simonw/datasette/issues/806,641637696,MDEyOklzc3VlQ29tbWVudDY0MTYzNzY5Ng==,9599,simonw,2020-06-09T23:46:00Z,2020-06-09T23:46:00Z,OWNER,"The issues that should be referenced from this release are:
#395, #519, #576, #699, #706, #774, #777, #781, #784, #788, #790, #797, #798, #800, #802, #804, #819, #822 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/806#issuecomment-641634749,https://api.github.com/repos/simonw/datasette/issues/806,641634749,MDEyOklzc3VlQ29tbWVudDY0MTYzNDc0OQ==,9599,simonw,2020-06-09T23:34:52Z,2020-06-09T23:34:52Z,OWNER,Preview of the release notes is now available here: https://datasette.readthedocs.io/en/latest/changelog.html#v0-44,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/795#issuecomment-641616185,https://api.github.com/repos/simonw/datasette/issues/795,641616185,MDEyOklzc3VlQ29tbWVudDY0MTYxNjE4NQ==,9599,simonw,2020-06-09T22:33:33Z,2020-06-09T22:33:33Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/internals.html#setting-cookies-with-response-set-cookie,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629541395,response.set_cookie() method,
https://github.com/simonw/datasette/issues/826#issuecomment-641616060,https://api.github.com/repos/simonw/datasette/issues/826,641616060,MDEyOklzc3VlQ29tbWVudDY0MTYxNjA2MA==,9599,simonw,2020-06-09T22:33:12Z,2020-06-09T22:33:12Z,OWNER,https://datasette.readthedocs.io/en/latest/authentication.html#the-ds-actor-cookie,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635519358,Document the ds_actor signed cookie,
https://github.com/simonw/datasette/issues/806#issuecomment-641604210,https://api.github.com/repos/simonw/datasette/issues/806,641604210,MDEyOklzc3VlQ29tbWVudDY0MTYwNDIxMA==,9599,simonw,2020-06-09T21:59:33Z,2020-06-09T22:00:11Z,OWNER,"AWS IAM uses action and resource terminology: https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction_access-management.html - I think that's where I got that language:
> ```json
> {
> ""Version"": ""2012-10-17"",
> ""Statement"": {
> ""Effect"": ""Allow"",
> ""Action"": ""dynamodb:*"",
> ""Resource"": ""arn:aws:dynamodb:us-east-2:123456789012:table/Books""
> }
> }
> ```
I'm going to stick with ""action"" in its current meaning.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/806#issuecomment-641603457,https://api.github.com/repos/simonw/datasette/issues/806,641603457,MDEyOklzc3VlQ29tbWVudDY0MTYwMzQ1Nw==,9599,simonw,2020-06-09T21:57:32Z,2020-06-09T21:57:32Z,OWNER,"operation, procedure, process as alternative words for those menu items?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/806#issuecomment-641602794,https://api.github.com/repos/simonw/datasette/issues/806,641602794,MDEyOklzc3VlQ29tbWVudDY0MTYwMjc5NA==,9599,simonw,2020-06-09T21:55:45Z,2020-06-09T21:55:45Z,OWNER,"Last-minute thought:
Should I worry about calling permissions ""actions"", when I have an idea for a future plugin hook that allows plugins to add something I was going to call ""actions"" to database, table and row pages?
Those actions would take the form of menu item commands that Do Something to the selected object. If I use ""actions"" to mean permission names, will I be able to find a good alternative name for these dynamic menu items?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/804#issuecomment-641538982,https://api.github.com/repos/simonw/datasette/issues/804,641538982,MDEyOklzc3VlQ29tbWVudDY0MTUzODk4Mg==,9599,simonw,2020-06-09T20:01:30Z,2020-06-09T20:01:30Z,OWNER,Now fully documented here: https://datasette.readthedocs.io/en/latest/contributing.html#setting-up-a-development-environment,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632673972,python tests/fixtures.py command has a bug,
https://github.com/simonw/datasette/issues/804#issuecomment-641538799,https://api.github.com/repos/simonw/datasette/issues/804,641538799,MDEyOklzc3VlQ29tbWVudDY0MTUzODc5OQ==,9599,simonw,2020-06-09T20:01:08Z,2020-06-09T20:01:08Z,OWNER," $ python tests/fixtures.py fixtures.db fixtures-metadata.json fixtures-plugins
Test tables written to fixtures.db
- metadata written to fixtures-metadata.json
Wrote plugin: fixtures-plugins/register_output_renderer.py
Wrote plugin: fixtures-plugins/view_name.py
Wrote plugin: fixtures-plugins/my_plugin.py
Wrote plugin: fixtures-plugins/messages_output_renderer.py
Wrote plugin: fixtures-plugins/my_plugin_2.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632673972,python tests/fixtures.py command has a bug,
https://github.com/simonw/datasette/issues/804#issuecomment-641528737,https://api.github.com/repos/simonw/datasette/issues/804,641528737,MDEyOklzc3VlQ29tbWVudDY0MTUyODczNw==,9599,simonw,2020-06-09T19:39:24Z,2020-06-09T19:39:24Z,OWNER,Switched to 0.44 milestone because I don't like shipping releases with known bugs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632673972,python tests/fixtures.py command has a bug,
https://github.com/simonw/datasette/issues/827#issuecomment-641528269,https://api.github.com/repos/simonw/datasette/issues/827,641528269,MDEyOklzc3VlQ29tbWVudDY0MTUyODI2OQ==,9599,simonw,2020-06-09T19:38:30Z,2020-06-09T19:38:30Z,OWNER,https://datasette.readthedocs.io/en/latest/internals.html#csrf-protection,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635696400,Document CSRF protection (for plugins),
https://github.com/simonw/datasette/issues/825#issuecomment-641406944,https://api.github.com/repos/simonw/datasette/issues/825,641406944,MDEyOklzc3VlQ29tbWVudDY0MTQwNjk0NA==,9599,simonw,2020-06-09T16:12:02Z,2020-06-09T17:19:19Z,OWNER,"Alternative design: leave actor alone. Instead specify that allow blocks can look like this:
```json
{
""allow"": {
""unauthenticated"": true
}
}
```
I like this: the above block is very self-documenting.
The `""id"": ""*""` mechanism means there is already precedent for allow keys with special meaning.
**I'm going with this design.**","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635147716,Way to enable a default=False permission for anonymous users,
https://github.com/simonw/datasette/issues/825#issuecomment-641452563,https://api.github.com/repos/simonw/datasette/issues/825,641452563,MDEyOklzc3VlQ29tbWVudDY0MTQ1MjU2Mw==,9599,simonw,2020-06-09T17:08:00Z,2020-06-09T17:08:00Z,OWNER,https://datasette.readthedocs.io/en/latest/authentication.html#defining-permissions-with-allow-blocks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635147716,Way to enable a default=False permission for anonymous users,
https://github.com/simonw/datasette/issues/825#issuecomment-641449725,https://api.github.com/repos/simonw/datasette/issues/825,641449725,MDEyOklzc3VlQ29tbWVudDY0MTQ0OTcyNQ==,9599,simonw,2020-06-09T17:02:31Z,2020-06-09T17:02:31Z,OWNER,Documented at the bottom of this section: https://github.com/simonw/datasette/blob/7633b9ab249b2dce5ee0b4fcf9542c13a1703ef0/docs/authentication.rst#defining-permissions-with-allow-blocks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635147716,Way to enable a default=False permission for anonymous users,
https://github.com/simonw/datasette/issues/825#issuecomment-641412424,https://api.github.com/repos/simonw/datasette/issues/825,641412424,MDEyOklzc3VlQ29tbWVudDY0MTQxMjQyNA==,9599,simonw,2020-06-09T16:22:07Z,2020-06-09T16:22:07Z,OWNER,"When I implement this I should also document default allow vs default deny as a concept, and specify that default next to every documented permission.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635147716,Way to enable a default=False permission for anonymous users,
https://github.com/simonw/datasette/issues/795#issuecomment-641361311,https://api.github.com/repos/simonw/datasette/issues/795,641361311,MDEyOklzc3VlQ29tbWVudDY0MTM2MTMxMQ==,9599,simonw,2020-06-09T15:11:50Z,2020-06-09T15:11:50Z,OWNER,Also: https://github.com/simonw/datasette/blob/dfff34e1987976e72f58ee7b274952840b1f4b71/datasette/views/special.py#L63-L76,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629541395,response.set_cookie() method,
https://github.com/simonw/datasette/issues/826#issuecomment-641360187,https://api.github.com/repos/simonw/datasette/issues/826,641360187,MDEyOklzc3VlQ29tbWVudDY0MTM2MDE4Nw==,9599,simonw,2020-06-09T15:10:00Z,2020-06-09T15:11:24Z,OWNER,Also a good reminder that I need a `set_cookie()` function (#795) so I don't have to mess around with `SimpleCookie` directly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635519358,Document the ds_actor signed cookie,
https://github.com/simonw/datasette/issues/826#issuecomment-641359103,https://api.github.com/repos/simonw/datasette/issues/826,641359103,MDEyOklzc3VlQ29tbWVudDY0MTM1OTEwMw==,9599,simonw,2020-06-09T15:08:07Z,2020-06-09T15:10:33Z,OWNER,"I should probably add a utility function for setting that cookie - right now the only code that does that is here:
https://github.com/simonw/datasette/blob/dfff34e1987976e72f58ee7b274952840b1f4b71/datasette/views/special.py#L63-L76","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635519358,Document the ds_actor signed cookie,
https://github.com/simonw/datasette/issues/812#issuecomment-641353729,https://api.github.com/repos/simonw/datasette/issues/812,641353729,MDEyOklzc3VlQ29tbWVudDY0MTM1MzcyOQ==,9599,simonw,2020-06-09T14:59:25Z,2020-06-09T14:59:25Z,OWNER,I'm going to figure this out by working with https://github.com/simonw/datasette-auth-github/issues/62,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634112607,Ability to customize what happens when a view permission fails,
https://github.com/simonw/datasette/issues/823#issuecomment-641353186,https://api.github.com/repos/simonw/datasette/issues/823,641353186,MDEyOklzc3VlQ29tbWVudDY0MTM1MzE4Ng==,9599,simonw,2020-06-09T14:58:36Z,2020-06-09T14:58:36Z,OWNER,"Docs now say:
> The actor dictionary can be any shape - the design of that data structure is left up to the plugins. A useful convention is to include an `""id""` string, as demonstrated by the ""root"" actor below.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635107393,"Documentation is inconsistent about ""id"" as required field on actor",
https://github.com/simonw/datasette/issues/825#issuecomment-641320947,https://api.github.com/repos/simonw/datasette/issues/825,641320947,MDEyOklzc3VlQ29tbWVudDY0MTMyMDk0Nw==,9599,simonw,2020-06-09T14:06:46Z,2020-06-09T14:06:46Z,OWNER,"I'm torn between `anonymous` and `anon` - because the latter is less typing, and I envisage people writing a lot of code like this:
```python
if actor.get(""anonymous""):
# ...
```
I'm going with `anonymous` because it's that tiny bit clearer than `anon`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635147716,Way to enable a default=False permission for anonymous users,
https://github.com/simonw/datasette/issues/825#issuecomment-641062164,https://api.github.com/repos/simonw/datasette/issues/825,641062164,MDEyOklzc3VlQ29tbWVudDY0MTA2MjE2NA==,9599,simonw,2020-06-09T06:30:24Z,2020-06-09T14:05:33Z,OWNER,"Idea: the anonymous actor could be passed to `actor_matches_allow()` as:
```json
{""anonymous"": true}
```
Then allow blocks like this could be used to allow them:
```json
{
""plugins"": {
""datasette-upload-csvs"": {
""allow"": {
""anonymous"": true
}
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635147716,Way to enable a default=False permission for anonymous users,
https://github.com/simonw/datasette/issues/823#issuecomment-641059221,https://api.github.com/repos/simonw/datasette/issues/823,641059221,MDEyOklzc3VlQ29tbWVudDY0MTA1OTIyMQ==,9599,simonw,2020-06-09T06:23:51Z,2020-06-09T06:24:09Z,OWNER,"I don't like the ""id"" requirement.
I can think of plenty of situations where a unique ID might not be available:
- auth against an external token - an email address or a phone number for example
- auth using encrypted tokens - where decrypting the token tells you exactly what permissions that token should have, like in https://blog.thea.codes/building-a-stateless-api-proxy/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635107393,"Documentation is inconsistent about ""id"" as required field on actor",
https://github.com/simonw/datasette/issues/806#issuecomment-641026726,https://api.github.com/repos/simonw/datasette/issues/806,641026726,MDEyOklzc3VlQ29tbWVudDY0MTAyNjcyNg==,9599,simonw,2020-06-09T04:52:07Z,2020-06-09T04:52:07Z,OWNER,Changelog for this is going to be huge - 96 commits since 0.43 already! https://github.com/simonw/datasette/compare/0.43...master,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/818#issuecomment-641026230,https://api.github.com/repos/simonw/datasette/issues/818,641026230,MDEyOklzc3VlQ29tbWVudDY0MTAyNjIzMA==,9599,simonw,2020-06-09T04:50:24Z,2020-06-09T04:50:24Z,OWNER,I'm dropping this from the 0.44 milestone.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/823#issuecomment-641025760,https://api.github.com/repos/simonw/datasette/issues/823,641025760,MDEyOklzc3VlQ29tbWVudDY0MTAyNTc2MA==,9599,simonw,2020-06-09T04:48:40Z,2020-06-09T04:48:40Z,OWNER,"I should assert that `""id""` exists and is a string in the code that calls the `actor_from_request` hook.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635107393,"Documentation is inconsistent about ""id"" as required field on actor",
https://github.com/simonw/datasette/issues/805#issuecomment-641017851,https://api.github.com/repos/simonw/datasette/issues/805,641017851,MDEyOklzc3VlQ29tbWVudDY0MTAxNzg1MQ==,9599,simonw,2020-06-09T04:17:00Z,2020-06-09T04:17:00Z,OWNER,I can't get Datasette working on Glitch installed from a URL - I'm going to try this on Glitch once I've shipped the 0.44 release in #806.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/805#issuecomment-641017721,https://api.github.com/repos/simonw/datasette/issues/805,641017721,MDEyOklzc3VlQ29tbWVudDY0MTAxNzcyMQ==,9599,simonw,2020-06-09T04:16:28Z,2020-06-09T04:16:28Z,OWNER,"Create `data.db` with:
```
echo '{""emoji"": ""🐯"", ""score"": 0}' | sqlite-utils insert data.db emojis --pk=emoji -
echo '{""emoji"": ""🐺"", ""score"": 0}' | sqlite-utils insert data.db emojis --pk=emoji -
```
Then run Datasette with this `metadata.yaml`:
```yaml
title: Datasette Poll
databases:
data:
queries:
vote:
sql: |-
update emojis set score = score + 1 where emoji = :emoji
write: true
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/797#issuecomment-638301073,https://api.github.com/repos/simonw/datasette/issues/797,638301073,MDEyOklzc3VlQ29tbWVudDYzODMwMTA3Mw==,9599,simonw,2020-06-03T16:14:54Z,2020-06-09T04:00:40Z,OWNER,I want a unit test that exercises this for both writable and regular canned queries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",630120235,"Documentation for new ""params"" setting for canned queries",
https://github.com/simonw/datasette/issues/818#issuecomment-641013524,https://api.github.com/repos/simonw/datasette/issues/818,641013524,MDEyOklzc3VlQ29tbWVudDY0MTAxMzUyNA==,9599,simonw,2020-06-09T03:57:38Z,2020-06-09T04:00:24Z,OWNER,"Problem with that is it's more of a `actor_from_request` opportunity than `permission_allowed`. You could use `actor_from_request` to authenticate API clients from their `Authorization:` header, then use the regular `""allow""` blocks in `metadata.json` to actually assign their permissions.
The most interesting permissions plugin would be one that implements permissions against some kind of database schema, hence allowing admins to edit permissions through writable canned queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/818#issuecomment-641009744,https://api.github.com/repos/simonw/datasette/issues/818,641009744,MDEyOklzc3VlQ29tbWVudDY0MTAwOTc0NA==,9599,simonw,2020-06-09T03:43:18Z,2020-06-09T03:43:18Z,OWNER,`datasette-auth-bearer` perhaps?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/818#issuecomment-641009442,https://api.github.com/repos/simonw/datasette/issues/818,641009442,MDEyOklzc3VlQ29tbWVudDY0MTAwOTQ0Mg==,9599,simonw,2020-06-09T03:41:55Z,2020-06-09T03:41:55Z,OWNER,I want to build a plugin that does `Authorization: Bearer xxx` API key authentication.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/822#issuecomment-641003291,https://api.github.com/repos/simonw/datasette/issues/822,641003291,MDEyOklzc3VlQ29tbWVudDY0MTAwMzI5MQ==,9599,simonw,2020-06-09T03:17:43Z,2020-06-09T03:17:43Z,OWNER,I'm leaning towards `request.url_vars`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635077656,request.url_vars helper property,
https://github.com/simonw/datasette/issues/822#issuecomment-641003237,https://api.github.com/repos/simonw/datasette/issues/822,641003237,MDEyOklzc3VlQ29tbWVudDY0MTAwMzIzNw==,9599,simonw,2020-06-09T03:17:32Z,2020-06-09T03:17:32Z,OWNER,Currently querystring parameters are accessed through `request.args` and POST variables through `request.post_vars()`. Would be good to have a name that was somewhat consistent with those.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635077656,request.url_vars helper property,
https://github.com/simonw/datasette/issues/215#issuecomment-641002504,https://api.github.com/repos/simonw/datasette/issues/215,641002504,MDEyOklzc3VlQ29tbWVudDY0MTAwMjUwNA==,9599,simonw,2020-06-09T03:14:32Z,2020-06-09T03:14:32Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/plugins.html#register-routes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/820#issuecomment-640982533,https://api.github.com/repos/simonw/datasette/issues/820,640982533,MDEyOklzc3VlQ29tbWVudDY0MDk4MjUzMw==,9599,simonw,2020-06-09T02:00:21Z,2020-06-09T02:00:21Z,OWNER,In the case of registering API tokens it would be useful if the plugin could call a writable canned query which knows how to insert a randomly generated value. This could be achieved using a custom SQL function.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",635049296,Idea: Plugin hook for registering canned queries,
https://github.com/simonw/datasette/issues/215#issuecomment-640972952,https://api.github.com/repos/simonw/datasette/issues/215,640972952,MDEyOklzc3VlQ29tbWVudDY0MDk3Mjk1Mg==,9599,simonw,2020-06-09T01:24:52Z,2020-06-09T01:25:33Z,OWNER,WIP documentation: https://github.com/simonw/datasette/blob/770dedb21adfc706592e6b5cdf5e751a8720fdf9/docs/plugins.rst#register_routes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640971470,https://api.github.com/repos/simonw/datasette/issues/215,640971470,MDEyOklzc3VlQ29tbWVudDY0MDk3MTQ3MA==,9599,simonw,2020-06-09T01:19:44Z,2020-06-09T01:19:44Z,OWNER,I'll need to add documentation of the `Response` object (and `Response.html()` and `Response.text()` class methods - I should add `Response.json()` too) to the internals page https://datasette.readthedocs.io/en/stable/internals.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640960667,https://api.github.com/repos/simonw/datasette/issues/215,640960667,MDEyOklzc3VlQ29tbWVudDY0MDk2MDY2Nw==,9599,simonw,2020-06-09T00:41:35Z,2020-06-09T00:41:35Z,OWNER,I'm going to implement this one documentation-first in a pull request.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640960553,https://api.github.com/repos/simonw/datasette/issues/215,640960553,MDEyOklzc3VlQ29tbWVudDY0MDk2MDU1Mw==,9599,simonw,2020-06-09T00:41:09Z,2020-06-09T00:41:09Z,OWNER,"I'm going to imitate `register_output_renderer` and `register_facet_classes` - both return a list of things to register.
So I'll do this:
```python
@hookspec
def register_routes():
""Register URL routes. Return a list of (regex, view_function) pairs""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/777#issuecomment-640957423,https://api.github.com/repos/simonw/datasette/issues/777,640957423,MDEyOklzc3VlQ29tbWVudDY0MDk1NzQyMw==,9599,simonw,2020-06-09T00:29:03Z,2020-06-09T00:29:03Z,OWNER,"Here's why:
https://github.com/simonw/datasette/blob/49d6d2f7b0f6cb02e25022e1c9403811f1fa0a7c/datasette/app.py#L1024-L1029
404 errors are rendered by looking for a template from `[""404.html"", ""500.html""]`.
`404.html` doesn't actually ship with Datasette (plugins or custom template directories can provide it). So the `500.html` template is used.
That template extends `base.html`, which expects there to be `base_url` and `app_css_hash` variables. But as you can see in the excerpt above, those variables are not being passed to the template context when the error page is rendered.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626171242,Error pages not correctly loading CSS,
https://github.com/simonw/datasette/issues/777#issuecomment-640955788,https://api.github.com/repos/simonw/datasette/issues/777,640955788,MDEyOklzc3VlQ29tbWVudDY0MDk1NTc4OA==,9599,simonw,2020-06-09T00:23:26Z,2020-06-09T00:23:57Z,OWNER,"Clue: https://latest.datasette.io/404 displays correctly but https://latest.datasette.io/fixtures/404 does not.
That's because `` does the correct thing if you are on the root of the site but not if you are in a sub-directory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626171242,Error pages not correctly loading CSS,
https://github.com/simonw/datasette/issues/813#issuecomment-640951947,https://api.github.com/repos/simonw/datasette/issues/813,640951947,MDEyOklzc3VlQ29tbWVudDY0MDk1MTk0Nw==,9599,simonw,2020-06-09T00:09:56Z,2020-06-09T00:09:56Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/authentication.html#controlling-the-ability-to-execute-arbitrary-sql,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634139848,Mechanism for specifying allow_sql permission in metadata.json,
https://github.com/simonw/datasette/issues/818#issuecomment-640929693,https://api.github.com/repos/simonw/datasette/issues/818,640929693,MDEyOklzc3VlQ29tbWVudDY0MDkyOTY5Mw==,9599,simonw,2020-06-08T22:56:38Z,2020-06-08T22:56:38Z,OWNER,https://datasette.readthedocs.io/en/latest/plugins.html#permission-allowed-datasette-actor-action-resource has a couple of examples now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin,
https://github.com/simonw/datasette/issues/777#issuecomment-640925018,https://api.github.com/repos/simonw/datasette/issues/777,640925018,MDEyOklzc3VlQ29tbWVudDY0MDkyNTAxOA==,9599,simonw,2020-06-08T22:41:42Z,2020-06-08T22:41:42Z,OWNER,This is particularly worth fixing now that 403 forbidden pages are much more likely due to #811.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626171242,Error pages not correctly loading CSS,
https://github.com/simonw/datasette/issues/493#issuecomment-640924558,https://api.github.com/repos/simonw/datasette/issues/493,640924558,MDEyOklzc3VlQ29tbWVudDY0MDkyNDU1OA==,9599,simonw,2020-06-08T22:40:01Z,2020-06-08T22:40:01Z,OWNER,I'll also rename `--config` to `--setting`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",449886319,Rename metadata.json to config.json,
https://github.com/simonw/datasette/issues/493#issuecomment-640924482,https://api.github.com/repos/simonw/datasette/issues/493,640924482,MDEyOklzc3VlQ29tbWVudDY0MDkyNDQ4Mg==,9599,simonw,2020-06-08T22:39:45Z,2020-06-08T22:39:45Z,OWNER,"I'm definitely doing this rename, now that `metadata.json` is used for `allow` permissions configuration as well as-of #811.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",449886319,Rename metadata.json to config.json,
https://github.com/simonw/datasette/issues/806#issuecomment-640916991,https://api.github.com/repos/simonw/datasette/issues/806,640916991,MDEyOklzc3VlQ29tbWVudDY0MDkxNjk5MQ==,9599,simonw,2020-06-08T22:18:45Z,2020-06-08T22:18:45Z,OWNER,Reminder for release notes: I removed `--config allow_sql:0` - see https://github.com/simonw/datasette/issues/813#issuecomment-640916807,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632753851,Release Datasette 0.44,
https://github.com/simonw/datasette/issues/813#issuecomment-640916807,https://api.github.com/repos/simonw/datasette/issues/813,640916807,MDEyOklzc3VlQ29tbWVudDY0MDkxNjgwNw==,9599,simonw,2020-06-08T22:18:09Z,2020-06-08T22:18:09Z,OWNER,"I could retire the `--config allow_sql:0` option entirely, since the new `metadata.json` mechanism can be used to achieve the exact same thing.
I'm going to do that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634139848,Mechanism for specifying allow_sql permission in metadata.json,
https://github.com/simonw/datasette/issues/813#issuecomment-640916290,https://api.github.com/repos/simonw/datasette/issues/813,640916290,MDEyOklzc3VlQ29tbWVudDY0MDkxNjI5MA==,9599,simonw,2020-06-08T22:16:39Z,2020-06-08T22:17:32Z,OWNER,"Naming problem: Datasette already has a config option with this name:
$ datasette serve data.db --config allow_sql:1
https://datasette.readthedocs.io/en/stable/config.html#allow-sql
It's confusing to have two things called `allow_sql` that do slightly different things.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634139848,Mechanism for specifying allow_sql permission in metadata.json,
https://github.com/simonw/datasette/issues/801#issuecomment-640905609,https://api.github.com/repos/simonw/datasette/issues/801,640905609,MDEyOklzc3VlQ29tbWVudDY0MDkwNTYwOQ==,9599,simonw,2020-06-08T21:48:44Z,2020-06-08T21:48:44Z,OWNER,"Dropping this out of Datasette 0.44 again - I have enough other stuff to finish, this can wait.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631932926,allow_by_query setting for configuring permissions with a SQL statement,
https://github.com/simonw/datasette/issues/813#issuecomment-640837908,https://api.github.com/repos/simonw/datasette/issues/813,640837908,MDEyOklzc3VlQ29tbWVudDY0MDgzNzkwOA==,9599,simonw,2020-06-08T19:33:03Z,2020-06-08T19:33:03Z,OWNER,Don't forget to link to the `allow_sql` docs from the warning block here: https://github.com/simonw/datasette/blob/54370853828bdf87ca844fd0fc00900e0e2e659d/docs/authentication.rst#controlling-access-to-specific-tables-and-views,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634139848,Mechanism for specifying allow_sql permission in metadata.json,
https://github.com/simonw/datasette/issues/813#issuecomment-640831842,https://api.github.com/repos/simonw/datasette/issues/813,640831842,MDEyOklzc3VlQ29tbWVudDY0MDgzMTg0Mg==,9599,simonw,2020-06-08T19:27:47Z,2020-06-08T19:27:47Z,OWNER,"This needs to be ready for Datasette 0.44 because without it the ""view-table"" permission is useless - it will protect the https://latest.datasette.io/fixtures/facetable page but will not prevent users from executing https://latest.datasette.io/fixtures?sql=select+*+from+facetable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634139848,Mechanism for specifying allow_sql permission in metadata.json,
https://github.com/simonw/datasette/issues/813#issuecomment-640830088,https://api.github.com/repos/simonw/datasette/issues/813,640830088,MDEyOklzc3VlQ29tbWVudDY0MDgzMDA4OA==,9599,simonw,2020-06-08T19:26:15Z,2020-06-08T19:26:15Z,OWNER,This needs to affect the `?_where=` parameter on table pages as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634139848,Mechanism for specifying allow_sql permission in metadata.json,
https://github.com/simonw/datasette/issues/816#issuecomment-640815550,https://api.github.com/repos/simonw/datasette/issues/816,640815550,MDEyOklzc3VlQ29tbWVudDY0MDgxNTU1MA==,9599,simonw,2020-06-08T19:06:44Z,2020-06-08T19:06:44Z,OWNER,https://github.com/simonw/datasette/blob/c7d145e016522dd6ee229d4d0b3ba79a7a8877c1/docs/plugins.rst#extra_template_varstemplate-database-table-view_name-request-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634783573,Come up with a new example for extra_template_vars plugin,
https://github.com/simonw/datasette/issues/817#issuecomment-640808161,https://api.github.com/repos/simonw/datasette/issues/817,640808161,MDEyOklzc3VlQ29tbWVudDY0MDgwODE2MQ==,9599,simonw,2020-06-08T18:51:42Z,2020-06-08T18:54:37Z,OWNER,I'm also going to rename `resource_identifier` to just `resource`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634844634,Drop resource_type from permission_allowed system,
https://github.com/simonw/datasette/issues/816#issuecomment-640763899,https://api.github.com/repos/simonw/datasette/issues/816,640763899,MDEyOklzc3VlQ29tbWVudDY0MDc2Mzg5OQ==,9599,simonw,2020-06-08T17:21:59Z,2020-06-08T17:21:59Z,OWNER,I'm going to show how to display the current user's user-agent.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634783573,Come up with a new example for extra_template_vars plugin,
https://github.com/simonw/datasette/issues/815#issuecomment-640673405,https://api.github.com/repos/simonw/datasette/issues/815,640673405,MDEyOklzc3VlQ29tbWVudDY0MDY3MzQwNQ==,9599,simonw,2020-06-08T14:41:55Z,2020-06-08T14:41:55Z,OWNER,"I want to be able to display the HTTP path and verb - `GET /fixtures`, `POST /fixtures/myquery` etc. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/815#issuecomment-640673138,https://api.github.com/repos/simonw/datasette/issues/815,640673138,MDEyOklzc3VlQ29tbWVudDY0MDY3MzEzOA==,9599,simonw,2020-06-08T14:41:24Z,2020-06-08T14:41:24Z,OWNER,I could reuse that `get_task_id()` function though (I can move it to utils).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/815#issuecomment-640672540,https://api.github.com/repos/simonw/datasette/issues/815,640672540,MDEyOklzc3VlQ29tbWVudDY0MDY3MjU0MA==,9599,simonw,2020-06-08T14:40:22Z,2020-06-08T14:40:22Z,OWNER,"Here's the current tracer mechanism. Note that it captures a stacktrace (which is expensive) - but only if the tracer system has been enabled for a request.
https://github.com/simonw/datasette/blob/1c063fae9dba70f70244db010d55a18846640f07/datasette/tracer.py#L27-L51
For permissions checks I want to ALWAYS track those calls, not just on requests that have opted in.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/815#issuecomment-640671398,https://api.github.com/repos/simonw/datasette/issues/815,640671398,MDEyOklzc3VlQ29tbWVudDY0MDY3MTM5OA==,9599,simonw,2020-06-08T14:38:20Z,2020-06-08T14:38:20Z,OWNER,But `ds._permission_checks` is also used for unit tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/815#issuecomment-640671241,https://api.github.com/repos/simonw/datasette/issues/815,640671241,MDEyOklzc3VlQ29tbWVudDY0MDY3MTI0MQ==,9599,simonw,2020-06-08T14:38:04Z,2020-06-08T14:38:04Z,OWNER,"Alternative to a correlation ID would be to use the existing `AsgiTracer` / `capture_traces` mechanism. That's probably smarter.
It could even start logging SQL queries to an in-memory deque too, so a debug tool could show you queries executed by other requests!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/815#issuecomment-640656143,https://api.github.com/repos/simonw/datasette/issues/815,640656143,MDEyOklzc3VlQ29tbWVudDY0MDY1NjE0Mw==,9599,simonw,2020-06-08T14:25:48Z,2020-06-08T14:26:45Z,OWNER,Will we need a request correlation ID for this? Multiple asyncio threads can write things to the `ds._permission_checks` deque at the same time.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/814#issuecomment-640638057,https://api.github.com/repos/simonw/datasette/issues/814,640638057,MDEyOklzc3VlQ29tbWVudDY0MDYzODA1Nw==,9599,simonw,2020-06-08T14:11:51Z,2020-06-08T14:12:12Z,OWNER,"The only impact it has at all is on this code here:
https://github.com/simonw/datasette/blob/cc218fa9be55842656d030545c308392e3736053/datasette/views/base.py#L515-L527
That `ds.cache_headers` property looks like it needs rethinking too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634651079,Remove --debug option from datasette serve,
https://github.com/simonw/datasette/issues/811#issuecomment-640362879,https://api.github.com/repos/simonw/datasette/issues/811,640362879,MDEyOklzc3VlQ29tbWVudDY0MDM2Mjg3OQ==,9599,simonw,2020-06-08T04:42:28Z,2020-06-08T13:39:46Z,OWNER,"I'm finding myself repeating this pattern a lot:
```python
for table in table_counts:
allowed = await self.ds.permission_allowed(
request.scope.get(""actor""),
""view-table"",
resource_type=""table"",
resource_identifier=(database, table),
default=True,
)
if not allowed:
continue
private = not await self.ds.permission_allowed(
None,
""view-table"",
resource_type=""table"",
resource_identifier=(database, table),
)
```
I use a similar pattern for lists of databases and lists of queries, and I'll be doing the same thing for lists of SQL views too.
An abstraction around this would be useful.
Idea:
```python
visible, private = await check_visibility(
self.ds, actor, ""view-table"", ""table"", (database, table)
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640367128,https://api.github.com/repos/simonw/datasette/issues/811,640367128,MDEyOklzc3VlQ29tbWVudDY0MDM2NzEyOA==,9599,simonw,2020-06-08T05:00:13Z,2020-06-08T05:00:49Z,OWNER,"Should the padlock show up on tables that are private only because they inherited their privacy from their parent database or even the parent instance?
Interesting question. If an instance is private, I'm not sure it makes sense to show padlocks on absolutely everything.
Likewise, a list of tables shown on the database table with a padlock next to every single table (when the database itself is private) doesn't seem to add any useful information.
I think ""Show 🔒 in header on private database page"" will resolve this for me. I'll always show the padlock in the header of a database/table page even if that privacy is inherited - but I won't do that for padlocks shown in the list of tables or list of databases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640365512,https://api.github.com/repos/simonw/datasette/issues/811,640365512,MDEyOklzc3VlQ29tbWVudDY0MDM2NTUxMg==,9599,simonw,2020-06-08T04:53:49Z,2020-06-08T04:53:49Z,OWNER,"I really like the padlocks. I should include a screenshot in the documentation that illustrates them.
![]()
Maybe I should figure out a way to have the https://latest.datasette.io/ demo illustrate both a logged-in and a logged-out state.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640348785,https://api.github.com/repos/simonw/datasette/issues/811,640348785,MDEyOklzc3VlQ29tbWVudDY0MDM0ODc4NQ==,9599,simonw,2020-06-08T03:51:50Z,2020-06-08T03:51:50Z,OWNER,"New convention: the 🔒 icon is now shown next to resources that are private - that are visible to you now, but would not be visible to the anonymous user.
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640345115,https://api.github.com/repos/simonw/datasette/issues/811,640345115,MDEyOklzc3VlQ29tbWVudDY0MDM0NTExNQ==,9599,simonw,2020-06-08T03:37:33Z,2020-06-08T03:37:33Z,OWNER,Per-table permissions is pretty interesting for large installations though - an organization might have hundreds of CSV files imported into Datasette and then allow users to specify which exact users within that organization are allowed to see which CSV.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640344950,https://api.github.com/repos/simonw/datasette/issues/811,640344950,MDEyOklzc3VlQ29tbWVudDY0MDM0NDk1MA==,9599,simonw,2020-06-08T03:36:49Z,2020-06-08T03:36:49Z,OWNER,"Oh this is a bit awkward - should I be running per-table permission checks for every table that might be shown on the index page?
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/801#issuecomment-640339828,https://api.github.com/repos/simonw/datasette/issues/801,640339828,MDEyOklzc3VlQ29tbWVudDY0MDMzOTgyOA==,9599,simonw,2020-06-08T03:18:47Z,2020-06-08T03:18:47Z,OWNER,"Example. This will only allow users to access the `fixtures` database if the logged-in actor's ID value appears for a record in the `users` table which has `admin` = 1.
```json
{
""databases"": {
""fixtures"": {
""allow_by_query"": ""select * from users where id = :id and admin = 1""
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631932926,allow_by_query setting for configuring permissions with a SQL statement,
https://github.com/simonw/datasette/issues/811#issuecomment-640339674,https://api.github.com/repos/simonw/datasette/issues/811,640339674,MDEyOklzc3VlQ29tbWVudDY0MDMzOTY3NA==,9599,simonw,2020-06-08T03:18:15Z,2020-06-08T03:18:15Z,OWNER,I should take these permissions into account when displaying a list of tables or a list of databases (like I do right now when displaying a list of queries).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/801#issuecomment-640339117,https://api.github.com/repos/simonw/datasette/issues/801,640339117,MDEyOklzc3VlQ29tbWVudDY0MDMzOTExNw==,9599,simonw,2020-06-08T03:16:16Z,2020-06-08T03:16:16Z,OWNER,"I'm going to call this key `""allow_by_query""` - I think I need `allow_sql` for something else (for configuring if users are allowed to execute arbitrary SQL queries).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631932926,allow_by_query setting for configuring permissions with a SQL statement,
https://github.com/simonw/datasette/issues/811#issuecomment-640338347,https://api.github.com/repos/simonw/datasette/issues/811,640338347,MDEyOklzc3VlQ29tbWVudDY0MDMzODM0Nw==,9599,simonw,2020-06-08T03:13:23Z,2020-06-08T03:13:23Z,OWNER,Do row-level permissions even make sense? Might be a good idea to remove those until I have a good use-case for them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640338151,https://api.github.com/repos/simonw/datasette/issues/811,640338151,MDEyOklzc3VlQ29tbWVudDY0MDMzODE1MQ==,9599,simonw,2020-06-08T03:12:41Z,2020-06-08T03:12:41Z,OWNER,"Also need to expand the docs on https://datasette.readthedocs.io/en/latest/authentication.html to explain where you can put `allow` blocks to control access to the instance, database or table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640337951,https://api.github.com/repos/simonw/datasette/issues/811,640337951,MDEyOklzc3VlQ29tbWVudDY0MDMzNzk1MQ==,9599,simonw,2020-06-08T03:11:58Z,2020-06-08T03:11:58Z,OWNER,"I'd like to be able to apply permissions for the ability to run a SQL query - but I'm not sure where the best place for that `""allow""` block to live would be.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640287967,https://api.github.com/repos/simonw/datasette/issues/811,640287967,MDEyOklzc3VlQ29tbWVudDY0MDI4Nzk2Nw==,9599,simonw,2020-06-07T22:16:10Z,2020-06-07T22:16:10Z,OWNER,The tests in test_permissions.py could check the .json variants and assert that permission checks were carried out too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/395#issuecomment-640280741,https://api.github.com/repos/simonw/datasette/issues/395,640280741,MDEyOklzc3VlQ29tbWVudDY0MDI4MDc0MQ==,9599,simonw,2020-06-07T21:12:57Z,2020-06-07T21:12:57Z,OWNER,"This is a pattern I like:
```python
with make_app_client(
template_dir=str(pathlib.Path(__file__).parent / ""test_templates"")
) as client:
response = client.get(""/-/metadata"")
assert response.status == 200
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",396215043,Find a cleaner pattern for fixtures with arguments,
https://github.com/simonw/datasette/issues/801#issuecomment-640277775,https://api.github.com/repos/simonw/datasette/issues/801,640277775,MDEyOklzc3VlQ29tbWVudDY0MDI3Nzc3NQ==,9599,simonw,2020-06-07T20:49:40Z,2020-06-07T20:49:40Z,OWNER,"I'm going to pass the entire actor object as a dictionary of available named query parameters. So if the actor looks like this:
```json
{
""id"": ""simonw"",
""roles"": [""staff"", ""developer""]
}
```
Then the SQL query will be called like this:
```python
conn.execute(sql, {
""id"": ""simonw"",
""roles: '[""staff"", ""developer""]',
})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631932926,allow_by_query setting for configuring permissions with a SQL statement,
https://github.com/simonw/datasette/issues/801#issuecomment-640277557,https://api.github.com/repos/simonw/datasette/issues/801,640277557,MDEyOklzc3VlQ29tbWVudDY0MDI3NzU1Nw==,9599,simonw,2020-06-07T20:48:00Z,2020-06-07T20:48:00Z,OWNER,"Now that I'm expanding permission checks to everything else too (#811), not just canned queries, I think it makes sense to re-prioritize this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631932926,allow_by_query setting for configuring permissions with a SQL statement,
https://github.com/simonw/datasette/issues/811#issuecomment-640274171,https://api.github.com/repos/simonw/datasette/issues/811,640274171,MDEyOklzc3VlQ29tbWVudDY0MDI3NDE3MQ==,9599,simonw,2020-06-07T20:21:14Z,2020-06-07T20:21:14Z,OWNER,"Next step: fix this
```
- # TODO: fix this to use that permission check
- if not actor_matches_allow(
- request.scope.get(""actor"", None), metadata.get(""allow"")
- ):
- return Response(""Permission denied"", status=403)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640273945,https://api.github.com/repos/simonw/datasette/issues/811,640273945,MDEyOklzc3VlQ29tbWVudDY0MDI3Mzk0NQ==,9599,simonw,2020-06-07T20:19:15Z,2020-06-07T20:19:15Z,OWNER,I'm going to add a `test_permissions.py` module that checks for 403 errors against different patterns of the `actors` block at different levels in `metadata.json`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640270178,https://api.github.com/repos/simonw/datasette/issues/811,640270178,MDEyOklzc3VlQ29tbWVudDY0MDI3MDE3OA==,9599,simonw,2020-06-07T19:48:39Z,2020-06-07T19:48:39Z,OWNER,"Testing pattern:
```python
def test_canned_query_with_custom_metadata(app_client):
response = app_client.get(""/fixtures/neighborhood_search?text=town"")
assert_permissions_checked(
app_client.ds,
[
""view-instance"",
(""view-database"", ""database"", ""fixtures""),
(""view-query"", ""query"", (""fixtures"", ""neighborhood_search"")),
],
)
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/811#issuecomment-640248972,https://api.github.com/repos/simonw/datasette/issues/811,640248972,MDEyOklzc3VlQ29tbWVudDY0MDI0ODk3Mg==,9599,simonw,2020-06-07T17:04:22Z,2020-06-07T17:04:22Z,OWNER,I'll need a neat testing pattern for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/810#issuecomment-640248864,https://api.github.com/repos/simonw/datasette/issues/810,640248864,MDEyOklzc3VlQ29tbWVudDY0MDI0ODg2NA==,9599,simonw,2020-06-07T17:03:15Z,2020-06-07T17:03:15Z,OWNER,This is obsoleted by #811.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633066114,Refactor permission check for canned query,
https://github.com/simonw/datasette/issues/811#issuecomment-640248669,https://api.github.com/repos/simonw/datasette/issues/811,640248669,MDEyOklzc3VlQ29tbWVudDY0MDI0ODY2OQ==,9599,simonw,2020-06-07T17:01:44Z,2020-06-07T17:01:44Z,OWNER,"If the allow block at the database level forbids access this needs to cascade down to the table, query and row levels as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",633578769,"Support ""allow"" block on root, databases and tables, not just queries",
https://github.com/simonw/datasette/issues/215#issuecomment-640121917,https://api.github.com/repos/simonw/datasette/issues/215,640121917,MDEyOklzc3VlQ29tbWVudDY0MDEyMTkxNw==,9599,simonw,2020-06-06T21:42:58Z,2020-06-07T05:58:36Z,OWNER,"I might use some dependency injection here, with `call_with_supported_arguments()` from https://github.com/simonw/datasette/commit/41a0cd7b6afe0397efbbf27ad822679fc574811a#diff-942305c83055fdc0ff5f4e7d6ab06b29
Maybe a view function can take `request` and optionally also take `datasette`? Or `scope` or `receive` or `send`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/800#issuecomment-640160487,https://api.github.com/repos/simonw/datasette/issues/800,640160487,MDEyOklzc3VlQ29tbWVudDY0MDE2MDQ4Nw==,9599,simonw,2020-06-07T05:34:07Z,2020-06-07T05:34:07Z,OWNER,See #810 for work to finish this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/808#issuecomment-640157216,https://api.github.com/repos/simonw/datasette/issues/808,640157216,MDEyOklzc3VlQ29tbWVudDY0MDE1NzIxNg==,9599,simonw,2020-06-07T04:58:40Z,2020-06-07T04:58:40Z,OWNER,... and I want a unit test which confirms that all permissions are documented.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632918799,Permission check for every view in Datasette (plus docs),
https://github.com/simonw/datasette/issues/808#issuecomment-640152036,https://api.github.com/repos/simonw/datasette/issues/808,640152036,MDEyOklzc3VlQ29tbWVudDY0MDE1MjAzNg==,9599,simonw,2020-06-07T03:38:07Z,2020-06-07T03:38:07Z,OWNER,I'm going to need to add permissions documentation for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632918799,Permission check for every view in Datasette (plus docs),
https://github.com/simonw/datasette/issues/807#issuecomment-640135332,https://api.github.com/repos/simonw/datasette/issues/807,640135332,MDEyOklzc3VlQ29tbWVudDY0MDEzNTMzMg==,9599,simonw,2020-06-07T00:13:51Z,2020-06-07T00:13:51Z,OWNER,"These should not be shipped as the latest version on Docker Hub. They also should not become the ""stable"" release on ReadTheDocs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632843030,Ability to ship alpha and beta releases,
https://github.com/simonw/datasette/issues/800#issuecomment-640123488,https://api.github.com/repos/simonw/datasette/issues/800,640123488,MDEyOklzc3VlQ29tbWVudDY0MDEyMzQ4OA==,9599,simonw,2020-06-06T21:59:14Z,2020-06-06T21:59:14Z,OWNER,I didn't build this quite right: it should be using the permissions plugin hook.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/805#issuecomment-640122664,https://api.github.com/repos/simonw/datasette/issues/805,640122664,MDEyOklzc3VlQ29tbWVudDY0MDEyMjY2NA==,9599,simonw,2020-06-06T21:50:41Z,2020-06-06T21:50:41Z,OWNER,Part of #806 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/215#issuecomment-504881900,https://api.github.com/repos/simonw/datasette/issues/215,504881900,MDEyOklzc3VlQ29tbWVudDUwNDg4MTkwMA==,9599,simonw,2019-06-24T06:51:29Z,2020-06-06T21:47:11Z,OWNER,See also #520 - asgi_wrapper plugin hook.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-398826108,https://api.github.com/repos/simonw/datasette/issues/215,398826108,MDEyOklzc3VlQ29tbWVudDM5ODgyNjEwOA==,9599,simonw,2018-06-20T17:09:18Z,2020-06-06T21:46:51Z,OWNER,This depends on #272 - Datasette ported to ASGI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640122120,https://api.github.com/repos/simonw/datasette/issues/215,640122120,MDEyOklzc3VlQ29tbWVudDY0MDEyMjEyMA==,9599,simonw,2020-06-06T21:45:13Z,2020-06-06T21:45:52Z,OWNER,"Stretch goal: make it easy for plugin views to implement formats, so they can produce HTML by default and .json or .csv etc as alternative outputs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640121036,https://api.github.com/repos/simonw/datasette/issues/215,640121036,MDEyOklzc3VlQ29tbWVudDY0MDEyMTAzNg==,9599,simonw,2020-06-06T21:34:03Z,2020-06-06T21:34:03Z,OWNER,"I'll refactor existing code to register views using the same mechanism that plugins will have access to.
Maybe plugins get to register their routes first? That would allow plugins to do things like entirely take over the / page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640119259,https://api.github.com/repos/simonw/datasette/issues/215,640119259,MDEyOklzc3VlQ29tbWVudDY0MDExOTI1OQ==,9599,simonw,2020-06-06T21:16:46Z,2020-06-06T21:16:46Z,OWNER,"I deprioritised this a while ago because the asgi_wrapper hook allowed me to set up new URL routes: https://datasette.readthedocs.io/en/0.43/plugins.html#asgi-wrapper-datasette
But... those were pretty low level, for example this code here: https://github.com/simonw/datasette-auth-github/blob/6c971064f6f4e6857bade5c6b88842f9cdeca9d9/datasette_auth_github/github_auth.py#L104-L113
Now that Datasette has a documented request object #706 and that object is used by things like the flash messages system (#790) - https://datasette.readthedocs.io/en/latest/internals.html#add-message-request-message-message-type-datasette-info - I find myself wanting to add views which get a request, as opposed to an ASGI scope.
So I'm re-prioritising this, with the main need being a way for plugins to hook up their own view functions that can accept a request and return a response. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/215#issuecomment-640118802,https://api.github.com/repos/simonw/datasette/issues/215,640118802,MDEyOklzc3VlQ29tbWVudDY0MDExODgwMg==,9599,simonw,2020-06-06T21:12:41Z,2020-06-06T21:12:41Z,OWNER,@clausjuhl your use-case there is now covered by custom pages from Datasette 0.41 https://datasette.readthedocs.io/en/stable/changelog.html#v0-41,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314506669,Allow plugins to define additional URL routes and views,
https://github.com/simonw/datasette/issues/805#issuecomment-640116970,https://api.github.com/repos/simonw/datasette/issues/805,640116970,MDEyOklzc3VlQ29tbWVudDY0MDExNjk3MA==,9599,simonw,2020-06-06T20:55:03Z,2020-06-06T20:55:03Z,OWNER,"Would be useful if I had a plugin that could authenticate users based on a secret environment variable (maybe for a password) - that way I could have an ""admin"" account on the Glitch app that is allowed to setup new polls, while anonymous users can only vote on them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/805#issuecomment-640116842,https://api.github.com/repos/simonw/datasette/issues/805,640116842,MDEyOklzc3VlQ29tbWVudDY0MDExNjg0Mg==,9599,simonw,2020-06-06T20:53:51Z,2020-06-06T20:53:51Z,OWNER,"I'd like to illustrate writable canned queries without the risk of someone abusing and breaking it (or filling it with bad content).
I don't want to have to monitor it, so an application that won't run out of disk space after a few months would be good too.
Maybe a polling app? If I'm only tracking integer numbers of votes it shouldn't ever run out of space.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/791#issuecomment-640116494,https://api.github.com/repos/simonw/datasette/issues/791,640116494,MDEyOklzc3VlQ29tbWVudDY0MDExNjQ5NA==,9599,simonw,2020-06-06T20:50:41Z,2020-06-06T20:50:41Z,OWNER,"I have a better idea: a feed reader! You can insert URLs to feeds, then have a command which fetches the latest entries from them into a separate table. Then implement favorites as a canned query, let you search your favorites, etc.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",628572716,Tutorial: building a something-interesting with writable canned queries,
https://github.com/simonw/datasette/issues/787#issuecomment-640111383,https://api.github.com/repos/simonw/datasette/issues/787,640111383,MDEyOklzc3VlQ29tbWVudDY0MDExMTM4Mw==,9599,simonw,2020-06-06T20:04:20Z,2020-06-06T20:04:20Z,OWNER,"I should let people running the 'publish' command set this explicitly if they want to, so they can re-deploy a published Datasette without invalidating every user's cookies.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628089318,"""datasette publish"" should bake in a random --secret",
https://github.com/simonw/datasette/issues/698#issuecomment-640108942,https://api.github.com/repos/simonw/datasette/issues/698,640108942,MDEyOklzc3VlQ29tbWVudDY0MDEwODk0Mg==,9599,simonw,2020-06-06T19:43:48Z,2020-06-06T19:43:48Z,OWNER,"Landed - documentation is here: https://datasette.readthedocs.io/en/latest/sql_queries.html#writable-canned-queries
See also https://datasette.readthedocs.io/en/latest/authentication.html#permissions-for-canned-queries","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/800#issuecomment-640108835,https://api.github.com/repos/simonw/datasette/issues/800,640108835,MDEyOklzc3VlQ29tbWVudDY0MDEwODgzNQ==,9599,simonw,2020-06-06T19:42:46Z,2020-06-06T19:42:46Z,OWNER,This is implemented and documented: https://datasette.readthedocs.io/en/latest/authentication.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/699#issuecomment-640108763,https://api.github.com/repos/simonw/datasette/issues/699,640108763,MDEyOklzc3VlQ29tbWVudDY0MDEwODc2Mw==,9599,simonw,2020-06-06T19:42:11Z,2020-06-06T19:42:11Z,OWNER,I landed canned query writes. This feature can now be considered complete: https://datasette.readthedocs.io/en/latest/authentication.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-640106668,https://api.github.com/repos/simonw/datasette/issues/699,640106668,MDEyOklzc3VlQ29tbWVudDY0MDEwNjY2OA==,9599,simonw,2020-06-06T19:22:36Z,2020-06-06T19:22:36Z,OWNER,The canned queries feature is gaining permissions support in #800.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/804#issuecomment-640106569,https://api.github.com/repos/simonw/datasette/issues/804,640106569,MDEyOklzc3VlQ29tbWVudDY0MDEwNjU2OQ==,9599,simonw,2020-06-06T19:21:41Z,2020-06-06T19:21:41Z,OWNER,I don't think this is fully documented either. Current partial documentation is on https://datasette.readthedocs.io/en/stable/contributing.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632673972,python tests/fixtures.py command has a bug,
https://github.com/simonw/datasette/issues/804#issuecomment-640106342,https://api.github.com/repos/simonw/datasette/issues/804,640106342,MDEyOklzc3VlQ29tbWVudDY0MDEwNjM0Mg==,9599,simonw,2020-06-06T19:19:33Z,2020-06-06T19:19:33Z,OWNER,I should replace the bodged-together argument passing with Click while I'm fixing this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632673972,python tests/fixtures.py command has a bug,
https://github.com/simonw/datasette/issues/804#issuecomment-640106202,https://api.github.com/repos/simonw/datasette/issues/804,640106202,MDEyOklzc3VlQ29tbWVudDY0MDEwNjIwMg==,9599,simonw,2020-06-06T19:18:23Z,2020-06-06T19:18:43Z,OWNER,"I broke this in #775 https://github.com/simonw/datasette/commit/446e5de65d1b9c6c877e38b0ef13bc9285c465a1
Here's the now-broken code (I removed the `PLUGIN1` and `PLUGIN2` constants): https://github.com/simonw/datasette/blob/9c563d6aed072f14d3d25f58e84659f9caa1a243/tests/fixtures.py#L828-L835","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632673972,python tests/fixtures.py command has a bug,
https://github.com/simonw/datasette/issues/800#issuecomment-640103204,https://api.github.com/repos/simonw/datasette/issues/800,640103204,MDEyOklzc3VlQ29tbWVudDY0MDEwMzIwNA==,9599,simonw,2020-06-06T18:52:56Z,2020-06-06T18:52:56Z,OWNER,"I'm also going to add an indicator to the UI next to queries that you can only execute because you are signed in:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640102200,https://api.github.com/repos/simonw/datasette/issues/800,640102200,MDEyOklzc3VlQ29tbWVudDY0MDEwMjIwMA==,9599,simonw,2020-06-06T18:45:11Z,2020-06-06T18:45:11Z,OWNER,"In the code that's:
https://github.com/simonw/datasette/blob/9c563d6aed072f14d3d25f58e84659f9caa1a243/datasette/views/database.py#L56-L64
And:
https://github.com/simonw/datasette/blob/9c563d6aed072f14d3d25f58e84659f9caa1a243/datasette/views/database.py#L98-L112
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640101762,https://api.github.com/repos/simonw/datasette/issues/800,640101762,MDEyOklzc3VlQ29tbWVudDY0MDEwMTc2Mg==,9599,simonw,2020-06-06T18:41:20Z,2020-06-06T18:41:20Z,OWNER,Now the actual permission checks. I need these in two places: the code that generates the list of available queries on https://latest.datasette.io/fixtures#queries and the query page itself at https://latest.datasette.io/fixtures/pragma_cache_size,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640101625,https://api.github.com/repos/simonw/datasette/issues/800,640101625,MDEyOklzc3VlQ29tbWVudDY0MDEwMTYyNQ==,9599,simonw,2020-06-06T18:40:09Z,2020-06-06T18:40:09Z,OWNER,Documentation for `actor_matches_allow`: https://github.com/simonw/datasette/blob/14f6b4d200f24940a795ddc0825319ab2891bde2/docs/authentication.rst#actor_matches_allow,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640099707,https://api.github.com/repos/simonw/datasette/issues/800,640099707,MDEyOklzc3VlQ29tbWVudDY0MDA5OTcwNw==,9599,simonw,2020-06-06T18:24:54Z,2020-06-06T18:24:54Z,OWNER,Next step: a utility function and tests for matching actors to allow blocks.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640099404,https://api.github.com/repos/simonw/datasette/issues/800,640099404,MDEyOklzc3VlQ29tbWVudDY0MDA5OTQwNA==,9599,simonw,2020-06-06T18:22:10Z,2020-06-06T18:24:26Z,OWNER,Docs here: https://github.com/simonw/datasette/blob/d4c7b85f556230923d37ff327a068ed08aa9b62b/docs/authentication.rst#setting-permissions-for-canned-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640099434,https://api.github.com/repos/simonw/datasette/issues/800,640099434,MDEyOklzc3VlQ29tbWVudDY0MDA5OTQzNA==,9599,simonw,2020-06-06T18:22:29Z,2020-06-06T18:22:29Z,OWNER,I should add the '*' bit to the docs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/786#issuecomment-640099333,https://api.github.com/repos/simonw/datasette/issues/786,640099333,MDEyOklzc3VlQ29tbWVudDY0MDA5OTMzMw==,9599,simonw,2020-06-06T18:21:36Z,2020-06-06T18:21:36Z,OWNER,"This is done but currently lives in a branch, will close this issue when that branch lands: Implemented in this branch: https://github.com/simonw/datasette/blob/30a8132d58a89fed0e034e058b62fab5180fae0f/docs/authentication.rst","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628087971,Documentation page describing Datasette's authentication system,
https://github.com/simonw/datasette/issues/800#issuecomment-640090575,https://api.github.com/repos/simonw/datasette/issues/800,640090575,MDEyOklzc3VlQ29tbWVudDY0MDA5MDU3NQ==,9599,simonw,2020-06-06T17:06:28Z,2020-06-06T17:06:28Z,OWNER,I'm going to implement this documentation-first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-640090343,https://api.github.com/repos/simonw/datasette/issues/800,640090343,MDEyOklzc3VlQ29tbWVudDY0MDA5MDM0Mw==,9599,simonw,2020-06-06T17:04:36Z,2020-06-06T17:04:36Z,OWNER,I like this mechanism better than the SQL query one. Constructing SQL queries that return true if a particular string is embedded inside a JSON list in a larger object is decidedly non-trivial.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/802#issuecomment-639895450,https://api.github.com/repos/simonw/datasette/issues/802,639895450,MDEyOklzc3VlQ29tbWVudDYzOTg5NTQ1MA==,9599,simonw,2020-06-05T23:33:52Z,2020-06-05T23:33:52Z,OWNER,"https://github.com/simonw/datasette/blob/033a1bb22c70a955d9fd1d3b4675a0e2e5c8b8cd/datasette/cli.py#L126-L129
But I changed the `.plugins()` method to this:
https://github.com/simonw/datasette/blob/033a1bb22c70a955d9fd1d3b4675a0e2e5c8b8cd/datasette/app.py#L628-L633","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632056825,"""datasette plugins"" command is broken",
https://github.com/simonw/datasette/issues/800#issuecomment-639803719,https://api.github.com/repos/simonw/datasette/issues/800,639803719,MDEyOklzc3VlQ29tbWVudDYzOTgwMzcxOQ==,9599,simonw,2020-06-05T20:40:34Z,2020-06-05T20:40:34Z,OWNER,It's a bit obscure though. I'll try building both and see how they feel in practice.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/800#issuecomment-639803099,https://api.github.com/repos/simonw/datasette/issues/800,639803099,MDEyOklzc3VlQ29tbWVudDYzOTgwMzA5OQ==,9599,simonw,2020-06-05T20:39:34Z,2020-06-05T20:39:34Z,OWNER,"Maybe #801 (configuring permissions with a SQL query) is enough here - might not need this mechanism at all, since that mechanism covers it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631931408,Canned query permissions mechanism,
https://github.com/simonw/datasette/issues/698#issuecomment-639788562,https://api.github.com/repos/simonw/datasette/issues/698,639788562,MDEyOklzc3VlQ29tbWVudDYzOTc4ODU2Mg==,9599,simonw,2020-06-05T20:27:49Z,2020-06-05T20:27:49Z,OWNER,"There can be a detailed section explaining these different mechanisms on the authentication documentation page.
I imagine they will end up applying to more than just canned queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-639787304,https://api.github.com/repos/simonw/datasette/issues/698,639787304,MDEyOklzc3VlQ29tbWVudDYzOTc4NzMwNA==,9599,simonw,2020-06-05T20:26:57Z,2020-06-05T20:26:57Z,OWNER,"Idea: an `""allow_sql""` key with a SQL query that gets passed the actor JSON as `:actor` and can extract the relevant keys from it and return 1 or 0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-639785878,https://api.github.com/repos/simonw/datasette/issues/698,639785878,MDEyOklzc3VlQ29tbWVudDYzOTc4NTg3OA==,9599,simonw,2020-06-05T20:25:55Z,2020-06-05T20:25:55Z,OWNER,"I'd really like to support SQL query defined permissions too, mainly to set an example for how plugins could do something similar.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-639784651,https://api.github.com/repos/simonw/datasette/issues/698,639784651,MDEyOklzc3VlQ29tbWVudDYzOTc4NDY1MQ==,9599,simonw,2020-06-05T20:25:02Z,2020-06-05T20:25:02Z,OWNER,"Idea: default is anyone can execute a query.
Or you can specify the following:
```json
{
""databases"": {
""my-database"": {
""queries"": {
""add_twitter_handle"": {
""sql"": ""insert into twitter_handles (username) values (:username)"",
""write"": true,
""allow"": {
""id"": [""simon""],
""role"": [""staff""]
}
}
}
}
}
}
```
These get matched against the actor JSON. If any of the fields in any of the keys of `""allow""` match a key on the actor, the query is allowed.
`""id"": ""*""` matches any actor with an `id` key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-639779403,https://api.github.com/repos/simonw/datasette/issues/698,639779403,MDEyOklzc3VlQ29tbWVudDYzOTc3OTQwMw==,9599,simonw,2020-06-05T20:20:12Z,2020-06-05T20:20:12Z,OWNER,CSRF is done. Last step: figure out a smart way to integrate this with permissions and authentication.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/pull/798#issuecomment-639712835,https://api.github.com/repos/simonw/datasette/issues/798,639712835,MDEyOklzc3VlQ29tbWVudDYzOTcxMjgzNQ==,9599,simonw,2020-06-05T18:53:32Z,2020-06-05T18:53:32Z,OWNER,Add unit tests illustrating the `Vary: Cookie` header and I'm done here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631300342,CSRF protection,
https://github.com/simonw/datasette/pull/798#issuecomment-639685550,https://api.github.com/repos/simonw/datasette/issues/798,639685550,MDEyOklzc3VlQ29tbWVudDYzOTY4NTU1MA==,9599,simonw,2020-06-05T18:20:34Z,2020-06-05T18:20:34Z,OWNER,I'm solving the compatibility with caching problem in this ticket: https://github.com/simonw/asgi-csrf/issues/7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631300342,CSRF protection,
https://github.com/simonw/datasette/issues/799#issuecomment-639661014,https://api.github.com/repos/simonw/datasette/issues/799,639661014,MDEyOklzc3VlQ29tbWVudDYzOTY2MTAxNA==,9599,simonw,2020-06-05T17:43:41Z,2020-06-05T17:43:41Z,OWNER,I'm going to rename that `MultiParams` and use it in both places.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631789422,TestResponse needs to handle multiple set-cookie headers,
https://github.com/simonw/datasette/issues/799#issuecomment-639660667,https://api.github.com/repos/simonw/datasette/issues/799,639660667,MDEyOklzc3VlQ29tbWVudDYzOTY2MDY2Nw==,9599,simonw,2020-06-05T17:43:08Z,2020-06-05T17:43:08Z,OWNER,"This really needs a `MultiValueDict` ala Django: https://github.com/django/django/blob/24b82cd201e21060fbc02117dc16d1702877a1f3/django/utils/datastructures.py#L42
Turns out I have one of these in Datasette already - `RequestParameters` from https://github.com/simonw/datasette/commit/81be31322a968d23cf57cee62b58df55433385e3
The name isn't quite right though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631789422,TestResponse needs to handle multiple set-cookie headers,
https://github.com/simonw/datasette/pull/798#issuecomment-639269994,https://api.github.com/repos/simonw/datasette/issues/798,639269994,MDEyOklzc3VlQ29tbWVudDYzOTI2OTk5NA==,9599,simonw,2020-06-05T05:36:35Z,2020-06-05T05:38:25Z,OWNER,"Django docs on CSRF and caching: https://docs.djangoproject.com/en/3.0/ref/csrf/#caching
> If the csrf_token template tag is used by a template (or the get_token function is called some other way), CsrfViewMiddleware will add a cookie and a Vary: Cookie header to the response. This means that the middleware will play well with the cache middleware if it is used as instructed
So the cookie is only set for pages that included a hidden csrftoken form field! This could work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631300342,CSRF protection,
https://github.com/simonw/datasette/pull/798#issuecomment-639269559,https://api.github.com/repos/simonw/datasette/issues/798,639269559,MDEyOklzc3VlQ29tbWVudDYzOTI2OTU1OQ==,9599,simonw,2020-06-05T05:34:56Z,2020-06-05T05:35:23Z,OWNER,"I don't want to set a cookie on a page response that is being cached.
Right now the ASGI middleware will be doing exactly that, which is bad.
But how do I get certainty that when you load a page with a form that will be CSRF protected you have been served the cookie?
Maybe those pages should do something explicit to the request object indicating that the cookie is needed?
That works for Datasette (since it has mutable request objects) but I'm not sure how it would work in the asgi-csrf pure ASGI middleware context.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631300342,CSRF protection,
https://github.com/simonw/datasette/pull/798#issuecomment-639249743,https://api.github.com/repos/simonw/datasette/issues/798,639249743,MDEyOklzc3VlQ29tbWVudDYzOTI0OTc0Mw==,9599,simonw,2020-06-05T04:23:01Z,2020-06-05T04:23:01Z,OWNER,"Needs unit tests.
More importantly: needs very, very careful consideration of how this plays with HTTP caching.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",631300342,CSRF protection,
https://github.com/simonw/datasette/issues/684#issuecomment-639053707,https://api.github.com/repos/simonw/datasette/issues/684,639053707,MDEyOklzc3VlQ29tbWVudDYzOTA1MzcwNw==,9599,simonw,2020-06-04T18:56:15Z,2020-06-04T18:56:15Z,OWNER,This documentation is live here: https://datasette.readthedocs.io/en/latest/internals.html#database-introspection,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570301333,Add documentation on Database introspection methods to internals.rst,
https://github.com/simonw/datasette/issues/119#issuecomment-639047315,https://api.github.com/repos/simonw/datasette/issues/119,639047315,MDEyOklzc3VlQ29tbWVudDYzOTA0NzMxNQ==,9599,simonw,2020-06-04T18:46:39Z,2020-06-04T18:46:39Z,OWNER,"The OAuth dance needed for this is a pretty nasty barrier to plugin installation and configuration.
I'm going to focus on making it easy to copy and paste data into sheets instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275082158,"Build an ""export this data to google sheets"" plugin",
https://github.com/simonw/datasette/issues/793#issuecomment-638462052,https://api.github.com/repos/simonw/datasette/issues/793,638462052,MDEyOklzc3VlQ29tbWVudDYzODQ2MjA1Mg==,9599,simonw,2020-06-03T21:07:39Z,2020-06-03T21:07:39Z,OWNER,I need to land and release the fix for signing cookies in https://github.com/simonw/asgi-csrf/issues/2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629524205,CSRF protection for /-/messages tool and writable canned queries,
https://github.com/simonw/datasette/issues/797#issuecomment-638461797,https://api.github.com/repos/simonw/datasette/issues/797,638461797,MDEyOklzc3VlQ29tbWVudDYzODQ2MTc5Nw==,9599,simonw,2020-06-03T21:07:06Z,2020-06-03T21:07:06Z,OWNER,"Docs here (search for ""params""): https://datasette.readthedocs.io/en/latest/sql_queries.html#canned-queries-named-parameters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",630120235,"Documentation for new ""params"" setting for canned queries",
https://github.com/simonw/datasette/issues/797#issuecomment-638289878,https://api.github.com/repos/simonw/datasette/issues/797,638289878,MDEyOklzc3VlQ29tbWVudDYzODI4OTg3OA==,9599,simonw,2020-06-03T15:57:47Z,2020-06-03T15:57:47Z,OWNER,Also mention ability to pre-fill the form for writable canned queries using the querystring.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",630120235,"Documentation for new ""params"" setting for canned queries",
https://github.com/simonw/datasette/issues/698#issuecomment-638266171,https://api.github.com/repos/simonw/datasette/issues/698,638266171,MDEyOklzc3VlQ29tbWVudDYzODI2NjE3MQ==,9599,simonw,2020-06-03T15:18:49Z,2020-06-03T15:18:49Z,OWNER,Landed the work so far from #796! Here's the documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#writable-canned-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/793#issuecomment-638265394,https://api.github.com/repos/simonw/datasette/issues/793,638265394,MDEyOklzc3VlQ29tbWVudDYzODI2NTM5NA==,9599,simonw,2020-06-03T15:17:35Z,2020-06-03T15:17:51Z,OWNER,I need this for writable canned queries in #698 and #796 too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629524205,CSRF protection for /-/messages tool and writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638257697,https://api.github.com/repos/simonw/datasette/issues/796,638257697,MDEyOklzc3VlQ29tbWVudDYzODI1NzY5Nw==,9599,simonw,2020-06-03T15:05:07Z,2020-06-03T15:05:07Z,OWNER,"I'm going to document this, land it and then continue to work on the other pieces - CSRF protection and .json mode - in separate tickets.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638249652,https://api.github.com/repos/simonw/datasette/issues/796,638249652,MDEyOklzc3VlQ29tbWVudDYzODI0OTY1Mg==,9599,simonw,2020-06-03T14:51:29Z,2020-06-03T14:51:51Z,OWNER,"Consider this one:
```
""delete_name"": {
""sql"": ""delete from names where rowid = :rowid"",
""write"": True,
""on_success_message"": ""Name deleted"",
},
```
If the user enters an invalid `rowid` the query will still execute without errors and hence the success message will still be displayed. Can I address this?
Maybe allow an optional `""rowcount_expected"": 1` property? And if that count isn't matched treat the query as an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638241738,https://api.github.com/repos/simonw/datasette/issues/796,638241738,MDEyOklzc3VlQ29tbWVudDYzODI0MTczOA==,9599,simonw,2020-06-03T14:38:45Z,2020-06-03T14:38:45Z,OWNER,Violating a unique constraint will throw an error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638241366,https://api.github.com/repos/simonw/datasette/issues/796,638241366,MDEyOklzc3VlQ29tbWVudDYzODI0MTM2Ng==,9599,simonw,2020-06-03T14:38:03Z,2020-06-03T14:38:31Z,OWNER,"Maybe I need some kind of optional validation mechanism? SQLite lets me insert text or floating point numbers into integer columns right now, which feels wrong.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638240919,https://api.github.com/repos/simonw/datasette/issues/796,638240919,MDEyOklzc3VlQ29tbWVudDYzODI0MDkxOQ==,9599,simonw,2020-06-03T14:37:22Z,2020-06-03T14:37:22Z,OWNER,"I'm having trouble coming up with a canned SQL query that will only error with certain parameters, which I need in order to test out the `on_error_redirect` and `on_error_message` properties.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638238144,https://api.github.com/repos/simonw/datasette/issues/796,638238144,MDEyOklzc3VlQ29tbWVudDYzODIzODE0NA==,9599,simonw,2020-06-03T14:32:54Z,2020-06-03T14:33:11Z,OWNER,I'm going to have `on_success_redirect` and `on_error_redirect` properties for specifying where the redirect should go too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638188196,https://api.github.com/repos/simonw/datasette/issues/796,638188196,MDEyOklzc3VlQ29tbWVudDYzODE4ODE5Ng==,9599,simonw,2020-06-03T13:13:27Z,2020-06-03T14:32:27Z,OWNER,"""Query executed"" is the default message, but it's pretty bland:
![]()
How about letting queries define custom success messages in their metadata configuration?
`""on_success_message""` and `""on_error_message""`
How can the system tell if an ""update"" query was actually successful? Maybe I should expose `.rowcount` somehow, so I can report back on how many rows were updated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638206851,https://api.github.com/repos/simonw/datasette/issues/796,638206851,MDEyOklzc3VlQ29tbWVudDYzODIwNjg1MQ==,9599,simonw,2020-06-03T13:44:29Z,2020-06-03T13:44:29Z,OWNER,"Default message is now ""Query executed, 1 row affected"" as of https://github.com/simonw/datasette/pull/796/commits/f45c44ac8f3a9a2182d76c6bda44a06676499e4b","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/pull/796#issuecomment-638205923,https://api.github.com/repos/simonw/datasette/issues/796,638205923,MDEyOklzc3VlQ29tbWVudDYzODIwNTkyMw==,9599,simonw,2020-06-03T13:43:12Z,2020-06-03T13:43:12Z,OWNER,For `.json` mode (when the URL has a `.json` suffix) I'm going to handle both form-encoded and JSON encoded inputs and return a 200 status code with JSON that tells you if the query executed successfully and how many rows were affected.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629595228,New WIP writable canned queries,
https://github.com/simonw/datasette/issues/698#issuecomment-638183337,https://api.github.com/repos/simonw/datasette/issues/698,638183337,MDEyOklzc3VlQ29tbWVudDYzODE4MzMzNw==,9599,simonw,2020-06-03T13:05:03Z,2020-06-03T13:05:03Z,OWNER,"One challenge with this feature is that it confuses the messaging about what Datasette does somewhat.
Prior to shipping this, Datasette's core value proposition is as a way to publish read-only data.
That changed a little [in 0.37 in February](https://datasette.readthedocs.io/en/stable/changelog.html#v0-37) when plugins gained the supported ability to execute writes, but there was no way of doing that without a plugin.
With this feature, Datasette becomes a read-write database solution.
I should update the documentation to help explain this. Essentially the message is that Datasette can be used in one of two ""modes"" - it can be used just for sharing/publishing data, or you can use it to collect and manage data, most likely still in collaboration with plugins for things like authentication.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-637934813,https://api.github.com/repos/simonw/datasette/issues/698,637934813,MDEyOklzc3VlQ29tbWVudDYzNzkzNDgxMw==,9599,simonw,2020-06-03T03:45:07Z,2020-06-03T03:45:07Z,OWNER,"Some extra thoughts now that this is mostly working:
- ""Edit this row"" is such an obvious use-case. Could I automatically support row editing where every column except the primary key can be updated?
- It would be useful to be able to link to a query in a way that pre-populates various form fields. The ""edit"" interface could then be a link that pre-populates the form with all of the existing values.
- Can the redirect URL be configured to include values from the form submission? So you could e.g. add a blog post with a unique slug and then redirect to that URL?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-637879242,https://api.github.com/repos/simonw/datasette/issues/698,637879242,MDEyOklzc3VlQ29tbWVudDYzNzg3OTI0Mg==,9599,simonw,2020-06-03T00:10:30Z,2020-06-03T00:10:30Z,OWNER,Started a fresh pull request for this in #796 - the one in #703 got a bit untidy.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/pull/703#issuecomment-637875307,https://api.github.com/repos/simonw/datasette/issues/703,637875307,MDEyOklzc3VlQ29tbWVudDYzNzg3NTMwNw==,9599,simonw,2020-06-02T23:57:35Z,2020-06-02T23:57:35Z,OWNER,This pull request got too messy. I'm going to abandon this and start a new one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585597133,WIP implementation of writable canned queries,
https://github.com/simonw/datasette/issues/790#issuecomment-637843494,https://api.github.com/repos/simonw/datasette/issues/790,637843494,MDEyOklzc3VlQ29tbWVudDYzNzg0MzQ5NA==,9599,simonw,2020-06-02T22:35:22Z,2020-06-02T22:35:22Z,OWNER,Message CSS is now demonstrated on https://latest.datasette.io/-/patterns,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/794#issuecomment-637841942,https://api.github.com/repos/simonw/datasette/issues/794,637841942,MDEyOklzc3VlQ29tbWVudDYzNzg0MTk0Mg==,9599,simonw,2020-06-02T22:30:17Z,2020-06-02T22:30:17Z,OWNER,Another demo: https://til.simonwillison.net/-/plugins,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629535669,Show hooks implemented by each plugin on /-/plugins,
https://github.com/simonw/datasette/issues/794#issuecomment-637832651,https://api.github.com/repos/simonw/datasette/issues/794,637832651,MDEyOklzc3VlQ29tbWVudDYzNzgzMjY1MQ==,9599,simonw,2020-06-02T22:10:34Z,2020-06-02T22:10:34Z,OWNER,"Demo: https://latest.datasette.io/-/plugins?all=1
```json
[
{
""name"": ""datasette.facets"",
""static"": false,
""templates"": false,
""version"": null,
""hooks"": [
""register_facet_classes""
]
},
{
""name"": ""datasette.publish.cloudrun"",
""static"": false,
""templates"": false,
""version"": null,
""hooks"": [
""publish_subcommand""
]
},
{
""name"": ""datasette.actor_auth_cookie"",
""static"": false,
""templates"": false,
""version"": null,
""hooks"": [
""actor_from_request""
]
},
{
""name"": ""datasette.default_permissions"",
""static"": false,
""templates"": false,
""version"": null,
""hooks"": [
""permission_allowed""
]
},
{
""name"": ""datasette.sql_functions"",
""static"": false,
""templates"": false,
""version"": null,
""hooks"": [
""prepare_connection""
]
},
{
""name"": ""datasette.publish.heroku"",
""static"": false,
""templates"": false,
""version"": null,
""hooks"": [
""publish_subcommand""
]
}
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629535669,Show hooks implemented by each plugin on /-/plugins,
https://github.com/simonw/datasette/issues/699#issuecomment-637819025,https://api.github.com/repos/simonw/datasette/issues/699,637819025,MDEyOklzc3VlQ29tbWVudDYzNzgxOTAyNQ==,9599,simonw,2020-06-02T21:34:31Z,2020-06-02T21:34:31Z,OWNER,I can close this issue once I've expanded out this page of documentation https://datasette.readthedocs.io/en/latest/authentication.html - and published at least one plugin and/or feature that takes advantage of this new mechanism.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/793#issuecomment-637813993,https://api.github.com/repos/simonw/datasette/issues/793,637813993,MDEyOklzc3VlQ29tbWVudDYzNzgxMzk5Mw==,9599,simonw,2020-06-02T21:22:50Z,2020-06-02T21:22:50Z,OWNER,"This is a minor security issue with `master` at the moment, but I'll resolve this before I ship the next release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629524205,CSRF protection for /-/messages tool and writable canned queries,
https://github.com/simonw/datasette/issues/790#issuecomment-637813616,https://api.github.com/repos/simonw/datasette/issues/790,637813616,MDEyOklzc3VlQ29tbWVudDYzNzgxMzYxNg==,9599,simonw,2020-06-02T21:22:02Z,2020-06-02T21:22:02Z,OWNER,"Debug tool is live here: https://latest.datasette.io/-/messages
Documentation is here: https://github.com/simonw/datasette/blob/master/docs/internals.rst#add_messagerequest-message-message_typedatasetteinfo","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-637793590,https://api.github.com/repos/simonw/datasette/issues/790,637793590,MDEyOklzc3VlQ29tbWVudDYzNzc5MzU5MA==,9599,simonw,2020-06-02T20:40:02Z,2020-06-02T20:40:02Z,OWNER,"From https://github.com/simonw/datasette/issues/698#issuecomment-621037724
> Concept for displaying a success message:
>
> ![]()
> CSS:
>
> ```css
> .success {
> padding: 1em;
> border: 1px solid green;
> background-color: #c7fbc7;
> }","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-637790860,https://api.github.com/repos/simonw/datasette/issues/790,637790860,MDEyOklzc3VlQ29tbWVudDYzNzc5MDg2MA==,9599,simonw,2020-06-02T20:34:15Z,2020-06-02T20:34:15Z,OWNER,The `/-/messages` debug tool will need CSRF protection or people will be able to add messages using a hidden form on another website.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-637708090,https://api.github.com/repos/simonw/datasette/issues/790,637708090,MDEyOklzc3VlQ29tbWVudDYzNzcwODA5MA==,9599,simonw,2020-06-02T17:52:30Z,2020-06-02T17:52:30Z,OWNER,I need to make sure that any time cookies are set there's no cache-control header (or it is set to private).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-637699337,https://api.github.com/repos/simonw/datasette/issues/790,637699337,MDEyOklzc3VlQ29tbWVudDYzNzY5OTMzNw==,9599,simonw,2020-06-02T17:34:47Z,2020-06-02T17:34:47Z,OWNER,"I'm going to use a output renderer plugin to test this, since then my unit tests can run against custom code that both sets and displays messages.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-637066496,https://api.github.com/repos/simonw/datasette/issues/790,637066496,MDEyOklzc3VlQ29tbWVudDYzNzA2NjQ5Ng==,9599,simonw,2020-06-01T19:48:20Z,2020-06-01T19:48:20Z,OWNER,"I'm going to stash these on the `request` object after all, so the memory used for the messages gets automatically cleaned up at the end of the request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-637009509,https://api.github.com/repos/simonw/datasette/issues/790,637009509,MDEyOklzc3VlQ29tbWVudDYzNzAwOTUwOQ==,9599,simonw,2020-06-01T17:44:55Z,2020-06-01T17:46:18Z,OWNER,"Problem with `datasette.fetch_and_clear_messages(request, response)` is that I want to call it from the template (so we only clear messages if they have been displayed) - but by that point in the code the `Response` object has not yet been created, so it can't have cookie set on it to clear the list of messages.
Solution: call it `datasette.show_messages(request)` and have it update internal state on the `datasette` object such that a later call to `write_messages_to_response(request, response)` knows to clear the cookie.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636978065,https://api.github.com/repos/simonw/datasette/issues/790,636978065,MDEyOklzc3VlQ29tbWVudDYzNjk3ODA2NQ==,9599,simonw,2020-06-01T16:42:59Z,2020-06-01T17:44:12Z,OWNER,"`datasette.add_message(request, message, type=datasette.INFO)`
Then later:
`datasette.write_messages_to_response(request, response)`
Writes the messages as cookies in the response.
`datasette.fetch_and_clear_messages(request, response)`
To display messages and clears them from the response.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/791#issuecomment-636973355,https://api.github.com/repos/simonw/datasette/issues/791,636973355,MDEyOklzc3VlQ29tbWVudDYzNjk3MzM1NQ==,9599,simonw,2020-06-01T16:33:33Z,2020-06-01T16:33:33Z,OWNER,"A fun thing about this tutorial is that it can start with a classic, basic todo list - and then start growing all kinds of outlandish features to help demonstrate various Datasette plugins and approaches.
Your TODOs on a map. URLs in TODOs that have been unfurled. Tag your TODOs and browse them with facets. Vega graphs showing your progress. Etc etc etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628572716,Tutorial: building a something-interesting with writable canned queries,
https://github.com/simonw/datasette/issues/790#issuecomment-636959774,https://api.github.com/repos/simonw/datasette/issues/790,636959774,MDEyOklzc3VlQ29tbWVudDYzNjk1OTc3NA==,9599,simonw,2020-06-01T16:15:33Z,2020-06-01T16:15:33Z,OWNER,It would be neat if this was driven by a method on `datasette` just because that's already the object passed to plugins as a documented API.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636934016,https://api.github.com/repos/simonw/datasette/issues/790,636934016,MDEyOklzc3VlQ29tbWVudDYzNjkzNDAxNg==,9599,simonw,2020-06-01T15:49:26Z,2020-06-01T15:49:26Z,OWNER,"Flask and Django both support ""types"" of message - info, warning etc. I think I should do the same.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636925354,https://api.github.com/repos/simonw/datasette/issues/790,636925354,MDEyOklzc3VlQ29tbWVudDYzNjkyNTM1NA==,9599,simonw,2020-06-01T15:32:02Z,2020-06-01T15:32:02Z,OWNER,"If `scope` had an immutable correlation ID I could use that with a dict somewhere mapping `correlation_id` to a messages object.
The problem then is how do I know to clean up the memory used by that dictionary when the request flows out of the system? I guess the code that updates the cookies in the response could do that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636922104,https://api.github.com/repos/simonw/datasette/issues/790,636922104,MDEyOklzc3VlQ29tbWVudDYzNjkyMjEwNA==,9599,simonw,2020-06-01T15:25:39Z,2020-06-01T15:25:39Z,OWNER,"What if I use a mutable key on `scope` to track messages for the duration of the request? Is that an OK thing to do?
ASGI spec says this: https://asgi.readthedocs.io/en/latest/specs/main.html#middleware
> ### Middleware
>
> It is possible to have ASGI ""middleware"" - code that plays the role of both server and application, taking in a scope and the send/receive awaitables, potentially modifying them, and then calling an inner application.
>
> When middleware is modifying the scope, it should make a copy of the scope object before mutating it and passing it to the inner application, as changes may leak upstream otherwise. In particular, you should not assume that the copy of the scope you pass down to the application is the one that it ends up using, as there may be other middleware in the way; thus, do not keep a reference to it and try to mutate it outside of the initial ASGI constructor callable that receives `scope`. Your one and only chance to add to it is before you hand control to the child application.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636920304,https://api.github.com/repos/simonw/datasette/issues/790,636920304,MDEyOklzc3VlQ29tbWVudDYzNjkyMDMwNA==,9599,simonw,2020-06-01T15:22:15Z,2020-06-01T15:22:15Z,OWNER,"Here's how the Django stuff works: https://github.com/django/django/blob/master/django/contrib/messages/storage/base.py
Notably the messages are mostly dealt with on the request object, with a piece of middleware that reads from the request and modifies the response (to set or clear cookies) right at the end: https://github.com/django/django/blob/master/django/contrib/messages/middleware.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636916107,https://api.github.com/repos/simonw/datasette/issues/790,636916107,MDEyOklzc3VlQ29tbWVudDYzNjkxNjEwNw==,9599,simonw,2020-06-01T15:14:30Z,2020-06-01T15:15:52Z,OWNER,"Alternative: `datasette.add_message(message)` and `datasette.read_and_clear_messages()` - these would need some kind of dark magic to ensure that the message was associated with the current request flowing through the system though, since that `datasette` object is shared my multiple concurrent requests.
Maybe use a request correlation ID that gets added to the scope? This is all getting a bit messy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636915499,https://api.github.com/repos/simonw/datasette/issues/790,636915499,MDEyOklzc3VlQ29tbWVudDYzNjkxNTQ5OQ==,9599,simonw,2020-06-01T15:13:40Z,2020-06-01T15:13:40Z,OWNER,"Maybe two utility functions:
`add_message(request, message)` - adds a Flash message (will be set later)
`read_and_clear_messages(request)` - reads messages and sets them to be cleared
Problem: the `request` object isn't created at the very top of the stack - it's actually created within each view. So maybe I need to move its creation up to the top of the routing stuff so that the code that returns the response can see it?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636912730,https://api.github.com/repos/simonw/datasette/issues/790,636912730,MDEyOklzc3VlQ29tbWVudDYzNjkxMjczMA==,9599,simonw,2020-06-01T15:08:13Z,2020-06-01T15:08:13Z,OWNER,"I'm going to build the first version of this with signed cookies.
I'm inclined to do this all on the request object, since it's the object representing the current request as it flows through the application. I need the ability to remember which messages were set and which need to be cleared, so I need to do that on something that is available for the lifetime of the request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636908972,https://api.github.com/repos/simonw/datasette/issues/790,636908972,MDEyOklzc3VlQ29tbWVudDYzNjkwODk3Mg==,9599,simonw,2020-06-01T15:01:00Z,2020-06-01T15:01:00Z,OWNER,"Setting messages just needs access to the response. Reading messages needs access to both request AND response, since it needs to clear the messages that are being displayed.
That's if the messages are persisted exclusively in cookies - which makes sense for Django since it's designed to run as many different load-balanced processes.
Since Datasette is a single process which can access an on-file database, maybe consider storing the flash messages within Datasette memory itself - a sort of session mechanism?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636906773,https://api.github.com/repos/simonw/datasette/issues/790,636906773,MDEyOklzc3VlQ29tbWVudDYzNjkwNjc3Mw==,9599,simonw,2020-06-01T14:57:02Z,2020-06-01T14:58:14Z,OWNER,"Actually I'm inclined to use cookies now, ala Django: https://docs.djangoproject.com/en/3.0/ref/contrib/messages/
> This class stores the message data in a cookie (signed with a secret hash to prevent manipulation) to persist notifications across requests. Old messages are dropped if the cookie data size would exceed 2048 bytes.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/790#issuecomment-636906581,https://api.github.com/repos/simonw/datasette/issues/790,636906581,MDEyOklzc3VlQ29tbWVudDYzNjkwNjU4MQ==,9599,simonw,2020-06-01T14:56:42Z,2020-06-01T14:56:42Z,OWNER,I can use the new signed values support from #785 to help build this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628499086,"""flash messages"" mechanism",
https://github.com/simonw/datasette/issues/698#issuecomment-636617140,https://api.github.com/repos/simonw/datasette/issues/698,636617140,MDEyOklzc3VlQ29tbWVudDYzNjYxNzE0MA==,9599,simonw,2020-06-01T05:14:39Z,2020-06-01T05:14:39Z,OWNER,Here's the new `default_permissions.py` file I can add this permission check to: https://github.com/simonw/datasette/blob/dfdbdf378aba9afb66666f66b78df2f2069d2595/datasette/default_permissions.py#L1-L7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/788#issuecomment-636616638,https://api.github.com/repos/simonw/datasette/issues/788,636616638,MDEyOklzc3VlQ29tbWVudDYzNjYxNjYzOA==,9599,simonw,2020-06-01T05:12:30Z,2020-06-01T05:12:30Z,OWNER,"Looks like this (at the moment):
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628121234, /-/permissions debugging tool,
https://github.com/simonw/datasette/issues/789#issuecomment-636616307,https://api.github.com/repos/simonw/datasette/issues/789,636616307,MDEyOklzc3VlQ29tbWVudDYzNjYxNjMwNw==,9599,simonw,2020-06-01T05:11:03Z,2020-06-01T05:11:03Z,OWNER,Or I could get fancy and implement my own `pm.trace.root.setwriter` function which collects data that can be appended to the `?_trace=1` dump.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628156527,Mechanism for enabling pluggy tracing,
https://github.com/simonw/datasette/issues/789#issuecomment-636616155,https://api.github.com/repos/simonw/datasette/issues/789,636616155,MDEyOklzc3VlQ29tbWVudDYzNjYxNjE1NQ==,9599,simonw,2020-06-01T05:10:27Z,2020-06-01T05:10:27Z,OWNER,Easiest way to do this would be with an environment variable.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628156527,Mechanism for enabling pluggy tracing,
https://github.com/simonw/datasette/issues/786#issuecomment-636614062,https://api.github.com/repos/simonw/datasette/issues/786,636614062,MDEyOklzc3VlQ29tbWVudDYzNjYxNDA2Mg==,9599,simonw,2020-06-01T05:02:18Z,2020-06-01T05:02:18Z,OWNER,The skeleton of this page now exists at https://datasette.readthedocs.io/en/latest/authentication.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628087971,Documentation page describing Datasette's authentication system,
https://github.com/simonw/datasette/issues/788#issuecomment-636598949,https://api.github.com/repos/simonw/datasette/issues/788,636598949,MDEyOklzc3VlQ29tbWVudDYzNjU5ODk0OQ==,9599,simonw,2020-06-01T03:53:00Z,2020-06-01T03:53:00Z,OWNER,I can use a deque with a max length for this: https://docs.python.org/3/library/collections.html#deque-objects,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628121234, /-/permissions debugging tool,
https://github.com/simonw/datasette/issues/699#issuecomment-636576603,https://api.github.com/repos/simonw/datasette/issues/699,636576603,MDEyOklzc3VlQ29tbWVudDYzNjU3NjYwMw==,9599,simonw,2020-06-01T02:13:26Z,2020-06-01T03:13:31Z,OWNER,"Debugging tool idea: `/-/permissions` page which shows you the actor and lets you type in the strings for `action`, `resource_type` and `resource_identifier` - then shows you EVERY plugin hook that would have executed and what it would have said, plus when the chain would have terminated.
Bonus: if you're logged in as the `root` user (or a user that matches some kind of permission check, maybe a check for `permissions_debug`) you get to see a rolling log of the last 30 permission checks and what the results were across the whole of Datasette. This should make figuring out permissions policies a whole lot easier.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636576252,https://api.github.com/repos/simonw/datasette/issues/699,636576252,MDEyOklzc3VlQ29tbWVudDYzNjU3NjI1Mg==,9599,simonw,2020-06-01T02:11:40Z,2020-06-01T02:11:40Z,OWNER,"Plugin idea: `datasette-allow-all` - really simple plugin which just says ""yes"" to every permission check.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/698#issuecomment-636569917,https://api.github.com/repos/simonw/datasette/issues/698,636569917,MDEyOklzc3VlQ29tbWVudDYzNjU2OTkxNw==,9599,simonw,2020-06-01T01:39:44Z,2020-06-01T01:39:44Z,OWNER,"Idea for the authentication piece: I'll have the canned query code execute the following:
```python
if await datasette.permission_allowed(
request.scope.get(""actor""), ""execute_query"", ""canned_query"", query_name, default=True
):
```
Then I'll add a default plugin to Datasette which implements that plugin hook, looks at the Datasette metadata for that query, and says ""No"" if the following (and `request.scope[""actor""]` is empty):
```json
{
""databases"": {
""my-database"": {
""queries"": {
""add_twitter_handle"": {
""sql"": ""insert into twitter_handles (username) values (:username)"",
""write"": true,
""requires_actor"": true
}
}
}
}
}
```
I think I'll support this too:
```json
""allowed_actors"": [""root""]
```
So you can configure queries to only be available to specific `{""id"": xxx}` actors.
This will be the first time the new `permission_allowed` mechanism from #699 will be exercised in Datasette core.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/699#issuecomment-636566616,https://api.github.com/repos/simonw/datasette/issues/699,636566616,MDEyOklzc3VlQ29tbWVudDYzNjU2NjYxNg==,9599,simonw,2020-06-01T01:23:48Z,2020-06-01T01:23:48Z,OWNER,https://latest.datasette.io/-/actor is now live (it returns `null` because there's no current way to sign into the `latest.datasette.io` site - not even with a fake `ds_actor` cookie because there's no way to know what that site's random secret is).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636566433,https://api.github.com/repos/simonw/datasette/issues/699,636566433,MDEyOklzc3VlQ29tbWVudDYzNjU2NjQzMw==,9599,simonw,2020-06-01T01:22:59Z,2020-06-01T01:22:59Z,OWNER,"Some next steps:
- Try out a branch of `datasette-auth-github` that builds on these new plugin hooks
- Build a `datasette-api-tokens` plugin which implements `Authorization: bearer xxx` token support for API access
- Maybe prototype up a `datasette-user-accounts` plugin which supports username/password accounts and allows an admin user to create/delete them
- Do more work on writable canned queries in #698 and see what they look like if they take advantage of the permissions hook (to restrict some to only allowing authenticated users)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636565610,https://api.github.com/repos/simonw/datasette/issues/699,636565610,MDEyOklzc3VlQ29tbWVudDYzNjU2NTYxMA==,9599,simonw,2020-06-01T01:19:45Z,2020-06-01T01:19:45Z,OWNER,I rebased in #783 so all of this is on master now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/784#issuecomment-636565242,https://api.github.com/repos/simonw/datasette/issues/784,636565242,MDEyOklzc3VlQ29tbWVudDYzNjU2NTI0Mg==,9599,simonw,2020-06-01T01:18:20Z,2020-06-01T01:18:20Z,OWNER,I'm considering this done. I'm going to leave it to plugins to implement a web-based sign-in flow for accounts (at least for the moment).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628003707,Ability to sign in to Datasette as a root account,
https://github.com/simonw/datasette/issues/699#issuecomment-636562999,https://api.github.com/repos/simonw/datasette/issues/699,636562999,MDEyOklzc3VlQ29tbWVudDYzNjU2Mjk5OQ==,9599,simonw,2020-06-01T01:09:47Z,2020-06-01T01:09:47Z,OWNER,I should add an entire page to the documentation describing Datasette authentication.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636562658,https://api.github.com/repos/simonw/datasette/issues/699,636562658,MDEyOklzc3VlQ29tbWVudDYzNjU2MjY1OA==,9599,simonw,2020-06-01T01:08:20Z,2020-06-01T01:08:54Z,OWNER,"OK, the implementation in PR #783 is in a good state now - it implements the new plugin hooks with tests and documentation, plus it implements this:
$ datasette . --root
http://127.0.0.1:8001/-/auth-token?token=3ca9ee460a6451142389351d19b147bce27d2a785dfb6b5a74f82211be1ede49
...
That URL, when clicked, will set a cookie for the `{""id"": ""root""}` user. The cookie is respected and used to populate `scope[""actor""]`.
I'm going to merge that pull request and continue working on this stuff on master.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/784#issuecomment-636554258,https://api.github.com/repos/simonw/datasette/issues/784,636554258,MDEyOklzc3VlQ29tbWVudDYzNjU1NDI1OA==,9599,simonw,2020-06-01T00:21:33Z,2020-06-01T00:21:33Z,OWNER,"The URL for this will be:
`/-/auth-token?token=xxx`
The token will be generated by Datasette on startup and will only be valid for a single request, at which point it will be used to set a signed `ds_actor` cookie and then redirect to the homepage.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628003707,Ability to sign in to Datasette as a root account,
https://github.com/simonw/datasette/issues/785#issuecomment-636553736,https://api.github.com/repos/simonw/datasette/issues/785,636553736,MDEyOklzc3VlQ29tbWVudDYzNjU1MzczNg==,9599,simonw,2020-06-01T00:18:40Z,2020-06-01T00:18:40Z,OWNER,That documentation: https://github.com/simonw/datasette/blob/c818de88a9c2683437875f788e325d911c8b767b/docs/config.rst#configuring-the-secret,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636541827,https://api.github.com/repos/simonw/datasette/issues/785,636541827,MDEyOklzc3VlQ29tbWVudDYzNjU0MTgyNw==,9599,simonw,2020-05-31T22:46:34Z,2020-06-01T00:17:35Z,OWNER,This is nearly ready to close. I'm going to add documentation for `--secret` and the `DATASETTE_SECRET` environment variable.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636541929,https://api.github.com/repos/simonw/datasette/issues/785,636541929,MDEyOklzc3VlQ29tbWVudDYzNjU0MTkyOQ==,9599,simonw,2020-05-31T22:47:17Z,2020-05-31T22:47:17Z,OWNER,I'll add a section about secrets to this page: https://datasette.readthedocs.io/en/latest/config.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636541630,https://api.github.com/repos/simonw/datasette/issues/785,636541630,MDEyOklzc3VlQ29tbWVudDYzNjU0MTYzMA==,9599,simonw,2020-05-31T22:45:07Z,2020-05-31T22:45:07Z,OWNER,Documentation for those new methods: https://github.com/simonw/datasette/blob/e28207e76ec3b26b2c396370fd3fb325a60bfd49/docs/internals.rst#signvalue-namespacedefault,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636539295,https://api.github.com/repos/simonw/datasette/issues/785,636539295,MDEyOklzc3VlQ29tbWVudDYzNjUzOTI5NQ==,9599,simonw,2020-05-31T22:24:14Z,2020-05-31T22:28:27Z,OWNER,"I'll add two utility methods to the Datasette class:
- `datasette.sign(value, ""namespace"")` - returns signed string
- `datasette.unsign(signed, ""namespace"")` - returns value OR raises `BadSignature`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636538298,https://api.github.com/repos/simonw/datasette/issues/785,636538298,MDEyOklzc3VlQ29tbWVudDYzNjUzODI5OA==,9599,simonw,2020-05-31T22:14:43Z,2020-05-31T22:15:01Z,OWNER,"... actually no I'll do it using a CLI option that can also be in an environment variable:
https://click.palletsprojects.com/en/7.x/options/#values-from-environment-variables
```python
@click.command()
@click.option('--secret', envvar='DATASETTE_SECRET')
def greet(secret):
...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636537921,https://api.github.com/repos/simonw/datasette/issues/785,636537921,MDEyOklzc3VlQ29tbWVudDYzNjUzNzkyMQ==,9599,simonw,2020-05-31T22:11:29Z,2020-05-31T22:11:29Z,OWNER,First version of cookie signing will use a secret that is either pulled from `DATASETTE_SECRET` environment variable or generated every time the server starts. I'll add a non-environment-variable based secret later.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636537679,https://api.github.com/repos/simonw/datasette/issues/785,636537679,MDEyOklzc3VlQ29tbWVudDYzNjUzNzY3OQ==,9599,simonw,2020-05-31T22:09:23Z,2020-05-31T22:09:23Z,OWNER,"I'm going to use https://github.com/pallets/itsdangerous for this.
Annoyingly they're very close to release v2.0 which adds support for key rotation... but it's not quite out of pre-release yet. I'll go with 1.1.0 for the moment and upgrade to 2.0 as soon as that is out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636515763,https://api.github.com/repos/simonw/datasette/issues/785,636515763,MDEyOklzc3VlQ29tbWVudDYzNjUxNTc2Mw==,9599,simonw,2020-05-31T19:19:03Z,2020-05-31T19:19:13Z,OWNER,Maybe Datasette should have a `--secrets=path/to/secrets.json` command-line option for storing these?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636515671,https://api.github.com/repos/simonw/datasette/issues/785,636515671,MDEyOklzc3VlQ29tbWVudDYzNjUxNTY3MQ==,9599,simonw,2020-05-31T19:18:18Z,2020-05-31T19:18:18Z,OWNER,That `user_state_dir` solution may have been more trouble than it was worth though - I seem to remember it causing issues on some hosting providers.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/785#issuecomment-636515599,https://api.github.com/repos/simonw/datasette/issues/785,636515599,MDEyOklzc3VlQ29tbWVudDYzNjUxNTU5OQ==,9599,simonw,2020-05-31T19:17:43Z,2020-05-31T19:17:43Z,OWNER,"I previously solved this for the `datasette-auth-existing-cookies` plugin as described in this issue: https://github.com/simonw/datasette-auth-existing-cookies/issues/1
> Concrete plan: you have to pass a secret to the class constructor. The Datasette plugin (the code in `__init__.py`) uses the following in order of preference (first things are most preferred):
>
> - A plugin configuration option called `cookie_secret` - which can be protected by this mechanism: https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values
> - A JSON configuration file in the `user_state_dir` file, if it exists
> - If that does not exist, a secret is generated and written to that JSON file
>
> I originally planned to have separate support for an environment variable, but the existence of the [secret configuration values](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values) mechanism means this is already handled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628025100,Datasette secret mechanism - initially for signed cookies,
https://github.com/simonw/datasette/issues/784#issuecomment-636514974,https://api.github.com/repos/simonw/datasette/issues/784,636514974,MDEyOklzc3VlQ29tbWVudDYzNjUxNDk3NA==,9599,simonw,2020-05-31T19:12:48Z,2020-05-31T19:12:48Z,OWNER,"For the first version of this I'm not going to use passwords at all. I'll implement this:
$ datasette fixtures.db --root
The `--root` option will cause Datasette to output a URL with a one-time-use token in it which, when clicked, will authenticate the user as the root account (by setting a signed cookie).
Signed cookie means Datasette needs a secrets recipe. I'll open a new issue for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628003707,Ability to sign in to Datasette as a root account,
https://github.com/simonw/datasette/issues/784#issuecomment-636510838,https://api.github.com/repos/simonw/datasette/issues/784,636510838,MDEyOklzc3VlQ29tbWVudDYzNjUxMDgzOA==,9599,simonw,2020-05-31T18:39:08Z,2020-05-31T18:39:08Z,OWNER,"I'm calling this the `root` account now, for reasons discussed in these two comments: https://github.com/simonw/datasette/issues/699#issuecomment-636510647","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628003707,Ability to sign in to Datasette as a root account,
https://github.com/simonw/datasette/issues/699#issuecomment-636510761,https://api.github.com/repos/simonw/datasette/issues/699,636510761,MDEyOklzc3VlQ29tbWVudDYzNjUxMDc2MQ==,9599,simonw,2020-05-31T18:38:30Z,2020-05-31T18:38:30Z,OWNER,"I quite like `root` - it supports the idea that best practice is to NOT do things as the root account, but to use a plugin to set up separate accounts for different purposes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636510647,https://api.github.com/repos/simonw/datasette/issues/699,636510647,MDEyOklzc3VlQ29tbWVudDYzNjUxMDY0Nw==,9599,simonw,2020-05-31T18:37:39Z,2020-05-31T18:37:39Z,OWNER,Maybe the default single account should be called something other than `admin`? The problem with `admin` is that it sounds like more of a role - in larger installations one can expect multiple admins. `root` may be better since there's clearly only one root account. Bit of a technical term though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636510398,https://api.github.com/repos/simonw/datasette/issues/699,636510398,MDEyOklzc3VlQ29tbWVudDYzNjUxMDM5OA==,9599,simonw,2020-05-31T18:35:57Z,2020-05-31T18:36:05Z,OWNER,Again I will use exploratory prototyping to inform a decision on the minimum subset design for the `actor` dictionary.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636510303,https://api.github.com/repos/simonw/datasette/issues/699,636510303,MDEyOklzc3VlQ29tbWVudDYzNjUxMDMwMw==,9599,simonw,2020-05-31T18:35:17Z,2020-05-31T18:35:17Z,OWNER,"Keeping the structure of the actor dictionary completely undefined doesn't make sense if Datasette is going to ship with a default authentication mechanism for admin users.
I'm going to define a small set of required keys for the actor dictionary, and enforce them in code.
But which keys? I feel I need a unique key representing the identity of the actor, plus a key that can be displayed in the ""You are logged in as X"" navigation. Maybe these are the same key?
So the single required key could be `id`. Problem is: is that a string or an integer? Some use-cases may call for an integer, which matches to how SQLite auto incrementing primary keys work. `admin` is a string.
Maybe `id` is required, `name` is optional - but if `name` is present then the ""You are logged in as..."" uses that in preference to `id`. `id` has to be a string, and if you want to store integer IDs in your database you need to remember to convert them to a string in your `actor_from_request` implementation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636498913,https://api.github.com/repos/simonw/datasette/issues/699,636498913,MDEyOklzc3VlQ29tbWVudDYzNjQ5ODkxMw==,9599,simonw,2020-05-31T17:04:50Z,2020-05-31T17:06:40Z,OWNER,"This also means some writable canned queries can allow writes from unauthenticated users (for stuff like feedback forms), while others can require an authenticated user - all with core Datasette without any plugins needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636499075,https://api.github.com/repos/simonw/datasette/issues/699,636499075,MDEyOklzc3VlQ29tbWVudDYzNjQ5OTA3NQ==,9599,simonw,2020-05-31T17:06:09Z,2020-05-31T17:06:09Z,OWNER,"I believe that this plugin hook design is flexible enough that role-based permissions could be built on top of it as a separate plugin.
Would be good to check that with a proof of concept though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636498770,https://api.github.com/repos/simonw/datasette/issues/699,636498770,MDEyOklzc3VlQ29tbWVudDYzNjQ5ODc3MA==,9599,simonw,2020-05-31T17:03:38Z,2020-05-31T17:03:38Z,OWNER,"I'm going to draw the line here: default Datasette supports authentication but only for a single user account (""admin""). Plugins can then add support for multiple user accounts, social auth, SSO etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636495124,https://api.github.com/repos/simonw/datasette/issues/699,636495124,MDEyOklzc3VlQ29tbWVudDYzNjQ5NTEyNA==,9599,simonw,2020-05-31T16:36:08Z,2020-05-31T16:36:08Z,OWNER,HTTP Basic auth would be a good default option. No need to build a custom login UI for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636495005,https://api.github.com/repos/simonw/datasette/issues/699,636495005,MDEyOklzc3VlQ29tbWVudDYzNjQ5NTAwNQ==,9599,simonw,2020-05-31T16:35:10Z,2020-05-31T16:35:26Z,OWNER,I think I want to keep full username/password authentication against a database table as a plugin. I'll experiment with Jupyter-style URLs as a starting point.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636494374,https://api.github.com/repos/simonw/datasette/issues/699,636494374,MDEyOklzc3VlQ29tbWVudDYzNjQ5NDM3NA==,9599,simonw,2020-05-31T16:29:48Z,2020-05-31T16:29:48Z,OWNER,"If Datasette were to support authentication out-of-the-box, without plugins (which makes more sense with writable canned queries, #698) what would that look like?
Some options:
- Jupyter notebook style: output a magic URL on the console with a one-time token to authenticate the user as an ""admin""
- Really simple password authentication - via an environment variable perhaps?
- SQL based authentication: I was going to do this as a plugin, but maybe it should be default? A way of configuring a SQL query which can be used to authenticate a user based on their username and password.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636395263,https://api.github.com/repos/simonw/datasette/issues/699,636395263,MDEyOklzc3VlQ29tbWVudDYzNjM5NTI2Mw==,9599,simonw,2020-05-30T22:54:09Z,2020-05-30T22:54:09Z,OWNER,"Idea: add a `/-/actor.json` special page which JSON dumps out the current `request.scope[""actor""]` - so you can easily test how your request has been authenticated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636393204,https://api.github.com/repos/simonw/datasette/issues/699,636393204,MDEyOklzc3VlQ29tbWVudDYzNjM5MzIwNA==,9599,simonw,2020-05-30T22:29:44Z,2020-05-30T22:30:15Z,OWNER,"Robust testing of permissions is really important. I should think about utilities I may be able to add to Datasette's unit testing tools that make it as easy as possible to confirm which permission checks were carried out on a specific HTTP request.
That way I can set a good example that any Datasette plugin which makes permission checks can follow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636392850,https://api.github.com/repos/simonw/datasette/issues/699,636392850,MDEyOklzc3VlQ29tbWVudDYzNjM5Mjg1MA==,9599,simonw,2020-05-30T22:25:19Z,2020-05-30T22:25:19Z,OWNER,The branch is now usable! Next step: write some experimental plugins that exercise some real authentication use-cases with it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636391331,https://api.github.com/repos/simonw/datasette/issues/699,636391331,MDEyOklzc3VlQ29tbWVudDYzNjM5MTMzMQ==,9599,simonw,2020-05-30T22:08:21Z,2020-05-30T22:08:21Z,OWNER,"I'm going to add an awaitable utility method to the Datasette class for checking permissions:
await datasette.permission_allowed(actor, action, resource_type, resource_identifier)
The second two arguments will be optional.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636388288,https://api.github.com/repos/simonw/datasette/issues/699,636388288,MDEyOklzc3VlQ29tbWVudDYzNjM4ODI4OA==,9599,simonw,2020-05-30T21:34:50Z,2020-05-30T21:34:50Z,OWNER,"Debugging permissions is going to be important. Optional tooling that supports the following would be useful:
- Log every check to `permission_allowed` to the console - optionally with tracebacks showing where in the code the check was made
- Log every check to the https://latest.datasette.io/?_trace=1 output
- A tool that shows you exactly what permissions the current authenticated user/entity has
- A tool showing all available permissions
That last one is tricky if permissions are just strings that might be passed to `permission_allowed` - so maybe there needs to be a plugin hook that lets plugins register their permissions, such that they can be introspected later on? A `register_permission_actions()` hook that returns a list of permission action strings (or objects of some sort) perhaps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636379067,https://api.github.com/repos/simonw/datasette/issues/699,636379067,MDEyOklzc3VlQ29tbWVudDYzNjM3OTA2Nw==,9599,simonw,2020-05-30T20:12:47Z,2020-05-30T20:40:42Z,OWNER,"I could bake some permission checks into default Datasette, which are all treated as allow by default but can then be locked down by plugins. Maybe the following:
permission_allowed(request.actor, ""execute-sql"", ""database"", ""name-of-database"")
Checks that current user can execute arbitrary SQL queries against a specific database (or use the `?_where=` feature). Equivalent to current [allow_sql](https://datasette.readthedocs.io/en/0.43/config.html#allow-sql) setting.
permission_allowed(request.actor, ""download-database"", ""database"", ""name-of-database"")
Can the user download the database file? Like [allow_download](https://datasette.readthedocs.io/en/0.43/config.html#allow-download).
Maybe one for [allow_csv_stream](https://datasette.readthedocs.io/en/0.43/config.html#allow-csv-stream) too.
Having a permission check (defaulting to True) on every single ""view"" would be useful:
- view_index
- view_database
- view_table
- view_row
- view_query
- view_special (for `/-/versions` and so on)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636381732,https://api.github.com/repos/simonw/datasette/issues/699,636381732,MDEyOklzc3VlQ29tbWVudDYzNjM4MTczMg==,9599,simonw,2020-05-30T20:32:11Z,2020-05-30T20:39:11Z,OWNER,I started sketching this out in the [authentication](https://github.com/simonw/datasette/tree/authentication) branch. Here's the documentation so far: https://github.com/simonw/datasette/blob/8871c20/docs/plugins.rst#actor_from_requestdatasette-request,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636376209,https://api.github.com/repos/simonw/datasette/issues/699,636376209,MDEyOklzc3VlQ29tbWVudDYzNjM3NjIwOQ==,9599,simonw,2020-05-30T19:53:28Z,2020-05-30T20:09:10Z,OWNER,"I think there are two hooks here:
`actor_from_request(datasette, request)` - returns `None` or a dictionary.
- `datasette` is a Datasette instance - useful for things like reading plugin configuration or executing queries
- `request` is a [Request object](https://datasette.readthedocs.io/en/latest/internals.html#request-object) - which means ASGI scope can be accessed as `request.scope`
A non-None value means the request is authenticated in some way. The shape of that dictionary is entirely undefined.
The second hook is for checking permissions. It can look something like this:
`permission_allowed(actor, action, resource_type, resource_identifier)`
- `actor` = the dictionary that was returned by `actor_from_scope`
- `action` = a string representing the action to be performed, e.g. `edit-schema`
- `resource_type` = a string representing the type of resource being acted on, e.g. `table`
- `resource_identifier` = a string (or maybe tuple?) representing the specific resource, e.g. the table name
I don't know if Datasette should provide default implementations of these hooks. It may be that leaving them completely up to plugins is the way to go.
I think I need to prototype this quickly to start feeling for how well it might work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636376893,https://api.github.com/repos/simonw/datasette/issues/699,636376893,MDEyOklzc3VlQ29tbWVudDYzNjM3Njg5Mw==,9599,simonw,2020-05-30T19:57:54Z,2020-05-30T20:09:05Z,OWNER,"`auth_from_scope(datasette, scope)` needs to be able to return an awaitable which is then awaited - so it can execute database queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636376974,https://api.github.com/repos/simonw/datasette/issues/699,636376974,MDEyOklzc3VlQ29tbWVudDYzNjM3Njk3NA==,9599,simonw,2020-05-30T19:58:40Z,2020-05-30T20:08:59Z,OWNER,"Maybe call that `actor_from_request(datasette, request)` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636378228,https://api.github.com/repos/simonw/datasette/issues/699,636378228,MDEyOklzc3VlQ29tbWVudDYzNjM3ODIyOA==,9599,simonw,2020-05-30T20:07:25Z,2020-05-30T20:07:25Z,OWNER,"I like ""actor"" better than ""entity"" to mean ""the user or API key that is authenticated for this request"".
I'm going to use ""resource"" instead of ""subject"" - updating the design comment again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636378121,https://api.github.com/repos/simonw/datasette/issues/699,636378121,MDEyOklzc3VlQ29tbWVudDYzNjM3ODEyMQ==,9599,simonw,2020-05-30T20:06:47Z,2020-05-30T20:06:47Z,OWNER,"In AWS IAM world the following terminology is used: https://aws.amazon.com/iam/features/manage-permissions/
> Permissions are granted to IAM **entities** (users, groups, and roles) [...]
>
> To assign permissions to a user, group, role, or resource, you create a policy that lets you specify:
>
> * **Actions** – Which AWS service actions you allow. For example, you might allow a user to call the Amazon S3 ListBucket action. Any actions that you don't explicitly allow are denied.
> * **Resources** – Which AWS resources you allow the action on. For example, what Amazon S3 buckets will you allow the user to perform the ListBucket action on? Users cannot access any resources that you do not explicitly grant permissions to.
> * **Effect** – Whether to allow or deny access. Because access is denied by default, you typically write policies where the effect is to allow.
> * **Conditions** – Which conditions must be present for the policy to take effect. For example, you might allow access only to the specific S3 buckets if the user is connecting from a specific IP range or has used multi-factor authentication at login.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636377755,https://api.github.com/repos/simonw/datasette/issues/699,636377755,MDEyOklzc3VlQ29tbWVudDYzNjM3Nzc1NQ==,9599,simonw,2020-05-30T20:04:23Z,2020-05-30T20:04:23Z,OWNER,"My usage of the term `subject` here to mean ""the thing I am checking I have permission to interact with, e.g. a database table"" may be misleading. https://stackoverflow.com/questions/4989063/what-is-the-meaning-and-difference-between-subject-user-and-principal for example shows that JAAS (Java Authentication and Authorization Service) defines subject as ""The purpose of the Subject is to represent the authenticated user"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-636377235,https://api.github.com/repos/simonw/datasette/issues/699,636377235,MDEyOklzc3VlQ29tbWVudDYzNjM3NzIzNQ==,9599,simonw,2020-05-30T20:00:42Z,2020-05-30T20:01:35Z,OWNER,I'm changing `auth` to `actor` and updating the above [design comment](https://github.com/simonw/datasette/issues/699#issuecomment-636376209).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/782#issuecomment-636370064,https://api.github.com/repos/simonw/datasette/issues/782,636370064,MDEyOklzc3VlQ29tbWVudDYzNjM3MDA2NA==,9599,simonw,2020-05-30T18:51:19Z,2020-05-30T18:51:19Z,OWNER,"https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_shape=array returns this:
```json
[
{
""pk1"": ""a"",
""pk2"": ""a"",
""pk3"": ""a"",
""content"": ""a-a-a""
},
{
""pk1"": ""a"",
""pk2"": ""a"",
""pk3"": ""b"",
""content"": ""a-a-b""
}
]
```
There's one big problem with this format: it doesn't provide any space for pagination information.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-636369978,https://api.github.com/repos/simonw/datasette/issues/782,636369978,MDEyOklzc3VlQ29tbWVudDYzNjM2OTk3OA==,9599,simonw,2020-05-30T18:50:31Z,2020-05-30T18:50:31Z,OWNER,"Here's the default JSON at the moment: https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2
```json
{
""database"": ""fixtures"",
""table"": ""compound_three_primary_keys"",
""is_view"": false,
""human_description_en"": """",
""rows"": [
[
""a"",
""a"",
""a"",
""a-a-a""
],
[
""a"",
""a"",
""b"",
""a-a-b""
]
],
""truncated"": false,
""filtered_table_rows_count"": 1001,
""expanded_columns"": [],
""expandable_columns"": [],
""columns"": [
""pk1"",
""pk2"",
""pk3"",
""content""
],
""primary_keys"": [
""pk1"",
""pk2"",
""pk3""
],
""units"": {},
""query"": {
""sql"": ""select pk1, pk2, pk3, content from compound_three_primary_keys order by pk1, pk2, pk3 limit 3"",
""params"": {}
},
""facet_results"": {},
""suggested_facets"": [
{
""name"": ""pk1"",
""toggle_url"": ""http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_facet=pk1""
},
{
""name"": ""pk2"",
""toggle_url"": ""http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_facet=pk2""
},
{
""name"": ""pk3"",
""toggle_url"": ""http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_facet=pk3""
}
],
""next"": ""a,a,b"",
""next_url"": ""http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_next=a%2Ca%2Cb"",
""query_ms"": 17.56119728088379,
""source"": ""tests/fixtures.py"",
""source_url"": ""https://github.com/simonw/datasette/blob/master/tests/fixtures.py"",
""license"": ""Apache License 2.0"",
""license_url"": ""https://github.com/simonw/datasette/blob/master/LICENSE""
}
```
There's a lot of stuff in there. This increases the risk that future minor changes might break existing API consumers.
It returns rows as a list of lists of values, and expects you to correlate these with the list of columns. I originally designed it like this because I thought this was a more efficient representation than repeating the column names in a dictionary for every row. With hindsight this was a bad optimization - I _always_ use `?shape=array` because it's more convenient, and gzip encoding of the response means there's no bandwidth saving. Users who want that efficiency should request it using a custom `?_shape=`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/576#issuecomment-636340618,https://api.github.com/repos/simonw/datasette/issues/576,636340618,MDEyOklzc3VlQ29tbWVudDYzNjM0MDYxOA==,9599,simonw,2020-05-30T14:46:04Z,2020-05-30T18:41:41Z,OWNER,"I should also think about the class properties (as opposed to methods) that are setup in the Datasette constructor. Many of these should be private, some should be documented.
- cache_headers
- cors
- databases
- executor
- files
- immutables
- inspect_data
- jinja_env
- max_returned_rows
- page_size
- plugins_dir
- renderers
- sql_time_limit_ms
- sqlite_extensions
- sqlite_functions
- static_mounts
- template_dir
- version_note
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/37#issuecomment-636360861,https://api.github.com/repos/simonw/datasette/issues/37,636360861,MDEyOklzc3VlQ29tbWVudDYzNjM2MDg2MQ==,9599,simonw,2020-05-30T17:29:20Z,2020-05-30T17:29:20Z,OWNER,I'm not going to do this: 2.5 years later I have yet to run into anything that makes me think that JSON serialization performance is worth any extra work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",268453968,Ability to serialize massive JSON without blocking event loop,
https://github.com/simonw/datasette/issues/498#issuecomment-636360574,https://api.github.com/repos/simonw/datasette/issues/498,636360574,MDEyOklzc3VlQ29tbWVudDYzNjM2MDU3NA==,9599,simonw,2020-05-30T17:26:02Z,2020-05-30T17:26:02Z,OWNER,"I released a plugin that implements an early version of this a while ago: https://github.com/simonw/datasette-search-all
I'm still thinking about how a plugin / separate tool could work that builds a single FTS index over the content from multiple tables such that you can run relevance-calculated queries against those multiple tables.
Since that's going to be a plugin / separate tool too, I'm closing this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",451513541,Full text search of all tables at once?,
https://github.com/simonw/datasette/issues/576#issuecomment-636332183,https://api.github.com/repos/simonw/datasette/issues/576,636332183,MDEyOklzc3VlQ29tbWVudDYzNjMzMjE4Mw==,9599,simonw,2020-05-30T13:37:51Z,2020-05-30T13:38:35Z,OWNER,"**\_\_init\_\_**(self, files, immutables=None, cache_headers=True, cors=False, inspect_data=None, metadata=None, sqlite_extensions=None, template_dir=None, plugins_dir=None, static_mounts=None, memory=False, config=None, version_note=None, config_dir=None)
`Initialize self. See help(type(self)) for accurate signature.`
**absolute_url**(self, request, path)
**add_database**(self, name, db)
**app**(self)
`Returns an ASGI app function that serves the whole of [Datasette](http://localhost:8066/datasette.app.html#Datasette)`
**app_css_hash**(self)
**config**(self, key)
**config_dict**(self)
**connected_databases**(self)
**execute**(self, db_name, sql, params=None, truncate=False, custom_time_limit=None, page_size=None, log_sql_errors=True)
**expand_foreign_keys**(self, database, table, column, values)
`Returns dict mapping (column, value) -> label`
**get_canned_queries**(self, database_name)
**get_canned_query**(self, database_name, query_name)
**metadata**(self, key=None, database=None, table=None, fallback=True)
`Looks up metadata, cascading backwards from specified level.\
Returns None if metadata value is not found.`
**plugin_config**(self, plugin_name, database=None, table=None, fallback=True)
`Return config for plugin, falling back from specified database/table`
**plugins**(self, show_all=False)
**prepare_connection**(self, conn, database)
**register_custom_units**(self)
`Register any custom units defined in the metadata.json with Pint`
**register_renderers**(self)
`Register output renderers which output data in custom formats.`
**remove_database**(self, name)
**render_template**(self, templates, context=None, request=None, view_name=None)
**table_metadata**(self, database, table)
`Fetch table-specific metadata.`
**threads**(self)
**update_with_inherited_metadata**(self, metadata)
**versions**(self)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/576#issuecomment-636332083,https://api.github.com/repos/simonw/datasette/issues/576,636332083,MDEyOklzc3VlQ29tbWVudDYzNjMzMjA4Mw==,9599,simonw,2020-05-30T13:36:55Z,2020-05-30T13:36:55Z,OWNER,"Here's the current `Datasette` class as introspected using the webserver run by `pydoc -p 8000`:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/576#issuecomment-636330843,https://api.github.com/repos/simonw/datasette/issues/576,636330843,MDEyOklzc3VlQ29tbWVudDYzNjMzMDg0Mw==,9599,simonw,2020-05-30T13:26:38Z,2020-05-30T13:26:38Z,OWNER,There's a bunch of methods on that class which I could add a `_` prefix to and leave undocumented. `datasette.register_renderers()` should be `datasette._register_renderers()` for example.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/576#issuecomment-636329095,https://api.github.com/repos/simonw/datasette/issues/576,636329095,MDEyOklzc3VlQ29tbWVudDYzNjMyOTA5NQ==,9599,simonw,2020-05-30T13:11:39Z,2020-05-30T13:25:10Z,OWNER,"I need to document the `.databases` property - both how to get access to specific databases and how to access the first/only database.
Idea: `datasette.get_database(name)` method (consistent with existing `.add_database()` and `.remove_database()` methods) - but `name` parameter is optional, returns first database if you omit that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/576#issuecomment-636330438,https://api.github.com/repos/simonw/datasette/issues/576,636330438,MDEyOklzc3VlQ29tbWVudDYzNjMzMDQzOA==,9599,simonw,2020-05-30T13:23:04Z,2020-05-30T13:23:04Z,OWNER,Need to document `datasette.metadata()` - see #780,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/576#issuecomment-586053947,https://api.github.com/repos/simonw/datasette/issues/576,586053947,MDEyOklzc3VlQ29tbWVudDU4NjA1Mzk0Nw==,9599,simonw,2020-02-14T01:29:48Z,2020-05-30T13:22:09Z,OWNER,"OK, I've made a start on this now in 3ffb8f3b98252531d11897fd431711e9b8045ace - still plenty more methods to document. More importantly that class has a LOT of junk methods on that no-one should ever call from a plugin, so I need to decide what to do about those.
https://datasette.readthedocs.io/en/latest/internals.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/519#issuecomment-636329468,https://api.github.com/repos/simonw/datasette/issues/519,636329468,MDEyOklzc3VlQ29tbWVudDYzNjMyOTQ2OA==,9599,simonw,2020-05-30T13:14:52Z,2020-05-30T13:21:03Z,OWNER,"I've made a lot of progress towards this recently:
- Started internals documentation #576
- Improved design of the request object #706
- Redesigned register_output_renderer plugin hook #581","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459590021,Decide what goes into Datasette 1.0,
https://github.com/simonw/datasette/issues/519#issuecomment-636330023,https://api.github.com/repos/simonw/datasette/issues/519,636330023,MDEyOklzc3VlQ29tbWVudDYzNjMzMDAyMw==,9599,simonw,2020-05-30T13:19:24Z,2020-05-30T13:19:24Z,OWNER,"Goals for Datasette 1.0:
- Generally signify confidence in the quality/stability of Datasette. I think I'm there already.
- Plugin authors can have confidence that their plugins will work for the whole 1.x release cycle
- Developers building against Datasette JSON APIs should have confidence in 1.x compatibility
- Template authors and CSS theme authors should have that confidence too
I think I'm very nearly there for the plugin API. The harder ones are JSON APIs and template authors.
For JSON APIs: The default JSON just isn't right. I find myself using `?_shape=array` for almost everything I build.
The template part is harder. I think I need to fully document the template variables for every view - and add protective unit tests that match that documentation.
The CSS theme part is so hard it may not be possible. How do you guarantee stable HTML that won't break with custom CSS when you'll be adding new features during the 1.x release run? The pattern portfolio from #151 should hopefully help a lot there, but I still don't have a complete idea of what this entails or how feasible it is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459590021,Decide what goes into Datasette 1.0,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-636322089,https://api.github.com/repos/simonw/sqlite-utils/issues/114,636322089,MDEyOklzc3VlQ29tbWVudDYzNjMyMjA4OQ==,9599,simonw,2020-05-30T12:08:43Z,2020-05-30T12:08:43Z,OWNER,"Idea: use a chained API to define a complex transition and then execute it all at once. For example:
```python
db[""mytable""].transform().rename(""col1"", ""col_1"") \
.change_type(""col1"", float) \
.execute()
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/datasette/issues/774#issuecomment-636234759,https://api.github.com/repos/simonw/datasette/issues/774,636234759,MDEyOklzc3VlQ29tbWVudDYzNjIzNDc1OQ==,9599,simonw,2020-05-29T23:27:35Z,2020-05-29T23:27:35Z,OWNER,"Oh dear... it looks like `.raw_args` is used in my TIL script, which has been copied by a few people!
https://github.com/search?q=request+raw_args+datasette&type=Code
I'll fix it in mine and file pull requests against other pieces before this code gets released.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/774#issuecomment-636234067,https://api.github.com/repos/simonw/datasette/issues/774,636234067,MDEyOklzc3VlQ29tbWVudDYzNjIzNDA2Nw==,9599,simonw,2020-05-29T23:24:34Z,2020-05-29T23:24:34Z,OWNER,Updated documentation for `RequestParameters`: https://datasette.readthedocs.io/en/latest/internals.html#the-requestparameters-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/774#issuecomment-636229764,https://api.github.com/repos/simonw/datasette/issues/774,636229764,MDEyOklzc3VlQ29tbWVudDYzNjIyOTc2NA==,9599,simonw,2020-05-29T23:05:48Z,2020-05-29T23:05:48Z,OWNER,"I'm going to rebuild `RequestParameters` to no longer subclass `dict`. I'll keep the following methods:
- `__contains__()`
- `__getitem__()` (with the new behaviour)
- `keys()`
- iterating iterates keys
- `__len__`
- `get`
- `getlist`
It won't support writing, so it will effectively be immutable after you have constructed it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/774#issuecomment-636228656,https://api.github.com/repos/simonw/datasette/issues/774,636228656,MDEyOklzc3VlQ29tbWVudDYzNjIyODY1Ng==,9599,simonw,2020-05-29T23:01:22Z,2020-05-29T23:01:22Z,OWNER,"As far as I can tell the only code I've ever written that would break if I made this change is in `russian-ira-facebook-ads`:
https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/e7106710abdd7bdcae035bedd8bdaba75ae56a12/plugins/target.py#L22
https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/b8a22348c6b315ab94ddba69e8117dfdfd9573dc/plugins/regexp.py#L17
That doesn't work against latest Datasette anyway, so I think I can safely make this change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/774#issuecomment-636227927,https://api.github.com/repos/simonw/datasette/issues/774,636227927,MDEyOklzc3VlQ29tbWVudDYzNjIyNzkyNw==,9599,simonw,2020-05-29T22:58:32Z,2020-05-29T22:58:32Z,OWNER,"I think I want `request.args[""key""]` to return the FIRST item for that key or raise a `KeyError` if none are found.
Right now it returns the full list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/774#issuecomment-635702385,https://api.github.com/repos/simonw/datasette/issues/774,635702385,MDEyOklzc3VlQ29tbWVudDYzNTcwMjM4NQ==,9599,simonw,2020-05-29T01:21:15Z,2020-05-29T01:21:15Z,OWNER,"I think `request.args.getlist()` should return a list, not None, if the key does not exist.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/774#issuecomment-635702201,https://api.github.com/repos/simonw/datasette/issues/774,635702201,MDEyOklzc3VlQ29tbWVudDYzNTcwMjIwMQ==,9599,simonw,2020-05-29T01:20:34Z,2020-05-29T01:20:34Z,OWNER,"Or change `request.args` to behave more like `request.raw_args` - mainly to return a single value when you look things up by key.
It's currently defined like this: https://github.com/simonw/datasette/blob/3c1a60589e14849344acd8aa6da0a60b40fbfc60/datasette/utils/__init__.py#L756-L766","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/351#issuecomment-635533807,https://api.github.com/repos/simonw/datasette/issues/351,635533807,MDEyOklzc3VlQ29tbWVudDYzNTUzMzgwNw==,9599,simonw,2020-05-28T18:56:15Z,2020-05-28T18:56:15Z,OWNER,I'm not going to do this. I have a good enough solution now pasting the rendered HTML from the release notes into https://euangoddard.github.io/clipboard2markdown/ to get the Markdown.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",345469355,Automatically create a GitHub release linking to release notes for every tagged release,
https://github.com/simonw/datasette/issues/508#issuecomment-635532216,https://api.github.com/repos/simonw/datasette/issues/508,635532216,MDEyOklzc3VlQ29tbWVudDYzNTUzMjIxNg==,9599,simonw,2020-05-28T18:53:02Z,2020-05-28T18:53:02Z,OWNER,I fixed this a while ago in #702: https://datasette.readthedocs.io/en/stable/metadata.html#setting-a-default-sort-order,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455965174,Ability to set default sort order for a table or view in metadata.json,
https://github.com/simonw/datasette/issues/774#issuecomment-635530876,https://api.github.com/repos/simonw/datasette/issues/774,635530876,MDEyOklzc3VlQ29tbWVudDYzNTUzMDg3Ng==,9599,simonw,2020-05-28T18:50:18Z,2020-05-28T18:50:18Z,OWNER,"How about moving this functionality to the request object itself?
```python
q = request[""q""] # Raises KeyError if missing, otherwise returns first
q = request.get(""q"", ""default"") # Returns first, or optional default or None
facets = request.getlist(""_facet"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626078521,Consolidate request.raw_args and request.args,
https://github.com/simonw/datasette/issues/777#issuecomment-635513983,https://api.github.com/repos/simonw/datasette/issues/777,635513983,MDEyOklzc3VlQ29tbWVudDYzNTUxMzk4Mw==,63653929,thisismyfuckingusername,2020-05-28T18:16:49Z,2020-05-28T18:16:49Z,NONE," think, because the given URL of the CSS file doesn't have any complete parameters after query
Try to complete the parameter
``","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626171242,Error pages not correctly loading CSS,
https://github.com/simonw/datasette/issues/781#issuecomment-635494730,https://api.github.com/repos/simonw/datasette/issues/781,635494730,MDEyOklzc3VlQ29tbWVudDYzNTQ5NDczMA==,9599,simonw,2020-05-28T17:39:54Z,2020-05-28T17:39:54Z,OWNER,https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fwww.niche-museums.com%2Fbrowse%2Ffeed.atom validates now!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626663119,request.url and request.scheme should obey force_https_urls config setting,
https://github.com/simonw/datasette/issues/780#issuecomment-635480948,https://api.github.com/repos/simonw/datasette/issues/780,635480948,MDEyOklzc3VlQ29tbWVudDYzNTQ4MDk0OA==,9599,simonw,2020-05-28T17:12:45Z,2020-05-28T17:12:45Z,OWNER,This is a good opportunity to reconsider the design of this function and see if I'm happy with it as-is or if there are some improvements I want to make before adding it to the documented API.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626593402,Internals documentation for datasette.metadata() method,
https://github.com/simonw/datasette/issues/781#issuecomment-635471007,https://api.github.com/repos/simonw/datasette/issues/781,635471007,MDEyOklzc3VlQ29tbWVudDYzNTQ3MTAwNw==,9599,simonw,2020-05-28T16:58:47Z,2020-05-28T16:58:47Z,OWNER,"I'm inclined to do this at the earliest possible moment. I think that's probably in the `DatasetteRouter` class, which should see every `scope` first and be able to apply that setting to it. It already has some special case logic to deal with the `base_url` setting here:
https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/datasette/app.py#L779-L789","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626663119,request.url and request.scheme should obey force_https_urls config setting,
https://github.com/simonw/datasette/issues/781#issuecomment-635468994,https://api.github.com/repos/simonw/datasette/issues/781,635468994,MDEyOklzc3VlQ29tbWVudDYzNTQ2ODk5NA==,9599,simonw,2020-05-28T16:55:35Z,2020-05-28T16:55:35Z,OWNER,"I think the right way to fix this is to modify the `scope[""scheme""]` ASGI key before the Request object is constructed - otherwise that Request class will have to gain knowledge of Datasette's configuration mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626663119,request.url and request.scheme should obey force_https_urls config setting,
https://github.com/simonw/datasette/issues/780#issuecomment-635413437,https://api.github.com/repos/simonw/datasette/issues/780,635413437,MDEyOklzc3VlQ29tbWVudDYzNTQxMzQzNw==,9599,simonw,2020-05-28T15:15:03Z,2020-05-28T15:15:03Z,OWNER,Also: I think I should add a `query=` parameter to help lookup metadata about a specific named canned query.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626593402,Internals documentation for datasette.metadata() method,
https://github.com/simonw/datasette/issues/744#issuecomment-635386935,https://api.github.com/repos/simonw/datasette/issues/744,635386935,MDEyOklzc3VlQ29tbWVudDYzNTM4NjkzNQ==,30607,aborruso,2020-05-28T14:32:53Z,2020-05-28T14:32:53Z,NONE,"Wow, I'm in some way very proud!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-635384739,https://api.github.com/repos/simonw/datasette/issues/744,635384739,MDEyOklzc3VlQ29tbWVudDYzNTM4NDczOQ==,9599,simonw,2020-05-28T14:28:58Z,2020-05-28T14:28:58Z,OWNER,This is now released. `pip install datasette==0.43` should get the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/758#issuecomment-635195322,https://api.github.com/repos/simonw/datasette/issues/758,635195322,MDEyOklzc3VlQ29tbWVudDYzNTE5NTMyMg==,2181410,clausjuhl,2020-05-28T08:23:27Z,2020-05-28T08:23:27Z,NONE,@simonw I would prefer just the 7 character hash. No need to make the urls any longer than they need to be :),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612382643,Question: Access to immutable database-path,
https://github.com/simonw/datasette/issues/706#issuecomment-635131217,https://api.github.com/repos/simonw/datasette/issues/706,635131217,MDEyOklzc3VlQ29tbWVudDYzNTEzMTIxNw==,9599,simonw,2020-05-28T06:20:44Z,2020-05-28T06:20:44Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#canned-queries-default-fragment,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585633142,"Documentation for the ""request"" object",
https://github.com/simonw/datasette/issues/758#issuecomment-635102675,https://api.github.com/repos/simonw/datasette/issues/758,635102675,MDEyOklzc3VlQ29tbWVudDYzNTEwMjY3NQ==,9599,simonw,2020-05-28T05:04:07Z,2020-05-28T05:04:07Z,OWNER,"@clausjuhl do you have any thoughts on what would be most useful for you in these JSON responses? The full `/databasename-hash` path, just the 7 character hash, or something else?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612382643,Question: Access to immutable database-path,
https://github.com/simonw/datasette/issues/751#issuecomment-635101438,https://api.github.com/repos/simonw/datasette/issues/751,635101438,MDEyOklzc3VlQ29tbWVudDYzNTEwMTQzOA==,9599,simonw,2020-05-28T05:00:21Z,2020-05-28T05:00:21Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/metadata.html#setting-a-custom-page-size,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611540797,Ability to set custom default _size on a per-table basis,
https://github.com/simonw/datasette/issues/770#issuecomment-634880090,https://api.github.com/repos/simonw/datasette/issues/770,634880090,MDEyOklzc3VlQ29tbWVudDYzNDg4MDA5MA==,9599,simonw,2020-05-27T19:10:57Z,2020-05-28T04:22:47Z,OWNER,"This `can_render` callback should take the same arguments as the redesigned `render` (previously called `callback`): https://github.com/simonw/datasette/issues/581#issuecomment-634879258
- `datasette` - a Datasette instance
- `columns` - the list of columns
- `rows` - the list of rows (each one a SQLite `Row` object)
- `sql` - the SQL query being executed
- `query_name` - the name of the canned query, if this is one
- `database` - the database name
- `table` - the table or view name
- `request` - the request object (to be documented in #706)
- `view_name` - the name of the view","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625930207,register_output_renderer can_render mechanism,
https://github.com/simonw/datasette/issues/773#issuecomment-635069987,https://api.github.com/repos/simonw/datasette/issues/773,635069987,MDEyOklzc3VlQ29tbWVudDYzNTA2OTk4Nw==,9599,simonw,2020-05-28T03:13:27Z,2020-05-28T03:13:27Z,OWNER,`register_output_renderer` test added in https://github.com/simonw/datasette/commit/52c4387c7d37c867104e3728cc1f4c4d1e100642#diff-56f7d7b4778bac73b6b655c02c8467aaR336,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626001501,All plugin hooks should have unit tests,
https://github.com/simonw/datasette/issues/770#issuecomment-635056520,https://api.github.com/repos/simonw/datasette/issues/770,635056520,MDEyOklzc3VlQ29tbWVudDYzNTA1NjUyMA==,9599,simonw,2020-05-28T02:28:00Z,2020-05-28T02:28:00Z,OWNER,"This should be optionally awaitable, as in #776","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625930207,register_output_renderer can_render mechanism,
https://github.com/simonw/datasette/issues/776#issuecomment-635056357,https://api.github.com/repos/simonw/datasette/issues/776,635056357,MDEyOklzc3VlQ29tbWVudDYzNTA1NjM1Nw==,9599,simonw,2020-05-28T02:27:26Z,2020-05-28T02:27:26Z,OWNER,"Example code from elsewhere:
https://github.com/simonw/datasette/blob/52c4387c7d37c867104e3728cc1f4c4d1e100642/datasette/views/base.py#L416-L420","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626163974,register_output_renderer render callback should be optionally awaitable,
https://github.com/simonw/datasette/issues/581#issuecomment-635055346,https://api.github.com/repos/simonw/datasette/issues/581,635055346,MDEyOklzc3VlQ29tbWVudDYzNTA1NTM0Ng==,9599,simonw,2020-05-28T02:24:14Z,2020-05-28T02:24:14Z,OWNER,Updated documentation is here: https://datasette.readthedocs.io/en/latest/plugins.html#register-output-renderer-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/645#issuecomment-635054690,https://api.github.com/repos/simonw/datasette/issues/645,635054690,MDEyOklzc3VlQ29tbWVudDYzNTA1NDY5MA==,9599,simonw,2020-05-28T02:22:12Z,2020-05-28T02:22:12Z,OWNER,"This is a duplicate of a more recent, more developed issue: #770","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",530653633,Mechanism for register_output_renderer to suggest extension or not,
https://github.com/simonw/datasette/issues/581#issuecomment-634879258,https://api.github.com/repos/simonw/datasette/issues/581,634879258,MDEyOklzc3VlQ29tbWVudDYzNDg3OTI1OA==,9599,simonw,2020-05-27T19:09:22Z,2020-05-28T01:33:45Z,OWNER,"OK, the new design: your callback function can take any of the following arguments:
- `datasette` - a Datasette instance
- `columns` - the list of columns
- `rows` - the list of rows (each one a SQLite `Row` object)
- `sql` - the SQL query being executed
- `query_name` - the name of the canned query, if this is one
- `database` - the database name
- `table` - the table or view name
- `request` - the request object (to be documented in #706)
- `view_name` - the name of the view
We will also continue to support the existing `args` and `data` arguments, but these will be undocumented and will be deprecated in Datasette 1.0. UPDATE: Decided against this, see https://github.com/simonw/datasette/issues/581#issuecomment-634946197","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/775#issuecomment-635023389,https://api.github.com/repos/simonw/datasette/issues/775,635023389,MDEyOklzc3VlQ29tbWVudDYzNTAyMzM4OQ==,9599,simonw,2020-05-28T00:47:32Z,2020-05-28T00:47:32Z,OWNER,"These:
https://github.com/simonw/datasette/blob/4b96857f170e329a73186e703cc0d9ca4e8719cc/tests/fixtures.py#L337-L506","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626131309,Move test plugins into datasette/tests/plugins/ directory,
https://github.com/simonw/datasette/issues/770#issuecomment-634980179,https://api.github.com/repos/simonw/datasette/issues/770,634980179,MDEyOklzc3VlQ29tbWVudDYzNDk4MDE3OQ==,9599,simonw,2020-05-27T22:37:19Z,2020-05-27T22:37:19Z,OWNER,"Can I come up with a better name than `should_suggest`?
It's a check that sees if the current query is supported by the renderer plugin. Some options:
- `can_render`
- `supports_query`
- `is_supported`
I like `can_render`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625930207,register_output_renderer can_render mechanism,
https://github.com/simonw/datasette/issues/581#issuecomment-634978388,https://api.github.com/repos/simonw/datasette/issues/581,634978388,MDEyOklzc3VlQ29tbWVudDYzNDk3ODM4OA==,9599,simonw,2020-05-27T22:32:03Z,2020-05-27T22:32:03Z,OWNER,Request object is now documented: https://datasette.readthedocs.io/en/latest/internals.html#request-object,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/706#issuecomment-634975252,https://api.github.com/repos/simonw/datasette/issues/706,634975252,MDEyOklzc3VlQ29tbWVudDYzNDk3NTI1Mg==,9599,simonw,2020-05-27T22:23:26Z,2020-05-27T22:30:05Z,OWNER,I'm going to leave `.raw_args` in for the moment but deliberately not document it. I'll hope to phase it out entirely at a later date.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585633142,"Documentation for the ""request"" object",
https://github.com/simonw/datasette/issues/706#issuecomment-634974819,https://api.github.com/repos/simonw/datasette/issues/706,634974819,MDEyOklzc3VlQ29tbWVudDYzNDk3NDgxOQ==,9599,simonw,2020-05-27T22:22:20Z,2020-05-27T22:22:20Z,OWNER,"What would a better name be?
- `.simple_args`
- `.kv_args`
- `.pair_args`
- `.dict_args`
- `.args_dict`
I dislike the last two the least.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585633142,"Documentation for the ""request"" object",
https://github.com/simonw/datasette/issues/706#issuecomment-634974088,https://api.github.com/repos/simonw/datasette/issues/706,634974088,MDEyOklzc3VlQ29tbWVudDYzNDk3NDA4OA==,9599,simonw,2020-05-27T22:20:20Z,2020-05-27T22:20:20Z,OWNER,It looks like I inherited `.raw_args` from Sanic - I use it in a few places: https://github.com/search?q=user%3Asimonw+raw_args&type=Code,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585633142,"Documentation for the ""request"" object",
https://github.com/simonw/datasette/issues/706#issuecomment-634973596,https://api.github.com/repos/simonw/datasette/issues/706,634973596,MDEyOklzc3VlQ29tbWVudDYzNDk3MzU5Ng==,9599,simonw,2020-05-27T22:19:02Z,2020-05-27T22:19:02Z,OWNER,"New documentation can be seen here: https://github.com/simonw/datasette/blob/6d7cb02f00010d3cb4b4bac0460d41277652b80e/docs/internals.rst#request-object
It's inspired me to reconsider the name of the `.raw_args` property, which isn't particularly clear.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585633142,"Documentation for the ""request"" object",
https://github.com/simonw/datasette/issues/706#issuecomment-634965148,https://api.github.com/repos/simonw/datasette/issues/706,634965148,MDEyOklzc3VlQ29tbWVudDYzNDk2NTE0OA==,9599,simonw,2020-05-27T21:59:07Z,2020-05-27T21:59:07Z,OWNER,"This is the full current implementation of the request object:
https://github.com/simonw/datasette/blob/9424687e9e94401438896116898a071702b09d40/datasette/utils/asgi.py#L15-L95","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585633142,"Documentation for the ""request"" object",
https://github.com/simonw/datasette/issues/581#issuecomment-634964457,https://api.github.com/repos/simonw/datasette/issues/581,634964457,MDEyOklzc3VlQ29tbWVudDYzNDk2NDQ1Nw==,9599,simonw,2020-05-27T21:57:35Z,2020-05-27T21:57:35Z,OWNER,(I wonder if this would be enough to allow really smart plugins to implement ETag/conditional get),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634964294,https://api.github.com/repos/simonw/datasette/issues/581,634964294,MDEyOklzc3VlQ29tbWVudDYzNDk2NDI5NA==,9599,simonw,2020-05-27T21:57:10Z,2020-05-27T21:57:10Z,OWNER,"Right now a rendering callback returns the following:
```
body - string or bytes, optional
The response body, default empty
content_type - string, optional
The Content-Type header, default text/plain
status_code - integer, optional
The HTTP status code, default 200
```
I'm going to add an optional `headers` dictionary key, too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/758#issuecomment-634951605,https://api.github.com/repos/simonw/datasette/issues/758,634951605,MDEyOklzc3VlQ29tbWVudDYzNDk1MTYwNQ==,9599,simonw,2020-05-27T21:29:19Z,2020-05-27T21:29:19Z,OWNER,"But... https://datasette-hash-urls-j7hipcg4aq-uw.a.run.app/fixtures-bda7daa.json doesn't expose that hash:
```
{
""database"": ""fixtures"",
""size"": 258048,
""tables"": [
{
""name"": ""123_starts_with_digits"",
```
Likewise https://datasette-hash-urls-j7hipcg4aq-uw.a.run.app/fixtures-bda7daa/complex_foreign_keys.json
```
{
""database"": ""fixtures"",
""table"": ""complex_foreign_keys"",
""is_view"": false,
""human_description_en"": """",
""rows"": [
[
""1"",
""1"",
""2"",
""1""
]
],
```
And https://datasette-hash-urls-j7hipcg4aq-uw.a.run.app/fixtures-bda7daa/complex_foreign_keys/1.json
```
{
""database"": ""fixtures"",
""table"": ""complex_foreign_keys"",
""rows"": [
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612382643,Question: Access to immutable database-path,
https://github.com/simonw/datasette/issues/758#issuecomment-634950200,https://api.github.com/repos/simonw/datasette/issues/758,634950200,MDEyOklzc3VlQ29tbWVudDYzNDk1MDIwMA==,9599,simonw,2020-05-27T21:26:37Z,2020-05-27T21:26:37Z,OWNER,"https://latest.datasette.io/.json currently returns:
```
{
""fixtures"": {
""name"": ""fixtures"",
""hash"": ""87b3f2c55dfb81ff1452dd306c2623fa5550b90982cfa32bad404c4d8bbedde2"",
""color"": ""87b3f2"",
""path"": ""/fixtures"",
""tables_and_views_truncated"": [
```
I published `fixtures.db` here like this:
datasette publish cloudrun fixtures.db --service datasette-hash-urls --extra-options '--config hash_urls:1'
https://datasette-hash-urls-j7hipcg4aq-uw.a.run.app/.json
```
{
""fixtures"": {
""name"": ""fixtures"",
""hash"": ""bda7daa889c23f9a8f06e46d7d280dd423c76275e9593c4c1cad7c53b19032fe"",
""color"": ""bda7da"",
""path"": ""/fixtures-bda7daa"",
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612382643,Question: Access to immutable database-path,
https://github.com/simonw/datasette/issues/581#issuecomment-634946319,https://api.github.com/repos/simonw/datasette/issues/581,634946319,MDEyOklzc3VlQ29tbWVudDYzNDk0NjMxOQ==,9599,simonw,2020-05-27T21:18:50Z,2020-05-27T21:18:50Z,OWNER,(I used GitHub code search to find code using this plugin hook: https://github.com/search?q=register_output_renderer&type=Code ),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634946197,https://api.github.com/repos/simonw/datasette/issues/581,634946197,MDEyOklzc3VlQ29tbWVudDYzNDk0NjE5Nw==,9599,simonw,2020-05-27T21:18:30Z,2020-05-27T21:18:30Z,OWNER,"I'm going to break backwards compatibility directly here, without waiting for Datasette 1.0.
The reason is that https://github.com/russss/datasette-geo hasn't been updated in 13 months so is already broken against current Datasette, and the other two plugins using this hook are owned by me so I can upgrade them myself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634944832,https://api.github.com/repos/simonw/datasette/issues/581,634944832,MDEyOklzc3VlQ29tbWVudDYzNDk0NDgzMg==,9599,simonw,2020-05-27T21:15:50Z,2020-05-27T21:16:28Z,OWNER,"It bothers me that `query_name` here means the configured name of the canned query, but `view_name` means the name of the Datasette view class, NOT the name of an associated SQL view. That's in `table`.
Can I come up with clearer names for these?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634943336,https://api.github.com/repos/simonw/datasette/issues/581,634943336,MDEyOklzc3VlQ29tbWVudDYzNDk0MzMzNg==,9599,simonw,2020-05-27T21:13:04Z,2020-05-27T21:13:04Z,OWNER,Since I'm passing `request` I won't pass `scope` - if people want that they can access `request.scope`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-592621235,https://api.github.com/repos/simonw/datasette/issues/581,592621235,MDEyOklzc3VlQ29tbWVudDU5MjYyMTIzNQ==,9599,simonw,2020-02-28T17:24:06Z,2020-05-27T21:12:21Z,OWNER,"Rather than pass a request object (hence promoting that object into part of the documented, stable API) I think I'll pass the ASGI scope - that's already a stable, documented standard.
UPDATE: changed my mind since `request` is used by other plugins too, see #706.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/773#issuecomment-634940522,https://api.github.com/repos/simonw/datasette/issues/773,634940522,MDEyOklzc3VlQ29tbWVudDYzNDk0MDUyMg==,9599,simonw,2020-05-27T21:07:48Z,2020-05-27T21:07:48Z,OWNER,Remove this `xfail` decorator once they are all tested: https://github.com/simonw/datasette/blob/da87e963bff24e47878a5bc2025c8bfc63d4bc93/tests/test_plugins.py#L23-L28,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626001501,All plugin hooks should have unit tests,
https://github.com/simonw/datasette/issues/581#issuecomment-634921101,https://api.github.com/repos/simonw/datasette/issues/581,634921101,MDEyOklzc3VlQ29tbWVudDYzNDkyMTEwMQ==,9599,simonw,2020-05-27T20:27:36Z,2020-05-27T20:27:36Z,OWNER,Actually passing the `request` object would be OK if I document it see https://github.com/simonw/datasette/issues/706,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/771#issuecomment-634916313,https://api.github.com/repos/simonw/datasette/issues/771,634916313,MDEyOklzc3VlQ29tbWVudDYzNDkxNjMxMw==,9599,simonw,2020-05-27T20:17:13Z,2020-05-27T20:17:13Z,OWNER,Closed in da87e963bff24e47878a5bc2025c8bfc63d4bc93,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625980317,Unit test that checks that all plugin hooks have corresponding unit tests,
https://github.com/simonw/datasette/issues/771#issuecomment-634915104,https://api.github.com/repos/simonw/datasette/issues/771,634915104,MDEyOklzc3VlQ29tbWVudDYzNDkxNTEwNA==,9599,simonw,2020-05-27T20:14:32Z,2020-05-27T20:14:32Z,OWNER,"```
$ pytest -k test_plugin_hooks_have_tests -vv
====================================== test session starts ======================================
platform darwin -- Python 3.7.7, pytest-5.2.4, py-1.8.1, pluggy-0.13.1 -- /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/python
cachedir: .pytest_cache
rootdir: /Users/simon/Dropbox/Development/datasette, inifile: pytest.ini
plugins: asyncio-0.10.0
collected 486 items / 475 deselected / 11 selected
tests/test_plugins.py::test_plugin_hooks_have_tests[asgi_wrapper] XPASS [ 9%]
tests/test_plugins.py::test_plugin_hooks_have_tests[extra_body_script] XPASS [ 18%]
tests/test_plugins.py::test_plugin_hooks_have_tests[extra_css_urls] XPASS [ 27%]
tests/test_plugins.py::test_plugin_hooks_have_tests[extra_js_urls] XPASS [ 36%]
tests/test_plugins.py::test_plugin_hooks_have_tests[extra_template_vars] XPASS [ 45%]
tests/test_plugins.py::test_plugin_hooks_have_tests[prepare_connection] XPASS [ 54%]
tests/test_plugins.py::test_plugin_hooks_have_tests[prepare_jinja2_environment] XFAIL [ 63%]
tests/test_plugins.py::test_plugin_hooks_have_tests[publish_subcommand] XFAIL [ 72%]
tests/test_plugins.py::test_plugin_hooks_have_tests[register_facet_classes] XFAIL [ 81%]
tests/test_plugins.py::test_plugin_hooks_have_tests[register_output_renderer] XFAIL [ 90%]
tests/test_plugins.py::test_plugin_hooks_have_tests[render_cell] XPASS [100%]
========================= 475 deselected, 4 xfailed, 7 xpassed in 1.70s =========================","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625980317,Unit test that checks that all plugin hooks have corresponding unit tests,
https://github.com/simonw/datasette/issues/771#issuecomment-634909818,https://api.github.com/repos/simonw/datasette/issues/771,634909818,MDEyOklzc3VlQ29tbWVudDYzNDkwOTgxOA==,9599,simonw,2020-05-27T20:02:52Z,2020-05-27T20:02:52Z,OWNER,Actually I'll land this using `@pytest.mark.xfail`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625980317,Unit test that checks that all plugin hooks have corresponding unit tests,
https://github.com/simonw/datasette/issues/771#issuecomment-634909347,https://api.github.com/repos/simonw/datasette/issues/771,634909347,MDEyOklzc3VlQ29tbWVudDYzNDkwOTM0Nw==,9599,simonw,2020-05-27T20:01:52Z,2020-05-27T20:01:52Z,OWNER,I'll do the work for this in the pull request #772.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625980317,Unit test that checks that all plugin hooks have corresponding unit tests,
https://github.com/simonw/datasette/issues/771#issuecomment-634900776,https://api.github.com/repos/simonw/datasette/issues/771,634900776,MDEyOklzc3VlQ29tbWVudDYzNDkwMDc3Ng==,9599,simonw,2020-05-27T19:44:25Z,2020-05-27T19:44:25Z,OWNER,"This seems to work:
```diff
diff --git a/tests/test_plugins.py b/tests/test_plugins.py
index 8b6a6b4..e9a40aa 100644
--- a/tests/test_plugins.py
+++ b/tests/test_plugins.py
@@ -7,7 +7,7 @@ from .fixtures import (
TestClient as _TestClient,
) # noqa
from datasette.app import Datasette
-from datasette.plugins import get_plugins, DEFAULT_PLUGINS
+from datasette.plugins import get_plugins, DEFAULT_PLUGINS, pm
from datasette.utils import sqlite3
import base64
import json
@@ -20,6 +20,21 @@ import pytest
import urllib
+def test_plugin_hooks_have_tests():
+ ""Every plugin hook should be referenced in this test module""
+ hooks = [name for name in dir(pm.hook) if not name.startswith(""_"")]
+ tests_in_this_module = [t for t in globals().keys() if t.startswith('test_')]
+ untested = []
+ for hook in hooks:
+ ok = False
+ for test in tests_in_this_module:
+ if hook in test:
+ ok = True
+ if not ok:
+ untested.append(hook)
+ assert not untested, 'These plugin hooks are missing tests: {}'.format(untested)
+
+
def test_plugins_dir_plugin_prepare_connection(app_client):
response = app_client.get(
""/fixtures.json?sql=select+convert_units(100%2C+'m'%2C+'ft')""
```
Based on how the documentation unit tests work.
Currently fails with:
AssertionError: These plugin hooks are missing tests:
['prepare_jinja2_environment', 'publish_subcommand', 'register_facet_classes', 'register_output_renderer']
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625980317,Unit test that checks that all plugin hooks have corresponding unit tests,
https://github.com/simonw/datasette/issues/581#issuecomment-634893744,https://api.github.com/repos/simonw/datasette/issues/581,634893744,MDEyOklzc3VlQ29tbWVudDYzNDg5Mzc0NA==,9599,simonw,2020-05-27T19:32:08Z,2020-05-27T19:32:08Z,OWNER,Need to figure out how best to unit test this plugin hook.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634888582,https://api.github.com/repos/simonw/datasette/issues/581,634888582,MDEyOklzc3VlQ29tbWVudDYzNDg4ODU4Mg==,9599,simonw,2020-05-27T19:23:23Z,2020-05-27T19:23:23Z,OWNER,"Here's the function I just wrote for this:
```python
def call_with_supported_arguments(fn, **kwargs):
parameters = inspect.signature(fn).parameters.keys()
call_with = []
for parameter in parameters:
if parameter not in kwargs:
raise TypeError(""{} requires parameters {}"".format(fn, tuple(parameters)))
call_with.append(kwargs[parameter])
return fn(*call_with)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634882770,https://api.github.com/repos/simonw/datasette/issues/581,634882770,MDEyOklzc3VlQ29tbWVudDYzNDg4Mjc3MA==,9599,simonw,2020-05-27T19:16:19Z,2020-05-27T19:16:19Z,OWNER,"```
In [1]: import inspect
In [2]: def foo(view, sql, inspect):
...: pass
...:
In [3]: inspect.signature(foo)
Out[3]:
In [4]: inspect.signature(foo).parameters
Out[4]:
mappingproxy({'view': ,
'sql': ,
'inspect': })
In [5]: inspect.signature(foo).parameters.keys()
Out[5]: odict_keys(['view', 'sql', 'inspect'])
In [6]: set(inspect.signature(foo).parameters.keys())
Out[6]: {'inspect', 'sql', 'view'}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634882112,https://api.github.com/repos/simonw/datasette/issues/581,634882112,MDEyOklzc3VlQ29tbWVudDYzNDg4MjExMg==,9599,simonw,2020-05-27T19:14:55Z,2020-05-27T19:14:55Z,OWNER,"https://docs.python.org/3/library/inspect.html#introspecting-callables-with-the-signature-object
> New in version 3.3.
>
> The Signature object represents the call signature of a callable object and its return annotation. To retrieve a Signature object, use the [`signature()`](https://docs.python.org/3/library/inspect.html#inspect.signature ""inspect.signature"") function.
>
> `inspect.``signature`(*callable*, ***, *follow_wrapped=True*)
>
> Return a [`Signature`](https://docs.python.org/3/library/inspect.html#inspect.Signature ""inspect.Signature"") object for the given `callable`:
>
> ```
> >>> from inspect import signature
> >>> def foo(a, *, b:int, **kwargs):
> ... pass
>
> >>> sig = signature(foo)
>
> >>> str(sig)
> '(a, *, b:int, **kwargs)'
>
> >>> str(sig.parameters['b'])
> 'b:int'
>
> >>> sig.parameters['b'].annotation
>
> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634881287,https://api.github.com/repos/simonw/datasette/issues/581,634881287,MDEyOklzc3VlQ29tbWVudDYzNDg4MTI4Nw==,9599,simonw,2020-05-27T19:13:09Z,2020-05-27T19:13:09Z,OWNER,"I think I need a utility function for ""call this function with this dictionary of arguments, but only pass the arguments which are inspected by the function"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634879734,https://api.github.com/repos/simonw/datasette/issues/581,634879734,MDEyOklzc3VlQ29tbWVudDYzNDg3OTczNA==,9599,simonw,2020-05-27T19:10:17Z,2020-05-27T19:12:36Z,OWNER,The `should_suggest` callback will take the same arguments: https://github.com/simonw/datasette/issues/770#issuecomment-634880090,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/770#issuecomment-634880474,https://api.github.com/repos/simonw/datasette/issues/770,634880474,MDEyOklzc3VlQ29tbWVudDYzNDg4MDQ3NA==,9599,simonw,2020-05-27T19:11:39Z,2020-05-27T19:11:39Z,OWNER,I'm going to rename `callback` to `render` but continue supporting `callback` until Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",625930207,register_output_renderer can_render mechanism,
https://github.com/simonw/datasette/issues/581#issuecomment-634865620,https://api.github.com/repos/simonw/datasette/issues/581,634865620,MDEyOklzc3VlQ29tbWVudDYzNDg2NTYyMA==,9599,simonw,2020-05-27T18:44:06Z,2020-05-27T18:44:06Z,OWNER,"The existing render callback takes the following arguments:
> `args` - dictionary
> The GET parameters of the request
>
> `data` - dictionary
> The data to be rendered
>
> `view_name` - string
> The name of the view where the renderer is being called. (`index`, `database`, `table`, and `row` are the most important ones.)
The `data` argument is a bit of a problem, because it tightly couples plugins to a currently undocumented datastructure within Datasette. Here's how `datasette-atom` picks that apart for example: https://github.com/simonw/datasette-atom/blob/095941c23c81b70c4787cdeef873c556b573b5fa/datasette_atom/__init__.py#L15-L66 - it does things like access `data[""query""][""sql""]` to figure out the SQL query that was used.
I'm going to change the design of part of this ticket. I won't break the old `data` value just yet, but I'll mark it to be deprecated by Datasette 1.0.
I think the only plugins using it right now are my `datasette-atom` and `datasette-ics` and @russss's [datasette-geo](https://github.com/russss/datasette-geo/blob/0d24c9fd782eeae8d136ee791143ee8cbf49ebdf/datasette_plugin_geo/__init__.py#L94-L101) so hopefully changing this won't cause any wider damage.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634857975,https://api.github.com/repos/simonw/datasette/issues/581,634857975,MDEyOklzc3VlQ29tbWVudDYzNDg1Nzk3NQ==,9599,simonw,2020-05-27T18:30:29Z,2020-05-27T18:30:29Z,OWNER,I'll use #770 for the `should_suggest` mechanism - this issue is for the extra arguments passed to the rendering callback.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634856748,https://api.github.com/repos/simonw/datasette/issues/581,634856748,MDEyOklzc3VlQ29tbWVudDYzNDg1Njc0OA==,9599,simonw,2020-05-27T18:28:32Z,2020-05-27T18:28:32Z,OWNER,"Here's the code that passes a list of renderers to the template:
https://github.com/simonw/datasette/blob/2d099ad9c657d2cab59de91cdb8bfed2da236ef6/datasette/views/base.py#L411-L423
A renderer is currently defined as a two-key dictionary:
```python
@hookimpl
def register_output_renderer(datasette):
return {
'extension': 'test',
'callback': render_test
}
```
I can add a third key, `""should_suggest""` which is a function that returns `True` or `False` for a given query. If that key is missing it is assumed to return `True`.
One catch: what arguments should be passed to the `should_suggest(...)` function?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634853296,https://api.github.com/repos/simonw/datasette/issues/581,634853296,MDEyOklzc3VlQ29tbWVudDYzNDg1MzI5Ng==,9599,simonw,2020-05-27T18:22:46Z,2020-05-27T18:22:46Z,OWNER,"While I'm doing this, another feature I would like is the ability for renderers to opt-in / opt-out of being displayed as options on the page.
https://www.niche-museums.com/browse/museums for example shows a `atom` link because the `datasette-atom` plugin is installed... but clicking it will give you a 400 error because the correct columns are not present:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/581#issuecomment-634852196,https://api.github.com/repos/simonw/datasette/issues/581,634852196,MDEyOklzc3VlQ29tbWVudDYzNDg1MjE5Ng==,9599,simonw,2020-05-27T18:20:46Z,2020-05-27T18:20:46Z,OWNER,"Here's the code that calls the renderers - this needs to be expanded to check for those extra optional arguments:
https://github.com/simonw/datasette/blob/2d099ad9c657d2cab59de91cdb8bfed2da236ef6/datasette/views/base.py#L387-L398","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",502993509,Redesign register_output_renderer callback,
https://github.com/simonw/datasette/issues/744#issuecomment-634850676,https://api.github.com/repos/simonw/datasette/issues/744,634850676,MDEyOklzc3VlQ29tbWVudDYzNDg1MDY3Ng==,9599,simonw,2020-05-27T18:18:01Z,2020-05-27T18:18:01Z,OWNER,Thanks for helping test this @aborruso!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-634446887,https://api.github.com/repos/simonw/datasette/issues/744,634446887,MDEyOklzc3VlQ29tbWVudDYzNDQ0Njg4Nw==,30607,aborruso,2020-05-27T06:01:28Z,2020-05-27T06:01:28Z,NONE,"Dear @simonw thank you for your time, now IT WORKS!!!
I hope that this edit to datasette code is not for an exceptional case (my PC configuration) and that it will be useful to other users.
Thank you again!!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-634395343,https://api.github.com/repos/simonw/datasette/issues/744,634395343,MDEyOklzc3VlQ29tbWVudDYzNDM5NTM0Mw==,9599,simonw,2020-05-27T02:49:26Z,2020-05-27T02:49:26Z,OWNER,"OK, here's a new branch you can try. Install it like this:
pip install https://github.com/simonw/datasette/archive/shutil-backport.zip
If it works for you I'll merge that branch into master.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-634283355,https://api.github.com/repos/simonw/datasette/issues/744,634283355,MDEyOklzc3VlQ29tbWVudDYzNDI4MzM1NQ==,30607,aborruso,2020-05-26T21:15:34Z,2020-05-26T21:15:34Z,NONE,"> Oh no! It looks like `dirs_exist_ok` is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.
Thank you very much","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-634254943,https://api.github.com/repos/simonw/datasette/issues/744,634254943,MDEyOklzc3VlQ29tbWVudDYzNDI1NDk0Mw==,9599,simonw,2020-05-26T20:15:11Z,2020-05-26T20:15:11Z,OWNER,"Oh no! It looks like `dirs_exist_ok` is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-634254700,https://api.github.com/repos/simonw/datasette/issues/744,634254700,MDEyOklzc3VlQ29tbWVudDYzNDI1NDcwMA==,9599,simonw,2020-05-26T20:14:35Z,2020-05-26T20:14:35Z,OWNER,What version of Python are you running?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633704127,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633704127,MDEyOklzc3VlQ29tbWVudDYzMzcwNDEyNw==,9599,simonw,2020-05-25T20:14:22Z,2020-05-25T20:14:22Z,MEMBER,https://github.com/dogsheep/dogsheep-photos/blob/0.4.1/README.md#serving-photos-locally-with-datasette-media,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633644225,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633644225,MDEyOklzc3VlQ29tbWVudDYzMzY0NDIyNQ==,9599,simonw,2020-05-25T16:30:44Z,2020-05-25T16:30:44Z,MEMBER,I'll add docs on using `datasette-json-html` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633643921,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633643921,MDEyOklzc3VlQ29tbWVudDYzMzY0MzkyMQ==,9599,simonw,2020-05-25T16:29:44Z,2020-05-25T16:29:44Z,MEMBER,https://github.com/dogsheep/dogsheep-photos/blob/dc43fa8653cb9c7238a36f52239b91d1ec916d5c/README.md#serving-photos-locally-with-datasette-media,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633629944,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633629944,MDEyOklzc3VlQ29tbWVudDYzMzYyOTk0NA==,9599,simonw,2020-05-25T15:47:42Z,2020-05-25T15:47:42Z,MEMBER,"I'll add a proper section to the README, but for the moment here's how I do this.
First, install `datasette` and the `datasette-media` plugin.
Create a `metadata.yaml` file with the following content:
```yaml
plugins:
datasette-media:
photo:
sql: |-
select path as filepath, 200 as resize_height from apple_photos where uuid = :key
photo-big:
sql: |-
select path as filepath, 1024 as resize_height from apple_photos where uuid = :key
```
Now run `datasette -m metadata.yaml photos.db` - thumbnails will be served at http://127.0.0.1:8001/-/media/photo/F4469918-13F3-43D8-9EC1-734C0E6B60AD and larger sizes of the image at http://127.0.0.1:8001/-/media/photo-big/A8B02C7D-365E-448B-9510-69F80C26304D
I also made myself two custom pages, one showing recent images and one showing random images.
To do this, install the `datasette-template-sql` plugin and then create a `templates/pages` directory and add these files:
`recent-photos.html`
```html
sys.exit(cli())
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py"", line 103, in heroku
extra_metadata,
File ""/usr/lib/python3.7/contextlib.py"", line 112, in __enter__
return next(self.gen)
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py"", line 191, in temporary_heroku_directory
os.path.join(tmp.name, ""templates""),
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py"", line 605, in link_or_copy_directory
shutil.copytree(src, dst, copy_function=os.link, dirs_exist_ok=True)
TypeError: copytree() got an unexpected keyword argument 'dirs_exist_ok'
```
Do I must open a new issue?
Thank you","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632255088,https://api.github.com/repos/simonw/datasette/issues/744,632255088,MDEyOklzc3VlQ29tbWVudDYzMjI1NTA4OA==,30607,aborruso,2020-05-21T17:58:51Z,2020-05-21T17:58:51Z,NONE,"Thank you very much!!
I will try and I write you here","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632253073,https://api.github.com/repos/simonw/datasette/issues/744,632253073,MDEyOklzc3VlQ29tbWVudDYzMjI1MzA3Mw==,9599,simonw,2020-05-21T17:54:55Z,2020-05-21T17:54:55Z,OWNER,"I think this is the fix! I just landed it in master.
@aborruso you can install Datasette master to test it out like this:
pip install https://github.com/simonw/datasette/archive/cee671a58f417f827d1735b1abaa40716534ea67.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632249565,https://api.github.com/repos/simonw/datasette/issues/744,632249565,MDEyOklzc3VlQ29tbWVudDYzMjI0OTU2NQ==,30607,aborruso,2020-05-21T17:47:40Z,2020-05-21T17:47:40Z,NONE,"@simonw can I test it know? What I must do to update it?
Thank you","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632237195,https://api.github.com/repos/simonw/datasette/issues/744,632237195,MDEyOklzc3VlQ29tbWVudDYzMjIzNzE5NQ==,9599,simonw,2020-05-21T17:23:39Z,2020-05-21T17:23:48Z,OWNER,Actually maybe the answer here is to use `dirs_exist_ok=True` when calling `shutil.copytree`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-632233393,https://api.github.com/repos/simonw/datasette/issues/744,632233393,MDEyOklzc3VlQ29tbWVudDYzMjIzMzM5Mw==,9599,simonw,2020-05-21T17:16:15Z,2020-05-21T17:17:43Z,OWNER,"I just hit this bug myself, or at least a variant of it!
https://github.com/simonw/museums/runs/697063068?check_suite_focus=true
```
2020-05-21T17:14:02.2596471Z Traceback (most recent call last):
2020-05-21T17:14:02.2599146Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py"", line 605, in link_or_copy_directory
2020-05-21T17:14:02.2599861Z shutil.copytree(src, dst, copy_function=os.link)
2020-05-21T17:14:02.2600377Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py"", line 554, in copytree
2020-05-21T17:14:02.2600786Z return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
2020-05-21T17:14:02.2601196Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py"", line 510, in _copytree
2020-05-21T17:14:02.2601580Z raise Error(errors)
2020-05-21T17:14:02.2604565Z shutil.Error: [('/home/runner/work/museums/museums/templates/row-browse-museums.html', '/tmp/tmpq47xi96y/datasette/templates/row-browse-museums.html',
""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/row-browse-museums.html' -> '/tmp/tmpq47xi96y/datasette/templates/row-browse-museums.html'""), ('/home/runner/work/museums/museums/templates/index.html', '/tmp/tmpq47xi96y/datasette/templates/index.html',
""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/index.html' -> '/tmp/tmpq47xi96y/datasette/templates/index.html'""), ('/home/runner/work/museums/museums/templates/query-browse-search.html', '/tmp/tmpq47xi96y/datasette/templates/query-browse-search.html',
""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/query-browse-search.html' -> '/tmp/tmpq47xi96y/datasette/templates/query-browse-search.html'""), ('/home/runner/work/museums/museums/templates/_analytics.html', '/tmp/tmpq47xi96y/datasette/templates/_analytics.html', ""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/_analytics.html' -> '/tmp/tmpq47xi96y/datasette/templates/_analytics.html'""), ('/home/runner/work/museums/museums/templates/_museum_card.html', '/tmp/tmpq47xi96y/datasette/templates/_museum_card.html', ""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/_museum_card.html' -> '/tmp/tmpq47xi96y/datasette/templates/_museum_card.html'""), ('/home/runner/work/museums/museums/templates/pages/about.html', '/tmp/tmpq47xi96y/datasette/templates/pages/about.html', ""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/pages/about.html' -> '/tmp/tmpq47xi96y/datasette/templates/pages/about.html'""), ('/home/runner/work/museums/museums/templates/pages/map.html', '/tmp/tmpq47xi96y/datasette/templates/pages/map.html', ""[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/pages/map.html' -> '/tmp/tmpq47xi96y/datasette/templates/pages/map.html'"")]
2020-05-21T17:14:02.2605437Z
2020-05-21T17:14:02.2605797Z During handling of the above exception, another exception occurred:
2020-05-21T17:14:02.2606102Z
2020-05-21T17:14:02.2606423Z Traceback (most recent call last):
2020-05-21T17:14:02.2606817Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette"", line 8, in
2020-05-21T17:14:02.2607189Z sys.exit(cli())
2020-05-21T17:14:02.2607907Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
2020-05-21T17:14:02.2608347Z return self.main(*args, **kwargs)
2020-05-21T17:14:02.2609024Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 782, in main
2020-05-21T17:14:02.2609451Z rv = self.invoke(ctx)
2020-05-21T17:14:02.2610116Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
2020-05-21T17:14:02.2610550Z return _process_result(sub_ctx.command.invoke(sub_ctx))
2020-05-21T17:14:02.2611451Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
2020-05-21T17:14:02.2611989Z return _process_result(sub_ctx.command.invoke(sub_ctx))
2020-05-21T17:14:02.2612682Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
2020-05-21T17:14:02.2613117Z return ctx.invoke(self.callback, **ctx.params)
2020-05-21T17:14:02.2613801Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
2020-05-21T17:14:02.2614235Z return callback(*args, **kwargs)
2020-05-21T17:14:02.2614946Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/cloudrun.py"", line 111, in cloudrun
2020-05-21T17:14:02.2615371Z with temporary_docker_directory(
2020-05-21T17:14:02.2615769Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/contextlib.py"", line 113, in __enter__
2020-05-21T17:14:02.2616135Z return next(self.gen)
2020-05-21T17:14:02.2616957Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py"", line 392, in temporary_docker_directory
2020-05-21T17:14:02.2617403Z link_or_copy_directory(
2020-05-21T17:14:02.2618298Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py"", line 607, in link_or_copy_directory
2020-05-21T17:14:02.2619270Z shutil.copytree(src, dst)
2020-05-21T17:14:02.2619872Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py"", line 554, in copytree
2020-05-21T17:14:02.2622179Z return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
2020-05-21T17:14:02.2622609Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py"", line 455, in _copytree
2020-05-21T17:14:02.2622878Z os.makedirs(dst, exist_ok=dirs_exist_ok)
2020-05-21T17:14:02.2623157Z File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/os.py"", line 223, in makedirs
2020-05-21T17:14:02.2623405Z mkdir(name, mode)
2020-05-21T17:14:02.2623978Z FileExistsError: [Errno 17] File exists: '/tmp/tmpq47xi96y/datasette/templates'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631251707,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631251707,MDEyOklzc3VlQ29tbWVudDYzMTI1MTcwNw==,9599,simonw,2020-05-20T05:49:27Z,2020-05-21T15:58:42Z,MEMBER,Renaming this demo to `dogsheep-photos.dogsheep.net`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631127454,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631127454,MDEyOklzc3VlQ29tbWVudDYzMTEyNzQ1NA==,9599,simonw,2020-05-19T22:48:00Z,2020-05-21T15:58:32Z,MEMBER,"I built #23 to help with this.
$ dogsheep-photos create-subset photos.db public.db \
""select sha256 from apple_photos where albums like '%Public%'""
And publish with Vercel:
$ datasette publish now public.db --project dogsheep-photos \
--about=dogsheep/dogsheep-photos \
--about_url=""https://github.com/dogsheep/dogsheep-photos"" \
--install=datasette-json-html \
--install=datasette-cluster-map","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/24#issuecomment-631255206,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/24,631255206,MDEyOklzc3VlQ29tbWVudDYzMTI1NTIwNg==,9599,simonw,2020-05-20T06:00:25Z,2020-05-20T06:00:25Z,MEMBER,This needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621323348,Configurable URL for images,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253248,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631253248,MDEyOklzc3VlQ29tbWVudDYzMTI1MzI0OA==,9599,simonw,2020-05-20T05:54:18Z,2020-05-20T05:54:18Z,MEMBER,https://dogsheep-photos.dogsheep.net/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253136,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631253136,MDEyOklzc3VlQ29tbWVudDYzMTI1MzEzNg==,9599,simonw,2020-05-20T05:53:58Z,2020-05-20T05:53:58Z,MEMBER,"Updated deploy command:
```
datasette publish now public.db --project dogsheep-photos \
--about=dogsheep/dogsheep-photos \
--about_url=""https://github.com/dogsheep/dogsheep-photos"" \
--install=datasette-json-html \
--install=datasette-cluster-map \
--title ""Dogsheep Photos demo""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229485,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631229485,MDEyOklzc3VlQ29tbWVudDYzMTIyOTQ4NQ==,9599,simonw,2020-05-20T04:31:02Z,2020-05-20T04:31:02Z,MEMBER,https://pypi.org/project/dogsheep-photos/ is live.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631229409,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631229409,MDEyOklzc3VlQ29tbWVudDYzMTIyOTQwOQ==,9599,simonw,2020-05-20T04:30:40Z,2020-05-20T04:30:40Z,MEMBER,https://pypi.org/project/photos-to-sqlite/ now links to dogsheep-photos.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227245,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631227245,MDEyOklzc3VlQ29tbWVudDYzMTIyNzI0NQ==,9599,simonw,2020-05-20T04:21:38Z,2020-05-20T04:21:38Z,MEMBER,I'm going to release 0.4 now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227020,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631227020,MDEyOklzc3VlQ29tbWVudDYzMTIyNzAyMA==,9599,simonw,2020-05-20T04:20:48Z,2020-05-20T04:21:16Z,MEMBER,Next time I push a release it will create `dogsheep-photos` on PyPI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631227105,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631227105,MDEyOklzc3VlQ29tbWVudDYzMTIyNzEwNQ==,9599,simonw,2020-05-20T04:21:06Z,2020-05-20T04:21:06Z,MEMBER,Then I just need to push a final photos-to-sqlite release that updates the README to tell people about the name change.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226953,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631226953,MDEyOklzc3VlQ29tbWVudDYzMTIyNjk1Mw==,9599,simonw,2020-05-20T04:20:34Z,2020-05-20T04:20:34Z,MEMBER,"Huh, it looks like Circle CI picked up the name change automatically. https://app.circleci.com/pipelines/github/dogsheep/dogsheep-photos","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226572,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631226572,MDEyOklzc3VlQ29tbWVudDYzMTIyNjU3Mg==,9599,simonw,2020-05-20T04:18:52Z,2020-05-20T04:18:52Z,MEMBER,Need to reconfigure Circle CI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/26#issuecomment-631226481,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26,631226481,MDEyOklzc3VlQ29tbWVudDYzMTIyNjQ4MQ==,9599,simonw,2020-05-20T04:18:29Z,2020-05-20T04:18:29Z,MEMBER,I just renamed the repository.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621444763,Rename project to dogsheep-photos,
https://github.com/dogsheep/dogsheep-photos/issues/23#issuecomment-631120771,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/23,631120771,MDEyOklzc3VlQ29tbWVudDYzMTEyMDc3MQ==,9599,simonw,2020-05-19T22:32:48Z,2020-05-19T22:32:48Z,MEMBER,Documentation: https://github.com/dogsheep/photos-to-sqlite/blob/e2fab012551eed05278040b5d57e7373a1b9a0bf/README.md#creating-a-subset-database,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621280529,create-subset command for creating a publishable subset of a photos database,
https://github.com/simonw/sqlite-utils/issues/113#issuecomment-631084748,https://api.github.com/repos/simonw/sqlite-utils/issues/113,631084748,MDEyOklzc3VlQ29tbWVudDYzMTA4NDc0OA==,9599,simonw,2020-05-19T21:11:22Z,2020-05-19T21:11:22Z,OWNER,"Here's how `twitter-to-sqlite` does it: https://github.com/dogsheep/twitter-to-sqlite/blob/6be0ffcee24504fa2964b0e526842b8cfce7567b/twitter_to_sqlite/utils.py#L506-L511
```python
attach_sql = """"""
ATTACH DATABASE '{}' AS [{}];
"""""".format(
str(pathlib.Path(filepath).resolve()), alias
)
db.conn.execute(attach_sql)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621286870,Syntactic sugar for ATTACH DATABASE,
https://github.com/simonw/datasette/issues/729#issuecomment-630323870,https://api.github.com/repos/simonw/datasette/issues/729,630323870,MDEyOklzc3VlQ29tbWVudDYzMDMyMzg3MA==,9599,simonw,2020-05-18T17:20:02Z,2020-05-18T17:20:02Z,OWNER,"Deployed here: https://covid-19.datasettes.com/covid/economist_excess_deaths - and it helped me spot a bug! The floating point numbers there were being treated as strings:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629050775,https://api.github.com/repos/simonw/datasette/issues/729,629050775,MDEyOklzc3VlQ29tbWVudDYyOTA1MDc3NQ==,9599,simonw,2020-05-15T06:17:12Z,2020-05-15T06:17:12Z,OWNER,"
I don't like how the column headers themselves are no longer black in mobile view.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629043480,https://api.github.com/repos/simonw/datasette/issues/729,629043480,MDEyOklzc3VlQ29tbWVudDYyOTA0MzQ4MA==,9599,simonw,2020-05-15T05:55:44Z,2020-05-15T05:55:44Z,OWNER,Live demo: https://latest.datasette.io/fixtures/sortable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629042437,https://api.github.com/repos/simonw/datasette/issues/729,629042437,MDEyOklzc3VlQ29tbWVudDYyOTA0MjQzNw==,9599,simonw,2020-05-15T05:52:39Z,2020-05-15T05:52:39Z,OWNER,`type-pk` is a more accurate name than `type-link` - since people might assume that `type-link` refers to columns containing a link or a URL or similar.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629033753,https://api.github.com/repos/simonw/datasette/issues/729,629033753,MDEyOklzc3VlQ29tbWVudDYyOTAzMzc1Mw==,9599,simonw,2020-05-15T05:21:42Z,2020-05-15T05:21:51Z,OWNER,I'm going to make all integers AND floats show up as `#666666`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629033619,https://api.github.com/repos/simonw/datasette/issues/729,629033619,MDEyOklzc3VlQ29tbWVudDYyOTAzMzYxOQ==,9599,simonw,2020-05-15T05:21:17Z,2020-05-15T05:21:30Z,OWNER,"Turns out `1.0` and `2` stored in a float column already displays like that:
```
echo '[{""foo"": 1.0, ""bar"": 1}, {""foo"": 2, ""bar"": 2}]' | sqlite-utils insert /tmp/db.db t -
```
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629032619,https://api.github.com/repos/simonw/datasette/issues/729,629032619,MDEyOklzc3VlQ29tbWVudDYyOTAzMjYxOQ==,9599,simonw,2020-05-15T05:17:53Z,2020-05-15T05:18:29Z,OWNER,"I like the gray. The purple is a bit confusing.
I could have both integers and floats show up as gray? The floating point should help show which is which.
Maybe I could default floating point columns with integer values in them to `11.0` just to make it clear that they are floats?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/simonw/datasette/issues/729#issuecomment-629032425,https://api.github.com/repos/simonw/datasette/issues/729,629032425,MDEyOklzc3VlQ29tbWVudDYyOTAzMjQyNQ==,9599,simonw,2020-05-15T05:17:09Z,2020-05-15T05:17:39Z,OWNER,"Not sure the best way to do this. Here's an example using very subtle colour changes - but I'm worried it's not at all discoverable:
![]()
That's with `#4c0077` for floats and `#666666` for integers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603295970,Visually distinguish integer and text columns,
https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-628405453,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,628405453,MDEyOklzc3VlQ29tbWVudDYyODQwNTQ1Mw==,41546558,RhetTbull,2020-05-14T05:59:53Z,2020-05-14T05:59:53Z,CONTRIBUTOR,"I've added support for the above exif data to [v0.28.17](https://github.com/RhetTbull/osxphotos/releases/tag/v0.28.17) of osxphotos. `PhotoInfo.exif_info` will return an `ExifInfo` [dataclass](https://docs.python.org/3/library/dataclasses.html) object with the following properties:
```python
flash_fired: bool
iso: int
metering_mode: int
sample_rate: int
track_format: int
white_balance: int
aperture: float
bit_rate: float
duration: float
exposure_bias: float
focal_length: float
fps: float
latitude: float
longitude: float
shutter_speed: float
camera_make: str
camera_model: str
codec: str
lens_model: str
```
It's not all the EXIF data available in most files but is the data Photos deems important to save. Of course, you can get all the exif_data
Note: this only works in Photos 5. As best as I can tell, EXIF data is not stored in the database for earlier versions. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118,Try out ExifReader,
https://github.com/simonw/sqlite-utils/issues/112#issuecomment-627036749,https://api.github.com/repos/simonw/sqlite-utils/issues/112,627036749,MDEyOklzc3VlQ29tbWVudDYyNzAzNjc0OQ==,9599,simonw,2020-05-12T00:27:24Z,2020-05-12T00:27:24Z,OWNER,Here's an example of some code that would be cleaner with this mechanism: https://github.com/dogsheep/swarm-to-sqlite/blob/f4a82633da927cde672c9d9af92930bfca2e3ddf/swarm_to_sqlite/utils.py#L120-L143,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616271236,"add_foreign_key(...., ignore=True)",
https://github.com/simonw/sqlite-utils/issues/112#issuecomment-627036475,https://api.github.com/repos/simonw/sqlite-utils/issues/112,627036475,MDEyOklzc3VlQ29tbWVudDYyNzAzNjQ3NQ==,9599,simonw,2020-05-12T00:26:25Z,2020-05-12T00:26:48Z,OWNER,"Question: if you use `ignore=True` should it still raise an error if one of the columns you referenced does not exist?
That's tricky. I'm leaning towards ""that's still an error"".
Are we ignoring an already existing foreign key, or ignoring all errors?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616271236,"add_foreign_key(...., ignore=True)",
https://github.com/simonw/sqlite-utils/issues/112#issuecomment-627035928,https://api.github.com/repos/simonw/sqlite-utils/issues/112,627035928,MDEyOklzc3VlQ29tbWVudDYyNzAzNTkyOA==,9599,simonw,2020-05-12T00:24:16Z,2020-05-12T00:24:51Z,OWNER,"I would add `ignore=True/False` as a parameter here:
https://github.com/simonw/sqlite-utils/blob/af3f81b540923f2cf04c76cfa81b0d811c0084bf/sqlite_utils/db.py#L731-L761","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616271236,"add_foreign_key(...., ignore=True)",
https://github.com/simonw/sqlite-utils/issues/112#issuecomment-627036010,https://api.github.com/repos/simonw/sqlite-utils/issues/112,627036010,MDEyOklzc3VlQ29tbWVudDYyNzAzNjAxMA==,9599,simonw,2020-05-12T00:24:37Z,2020-05-12T00:24:37Z,OWNER,"Here's how it works for `create_view`:
https://github.com/simonw/sqlite-utils/blob/af3f81b540923f2cf04c76cfa81b0d811c0084bf/sqlite_utils/db.py#L327-L336","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616271236,"add_foreign_key(...., ignore=True)",
https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-627007458,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,627007458,MDEyOklzc3VlQ29tbWVudDYyNzAwNzQ1OA==,41546558,RhetTbull,2020-05-11T22:51:52Z,2020-05-11T22:52:26Z,CONTRIBUTOR,"I'm not familiar with `ExifReader`. I wrote my own wrapper around `exiftool` because I wanted a simple way to write EXIF data when exporting photos (e.g. writing out to PersonInImage and keywords to IPTC:Keywords) and the existing python packages like [pyexiftool](https://github.com/smarnach/pyexiftool) didn't do quite what I wanted. If all you're after is the camera and shot info, that's available in `ZEXTENDEDATTRIBUTES` table. I've got an open issue [#11](https://github.com/RhetTbull/osxphotos/issues/11) to add this to osxphotos but it hasn't bubbled to the top of my backlog yet.
osxphotos will give you the location info: `PhotoInfo.location` returns a tuple of (lat, lon) though this info is in ZEXTENDEDATTRIBUTES too (though it might not be correct as I believe Photos creates this table at import and the user might have changed the location of a photo, e.g. if camera didn't have GPS).
```sql
CREATE TABLE ZEXTENDEDATTRIBUTES (
Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER,
Z_OPT INTEGER, ZFLASHFIRED INTEGER,
ZISO INTEGER, ZMETERINGMODE INTEGER,
ZSAMPLERATE INTEGER, ZTRACKFORMAT INTEGER,
ZWHITEBALANCE INTEGER, ZASSET INTEGER,
ZAPERTURE FLOAT, ZBITRATE FLOAT, ZDURATION FLOAT,
ZEXPOSUREBIAS FLOAT, ZFOCALLENGTH FLOAT,
ZFPS FLOAT, ZLATITUDE FLOAT, ZLONGITUDE FLOAT,
ZSHUTTERSPEED FLOAT, ZCAMERAMAKE VARCHAR,
ZCAMERAMODEL VARCHAR, ZCODEC VARCHAR,
ZLENSMODEL VARCHAR
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118,Try out ExifReader,
https://github.com/simonw/datasette/issues/699#issuecomment-626991001,https://api.github.com/repos/simonw/datasette/issues/699,626991001,MDEyOklzc3VlQ29tbWVudDYyNjk5MTAwMQ==,8431341,zeluspudding,2020-05-11T22:06:34Z,2020-05-11T22:06:34Z,NONE,Very nice! Thank you for sharing that :+1: :) Will try it out!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-626945281,https://api.github.com/repos/simonw/datasette/issues/699,626945281,MDEyOklzc3VlQ29tbWVudDYyNjk0NTI4MQ==,9599,simonw,2020-05-11T20:32:33Z,2020-05-11T20:32:33Z,OWNER,"I did have a bit of trouble with this one-off plugin getting it to load in the correct order - since I need authentication to work if EITHER the one-off plugin spots a token or my `datasette-auth-github` plugin authenticates the user.
That's why I want authentication as a core Datasette concept - so plugins like these can easily play together in a predictable manner.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-626943809,https://api.github.com/repos/simonw/datasette/issues/699,626943809,MDEyOklzc3VlQ29tbWVudDYyNjk0MzgwOQ==,9599,simonw,2020-05-11T20:30:07Z,2020-05-11T20:31:18Z,OWNER,"I implemented bearer tokens in a private project of mine as a one-off plugin. I'm going to extract that out into a installable plugin soon. For the moment, my `plugins/token_auth.py` file looks like this:
```python
from datasette import hookimpl
import secrets
class TokenAuth:
def __init__(
self, app, secret, auth,
):
self.app = app
self.secret = secret
self.auth = auth
async def __call__(self, scope, receive, send):
if scope.get(""type"") != ""http"":
return await self.app(scope, receive, send)
authorization = dict(scope.get(""headers"") or {}).get(b""authorization"") or b""""
expected = ""Bearer {}"".format(self.secret).encode(""utf8"")
if secrets.compare_digest(authorization, expected):
scope = dict(scope, auth=self.auth)
return await self.app(scope, receive, send)
@hookimpl(trylast=True)
def asgi_wrapper(datasette):
config = datasette.plugin_config(""token-auth"") or {}
secret = config.get(""secret"")
auth = config.get(""auth"")
def wrap_with_asgi_auth(app):
return TokenAuth(app, secret=secret, auth=auth,)
return wrap_with_asgi_auth
```
Then I have the following in `metadata.json`:
```json
{
""plugins"": {
""token-auth"": {
""auth"": {
""name"": ""token-bot""
},
""secret"": {
""$env"": ""TOKEN_SECRET""
}
}
}
}
```
And a `TOKEN_SECRET` environment variable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-626941278,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,626941278,MDEyOklzc3VlQ29tbWVudDYyNjk0MTI3OA==,9599,simonw,2020-05-11T20:25:58Z,2020-05-11T20:25:58Z,MEMBER,"Interesting - do you know if there's anything the `exiftool` process handles that `ExifReader` doesn't?
I'm actually just going to extract a subset of the EXIF data at first - since the original photo files will always be available I don't feel the need to get everything out for the first step.
My plan is to use EXIF to help support photo collections that aren't in Apple Photos - I'm going to build a database table keyed by the `sha256` of each photo that extracts the camera make, lens, a few settings (ISO, aperture etc) and the GPS lat/lon.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118,Try out ExifReader,
https://github.com/simonw/sqlite-utils/issues/30#issuecomment-626903632,https://api.github.com/repos/simonw/sqlite-utils/issues/30,626903632,MDEyOklzc3VlQ29tbWVudDYyNjkwMzYzMg==,9599,simonw,2020-05-11T19:17:17Z,2020-05-11T19:17:17Z,OWNER,I don't think this is a useful feature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",461215118,Option to open database in read-only mode,
https://github.com/simonw/datasette/issues/765#issuecomment-626874374,https://api.github.com/repos/simonw/datasette/issues/765,626874374,MDEyOklzc3VlQ29tbWVudDYyNjg3NDM3NA==,9599,simonw,2020-05-11T18:25:43Z,2020-05-11T18:25:43Z,OWNER,"Trickiness here is what tag to use. Do we use the tag of the installed copy of Datasette that is running `datasette publish`? That would mean you don't get the latest features in your deployed release.
Could hit an API to figure out the most recent version? Bit odd.
Could we output the version of Datasette that was deployed and tell people they can run `--latest` to force the most recent release?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616087149,publish heroku should default to currently tagged version,
https://github.com/simonw/datasette/issues/764#issuecomment-626813924,https://api.github.com/repos/simonw/datasette/issues/764,626813924,MDEyOklzc3VlQ29tbWVudDYyNjgxMzkyNA==,9599,simonw,2020-05-11T16:36:06Z,2020-05-11T16:36:06Z,OWNER,Made a TIL: https://github.com/simonw/til/blob/master/pypi/project-links.md,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616012427,Add PyPI project urls to setup.py,
https://github.com/simonw/datasette/issues/764#issuecomment-626810129,https://api.github.com/repos/simonw/datasette/issues/764,626810129,MDEyOklzc3VlQ29tbWVudDYyNjgxMDEyOQ==,9599,simonw,2020-05-11T16:28:43Z,2020-05-11T16:28:43Z,OWNER,"For Datasette I'll go with:
* Documentation
* Changelog
* Live demo
* Source code
* Issues
* CI","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616012427,Add PyPI project urls to setup.py,
https://github.com/simonw/datasette/issues/764#issuecomment-626808805,https://api.github.com/repos/simonw/datasette/issues/764,626808805,MDEyOklzc3VlQ29tbWVudDYyNjgwODgwNQ==,9599,simonw,2020-05-11T16:26:17Z,2020-05-11T16:27:27Z,OWNER,"The keys here can be anything: https://packaging.python.org/guides/distributing-packages-using-setuptools/#project-urls
So where do the icons come from? Turns out the PyPI site has special case rules for the icons here: https://github.com/pypa/warehouse/blob/2f00f4a9f208546ff0ebb6a6e61439021ca60a43/warehouse/templates/packaging/detail.html#L16-L60","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",616012427,Add PyPI project urls to setup.py,
https://github.com/simonw/datasette/issues/699#issuecomment-626807487,https://api.github.com/repos/simonw/datasette/issues/699,626807487,MDEyOklzc3VlQ29tbWVudDYyNjgwNzQ4Nw==,8431341,zeluspudding,2020-05-11T16:23:57Z,2020-05-11T16:24:59Z,NONE,`Authorization: bearer xxx` auth for API keys is a plus plus for me. Looked into just adding this into your `Flask` logic but learned this project doesn't use flask. Interesting 🤔,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/dogsheep/dogsheep-photos/issues/22#issuecomment-626667235,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22,626667235,MDEyOklzc3VlQ29tbWVudDYyNjY2NzIzNQ==,41546558,RhetTbull,2020-05-11T12:20:34Z,2020-05-11T12:20:34Z,CONTRIBUTOR,"@simonw FYI, osxphotos includes a built in ExifTool class that uses [exiftool](https://exiftool.org/) to read and write exif data. It's not exposed yet in the docs because I really only use it right now in the osphotos command line interface to write tags when exporting. In v0.28.16 (just pushed) I added an ExifTool.as_dict() method which will give you a dict with all the exif tags in a file. For example:
```python
import osxphotos
photos = osxphotos.PhotosDB().photos()
exiftool = osxphotos.exiftool.ExifTool(photos[0].path)
exifdata = exiftool.as_dict()
tags = exifdata[""IPTC:Keywords""]
```
Not as elegant perhaps as a python only implementation because ExifTool has to make subprocess calls to an external tool but exiftool is by far the best tool available for reading and writing EXIF data and it does support HEIC.
As for implementation, ExifTool uses a singleton pattern so the first time you instantiate it, it spawns an IPC to exiftool but then keeps it open and uses the same process for any subsequent calls (even on different files). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615626118,Try out ExifReader,
https://github.com/simonw/sqlite-utils/issues/111#issuecomment-626431562,https://api.github.com/repos/simonw/sqlite-utils/issues/111,626431562,MDEyOklzc3VlQ29tbWVudDYyNjQzMTU2Mg==,9599,simonw,2020-05-11T01:58:36Z,2020-05-11T01:58:36Z,OWNER,Released in 2.9 https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v2-9,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615477131,sqlite-utils drop-table and drop-view commands,
https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626431484,https://api.github.com/repos/simonw/sqlite-utils/issues/110,626431484,MDEyOklzc3VlQ29tbWVudDYyNjQzMTQ4NA==,9599,simonw,2020-05-11T01:58:20Z,2020-05-11T01:58:20Z,OWNER,Released in 2.9 https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v2-9,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613755043,Support decimal.Decimal type,
https://github.com/simonw/sqlite-utils/issues/111#issuecomment-626417220,https://api.github.com/repos/simonw/sqlite-utils/issues/111,626417220,MDEyOklzc3VlQ29tbWVudDYyNjQxNzIyMA==,9599,simonw,2020-05-11T00:46:04Z,2020-05-11T00:46:04Z,OWNER,"Docs:
* https://sqlite-utils.readthedocs.io/en/latest/cli.html#dropping-tables
* https://sqlite-utils.readthedocs.io/en/latest/cli.html#dropping-views","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615477131,sqlite-utils drop-table and drop-view commands,
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626396379,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626396379,MDEyOklzc3VlQ29tbWVudDYyNjM5NjM3OQ==,41546558,RhetTbull,2020-05-10T22:01:48Z,2020-05-10T22:01:48Z,CONTRIBUTOR,"Frustrates me when package authors create a ""drop in"" replacement with the same import name...this kind of thing has bitten me more than once! Would've been nicer I think for bpylist2 to do ""import bpylist2 as bpylist""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395781,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395781,MDEyOklzc3VlQ29tbWVudDYyNjM5NTc4MQ==,9599,simonw,2020-05-10T21:57:09Z,2020-05-10T21:57:09Z,MEMBER,"Yes, I just recreated my virtual environment from scratch and the error went away.
The problem occurred when I ran `pip install datasette-bplist` in the same virtual environment - https://github.com/simonw/datasette-bplist/blob/master/setup.py depends on `bpylist` which is incompatible with `bpylist2`.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395641,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395641,MDEyOklzc3VlQ29tbWVudDYyNjM5NTY0MQ==,41546558,RhetTbull,2020-05-10T21:55:54Z,2020-05-10T21:55:54Z,CONTRIBUTOR,Did removing old bpylist solve the original problem or do you still have a photo that throws circular reference?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395507,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395507,MDEyOklzc3VlQ29tbWVudDYyNjM5NTUwNw==,41546558,RhetTbull,2020-05-10T21:54:45Z,2020-05-10T21:54:45Z,CONTRIBUTOR,"@simonw does Photos show valid reverse geolocation info? Are you sure you're using [bpylist2](https://github.com/xa4a/bpylist2) and not bpylist? They're both unfortunately imported as ""bpylist"" so if you somehow got the wrong (original bpylist) version installed, it could be the issue. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395209,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395209,MDEyOklzc3VlQ29tbWVudDYyNjM5NTIwOQ==,9599,simonw,2020-05-10T21:52:42Z,2020-05-10T21:52:42Z,MEMBER,"Aha! It looks like I accidentally installed the old bplist into the same environment:
```
$ pip freeze | grep bpylist
bpylist==0.1.4
bpylist2==3.0.0
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626395103,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626395103,MDEyOklzc3VlQ29tbWVudDYyNjM5NTEwMw==,9599,simonw,2020-05-10T21:51:36Z,2020-05-10T21:51:36Z,MEMBER,"@RhetTbull I tried that workaround and it turns out I'm getting this error on ALL of my photos now!
It's weird: a few day ago this wasn't happening. Now it's happening to everything. I'm not sure what I might have changed. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626390317,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626390317,MDEyOklzc3VlQ29tbWVudDYyNjM5MDMxNw==,41546558,RhetTbull,2020-05-10T21:11:24Z,2020-05-10T21:50:58Z,CONTRIBUTOR,"Ugh....Yeah, I think easiest is to catch the exception and return no place as you suggest. This particular bit of code involves un-archiving a serialized NSKeyedArchiver which uses an object table and it is certainly possible to create a circular reference that way. Because this is happening in the decode, the circular reference must be in the original data. Does Photos show valid reverse geolocation info for the photo in question? If so, Photos may be doing something beyond a simple decode of the binary plist. For now, I'll push a patch to catch the exception.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626394989,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626394989,MDEyOklzc3VlQ29tbWVudDYyNjM5NDk4OQ==,9599,simonw,2020-05-10T21:50:36Z,2020-05-10T21:50:36Z,MEMBER,https://github.com/Marketcircle/bpylist/pull/2 looks relevant here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391307,https://api.github.com/repos/simonw/sqlite-utils/issues/110,626391307,MDEyOklzc3VlQ29tbWVudDYyNjM5MTMwNw==,9599,simonw,2020-05-10T21:19:04Z,2020-05-10T21:19:04Z,OWNER,"I'm going to set it up so that Python `decimal.Decimal` is treated in a FLOAT column, until someone convinces me otherwise!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613755043,Support decimal.Decimal type,
https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391217,https://api.github.com/repos/simonw/sqlite-utils/issues/110,626391217,MDEyOklzc3VlQ29tbWVudDYyNjM5MTIxNw==,9599,simonw,2020-05-10T21:18:28Z,2020-05-10T21:18:28Z,OWNER,"`sqlite-utils` currently treats the SQLite NUMERIC concept as a float:
https://github.com/simonw/sqlite-utils/blob/daf2a245aa4e0b0cf62a94c1232cfb858821803b/tests/test_column_affinity.py#L28-L30","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613755043,Support decimal.Decimal type,
https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391063,https://api.github.com/repos/simonw/sqlite-utils/issues/110,626391063,MDEyOklzc3VlQ29tbWVudDYyNjM5MTA2Mw==,9599,simonw,2020-05-10T21:17:16Z,2020-05-10T21:17:16Z,OWNER,"From https://www.sqlite.org/datatype3.html#type_affinity :
> A column with NUMERIC affinity may contain values using all five storage classes. When text data is inserted into a NUMERIC column, the storage class of the text is converted to INTEGER or REAL (in order of preference) if the text is a well-formed integer or real literal, respectively. If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL. For conversions between TEXT and REAL storage classes, only the first 15 significant decimal digits of the number are preserved. If the TEXT value is not a well-formed integer or real literal, then the value is stored as TEXT.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613755043,Support decimal.Decimal type,
https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626390822,https://api.github.com/repos/simonw/sqlite-utils/issues/110,626390822,MDEyOklzc3VlQ29tbWVudDYyNjM5MDgyMg==,9599,simonw,2020-05-10T21:15:28Z,2020-05-10T21:15:28Z,OWNER,"https://www.sqlite.org/datatype3.html#affinity_name_examples suggests that `DECIMAL(10,5)` should be mapped to the SQLite affinity of `NUMERIC` - which I've not worked with before.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613755043,Support decimal.Decimal type,
https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626390456,https://api.github.com/repos/simonw/sqlite-utils/issues/110,626390456,MDEyOklzc3VlQ29tbWVudDYyNjM5MDQ1Ng==,9599,simonw,2020-05-10T21:12:40Z,2020-05-10T21:12:40Z,OWNER,"It definitely makes sense to me that this library should support `decimal.Decimal`.
Here are the current supported types:
https://github.com/simonw/sqlite-utils/blob/396bee92364fc3a88f6c76969366dd1c4c9c944d/sqlite_utils/db.py#L35-L55
Should `decimal.Decimal` be stored as a `text` or `float` in SQLite I wonder?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613755043,Support decimal.Decimal type,
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388837,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626388837,MDEyOklzc3VlQ29tbWVudDYyNjM4ODgzNw==,9599,simonw,2020-05-10T20:59:32Z,2020-05-10T20:59:32Z,MEMBER,So it appears it's possible for `photo.place` to raise that exception. A workaround could be to catch that and treat those photos as not having a place.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-626388764,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,626388764,MDEyOklzc3VlQ29tbWVudDYyNjM4ODc2NA==,9599,simonw,2020-05-10T20:58:52Z,2020-05-10T20:58:52Z,MEMBER,"More from the debugger:
```
> /Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py(614)place()
-> self._place = PlaceInfo5(self._info[""reverse_geolocation""])
```
And:
```
> /Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/utils.py(91)osxphoto_to_row()
-> place = photo.place
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/simonw/datasette/issues/254#issuecomment-626340387,https://api.github.com/repos/simonw/datasette/issues/254,626340387,MDEyOklzc3VlQ29tbWVudDYyNjM0MDM4Nw==,247131,philroche,2020-05-10T14:54:13Z,2020-05-10T14:54:13Z,NONE,"This has now been resolved and is not present in current version of datasette.
Sample query @simonw mentioned now returns as expected.
https://aggreg8streams.tinyviking.ie/simplestreams?sql=select+*+from+cloudimage+where+%22content_id%22+%3D+%22com.ubuntu.cloud%3Areleased%3Adownload%22+order+by+id+limit+10","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",322283067,Escaping named parameters in canned queries,
https://github.com/simonw/datasette/issues/619#issuecomment-626006493,https://api.github.com/repos/simonw/datasette/issues/619,626006493,MDEyOklzc3VlQ29tbWVudDYyNjAwNjQ5Mw==,412005,davidszotten,2020-05-08T20:29:12Z,2020-05-08T20:29:12Z,NONE,"just trying out datasette and quite like it, thanks! i found this issue annoying enough to have a go at a fix. have you any thoughts on a good approach? (i'm happy to dig in myself if you haven't thought about it yet, but wanted to check if you had an idea for how to fix when you raised the issue)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-625947133,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,625947133,MDEyOklzc3VlQ29tbWVudDYyNTk0NzEzMw==,9599,simonw,2020-05-08T18:13:06Z,2020-05-08T18:13:06Z,MEMBER,`datasette-media` will be able to handle this once I implement https://github.com/simonw/datasette-media/issues/3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/simonw/datasette/issues/685#issuecomment-625890190,https://api.github.com/repos/simonw/datasette/issues/685,625890190,MDEyOklzc3VlQ29tbWVudDYyNTg5MDE5MA==,9599,simonw,2020-05-08T16:09:44Z,2020-05-08T16:09:56Z,OWNER,Re-opening this ticket because I forgot to document `execute_fn()` (the old `execute_against_connection_in_thread()` method).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625889673,https://api.github.com/repos/simonw/datasette/issues/685,625889673,MDEyOklzc3VlQ29tbWVudDYyNTg4OTY3Mw==,9599,simonw,2020-05-08T16:08:40Z,2020-05-08T16:08:40Z,OWNER,I'm going to ship a release with just this change purely so I can start depending on it from my in-development https://github.com/simonw/datasette-media plugin.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625889303,https://api.github.com/repos/simonw/datasette/issues/685,625889303,MDEyOklzc3VlQ29tbWVudDYyNTg4OTMwMw==,9599,simonw,2020-05-08T16:07:55Z,2020-05-08T16:07:55Z,OWNER,New documentation is live here: https://datasette.readthedocs.io/en/latest/internals.html#database-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625887196,https://api.github.com/repos/simonw/datasette/issues/685,625887196,MDEyOklzc3VlQ29tbWVudDYyNTg4NzE5Ng==,9599,simonw,2020-05-08T16:03:16Z,2020-05-08T16:03:16Z,OWNER,[ Fun aside: I implemented and shipped this entire branch in my browser using the beta of GitHub's new [Codespaces](https://github.com/features/codespaces) ],"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625868416,https://api.github.com/repos/simonw/datasette/issues/685,625868416,MDEyOklzc3VlQ29tbWVudDYyNTg2ODQxNg==,9599,simonw,2020-05-08T15:23:04Z,2020-05-08T15:23:04Z,OWNER,I'll move the code into `datasette/database.py`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625866419,https://api.github.com/repos/simonw/datasette/issues/685,625866419,MDEyOklzc3VlQ29tbWVudDYyNTg2NjQxOQ==,9599,simonw,2020-05-08T15:18:55Z,2020-05-08T15:18:55Z,OWNER,"This code should live somewhere other than `datasette/utils/__init__.py`. Especially the exceptions, since calling code needs to be able to sensibly import them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625865434,https://api.github.com/repos/simonw/datasette/issues/685,625865434,MDEyOklzc3VlQ29tbWVudDYyNTg2NTQzNA==,9599,simonw,2020-05-08T15:16:43Z,2020-05-08T15:16:43Z,OWNER,Documentation for improved design: https://github.com/simonw/datasette/pull/763/files#diff-87703da03a02e0f8dae01f39abd6250f,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625862578,https://api.github.com/repos/simonw/datasette/issues/685,625862578,MDEyOklzc3VlQ29tbWVudDYyNTg2MjU3OA==,9599,simonw,2020-05-08T15:10:16Z,2020-05-08T15:10:16Z,OWNER,I'm not going to do ``.last()`` because it could be confusing if truncation has come into play.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625850152,https://api.github.com/repos/simonw/datasette/issues/685,625850152,MDEyOklzc3VlQ29tbWVudDYyNTg1MDE1Mg==,9599,simonw,2020-05-08T14:44:37Z,2020-05-08T14:44:37Z,OWNER,"I'll write the API documentation first, in a branch.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625849497,https://api.github.com/repos/simonw/datasette/issues/685,625849497,MDEyOklzc3VlQ29tbWVudDYyNTg0OTQ5Nw==,9599,simonw,2020-05-08T14:43:13Z,2020-05-08T14:43:13Z,OWNER,"I'm going to add `.first()` and `.last()` methods too. These will return the first or last row, or `None` if the results are empty (rather than raising an `IndexError`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625841397,https://api.github.com/repos/simonw/datasette/issues/685,625841397,MDEyOklzc3VlQ29tbWVudDYyNTg0MTM5Nw==,9599,simonw,2020-05-08T14:25:56Z,2020-05-08T14:27:15Z,OWNER,"Maybe all I really want here is to add length and access-by-index methods to the Results class? So I don't have to do `results.rows[0][0]` to get at a single returned value.
Also how about a `results.single_value()` method which returns a value only if there is a single row with a single column, otherwise raises an exception?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625839046,https://api.github.com/repos/simonw/datasette/issues/685,625839046,MDEyOklzc3VlQ29tbWVudDYyNTgzOTA0Ng==,9599,simonw,2020-05-08T14:20:43Z,2020-05-08T14:25:05Z,OWNER,"The API design for the `.execute()` function is actually fine, I think it's more the API of the returned `Results` object that I want to improve.
That object encapsulates the returned data, the named columns and whether or not the result was truncated.
return Results(rows, truncated, cursor.description)
The `rows` argument comes from either `rows = cursor.fetchmany(max_returned_rows + 1)` or `rows = cursor.fetchall()`. It's a Python list of `sqlite3.Row` objects.
Here's the current class implementation:
https://github.com/simonw/datasette/blob/0784f2ef9d3ff6dd9df05f54cb51de29a6d11764/datasette/utils/__init__.py#L54-L68","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625817999,https://api.github.com/repos/simonw/datasette/issues/685,625817999,MDEyOklzc3VlQ29tbWVudDYyNTgxNzk5OQ==,9599,simonw,2020-05-08T13:33:23Z,2020-05-08T13:33:23Z,OWNER,I'm going to rename `execute_against_connection_in_thread()` to just `execute_fn()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/685#issuecomment-625603221,https://api.github.com/repos/simonw/datasette/issues/685,625603221,MDEyOklzc3VlQ29tbWVudDYyNTYwMzIyMQ==,9599,simonw,2020-05-08T03:00:23Z,2020-05-08T03:00:23Z,OWNER,Working with this in https://github.com/simonw/datasette-media/issues/2 made me really want to redesign this API: https://github.com/simonw/datasette-media/commit/be23ec3d05900b69a3d47bc1e0dd056c333125f4,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570309546,Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread(),
https://github.com/simonw/datasette/issues/648#issuecomment-625321121,https://api.github.com/repos/simonw/datasette/issues/648,625321121,MDEyOklzc3VlQ29tbWVudDYyNTMyMTEyMQ==,28694175,chekos,2020-05-07T15:21:19Z,2020-05-07T15:21:19Z,NONE,"It seems that heroku wasn't updating to 0.41 on deployment.
Had to add `--branch 0.41` and that solved it! Heroku caches dependencies
![]()
and (i think) because the `requirements.txt` doesn't specify the datasette version, it didn't update from 0.40 to 0.41 on heroku even though it was specified on my local requirements file as `datasette >= 0.41`
These are the lines that gave me an idea on how to solve it:
https://github.com/simonw/datasette/blob/182e5c8745c94576718315f7596ccc81e5e2417b/datasette/publish/heroku.py#L164-L186","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534492501,Mechanism for adding arbitrary pages like /about,
https://github.com/simonw/datasette/issues/648#issuecomment-625286519,https://api.github.com/repos/simonw/datasette/issues/648,625286519,MDEyOklzc3VlQ29tbWVudDYyNTI4NjUxOQ==,28694175,chekos,2020-05-07T14:23:22Z,2020-05-07T14:28:33Z,NONE,"Hi! I'm using datasette on this repository: https://github.com/chekos/RIPA-2018-datasette
and on my local machine i can see an /about page i created but when i deploy to heroku i get a 404 (http://ripa-2018-db.herokuapp.com)
![]()
I bumped datasette in my requirements file to 0.41 so I'm 100% what the issue is 🤔
Do you have any idea what could be the problem? 👀
EDIT: for context, I have a templates directory with a pages/about.html file in https://github.com/chekos/RIPA-2018-datasette/tree/master/datasette/templates","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534492501,Mechanism for adding arbitrary pages like /about,
https://github.com/simonw/datasette/issues/744#issuecomment-625091976,https://api.github.com/repos/simonw/datasette/issues/744,625091976,MDEyOklzc3VlQ29tbWVudDYyNTA5MTk3Ng==,30607,aborruso,2020-05-07T07:51:25Z,2020-05-07T07:51:25Z,NONE,"I have installed `heroku plugins:install heroku-builds`, but I have the same error.
Then I have removed from `datasette\publish\heroku.py`
```python
# Check for heroku-builds plugin
plugins = [
line.split()[0] for line in check_output([""heroku"", ""plugins""]).splitlines()
]
if b""heroku-builds"" not in plugins:
click.echo(
""Publishing to Heroku requires the heroku-builds plugin to be installed.""
)
click.confirm(
""Install it? (this will run `heroku plugins:install heroku-builds`)"",
abort=True,
)
call([""heroku"", ""plugins:install"", ""heroku-builds""])
```
And now I have
```
Traceback (most recent call last):
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py"", line 210, in temporary_heroku_directory
yield
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py"", line 96, in heroku
list_output = check_output([""heroku"", ""apps:list"", ""--json""]).decode(
File ""c:\python37\lib\subprocess.py"", line 395, in check_output
**kwargs).stdout
File ""c:\python37\lib\subprocess.py"", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File ""c:\python37\lib\subprocess.py"", line 775, in __init__
restore_signals, start_new_session)
File ""c:\python37\lib\subprocess.py"", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The specified file could not be found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ""c:\python37\lib\runpy.py"", line 193, in _run_module_as_main
""__main__"", mod_spec)
File ""c:\python37\lib\runpy.py"", line 85, in _run_code
exec(code, run_globals)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\Scripts\datasette.exe\__main__.py"", line 9, in
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 782, in main
rv = self.invoke(ctx)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py"", line 120, in heroku
call([""heroku"", ""builds:create"", ""-a"", app_name, ""--include-vcs-ignore""])
File ""c:\python37\lib\contextlib.py"", line 130, in __exit__
self.gen.throw(type, value, traceback)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py"", line 213, in temporary_heroku_directory
tmp.cleanup()
File ""c:\python37\lib\tempfile.py"", line 809, in cleanup
_shutil.rmtree(self.name)
File ""c:\python37\lib\shutil.py"", line 513, in rmtree
return _rmtree_unsafe(path, onerror)
File ""c:\python37\lib\shutil.py"", line 401, in _rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File ""c:\python37\lib\shutil.py"", line 399, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 32] Unable to access file. The file is being used by another process: 'C:\\Users\\aborr\\AppData\\Local\\Temp\\tmpkcxy8i_q'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-625083715,https://api.github.com/repos/simonw/datasette/issues/744,625083715,MDEyOklzc3VlQ29tbWVudDYyNTA4MzcxNQ==,30607,aborruso,2020-05-07T07:34:18Z,2020-05-07T07:34:18Z,NONE,"In Windows I'm not very strong. I use debian (inside WSL).
However these are the possible steps:
- I have installed Python 3 for win (I have 3.7.3);
- I have installed heroku cli for win64 and logged in;
- I have installed datasette running `python -m pip install --upgrade --user datasette`.
It's a very basic Python env that I do not use. This time only to reach my goal: try to publish using custom template","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-625066876,https://api.github.com/repos/simonw/datasette/issues/744,625066876,MDEyOklzc3VlQ29tbWVudDYyNTA2Njg3Ng==,9599,simonw,2020-05-07T06:55:04Z,2020-05-07T06:55:04Z,OWNER,I have a Windows 10 gaming PC in the house: what steps can I take to get an environment in there that matches yours? I've not installed Python on Windows before.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-625066073,https://api.github.com/repos/simonw/datasette/issues/744,625066073,MDEyOklzc3VlQ29tbWVudDYyNTA2NjA3Mw==,30607,aborruso,2020-05-07T06:53:09Z,2020-05-07T06:53:09Z,NONE,"@simonw another error starting from Windows.
I run
```
datasette publish heroku -n comunepa --template-dir template commissioniComunePalermo.db
```
And I have
```
Traceback (most recent call last):
File ""c:\python37\lib\runpy.py"", line 193, in _run_module_as_main
""__main__"", mod_spec)
File ""c:\python37\lib\runpy.py"", line 85, in _run_code
exec(code, run_globals)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\Scripts\datasette.exe\__main__.py"", line 9, in
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 782, in main
rv = self.invoke(ctx)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py"", line 53, in heroku
line.split()[0] for line in check_output([""heroku"", ""plugins""]).splitlines()
File ""c:\python37\lib\subprocess.py"", line 395, in check_output
**kwargs).stdout
File ""c:\python37\lib\subprocess.py"", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File ""c:\python37\lib\subprocess.py"", line 775, in __init__
restore_signals, start_new_session)
File ""c:\python37\lib\subprocess.py"", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The specified file could not be found
```
[files.zip](https://github.com/simonw/datasette/files/4591173/files.zip)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-625061250,https://api.github.com/repos/simonw/datasette/issues/744,625061250,MDEyOklzc3VlQ29tbWVudDYyNTA2MTI1MA==,9599,simonw,2020-05-07T06:40:09Z,2020-05-07T06:40:09Z,OWNER,Have a try and let me know what happens! I'd like this stuff to just work on Windows but I need to figure out how to get an environment working.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-625060561,https://api.github.com/repos/simonw/datasette/issues/744,625060561,MDEyOklzc3VlQ29tbWVudDYyNTA2MDU2MQ==,30607,aborruso,2020-05-07T06:38:24Z,2020-05-07T06:38:24Z,NONE,"Hi @simonw probably I could try to do it in Python for windows. I do not like to do these things in win enviroment.
Because probably WSL Linux env (in which I do a lot of great things) is not an environment that will be tested for datasette.
In win I shouldn't have any problems. Am I right?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/761#issuecomment-625003013,https://api.github.com/repos/simonw/datasette/issues/761,625003013,MDEyOklzc3VlQ29tbWVudDYyNTAwMzAxMw==,9599,simonw,2020-05-07T03:07:55Z,2020-05-07T03:09:05Z,OWNER,"Here's another demo I built using this feature: https://observablehq.com/@simonw/datasette-table-diagram
![]()
Running on this query:
```sql
select
sqlite_master.name as table_from,
fk_info.[from] as column_from,
fk_info.[table] as table_to,
fk_info.[to] as column_to
from
sqlite_master
join pragma_foreign_key_list(sqlite_master.name) as fk_info
order by
sqlite_master.name
```
See https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++sqlite_master.name+as+table_from%2C%0D%0A++fk_info.%5Bfrom%5D+as+column_from%2C%0D%0A++fk_info.%5Btable%5D+as+table_to%2C%0D%0A++fk_info.%5Bto%5D+as+column_to%0D%0Afrom%0D%0A++sqlite_master%0D%0A++join+pragma_foreign_key_list%28sqlite_master.name%29+as+fk_info%0D%0Aorder+by%0D%0A++sqlite_master.name","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613467382,Allow-list pragma_table_info(tablename) and similar,
https://github.com/simonw/datasette/issues/760#issuecomment-624949809,https://api.github.com/repos/simonw/datasette/issues/760,624949809,MDEyOklzc3VlQ29tbWVudDYyNDk0OTgwOQ==,9599,simonw,2020-05-06T23:49:06Z,2020-05-06T23:49:06Z,OWNER,"```sql
select
sqlite_master.name as table_name,
table_info.*
from
sqlite_master
join pragma_table_info(sqlite_master.name) as table_info
order by
sqlite_master.name,
table_info.cid
```
https://latest.datasette.io/fixtures?sql=select%0D%0A++sqlite_master.name+as+table_name%2C%0D%0A++table_info.*%0D%0Afrom%0D%0A++sqlite_master%0D%0A++join+pragma_table_info%28sqlite_master.name%29+as+table_info%0D%0Aorder+by%0D%0A++sqlite_master.name%2C%0D%0A++table_info.cid","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613422636,Way of seeing full schema for a database,
https://github.com/simonw/datasette/issues/759#issuecomment-624860451,https://api.github.com/repos/simonw/datasette/issues/759,624860451,MDEyOklzc3VlQ29tbWVudDYyNDg2MDQ1MQ==,133845,Krazybug,2020-05-06T20:03:01Z,2020-05-06T20:04:42Z,NONE,"Thank you. Now it's ok with the url
http://localhost:8001/index/summary?_search=language%3Aeng&_sort=title&_searchmode=raw
But I'm not able to manage it in the metadata file. Here is mine (note that the sort column is taken into account)
Here it is:
```
{
""databases"": {
""index"": {
""tables"": {
""summary"": {
""sort"": ""title"",
""searchmode"": ""raw""
}
}
}
}
}
```
Any idea ?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612673948,fts search on a column doesn't work anymore due to escape_fts,
https://github.com/simonw/datasette/issues/757#issuecomment-624821090,https://api.github.com/repos/simonw/datasette/issues/757,624821090,MDEyOklzc3VlQ29tbWVudDYyNDgyMTA5MA==,9599,simonw,2020-05-06T18:41:29Z,2020-05-06T18:41:29Z,OWNER,"OK, I just released 0.41 with that and a bunch of other stuff: https://datasette.readthedocs.io/en/latest/changelog.html#v0-41","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612378203,Question: Any fixed date for the release with the uft8-encoding fix?,
https://github.com/simonw/datasette/issues/757#issuecomment-624798540,https://api.github.com/repos/simonw/datasette/issues/757,624798540,MDEyOklzc3VlQ29tbWVudDYyNDc5ODU0MA==,9599,simonw,2020-05-06T17:56:34Z,2020-05-06T17:56:34Z,OWNER,Actually I'm going to put that release out today. I was hoping to finish #698 first but that shouldn't delay those other features any longer.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612378203,Question: Any fixed date for the release with the uft8-encoding fix?,
https://github.com/simonw/datasette/issues/757#issuecomment-624798182,https://api.github.com/repos/simonw/datasette/issues/757,624798182,MDEyOklzc3VlQ29tbWVudDYyNDc5ODE4Mg==,9599,simonw,2020-05-06T17:55:50Z,2020-05-06T17:55:50Z,OWNER,"I'll definitely get that out this week!
For the moment a trick I often use is to put a URL to the most recent commit in my `requirements.txt` - e.g. https://github.com/simonw/datasette/archive/0784f2ef9d3ff6dd9df05f54cb51de29a6d11764.zip
This should be safe because nothing lands on Datasette master without the full unit test suite passing. But you're right, there's a bunch of stuff now that needs to go out in a release: https://github.com/simonw/datasette/compare/0.40...0784f2ef9d3ff6dd9df05f54cb51de29a6d11764","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612378203,Question: Any fixed date for the release with the uft8-encoding fix?,
https://github.com/simonw/datasette/issues/758#issuecomment-624797119,https://api.github.com/repos/simonw/datasette/issues/758,624797119,MDEyOklzc3VlQ29tbWVudDYyNDc5NzExOQ==,9599,simonw,2020-05-06T17:53:46Z,2020-05-06T17:53:46Z,OWNER,It's interesting to hear from someone who's using this feature - I'm considering moving it out into a plugin #647.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612382643,Question: Access to immutable database-path,
https://github.com/simonw/datasette/issues/758#issuecomment-624796685,https://api.github.com/repos/simonw/datasette/issues/758,624796685,MDEyOklzc3VlQ29tbWVudDYyNDc5NjY4NQ==,9599,simonw,2020-05-06T17:52:54Z,2020-05-06T17:52:54Z,OWNER,Thanks for the suggestion! I'll add this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612382643,Question: Access to immutable database-path,
https://github.com/simonw/datasette/issues/759#issuecomment-624795695,https://api.github.com/repos/simonw/datasette/issues/759,624795695,MDEyOklzc3VlQ29tbWVudDYyNDc5NTY5NQ==,9599,simonw,2020-05-06T17:50:57Z,2020-05-06T17:52:07Z,OWNER,"This was a deliberate change from #651.
The `_search_colname=` alternative argument for doing this still works - compare these two:
* https://latest.datasette.io/fixtures/searchable?_search=dog
* https://latest.datasette.io/fixtures/searchable?_search_tex1=dog
If you want to use advanced search syntax on those pages you can do so using the `&_searchmode=raw` option - I added better documentation for this the other day: https://datasette.readthedocs.io/en/latest/full_text_search.html#advanced-sqlite-search-queries
Example: https://latest.datasette.io/fixtures/searchable?_search=text1:dog&_searchmode=raw","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612673948,fts search on a column doesn't work anymore due to escape_fts,
https://github.com/simonw/datasette/issues/761#issuecomment-624790887,https://api.github.com/repos/simonw/datasette/issues/761,624790887,MDEyOklzc3VlQ29tbWVudDYyNDc5MDg4Nw==,9599,simonw,2020-05-06T17:41:21Z,2020-05-06T17:41:21Z,OWNER,More demos here: https://github.com/simonw/til/blob/master/sqlite/list-all-columns-in-a-database.md,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613467382,Allow-list pragma_table_info(tablename) and similar,
https://github.com/simonw/datasette/issues/760#issuecomment-624787678,https://api.github.com/repos/simonw/datasette/issues/760,624787678,MDEyOklzc3VlQ29tbWVudDYyNDc4NzY3OA==,9599,simonw,2020-05-06T17:35:05Z,2020-05-06T17:35:05Z,OWNER,Potential recipe in https://github.com/simonw/til/blob/master/sqlite/list-all-columns-in-a-database.md,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613422636,Way of seeing full schema for a database,
https://github.com/simonw/datasette/issues/761#issuecomment-624783996,https://api.github.com/repos/simonw/datasette/issues/761,624783996,MDEyOklzc3VlQ29tbWVudDYyNDc4Mzk5Ng==,9599,simonw,2020-05-06T17:28:20Z,2020-05-06T17:28:20Z,OWNER,"Interestingly https://latest.datasette.io/fixtures?sql=select+*+from+pragma_function_list() doesn't work, when it DOES work on my laptop.
`latest.datasette.io` currently runs SQLite `3.27.2` while my laptop runs `3.31.1`
https://www.sqlite.org/changes.html#version_3_30_0 says that as-of 3.30.0:
> The PRAGMA function_list, PRAGMA module_list, and PRAGMA pragma_list commands are now enabled in all builds by default. Disable them using -DSQLITE_OMIT_INTROSPECTION_PRAGMAS. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613467382,Allow-list pragma_table_info(tablename) and similar,
https://github.com/simonw/datasette/issues/761#issuecomment-624782775,https://api.github.com/repos/simonw/datasette/issues/761,624782775,MDEyOklzc3VlQ29tbWVudDYyNDc4Mjc3NQ==,9599,simonw,2020-05-06T17:26:05Z,2020-05-06T17:26:05Z,OWNER,"Some demos:
* https://latest.datasette.io/fixtures?sql=select+*+from+pragma_database_list%28%29
* https://latest.datasette.io/fixtures?sql=select+*+from+pragma_foreign_key_list%28%27complex_foreign_keys%27%29
* https://latest.datasette.io/fixtures?sql=select+*+from+pragma_function_list%28%29
* https://latest.datasette.io/fixtures?sql=select+*+from+pragma_index_info%28%27idx_compound_three_primary_keys_content%27%29
* https://latest.datasette.io/fixtures?sql=select+*+from+pragma_index_list%28%27compound_three_primary_keys%27%29
* https://latest.datasette.io/fixtures?sql=select+*+from+pragma_index_xinfo%28%27idx_compound_three_primary_keys_content%27%29
* https://latest.datasette.io/fixtures?sql=select+*+from+pragma_page_count%28%29
* https://latest.datasette.io/fixtures?sql=select+*+from+pragma_max_page_count%28%29
* https://latest.datasette.io/fixtures?sql=select+*+from+pragma_page_size%28%29
* https://latest.datasette.io/fixtures?sql=select+*+from+pragma_schema_version%28%29
* https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_info%28%27complex_foreign_keys%27%29
* https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_xinfo%28%27complex_foreign_keys%27%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613467382,Allow-list pragma_table_info(tablename) and similar,
https://github.com/simonw/datasette/issues/761#issuecomment-624774928,https://api.github.com/repos/simonw/datasette/issues/761,624774928,MDEyOklzc3VlQ29tbWVudDYyNDc3NDkyOA==,9599,simonw,2020-05-06T17:11:15Z,2020-05-06T17:11:15Z,OWNER,"For the moment I'll allow-list the following:
* `pragma_database_list`
* `pragma_foreign_key_list`
* `pragma_function_list`
* `pragma_index_info`
* `pragma_index_list`
* `pragma_index_xinfo`
* `pragma_page_count`
* `pragma_max_page_count`
* `pragma_page_size`
* `pragma_schema_version`
* `pragma_table_info`
* `pragma_table_xinfo`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613467382,Allow-list pragma_table_info(tablename) and similar,
https://github.com/simonw/datasette/issues/761#issuecomment-624766424,https://api.github.com/repos/simonw/datasette/issues/761,624766424,MDEyOklzc3VlQ29tbWVudDYyNDc2NjQyNA==,9599,simonw,2020-05-06T16:54:38Z,2020-05-06T17:01:02Z,OWNER,"I could allow-list some other useful `pragma_x` tables too.
SQLite calls these ""pragma functions"" - documented here: https://www.sqlite.org/pragma.html#pragfunc
They sound safe:
> Table-valued functions exist only for PRAGMAs that return results and that have no side-effects. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613467382,Allow-list pragma_table_info(tablename) and similar,
https://github.com/simonw/datasette/issues/761#issuecomment-624768744,https://api.github.com/repos/simonw/datasette/issues/761,624768744,MDEyOklzc3VlQ29tbWVudDYyNDc2ODc0NA==,9599,simonw,2020-05-06T16:59:01Z,2020-05-06T16:59:01Z,OWNER,"Maybe use a negative lookahead assertion? https://docs.python.org/3/library/re.html#index-20
> `(?!...)`
>
> Matches if `...` doesn’t match next. This is a negative lookahead assertion. For example, `Isaac (?!Asimov)` will match 'Isaac ' only if it’s not followed by 'Asimov'.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613467382,Allow-list pragma_table_info(tablename) and similar,
https://github.com/simonw/datasette/issues/761#issuecomment-624767466,https://api.github.com/repos/simonw/datasette/issues/761,624767466,MDEyOklzc3VlQ29tbWVudDYyNDc2NzQ2Ng==,9599,simonw,2020-05-06T16:56:40Z,2020-05-06T16:57:03Z,OWNER,"The rationale for blocking `pragma` entirely from statements is that it can be used to change the state of the SQLite database, e.g. from https://www.sqlite.org/pragma.html :
```
PRAGMA schema.application_id;
PRAGMA schema.application_id = integer ;
```
That second line is unsafe. I don't think it's possible to use the `pragma_table_x` variants to make writes in this way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613467382,Allow-list pragma_table_info(tablename) and similar,
https://github.com/simonw/datasette/issues/760#issuecomment-624729459,https://api.github.com/repos/simonw/datasette/issues/760,624729459,MDEyOklzc3VlQ29tbWVudDYyNDcyOTQ1OQ==,9599,simonw,2020-05-06T15:47:44Z,2020-05-06T15:47:44Z,OWNER,"`select * from pragma_table_info(tablename);` is currently disallowed for user-provided queries via a regex restriction - but could help here too.
https://github.com/simonw/datasette/blob/d349d57cdf3d577afb62bdf784af342a4d5be660/datasette/utils/__init__.py#L174","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613422636,Way of seeing full schema for a database,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408738,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,624408738,MDEyOklzc3VlQ29tbWVudDYyNDQwODczOA==,9599,simonw,2020-05-06T02:21:05Z,2020-05-06T02:21:32Z,MEMBER,"Here's rendering code from my hacked-together not-yet-released S3 image proxy:
```python
from starlette.responses import Response
from PIL import Image, ExifTags
import pyheif
for ORIENTATION_TAG in ExifTags.TAGS.keys():
if ExifTags.TAGS[ORIENTATION_TAG] == ""Orientation"":
break
...
# Load it into Pillow
if ext == ""heic"":
heic = pyheif.read_heif(image_response.content)
image = Image.frombytes(mode=heic.mode, size=heic.size, data=heic.data)
else:
image = Image.open(io.BytesIO(image_response.content))
# Does EXIF tell us to rotate it?
try:
exif = dict(image._getexif().items())
if exif[ORIENTATION_TAG] == 3:
image = image.rotate(180, expand=True)
elif exif[ORIENTATION_TAG] == 6:
image = image.rotate(270, expand=True)
elif exif[ORIENTATION_TAG] == 8:
image = image.rotate(90, expand=True)
except (AttributeError, KeyError, IndexError):
pass
# Resize based on ?w= and ?h=, if set
width, height = image.size
w = request.query_params.get(""w"")
h = request.query_params.get(""h"")
if w is not None or h is not None:
if h is None:
# Set h based on w
w = int(w)
h = int((float(height) / width) * w)
elif w is None:
h = int(h)
# Set w based on h
w = int((float(width) / height) * h)
w = int(w)
h = int(h)
image.thumbnail((w, h))
# ?bw= converts to black and white
if request.query_params.get(""bw""):
image = image.convert(""L"")
# ?q= sets the quality - defaults to 75
quality = 75
q = request.query_params.get(""q"")
if q and q.isdigit() and 1 <= int(q) <= 100:
quality = int(q)
# Output as JPEG or PNG
output_image = io.BytesIO()
image_type = ""JPEG""
kwargs = {""quality"": quality}
if image.format == ""PNG"":
image_type = ""PNG""
kwargs = {}
image.save(output_image, image_type, **kwargs)
return Response(
output_image.getvalue(),
media_type=""image/jpeg"",
headers={""cache-control"": ""s-maxage={}, public"".format(365 * 24 * 60 * 60)},
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408370,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,624408370,MDEyOklzc3VlQ29tbWVudDYyNDQwODM3MA==,9599,simonw,2020-05-06T02:19:27Z,2020-05-06T02:19:27Z,MEMBER,"The plugin can be generalized: it can be configured to know how to take the URL path, look it up in ANY table (via a custom SQL query) to get a path on disk and then serve that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-624408220,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,624408220,MDEyOklzc3VlQ29tbWVudDYyNDQwODIyMA==,9599,simonw,2020-05-06T02:18:47Z,2020-05-06T02:18:47Z,MEMBER,"The `apple_photos` table has an indexed `uuid` column and a `path` column which stores the full path to that photo file on disk.
I can write a custom Datasette plugin which takes the `uuid` from the URL, looks up the path, then serves up a thumbnail of the jpeg or heic image file.
I'll prototype this is a one-off plugin first, then package it on PyPI for other people to install.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/19#issuecomment-624406285,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/19,624406285,MDEyOklzc3VlQ29tbWVudDYyNDQwNjI4NQ==,9599,simonw,2020-05-06T02:10:03Z,2020-05-06T02:10:03Z,MEMBER,"Most annoying part of this is the difficulty of actually showing a photo.
Maybe I need to run a local proxy that I can link to? A custom Datasette plugin perhaps?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613002220,apple-photos command should work even if upload has not run,
https://github.com/dogsheep/dogsheep-photos/issues/18#issuecomment-624364557,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18,624364557,MDEyOklzc3VlQ29tbWVudDYyNDM2NDU1Nw==,9599,simonw,2020-05-05T23:49:18Z,2020-05-05T23:49:18Z,MEMBER,Label is `macos-latest`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612860758,Switch CI solution to GitHub Actions with a macOS runner,
https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624284539,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17,624284539,MDEyOklzc3VlQ29tbWVudDYyNDI4NDUzOQ==,41546558,RhetTbull,2020-05-05T20:20:05Z,2020-05-05T20:20:05Z,CONTRIBUTOR,"FYI, I've got an [issue](https://github.com/RhetTbull/osxphotos/issues/25) to make osxphotos cross-platform but it's low on my priority list. About 90% of the functionality could be done cross-platform but right now the MacOS specific stuff is embedded throughout and would take some work. Though I try to minimize it, there's sprinklings of ObjC & Applescript throughout osxphotos.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612860531,Only install osxphotos if running on macOS,
https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624278714,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17,624278714,MDEyOklzc3VlQ29tbWVudDYyNDI3ODcxNA==,9599,simonw,2020-05-05T20:07:19Z,2020-05-05T20:07:19Z,MEMBER,"From https://hynek.me/articles/conditional-python-dependencies/ I think this will look like:
```python
setup(
# ...
install_requires=[
# ...
""osxphotos>=0.28.13 ; sys_platform=='darwin'"",
]
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612860531,Only install osxphotos if running on macOS,
https://github.com/dogsheep/dogsheep-photos/issues/17#issuecomment-624278090,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17,624278090,MDEyOklzc3VlQ29tbWVudDYyNDI3ODA5MA==,9599,simonw,2020-05-05T20:06:01Z,2020-05-05T20:06:01Z,MEMBER,https://www.python.org/dev/peps/pep-0508/#environment-markers I think I want `sys_platform` of `darwin`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612860531,Only install osxphotos if running on macOS,
https://github.com/simonw/sqlite-utils/issues/109#issuecomment-624099524,https://api.github.com/repos/simonw/sqlite-utils/issues/109,624099524,MDEyOklzc3VlQ29tbWVudDYyNDA5OTUyNA==,9599,simonw,2020-05-05T14:46:53Z,2020-05-05T14:46:53Z,OWNER,"This already exists, it's just called `if_not_exists=True`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612658444,"table.create_index(..., ignore=True)",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623865250,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623865250,MDEyOklzc3VlQ29tbWVudDYyMzg2NTI1MA==,9599,simonw,2020-05-05T05:38:16Z,2020-05-05T05:38:16Z,MEMBER,It looks like `groups.content_string` often has a null byte in it. I should clean this up as part of the import.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623863902,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623863902,MDEyOklzc3VlQ29tbWVudDYyMzg2MzkwMg==,9599,simonw,2020-05-05T05:31:53Z,2020-05-05T05:31:53Z,MEMBER,"Yes! Turning those `rowid` values into `id` with this script did the job:
```python
import sqlite3
import sqlite_utils
conn = sqlite3.connect(
""/Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite""
)
def all_rows(table):
result = conn.execute(""select rowid as id, * from {}"".format(table))
cols = [c[0] for c in result.description]
for row in result.fetchall():
yield dict(zip(cols, row))
if __name__ == ""__main__"":
db = sqlite_utils.Database(""psi_copy.db"")
for table in (""assets"", ""collections"", ""ga"", ""gc"", ""groups""):
db[table].upsert_all(all_rows(table), pk=""id"", alter=True)
```
Then I ran this query:
```sql
select
json_object('img_src', 'https://photos.simonwillison.net/i/' || photos.sha256 || '.' || photos.ext || '?w=400') as photo,
group_concat(strip_null_chars(groups.content_string), ' ') as words, assets.uuid_0, assets.uuid_1, to_uuid(assets.uuid_0, assets.uuid_1) as uuid
from assets join ga on assets.id = ga.assetid
join groups on ga.groupid = groups.id
join photos on photos.uuid = to_uuid(assets.uuid_0, assets.uuid_1)
where groups.category = 2024
group by assets.id
order by random() limit 10
```
And got these results!
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623857417,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623857417,MDEyOklzc3VlQ29tbWVudDYyMzg1NzQxNw==,9599,simonw,2020-05-05T05:01:47Z,2020-05-05T05:01:47Z,MEMBER,"Even that didn't work - it didn't copy across the rowid values. I'm pretty sure that's what's wrong here:
```
sqlite3 /Users/simon/Pictures/Photos\ Library.photoslibrary/database/search/psi.sqlite 'select rowid, uuid_0, uuid_1 from assets limit 10'
1619605|-9205353363298198838|4814875488794983828
1641378|-9205348195631362269|390804289838822030
1634974|-9205331524553603243|-3834026796261633148
1619083|-9205326176986145401|7563404215614709654
22131|-9205315724827218763|8370531509591906734
1645633|-9205247376092758131|-1311540150497601346
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855885,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623855885,MDEyOklzc3VlQ29tbWVudDYyMzg1NTg4NQ==,9599,simonw,2020-05-05T04:54:39Z,2020-05-05T04:54:53Z,MEMBER,"Trying this import mechanism instead:
`sqlite3 /Users/simon/Pictures/Photos\ Library.photoslibrary/database/search/psi.sqlite .dump | grep -v 'CREATE INDEX' | grep -v 'CREATE TRIGGER' | grep -v 'CREATE VIRTUAL TABLE' | sqlite3 search.db`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623855841,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623855841,MDEyOklzc3VlQ29tbWVudDYyMzg1NTg0MQ==,9599,simonw,2020-05-05T04:54:28Z,2020-05-05T04:54:28Z,MEMBER,"Things were not matching up for me correctly:
![]()
I think that's because my import script didn't correctly import the existing `rowid` values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623846880,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623846880,MDEyOklzc3VlQ29tbWVudDYyMzg0Njg4MA==,9599,simonw,2020-05-05T04:06:08Z,2020-05-05T04:06:08Z,MEMBER,"This function seems to convert them into UUIDs that match my photos:
```python
def to_uuid(uuid_0, uuid_1):
b = uuid_0.to_bytes(8, 'little', signed=True) + uuid_1.to_bytes(8, 'little', signed=True)
return str(uuid.UUID(bytes=b)).upper()
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623845014,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623845014,MDEyOklzc3VlQ29tbWVudDYyMzg0NTAxNA==,41546558,RhetTbull,2020-05-05T03:55:14Z,2020-05-05T03:56:24Z,CONTRIBUTOR,"I'm traveling w/o access to my Mac so can't help with any code right now. I suspected ZSCENEIDENTIFIER was a foreign key into one of these psi.sqlite tables. But looks like you're on to something connecting groups to assets. As for the UUID, I think there's two ints because each is 64-bits but UUIDs are 128-bits. Thus they need to be combined to get the 128 bit UUID. You might be able to use Apple's [NSUUID](https://developer.apple.com/documentation/foundation/nsuuid?language=objc), for example, by wrapping with pyObjC. Here's one [example](https://github.com/ronaldoussoren/pyobjc/blob/881c82a7ba90f193934b52b44143360c80dce5e5/pyobjc-framework-Cocoa/PyObjCTest/test_nsuuid.py) of using this in PyObjC's test suite. Interesting it's stored this way instead of a UUIDString as in Photos.sqlite. Perhaps it for faster indexing.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623811131,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623811131,MDEyOklzc3VlQ29tbWVudDYyMzgxMTEzMQ==,9599,simonw,2020-05-05T03:16:18Z,2020-05-05T03:16:18Z,MEMBER,"Here's how to convert two integers unto a UUID using Java. Not sure if it's the solution I need though (or how to do the same thing in Python):
https://repl.it/repls/EuphoricSomberClasslibrary
![]()
```java
import java.util.UUID;
class Main {
public static void main(String[] args) {
java.util.UUID uuid = new java.util.UUID(
2544182952487526660L,
-3640314103732024685L
);
System.out.println(
uuid
);
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623807568,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623807568,MDEyOklzc3VlQ29tbWVudDYyMzgwNzU2OA==,9599,simonw,2020-05-05T02:56:06Z,2020-05-05T02:56:06Z,MEMBER,"I'm pretty sure this is what I'm after. The `groups` table has what looks like identified labels in the rows with category = 2025:
![]()
Then there's a `ga` table that maps groups to assets:
![]()
And an `assets` table which looks like it has one row for every one of my photos:
![]()
One major challenge: these UUIDs are split into two integer numbers, `uuid_0` and `uuid_1` - but the main photos database uses regular UUIDs like this:
I need to figure out how to match up these two different UUID representations. I asked on Twitter if anyone has any ideas: https://twitter.com/simonw/status/1257500689019703296","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806687,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623806687,MDEyOklzc3VlQ29tbWVudDYyMzgwNjY4Nw==,9599,simonw,2020-05-05T02:51:16Z,2020-05-05T02:51:16Z,MEMBER,"Running datasette against it directly doesn't work:
```
simon@Simons-MacBook-Pro search % datasette psi.sqlite
Serve! files=('psi.sqlite',) (immutables=()) on port 8001
Usage: datasette serve [OPTIONS] [FILES]...
Error: Connection to psi.sqlite failed check: no such tokenizer: PSITokenizer
```
Instead, I created a new SQLite database with a copy of some of the key tables, like this:
```
sqlite-utils rows psi.sqlite groups | sqlite-utils insert /tmp/search.db groups -
sqlite-utils rows psi.sqlite assets | sqlite-utils insert /tmp/search.db assets -
sqlite-utils rows psi.sqlite ga | sqlite-utils insert /tmp/search.db ga -
sqlite-utils rows psi.sqlite collections | sqlite-utils insert /tmp/search.db collections -
sqlite-utils rows psi.sqlite gc | sqlite-utils insert /tmp/search.db gc -
sqlite-utils rows psi.sqlite lookup | sqlite-utils insert /tmp/search.db lookup -
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806533,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623806533,MDEyOklzc3VlQ29tbWVudDYyMzgwNjUzMw==,9599,simonw,2020-05-05T02:50:16Z,2020-05-05T02:50:16Z,MEMBER,"I figured there must be a separate database that Photos uses to store the text of the identified labels.
I used ""Open Files and Ports"" in Activity Monitor against the Photos app to try and spot candidates... and found `/Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite` - a 53MB SQLite database file.
![]()
Here's the schema of that file:
```
$ sqlite3 psi.sqlite .schema
CREATE TABLE word_embedding(word TEXT, extended_word TEXT, score DOUBLE);
CREATE INDEX word_embedding_index ON word_embedding(word);
CREATE VIRTUAL TABLE word_embedding_prefix USING fts5(extended_word)
/* word_embedding_prefix(extended_word) */;
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_content'(id INTEGER PRIMARY KEY, c0);
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE IF NOT EXISTS 'word_embedding_prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID;
CREATE TABLE groups(category INT2, owning_groupid INT, content_string TEXT, normalized_string TEXT, lookup_identifier TEXT, token_ranges_0 INT8, token_ranges_1 INT8, UNIQUE(category, owning_groupid, content_string, lookup_identifier, token_ranges_0, token_ranges_1));
CREATE TABLE assets(uuid_0 INT, uuid_1 INT, creationDate INT, UNIQUE(uuid_0, uuid_1));
CREATE TABLE ga(groupid INT, assetid INT, PRIMARY KEY(groupid, assetid));
CREATE TABLE collections(uuid_0 INT, uuid_1 INT, startDate INT, endDate INT, title TEXT, subtitle TEXT, keyAssetUUID_0 INT, keyAssetUUID_1 INT, typeAndNumberOfAssets INT32, sortDate DOUBLE, UNIQUE(uuid_0, uuid_1));
CREATE TABLE gc(groupid INT, collectionid INT, PRIMARY KEY(groupid, collectionid));
CREATE VIRTUAL TABLE prefix USING fts5(content='groups', normalized_string, category UNINDEXED, tokenize = 'PSITokenizer');
CREATE TABLE IF NOT EXISTS 'prefix_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE IF NOT EXISTS 'prefix_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'prefix_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE IF NOT EXISTS 'prefix_config'(k PRIMARY KEY, v) WITHOUT ROWID;
CREATE TABLE lookup(identifier TEXT PRIMARY KEY, category INT2);
CREATE TRIGGER trigger_groups_insert AFTER INSERT ON groups BEGIN INSERT INTO prefix(rowid, normalized_string, category) VALUES (new.rowid, new.normalized_string, new.category); END;
CREATE TRIGGER trigger_groups_delete AFTER DELETE ON groups BEGIN INSERT INTO prefix(prefix, rowid, normalized_string, category) VALUES('delete', old.rowid, old.normalized_string, old.category); END;
CREATE INDEX group_pk ON groups(category, content_string, normalized_string, lookup_identifier);
CREATE INDEX asset_pk ON assets(uuid_0, uuid_1);
CREATE INDEX ga_assetid ON ga(assetid, groupid);
CREATE INDEX collection_pk ON collections(uuid_0, uuid_1);
CREATE INDEX gc_collectionid ON gc(collectionid);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623806085,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623806085,MDEyOklzc3VlQ29tbWVudDYyMzgwNjA4NQ==,9599,simonw,2020-05-05T02:47:18Z,2020-05-05T02:47:18Z,MEMBER,"In https://github.com/RhetTbull/osxphotos/issues/121#issuecomment-623249263 Rhet Turnbull spotted a table called `ZSCENEIDENTIFIER` which looked like it might have the right data, but the columns in it aren't particularly helpful:
```
Z_PK,Z_ENT,Z_OPT,ZSCENEIDENTIFIER,ZASSETATTRIBUTES,ZCONFIDENCE
8,49,1,731,5,0.11834716796875
9,49,1,684,6,0.0233648251742125
10,49,1,1702,1,0.026153564453125
```
I love the look of those confidence scores, but what do the numbers mean?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/16#issuecomment-623805823,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16,623805823,MDEyOklzc3VlQ29tbWVudDYyMzgwNTgyMw==,9599,simonw,2020-05-05T02:45:56Z,2020-05-05T02:45:56Z,MEMBER,I filed an issue with `osxphotos` about this here: https://github.com/RhetTbull/osxphotos/issues/121,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612287234,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623739934,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,623739934,MDEyOklzc3VlQ29tbWVudDYyMzczOTkzNA==,9599,simonw,2020-05-04T22:24:26Z,2020-05-04T22:24:26Z,MEMBER,Twitter thread with some examples of photos that are coming up from queries against these scores: https://twitter.com/simonw/status/1257434670750408705,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623730934,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,623730934,MDEyOklzc3VlQ29tbWVudDYyMzczMDkzNA==,9599,simonw,2020-05-04T22:00:38Z,2020-05-04T22:00:48Z,MEMBER,"Here's the query to create the new table:
```sql
create table apple_photos_scores as select
ZGENERICASSET.ZUUID,
ZGENERICASSET.ZOVERALLAESTHETICSCORE,
ZGENERICASSET.ZCURATIONSCORE,
ZGENERICASSET.ZPROMOTIONSCORE,
ZGENERICASSET.ZHIGHLIGHTVISIBILITYSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZBEHAVIORALSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZFAILURESCORE,
ZCOMPUTEDASSETATTRIBUTES.ZHARMONIOUSCOLORSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZIMMERSIVENESSSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZINTERACTIONSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZINTERESTINGSUBJECTSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZINTRUSIVEOBJECTPRESENCESCORE,
ZCOMPUTEDASSETATTRIBUTES.ZLIVELYCOLORSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZLOWLIGHT,
ZCOMPUTEDASSETATTRIBUTES.ZNOISESCORE,
ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCAMERATILTSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTCOMPOSITIONSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTLIGHTINGSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPATTERNSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPERSPECTIVESCORE,
ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTPOSTPROCESSINGSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTREFLECTIONSSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZPLEASANTSYMMETRYSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZSHARPLYFOCUSEDSUBJECTSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZTASTEFULLYBLURREDSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZWELLCHOSENSUBJECTSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZWELLFRAMEDSUBJECTSCORE,
ZCOMPUTEDASSETATTRIBUTES.ZWELLTIMEDSHOTSCORE
from
attached.ZGENERICASSET
join attached.ZCOMPUTEDASSETATTRIBUTES on
attached.ZGENERICASSET.Z_PK = attached.ZCOMPUTEDASSETATTRIBUTES.Z_PK;
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723687,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,623723687,MDEyOklzc3VlQ29tbWVudDYyMzcyMzY4Nw==,9599,simonw,2020-05-04T21:43:06Z,2020-05-04T21:43:06Z,MEMBER,It looks like I can map the photos I'm importing to these tables using the `ZUUID` column on `ZGENERICASSET` to get a `Z_PK` which then maps to the rows in `ZGENERICASSET`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-623723026,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,623723026,MDEyOklzc3VlQ29tbWVudDYyMzcyMzAyNg==,9599,simonw,2020-05-04T21:41:30Z,2020-05-04T21:41:30Z,MEMBER,"I'm going to put these in a table called `apple_photos_scores` - I'll also pull in the following columns from the `ZGENERICASSET` table:
* `ZOVERALLAESTHETICSCORE`
* `ZCURATIONSCORE`
* `ZHIGHLIGHTVISIBILITYSCORE`
* `ZPROMOTIONSCORE`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/simonw/datasette/issues/756#issuecomment-623654436,https://api.github.com/repos/simonw/datasette/issues/756,623654436,MDEyOklzc3VlQ29tbWVudDYyMzY1NDQzNg==,9599,simonw,2020-05-04T19:19:06Z,2020-05-04T19:19:06Z,OWNER,New documentation is here: https://datasette.readthedocs.io/en/latest/installation.html#install-using-pipx,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612089949,Add pipx to installation documentation,
https://github.com/simonw/datasette/issues/756#issuecomment-623650324,https://api.github.com/repos/simonw/datasette/issues/756,623650324,MDEyOklzc3VlQ29tbWVudDYyMzY1MDMyNA==,9599,simonw,2020-05-04T19:10:23Z,2020-05-04T19:10:23Z,OWNER,https://github.com/pipxproject/pipx/issues/79 showed me how to upgrade plugins within the pipx virtual environment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612089949,Add pipx to installation documentation,
https://github.com/simonw/datasette/issues/755#issuecomment-623635525,https://api.github.com/repos/simonw/datasette/issues/755,623635525,MDEyOklzc3VlQ29tbWVudDYyMzYzNTUyNQ==,9599,simonw,2020-05-04T18:40:15Z,2020-05-04T18:40:15Z,OWNER,"```
$ pytest tests/test_database.py -v
================================================================================ test session starts =================================================================================
platform darwin -- Python 3.7.7, pytest-5.2.4, py-1.8.1, pluggy-0.13.1 -- /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/python
cachedir: .pytest_cache
rootdir: /Users/simon/Dropbox/Development/datasette, inifile: pytest.ini
plugins: asyncio-0.10.0
collected 9 items
tests/test_database.py::test_table_exists[tables0-True] PASSED [ 11%]
tests/test_database.py::test_table_exists[tables1-False] PASSED [ 22%]
tests/test_database.py::test_get_all_foreign_keys PASSED [ 33%]
tests/test_database.py::test_table_names PASSED [ 44%]
tests/test_database.py::test_execute_write_block_true PASSED [ 55%]
tests/test_database.py::test_execute_write_block_false PASSED [ 66%]
tests/test_database.py::test_execute_write_fn_block_false no such column: id
PASSED [ 77%]
tests/test_database.py::test_execute_write_fn_block_true PASSED [ 88%]
tests/test_database.py::test_execute_write_fn_exception PASSED [100%]
================================================================================= 9 passed in 1.90s ==================================================================================
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612082842,"Fix ""no such column: id"" output in tests",
https://github.com/simonw/datasette/pull/725#issuecomment-623623696,https://api.github.com/repos/simonw/datasette/issues/725,623623696,MDEyOklzc3VlQ29tbWVudDYyMzYyMzY5Ng==,4312421,stonebig,2020-05-04T18:16:54Z,2020-05-04T18:16:54Z,NONE,"thanks a lot, Simon ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",598891570,"Update aiofiles requirement from ~=0.4.0 to >=0.4,<0.6",
https://github.com/simonw/datasette/issues/752#issuecomment-623598224,https://api.github.com/repos/simonw/datasette/issues/752,623598224,MDEyOklzc3VlQ29tbWVudDYyMzU5ODIyNA==,9599,simonw,2020-05-04T17:28:11Z,2020-05-04T17:28:20Z,OWNER,"Good catch: it looks like we're not specifying the charset for those pages:
```
$ curl -i 'https://latest.datasette.io/404'
HTTP/1.1 404 Not Found
access-control-allow-origin: *
content-type: text/html
Date: Mon, 04 May 2020 16:23:25 GMT
Server: Google Frontend
Content-Length: 605
Error 404
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611835285,Non-utf8 encoding in exceptionhandlers and custom-pages,
https://github.com/simonw/datasette/pull/719#issuecomment-623591293,https://api.github.com/repos/simonw/datasette/issues/719,623591293,MDEyOklzc3VlQ29tbWVudDYyMzU5MTI5Mw==,9599,simonw,2020-05-04T17:14:23Z,2020-05-04T17:14:23Z,OWNER,"Thanks, this is a smart fix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594553553,asgi: check raw_path is not None,
https://github.com/simonw/datasette/issues/754#issuecomment-623561480,https://api.github.com/repos/simonw/datasette/issues/754,623561480,MDEyOklzc3VlQ29tbWVudDYyMzU2MTQ4MA==,9599,simonw,2020-05-04T16:17:42Z,2020-05-04T16:17:42Z,OWNER,Sure enough this run https://travis-ci.org/github/simonw/datasette/jobs/674415040 from that PR no longer shows the warnings.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611997130,Clean up aiofiles warnings on 3.8,
https://github.com/simonw/datasette/pull/725#issuecomment-623560732,https://api.github.com/repos/simonw/datasette/issues/725,623560732,MDEyOklzc3VlQ29tbWVudDYyMzU2MDczMg==,9599,simonw,2020-05-04T16:16:21Z,2020-05-04T16:16:21Z,OWNER,This may fix #754 warnings.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",598891570,"Update aiofiles requirement from ~=0.4.0 to >=0.4,<0.6",
https://github.com/simonw/datasette/issues/754#issuecomment-623560217,https://api.github.com/repos/simonw/datasette/issues/754,623560217,MDEyOklzc3VlQ29tbWVudDYyMzU2MDIxNw==,9599,simonw,2020-05-04T16:15:28Z,2020-05-04T16:15:28Z,OWNER,"I think those will be fixed by an upgrade to aiofiles 0.5.0:
https://github.com/Tinche/aiofiles/compare/v0.4.0...v0.5.0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611997130,Clean up aiofiles warnings on 3.8,
https://github.com/simonw/datasette/pull/730#issuecomment-623463200,https://api.github.com/repos/simonw/datasette/issues/730,623463200,MDEyOklzc3VlQ29tbWVudDYyMzQ2MzIwMA==,27856297,dependabot-preview[bot],2020-05-04T13:27:22Z,2020-05-04T13:27:22Z,CONTRIBUTOR,Superseded by #753.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604001627,"Update pytest-asyncio requirement from ~=0.10.0 to >=0.10,<0.12",
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623232984,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623232984,MDEyOklzc3VlQ29tbWVudDYyMzIzMjk4NA==,9599,simonw,2020-05-04T02:41:32Z,2020-05-04T02:41:32Z,MEMBER,Needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/simonw/datasette/issues/751#issuecomment-623207741,https://api.github.com/repos/simonw/datasette/issues/751,623207741,MDEyOklzc3VlQ29tbWVudDYyMzIwNzc0MQ==,9599,simonw,2020-05-04T00:14:35Z,2020-05-04T00:14:35Z,OWNER,"I'm calling it `""size""` for consistency with the querystring parameter `_size=`, even though `""page_size""` might be a more accurate name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611540797,Ability to set custom default _size on a per-table basis,
https://github.com/simonw/datasette/issues/751#issuecomment-623207636,https://api.github.com/repos/simonw/datasette/issues/751,623207636,MDEyOklzc3VlQ29tbWVudDYyMzIwNzYzNg==,9599,simonw,2020-05-04T00:13:46Z,2020-05-04T00:13:46Z,OWNER,"Syntax suggestion:
```json
{
""databases"": {
""mydatabase"": {
""tables"": {
""example_table"": {
""sort"": ""created"",
""size"": 10
}
}
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611540797,Ability to set custom default _size on a per-table basis,
https://github.com/simonw/datasette/issues/751#issuecomment-623207541,https://api.github.com/repos/simonw/datasette/issues/751,623207541,MDEyOklzc3VlQ29tbWVudDYyMzIwNzU0MQ==,9599,simonw,2020-05-04T00:13:09Z,2020-05-04T00:13:09Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/views/table.py#L519-L520,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611540797,Ability to set custom default _size on a per-table basis,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199750,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623199750,MDEyOklzc3VlQ29tbWVudDYyMzE5OTc1MA==,9599,simonw,2020-05-03T23:17:58Z,2020-05-03T23:17:58Z,MEMBER,Reading this source code is really useful for figuring out how to store a photo in a DB table: https://github.com/RhetTbull/osxphotos/blob/7444b6d173918a3ad2a07aefce5ecf054786c787/osxphotos/photoinfo.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199701,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623199701,MDEyOklzc3VlQ29tbWVudDYyMzE5OTcwMQ==,9599,simonw,2020-05-03T23:17:38Z,2020-05-03T23:17:38Z,MEMBER,"Record burst_uuid as a column:
```
(Pdb) with_bursts[0]._info[""burstUUID""]
'703FAA23-57BF-40B4-8A33-D9CEB143391B'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623199214,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623199214,MDEyOklzc3VlQ29tbWVudDYyMzE5OTIxNA==,9599,simonw,2020-05-03T23:14:08Z,2020-05-03T23:14:08Z,MEMBER,"Albums have UUIDs:
```
(Pdb) photo.album_info[0].__dict__
{'_uuid': '17816791-ABF3-447B-942C-9FA8065EEBBA', '_db': osxphotos.PhotosDB(dbfile='/Users/simon/Pictures/Photos Library.photoslibrary/database/photos.db'), '_title': 'Geotaggable Photos geotagged'}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198986,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623198986,MDEyOklzc3VlQ29tbWVudDYyMzE5ODk4Ng==,9599,simonw,2020-05-03T23:12:31Z,2020-05-03T23:12:46Z,MEMBER,"To get the taken date in UTC:
```
from datetime import timezone
(Pdb) photo.date.astimezone(timezone.utc).isoformat()
'2018-02-13T20:21:31.620000+00:00'
(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(""."")
['2018-02-13T20:21:31', '620000+00:00']
(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(""."")[0]
'2018-02-13T20:21:31'
(Pdb) photo.date.astimezone(timezone.utc).isoformat().split(""."")[0] + ""+00:00""
'2018-02-13T20:21:31+00:00'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623198653,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623198653,MDEyOklzc3VlQ29tbWVudDYyMzE5ODY1Mw==,9599,simonw,2020-05-03T23:09:57Z,2020-05-03T23:09:57Z,MEMBER,"For locations: I'll add `place_x` columns for all of these:
```
(Pdb) photo.place.address._asdict()
{'street': None, 'sub_locality': None, 'city': 'Loreto', 'sub_administrative_area': 'Loreto', 'state_province': 'BCS', 'postal_code': None, 'country': 'Mexico', 'iso_country_code': 'MX'}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623195197,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623195197,MDEyOklzc3VlQ29tbWVudDYyMzE5NTE5Nw==,9599,simonw,2020-05-03T22:44:33Z,2020-05-03T22:44:33Z,MEMBER,"Command will be this:
$ photos-to-sqlite apple-photos photos.db
This will populate a `apple_photos` table with the data imported by the `osxphotos` library, plus the calculated sha256.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/dogsheep/dogsheep-photos/issues/1#issuecomment-623193947,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1,623193947,MDEyOklzc3VlQ29tbWVudDYyMzE5Mzk0Nw==,9599,simonw,2020-05-03T22:36:17Z,2020-05-03T22:36:17Z,MEMBER,"I'm going to use [osxphotos](https://github.com/RhetTbull/osxphotos) for this.
Since I've already got code to upload photos and insert them into a table based on their `sha256` hash, my first go at this will be to import data using the tool and foreign-key it to the `sha256` hash in the existing table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533300,Import photo metadata from Apple Photos into SQLite,
https://github.com/simonw/sqlite-utils/issues/107#issuecomment-623128528,https://api.github.com/repos/simonw/sqlite-utils/issues/107,623128528,MDEyOklzc3VlQ29tbWVudDYyMzEyODUyOA==,9599,simonw,2020-05-03T15:36:58Z,2020-05-03T15:36:58Z,OWNER,Documentation: https://github.com/simonw/sqlite-utils/blob/d16097231c5e51ea857b58c700f97a17b68dc583/docs/cli.rst#creating-views,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611222968,sqlite-utils create-view CLI command,
https://github.com/simonw/sqlite-utils/issues/107#issuecomment-623127140,https://api.github.com/repos/simonw/sqlite-utils/issues/107,623127140,MDEyOklzc3VlQ29tbWVudDYyMzEyNzE0MA==,9599,simonw,2020-05-03T15:27:22Z,2020-05-03T15:27:22Z,OWNER,"Syntax:
$ sqlite-utils create-view my.db myview ""select sqlite_version()""
Also accepts `--replace` and `--ignore` options, same as `create-table`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611222968,sqlite-utils create-view CLI command,
https://github.com/simonw/sqlite-utils/issues/27#issuecomment-623126943,https://api.github.com/repos/simonw/sqlite-utils/issues/27,623126943,MDEyOklzc3VlQ29tbWVudDYyMzEyNjk0Mw==,9599,simonw,2020-05-03T15:25:55Z,2020-05-03T15:25:55Z,OWNER,Documentation: https://github.com/simonw/sqlite-utils/blob/78264b738cd72ffad6e5c32ede3f074f8aad0ca4/docs/cli.rst#creating-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455496504,sqlite-utils create-table command,
https://github.com/simonw/sqlite-utils/issues/27#issuecomment-623120166,https://api.github.com/repos/simonw/sqlite-utils/issues/27,623120166,MDEyOklzc3VlQ29tbWVudDYyMzEyMDE2Ng==,9599,simonw,2020-05-03T14:38:59Z,2020-05-03T14:38:59Z,OWNER,I'll stick with requiring all three fk arguments.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455496504,sqlite-utils create-table command,
https://github.com/simonw/sqlite-utils/issues/27#issuecomment-623051550,https://api.github.com/repos/simonw/sqlite-utils/issues/27,623051550,MDEyOklzc3VlQ29tbWVudDYyMzA1MTU1MA==,9599,simonw,2020-05-03T04:17:18Z,2020-05-03T04:17:18Z,OWNER,"Be nice if you could do `--fk colname` and have it guess the rest, but I'm not sure how to do that with a CLI option - they need a fixed number of arguments so that they don't consume the next batch of options.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455496504,sqlite-utils create-table command,
https://github.com/simonw/sqlite-utils/issues/27#issuecomment-623051447,https://api.github.com/repos/simonw/sqlite-utils/issues/27,623051447,MDEyOklzc3VlQ29tbWVudDYyMzA1MTQ0Nw==,9599,simonw,2020-05-03T04:15:52Z,2020-05-03T04:15:52Z,OWNER,Supporting `--fk colname othertable othercol` would be neat too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455496504,sqlite-utils create-table command,
https://github.com/simonw/sqlite-utils/issues/27#issuecomment-623051392,https://api.github.com/repos/simonw/sqlite-utils/issues/27,623051392,MDEyOklzc3VlQ29tbWVudDYyMzA1MTM5Mg==,9599,simonw,2020-05-03T04:15:05Z,2020-05-03T04:15:05Z,OWNER,"I need tests for what happens if table already exists, or if the user provides invalid column types.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455496504,sqlite-utils create-table command,
https://github.com/simonw/sqlite-utils/issues/108#issuecomment-623050458,https://api.github.com/repos/simonw/sqlite-utils/issues/108,623050458,MDEyOklzc3VlQ29tbWVudDYyMzA1MDQ1OA==,9599,simonw,2020-05-03T04:00:32Z,2020-05-03T04:00:32Z,OWNER,"```
from sqlite_utils import cli
cli.cli.commands.keys()
```
Outputs:
```
dict_keys(['tables', 'views', 'vacuum', 'optimize', 'add-column', 'add-foreign-key', 'index-foreign-keys', 'create-index', 'enable-fts', 'populate-fts', 'disable-fts', 'insert', 'upsert', 'create-table', 'query', 'rows'])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611326701,Documentation unit tests for CLI commands,
https://github.com/simonw/sqlite-utils/issues/108#issuecomment-623050375,https://api.github.com/repos/simonw/sqlite-utils/issues/108,623050375,MDEyOklzc3VlQ29tbWVudDYyMzA1MDM3NQ==,9599,simonw,2020-05-03T03:59:16Z,2020-05-03T03:59:27Z,OWNER,"```
import re
r = re.compile(r'\$ sqlite-utils (\S+) ')
rst = open(""docs/cli.rst"").read()
{command for command in r.findall(rst) if ""."" not in command and "":"" not in command}
```
Outputs:
```
{'add-column',
'add-foreign-key',
'create-index',
'disable-fts',
'enable-fts',
'index-foreign-keys',
'insert',
'optimize',
'populate-fts',
'query',
'rows',
'tables',
'vacuum',
'views'}
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611326701,Documentation unit tests for CLI commands,
https://github.com/simonw/sqlite-utils/issues/27#issuecomment-623049505,https://api.github.com/repos/simonw/sqlite-utils/issues/27,623049505,MDEyOklzc3VlQ29tbWVudDYyMzA0OTUwNQ==,9599,simonw,2020-05-03T03:45:32Z,2020-05-03T03:45:32Z,OWNER,"Could take `--ignore` to ignore if table already exists, and `--replace` to drop and replace it if it exists.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455496504,sqlite-utils create-table command,
https://github.com/simonw/sqlite-utils/issues/27#issuecomment-623048530,https://api.github.com/repos/simonw/sqlite-utils/issues/27,623048530,MDEyOklzc3VlQ29tbWVudDYyMzA0ODUzMA==,9599,simonw,2020-05-03T03:30:31Z,2020-05-03T03:30:31Z,OWNER,"Copy the design for `--not-null` and `--default` from the `insert` command:
```
$ sqlite-utils insert dogs.db dogs_with_scores dogs-with-scores.json \
--not-null=age \
--not-null=name \
--default age 2 \
--default score 5
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455496504,sqlite-utils create-table command,
https://github.com/simonw/datasette/issues/151#issuecomment-623047233,https://api.github.com/repos/simonw/datasette/issues/151,623047233,MDEyOklzc3VlQ29tbWVudDYyMzA0NzIzMw==,9599,simonw,2020-05-03T03:11:16Z,2020-05-03T03:11:16Z,OWNER,Now live at https://latest.datasette.io/-/patterns,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",276718605,Set up a pattern portfolio,
https://github.com/simonw/datasette/issues/151#issuecomment-623044858,https://api.github.com/repos/simonw/datasette/issues/151,623044858,MDEyOklzc3VlQ29tbWVudDYyMzA0NDg1OA==,9599,simonw,2020-05-03T02:37:03Z,2020-05-03T02:37:03Z,OWNER,"I'm going to put this at `/-/patterns`, which will render a template called `patterns.html`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",276718605,Set up a pattern portfolio,
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623044643,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623044643,MDEyOklzc3VlQ29tbWVudDYyMzA0NDY0Mw==,5779832,zzeleznick,2020-05-03T02:34:32Z,2020-05-03T02:34:32Z,NONE,"1. More than glad to share feedback from the sidelines as a [starrer](https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++starred_at%2C%0D%0A++starred_by%2C%0D%0A++full_name+as+repo_name%0D%0Afrom%0D%0A++repos_starred%0D%0Awhere%0D%0A++starred_by+%3D+%22zzeleznick%22%0D%0Aorder+by%0D%0A++starred_at+desc).
```
-- Motivation:
-- Datasette is a data hammer and I'm looking for nails
-- e.g. Find which repos a user has starred => trigger a TBD downstream action
select
starred_at,
starred_by,
full_name as repo_name
from
repos_starred
where
starred_by = ""zzeleznick""
order by
starred_at desc
```
| starred_at | starred_by | repo_name |
| --- | --- | --- |
| 2020-02-11T01:08:59Z | zzeleznick | dogsheep/twitter-to-sqlite |
| 2020-01-11T21:57:34Z | zzeleznick | simonw/datasette |
2. In my day job, I use [airflow](https://github.com/apache/airflow), and that's the mental model I'm bringing to [datasette](https://github.com/simonw/datasette).
3. I see your project like [twitter-to-sqlite](https://github.com/dogsheep/twitter-to-sqlite) akin to [Operators](https://airflow.apache.org/docs/stable/_api/index.html#pythonapi-operators) in Airflow world.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481,[Feature Request] Support Repo Name in Search 🥺,
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038378,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623038378,MDEyOklzc3VlQ29tbWVudDYyMzAzODM3OA==,9599,simonw,2020-05-03T01:21:13Z,2020-05-03T01:21:13Z,MEMBER,No this is really useful feedback! I'm so close to this project that I miss what's not obvious to people dropping in for the first time.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481,[Feature Request] Support Repo Name in Search 🥺,
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623038148,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623038148,MDEyOklzc3VlQ29tbWVudDYyMzAzODE0OA==,5779832,zzeleznick,2020-05-03T01:18:57Z,2020-05-03T01:18:57Z,NONE,"Thanks, @simonw!
I feel a little foolish in hindsight, but I'm on the same page now and am glad to have discovered first-hand a motivation for this `repos_starred` use case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481,[Feature Request] Support Repo Name in Search 🥺,
https://github.com/dogsheep/github-to-sqlite/issues/38#issuecomment-623027889,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38,623027889,MDEyOklzc3VlQ29tbWVudDYyMzAyNzg4OQ==,9599,simonw,2020-05-02T23:15:11Z,2020-05-02T23:15:11Z,MEMBER,"This is one of the use-cases for the `repos_starred` view: it allows you to easily run this kid of query without having to construct the SQL by hand. Here's a demo:
https://github-to-sqlite.dogsheep.net/github/repos_starred?name__contains=twitter
My philosophy here is to keep the raw tables (like `stars`) as normalized as possible, then use SQL views which expose the data in a form that's easier to query.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611284481,[Feature Request] Support Repo Name in Search 🥺,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623010272,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623010272,MDEyOklzc3VlQ29tbWVudDYyMzAxMDI3Mg==,9599,simonw,2020-05-02T20:39:14Z,2020-05-02T20:39:14Z,MEMBER,"Graph of cumulative stars for Datasette over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29#g.mark=line&g.x_column=yyyymmdd&g.x_type=temporal&g.y_column=cumulative_count&g.y_type=quantitative
![]()
Stars per day (as a label bar chart, so very wide):
https://github-to-sqlite.dogsheep.net/github?sql=%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++#g.mark=bar&g.x_column=yyyymmdd&g.x_type=ordinal&g.y_column=n&g.y_type=quantitative
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623007441,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623007441,MDEyOklzc3VlQ29tbWVudDYyMzAwNzQ0MQ==,9599,simonw,2020-05-02T20:13:37Z,2020-05-02T20:13:37Z,MEMBER,Datasette cumulative stars over time: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++yyyymmdd%2C%0D%0A++sum%28n%29+over+%28%0D%0A++++order+by%0D%0A++++++yyyymmdd+rows+unbounded+preceding%0D%0A++%29+as+cumulative_count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++substr%28starred_at%2C+0%2C+11%29+as+yyyymmdd%2C%0D%0A++++++count%28*%29+as+n%0D%0A++++from%0D%0A++++++stars%0D%0A++++where+repo+%3D+107914493%0D%0A++++group+by%0D%0A++++++yyyymmdd%0D%0A++%29,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623006154,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623006154,MDEyOklzc3VlQ29tbWVudDYyMzAwNjE1NA==,9599,simonw,2020-05-02T20:01:39Z,2020-05-02T20:01:54Z,MEMBER,Needs tests and documentation. I shipped it early to check that the live demo works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-623006004,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8,623006004,MDEyOklzc3VlQ29tbWVudDYyMzAwNjAwNA==,9599,simonw,2020-05-02T20:00:26Z,2020-05-02T20:00:26Z,MEMBER,I'm abandoning this in favour of a new implementation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516763727,"stargazers command, refs #4",
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623004836,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623004836,MDEyOklzc3VlQ29tbWVudDYyMzAwNDgzNg==,9599,simonw,2020-05-02T19:49:51Z,2020-05-02T19:49:51Z,MEMBER,"Alternative pattern:
```
sqlite-utils releases.db 'select full_name from repos' --csv --no-headers \
| tr -d '\r' \
| xargs github-to-sqlite stargazers stars.db
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-623000814,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,623000814,MDEyOklzc3VlQ29tbWVudDYyMzAwMDgxNA==,9599,simonw,2020-05-02T19:15:23Z,2020-05-02T19:15:23Z,MEMBER,"I'm not going to do the `--sql` bit just yet. I have patterns for working around this for other commands which are working fine:
https://github.com/dogsheep/github-to-sqlite/blob/d00a53061556dc403c166b443d141c4e1adbd64a/.github/workflows/deploy-demo.yml#L53-L70","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/simonw/datasette/issues/750#issuecomment-623000252,https://api.github.com/repos/simonw/datasette/issues/750,623000252,MDEyOklzc3VlQ29tbWVudDYyMzAwMDI1Mg==,9599,simonw,2020-05-02T19:10:44Z,2020-05-02T19:10:44Z,OWNER,"Demo: https://latest.datasette.io/fixtures/roadside_attractions?name__notlike=%25museum%25
Docs: https://datasette.readthedocs.io/en/latest/json_api.html#column-filter-arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611252244,Add notlike table filter,
https://github.com/simonw/datasette/issues/750#issuecomment-622999623,https://api.github.com/repos/simonw/datasette/issues/750,622999623,MDEyOklzc3VlQ29tbWVudDYyMjk5OTYyMw==,9599,simonw,2020-05-02T19:05:07Z,2020-05-02T19:05:07Z,OWNER,"![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611252244,Add notlike table filter,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622998813,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,622998813,MDEyOklzc3VlQ29tbWVudDYyMjk5ODgxMw==,9599,simonw,2020-05-02T18:58:17Z,2020-05-02T18:58:17Z,MEMBER,Faceting works now: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet_array=topics&topics__arraycontains=datasette-io&topics__arraycontains=sqlite&_facet=repo#facet-repo,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,Add this view for seeing new releases,
https://github.com/simonw/datasette/issues/750#issuecomment-622998529,https://api.github.com/repos/simonw/datasette/issues/750,622998529,MDEyOklzc3VlQ29tbWVudDYyMjk5ODUyOQ==,9599,simonw,2020-05-02T18:56:00Z,2020-05-02T18:56:00Z,OWNER,So that I can do this: https://github-to-sqlite.dogsheep.net/github/dependent_repos?_where=dependent%20not%20like%20%27%simonw/%%27,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611252244,Add notlike table filter,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622997410,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622997410,MDEyOklzc3VlQ29tbWVudDYyMjk5NzQxMA==,9599,simonw,2020-05-02T18:46:10Z,2020-05-02T18:46:10Z,MEMBER,Documented here: https://github.com/dogsheep/github-to-sqlite/blob/10fb34de41aaa35681f08b5991540d65bfcf2e2e/README.md#authentication,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,Fall back to authentication via ENV,
https://github.com/dogsheep/github-to-sqlite/issues/4#issuecomment-622990947,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4,622990947,MDEyOklzc3VlQ29tbWVudDYyMjk5MDk0Nw==,9599,simonw,2020-05-02T17:54:16Z,2020-05-02T17:54:16Z,MEMBER,"I could add that window function query as a view, but only if the detected version of SQLite supports window functions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",493670730,Command to fetch stargazers for one or more repos,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622989874,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,622989874,MDEyOklzc3VlQ29tbWVudDYyMjk4OTg3NA==,9599,simonw,2020-05-02T17:46:14Z,2020-05-02T17:46:14Z,MEMBER,Without the rowid column facet by topics breaks: https://github-to-sqlite.dogsheep.net/github/recent_releases?_facet=repo&_facet_array=topics&topics__arraycontains=datasette-io,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,Add this view for seeing new releases,
https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622982667,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622982667,MDEyOklzc3VlQ29tbWVudDYyMjk4MjY2Nw==,9599,simonw,2020-05-02T16:52:53Z,2020-05-02T16:52:53Z,MEMBER,Easiest option: use `db.index_foreign_keys()`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#adding-indexes-for-all-foreign-keys,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450,Create index on issue_comments(user) and other foreign keys,
https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622982346,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36,622982346,MDEyOklzc3VlQ29tbWVudDYyMjk4MjM0Ng==,9599,simonw,2020-05-02T16:50:31Z,2020-05-02T16:50:31Z,MEMBER,Demo: https://github-to-sqlite.dogsheep.net/github/dependent_repos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610842926,Add view for better display of dependent repos,
https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622980203,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10,622980203,MDEyOklzc3VlQ29tbWVudDYyMjk4MDIwMw==,9599,simonw,2020-05-02T16:34:29Z,2020-05-02T16:34:29Z,MEMBER,"Fixed definition:
```sql
select
stars.starred_at,
starring_user.login as starred_by,
repos.*
from
repos
join stars on repos.id = stars.repo
join users as starring_user on stars.user = starring_user.id
join users on repos.owner = users.id
order by
starred_at desc;
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516967682,Add this repos_starred view,
https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622978173,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622978173,MDEyOklzc3VlQ29tbWVudDYyMjk3ODE3Mw==,9599,simonw,2020-05-02T16:19:31Z,2020-05-02T16:19:47Z,MEMBER,"I can use the new `.create_view(..., replace=True)` parameter in `sqlite-utils` 2.7.2 for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136,Mechanism for creating views if they don't yet exist,
https://github.com/simonw/sqlite-utils/issues/106#issuecomment-622976185,https://api.github.com/repos/simonw/sqlite-utils/issues/106,622976185,MDEyOklzc3VlQ29tbWVudDYyMjk3NjE4NQ==,9599,simonw,2020-05-02T16:04:51Z,2020-05-02T16:04:51Z,OWNER,Docs: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#creating-views,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",611216862,"create_view(..., ignore=True, replace=True) parameters",
https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622599528,https://api.github.com/repos/simonw/sqlite-utils/issues/103,622599528,MDEyOklzc3VlQ29tbWVudDYyMjU5OTUyOA==,32605365,b0b5h4rp13,2020-05-01T22:49:12Z,2020-05-02T11:15:44Z,CONTRIBUTOR,"With SQLITE_MAX_VARS = 999, or even 899, This hits the problem with the batch rows causing a overflow (works fine if SQLITE_MAX_VARS = 799).
p.s. I have tried a few list of dicts to sqlite modules and this was the easiest to use/understand
------------- file begins ------------------
import sqlite_utils as su
data = [
{'tickerId': 913324382, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'CONSTELLATION B', 'symbol': 'STZ B', 'disSymbol': 'STZ-B', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '163.13', 'change': '6.46', 'changeRatio': '0.0412', 'marketValue': '31180699895.63', 'volume': '417', 'turnoverRate': '0.0000'},
{'tickerId': 913323791, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Molina Health', 'symbol': 'MOH', 'disSymbol': 'MOH', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '173.25', 'change': '9.28', 'changeRatio': '0.0566', 'pPrice': '173.25', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '10520341695.50', 'volume': '1281557', 'turnoverRate': '0.0202'},
{'tickerId': 913257501, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Seattle Genetics', 'symbol': 'SGEN', 'disSymbol': 'SGEN', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '145.64', 'change': '8.41', 'changeRatio': '0.0613', 'pPrice': '146.45', 'pChange': '0.8100', 'pChRatio': '0.0056', 'marketValue': '25117961347.60', 'volume': '2791411', 'turnoverRate': '0.0162'},
{'tickerId': 925381971, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Bandwidth', 'symbol': 'BAND', 'disSymbol': 'BAND', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '89.22', 'change': '7.66', 'changeRatio': '0.0939', 'pPrice': '89.00', 'pChange': '-0.2200', 'pChRatio': '-0.0025', 'marketValue': '2100025474.98', 'volume': '1508629', 'turnoverRate': '0.0641'},
{'tickerId': 913323935, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Magellan Health', 'symbol': 'MGLN', 'disSymbol': 'MGLN', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '68.00', 'change': '7.27', 'changeRatio': '0.1197', 'pPrice': '68.00', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '1697894040.00', 'volume': '448919', 'turnoverRate': '0.0180'},
{'tickerId': 913254854, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'On Assignment', 'symbol': 'ASGN', 'disSymbol': 'ASGN', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '53.04', 'change': '6.59', 'changeRatio': '0.1419', 'pPrice': '53.04', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '2811120000.00', 'volume': '1339771', 'turnoverRate': '0.0253'},
{'tickerId': 913255732, 'exchangeId': 95, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Arcturus', 'symbol': 'ARCT', 'disSymbol': 'ARCT', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NMS', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '40.86', 'change': '6.36', 'changeRatio': '0.1843', 'pPrice': '42.60', 'pChange': '1.740', 'pChRatio': '0.0426', 'marketValue': '812021444.46', 'volume': '1577508', 'turnoverRate': '0.0794'},
{'tickerId': 913256616, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'DexCom', 'symbol': 'DXCM', 'disSymbol': 'DXCM', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '341.52', 'change': '6.32', 'changeRatio': '0.0189', 'pPrice': '340.00', 'pChange': '-1.5200', 'pChRatio': '-0.0045', 'marketValue': '31522296000.00', 'volume': '1008849', 'turnoverRate': '0.0109'},
{'tickerId': 913255108, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Clorox', 'symbol': 'CLX', 'disSymbol': 'CLX', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '192.71', 'change': '6.27', 'changeRatio': '0.0336', 'pPrice': '192.95', 'pChange': '0.2400', 'pChRatio': '0.0012', 'marketValue': '24185773318.28', 'volume': '4996414', 'turnoverRate': '0.0398'},
{'tickerId': 925314627, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'FRANCO NEVADA', 'symbol': 'FNV', 'disSymbol': 'FNV', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '137.85', 'change': '5.64', 'changeRatio': '0.0427', 'pPrice': '138.50', 'pChange': '0.6500', 'pChRatio': '0.0047', 'marketValue': '26110405326.30', 'volume': '1047688', 'turnoverRate': '0.0055'},
{'tickerId': 913254955, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Aon Plc', 'symbol': 'AON', 'disSymbol': 'AON', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '178.21', 'change': '5.54', 'changeRatio': '0.0321', 'pPrice': '178.21', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '41181209117.22', 'volume': '2026234', 'turnoverRate': '0.0088'},
{'tickerId': 913324105, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Willis Towers', 'symbol': 'WLTW', 'disSymbol': 'WLTW', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '183.34', 'change': '5.05', 'changeRatio': '0.0283', 'pPrice': '183.34', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '23597461124.96', 'volume': '968943', 'turnoverRate': '0.0075'},
{'tickerId': 913254759, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'TELADOC HEALTH', 'symbol': 'TDOC', 'disSymbol': 'TDOC', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '169.43', 'change': '4.84', 'changeRatio': '0.0294', 'pPrice': '168.88', 'pChange': '-0.5500', 'pChRatio': '-0.0032', 'marketValue': '12614616858.38', 'volume': '2628946', 'turnoverRate': '0.0353'},
{'tickerId': 913255222, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Emergent Bio', 'symbol': 'EBS', 'disSymbol': 'EBS', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '78.70', 'change': '4.75', 'changeRatio': '0.0642', 'pPrice': '78.40', 'pChange': '-0.3000', 'pChRatio': '-0.0038', 'marketValue': '4113368277.10', 'volume': '783804', 'turnoverRate': '0.0150'},
{'tickerId': 913323443, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Pool', 'symbol': 'POOL', 'disSymbol': 'POOL', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '216.02', 'change': '4.36', 'changeRatio': '0.0206', 'pPrice': '216.02', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '8696077573.82', 'volume': '310837', 'turnoverRate': '0.0077'},
{'tickerId': 913257075, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Masimo', 'symbol': 'MASI', 'disSymbol': 'MASI', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '218.00', 'change': '4.09', 'changeRatio': '0.0191', 'pPrice': '217.00', 'pChange': '-1.0000', 'pChRatio': '-0.0046', 'marketValue': '11797070000.00', 'volume': '542131', 'turnoverRate': '0.0100'},
{'tickerId': 913253761, 'exchangeId': 10, 'type': 2, 'secType': [62], 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Pope Resources', 'symbol': 'POPE', 'disSymbol': 'POPE', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NAS', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '101.05', 'change': '3.95', 'changeRatio': '0.0407', 'pPrice': '99.90', 'pChange': '2.800', 'pChRatio': '0.0288', 'marketValue': '447370075.75', 'volume': '33138', 'turnoverRate': '0.0075'},
{'tickerId': 913323560, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Seneca Foods', 'symbol': 'SENEB', 'disSymbol': 'SENEB', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '40.04', 'change': '3.84', 'changeRatio': '0.1061', 'marketValue': '347950039.71', 'volume': '501'},
{'tickerId': 913324274, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Resmed', 'symbol': 'RMD', 'disSymbol': 'RMD', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '159.07', 'change': '3.75', 'changeRatio': '0.0241', 'pPrice': '159.07', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '23004217759.29', 'volume': '1267075', 'turnoverRate': '0.0088'},
{'tickerId': 913323736, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Vertex Pharms', 'symbol': 'VRTX', 'disSymbol': 'VRTX', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '254.90', 'change': '3.70', 'changeRatio': '0.0147', 'pPrice': '255.00', 'pChange': '0.1000', 'pChRatio': '0.0004', 'marketValue': '66062980780.10', 'volume': '1939843', 'turnoverRate': '0.0075'},
{'tickerId': 913323767, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'MCCORMICK VTG', 'symbol': 'MKC V', 'disSymbol': 'MKC-V', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '159.99', 'change': '3.42', 'changeRatio': '0.0218', 'marketValue': '21262671000.00', 'volume': '432', 'turnoverRate': '0.0000'},
{'tickerId': 950118595, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'ZOOM VIDEO', 'symbol': 'ZM', 'disSymbol': 'ZM', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '138.56', 'change': '3.39', 'changeRatio': '0.0251', 'pPrice': '138.99', 'pChange': '0.4300', 'pChRatio': '0.0031', 'marketValue': '38620532420.16', 'volume': '13786017', 'turnoverRate': '0.0495'},
{'tickerId': 916040738, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'WHEATON PRECIOUS', 'symbol': 'WPM', 'disSymbol': 'WPM', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '41.10', 'change': '3.34', 'changeRatio': '0.0885', 'pPrice': '41.09', 'pChange': '-0.0100', 'pChRatio': '-0.0002', 'marketValue': '18404536146.30', 'volume': '5019137', 'turnoverRate': '0.0112'},
{'tickerId': 913257174, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Royal Gold', 'symbol': 'RGLD', 'disSymbol': 'RGLD', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '125.86', 'change': '3.33', 'changeRatio': '0.0272', 'pPrice': '125.86', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '8253015011.08', 'volume': '853473', 'turnoverRate': '0.0130'},
{'tickerId': 913254394, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Fortune Brand', 'symbol': 'FBHS', 'disSymbol': 'FBHS', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '51.50', 'change': '3.30', 'changeRatio': '0.0685', 'pPrice': '51.50', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '7194870278.50', 'volume': '3004021', 'turnoverRate': '0.0214'},
{'tickerId': 913323312, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Liberty Global', 'symbol': 'LBTYK', 'disSymbol': 'LBTYK', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '21.49', 'change': '3.18', 'changeRatio': '0.1737', 'pPrice': '21.48', 'pChange': '-0.0100', 'pChRatio': '-0.0005', 'marketValue': '13594662302.41', 'volume': '19980228', 'turnoverRate': '0.0315'},
{'tickerId': 913323882, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Preformed Line', 'symbol': 'PLPC', 'disSymbol': 'PLPC', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '52.82', 'change': '3.14', 'changeRatio': '0.0632', 'pPrice': '52.10', 'pChange': '-0.7200', 'pChRatio': '-0.0136', 'marketValue': '264979981.20', 'volume': '9305', 'turnoverRate': '0.0018'},
{'tickerId': 913323248, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Discovery', 'symbol': 'DISCB', 'disSymbol': 'DISCB', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'status': 'A', 'close': '57.95', 'change': '23.63', 'changeRatio': '0.6884', 'pPrice': '54.26', 'pChange': '-3.6900', 'pChRatio': '-0.0637', 'marketValue': '29362894177.95', 'volume': '218305', 'turnoverRate': '0.0004'},
{'tickerId': 913323930, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'MercadoLibre', 'symbol': 'MELI', 'disSymbol': 'MELI', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '605.52', 'change': '22.01', 'changeRatio': '0.0377', 'pPrice': '603.69', 'pChange': '-1.8300', 'pChRatio': '-0.0030', 'marketValue': '30226598045.28', 'volume': '699008', 'turnoverRate': '0.0140'},
{'tickerId': 913257170, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Liberty Global', 'symbol': 'LBTYA', 'disSymbol': 'LBTYA', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '22.28', 'change': '2.86', 'changeRatio': '0.1473', 'pPrice': '22.29', 'pChange': '0.0100', 'pChRatio': '0.0004', 'marketValue': '14094419548.52', 'volume': '10534672', 'turnoverRate': '0.0167'},
{'tickerId': 913303991, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Liberty Brodband', 'symbol': 'LBRDK', 'disSymbol': 'LBRDK', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '125.44', 'change': '2.76', 'changeRatio': '0.0225', 'pPrice': '125.44', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '22817900904.96', 'volume': '926177', 'turnoverRate': '0.0042'},
{'tickerId': 913257082, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Helen of Troy', 'symbol': 'HELE', 'disSymbol': 'HELE', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '167.04', 'change': '2.76', 'changeRatio': '0.0168', 'pPrice': '167.04', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '4216707982.08', 'volume': '341465', 'turnoverRate': '0.0135'},
{'tickerId': 913256458, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Forrester', 'symbol': 'FORR', 'disSymbol': 'FORR', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '33.88', 'change': '2.58', 'changeRatio': '0.0824', 'marketValue': '635419400.00', 'volume': '85115', 'turnoverRate': '0.0045'},
{'tickerId': 950158952, 'exchangeId': 95, 'type': 2, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'LYRA THERAPEUTICS, INC.', 'symbol': 'LYRA', 'disSymbol': 'LYRA', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NMS', 'listStatus': 1, 'template': 'ipo', 'status': 'A', 'close': '18.56', 'change': '2.56', 'changeRatio': '0.1600', 'pPrice': '18.96', 'pChange': '0.4000', 'pChRatio': '0.0216', 'marketValue': '229705575.68', 'volume': '1738472', 'turnoverRate': '0.1405'},
{'tickerId': 913257570, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Bio-Techne', 'symbol': 'TECH', 'disSymbol': 'TECH', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '227.54', 'change': '2.54', 'changeRatio': '0.0113', 'pPrice': '227.54', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '8726538309.18', 'volume': '497006', 'turnoverRate': '0.0130'},
{'tickerId': 913323246, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Bel Fuse', 'symbol': 'BELFB', 'disSymbol': 'BELFB', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '9.99', 'change': '2.53', 'changeRatio': '0.3391', 'pPrice': '9.75', 'pChange': '-0.2400', 'pChRatio': '-0.0240', 'marketValue': '122562454.86', 'volume': '177634', 'turnoverRate': '0.0145'},
{'tickerId': 916040647, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Agnico Eagle', 'symbol': 'AEM', 'disSymbol': 'AEM', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '61.20', 'change': '2.52', 'changeRatio': '0.0429', 'pPrice': '61.10', 'pChange': '-0.1000', 'pChRatio': '-0.0016', 'marketValue': '14739911553.60', 'volume': '2820765', 'turnoverRate': '0.0117'},
{'tickerId': 913303768, 'exchangeId': 12, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'CHASE CORP', 'symbol': 'CCF', 'disSymbol': 'CCF', 'disExchangeCode': 'AMEX', 'exchangeCode': 'ASE', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '96.71', 'change': '2.45', 'changeRatio': '0.0260', 'marketValue': '916799598.60', 'volume': '29229', 'turnoverRate': '0.0031'},
{'tickerId': 913324557, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Allergan', 'symbol': 'AGN', 'disSymbol': 'AGN', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '189.74', 'change': '2.40', 'changeRatio': '0.0128', 'pPrice': '189.76', 'pChange': '0.0200', 'pChRatio': '0.0001', 'marketValue': '62424842326.10', 'volume': '5787032', 'turnoverRate': '0.0176'},
{'tickerId': 913324566, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'West Pharm Svc', 'symbol': 'WST', 'disSymbol': 'WST', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '191.64', 'change': '2.38', 'changeRatio': '0.0126', 'pPrice': '191.64', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '14078267117.08', 'volume': '352460', 'turnoverRate': '0.0042'}
]
db = su.Database(f""overnight hold.db"" )
db['active'].insert_all(data)
--------------- file ends ----------------------","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610517472,sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns,
https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622587177,https://api.github.com/repos/simonw/sqlite-utils/issues/103,622587177,MDEyOklzc3VlQ29tbWVudDYyMjU4NzE3Nw==,9599,simonw,2020-05-01T22:07:51Z,2020-05-01T22:07:51Z,OWNER,"This is my failed attempt to recreate the bug (plus some extra debugging output):
```diff
% git diff
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index dd49d5c..ea42aea 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -1013,7 +1013,11 @@ class Table(Queryable):
assert (
num_columns <= SQLITE_MAX_VARS
), ""Rows can have a maximum of {} columns"".format(SQLITE_MAX_VARS)
+ print(""default batch_size = "", batch_size)
batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))
+ print(""new batch_size = {},num_columns = {}, MAX_VARS // num_columns = {}"".format(
+ batch_size, num_columns, SQLITE_MAX_VARS // num_columns
+ ))
self.last_rowid = None
self.last_pk = None
for chunk in chunks(itertools.chain([first_record], records), batch_size):
@@ -1124,6 +1128,9 @@ class Table(Queryable):
)
flat_values = list(itertools.chain(*values))
queries_and_params = [(sql, flat_values)]
+ print(sql.count(""?""), len(flat_values))
+
+ # print(json.dumps(queries_and_params, indent=4))
with self.db.conn:
for query, params in queries_and_params:
diff --git a/tests/test_create.py b/tests/test_create.py
index 5290cd8..52940df 100644
--- a/tests/test_create.py
+++ b/tests/test_create.py
@@ -853,3 +853,33 @@ def test_create_with_nested_bytes(fresh_db):
record = {""id"": 1, ""data"": {""foo"": b""bytes""}}
fresh_db[""t""].insert(record)
assert [{""id"": 1, ""data"": '{""foo"": ""b\'bytes\'""}'}] == list(fresh_db[""t""].rows)
+
+
+def test_create_throws_useful_error_with_increasing_number_of_columns(fresh_db):
+ # https://github.com/simonw/sqlite-utils/issues/103
+ def rows():
+ yield {""name"": 0}
+ for i in range(1, 1001):
+ yield {
+ ""name"": i,
+ ""age"": i,
+ ""size"": i,
+ ""name2"": i,
+ ""age2"": i,
+ ""size2"": i,
+ ""name3"": i,
+ ""age3"": i,
+ ""size3"": i,
+ ""name4"": i,
+ ""age4"": i,
+ ""size4"": i,
+ ""name5"": i,
+ ""age5"": i,
+ ""size5"": i,
+ ""name6"": i,
+ ""age6"": i,
+ ""size6"": i,
+ }
+
+ fresh_db[""t""].insert_all(rows())
+ assert 1001 == fresh_db[""t""].count
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610517472,sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns,
https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622584433,https://api.github.com/repos/simonw/sqlite-utils/issues/103,622584433,MDEyOklzc3VlQ29tbWVudDYyMjU4NDQzMw==,9599,simonw,2020-05-01T21:57:52Z,2020-05-01T21:57:52Z,OWNER,@b0b5h4rp13 I'm having trouble creating a test that triggers this bug. Could you share a chunk of code that replicates what you're seeing here?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610517472,sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns,
https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622565276,https://api.github.com/repos/simonw/sqlite-utils/issues/103,622565276,MDEyOklzc3VlQ29tbWVudDYyMjU2NTI3Ng==,9599,simonw,2020-05-01T20:57:16Z,2020-05-01T20:57:16Z,OWNER,I'm reconsidering this: I think this is going to happen ANY time someone has at least one row that is wider than the first row. So at the very least I should show a more understandable error message.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610517472,sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns,
https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622563188,https://api.github.com/repos/simonw/sqlite-utils/issues/103,622563188,MDEyOklzc3VlQ29tbWVudDYyMjU2MzE4OA==,9599,simonw,2020-05-01T20:51:24Z,2020-05-01T20:51:29Z,OWNER,Hopefully anyone who runs into this problem in the future will search for and find this issue thread!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610517472,sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns,
https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622563059,https://api.github.com/repos/simonw/sqlite-utils/issues/103,622563059,MDEyOklzc3VlQ29tbWVudDYyMjU2MzA1OQ==,9599,simonw,2020-05-01T20:51:01Z,2020-05-01T20:51:01Z,OWNER,"I'm not sure what to do about this.
I was thinking the solution would be to look at ALL of the rows in a batch before deciding on the maximum number of columns, but that doesn't work because we calculate batch size based on the number of columns!
I think my recommendation here is to manually pass a `batch_size=` argument to `.insert_all()` if you run into this error.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610517472,sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns,
https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622561944,https://api.github.com/repos/simonw/sqlite-utils/issues/103,622561944,MDEyOklzc3VlQ29tbWVudDYyMjU2MTk0NA==,9599,simonw,2020-05-01T20:47:51Z,2020-05-01T20:47:51Z,OWNER,"Yup we only take the number of columns in the first record into account at the moment:
https://github.com/simonw/sqlite-utils/blob/d56029549acae0b0ea94c5a0f783e3b3895d9218/sqlite_utils/db.py#L1007-L1016","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610517472,sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns,
https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622561585,https://api.github.com/repos/simonw/sqlite-utils/issues/103,622561585,MDEyOklzc3VlQ29tbWVudDYyMjU2MTU4NQ==,9599,simonw,2020-05-01T20:46:50Z,2020-05-01T20:46:50Z,OWNER,The varying number of columns thing is interesting - I don't think the tests cover that case much if at all.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610517472,sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns,
https://github.com/simonw/sqlite-utils/issues/105#issuecomment-622558889,https://api.github.com/repos/simonw/sqlite-utils/issues/105,622558889,MDEyOklzc3VlQ29tbWVudDYyMjU1ODg4OQ==,9599,simonw,2020-05-01T20:40:06Z,2020-05-01T20:40:06Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#listing-views,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610853576,"""sqlite-utils views"" command",
https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461948,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622461948,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTk0OA==,9599,simonw,2020-05-01T16:36:42Z,2020-05-01T16:36:42Z,MEMBER,It should only create views if the underlying tables exist.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136,Mechanism for creating views if they don't yet exist,
https://github.com/dogsheep/github-to-sqlite/issues/37#issuecomment-622461537,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37,622461537,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTUzNw==,9599,simonw,2020-05-01T16:35:40Z,2020-05-01T16:35:40Z,MEMBER,"This will check if the view exists and has the exact same matching definition as the one we want. If it doesn't, we will drop it (if it exists) and recreate it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610843136,Mechanism for creating views if they don't yet exist,
https://github.com/dogsheep/github-to-sqlite/issues/12#issuecomment-622461223,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12,622461223,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTIyMw==,9599,simonw,2020-05-01T16:34:52Z,2020-05-01T16:34:52Z,MEMBER,Blocked on #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520756546,Add this view for seeing new releases,
https://github.com/dogsheep/github-to-sqlite/issues/10#issuecomment-622461122,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10,622461122,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTEyMg==,9599,simonw,2020-05-01T16:34:39Z,2020-05-01T16:34:39Z,MEMBER,Blocked on #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",516967682,Add this repos_starred view,
https://github.com/dogsheep/github-to-sqlite/issues/36#issuecomment-622461025,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36,622461025,MDEyOklzc3VlQ29tbWVudDYyMjQ2MTAyNQ==,9599,simonw,2020-05-01T16:34:24Z,2020-05-01T16:34:24Z,MEMBER,Blocked on #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610842926,Add view for better display of dependent repos,
https://github.com/simonw/datasette/issues/749#issuecomment-622450636,https://api.github.com/repos/simonw/datasette/issues/749,622450636,MDEyOklzc3VlQ29tbWVudDYyMjQ1MDYzNg==,9599,simonw,2020-05-01T16:08:46Z,2020-05-01T16:08:46Z,OWNER,"Proposed solution: on Cloud Run don't show the ""download database"" link if the database file is larger than 32MB.
I can do this with a new config setting, `max_db_mb`, which is automatically set by the `publish cloudrun` command.
This is consistent with the existing `max_csv_mb` setting: https://datasette.readthedocs.io/en/stable/config.html#max-csv-mb
I should set `max_csv_mb` to 32MB on Cloud Run deploys as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610829227,Cloud Run fails to serve database files larger than 32MB,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622279374,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622279374,MDEyOklzc3VlQ29tbWVudDYyMjI3OTM3NA==,2029,garethr,2020-05-01T07:12:47Z,2020-05-01T07:12:47Z,NONE,"I also go it working with:
```yaml
run: echo ${{ secrets.github_token }} | github-to-sqlite auth
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,Fall back to authentication via ENV,
https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622214262,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622214262,MDEyOklzc3VlQ29tbWVudDYyMjIxNDI2Mg==,9599,simonw,2020-05-01T02:10:32Z,2020-05-01T02:11:19Z,MEMBER,"This sped that query up even more - down to 4ms.
```sql
create index issue_comments_issue on issue_comments(issue);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450,Create index on issue_comments(user) and other foreign keys,
https://github.com/dogsheep/github-to-sqlite/issues/35#issuecomment-622213950,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35,622213950,MDEyOklzc3VlQ29tbWVudDYyMjIxMzk1MA==,9599,simonw,2020-05-01T02:09:04Z,2020-05-01T02:09:04Z,MEMBER,"It sped up this query a lot - 2.5s down to 300ms:
```sql
select
repos.full_name,
json_object(
'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number,
'label', '#' || issues.number
) as issue,
issues.title,
users.login,
users.id,
issues.state,
issues.locked,
issues.assignee,
issues.milestone,
issues.comments,
issues.created_at,
issues.updated_at,
issues.closed_at,
issues.author_association,
issues.pull_request,
issues.repo,
issues.type
from
issues
join repos on repos.id = issues.repo
join users on issues.user = users.id
where
issues.state = 'open'
and issues.user not in (9599, 27856297)
and not exists (
select
id
from
issue_comments
where
issue_comments.user = 9599
and issues.id = issue_comments.issue
)
order by
issues.updated_at desc;
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610511450,Create index on issue_comments(user) and other foreign keys,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622171097,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622171097,MDEyOklzc3VlQ29tbWVudDYyMjE3MTA5Nw==,9599,simonw,2020-04-30T23:22:45Z,2020-04-30T23:23:57Z,MEMBER,"The `auth.json` mechanism this uses is standard across all of the other Dogsheep tools - it's actually designed so you can have one `auth.json` with a bunch of different credentials for different tools:
```json
{
""goodreads_personal_token"": ""..."",
""goodreads_user_id"": ""..."",
""github_personal_token"": ""..."",
""pocket_consumer_key"": ""..."",
""pocket_username"": ""..."",
""pocket_access_token"": ""...""
}
```
But... `github-to-sqlite` does feel like it deserves a special case here, since it's such a good fit for running inside of GitHub Actions - which even provide a `GITHUB_TOKEN` for you to use!
So I don't think it will harm the family of tools too much if this has an environment variable alternative to the `-a` file.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,Fall back to authentication via ENV,
https://github.com/dogsheep/github-to-sqlite/issues/33#issuecomment-622169728,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33,622169728,MDEyOklzc3VlQ29tbWVudDYyMjE2OTcyOA==,9599,simonw,2020-04-30T23:18:51Z,2020-04-30T23:18:51Z,MEMBER,"Sure, that sounds fine to me.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",609950090,Fall back to authentication via ENV,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622162835,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622162835,MDEyOklzc3VlQ29tbWVudDYyMjE2MjgzNQ==,9599,simonw,2020-04-30T22:59:26Z,2020-04-30T22:59:26Z,MEMBER,Documentation: https://github.com/dogsheep/github-to-sqlite/blob/c9f48404481882e8b3af06f35e4801a80ac79ed6/README.md#scraping-dependents-for-a-repository,"{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 2, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,Command for retrieving dependents for a repo,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622135654,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622135654,MDEyOklzc3VlQ29tbWVudDYyMjEzNTY1NA==,9599,simonw,2020-04-30T21:53:44Z,2020-04-30T21:56:06Z,MEMBER,"I think this is the neatest scraping pattern:
```python
[a[""href""].lstrip(""/"") for a in soup.select(""a[data-hovercard-type=repository]"")]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,Command for retrieving dependents for a repo,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622136585,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622136585,MDEyOklzc3VlQ29tbWVudDYyMjEzNjU4NQ==,9599,simonw,2020-04-30T21:55:51Z,2020-04-30T21:55:51Z,MEMBER,"And to find the ""Next"" pagination link:
```python
soup.select("".paginate-container"")[0].find(""a"", text=""Next"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,Command for retrieving dependents for a repo,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133775,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622133775,MDEyOklzc3VlQ29tbWVudDYyMjEzMzc3NQ==,9599,simonw,2020-04-30T21:49:27Z,2020-04-30T21:49:27Z,MEMBER,"Proposed command:
github-to-sqlite scrape-dependents github.db simonw/datasette
I'll pull full details of the scraped repos from the regular API. I'll also record when they were ""first seen"" by the command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,Command for retrieving dependents for a repo,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133422,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622133422,MDEyOklzc3VlQ29tbWVudDYyMjEzMzQyMg==,9599,simonw,2020-04-30T21:48:39Z,2020-04-30T21:48:39Z,MEMBER,It looks like the only option is to scrape them. I'll do that and then replace with an API as soon as one becomes available.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,Command for retrieving dependents for a repo,
https://github.com/dogsheep/github-to-sqlite/issues/34#issuecomment-622133298,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34,622133298,MDEyOklzc3VlQ29tbWVudDYyMjEzMzI5OA==,9599,simonw,2020-04-30T21:48:24Z,2020-04-30T21:48:24Z,MEMBER,"Unfortunately it's not available through any GitHub API - I managed to figure out how to get dependencies, but I need dependents. https://github.com/simonw/til/blob/master/github/dependencies-graphql-api.md","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610408908,Command for retrieving dependents for a repo,
https://github.com/simonw/datasette/issues/747#issuecomment-622043887,https://api.github.com/repos/simonw/datasette/issues/747,622043887,MDEyOklzc3VlQ29tbWVudDYyMjA0Mzg4Nw==,9599,simonw,2020-04-30T19:04:19Z,2020-04-30T19:04:19Z,OWNER,https://datasette.readthedocs.io/en/latest/config.html#configuration-directory-mode,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610192152,Directory configuration mode should support metadata.yaml,
https://github.com/simonw/datasette/issues/747#issuecomment-622036934,https://api.github.com/repos/simonw/datasette/issues/747,622036934,MDEyOklzc3VlQ29tbWVudDYyMjAzNjkzNA==,9599,simonw,2020-04-30T18:51:18Z,2020-04-30T18:51:18Z,OWNER,Needs docs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610192152,Directory configuration mode should support metadata.yaml,
https://github.com/simonw/datasette/issues/747#issuecomment-622003376,https://api.github.com/repos/simonw/datasette/issues/747,622003376,MDEyOklzc3VlQ29tbWVudDYyMjAwMzM3Ng==,9599,simonw,2020-04-30T17:45:32Z,2020-04-30T17:45:32Z,OWNER,"I can use this function to load the JSON-or-YAML:
https://github.com/simonw/datasette/blob/d349d57cdf3d577afb62bdf784af342a4d5be660/datasette/utils/__init__.py#L798-L806","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610192152,Directory configuration mode should support metadata.yaml,
https://github.com/simonw/datasette/issues/747#issuecomment-621948898,https://api.github.com/repos/simonw/datasette/issues/747,621948898,MDEyOklzc3VlQ29tbWVudDYyMTk0ODg5OA==,9599,simonw,2020-04-30T16:05:50Z,2020-04-30T16:05:50Z,OWNER,"Relevant code:
https://github.com/simonw/datasette/blob/e37f4077c0f1cd09d4102213d4e2a512af471b8d/datasette/app.py#L206-L208","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610192152,Directory configuration mode should support metadata.yaml,
https://github.com/simonw/datasette/pull/703#issuecomment-621045107,https://api.github.com/repos/simonw/datasette/issues/703,621045107,MDEyOklzc3VlQ29tbWVudDYyMTA0NTEwNw==,9599,simonw,2020-04-29T07:49:35Z,2020-04-29T07:49:35Z,OWNER,Adding some still-to-do items to a checklist in the description.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585597133,WIP implementation of writable canned queries,
https://github.com/simonw/datasette/issues/698#issuecomment-621041812,https://api.github.com/repos/simonw/datasette/issues/698,621041812,MDEyOklzc3VlQ29tbWVudDYyMTA0MTgxMg==,9599,simonw,2020-04-29T07:42:48Z,2020-04-29T07:42:48Z,OWNER,Need to figure out what the `.json` mode for this looks like - and if there's a `.csv` mode (I think not).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-621037724,https://api.github.com/repos/simonw/datasette/issues/698,621037724,MDEyOklzc3VlQ29tbWVudDYyMTAzNzcyNA==,9599,simonw,2020-04-29T07:34:02Z,2020-04-29T07:34:02Z,OWNER,"Concept for displaying a success message:
![]()
CSS:
```css
.success {
padding: 1em;
border: 1px solid green;
background-color: #c7fbc7;
}","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/698#issuecomment-621036032,https://api.github.com/repos/simonw/datasette/issues/698,621036032,MDEyOklzc3VlQ29tbWVudDYyMTAzNjAzMg==,9599,simonw,2020-04-29T07:29:52Z,2020-04-29T07:29:52Z,OWNER,"What should happen when a query has been successfully executed?
That depends on the query. Some queries may wish to redirect to another page. Other queries might want to show a custom message.
There should at least be a default message saying the query has been executed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582517965,Ability for a canned query to write to the database,
https://github.com/simonw/datasette/issues/744#issuecomment-621030783,https://api.github.com/repos/simonw/datasette/issues/744,621030783,MDEyOklzc3VlQ29tbWVudDYyMTAzMDc4Mw==,30607,aborruso,2020-04-29T07:16:27Z,2020-04-29T07:16:27Z,NONE,"Hi @simonw it's debian as Windows Subsystem for Linux
```
PRETTY_NAME=""Pengwin""
NAME=""Pengwin""
VERSION_ID=""10""
VERSION=""10 (buster)""
ID=debian
ID_LIKE=debian
HOME_URL=""https://github.com/whitewaterfoundry/Pengwin""
SUPPORT_URL=""https://github.com/whitewaterfoundry/Pengwin""
BUG_REPORT_URL=""https://github.com/whitewaterfoundry/Pengwin""
VERSION_CODENAME=buster
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-621027697,https://api.github.com/repos/simonw/datasette/issues/744,621027697,MDEyOklzc3VlQ29tbWVudDYyMTAyNzY5Nw==,9599,simonw,2020-04-29T07:08:09Z,2020-04-29T07:08:09Z,OWNER,What operating system are you using? I need to figure out a way to replicate this bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/pull/746#issuecomment-621027374,https://api.github.com/repos/simonw/datasette/issues/746,621027374,MDEyOklzc3VlQ29tbWVudDYyMTAyNzM3NA==,9599,simonw,2020-04-29T07:07:23Z,2020-04-29T07:07:23Z,OWNER,It looks like this isn't the right fix: https://github.com/simonw/datasette/issues/744#issuecomment-621008152,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608752766,"shutil.Error, not OSError",
https://github.com/simonw/datasette/issues/744#issuecomment-621011554,https://api.github.com/repos/simonw/datasette/issues/744,621011554,MDEyOklzc3VlQ29tbWVudDYyMTAxMTU1NA==,30607,aborruso,2020-04-29T06:17:26Z,2020-04-29T06:17:26Z,NONE,"A stupid note: I have no `tmpcqv_1i5d` folder in in `/tmp`.
It seems to me that it does not create any `/tmp/tmpcqv_1i5d/templates` folder (or other name folder, inside /tmp)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-621008152,https://api.github.com/repos/simonw/datasette/issues/744,621008152,MDEyOklzc3VlQ29tbWVudDYyMTAwODE1Mg==,30607,aborruso,2020-04-29T06:05:02Z,2020-04-29T06:05:02Z,NONE,"Hi @simonw , I have installed it and I have the below errors.
> Is it possible that your /tmp directory is on a different volume from the template folder? That could cause a problem with the symlinks.
No, /tmp folder is in the same volume.
Thank you
```
Traceback (most recent call last):
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py"", line 607, in link_or_copy_directory
shutil.copytree(src, dst, copy_function=os.link)
File ""/usr/lib/python3.7/shutil.py"", line 365, in copytree
raise Error(errors)
shutil.Error: [('/var/youtubeComunePalermo/processing/./template/base.html', '/tmp/tmpcqv_1i5d/templates/base.html', ""[Errno 18] Invalid cross-device link: '/var/youtubeComunePalermo/processing/./template/base.html' -> '/tmp/tmpcqv_1i5d/templates/base.html'""), ('/var/youtubeComunePalermo/processing/./template/index.html', '/tmp/tmpcqv_1i5d/templates/index.html', ""[Errno 18] Invalid cross-device link: '/var/youtubeComunePalermo/processing/./template/index.html' -> '/tmp/tmpcqv_1i5d/templates/index.html'"")]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ""/home/aborruso/.local/bin/datasette"", line 8, in
sys.exit(cli())
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/home/aborruso/.local/lib/python3.7/site-packages/click/core.py"", line 610, in invoke return callback(*args, **kwargs)
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py"", line 103, in heroku
extra_metadata,
File ""/usr/lib/python3.7/contextlib.py"", line 112, in __enter__
return next(self.gen)
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py"", line 191, in temporary_heroku_directory
os.path.join(tmp.name, ""templates""),
File ""/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py"", line 609, in link_or_copy_directory
shutil.copytree(src, dst)
File ""/usr/lib/python3.7/shutil.py"", line 321, in copytree
os.makedirs(dst)
File ""/usr/lib/python3.7/os.py"", line 221, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/tmp/tmpcqv_1i5d/templates'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/745#issuecomment-620972158,https://api.github.com/repos/simonw/datasette/issues/745,620972158,MDEyOklzc3VlQ29tbWVudDYyMDk3MjE1OA==,9599,simonw,2020-04-29T03:35:25Z,2020-04-29T03:35:25Z,OWNER,Relevant documentation: https://datasette.readthedocs.io/en/stable/performance.html#hashed-url-mode,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608613033,Extract the hash-URL mechanism out into a plugin,
https://github.com/simonw/datasette/issues/744#issuecomment-620971526,https://api.github.com/repos/simonw/datasette/issues/744,620971526,MDEyOklzc3VlQ29tbWVudDYyMDk3MTUyNg==,9599,simonw,2020-04-29T03:32:14Z,2020-04-29T03:32:14Z,OWNER,"@aborruso I think I have a branch with a fix - could you try it out?
Install the new branch like this:
```
pip install https://github.com/simonw/datasette/archive/issue-744.zip
```
Then try running `datasette publish heroku` again.
If this fixes your bug I'll merge that pull request!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/141#issuecomment-620970547,https://api.github.com/repos/simonw/datasette/issues/141,620970547,MDEyOklzc3VlQ29tbWVudDYyMDk3MDU0Nw==,9599,simonw,2020-04-29T03:27:37Z,2020-04-29T03:27:54Z,OWNER,This bug raised its head again in #744,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275814941,datasette publish can fail if /tmp is on a different device,
https://github.com/simonw/datasette/issues/744#issuecomment-620970475,https://api.github.com/repos/simonw/datasette/issues/744,620970475,MDEyOklzc3VlQ29tbWVudDYyMDk3MDQ3NQ==,9599,simonw,2020-04-29T03:27:20Z,2020-04-29T03:27:20Z,OWNER,This exception handling was introduced in #141,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-620970159,https://api.github.com/repos/simonw/datasette/issues/744,620970159,MDEyOklzc3VlQ29tbWVudDYyMDk3MDE1OQ==,9599,simonw,2020-04-29T03:25:45Z,2020-04-29T03:25:45Z,OWNER,"https://docs.python.org/3/library/shutil.html#shutil.copytree says that `shutil.Error` should be raised here, so I think this is a bug in the `link_or_copy_directory` function.
I don't have an environment that can replicate this bug. @aborruso I'm going to ship a fix in a branch that you can test against.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/744#issuecomment-620969639,https://api.github.com/repos/simonw/datasette/issues/744,620969639,MDEyOklzc3VlQ29tbWVudDYyMDk2OTYzOQ==,9599,simonw,2020-04-29T03:23:31Z,2020-04-29T03:23:31Z,OWNER,"Is it possible that your `/tmp` directory is on a different volume from the `template` folder? That could cause a problem with the symlinks.
Here's the code in question:
https://github.com/simonw/datasette/blob/89c4ddd4828623888e91a1d2cb396cba12d4e7b4/datasette/utils/__init__.py#L595-L609
It looks to me like we were expecting an `OSError` but in your environment you got a `shutil.Error` instead. So a fix could be to capture that error as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608058890,link_or_copy_directory() error - Invalid cross-device link,
https://github.com/simonw/datasette/issues/633#issuecomment-620841496,https://api.github.com/repos/simonw/datasette/issues/633,620841496,MDEyOklzc3VlQ29tbWVudDYyMDg0MTQ5Ng==,46165,nryberg,2020-04-28T20:37:50Z,2020-04-28T20:37:50Z,NONE,"Using the Heroku web interface, you can set the WEB_CONCURRENCY = 1
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",522334771,"Publish to Heroku is broken: ""WARNING: You must pass the application as an import string to enable 'reload' or 'workers""",
https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620774507,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620774507,MDEyOklzc3VlQ29tbWVudDYyMDc3NDUwNw==,9599,simonw,2020-04-28T18:19:06Z,2020-04-28T18:19:06Z,MEMBER,"The default timeout is a bit aggressive and sometimes failed for me if my resizing proxy took too long to fetch and resize the image.
`client.annotate_image(..., timeout=3.0)` may be worth trying.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747,Annotate photos using the Google Cloud Vision API,
https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620771067,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620771067,MDEyOklzc3VlQ29tbWVudDYyMDc3MTA2Nw==,9599,simonw,2020-04-28T18:12:34Z,2020-04-28T18:15:38Z,MEMBER,"Python library docs: https://googleapis.dev/python/vision/latest/index.html
I'm creating a new project for this called simonwillison-photos: https://console.cloud.google.com/projectcreate
https://console.cloud.google.com/home/dashboard?project=simonwillison-photos
Then I enabled the Vision API. The direct link to https://console.cloud.google.com/flows/enableapi?apiid=vision-json.googleapis.com which they provided in the docs didn't work - it gave me a ""You don't have sufficient permissions to use the requested API"" error - but starting at the ""Enable APIs"" page and searching for it worked fine.
I created a new service account as an ""owner"" of that project: https://console.cloud.google.com/apis/credentials/serviceaccountkey (and complained about it on Twitter and through their feedback form)
`pip install google-cloud-vision`
```python
from google.cloud import vision
client = vision.ImageAnnotatorClient.from_service_account_file(""simonwillison-photos-18c570b301fe.json"")
# Photo of a lemur
response = client.annotate_image(
{
""image"": {
""source"": {
""image_uri"": ""https://photos.simonwillison.net/i/1b3414ee9ade67ce04ade9042e6d4b433d1e523c9a16af17f490e2c0a619755b.jpeg""
}
},
""features"": [
{""type"": vision.enums.Feature.Type.IMAGE_PROPERTIES},
{""type"": vision.enums.Feature.Type.OBJECT_LOCALIZATION},
{""type"": vision.enums.Feature.Type.LABEL_DETECTION},
],
}
)
response
```
Output is:
```
label_annotations {
mid: ""/m/09686""
description: ""Vertebrate""
score: 0.9851104021072388
topicality: 0.9851104021072388
}
label_annotations {
mid: ""/m/04rky""
description: ""Mammal""
score: 0.975814163684845
topicality: 0.975814163684845
}
label_annotations {
mid: ""/m/01280g""
description: ""Wildlife""
score: 0.8973650336265564
topicality: 0.8973650336265564
}
label_annotations {
mid: ""/m/02f9pk""
description: ""Lemur""
score: 0.8270352482795715
topicality: 0.8270352482795715
}
label_annotations {
mid: ""/m/0fbf1m""
description: ""Terrestrial animal""
score: 0.7443860769271851
topicality: 0.7443860769271851
}
label_annotations {
mid: ""/m/06z_nw""
description: ""Tail""
score: 0.6934166550636292
topicality: 0.6934166550636292
}
label_annotations {
mid: ""/m/0b5gs""
description: ""Branch""
score: 0.6203985214233398
topicality: 0.6203985214233398
}
label_annotations {
mid: ""/m/05s2s""
description: ""Plant""
score: 0.585474967956543
topicality: 0.585474967956543
}
label_annotations {
mid: ""/m/089v3""
description: ""Zoo""
score: 0.5488107800483704
topicality: 0.5488107800483704
}
label_annotations {
mid: ""/m/02tcwp""
description: ""Trunk""
score: 0.5200017690658569
topicality: 0.5200017690658569
}
image_properties_annotation {
dominant_colors {
colors {
color {
red: 172.0
green: 146.0
blue: 116.0
}
score: 0.24523821473121643
pixel_fraction: 0.027533333748579025
}
colors {
color {
red: 54.0
green: 50.0
blue: 42.0
}
score: 0.10449723154306412
pixel_fraction: 0.12893334031105042
}
colors {
color {
red: 141.0
green: 121.0
blue: 97.0
}
score: 0.1391485631465912
pixel_fraction: 0.039133332669734955
}
colors {
color {
red: 28.0
green: 25.0
blue: 20.0
}
score: 0.08589499443769455
pixel_fraction: 0.11506666988134384
}
colors {
color {
red: 87.0
green: 82.0
blue: 74.0
}
score: 0.0845794677734375
pixel_fraction: 0.16113333404064178
}
colors {
color {
red: 121.0
green: 117.0
blue: 108.0
}
score: 0.05901569500565529
pixel_fraction: 0.13379999995231628
}
colors {
color {
red: 94.0
green: 83.0
blue: 66.0
}
score: 0.049011144787073135
pixel_fraction: 0.03946666792035103
}
colors {
color {
red: 155.0
green: 117.0
blue: 90.0
}
score: 0.04164913296699524
pixel_fraction: 0.0023333332501351833
}
colors {
color {
red: 178.0
green: 143.0
blue: 102.0
}
score: 0.02993861958384514
pixel_fraction: 0.0012666666880249977
}
colors {
color {
red: 61.0
green: 51.0
blue: 35.0
}
score: 0.027391711249947548
pixel_fraction: 0.01953333243727684
}
}
}
crop_hints_annotation {
crop_hints {
bounding_poly {
vertices {
x: 2073
}
vertices {
x: 4008
}
vertices {
x: 4008
y: 3455
}
vertices {
x: 2073
y: 3455
}
}
confidence: 0.65625
importance_fraction: 0.746666669845581
}
}
localized_object_annotations {
mid: ""/m/0jbk""
name: ""Animal""
score: 0.7008256912231445
bounding_poly {
normalized_vertices {
x: 0.0390297956764698
y: 0.26235100626945496
}
normalized_vertices {
x: 0.8466796875
y: 0.26235100626945496
}
normalized_vertices {
x: 0.8466796875
y: 0.9386426210403442
}
normalized_vertices {
x: 0.0390297956764698
y: 0.9386426210403442
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747,Annotate photos using the Google Cloud Vision API,
https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620772190,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620772190,MDEyOklzc3VlQ29tbWVudDYyMDc3MjE5MA==,9599,simonw,2020-04-28T18:14:43Z,2020-04-28T18:14:43Z,MEMBER,"Database schema for this will require some thought. Just dumping the output into a JSON column isn't going to be flexible enough - I want to be able to FTS against labels and OCR text, and potentially query against other characteristics too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747,Annotate photos using the Google Cloud Vision API,
https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620771698,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620771698,MDEyOklzc3VlQ29tbWVudDYyMDc3MTY5OA==,9599,simonw,2020-04-28T18:13:48Z,2020-04-28T18:13:48Z,MEMBER,"For face detection:
```
{""type"": vision.enums.Feature.Type.Type.FACE_DETECTION}
```
For OCR:
```
{""type"": vision.enums.Feature.Type.DOCUMENT_TEXT_DETECTION}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747,Annotate photos using the Google Cloud Vision API,
https://github.com/dogsheep/dogsheep-photos/issues/14#issuecomment-620769348,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14,620769348,MDEyOklzc3VlQ29tbWVudDYyMDc2OTM0OA==,9599,simonw,2020-04-28T18:09:21Z,2020-04-28T18:09:21Z,MEMBER,"Pricing is pretty good: free for first 1,000 calls per month, then $1.50 per thousand after that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608512747,Annotate photos using the Google Cloud Vision API,
https://github.com/simonw/datasette/issues/735#issuecomment-620401443,https://api.github.com/repos/simonw/datasette/issues/735,620401443,MDEyOklzc3VlQ29tbWVudDYyMDQwMTQ0Mw==,30607,aborruso,2020-04-28T06:10:20Z,2020-04-28T06:10:20Z,NONE,"It works in heroku, than might be a bug with datasette-publish-now.
Thank you","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605806386,"Error when I click on ""View and edit SQL""",
https://github.com/simonw/datasette/issues/736#issuecomment-620401172,https://api.github.com/repos/simonw/datasette/issues/736,620401172,MDEyOklzc3VlQ29tbWVudDYyMDQwMTE3Mg==,30607,aborruso,2020-04-28T06:09:28Z,2020-04-28T06:09:28Z,NONE,"> Would you mind trying publishing your database using one of the other options - Heroku, Cloud Run or https://fly.io/ - and see if you have the same bug there?
It works in heroku, than might be a bug with datasette-publish-now.
Thank you","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",606720674,strange behavior using accented characters,
https://github.com/dogsheep/dogsheep-photos/issues/13#issuecomment-620309185,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13,620309185,MDEyOklzc3VlQ29tbWVudDYyMDMwOTE4NQ==,9599,simonw,2020-04-28T00:39:45Z,2020-04-28T00:39:45Z,MEMBER,I'm going to leave this until I have the mechanism for associating a live photo video with the photo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607888367,Also upload movie files,
https://github.com/dogsheep/dogsheep-photos/issues/13#issuecomment-620273692,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13,620273692,MDEyOklzc3VlQ29tbWVudDYyMDI3MzY5Mg==,9599,simonw,2020-04-27T22:42:50Z,2020-04-27T22:42:50Z,MEMBER,"```
>>> def ext_counts(directory):
... counts = {}
... for path in pathlib.Path(directory).glob(""**/*""):
... ext = path.suffix
... counts[ext] = counts.get(ext, 0) + 1
... return counts
...
>>>
>>> ext_counts(""/Users/simon/Pictures/Photos Library.photoslibrary/originals"")
{'': 16, '.heic': 15478, '.jpeg': 21691, '.mov': 946, '.png': 2262, '.gif': 38, '.mp4': 116, '.aae': 2}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607888367,Also upload movie files,
https://github.com/simonw/datasette/issues/726#issuecomment-620260658,https://api.github.com/repos/simonw/datasette/issues/726,620260658,MDEyOklzc3VlQ29tbWVudDYyMDI2MDY1OA==,9599,simonw,2020-04-27T22:05:46Z,2020-04-27T22:05:46Z,OWNER,"Aah - yes I've seen this a few times before in my own projects. The problem is that the column types don't match up - your `oeuvre_id` column here is an integer, but if you look at the schema for `prelib_oeuvre` (at the bottom of this page: http://crbc-dataset.huma-num.fr/prelib/prelib_oeuvre) you'll see that it's defined as text:
```sql
CREATE TABLE ""prelib_oeuvre"" (
""id"" TEXT,
""titre"" TEXT,
""descriptif"" TEXT,
""wikidata"" TEXT,
""ark_bnf"" TEXT,
""notes"" TEXT
,PRIMARY KEY ([id])
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",600120439,Foreign key : case of a link to the associated row not displayed,
https://github.com/simonw/datasette/issues/743#issuecomment-620177365,https://api.github.com/repos/simonw/datasette/issues/743,620177365,MDEyOklzc3VlQ29tbWVudDYyMDE3NzM2NQ==,9599,simonw,2020-04-27T19:11:01Z,2020-04-27T19:11:30Z,OWNER,"Huh... turns out the documentation already claims that wildcards work! Closing this as wontfix:
https://datasette.readthedocs.io/en/stable/full_text_search.html#the-table-view-api
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607770595,escape_fts() does not correctly escape * wildcards,
https://github.com/simonw/datasette/issues/743#issuecomment-620174977,https://api.github.com/repos/simonw/datasette/issues/743,620174977,MDEyOklzc3VlQ29tbWVudDYyMDE3NDk3Nw==,9599,simonw,2020-04-27T19:05:56Z,2020-04-27T19:05:56Z,OWNER,"The other option would be to leave this as-is, and let people wildcard search all they want. I'm leaning in that direction.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607770595,escape_fts() does not correctly escape * wildcards,
https://github.com/simonw/datasette/issues/735#issuecomment-620172085,https://api.github.com/repos/simonw/datasette/issues/735,620172085,MDEyOklzc3VlQ29tbWVudDYyMDE3MjA4NQ==,9599,simonw,2020-04-27T19:00:30Z,2020-04-27T19:00:30Z,OWNER,"I think this may be a `datasette-publish-now` bug, as in https://github.com/simonw/datasette-publish-now/issues/20 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605806386,"Error when I click on ""View and edit SQL""",
https://github.com/simonw/datasette/issues/736#issuecomment-620171785,https://api.github.com/repos/simonw/datasette/issues/736,620171785,MDEyOklzc3VlQ29tbWVudDYyMDE3MTc4NQ==,9599,simonw,2020-04-27T18:59:54Z,2020-04-27T18:59:54Z,OWNER,"Would you mind trying publishing your database using one of the other options - Heroku, Cloud Run or https://fly.io/ - and see if you have the same bug there?
If you DO get the same bug with another option, please re-open this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",606720674,strange behavior using accented characters,
https://github.com/simonw/datasette/issues/736#issuecomment-620171434,https://api.github.com/repos/simonw/datasette/issues/736,620171434,MDEyOklzc3VlQ29tbWVudDYyMDE3MTQzNA==,9599,simonw,2020-04-27T18:59:16Z,2020-04-27T18:59:16Z,OWNER,I'm suspicious that this might be a bug with `datasette-publish-now` not with core Datasette. See https://github.com/simonw/datasette-publish-now/issues/20 - I think that might be the same issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",606720674,strange behavior using accented characters,
https://github.com/simonw/datasette/issues/743#issuecomment-620170826,https://api.github.com/repos/simonw/datasette/issues/743,620170826,MDEyOklzc3VlQ29tbWVudDYyMDE3MDgyNg==,9599,simonw,2020-04-27T18:58:04Z,2020-04-27T18:58:04Z,OWNER,"Maybe this is moot because you can't store a `*` character in a FTS table anyway, so it would never make sense to search for one? In which case maybe `escape_fts()` should just strip out `*` entirely?
Best source of information I could find was this tiny thread from 2014 about FTS4:
http://sqlite.1065341.n5.nabble.com/Escaping-conventions-for-FTS4-virtual-table-queries-td74589.html
> Dave Baggett wrote:
> > What if I want docids of documents containing the exact literal token ""any*""?
>
> You would have to use one of the Unicode tokenizers, and configure it to
> interpret * as a token character.
>
> > how do I escape the asterisk so that it's not interpreted as a wildcard?
>
> There are no escapes. When * is a token character, you lose the ability
> to do prefix searches.
I could investigate further by learning to use the fts5vocab virtual table debugging tool to see what's actually stored in those FTS5 indexes and check if `*` is indeed stripped by them.
https://www.sqlite.org/fts5.html#the_fts5vocab_virtual_table_module","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607770595,escape_fts() does not correctly escape * wildcards,
https://github.com/simonw/datasette/issues/743#issuecomment-620166959,https://api.github.com/repos/simonw/datasette/issues/743,620166959,MDEyOklzc3VlQ29tbWVudDYyMDE2Njk1OQ==,9599,simonw,2020-04-27T18:50:30Z,2020-04-27T18:50:30Z,OWNER,"Here's the `escape_fts()` function:
https://github.com/simonw/datasette/blob/89c4ddd4828623888e91a1d2cb396cba12d4e7b4/datasette/utils/__init__.py#L742-L753
https://latest.datasette.io/fixtures?sql=select+escape_fts%28%27bar%2A%27%29
![]()
So apparently wrapping a SQLite FTS word like `""bar*""` doesn't prevent SQLite from expanding the wildcard.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607770595,escape_fts() does not correctly escape * wildcards,
https://github.com/simonw/datasette/issues/742#issuecomment-620153909,https://api.github.com/repos/simonw/datasette/issues/742,620153909,MDEyOklzc3VlQ29tbWVudDYyMDE1MzkwOQ==,9599,simonw,2020-04-27T18:24:53Z,2020-04-27T18:24:53Z,OWNER,"I'm already using it extensively:
https://github.com/simonw/datasette/blob/25014ca25eb70b4c1217558ebd14db2845973bfb/tests/fixtures.py#L165-L189","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607243940,"Speed up tests with scope=""session""?",
https://github.com/simonw/datasette/issues/731#issuecomment-620095649,https://api.github.com/repos/simonw/datasette/issues/731,620095649,MDEyOklzc3VlQ29tbWVudDYyMDA5NTY0OQ==,9599,simonw,2020-04-27T16:32:44Z,2020-04-27T16:32:44Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/config.html#configuration-directory-mode,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605110015,Option to automatically configure based on directory layout,
https://github.com/simonw/datasette/issues/741#issuecomment-619708914,https://api.github.com/repos/simonw/datasette/issues/741,619708914,MDEyOklzc3VlQ29tbWVudDYxOTcwODkxNA==,9599,simonw,2020-04-27T04:30:02Z,2020-04-27T04:30:24Z,OWNER,"This can generate a `config.json` file that's included the published deployment, thanks to the work in #731 (pull request #739).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/741#issuecomment-619708745,https://api.github.com/repos/simonw/datasette/issues/741,619708745,MDEyOklzc3VlQ29tbWVudDYxOTcwODc0NQ==,9599,simonw,2020-04-27T04:29:26Z,2020-04-27T04:29:26Z,OWNER,I can show deprecation warnings for `--extra-options` and remove the option entirely in Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/pull/739#issuecomment-619700113,https://api.github.com/repos/simonw/datasette/issues/739,619700113,MDEyOklzc3VlQ29tbWVudDYxOTcwMDExMw==,9599,simonw,2020-04-27T03:58:09Z,2020-04-27T03:58:09Z,OWNER,"I think documentation goes on https://datasette.readthedocs.io/en/stable/config.html - but I'm going to need to re-arrange that page to fit in a section about configuration directories.
I'll move the existing list of settings into a ""configration options"" section.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607107849,Configuration directory mode,
https://github.com/simonw/datasette/issues/740#issuecomment-619698467,https://api.github.com/repos/simonw/datasette/issues/740,619698467,MDEyOklzc3VlQ29tbWVudDYxOTY5ODQ2Nw==,9599,simonw,2020-04-27T03:51:27Z,2020-04-27T03:51:27Z,OWNER,Spotted this while working on #731,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607211058,Don't throw 500 error on attempted directory browse,
https://github.com/simonw/datasette/pull/739#issuecomment-619678614,https://api.github.com/repos/simonw/datasette/issues/739,619678614,MDEyOklzc3VlQ29tbWVudDYxOTY3ODYxNA==,9599,simonw,2020-04-27T02:44:21Z,2020-04-27T02:44:21Z,OWNER,"I'm going to add a `Datasette(config_dir=""."")` parameter which initializes from the given directory, allowing other parameters to over-ride it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607107849,Configuration directory mode,
https://github.com/simonw/datasette/pull/739#issuecomment-619622311,https://api.github.com/repos/simonw/datasette/issues/739,619622311,MDEyOklzc3VlQ29tbWVudDYxOTYyMjMxMQ==,9599,simonw,2020-04-26T20:45:07Z,2020-04-26T20:45:07Z,OWNER,I think ALL explicit options should over-ride options that were detected in the directory structure. This means my current implementation isn't quite right - using a class method makes it hard to merge the details with the explicit options passed to the CLI function.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607107849,Configuration directory mode,
https://github.com/simonw/datasette/issues/731#issuecomment-619621388,https://api.github.com/repos/simonw/datasette/issues/731,619621388,MDEyOklzc3VlQ29tbWVudDYxOTYyMTM4OA==,9599,simonw,2020-04-26T20:38:09Z,2020-04-26T20:38:30Z,OWNER,Work now happening in the pull request: https://github.com/simonw/datasette/issues/739,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605110015,Option to automatically configure based on directory layout,
https://github.com/simonw/datasette/issues/648#issuecomment-619612675,https://api.github.com/repos/simonw/datasette/issues/648,619612675,MDEyOklzc3VlQ29tbWVudDYxOTYxMjY3NQ==,9599,simonw,2020-04-26T19:35:09Z,2020-04-26T19:35:09Z,OWNER,https://www.niche-museums.com/about now uses this mechanism. It used to be an empty `about.db` database but you can see from https://www.niche-museums.com/-/databases that it's now using the new mechanism.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534492501,Mechanism for adding arbitrary pages like /about,
https://github.com/simonw/datasette/issues/731#issuecomment-619611417,https://api.github.com/repos/simonw/datasette/issues/731,619611417,MDEyOklzc3VlQ29tbWVudDYxOTYxMTQxNw==,9599,simonw,2020-04-26T19:28:12Z,2020-04-26T19:28:12Z,OWNER,"I've shipped #648 now, so no reason not to go with the `static/` folder as the thing that gets detected and mounted as `/static`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605110015,Option to automatically configure based on directory layout,
https://github.com/simonw/datasette/issues/738#issuecomment-619605811,https://api.github.com/repos/simonw/datasette/issues/738,619605811,MDEyOklzc3VlQ29tbWVudDYxOTYwNTgxMQ==,9599,simonw,2020-04-26T18:58:56Z,2020-04-26T18:58:56Z,OWNER,"https://github.com/simonw/datasette/blob/304e7b1d9fd904ae1c35600bc03662eb90eeeae0/datasette/app.py#L781-L789
`request` is an optional argument here:
https://github.com/simonw/datasette/blob/304e7b1d9fd904ae1c35600bc03662eb90eeeae0/datasette/app.py#L556-L558","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607086780,Pass a request object to custom page templates,
https://github.com/simonw/datasette/issues/648#issuecomment-619604720,https://api.github.com/repos/simonw/datasette/issues/648,619604720,MDEyOklzc3VlQ29tbWVudDYxOTYwNDcyMA==,9599,simonw,2020-04-26T18:50:30Z,2020-04-26T18:50:30Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/custom_templates.html#custom-pages,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534492501,Mechanism for adding arbitrary pages like /about,
https://github.com/simonw/datasette/pull/737#issuecomment-619599904,https://api.github.com/repos/simonw/datasette/issues/737,619599904,MDEyOklzc3VlQ29tbWVudDYxOTU5OTkwNA==,9599,simonw,2020-04-26T18:21:06Z,2020-04-26T18:21:06Z,OWNER,Just needs documentation now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607067303,"Custom pages mechanism, refs #648",
https://github.com/simonw/datasette/pull/737#issuecomment-619593899,https://api.github.com/repos/simonw/datasette/issues/737,619593899,MDEyOklzc3VlQ29tbWVudDYxOTU5Mzg5OQ==,9599,simonw,2020-04-26T17:49:06Z,2020-04-26T17:49:06Z,OWNER,"I'm going with `custom_header(name, value)`, `custom_status(numeric_code)`, `custom_redirect(""URL"")`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607067303,"Custom pages mechanism, refs #648",
https://github.com/simonw/datasette/pull/737#issuecomment-619593811,https://api.github.com/repos/simonw/datasette/issues/737,619593811,MDEyOklzc3VlQ29tbWVudDYxOTU5MzgxMQ==,9599,simonw,2020-04-26T17:48:31Z,2020-04-26T17:48:31Z,OWNER,"Design challenge: the function that can be called in the custom template to return a custom status code and/or headers.
Proposed design:
```
{{ custom_headers({""x-foo"": ""bar""}) }}
{{ custom_status_code(404) }}
```
Or should I combine them into one function?
```
{{ custom_headers({""x-foo"": ""bar""}, status=404) }}
```
A common use-case here would be a redirect. How should that look?
```
{{ custom_redirect(""/foo"") }} - defaults to 301
{{ custom_redirect(""/foo"", status=302) }}
```
Maybe avoid dictionary syntax entirely:
```
{{ custom_header(""x-foo"", ""bar"") }}
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607067303,"Custom pages mechanism, refs #648",
https://github.com/simonw/datasette/pull/737#issuecomment-619591533,https://api.github.com/repos/simonw/datasette/issues/737,619591533,MDEyOklzc3VlQ29tbWVudDYxOTU5MTUzMw==,9599,simonw,2020-04-26T17:33:48Z,2020-04-26T17:33:48Z,OWNER,"> > Stretch goal: it would be neat if these pages could return custom HTTP headers (eg content-type) and maybe even status codes (eg for redirects) somehow.
>
> I think I could do that with a custom template function - if that function is called during the render then we follow those instructions instead of returning the rendered HTML.
https://github.com/simonw/datasette/issues/648#issuecomment-619591380","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607067303,"Custom pages mechanism, refs #648",
https://github.com/simonw/datasette/issues/648#issuecomment-619591380,https://api.github.com/repos/simonw/datasette/issues/648,619591380,MDEyOklzc3VlQ29tbWVudDYxOTU5MTM4MA==,9599,simonw,2020-04-26T17:33:04Z,2020-04-26T17:33:04Z,OWNER,"> > Stretch goal: it would be neat if these pages could return custom HTTP headers (eg content-type) and maybe even status codes (eg for redirects) somehow.
>
> I think I could do that with a custom template function - if that function is called during the render then we follow those instructions instead of returning the rendered HTML.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534492501,Mechanism for adding arbitrary pages like /about,
https://github.com/simonw/datasette/pull/725#issuecomment-619489720,https://api.github.com/repos/simonw/datasette/issues/725,619489720,MDEyOklzc3VlQ29tbWVudDYxOTQ4OTcyMA==,4312421,stonebig,2020-04-26T06:09:59Z,2020-04-26T06:10:13Z,NONE,"as a complementary remark: the versioning of datasette dependancies will become a problem when the new pip ""dependancy resolver"" will be activated. for now, it's just warnings via pip checks, later it will be a ""no"":
````
datasette 0.40 has requirement aiofiles~=0.4.0, but you have aiofiles 0.5.0.
datasette 0.40 has requirement Jinja2~=2.10.3, but you have jinja2 2.11.2.
````","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",598891570,"Update aiofiles requirement from ~=0.4.0 to >=0.4,<0.6",
https://github.com/simonw/datasette/issues/648#issuecomment-619263600,https://api.github.com/repos/simonw/datasette/issues/648,619263600,MDEyOklzc3VlQ29tbWVudDYxOTI2MzYwMA==,9599,simonw,2020-04-24T22:23:13Z,2020-04-24T22:23:13Z,OWNER,"Idea: `handle_404` can hand certain not found errors off to `handle_500` so it can render them correctly;
https://github.com/simonw/datasette/blob/227bb3e91fe34811a9374150798643a5af98ed79/datasette/app.py#L748-L765","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534492501,Mechanism for adding arbitrary pages like /about,
https://github.com/simonw/datasette/issues/648#issuecomment-619259859,https://api.github.com/repos/simonw/datasette/issues/648,619259859,MDEyOklzc3VlQ29tbWVudDYxOTI1OTg1OQ==,9599,simonw,2020-04-24T22:10:34Z,2020-04-24T22:10:34Z,OWNER,"The trickiest part here is the 404 logic. It's spread out through a couple of places right now. It's in `datasette.utils.asgi.AsgiRouter`:
https://github.com/simonw/datasette/blob/227bb3e91fe34811a9374150798643a5af98ed79/datasette/utils/asgi.py#L98-L134
Then also in the `DatasetteRouter subclass of that:
https://github.com/simonw/datasette/blob/227bb3e91fe34811a9374150798643a5af98ed79/datasette/app.py#L736-L757","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534492501,Mechanism for adding arbitrary pages like /about,
https://github.com/dogsheep/dogsheep-photos/issues/12#issuecomment-618796564,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/12,618796564,MDEyOklzc3VlQ29tbWVudDYxODc5NjU2NA==,9599,simonw,2020-04-24T04:35:25Z,2020-04-24T04:35:25Z,MEMBER,Code: https://github.com/dogsheep/photos-to-sqlite/blob/a388cf1f1b6b67752d669466cda8b171b6582171/photos_to_sqlite/cli.py#L109-L114,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",606033104,"If less than 500MB, show size in MB not GB",
https://github.com/simonw/datasette/issues/648#issuecomment-618775631,https://api.github.com/repos/simonw/datasette/issues/648,618775631,MDEyOklzc3VlQ29tbWVudDYxODc3NTYzMQ==,9599,simonw,2020-04-24T03:03:35Z,2020-04-24T03:03:35Z,OWNER,"> Stretch goal: it would be neat if these pages could return custom HTTP headers (eg content-type) and maybe even status codes (eg for redirects) somehow.
I think I could do that with a custom template function - if that function is called during the render then we follow those instructions instead of returning the rendered HTML.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534492501,Mechanism for adding arbitrary pages like /about,
https://github.com/simonw/datasette/issues/648#issuecomment-618775028,https://api.github.com/repos/simonw/datasette/issues/648,618775028,MDEyOklzc3VlQ29tbWVudDYxODc3NTAyOA==,9599,simonw,2020-04-24T03:01:32Z,2020-04-24T03:01:32Z,OWNER,Now that I've closed #577 this should be pretty easy to implement.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534492501,Mechanism for adding arbitrary pages like /about,
https://github.com/simonw/datasette/issues/731#issuecomment-618772317,https://api.github.com/repos/simonw/datasette/issues/731,618772317,MDEyOklzc3VlQ29tbWVudDYxODc3MjMxNw==,9599,simonw,2020-04-24T02:49:46Z,2020-04-24T02:49:46Z,OWNER,"Yeah I'll do that. My original thinking on allowing the `--static` option to mount other things to the root is that it would make it easy for people to add HTML ""pages"" to their site - like https://www.niche-museums.com/map for example.
But I've proposed a better solution for that here: #648","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605110015,Option to automatically configure based on directory layout,
https://github.com/simonw/datasette/issues/731#issuecomment-618758326,https://api.github.com/repos/simonw/datasette/issues/731,618758326,MDEyOklzc3VlQ29tbWVudDYxODc1ODMyNg==,25778,eyeseast,2020-04-24T01:55:00Z,2020-04-24T01:55:00Z,CONTRIBUTOR,Mounting `./static` at `/static` seems the simplest way. Saves you the trouble of deciding what else (`img` for example) gets special treatment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605110015,Option to automatically configure based on directory layout,
https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618725155,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9,618725155,MDEyOklzc3VlQ29tbWVudDYxODcyNTE1NQ==,9599,simonw,2020-04-23T23:39:14Z,2020-04-23T23:39:14Z,MEMBER,"A few minutes later...
```
Fetching existing keys from S3...
Got 22,446 existing keys
Calculating hashes [####################################] 100%
22,441 hashed files, 610 are not yet in S3
Uploading 0.99 GB
Uploading 610 photos [------------------------------------] 1/610 03:10:35
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605938063,"upload command should be resumable, should only upload photos not already uploaded",
https://github.com/dogsheep/dogsheep-photos/issues/9#issuecomment-618724149,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9,618724149,MDEyOklzc3VlQ29tbWVudDYxODcyNDE0OQ==,9599,simonw,2020-04-23T23:35:29Z,2020-04-23T23:35:29Z,MEMBER,"```
% photos-to-sqlite upload photos.db ~/Pictures/Photos\ Library.photoslibrary/originals
Fetching existing keys from S3...
Got 22,446 existing keys
Calculating hashes [####--------------------------------] 13% 00:04:14
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605938063,"upload command should be resumable, should only upload photos not already uploaded",
https://github.com/simonw/datasette/issues/731#issuecomment-618155472,https://api.github.com/repos/simonw/datasette/issues/731,618155472,MDEyOklzc3VlQ29tbWVudDYxODE1NTQ3Mg==,9599,simonw,2020-04-23T03:28:42Z,2020-04-23T03:28:56Z,OWNER,"As an alternative to `--static` this could work by letting you create the following:
- `static/css/`
- `static/js/`
Which would be automatically mounted at `/js/...` and `/css/...`
Or maybe just mount `static/` at `/static/` instead? ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605110015,Option to automatically configure based on directory layout,
https://github.com/simonw/datasette/issues/731#issuecomment-618126449,https://api.github.com/repos/simonw/datasette/issues/731,618126449,MDEyOklzc3VlQ29tbWVudDYxODEyNjQ0OQ==,25778,eyeseast,2020-04-23T01:38:55Z,2020-04-23T01:38:55Z,CONTRIBUTOR,"I've almost suggested this same thing a couple times. I tend to have Makefile (because I'm doing other `make` stuff anyway to get data prepped), and I end up putting all those CLI options in something like `make run`. But it would be way easier to just have all those typical options -- plugins, templates, metadata -- be defaults.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605110015,Option to automatically configure based on directory layout,
https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100658,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8,618100658,MDEyOklzc3VlQ29tbWVudDYxODEwMDY1OA==,9599,simonw,2020-04-23T00:03:35Z,2020-04-23T00:03:35Z,MEMBER,"Also MD5 isn't guaranteed for the ETag:
> If an object is created by either the Multipart Upload or Part Copy operation, the ETag is not an MD5 digest, regardless of the method of encryption.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605147638,Should I have used MD5 instead of SHA256?,
https://github.com/dogsheep/dogsheep-photos/issues/8#issuecomment-618100434,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8,618100434,MDEyOklzc3VlQ29tbWVudDYxODEwMDQzNA==,9599,simonw,2020-04-23T00:02:53Z,2020-04-23T00:02:53Z,MEMBER,"I don't think it matters one way or the other - I'm storing the sha256 in the filename, so the fact that I could read the MD5 back from the list bucket operation doesn't give me any benefits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605147638,Should I have used MD5 instead of SHA256?,
https://github.com/simonw/datasette/issues/731#issuecomment-618070791,https://api.github.com/repos/simonw/datasette/issues/731,618070791,MDEyOklzc3VlQ29tbWVudDYxODA3MDc5MQ==,9599,simonw,2020-04-22T22:21:27Z,2020-04-22T22:21:27Z,OWNER,I linked to this from https://github.com/zeit/now/discussions/4055,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605110015,Option to automatically configure based on directory layout,
https://github.com/simonw/datasette/issues/731#issuecomment-618070541,https://api.github.com/repos/simonw/datasette/issues/731,618070541,MDEyOklzc3VlQ29tbWVudDYxODA3MDU0MQ==,9599,simonw,2020-04-22T22:20:38Z,2020-04-22T22:20:38Z,OWNER,"I started thinking about this while building https://github.com/simonw/datasette-publish-now
The Vercel hosting platform is built on ASGI - so I need a way to provide an `app` ASGI object for it to use rather than Datasette's normal pattern of starting up the server using the `datasette serve` command.
The plugin currently does that by generating an `index.py` file that looks like this:
```python
from datasette.app import Datasette
import json
metadata = dict()
try:
metadata = json.load(open(""metadata.json""))
except Exception:
pass
app = Datasette([], [""fixtures.db""], metadata=metadata).app()
```
But that means I have to replicate the different CLI options - `--plugins-dir` etc - and figure out how to pass them to the `Datasette()` constructor.
This would be a whole lot easier if I could use `app = Datasette(configure_directory=""."")` and provide a magically configured instance based on the presence of files in the specified directory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",605110015,Option to automatically configure based on directory layout,
https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617491607,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,617491607,MDEyOklzc3VlQ29tbWVudDYxNzQ5MTYwNw==,9599,simonw,2020-04-22T01:20:19Z,2020-04-22T01:20:19Z,MEMBER,"https://github-to-sqlite.dogsheep.net/github/milestones now link to repo:
![]()
And so do issues: https://github-to-sqlite.dogsheep.net/github/issues
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862,Issue and milestone should have foreign key to repo,
https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617490914,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617490914,MDEyOklzc3VlQ29tbWVudDYxNzQ5MDkxNA==,9599,simonw,2020-04-22T01:17:44Z,2020-04-22T01:17:44Z,MEMBER,"https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101 now shows issues.
And https://github-to-sqlite.dogsheep.net/github/issue_comments links to them:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295,Issue comments don't appear to populate issues foreign key,
https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617369247,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617369247,MDEyOklzc3VlQ29tbWVudDYxNzM2OTI0Nw==,9599,simonw,2020-04-21T19:33:03Z,2020-04-21T19:33:03Z,MEMBER,Caused by #31.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295,Issue comments don't appear to populate issues foreign key,
https://github.com/dogsheep/github-to-sqlite/issues/32#issuecomment-617364956,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32,617364956,MDEyOklzc3VlQ29tbWVudDYxNzM2NDk1Ng==,9599,simonw,2020-04-21T19:24:45Z,2020-04-21T19:24:45Z,MEMBER,"That's because I just broke this code:
https://github.com/dogsheep/github-to-sqlite/blob/2cf75a0a036719eb7e57fdc7c5c2ea0f4c26978a/github_to_sqlite/utils.py#L131-L139
It expects the `repo` column to be `simonw/datasette` but it's now an ID instead.
I should add a test for this as part of the fix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",604222295,Issue comments don't appear to populate issues foreign key,
https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-617348174,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,617348174,MDEyOklzc3VlQ29tbWVudDYxNzM0ODE3NA==,9599,simonw,2020-04-21T18:50:29Z,2020-04-21T18:50:29Z,MEMBER,"Since this represents a breaking schema change for anyone running SQL queries against these tables, I'm going to do a major version bump to 2.0 when I release this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862,Issue and milestone should have foreign key to repo,
https://github.com/simonw/datasette/issues/176#issuecomment-617208503,https://api.github.com/repos/simonw/datasette/issues/176,617208503,MDEyOklzc3VlQ29tbWVudDYxNzIwODUwMw==,12976,nkirsch,2020-04-21T14:16:24Z,2020-04-21T14:16:24Z,NONE,"@eads I'm interested in helping, if there's still a need...","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",285168503,Add GraphQL endpoint,
https://github.com/dogsheep/github-to-sqlite/issues/31#issuecomment-616884647,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31,616884647,MDEyOklzc3VlQ29tbWVudDYxNjg4NDY0Nw==,9599,simonw,2020-04-21T00:49:16Z,2020-04-21T00:50:20Z,MEMBER,"The API just gives us the `repository_url`: https://api.github.com/repos/simonw/datasette/issues
We currently turn that into a `simonw/datasette` string here:
https://github.com/dogsheep/github-to-sqlite/blob/e0e8d8caa9657b04bfb8a2cf16c9b580f38b1805/github_to_sqlite/utils.py#L43-L46","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603624862,Issue and milestone should have foreign key to repo,
https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616883726,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30,616883726,MDEyOklzc3VlQ29tbWVudDYxNjg4MzcyNg==,9599,simonw,2020-04-21T00:45:23Z,2020-04-21T00:45:23Z,MEMBER,"Demo of fix: https://github-to-sqlite.dogsheep.net/github/issues?assignee__notblank=1&milestone__notblank=1
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603618244,Issues milestone column is the wrong type,
https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29,616883275,MDEyOklzc3VlQ29tbWVudDYxNjg4MzI3NQ==,9599,simonw,2020-04-21T00:43:28Z,2020-04-21T00:43:28Z,MEMBER,"I'm copying repo from issue, which surprisingly is a string, not an integer ID.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603617013,Milestones should have foreign key to creator and repo,
https://github.com/dogsheep/github-to-sqlite/issues/30#issuecomment-616879753,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30,616879753,MDEyOklzc3VlQ29tbWVudDYxNjg3OTc1Mw==,9599,simonw,2020-04-21T00:29:29Z,2020-04-21T00:29:29Z,MEMBER,`assignee` looks like it's the incorrect type too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",603618244,Issues milestone column is the wrong type,
https://github.com/dogsheep/twitter-to-sqlite/issues/45#issuecomment-616029262,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45,616029262,MDEyOklzc3VlQ29tbWVudDYxNjAyOTI2Mg==,9599,simonw,2020-04-19T04:39:21Z,2020-04-19T04:39:21Z,MEMBER,"
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602619330,Use raise_for_status() everywhere,
https://github.com/dogsheep/dogsheep-photos/issues/7#issuecomment-615993178,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7,615993178,MDEyOklzc3VlQ29tbWVudDYxNTk5MzE3OA==,9599,simonw,2020-04-19T00:37:08Z,2020-04-19T00:37:08Z,MEMBER,https://pypi.org/project/ImageHash/ Is one option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602585497,Integrate image content hashing,
https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615983393,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6,615983393,MDEyOklzc3VlQ29tbWVudDYxNTk4MzM5Mw==,9599,simonw,2020-04-18T23:53:10Z,2020-04-18T23:53:10Z,MEMBER,"```
$ photos-to-sqlite upload photos3.db ~/Pictures/Photos\ Library.photoslibrary/Masters/2020
Uploading 2.09 GB [##----------------------------------] 6% 00:36:37
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602575575,Add progress bar to upload command,
https://github.com/dogsheep/dogsheep-photos/issues/6#issuecomment-615979923,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6,615979923,MDEyOklzc3VlQ29tbWVudDYxNTk3OTkyMw==,9599,simonw,2020-04-18T23:36:02Z,2020-04-18T23:36:02Z,MEMBER,"I'll use a Click progress bar. To do this I need to first calculate the sum number of bytes in the photos that are going to be uploaded, then run the upload.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602575575,Add progress bar to upload command,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615957385,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615957385,MDEyOklzc3VlQ29tbWVudDYxNTk1NzM4NQ==,9599,simonw,2020-04-18T21:56:16Z,2020-04-18T21:58:11Z,MEMBER,Got this working! I'll do EXIF in a separate ticket #3.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/5#issuecomment-615949574,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/5,615949574,MDEyOklzc3VlQ29tbWVudDYxNTk0OTU3NA==,9599,simonw,2020-04-18T21:06:07Z,2020-04-18T21:06:07Z,MEMBER,"```
$ photos-to-sqlite s3-auth
Create S3 credentials and paste them here:
Access key ID: xxx
Secret access key: yyy
$ cat auth.json
{
""access_key_id"": ""xxx"",
""secret_access_key"": ""yyy""
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602551638,photos-to-sqlite s3-auth command,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615948102,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615948102,MDEyOklzc3VlQ29tbWVudDYxNTk0ODEwMg==,9599,simonw,2020-04-18T20:56:59Z,2020-04-18T20:56:59Z,MEMBER,"I'm going to start with this:
`photos-to-sqlite upload photos.db ~/path/to/directory`
This will scan the provided directory (and all sub-directories) for image files. It will then:
* Calculate a sha256 of the contents of that file
* Upload the file to a key that's `sha256.jpg` or `.heic`
* Upload a `sha256.json` file with the original path to the image
* Add that image to a `uploads` table in `photos.db`
Stretch goal: grab the EXIF data and include that in the `.json` upload AND the `uploads` database table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947370,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615947370,MDEyOklzc3VlQ29tbWVudDYxNTk0NzM3MA==,9599,simonw,2020-04-18T20:52:13Z,2020-04-18T20:52:13Z,MEMBER,"This is great! I now have a key that can upload photos, and a separate key that can download photos OR generate signed URLs to access those photos.
Next step: a script that starts uploading my photos.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615947229,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615947229,MDEyOklzc3VlQ29tbWVudDYxNTk0NzIyOQ==,9599,simonw,2020-04-18T20:51:26Z,2020-04-18T20:51:26Z,MEMBER,"Running the upload again like this resulted in the correct content-type:
```python
client.upload_file(
""/Users/simonw/Desktop/this_is_fine.jpg"",
""dogsheep-photos-simon"",
""this_is_fine.jpg"",
ExtraArgs={
""ContentType"": ""image/jpeg""
}
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533539,Upload all my photos to a secure S3 bucket,
https://github.com/dogsheep/dogsheep-photos/issues/4#issuecomment-615946537,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4,615946537,MDEyOklzc3VlQ29tbWVudDYxNTk0NjUzNw==,9599,simonw,2020-04-18T20:48:13Z,2020-04-18T20:48:13Z,MEMBER,"How about generating a signed URL?
```python
read_client.generate_presigned_url(
""get_object"",
Params={
""Bucket"": ""dogsheep-photos-simon"",
""Key"": ""this_is_fine.jpg"",
},
ExpiresIn=600
)
```
Gave me https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398
Which does this:
```
~ $ curl -i 'https://dogsheep-photos-simon.s3.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398'
HTTP/1.1 307 Temporary Redirect
x-amz-bucket-region: us-west-1
x-amz-request-id: E78CD859AEE21D33
x-amz-id-2: 648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=
Location: https://dogsheep-photos-simon.s3-us-west-1.amazonaws.com/this_is_fine.jpg?AWSAccessKeyId=AKIAWXFXAIOZNZ3JFO7I&Signature=x1zrS4w4OTGAACd7yHp9mYqXvN8%3D&Expires=1587243398
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Sat, 18 Apr 2020 20:47:21 GMT
Server: AmazonS3
Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests. dogsheep-photos-simon.s3-us-west-1.amazonaws.com dogsheep-photos-simon E78CD859AEE21D33 648mx+1+YSGga7NDOU7Q6isfsKnEPWOLC+DI4+x2o9FCc6pSCdIaoHJUbFMI8Vsuh1ADtx46ymU=
In [4]: c.execute('create table foo (bar integer);')
Out[4]:
In [5]: c.execute('insert into foo (bar) values (1)')
Out[5]:
In [6]: c.execute('select * from foo').fetchall()
Out[6]: [(1,)]
In [7]: c.execute('insert into foo (bar) values (1)')
Out[7]:
In [8]: c.execute('select * from foo').fetchall()
Out[8]: [(1,), (1,)]
In [9]: c.execute('insert into foo (bar) values (1)').lastrowid
Out[9]: 3
In [10]: c.execute('select * from foo').fetchall()
Out[10]: [(1,), (1,), (1,)]
In [11]: c.execute('select rowid, bar from foo').fetchall()
Out[11]: [(1, 1), (2, 1), (3, 1)]
In [12]: c.execute('insert into foo (bar) values (1)').lastrowid
Out[12]: 4
In [13]: c.execute('select rowid, bar from foo').fetchall()
Out[13]: [(1, 1), (2, 1), (3, 1), (4, 1)]
In [14]: r = c.execute('update foo set bar =2 where rowid = 1')
In [15]: r.lastrowid
Out[15]: 4
In [16]: c.execute('select rowid, bar from foo').fetchall()
Out[16]: [(1, 2), (2, 1), (3, 1), (4, 1)]
In [17]: r = c.execute('select rowid, bar from foo')
In [18]: r.fetchall()
Out[18]: [(1, 2), (2, 1), (3, 1), (4, 1)]
In [19]: r.lastrowid
Out[19]: 4
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records",
https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707828,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612707828,MDEyOklzc3VlQ29tbWVudDYxMjcwNzgyOA==,9599,simonw,2020-04-13T01:24:05Z,2020-04-13T01:24:16Z,OWNER,"Why do I even care about `lastrowid` here?
I'm trying to ensure that after you insert or upsert a row you can use `table.last_pk` to start doing things like building additional foreign key relationships.
So maybe it doesn't make sense to make `.last_pk` available _at all_ for cases where you called `.upsert_all()` or `.insert_all()` - it should just be populated for `.upsert()` and `.insert()`.
The documentation doesn't say it should work for `.upsert_all()` - it's only documented for the single actions.
https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1113-L1124","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records",
https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707293,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612707293,MDEyOklzc3VlQ29tbWVudDYxMjcwNzI5Mw==,9599,simonw,2020-04-13T01:21:22Z,2020-04-13T01:21:22Z,OWNER,"I have a hunch that the root of the problem here is that accessing `result.lastrowid` during my version of an `.upsert()` doesn't actually make sense:
https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1102-L1113
In the bug I'm seeing (which I still haven't reduced to a reproducible test) the debugger shows me this at that point:
```
(Pdb) query
'UPDATE [files] SET [createdAt] = ?, [ext] = ?, [updatedAt] = ?, [uri] = ?, [uriType] = ? WHERE [project] = ? AND [name] = ?'
(Pdb) params
['2020-03-04T04:04:40.152000+00:00', 'csv', '2020-03-04T04:04:40.152000+00:00', 'https://storage.googleapis.com/bln_prod/...', 'download', 'UHJvamVjdDo4MTgyMjU2Ny01ZjI0LTQxM2ItYWZmNi05NTlmNGY3MjExMjI=', 'loans_to_documentation.csv']
(Pdb) result.lastrowid
100
```
But here's the weird thing... there's no row in the table with a rowid of 100!
```
(Pdb) [r['rowid'] for r in self.db.execute_returning_dicts('select rowid, * from files')]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]
```
So what the heck is going on?
The last SQL statement I executed here was an `UPDATE`. The `lastrowid` docs say: https://kite.com/python/docs/sqlite3.Cursor.lastrowid
> This read-only attribute provides the rowid of the last modified row. It is only set if you issued a INSERT statement using the execute() method. For operations other than INSERT or when executemany() is called, lastrowid is set to None.
So where did that `100` come from? It should be `None`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records",
https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612258687,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612258687,MDEyOklzc3VlQ29tbWVudDYxMjI1ODY4Nw==,9599,simonw,2020-04-10T23:08:48Z,2020-04-10T23:08:48Z,OWNER,I need a test that reproduces this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records",
https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612173156,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612173156,MDEyOklzc3VlQ29tbWVudDYxMjE3MzE1Ng==,9599,simonw,2020-04-10T19:03:32Z,2020-04-10T23:08:28Z,OWNER,"Investigate this traceback:
```
Traceback (most recent call last):
File ""fetch_projects.py"", line 60, in
fetch_projects(db, token)
File ""fetch_projects.py"", line 41, in fetch_projects
db[""projects""].upsert(project, pk=""id"")
File ""/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1139, in upsert
conversions=conversions,
File ""/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1168, in upsert_all
upsert=True,
File ""/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1107, in insert_all
row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0]
IndexError: list index out of range
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records",
https://github.com/simonw/datasette/issues/236#issuecomment-612216820,https://api.github.com/repos/simonw/datasette/issues/236,612216820,MDEyOklzc3VlQ29tbWVudDYxMjIxNjgyMA==,193185,cldellow,2020-04-10T21:03:38Z,2020-04-10T21:03:38Z,CONTRIBUTOR,"I made a repo at https://github.com/code402/datasette-lambda to demonstrate the idea, and scratch my personal itch for this.
The demo relies on some central authority having already published a public, reusable Lambda layer with Datasette & its dependencies. I think that differs from the other publish plugins which seem to mainly publish Dockerfiles that the host will interpret to install deps from a requirements.txt file.
I chose that approach because `uvloop` appears to be a dependency with native code that needs to be compiled for the target runtime environment. In this case, that's Amazon Linux 2. I'm not 100% clear on whether that's still required, because:
- maybe `uvloop` is only needed for `uvicorn`, which the demo doesn't actually use since HTTP routing is handled by API Gateway
- it seems like `uvloop` may be an optional, drop-in optimization for `asyncio` in any case (but I may be misreading this; I'm very much a Python noob)
If it's the case that `uvloop` is truly optional, then I think the publish plugin could do the packaging on the user's machine, regardless of what flavour of operating system they're on. That'd be a bit slower for the user, but would provide the most long-term flexibility in terms of supporting plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/simonw/datasette/issues/724#issuecomment-612156367,https://api.github.com/repos/simonw/datasette/issues/724,612156367,MDEyOklzc3VlQ29tbWVudDYxMjE1NjM2Nw==,9599,simonw,2020-04-10T18:23:39Z,2020-04-10T18:23:39Z,OWNER,I think I'll use https://github.com/clarketm/mergedeep (MIT license),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",598013965,--plugin-secret over-rides existing metadata.json plugin config,
https://github.com/simonw/datasette/issues/724#issuecomment-612155297,https://api.github.com/repos/simonw/datasette/issues/724,612155297,MDEyOklzc3VlQ29tbWVudDYxMjE1NTI5Nw==,9599,simonw,2020-04-10T18:20:53Z,2020-04-10T18:20:53Z,OWNER,Lot's of recipes for deep dictionary merge on https://stackoverflow.com/q/7204805/6083,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",598013965,--plugin-secret over-rides existing metadata.json plugin config,
https://github.com/simonw/datasette/issues/724#issuecomment-612154749,https://api.github.com/repos/simonw/datasette/issues/724,612154749,MDEyOklzc3VlQ29tbWVudDYxMjE1NDc0OQ==,9599,simonw,2020-04-10T18:19:31Z,2020-04-10T18:19:31Z,OWNER,"Actually that code isn't the problem, since `extra_metadata` has not yet been read from `metadata.json` at that point.
The bug is here: https://github.com/simonw/datasette/blob/d55fe8cdfc2ce7bc6960bf2507766c1fcd1d31a7/datasette/utils/__init__.py#L362-L368
I need to merge dictionaries in a smarter way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",598013965,--plugin-secret over-rides existing metadata.json plugin config,
https://github.com/simonw/datasette/issues/699#issuecomment-611880250,https://api.github.com/repos/simonw/datasette/issues/699,611880250,MDEyOklzc3VlQ29tbWVudDYxMTg4MDI1MA==,9599,simonw,2020-04-10T05:08:54Z,2020-04-10T05:08:54Z,OWNER,"So maybe this is all handled by plugin hooks?
`auth_from_scope(datasette, scope)` would be the main one - for deciding if a user should be authenticated based on data from the scope.
How would a permissions hook work though?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/699#issuecomment-611879821,https://api.github.com/repos/simonw/datasette/issues/699,611879821,MDEyOklzc3VlQ29tbWVudDYxMTg3OTgyMQ==,9599,simonw,2020-04-10T05:06:56Z,2020-04-10T05:06:56Z,OWNER,"Another problem this would solve: if you want multiple authentication mechanisms - GitHub auth for users, `Authorization: bearer xxx` auth for API keys - the order in which they run might end up mattering.
I dealt with this a bit in https://github.com/simonw/datasette-auth-github/issues/59
But having an authentication plugin hook - where playing get to decide if a user should be authenticated based on the incoming ASGI scopes - would be neater.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",582526961,Authentication (and permissions) as a core concept,
https://github.com/simonw/datasette/issues/514#issuecomment-610186532,https://api.github.com/repos/simonw/datasette/issues/514,610186532,MDEyOklzc3VlQ29tbWVudDYxMDE4NjUzMg==,9599,simonw,2020-04-07T05:43:45Z,2020-04-07T05:43:45Z,OWNER,I've run Datasette in production directly on Ubuntu with systemd in a few places now. I think I'm ready to turn this into real documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459397625,Documentation with recommendations on running Datasette in production without using Docker,
https://github.com/simonw/datasette/issues/717#issuecomment-610076073,https://api.github.com/repos/simonw/datasette/issues/717,610076073,MDEyOklzc3VlQ29tbWVudDYxMDA3NjA3Mw==,9599,simonw,2020-04-06T22:47:21Z,2020-04-06T22:47:21Z,OWNER,I'm confident it's possible to create a plugin that deploys to Now v2 now. I'll do the rest of the work in a separate repo: https://github.com/simonw/datasette-publish-now,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594189527,See if I can get Datasette working on Zeit Now v2,
https://github.com/simonw/datasette/issues/717#issuecomment-610067097,https://api.github.com/repos/simonw/datasette/issues/717,610067097,MDEyOklzc3VlQ29tbWVudDYxMDA2NzA5Nw==,9599,simonw,2020-04-06T22:20:11Z,2020-04-06T22:20:11Z,OWNER,"Yes I think that's it - I created a table in `about.db`, then dropped that table and ran `vacuum` and the result was an empty database file that wasn't zero bytes. I deployed it and it worked: https://datasette-koo7abu7j.now.sh/ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594189527,See if I can get Datasette working on Zeit Now v2,
https://github.com/simonw/datasette/issues/717#issuecomment-610066089,https://api.github.com/repos/simonw/datasette/issues/717,610066089,MDEyOklzc3VlQ29tbWVudDYxMDA2NjA4OQ==,9599,simonw,2020-04-06T22:17:32Z,2020-04-06T22:17:53Z,OWNER,I think that error might be because Niche Museums uses two database files that are 0 bytes (about.db and map.db) - I just tried purely publishing the empty about.db file and got the same error: https://datasette-ks55wp251.now.sh/ - https://datasette-ks55wp251.now.sh/-/databases,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594189527,See if I can get Datasette working on Zeit Now v2,
https://github.com/simonw/datasette/issues/717#issuecomment-609905777,https://api.github.com/repos/simonw/datasette/issues/717,609905777,MDEyOklzc3VlQ29tbWVudDYwOTkwNTc3Nw==,9599,simonw,2020-04-06T16:41:04Z,2020-04-06T16:43:05Z,OWNER,"Full traceback from Zeit Now logs:
```
ERROR: conn=, sql = ""select name from sqlite_master where type='table'"", params = None: disk I/O error
Traceback (most recent call last):
File ""/var/task/datasette/utils/asgi.py"", line 121, in route_path
return await view(new_scope, receive, send)
File ""/var/task/datasette/utils/asgi.py"", line 193, in view
request, **scope[""url_route""][""kwargs""]
File ""/var/task/datasette/views/base.py"", line 61, in head
response = await self.get(*args, **kwargs)
File ""/var/task/datasette/views/index.py"", line 27, in get
table_names = await db.table_names()
File ""/var/task/datasette/database.py"", line 221, in table_names
""select name from sqlite_master where type='table'""
File ""/var/task/datasette/database.py"", line 167, in execute
sql_operation_in_thread
File ""/var/task/datasette/database.py"", line 114, in execute_against_connection_in_thread
self.ds.executor, in_thread
File ""/var/lang/lib/python3.6/concurrent/futures/thread.py"", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File ""/var/task/datasette/database.py"", line 111, in in_thread
return fn(conn)
File ""/var/task/datasette/database.py"", line 137, in sql_operation_in_thread
cursor.execute(sql, params or {})sqlite3.OperationalError: disk I/O error
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594189527,See if I can get Datasette working on Zeit Now v2,
https://github.com/simonw/datasette/issues/717#issuecomment-609905147,https://api.github.com/repos/simonw/datasette/issues/717,609905147,MDEyOklzc3VlQ29tbWVudDYwOTkwNTE0Nw==,9599,simonw,2020-04-06T16:39:50Z,2020-04-06T16:39:50Z,OWNER,"I tried deploying the whole of www.niche-museums.com with mixed results.
https://datasette-jm25oxd7s.now.sh/browse/museums works
https://datasette-jm25oxd7s.now.sh/ shows ""disk I/O error""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594189527,See if I can get Datasette working on Zeit Now v2,
https://github.com/simonw/datasette/issues/717#issuecomment-609889528,https://api.github.com/repos/simonw/datasette/issues/717,609889528,MDEyOklzc3VlQ29tbWVudDYwOTg4OTUyOA==,9599,simonw,2020-04-06T16:11:07Z,2020-04-06T16:11:07Z,OWNER,"https://now-2-datasette-e4l4whc9f.now.sh/-/versions
```json
{
""python"": {
""version"": ""3.6.10"",
""full"": ""3.6.10 (default, Mar 10 2020, 22:54:43) \n[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)]""
},
""datasette"": {
""version"": ""0.39""
},
""asgi"": ""3.0"",
""uvicorn"": ""0.11.3"",
""sqlite"": {
""version"": ""3.7.17"",
""fts_versions"": [
""FTS4"",
""FTS3""
],
""extensions"": {},
""compile_options"": [
""DISABLE_DIRSYNC"",
""ENABLE_COLUMN_METADATA"",
""ENABLE_FTS3"",
""ENABLE_RTREE"",
""ENABLE_UNLOCK_NOTIFY"",
""SECURE_DELETE"",
""TEMP_STORE=1"",
""THREADSAFE=1""
]
}
}
```
SQLite 3.7.17 is from May 2013: http://www.sqlite.org/releaselog/3_7_17.html - and there's no FTS5 (but we do get FTS4).
So the Now v2 version will be a bit limited (no support for table names containing `/` and an ancient SQLite version without FTS5) - but it's definitely enough for a `datasette-publish-now` plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594189527,See if I can get Datasette working on Zeit Now v2,
https://github.com/simonw/datasette/issues/717#issuecomment-609887635,https://api.github.com/repos/simonw/datasette/issues/717,609887635,MDEyOklzc3VlQ29tbWVudDYwOTg4NzYzNQ==,9599,simonw,2020-04-06T16:07:51Z,2020-04-06T16:08:47Z,OWNER,"Built myself a quick ASGI scope debugging tool: https://now-2-asgi-scope.now.sh/
Same `now.json` as above, but `index.py` is this:
```python
from pprint import pformat
async def app(scope, receive, send):
await send({
'type': 'http.response.start',
'status': 200,
'headers': [
[b'content-type', b'text/plain'],
],
})
await send({
'type': 'http.response.body',
'body': pformat(scope).encode('utf8'),
})
```
https://now-2-asgi-scope.now.sh/fixtures/table%2Fwith%2Fslashes.csv shows what's going on - those `%2F` have been decoded to `/` before they get to the ASGI app - probably by the Now routing layer:
```
...
'http_version': '1.1',
'method': 'GET',
'path': '/fixtures/table/with/slashes.csv',
'query_string': b'',
'raw_path': b'/fixtures/table/with/slashes.csv',
...
```
That `raw_path` there needs to be `b'/fixtures/table%2Fwith%2Fslashes.csv'` in order for Datasette to fully work here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594189527,See if I can get Datasette working on Zeit Now v2,
https://github.com/simonw/datasette/issues/717#issuecomment-609883545,https://api.github.com/repos/simonw/datasette/issues/717,609883545,MDEyOklzc3VlQ29tbWVudDYwOTg4MzU0NQ==,9599,simonw,2020-04-06T16:00:41Z,2020-04-06T16:03:21Z,OWNER,"https://now-2-datasette-e4l4whc9f.now.sh/ - first proof of concept!
It's just four files right now, all in the same directory and deployed by running `now`.
`now.json`
```json
{
""version"": 2,
""builds"": [
{
""src"": ""index.py"",
""use"": ""@now/python""
}
],
""routes"": [
{
""src"": ""(.*)"",
""dest"": ""index.py""
}
]
}
```
`requirements.txt`:
```
datasette
datasette-debug-asgi
```
`index.py`:
```python
from datasette.app import Datasette
app = Datasette([""fixtures.db""]).app()
```
Plus a copy of the standard `fixtures.db` file.
It almost entirely works! Just one bug so far:
https://now-2-datasette-e4l4whc9f.now.sh/fixtures/table%2Fwith%2Fslashes.csv returns a 404
https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv does not.
Plugins work too - here's the output from `datasette-debug-asgi`: https://now-2-datasette-e4l4whc9f.now.sh/-/asgi-scope","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594189527,See if I can get Datasette working on Zeit Now v2,
https://github.com/simonw/datasette/issues/409#issuecomment-609874145,https://api.github.com/repos/simonw/datasette/issues/409,609874145,MDEyOklzc3VlQ29tbWVudDYwOTg3NDE0NQ==,9599,simonw,2020-04-06T15:44:46Z,2020-04-06T15:44:46Z,OWNER,Closing this in favour of #717.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",408376825,Zeit API v1 does not work for new users - need to migrate to v2,
https://github.com/simonw/datasette/issues/716#issuecomment-609461331,https://api.github.com/repos/simonw/datasette/issues/716,609461331,MDEyOklzc3VlQ29tbWVudDYwOTQ2MTMzMQ==,9599,simonw,2020-04-05T18:31:32Z,2020-04-05T20:04:08Z,OWNER,"The test ended up being a bit fiddly - here it is:
https://github.com/simonw/datasette/blob/09253817dea3c131553494f9b2eb9c03f94ae761/tests/test_plugins.py#L266-L317
I used `tmp_path_factory` here because the `tmpdir` fixture I usually use isn't compatible with `scope=""session""`, and I wanted to only create those temporary plugins and templates directories once rather than create them for each run of the parametrized test function.
In writing this I realized that the `name` on the `QueryView` class wasn't being used, because that class is currently just used for its `.data()` method:
https://github.com/simonw/datasette/blob/09253817dea3c131553494f9b2eb9c03f94ae761/datasette/views/database.py#L27-L31
https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/views/table.py#L224-L227
So I removed the `name = ""query""` line since it was misleading.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594168758,extra_template_vars() sending wrong view_name for index,
https://github.com/simonw/datasette/issues/716#issuecomment-609456030,https://api.github.com/repos/simonw/datasette/issues/716,609456030,MDEyOklzc3VlQ29tbWVudDYwOTQ1NjAzMA==,9599,simonw,2020-04-05T17:52:39Z,2020-04-05T20:02:13Z,OWNER,"Found it. Prior to that change I passed `view_name` to the callbacks like this:
https://github.com/simonw/datasette/blob/286ed286b68793532c2a38436a08343b45cfbc91/datasette/views/base.py#L114-L132
I switched over to doing this:
https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/views/base.py#L95-L97
But forgot to pass in the optional `view_name=` argument to that `render_template()` method:
https://github.com/simonw/datasette/blob/2aaad72789c427875426673c1a43e67c86fc970e/datasette/app.py#L554-L556
Next step: write a failing test, then fix it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594168758,extra_template_vars() sending wrong view_name for index,
https://github.com/simonw/datasette/issues/689#issuecomment-609469440,https://api.github.com/repos/simonw/datasette/issues/689,609469440,MDEyOklzc3VlQ29tbWVudDYwOTQ2OTQ0MA==,9599,simonw,2020-04-05T19:28:59Z,2020-04-05T19:28:59Z,OWNER,"So I think the fix is to move the `""select_templates"": select_templates` context setting bit to here instead:
https://github.com/simonw/datasette/blob/e89b0ef2f9ae89eb3bde83b694f21452ea4858da/datasette/views/base.py#L75-L83
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573583971,"""Templates considered"" comment broken in >=0.35",
https://github.com/simonw/datasette/issues/689#issuecomment-609469318,https://api.github.com/repos/simonw/datasette/issues/689,609469318,MDEyOklzc3VlQ29tbWVudDYwOTQ2OTMxOA==,9599,simonw,2020-04-05T19:28:14Z,2020-04-05T19:28:14Z,OWNER,"Here's why: the `BaseView.render()` method is running `jinja_env.select_template()` now here:
https://github.com/simonw/datasette/blob/e89b0ef2f9ae89eb3bde83b694f21452ea4858da/datasette/views/base.py#L75-L88
Which means this logic is always called with a template, not a list of strings:
https://github.com/simonw/datasette/blob/e89b0ef2f9ae89eb3bde83b694f21452ea4858da/datasette/app.py#L555-L571","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573583971,"""Templates considered"" comment broken in >=0.35",
https://github.com/simonw/datasette/issues/689#issuecomment-609468485,https://api.github.com/repos/simonw/datasette/issues/689,609468485,MDEyOklzc3VlQ29tbWVudDYwOTQ2ODQ4NQ==,9599,simonw,2020-04-05T19:22:31Z,2020-04-05T19:22:31Z,OWNER,"```
$ git bisect start master 0.34
Bisecting: 32 revisions left to test after this (roughly 5 steps)
[dc80e779a2e708b2685fc641df99e6aae9ad6f97] Handle scope path if it is a string
$ git bisect run python check_templates_considered.py
running python check_templates_considered.py
Traceback (most recent call last):
...
AssertionError
Bisecting: 15 revisions left to test after this (roughly 4 steps)
[7c6a9c35299f251f9abfb03fd8e85143e4361709] Better tests for prepare_connection() plugin hook, refs #678
running python check_templates_considered.py
Traceback (most recent call last):
...
AssertionError
Bisecting: 7 revisions left to test after this (roughly 3 steps)
[0091dfe3e5a3db94af8881038d3f1b8312bb857d] More reliable tie-break ordering for facet results
running python check_templates_considered.py
Traceback (most recent call last):
...
AssertionError
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[ce12244037b60ba0202c814871218c1dab38d729] Release notes for 0.35
running python check_templates_considered.py
Traceback (most recent call last):
...
AssertionError
Bisecting: 1 revision left to test after this (roughly 1 step)
[70b915fb4bc214f9d064179f87671f8a378aa127] Datasette.render_template() method, closes #577
running python check_templates_considered.py
Traceback (most recent call last):
...
AssertionError
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[286ed286b68793532c2a38436a08343b45cfbc91] geojson-to-sqlite
running python check_templates_considered.py
70b915fb4bc214f9d064179f87671f8a378aa127 is the first bad commit
commit 70b915fb4bc214f9d064179f87671f8a378aa127
Author: Simon Willison
Date: Tue Feb 4 12:26:17 2020 -0800
Datasette.render_template() method, closes #577
Pull request #664.
:040000 040000 def9e31252e056845609de36c66d4320dd0c47f8 da19b7f8c26d50a4c05e5a7f05220b968429725c M datasette
bisect run success
```
It was 70b915fb4bc214f9d064179f87671f8a378aa127 - the same bad commit that caused #716!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573583971,"""Templates considered"" comment broken in >=0.35",
https://github.com/simonw/datasette/issues/689#issuecomment-609468180,https://api.github.com/repos/simonw/datasette/issues/689,609468180,MDEyOklzc3VlQ29tbWVudDYwOTQ2ODE4MA==,9599,simonw,2020-04-05T19:20:33Z,2020-04-05T19:20:33Z,OWNER,"```
git bisect start master 0.34
git bisect run python check_templates_considered.py
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573583971,"""Templates considered"" comment broken in >=0.35",
https://github.com/simonw/datasette/issues/689#issuecomment-609467876,https://api.github.com/repos/simonw/datasette/issues/689,609467876,MDEyOklzc3VlQ29tbWVudDYwOTQ2Nzg3Ng==,9599,simonw,2020-04-05T19:18:36Z,2020-04-05T19:18:36Z,OWNER,"Just need the one checking script to run with bisect this time:
`check_templates_considered.py`
```python
import asyncio
import pathlib
from datasette.app import Datasette
import httpx
async def run_check():
ds = Datasette([])
async with httpx.AsyncClient(app=ds.app()) as client:
response = await client.get(""http://localhost/"")
assert 200 == response.status_code
assert ""Templates considered"" in response.text
if __name__ == ""__main__"":
loop = asyncio.get_event_loop()
loop.run_until_complete(run_check())
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573583971,"""Templates considered"" comment broken in >=0.35",
https://github.com/simonw/datasette/issues/689#issuecomment-609467523,https://api.github.com/repos/simonw/datasette/issues/689,609467523,MDEyOklzc3VlQ29tbWVudDYwOTQ2NzUyMw==,9599,simonw,2020-04-05T19:16:13Z,2020-04-05T19:16:13Z,OWNER,I'm going to debug this using `git bisect run` - the same technique I used in #716 - https://github.com/simonw/datasette/issues/716#issuecomment-609453886,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573583971,"""Templates considered"" comment broken in >=0.35",
https://github.com/simonw/datasette/issues/693#issuecomment-609466998,https://api.github.com/repos/simonw/datasette/issues/693,609466998,MDEyOklzc3VlQ29tbWVudDYwOTQ2Njk5OA==,9599,simonw,2020-04-05T19:12:48Z,2020-04-05T19:12:48Z,OWNER,Fixed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",574043218,Variables from extra_template_vars() not exposed in _context=1,
https://github.com/simonw/datasette/issues/693#issuecomment-609461623,https://api.github.com/repos/simonw/datasette/issues/693,609461623,MDEyOklzc3VlQ29tbWVudDYwOTQ2MTYyMw==,9599,simonw,2020-04-05T18:33:46Z,2020-04-05T18:33:46Z,OWNER,So I should move the `template_debug` and `_context` logic into the `render_template()` method.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",574043218,Variables from extra_template_vars() not exposed in _context=1,
https://github.com/simonw/datasette/issues/716#issuecomment-609455243,https://api.github.com/repos/simonw/datasette/issues/716,609455243,MDEyOklzc3VlQ29tbWVudDYwOTQ1NTI0Mw==,9599,simonw,2020-04-05T17:47:33Z,2020-04-05T17:47:33Z,OWNER,"You start `git bisect` by giving it a known bad commit and a known good one:
```
git bisect start master 286ed28
```
Then you tell it to start running your script:
```
git bisect run python ../datasette-issue-716/check_view_name.py
```
Here's what I got:
```
(datasette) ~/Dropbox/Development/datasette $ git bisect start master 286ed28
Bisecting: 30 revisions left to test after this (roughly 5 steps)
[dc80e779a2e708b2685fc641df99e6aae9ad6f97] Handle scope path if it is a string
(datasette) ~/Dropbox/Development/datasette $ git bisect run python ../datasette-issue-716/check_view_name.py
running python ../datasette-issue-716/check_view_name.py
Traceback (most recent call last):
...
Bisecting: 15 revisions left to test after this (roughly 4 steps)
[7c6a9c35299f251f9abfb03fd8e85143e4361709] Better tests for prepare_connection() plugin hook, refs #678
running python ../datasette-issue-716/check_view_name.py
Traceback (most recent call last):
...
Bisecting: 7 revisions left to test after this (roughly 3 steps)
[0091dfe3e5a3db94af8881038d3f1b8312bb857d] More reliable tie-break ordering for facet results
running python ../datasette-issue-716/check_view_name.py
Traceback (most recent call last):
...
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[ce12244037b60ba0202c814871218c1dab38d729] Release notes for 0.35
running python ../datasette-issue-716/check_view_name.py
Traceback (most recent call last):
...
Bisecting: 0 revisions left to test after this (roughly 1 step)
[4d7dae9eb75e5430c3ee3c369bb5cd9ba0a148bc] Added a bunch more plugins to the Ecosystem page
running python ../datasette-issue-716/check_view_name.py
Traceback (most recent call last):
...
70b915fb4bc214f9d064179f87671f8a378aa127 is the first bad commit
commit 70b915fb4bc214f9d064179f87671f8a378aa127
Author: Simon Willison
Date: Tue Feb 4 12:26:17 2020 -0800
Datasette.render_template() method, closes #577
Pull request #664.
:040000 040000 def9e31252e056845609de36c66d4320dd0c47f8 da19b7f8c26d50a4c05e5a7f05220b968429725c M datasette
bisect run success
```
So 70b915fb4bc214f9d064179f87671f8a378aa127 introduced the bug!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",594168758,extra_template_vars() sending wrong view_name for index,
https://github.com/simonw/datasette/issues/716#issuecomment-609453886,https://api.github.com/repos/simonw/datasette/issues/716,609453886,MDEyOklzc3VlQ29tbWVudDYwOTQ1Mzg4Ng==,9599,simonw,2020-04-05T17:38:38Z,2020-04-05T17:42:14Z,OWNER,"OK, here's the test harness. Three files:
`check_view_name.py`
```python
import asyncio
import pathlib
from datasette.app import Datasette
import httpx
root = pathlib.Path(__file__).parent
async def run_check():
ds = Datasette(
[], template_dir=str(root / ""templates""), plugins_dir=str(root / ""plugins"")
)
async with httpx.AsyncClient(app=ds.app()) as client:
response = await client.get(""http://localhost/"")
assert 200 == response.status_code
assert b""view_name:index"" == response.content, response.content
if __name__ == ""__main__"":
loop = asyncio.get_event_loop()
loop.run_until_complete(run_check())
```
`templates/index.html`
```
view_name:{{ view_name }}
```
`plugins/extra_vars.py`
```python
from datasette import hookimpl
@hookimpl
def extra_template_vars(view_name):
return {""view_name"": view_name}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594168758,extra_template_vars() sending wrong view_name for index,
https://github.com/simonw/datasette/issues/716#issuecomment-609454072,https://api.github.com/repos/simonw/datasette/issues/716,609454072,MDEyOklzc3VlQ29tbWVudDYwOTQ1NDA3Mg==,9599,simonw,2020-04-05T17:39:47Z,2020-04-05T17:39:47Z,OWNER,"`python check_view_name.py` against 286ed28 exits cleanly.
`python check_view_name.py` against current master (e0e7a0fa) fails:
```
$ python check_view_name.py
Traceback (most recent call last):
File ""check_view_name.py"", line 16, in
loop.run_until_complete(run_check())
File ""/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py"", line 579, in run_until_complete
return future.result()
File ""check_view_name.py"", line 11, in run_check
assert b""view_name:index"" == response.content, response.content
AssertionError: b'view_name:None'
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594168758,extra_template_vars() sending wrong view_name for index,
https://github.com/simonw/datasette/pull/627#issuecomment-609393513,https://api.github.com/repos/simonw/datasette/issues/627,609393513,MDEyOklzc3VlQ29tbWVudDYwOTM5MzUxMw==,4312421,stonebig,2020-04-05T10:23:57Z,2020-04-05T10:23:57Z,NONE,is there any specific reason to stick to Jinja2~=2.10.3 when there is Jinja-2.11.1 ?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521323012,"Support Python 3.8, stop supporting Python 3.5",
https://github.com/simonw/datasette/issues/717#issuecomment-609306846,https://api.github.com/repos/simonw/datasette/issues/717,609306846,MDEyOklzc3VlQ29tbWVudDYwOTMwNjg0Ng==,9599,simonw,2020-04-05T04:18:48Z,2020-04-05T04:18:48Z,OWNER,"I need to route all paths to the same function. This should help: https://twitter.com/aboodman/status/1246605658067066882
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594189527,See if I can get Datasette working on Zeit Now v2,
https://github.com/simonw/datasette/issues/717#issuecomment-609111516,https://api.github.com/repos/simonw/datasette/issues/717,609111516,MDEyOklzc3VlQ29tbWVudDYwOTExMTUxNg==,9599,simonw,2020-04-05T00:57:29Z,2020-04-05T00:57:29Z,OWNER,"If I can get this working I think I'll release it as a plugin, similar to [datasette-publish-fly](https://github.com/simonw/datasette-publish-fly).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594189527,See if I can get Datasette working on Zeit Now v2,
https://github.com/simonw/datasette/issues/716#issuecomment-609109354,https://api.github.com/repos/simonw/datasette/issues/716,609109354,MDEyOklzc3VlQ29tbWVudDYwOTEwOTM1NA==,9599,simonw,2020-04-05T00:30:30Z,2020-04-05T00:31:55Z,OWNER,"In order to use `git bisect run` I need to write a standalone script that can tell if the bug is present or not.
I'm going to use something I can execute with `pytest`, inspired loosely by this code:
https://github.com/simonw/datasette/blob/7656fd64d8b6a32ebc34d89c1b8711cc5ea240f7/tests/test_plugins.py#L205-L227","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594168758,extra_template_vars() sending wrong view_name for index,
https://github.com/simonw/datasette/issues/716#issuecomment-609107679,https://api.github.com/repos/simonw/datasette/issues/716,609107679,MDEyOklzc3VlQ29tbWVudDYwOTEwNzY3OQ==,9599,simonw,2020-04-05T00:10:37Z,2020-04-05T00:11:06Z,OWNER,"Once I fix this bug I should update https://github.com/simonw/museums to deploy using the latest Datasette release as opposed to being anchored to 286ed28.
https://github.com/simonw/museums/blob/1bbed542617757e9e276a5098193d6288b7f421d/.github/workflows/push.yml#L61","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594168758,extra_template_vars() sending wrong view_name for index,
https://github.com/simonw/datasette/issues/716#issuecomment-609106252,https://api.github.com/repos/simonw/datasette/issues/716,609106252,MDEyOklzc3VlQ29tbWVudDYwOTEwNjI1Mg==,9599,simonw,2020-04-04T23:58:12Z,2020-04-04T23:58:12Z,OWNER,"I think I'll use `git bisect run` for this one, mainly to practice using it. https://git-scm.com/docs/git-bisect#_bisect_run","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",594168758,extra_template_vars() sending wrong view_name for index,
https://github.com/simonw/datasette/issues/710#issuecomment-608973849,https://api.github.com/repos/simonw/datasette/issues/710,608973849,MDEyOklzc3VlQ29tbWVudDYwODk3Mzg0OQ==,9599,simonw,2020-04-04T04:52:22Z,2020-04-04T04:52:44Z,OWNER,"Need to remove this entirely - it's going away in August (and new deployments will stop working in May).
From an email they just sent out:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",587322443,Remove Zeit Now v1 support,
https://github.com/simonw/datasette/issues/236#issuecomment-608716819,https://api.github.com/repos/simonw/datasette/issues/236,608716819,MDEyOklzc3VlQ29tbWVudDYwODcxNjgxOQ==,193185,cldellow,2020-04-03T22:19:00Z,2020-04-03T22:19:00Z,CONTRIBUTOR,"Hi Simon,
I'm thinking of attempting this. Can you clarify some questions I have?
1) I assume the goal is to have a CORS-friendly HTTPS endpoint that hosts the datasette service + user's db.
2) If that's the goal, I think Lambda alone is insufficient. Lambda provides the compute fabric, but not the HTTP routing. You'd also need to add Application Load Balancer or API Gateway to provide an HTTP endpoint that routes to the lambda function.
Do you have a preference between ALB or API GW? ALB has better economics at scale, but has a minimum monthly cost. API GW has worse per-request economics, but scales to zero when no requests are happening.
3) Does Datasette have any native components, or is it all pure python? If it has native bits, they'll likely need to be recompiled to work on Amazon Linux 2.
4) There are a few disparate services that need to be wired together to expose a Python service securely to the web. If I was doing this outside of the datasette publish system, I'd use an AWS CloudFormation template. Even within datasette, I think it still makes sense to use a CloudFormation template and just have the publish plugin invoke it (via the standard `aws` cli) with user-specified parameters. Does that sound reasonable to you?
Thanks for your help!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/simonw/datasette/pull/703#issuecomment-608137641,https://api.github.com/repos/simonw/datasette/issues/703,608137641,MDEyOklzc3VlQ29tbWVudDYwODEzNzY0MQ==,9599,simonw,2020-04-02T23:08:26Z,2020-04-02T23:08:26Z,OWNER,I'm going to split canned queries out from arbitrary queries to make the code easier to follow.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",585597133,WIP implementation of writable canned queries,
https://github.com/simonw/datasette/issues/698#issuecomment-608125928,https://api.github.com/repos/simonw/datasette/issues/698,608125928,MDEyOklzc3VlQ29tbWVudDYwODEyNTkyOA==,9599,simonw,2020-04-02T22:32:41Z,2020-04-02T22:33:10Z,OWNER,"I really want the option to use a `
View Datasette (Click on the stop button to close the Datasette server)
')) # Launch Datasette with Popen( [ 'python', '-m', 'datasette', '--', database, '--port', str(port), '--config', f'base_url:{proxy_url}' ], stdout=PIPE, stderr=PIPE, bufsize=1, universal_newlines=True ) as p: print(p.stdout.readline(), end='') while True: try: line = p.stderr.readline() if not line: break print(line, end='') exit_code = p.poll() except KeyboardInterrupt: p.send_signal(SIGINT) ``` Ideally, I'd like some extra magic to notify users when they are leaving the closing the notebook tab and make them terminate the running datasette processes. I'll be looking for it.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",396212021,base_url configuration setting, https://github.com/simonw/datasette/issues/818#issuecomment-642420375,https://api.github.com/repos/simonw/datasette/issues/818,642420375,MDEyOklzc3VlQ29tbWVudDY0MjQyMDM3NQ==,9599,simonw,2020-06-11T05:40:07Z,2020-06-11T05:40:07Z,OWNER,https://github.com/simonw/datasette-permissions-sql is now released as a 0.1a here: https://pypi.org/project/datasette-permissions-sql/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin, https://github.com/simonw/datasette/issues/832#issuecomment-642412017,https://api.github.com/repos/simonw/datasette/issues/832,642412017,MDEyOklzc3VlQ29tbWVudDY0MjQxMjAxNw==,9599,simonw,2020-06-11T05:13:59Z,2020-06-11T05:13:59Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/ce4958018ede00fbdadf0c37a99889b6901bfb9b/datasette/views/table.py#L267-L272","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table, https://github.com/simonw/datasette/issues/831#issuecomment-642324847,https://api.github.com/repos/simonw/datasette/issues/831,642324847,MDEyOklzc3VlQ29tbWVudDY0MjMyNDg0Nw==,9599,simonw,2020-06-10T23:50:55Z,2020-06-10T23:50:55Z,OWNER,"Actually I'm not sure about this. If `""allow"": null` means ""no-one can do this"", what's the allow block syntax for ""everyone can do this""? It could be `""allow"": {}` - but that's not intuitive because normally the allow block shows keys that need to match. `{}` suggests to me that no matches are possible. So I think I'm going to stick with the current mechanism, which is that `""allow"": null` means ""anyone can do this"" and `""allow"": {}` means ""no-one can do this"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636614868,"It would be more intuitive if ""allow"": none meant ""no-one can do this""", https://github.com/simonw/datasette/issues/818#issuecomment-642231871,https://api.github.com/repos/simonw/datasette/issues/818,642231871,MDEyOklzc3VlQ29tbWVudDY0MjIzMTg3MQ==,9599,simonw,2020-06-10T20:11:50Z,2020-06-10T20:11:50Z,OWNER,"`datasette-permissions-sql` ```yaml plugins: datasette-permissions-sql: view-instance: |- select count(*) from users where admin = 1 and id = :id ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin, https://github.com/simonw/datasette/issues/818#issuecomment-642230499,https://api.github.com/repos/simonw/datasette/issues/818,642230499,MDEyOklzc3VlQ29tbWVudDY0MjIzMDQ5OQ==,9599,simonw,2020-06-10T20:08:46Z,2020-06-10T20:09:26Z,OWNER,"What's a simple but useful plugin I could release that exercises this hook? Ideally one which executes permission checks against the database somehow. I could do a simplest-possible implementation of the idea in #801 (allow-by-query).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin, https://github.com/simonw/datasette/issues/818#issuecomment-642229899,https://api.github.com/repos/simonw/datasette/issues/818,642229899,MDEyOklzc3VlQ29tbWVudDY0MjIyOTg5OQ==,9599,simonw,2020-06-10T20:07:36Z,2020-06-10T20:07:36Z,OWNER,"New policy in 9f236c4 dictates that this should be in Milestone 0.44 after all: > * **New plugin hooks** should only be shipped if accompanied by a separate release of a non-demo plugin that uses them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634917088,Example permissions plugin, https://github.com/simonw/datasette/issues/829#issuecomment-642217520,https://api.github.com/repos/simonw/datasette/issues/829,642217520,MDEyOklzc3VlQ29tbWVudDY0MjIxNzUyMA==,9599,simonw,2020-06-10T19:41:35Z,2020-06-10T19:41:35Z,OWNER,"I didn't bother with the alternative epoch - it only shaves off two or three bytes from the cookie. Documentation for the new `ds_actor` cookie shape is here: https://datasette.readthedocs.io/en/latest/authentication.html#the-ds-actor-cookie","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636426530,Ability to set ds_actor cookie such that it expires, https://github.com/simonw/datasette/issues/829#issuecomment-642178604,https://api.github.com/repos/simonw/datasette/issues/829,642178604,MDEyOklzc3VlQ29tbWVudDY0MjE3ODYwNA==,9599,simonw,2020-06-10T18:18:36Z,2020-06-10T18:20:19Z,OWNER,"Even shorter: encode an integer that is the difference between that expiry timestamp and a more recent epoch - June 1st 2020 will do. ``` >>> import datetime, calendar >>> calendar.timegm(datetime.date(2020, 6, 1).timetuple()) 1590969600 >>> import baseconv >>> baseconv.base62.encode(int(time.time() - 1590969600)) '3XST' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636426530,Ability to set ds_actor cookie such that it expires, https://github.com/simonw/datasette/issues/829#issuecomment-642176180,https://api.github.com/repos/simonw/datasette/issues/829,642176180,MDEyOklzc3VlQ29tbWVudDY0MjE3NjE4MA==,9599,simonw,2020-06-10T18:14:02Z,2020-06-10T18:14:15Z,OWNER,"And the `e` key can be `null`or missing for ""never expires"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636426530,Ability to set ds_actor cookie such that it expires, https://github.com/simonw/datasette/issues/829#issuecomment-642175892,https://api.github.com/repos/simonw/datasette/issues/829,642175892,MDEyOklzc3VlQ29tbWVudDY0MjE3NTg5Mg==,9599,simonw,2020-06-10T18:13:26Z,2020-06-10T18:13:26Z,OWNER,"I'm going with `expires_at` - except to keep the cookies shorter the key will be called `e` and the actor will go in `a`, like this: ```json { ""e"": ""1UuHoo"", ""a"": {""id"": ""root""} } ``` That `e` value is a base64 encoded expiry integer timestamp (again for a shorter cookie) - using https://pypi.org/project/python-baseconv/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636426530,Ability to set ds_actor cookie such that it expires, https://github.com/simonw/datasette/issues/829#issuecomment-642174272,https://api.github.com/repos/simonw/datasette/issues/829,642174272,MDEyOklzc3VlQ29tbWVudDY0MjE3NDI3Mg==,9599,simonw,2020-06-10T18:10:13Z,2020-06-10T18:10:13Z,OWNER,"Some options: - Redesign the `ds_actor` cookie to be `{""expires_at"": 1591811250, ""actor"": ...}` - check if it has expired in that default `actor_from_request` hook - Let plugins set an additional cookie of some sort - Expect plugins that care about this to set a cookie with a different name and implement their own `actor_from_request` against that","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636426530,Ability to set ds_actor cookie such that it expires, https://github.com/simonw/datasette/issues/829#issuecomment-642161210,https://api.github.com/repos/simonw/datasette/issues/829,642161210,MDEyOklzc3VlQ29tbWVudDY0MjE2MTIxMA==,9599,simonw,2020-06-10T17:45:58Z,2020-06-10T17:45:58Z,OWNER,"`itsdangerous` has this ability but you specify the max-age when you call unsign: https://itsdangerous.palletsprojects.com/en/1.1.x/timed/ > s.unsign(string, max_age=5) > Traceback (most recent call last): > ... > itsdangerous.exc.SignatureExpired: Signature age 15 > 5 seconds I currently only decode the `ds_actor` cookie in one place: https://github.com/simonw/datasette/blob/d828abaddec0dce3ec4b4eeddc3a74384e52cf34/datasette/actor_auth_cookie.py#L5-L12 If plugins want to be able to set their own policies on how long the `ds_actor` cookie should remain valid, how do I know to listen to them when decoding the cookie here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636426530,Ability to set ds_actor cookie such that it expires, https://github.com/simonw/datasette/issues/394#issuecomment-641908346,https://api.github.com/repos/simonw/datasette/issues/394,641908346,MDEyOklzc3VlQ29tbWVudDY0MTkwODM0Ng==,127565,wragge,2020-06-10T10:22:54Z,2020-06-10T10:22:54Z,CONTRIBUTOR,"There's a working demo here: https://github.com/wragge/datasette-test And if you want something that's more than just proof-of-concept, here's a notebook which does some harvesting from web archives and then displays the results using Datasette: https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/explore_presentations.ipynb","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",396212021,base_url configuration setting, https://github.com/simonw/datasette/issues/394#issuecomment-641889565,https://api.github.com/repos/simonw/datasette/issues/394,641889565,MDEyOklzc3VlQ29tbWVudDY0MTg4OTU2NQ==,58298410,LVerneyPEReN,2020-06-10T09:49:34Z,2020-06-10T09:49:34Z,NONE,"Hi, I came across this issue while looking for a way to spawn Datasette as a SQLite files viewer in JupyterLab. I found https://github.com/simonw/jupyterserverproxy-datasette-demo which seems to be the most up to date proof of concept, but it seems to be failing to list the available db (at least in the Binder demo, https://hub.gke.mybinder.org/user/simonw-jupyters--datasette-demo-uw4dmlnn/datasette/, I only have `:memory`). Does anyone tried to improve on this proof of concept to have a Datasette visualization for SQLite files? Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",396212021,base_url configuration setting, https://github.com/simonw/datasette/issues/828#issuecomment-641713087,https://api.github.com/repos/simonw/datasette/issues/828,641713087,MDEyOklzc3VlQ29tbWVudDY0MTcxMzA4Nw==,9599,simonw,2020-06-10T04:28:17Z,2020-06-10T04:28:17Z,OWNER,"Fixed. https://datasette.readthedocs.io/en/latest/changelog.htmlRecent photos
{% for photo in sql(""select * from apple_photos order by date desc limit 100"") %}
![]()
{% endfor %}
```
`random-photos.html`
```html
Random photos
{% for photo in sql(""with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 100"") %}
![]()
{% endfor %}
```
Now run `datasette -m metadata.yaml photos.db --template-dir=templates/`
Visit http://127.0.0.1:8001/random-photos to see some random photos or http://127.0.0.1:8002/recent-photos for recent photos.
This is using this mechanism: https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633626741,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633626741,MDEyOklzc3VlQ29tbWVudDYzMzYyNjc0MQ==,9599,simonw,2020-05-25T15:38:55Z,2020-05-25T15:38:55Z,MEMBER,"Sure, I should absolutely document this!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/dogsheep/dogsheep-photos/issues/20#issuecomment-633234781,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20,633234781,MDEyOklzc3VlQ29tbWVudDYzMzIzNDc4MQ==,41439,dmd,2020-05-24T13:56:13Z,2020-05-24T13:56:13Z,NONE,"As that seems to be closed, can you give a hint on how to make this work?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613006393,Ability to serve thumbnailed Apple Photo from its place on disk,
https://github.com/simonw/datasette/issues/767#issuecomment-632555800,https://api.github.com/repos/simonw/datasette/issues/767,632555800,MDEyOklzc3VlQ29tbWVudDYzMjU1NTgwMA==,2657547,rixx,2020-05-22T08:00:23Z,2020-05-22T08:00:23Z,CONTRIBUTOR,That would be perfect!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",620969465,Allow to specify a URL fragment for canned queries,
https://github.com/simonw/datasette/issues/767#issuecomment-632398475,https://api.github.com/repos/simonw/datasette/issues/767,632398475,MDEyOklzc3VlQ29tbWVudDYzMjM5ODQ3NQ==,9599,simonw,2020-05-21T23:36:00Z,2020-05-21T23:36:00Z,OWNER,"Interesting. So this would take effect exclusively on pages that link to the queries, which I think just means this page: https://latest.datasette.io/fixtures
How about this for the design: there's a new optional key in the metadata configuration for a query called `""fragment""` - used like this:
```json
{
""databases"": {
""fixtures"": {
""queries"": {
""neighborhood_search"": {
""sql"": ""\nselect neighborhood, facet_cities.name, state\nfrom facetable\n join facet_cities\n on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%'\norder by neighborhood;\n"",
""title"": ""Search neighborhoods"",
""description_html"": ""Demonstrating simple like search"",
""name"": ""neighborhood_search"",
""fragment"": ""sql-format""
}
}
}
}
}
```
Because of the `""fragment"": ""sql-format""` bit there the link to that query from https://latest.datasette.io/fixtures would be to `https://latest.datasette.io/fixtures/neighborhood_search#sql-format`.
Does that cover what you need here? Have I missed anything?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",620969465,Allow to specify a URL fragment for canned queries,
https://github.com/dogsheep/dogsheep-photos/issues/25#issuecomment-631253852,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25,631253852,MDEyOklzc3VlQ29tbWVudDYzMTI1Mzg1Mg==,9599,simonw,2020-05-20T05:56:17Z,2020-05-21T22:26:16Z,MEMBER,"I have a `deploy-demo.sh` script now:
```bash
#!/bin/bash
if [ -f public.db ]; then
rm public.db
fi
pipenv run dogsheep-photos create-subset photos.db public.db \
""select sha256 from apple_photos where albums like '%Public%'""
pipenv run sqlite-utils create-view public.db photos_on_a_map \
""select
date,
latitude,
longitude,
apple_photos.sha256,
uploads.ext,
json_object(
'title',
'Taken on ' || date,
'image',
'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=400',
'link',
'https://photos.simonwillison.net/i/' || uploads.sha256 || '.' || uploads.ext || '?w=1200'
) as popup
from
apple_photos
join uploads on apple_photos.sha256 = uploads.sha256
where
latitude is not null
order by
date desc"" \
--replace
pipenv run datasette publish now public.db --project dogsheep-photos \
--about=dogsheep/dogsheep-photos \
--about_url=""https://github.com/dogsheep/dogsheep-photos"" \
--install=datasette-json-html \
--install=datasette-pretty-json \
--install=datasette-cluster-map>=0.10 \
--title ""Dogsheep Photos demo""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621332242,Create a public demo,
https://github.com/simonw/datasette/issues/744#issuecomment-632305868,https://api.github.com/repos/simonw/datasette/issues/744,632305868,MDEyOklzc3VlQ29tbWVudDYzMjMwNTg2OA==,30607,aborruso,2020-05-21T19:43:23Z,2020-05-21T19:43:23Z,NONE,"@simonw now I have
```
Traceback (most recent call last):
File ""/home/aborruso/.local/bin/datasette"", line 8, in TemporaryRedirect