id,node_id,number,title,user,user_label,state,locked,assignee,assignee_label,milestone,milestone_label,comments,created_at,updated_at,closed_at,author_association,pull_request,body,repo,repo_label,type,active_lock_reason,performed_via_github_app,reactions,draft,state_reason
795367402,MDU6SXNzdWU3OTUzNjc0MDI=,1209,v0.54 500 error from sql query in custom template; code worked in v0.53; found a workaround,11788561,jrdmb,open,0,,,,,1,2021-01-27T19:08:13Z,2021-01-28T23:00:27Z,,NONE,,"v0.54 500 error in sql query template; code worked in v0.53; found a workaround

**schema:**   
CREATE TABLE ""talks"" (""talk"" TEXT,""series"" INTEGER, ""talkdate"" TEXT)  
CREATE TABLE ""series"" (""id"" INTEGER PRIMARY KEY, ""series"" TEXT, talks_list TEXT default '', website TEXT default '');

**Live example of correctly rendered template in v.053:** https://cosmotalks-cy6xkkbezq-uw.a.run.app/cosmotalks/talks/1

**Description of problem:** I needed 'sql select' code in a custom row-mydatabase-mytable.html template to lookup the series name for a foreign key integer value in the talks table.  So `metadata.json` specifies the `datasette-template-sql` plugin.

The code below worked perfectly in v0.53 (just the relevant sql statement part is shown; full code is [here](https://github.com/jrdmb/cosmotalks-datasette/blob/main/templates/row-cosmotalks-talks.html)):

```
{# custom addition #}  
{% for row in display_rows %}  
    ...  
    {% set sname = sql(""select series from series where id = ?"", [row.series]) %}  
    <strong>Series name: {{ sname[0].series }}  
    ...  
{% endfor %}  
{# End of custom addition #}  
```

**In v0.54, that code resulted in a 500 error with a 'no such table series' message.**  A second query in that template also did not work but the above is fully illustrative of the problem.

All templates were up-to-date along with datasette v0.54.

**Workaround:** After fiddling around with trying different things, what worked was the syntax from [Querying a different database from the datasette-template-sql github repo](https://github.com/simonw/datasette-template-sql#querying-a-different-database) to add the database name to the sql statement:

`{% set sname = sql(""select series from series where id = ?"", [row.series], database=""cosmotalks"") %}`

Though this was found to work, it should not be necessary to add `database=""cosmotalks""` since per the `datasette-template-sql` README, it's only needed when querying a different database, but here it's a table within the same database.
",107914493,datasette,issue,,,,,
811054000,MDU6SXNzdWU4MTEwNTQwMDA=,1230,"Vega charts are plotted only for rows on the visible page, cluster maps only for rows in the remaining pages",7107523,Kabouik,open,0,,,,,1,2021-02-18T12:27:02Z,2021-02-18T15:22:15Z,,NONE,,"I filtered a data set on some criteria and obtain 265 results, split over three pages (100, 100, 65), and reazlized that Vega plots are only applied to the results displayed on the current page, instead of the whole filtered data, _e.g._, 100 on page 1, 100 on page 2, 65 on page 3. Is there a way to force the graphs to consider all results instead of just the page, considering that pages rarely represent sensible information?

Likewise, while the cluster map does show all results on the first page, if you go to next pages, it will show all remaining results except the previous page(s), _e.g._, 265 on page 1, 165 on page 2, 65 on page 3.

In both cases, I don't see many situations where one would like to represent the data this way, and it might even lead to interpretation errors when viewing the data. Am I missing some cases where this would be best? Perhaps a clickable option to subset visual representations according visible pages _vs._ display all search results would do?

[Edit] Oh, I just saw the ""Load all"" button under the cluster map as well as the [setting to alter the max number or results](https://docs.datasette.io/en/stable/settings.html#max-returned-rows). So I guess this issue only is about the Vega charts.",107914493,datasette,issue,,,,,
1616347574,I_kwDOJHON9s5gV4G2,1,Initial proof of concept with ChatGPT,9599,simonw,closed,0,,,,,3,2023-03-09T03:44:39Z,2023-03-09T03:51:55Z,2023-03-09T03:51:55Z,MEMBER,,I'm using ChatGPT to figure out enough AppleScript to get at my notes data.,611552758,apple-notes-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1617962395,I_kwDOJHON9s5gcCWb,10,Include schema in README,9599,simonw,closed,0,,,,,0,2023-03-09T20:38:59Z,2023-03-09T20:48:18Z,2023-03-09T20:48:18Z,MEMBER,,As seen in other tools like https://github.com/simonw/git-history,611552758,apple-notes-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/10/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1618130434,I_kwDOJHON9s5gcrYC,11,Implement a SQL view to make it easier to query files in a nested folder,9599,simonw,open,0,,,,,3,2023-03-09T23:19:28Z,2023-03-09T23:24:01Z,,MEMBER,,"Working with nested data in SQL is tricky, can I make it easier with a view or canned query?",611552758,apple-notes-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1650981564,I_kwDOJHON9s5iZ_q8,12,Error running pytest,14314871,amlestin,open,0,,,,,0,2023-04-02T15:02:36Z,2023-04-02T15:07:10Z,,NONE,,"`______________________________________________________ ERROR collecting tests/test_apple_notes_to_sqlite.py _______________________________________________________
ImportError while importing test module '/Users/lol/development/apple-notes-to-sqlite/tests/test_apple_notes_to_sqlite.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/homebrew/Cellar/python@3.9/3.9.16/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_apple_notes_to_sqlite.py:2: in <module>
    from apple_notes_to_sqlite.cli import cli, COUNT_SCRIPT, FOLDERS_SCRIPT
E   ModuleNotFoundError: No module named 'apple_notes_to_sqlite'`

Solution:
This is likely a PYTHONPATH issue due to having pytest installed both globally and in the venv. We can guarantee the tests run by adding the current directory to sys.path automatically using

`python -m pytest`

The alternative is to activate the venv, install pytest, deactivate, then activate the venv again (https://stackoverflow.com/questions/35045038/how-do-i-use-pytest-with-virtualenv)",611552758,apple-notes-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/12/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1650984552,PR_kwDOJHON9s5NbyYN,13,use universal command,14314871,amlestin,open,0,,,,,0,2023-04-02T15:10:54Z,2023-04-02T15:37:34Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/apple-notes-to-sqlite/pulls/13,,611552758,apple-notes-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/13/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1880968405,PR_kwDOJHON9s5ZhYny,14,fix: fix the problem of Chinese character garbling,2698003,barretlee,open,0,,,,,0,2023-09-04T23:48:28Z,2023-09-04T23:48:28Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/apple-notes-to-sqlite/pulls/14,"1. The code uses two different ways of writing encoding formats, `mac_roman` and `macroman`. It is uncertain whether there are any typo errors.
2. When there are Chinese characters in the content, exporting it results in garbled code. Changing it to `utf8` can fix the issue.",611552758,apple-notes-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/14/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1616354999,I_kwDOJHON9s5gV563,2,First working version,9599,simonw,closed,0,,,,,7,2023-03-09T03:53:00Z,2023-03-09T05:10:22Z,2023-03-09T05:10:22Z,MEMBER,,"It's going to shell out to `osascript` as seen in:
- #1

I'm going with that option because https://appscript.sourceforge.io/status.html warns against the other potential methods:

> Apple eliminated its Mac Automation department in 2016. The future of AppleScript and its related technologies is unclear. Caveat emptor.

But `osascript` looks pretty stable to me.",611552758,apple-notes-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1616422013,I_kwDOJHON9s5gWKR9,3,`apple-notes-to-sqlite --dump` option,9599,simonw,closed,0,,,,,0,2023-03-09T05:05:49Z,2023-03-09T05:06:14Z,2023-03-09T05:06:14Z,MEMBER,,"Option that doesn't write to the database at all, it just outputs all the notes to stdout as newline-delimited JSON.",611552758,apple-notes-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/3/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1616429236,I_kwDOJHON9s5gWMC0,4,Support incremental updates,9599,simonw,open,0,,,,,2,2023-03-09T05:14:00Z,2023-03-09T18:20:56Z,,MEMBER,,"Running this script can take several hours against a large notes database.

Would be neat if it could run against just the notes that have been modified since it last ran. Could pull the max `updated` date and then keep on looping until it finds one modified before then.

Problem is I don't actually know what order it iterates over the notes in.",611552758,apple-notes-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1616440856,I_kwDOJHON9s5gWO4Y,5,Configure full text search,9599,simonw,open,0,,,,,0,2023-03-09T05:20:46Z,2023-03-09T05:20:46Z,,MEMBER,,"FTS would be useful.

Maybe even extract the plain text from the notes to make that index easier to create, rather than creating it against the HTML. Can use the `plaintext` property for that.",611552758,apple-notes-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1617602868,I_kwDOJHON9s5gaqk0,6,Character encoding problem,9599,simonw,open,0,,,,,2,2023-03-09T16:44:34Z,2023-04-14T15:22:09Z,,MEMBER,,"I ran against a recent note with this in it:

> Or just ""Actions ⚙️ ""

And got back:

<img width=""367"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/224092586-e2475d8f-868c-489d-b495-2bfeb1a7db99.png"">

> `Actions ⚙️`

Pasting that into https://ftfy.vercel.app/?s=Actions+%E2%80%9A%C3%B6%C3%B4%C3%94%E2%88%8F%C3%A8+ gives this:

```python
s = 'Actions â\x80\x9aöôÃ\x94â\x88\x8fè'
s = s.encode('latin-1')
s = s.decode('utf-8')
s = s.encode('macroman')
s = s.decode('utf-8')
print(s)
```
<img width=""816"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/224092864-2b796c40-7c68-40b1-ae38-d0c61721b742.png"">
",611552758,apple-notes-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1617769847,I_kwDOJHON9s5gbTV3,7,Folder support,9599,simonw,closed,0,,,,,6,2023-03-09T18:21:33Z,2023-03-09T20:48:18Z,2023-03-09T20:48:18Z,MEMBER,,Notes can live in folders. These relationships should be exported too.,611552758,apple-notes-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1617823309,I_kwDOJHON9s5gbgZN,8,Increase performance using macnotesapp,41546558,RhetTbull,closed,0,,,,,1,2023-03-09T18:51:05Z,2023-03-14T22:00:22Z,2023-03-14T22:00:21Z,NONE,,"Neat project!  You can probably increase performance using my python interface to Notes, [macnotesapp](https://github.com/RhetTbull/macnotesapp), which uses Scripting Bridge and bulk queries for much better performance than AppleScript.

Another related project is [PyXA](https://github.com/SKaplanOfficial/PyXA) which uses Scripting Bridge to access Notes (and many other apps) and can return all the notes at once as opposed to calling AppleScript for each note.  

macnotesapp allows you to access multiple accounts and folders as well.

```python
from macnotesapp import NotesApp

# NotesApp() provides interface to Notes.app
notesapp = NotesApp()

# Get list of notes (Note objects for each note)
notes = notesapp.notes()
note = notes[0]
print(
    note.id,
    note.account,
    note.folder,
    note.name,
    note.body,
    note.plaintext,
    note.password_protected,
)

print(note.asdict())

```",611552758,apple-notes-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1617938730,I_kwDOJHON9s5gb8kq,9,"Default to just storing plaintext, store HTML if `--html` is passed",9599,simonw,open,0,,,,,0,2023-03-09T20:19:06Z,2023-03-09T20:19:06Z,,MEMBER,,"The full `body` version of the notes can get HUGE, due to embedded images. It turns out for my own purposes I'm usually happy with just the `plaintext` version.

I'm tempted to say you don't get HTML unless you pass a `--html` option.",611552758,apple-notes-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/9/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
689800307,MDU6SXNzdWU2ODk4MDAzMDc=,1,Add an index on the timestamp column,9599,simonw,closed,0,,,,,0,2020-09-01T04:33:37Z,2020-09-01T04:49:23Z,2020-09-01T04:49:23Z,MEMBER,,Since default view will likely be ordered by timestamp descending.,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
691557547,MDU6SXNzdWU2OTE1NTc1NDc=,10,Category 3: received,9599,simonw,closed,0,,,,,1,2020-09-03T01:40:36Z,2020-09-03T17:38:51Z,2020-09-03T17:38:51Z,MEMBER,,"A category for things that were sent to me: DMs, emails etc. Follows #7.",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/10/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
692125110,MDU6SXNzdWU2OTIxMjUxMTA=,11,Public / Private mechanism,9599,simonw,closed,0,,,,,1,2020-09-03T16:47:03Z,2020-09-03T17:33:52Z,2020-09-03T17:33:52Z,MEMBER,,"Some of the data in Dogsheep is stuff that was written publicly - tweets, blog posts, GitHub commits to public repos.

Some of it is private data - emails, photos, direct messages, Swarm checkins, commits to private repos.

Being able to filter for just one or the other (or both) would be useful. Especially when giving demos!",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/11/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
692202408,MDU6SXNzdWU2OTIyMDI0MDg=,12,Idea: maps and GeoJSON support,9599,simonw,open,0,,,,,0,2020-09-03T18:47:10Z,2020-09-04T01:45:03Z,,MEMBER,,"It would be cool if the `display_sql` could return a column populated with GeoJSON which would the automatically be displayed on a map in the results (or maybe default JS would look for a `class=""geojson""` element output by the `display` template) - ala https://github.com/simonw/datasette-leaflet-geojson

Then I could render workout routes on a map, or Swarm checkin points.",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/12/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
692386625,MDU6SXNzdWU2OTIzODY2MjU=,13,Support advanced FTS queries,9599,simonw,closed,0,,,,,1,2020-09-03T21:29:56Z,2020-09-03T21:40:51Z,2020-09-03T21:40:51Z,MEMBER,,`simon willison NOT screenshot` for example.,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/13/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
693318095,MDU6SXNzdWU2OTMzMTgwOTU=,14,On FTS exception rerun the query with quoting,9599,simonw,closed,0,,,,,0,2020-09-04T15:44:18Z,2020-09-05T16:23:01Z,2020-09-05T16:23:01Z,MEMBER,,"Searching for eg `#dogfest` currently throws an FTS exception - but I want to support advanced FTS query tricks as seen in #13.

https://dogsheep.simonwillison.net/-/beta?q=%23dogfest

> fts5: syntax error near ""#""

Idea: catch that error and re-run the query with FTS escaping applied!
",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/14/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
694136490,MDU6SXNzdWU2OTQxMzY0OTA=,15,Add a bunch of config examples,9599,simonw,open,0,,,,,1,2020-09-05T17:58:43Z,2020-09-18T23:17:39Z,,MEMBER,,I can bring these over from my personal Dogsheep.,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/15/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
694493566,MDU6SXNzdWU2OTQ0OTM1NjY=,16,Timeline view,9599,simonw,open,0,,,,,3,2020-09-06T19:13:58Z,2020-09-21T02:42:29Z,,MEMBER,,Ability to browse (and facet) by date.,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
694500679,MDU6SXNzdWU2OTQ1MDA2Nzk=,17,"Rename ""table"" to ""type""",9599,simonw,closed,0,,,,,2,2020-09-06T19:34:41Z,2020-09-09T03:03:22Z,2020-09-09T03:03:22Z,MEMBER,,"I think ""table"" is the wrong name for the concept I'm using it for here.

Two reasons: firstly, `table` is a reserved word in SQLite. More importantly, it turns out there's not a direct mapping from tables to types of search result. In particular, for GitHub I ended up having two different ""tables"" of repositories - one for repos created by me, another for repos that I have starred.",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
695553522,MDU6SXNzdWU2OTU1NTM1MjI=,18,Deleted records stay in the search index,9599,simonw,open,0,,,,,2,2020-09-08T05:14:23Z,2020-09-08T05:15:51Z,,MEMBER,,"Here's why: https://github.com/dogsheep/dogsheep-beta/blob/24f7898d41a39218058f174c75ba62f7c0fcfff6/dogsheep_beta/utils.py#L44-L53

That should probably do `DELETE FROM index1.search_index WHERE [table] = ?` first.",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
695556681,MDU6SXNzdWU2OTU1NTY2ODE=,19,Figure out incremental re-indexing,9599,simonw,open,0,,,,,2,2020-09-08T05:23:31Z,2020-09-08T05:27:07Z,,MEMBER,,As tables get bigger reindexing everything on a schedule (essentially recreating the entire index from scratch) will start to become a performance bottleneck.,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
689809225,MDU6SXNzdWU2ODk4MDkyMjU=,2,Apply porter stemming,9599,simonw,closed,0,,,,,2,2020-09-01T04:57:55Z,2020-09-01T20:42:00Z,2020-09-01T20:40:24Z,MEMBER,,This can be on by default. You can turn it off for a table in the config file using `stemming: none` - or maybe `tokenize: none` to match the terminology used by SQLite and `sqlite-utils`: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#enabling-full-text-search,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
697162939,MDU6SXNzdWU2OTcxNjI5Mzk=,20,Add more tags so people can find your project.,7902810,ran88dom99,open,0,,,,,0,2020-09-09T21:14:09Z,2020-09-09T21:14:09Z,,NONE,,"quantified-self habit-tracking google-fit time-tracking wearables quantifiedself 
for example",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/20/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 1, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
703951918,MDU6SXNzdWU3MDM5NTE5MTg=,21,Option to sort search results by date,9599,simonw,closed,0,,,,,0,2020-09-17T22:32:39Z,2020-09-17T22:55:35Z,2020-09-17T22:55:35Z,MEMBER,,"Sometimes I want to sort by date, not by relevance.",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/21/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
703962917,MDU6SXNzdWU3MDM5NjI5MTc=,22,Bug: UI says sorted by relevance in timeline view,9599,simonw,closed,0,,,,,0,2020-09-17T23:02:07Z,2020-09-17T23:13:14Z,2020-09-17T23:13:14Z,MEMBER,,"In regular timeline view sort defaults to newest, not relevance - so this UI is incorrect:

<img width=""837"" alt=""Dogsheep_Beta_and_The_latest_news_from_the_Datasette_ecosystem_of_tools_-_Datasette_Newsletter_and_Ability_to_invite_people_to_a_team_who_don_t_have_acounts_yet_·_Issue__307_·_simonw_datasettecloud"" src=""https://user-images.githubusercontent.com/9599/93536956-1facf900-f8ff-11ea-889b-bc8356e366df.png"">
",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/22/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
703970713,MDU6SXNzdWU3MDM5NzA3MTM=,23,Sort option should persist between multiple searches,9599,simonw,closed,0,,,,,0,2020-09-17T23:21:26Z,2020-09-18T22:39:12Z,2020-09-18T22:39:12Z,MEMBER,,Following #21 ,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/23/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
703970814,MDU6SXNzdWU3MDM5NzA4MTQ=,24,"the JSON object must be str, bytes or bytearray, not 'Undefined'",9599,simonw,closed,0,,,,,8,2020-09-17T23:21:41Z,2020-09-18T22:33:32Z,2020-09-18T22:33:32Z,MEMBER,,Got this on a search results page.,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
704685890,MDU6SXNzdWU3MDQ2ODU4OTA=,25,template_debug mechanism,9599,simonw,closed,0,,,,,2,2020-09-18T22:11:09Z,2020-09-18T22:12:21Z,2020-09-18T22:12:03Z,MEMBER,,"> I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible.
_Originally posted by @simonw in https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584_",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
705215230,MDU6SXNzdWU3MDUyMTUyMzA=,26,Pagination,9599,simonw,open,0,,,,,7,2020-09-21T00:14:37Z,2020-09-21T02:55:54Z,,MEMBER,,Useful for #16 (timeline view) since you can now filter to just the items on a specific day - but if there are more than 50 items you can't see them all.,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
709789634,MDU6SXNzdWU3MDk3ODk2MzQ=,27,Sort order is not persisted by facet filter links,9599,simonw,open,0,,,,,0,2020-09-27T18:22:07Z,2020-09-27T18:22:07Z,,MEMBER,,A link to `/-/beta?category=1&timestamp__date=2018-08-01&q=swedish` should be to `/-/beta?category=1&timestamp__date=2018-08-01&q=swedish&sort=newest`,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/27/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
723861683,MDU6SXNzdWU3MjM4NjE2ODM=,28,Switch to using datasette.client,9599,simonw,closed,0,,,,,1,2020-10-17T22:42:26Z,2020-10-17T23:00:47Z,2020-10-17T23:00:47Z,MEMBER,,"`datasette.client` is designed for this kind of thing, to replace this code: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L223-L232",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/28/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
724759588,MDU6SXNzdWU3MjQ3NTk1ODg=,29,Add search highlighting snippets,9599,simonw,open,0,,,,,5,2020-10-19T16:00:48Z,2021-08-26T20:23:11Z,,MEMBER,,Like on https://til.simonwillison.net/til/search?q=Snippet,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,
689810340,MDU6SXNzdWU2ODk4MTAzNDA=,3,"Datasette plugin to provide custom page for running faceted, ranked searches",9599,simonw,closed,0,,,,,3,2020-09-01T05:00:22Z,2020-09-03T21:01:41Z,2020-09-03T21:01:41Z,MEMBER,,"This will be a page at `/-/beta` which renders using a custom template.

It will offer a default timeline view plus search and facet by type/date.",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
769282206,MDU6SXNzdWU3NjkyODIyMDY=,30,Upgrade to sqlite-utils 3.0 (tests are failing),9599,simonw,closed,0,,,,,0,2020-12-16T21:25:15Z,2020-12-16T21:27:11Z,2020-12-16T21:27:10Z,MEMBER,,"```
        results = beta_db[""search_index""].search(""run"")
        if use_porter:
            assert results == [
                (
                    ""dogs.db/dogs"",
                    ""1"",
                    ""Cleo"",
                    ""2020-08-22 04:41:33"",
                    1,
                    0,
                    ""running"",
                    None,
                    None,
                )
            ]
        else:
>           assert results == []
E           assert <generator ob...x7f3c080a3e08> == []
E             +<generator object Table.search at 0x7f3c080a3e08>
E             -[]
E             Full diff:
E             - []
E             + <generator object Table.search at 0x7f3c080a3e08>
```
This was caused by a backwards incompatible change in sqlite-utils 3.0: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v3-0",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/30/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
771316301,MDU6SXNzdWU3NzEzMTYzMDE=,31,"Searching for ""github-to-sqlite"" throws an error",9599,simonw,closed,0,,,,,4,2020-12-19T06:07:20Z,2020-12-19T06:18:07Z,2020-12-19T06:18:07Z,MEMBER,,"https://datasette.io/-/beta?q=github-to-sqlite&sort=relevance&type=blog.db%2Fentries - ""no such column: to""",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
836923194,MDU6SXNzdWU4MzY5MjMxOTQ=,32,JSON API for search results,9599,simonw,open,0,,,,,0,2021-03-20T22:21:36Z,2021-03-20T22:21:36Z,,MEMBER,,Refs https://github.com/simonw/datasette/issues/878,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/32/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
919733213,MDU6SXNzdWU5MTk3MzMyMTM=,33,Searching for whitespace throws an error,9599,simonw,closed,0,,,,,0,2021-06-13T06:57:57Z,2021-06-13T14:36:39Z,2021-06-13T14:36:39Z,MEMBER,,"https://datasette.io/-/beta?q=+ returns a 500

> fts5: syntax error near """"",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/33/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
983221851,MDU6SXNzdWU5ODMyMjE4NTE=,34,Data folder as index command parameter,1223625,humrochagf,open,0,,,,,0,2021-08-30T21:29:33Z,2021-08-30T21:29:33Z,,NONE,,"Hi,

First of all, thank you for this wonderful project :smile:

I started to use dogsheep to make my personal data searchable, and by using the project I noticed an issue with the index command.

It always expects you are running it from the root folder from where the data is located, so I got some errors while trying to make it work on my setup.

I separate all databases inside a `data` folder (I published my setup to be easier to follow: https://github.com/humrochagf/my-dogsheep)

Before, I configured `dogsheep.yml` to add the data folder to its path like this:

```yml
data/twitter.db:
    tweets:
        sql: |-
...
```

And running the index command like this:

```
dogsheep-beta index data/dogsheep.db dogsheep.yml
```

It worked to the normal search feature with no problem this way, but when I started adding `display_sql` rules the app started to crash, because at datasette `get_database` it was looking for `data/twitter` and it only had a db called `twitter` there.

So my workaround to that was to cd into the data folder and run the indexer. You can check the way I'm doing it at this line of the makefile: https://github.com/humrochagf/my-dogsheep/blob/main/makefile#L3

It works but it would be nice to have an option to pass the path where the data is located to the index function.",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/34/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
987985935,MDExOlB1bGxSZXF1ZXN0NzI2OTkwNjgw,35,Support for Datasette's --base-url setting,2670795,brandonrobertz,open,0,,,,,0,2021-09-03T17:47:45Z,2021-09-03T17:47:45Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/dogsheep-beta/pulls/35,This makes it so you can use Dogsheep if you're using Datasette with the `--base-url /some-path/` setting.,197431109,dogsheep-beta,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/35/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1751214236,I_kwDOC8SPRc5oYWic,36,Getting sqlite_master may not be modified when creating dogsheep index,8711912,khushmeeet,open,0,,,,,0,2023-06-11T03:21:53Z,2023-06-11T03:21:53Z,,NONE,,"When creating a `dogsheep` index from `config.yml` file on pocket.db (created using pocket-to-sqlite), I am getting this error

```
Traceback (most recent call last):
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/bin/dogsheep-beta"", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py"", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py"", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py"", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py"", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/cli.py"", line 36, in index
    run_indexer(
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py"", line 32, in run_indexer
    ensure_table_and_indexes(db, tokenize)
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py"", line 91, in ensure_table_and_indexes
    table.add_foreign_key(*fk)
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py"", line 2155, in add_foreign_key
    self.db.add_foreign_keys([(self.name, column, other_table, other_column)])
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py"", line 1116, in add_foreign_keys
    cursor.execute(
sqlite3.OperationalError: table sqlite_master may not be modified
```

Command I ran to get this error
```
dogsheep-beta index pocket.db config.yml
```

Dogsheep version
```
dogsheep-beta, version 0.10.2
```

Python version 
```
Python 3.11.2
```",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/36/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1817281557,I_kwDOC8SPRc5sUYQV,37,cannot use jinja filters in display?,10352819,rprimet,closed,0,,,,,1,2023-07-23T20:09:54Z,2023-07-23T20:18:27Z,2023-07-23T20:18:26Z,NONE,,"Hi, I'm trying to have a display function in Dogsheep's `config.yml` that includes something like this:

```         
<h3>
  <a href=""{{ urls.row('my_database', 'my_table', key) }}"">{{ display.title }}</a>
  <a href=""{{ display.url }}🔗"" target=""_blank"">(source)</a>
</h3>
<p>{{ display.snippet|safe }}</p>
```

Unfortunately, rendering fails with a message 'urls is undefined'.

The same happens if I'm trying to build a row URL manually, using filters like `quote_plus` (as my keys are URLs).

Any hints?

Thanks!",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/37/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1888477283,I_kwDOC8SPRc5wj-Bj,38,Run `rebuild_fts` after building the index,9599,simonw,open,0,,,,,0,2023-09-08T23:17:45Z,2023-09-08T23:17:45Z,,MEMBER,,"In:
- https://github.com/simonw/datasette.io/issues/152#issuecomment-1712323347

This turned out to be the fix:

```bash
dogsheep-beta index dogsheep-index.db templates/dogsheep-beta.yml
sqlite-utils rebuild-fts dogsheep-index.db
```",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/38/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
689839399,MDU6SXNzdWU2ODk4MzkzOTk=,4,Optimize the FTS table,9599,simonw,closed,0,,,,,1,2020-09-01T05:58:17Z,2020-09-01T06:10:08Z,2020-09-01T06:10:08Z,MEMBER,,,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
689847361,MDU6SXNzdWU2ODk4NDczNjE=,5,Add a context column that's not searchable,9599,simonw,closed,0,,,,,1,2020-09-01T06:13:42Z,2020-09-03T18:43:50Z,2020-09-03T18:43:50Z,MEMBER,,"I sometimes like to configure titles that are things like ""Comment on issue X"" or ""Photo in Golden Gate Park"" - these shouldn't be included in the search index but should be stored so they can be displayed to provide context.

Add a column for this - probably called `context` - and make it so it can be populated.",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
689850810,MDU6SXNzdWU2ODk4NTA4MTA=,6,Set up a demo instance,9599,simonw,open,0,,,,,0,2020-09-01T06:20:24Z,2020-09-01T06:20:24Z,,MEMBER,,"Once I've got the Datasette plugin to a state where it's worth building a demo: #3

I can use data from my public https://github-to-sqlite.dogsheep.net/ demo plus the Pocket data subset I use for the demo in https://github.com/dogsheep/pocket-to-sqlite/issues/5 - I could pull in the https://dogsheep-photos.dogsheep.net/ photos data too.",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
691265198,MDU6SXNzdWU2OTEyNjUxOTg=,7,"Mechanism for differentiating between ""by me"" and ""liked by me""",9599,simonw,closed,0,,,,,6,2020-09-02T17:44:37Z,2020-09-02T21:06:28Z,2020-09-02T21:06:28Z,MEMBER,,"Some of the content I'm indexing is by me - photos I've taken, tweets I wrote, commits, comments I posted.

Some of it is stuff that I've ""liked"" or ""bookmarked"" in some way - favourited tweets, Pocket articles, starred GitHub repos.

It woud be useful to be able to differentiate between the two.",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
691369691,MDU6SXNzdWU2OTEzNjk2OTE=,8,Create a view for running faceted searches,9599,simonw,closed,0,,,,,1,2020-09-02T19:44:07Z,2020-09-02T19:50:47Z,2020-09-02T19:50:47Z,MEMBER,,"```sql
select
  search_index_fts.rank,
  search_index.rowid,
  search_index.[table],
  search_index.key,
  search_index.title,
  search_index.timestamp,
  search_index.search_1
from
  search_index join search_index_fts on search_index.rowid = search_index_fts.rowid
order by
  search_index_fts.rank, search_index.timestamp desc
```",197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
691521965,MDU6SXNzdWU2OTE1MjE5NjU=,9,Mechanism for defining custom display of results,9599,simonw,closed,0,,,,,8,2020-09-03T00:14:07Z,2020-09-03T21:12:14Z,2020-09-03T21:09:55Z,MEMBER,,Part of #3 - in particular I want to make sure my photos are displayed with a thumbnail.,197431109,dogsheep-beta,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
602533300,MDU6SXNzdWU2MDI1MzMzMDA=,1,Import photo metadata from Apple Photos into SQLite,9599,simonw,open,0,,,5324096,Apple Photos online and securely browsable,8,2020-04-18T19:23:26Z,2020-05-04T02:41:40Z,,MEMBER,,"Faces, albums, locations, that kind of thing.",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
606028272,MDU6SXNzdWU2MDYwMjgyNzI=,10,Speed up hashing step using threads,9599,simonw,closed,0,,,,,0,2020-04-24T04:20:08Z,2020-04-24T04:32:35Z,2020-04-24T04:32:35Z,MEMBER,,"This TODO from the code:

https://github.com/dogsheep/photos-to-sqlite/blob/2e7f2c67cc18b02c75bb64992a05b0196e507252/photos_to_sqlite/cli.py#L82-L90",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/10/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
606032950,MDU6SXNzdWU2MDYwMzI5NTA=,11,Try running S3 uploads in a thread pool,9599,simonw,closed,0,,,,,0,2020-04-24T04:34:31Z,2020-04-24T16:45:41Z,2020-04-24T16:45:41Z,MEMBER,,"Since #10 provided such a speedup, can the same thing be done for the actual uploads?

http://ls.pwd.io/2013/06/parallel-s3-uploads-using-boto-and-threads-in-python/ suggests it can really help performance.",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/11/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
606033104,MDU6SXNzdWU2MDYwMzMxMDQ=,12,"If less than 500MB, show size in MB not GB",9599,simonw,open,0,,,,,1,2020-04-24T04:35:01Z,2020-04-24T04:35:25Z,,MEMBER,,"Just saw this:
```
Uploading 0.05 GB
```",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/12/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
607888367,MDU6SXNzdWU2MDc4ODgzNjc=,13,Also upload movie files,9599,simonw,open,0,,,,,2,2020-04-27T22:11:25Z,2020-04-28T00:39:45Z,,MEMBER,,"The `upload` command currently only handles static images:

https://github.com/dogsheep/photos-to-sqlite/blob/d939455af00e07866686457ee2fcb9b2d1b7194e/photos_to_sqlite/utils.py#L26-L33

Need to cover movies taken by my phone and DSLR too.",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/13/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
608512747,MDU6SXNzdWU2MDg1MTI3NDc=,14,Annotate photos using the Google Cloud Vision API,9599,simonw,open,0,,,,,5,2020-04-28T18:09:03Z,2020-04-28T18:19:06Z,,MEMBER,,"It can detect faces, run OCR, do image labeling (it knows what a lemur is!) and do object localization where it identifies objects and returns bounding polygons for them.",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/14/reactions"", ""total_count"": 3, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,
612151767,MDU6SXNzdWU2MTIxNTE3Njc=,15,Expose scores from ZCOMPUTEDASSETATTRIBUTES,9599,simonw,closed,0,,,,,7,2020-05-04T20:36:07Z,2020-12-20T04:44:22Z,2020-05-05T00:11:45Z,MEMBER,,"The Apple Photos database has a `ZCOMPUTEDASSETATTRIBUTES` that looks absurdly interesting... it has calculated scores for every photo:

<img width=""273"" alt=""Photos__ZCOMPUTEDASSETATTRIBUTES"" src=""https://user-images.githubusercontent.com/9599/81011044-1f0e3d00-8e0c-11ea-84b6-f302b09e7bc9.png"">
",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
612287234,MDU6SXNzdWU2MTIyODcyMzQ=,16,"Import machine-learning detected labels (dog, llama etc) from Apple Photos",9599,simonw,open,0,,,,,13,2020-05-05T02:45:43Z,2020-05-05T05:38:16Z,,MEMBER,,"Follow-on from #1. Apple Photos runs some very sophisticated machine learning on-device to figure out if photos are of dogs, llamas and so on. I really want to extract those labels out into my own database.",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/16/reactions"", ""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
612860531,MDU6SXNzdWU2MTI4NjA1MzE=,17,Only install osxphotos if running on macOS,9599,simonw,closed,0,,,,,3,2020-05-05T20:03:26Z,2020-05-05T20:20:05Z,2020-05-05T20:11:23Z,MEMBER,,The build is broken right now because you can't `pip install osxphotos` on Ubuntu.,256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/17/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
612860758,MDU6SXNzdWU2MTI4NjA3NTg=,18,Switch CI solution to GitHub Actions with a macOS runner,9599,simonw,open,0,,,,,1,2020-05-05T20:03:50Z,2020-05-05T23:49:18Z,,MEMBER,,Refs #17.,256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/18/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
613002220,MDU6SXNzdWU2MTMwMDIyMjA=,19,apple-photos command should work even if upload has not run,9599,simonw,closed,0,,,,,1,2020-05-06T02:02:25Z,2020-05-19T20:59:59Z,2020-05-19T20:59:59Z,MEMBER,,"I want people to be able to query their Apple Photos metadata without having to first run `upload` to upload all of their files to their own S3 bucket.

To do this I can have `apple-photos` calculate SHA256 hashes of each photo if the `uploads` table does not yet exist (or does not contain that photo).",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/19/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
602533352,MDU6SXNzdWU2MDI1MzMzNTI=,2,Ability to convert HEIC images to JPEG,9599,simonw,closed,0,,,5324096,Apple Photos online and securely browsable,1,2020-04-18T19:23:43Z,2020-04-28T16:47:21Z,2020-04-28T16:47:21Z,MEMBER,,,256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
613006393,MDU6SXNzdWU2MTMwMDYzOTM=,20,Ability to serve thumbnailed Apple Photo from its place on disk,9599,simonw,closed,0,,,,,10,2020-05-06T02:17:50Z,2020-05-25T20:14:22Z,2020-05-25T20:09:41Z,MEMBER,,"A custom Datasette plugin that can be run locally on a Mac laptop which knows how to serve photos such that they can be seen in the browser.

_Originally posted by @simonw in https://github.com/dogsheep/photos-to-sqlite/issues/19#issuecomment-624406285_",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/20/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
615474990,MDU6SXNzdWU2MTU0NzQ5OTA=,21,bpylist.archiver.CircularReference: archive has a cycle with uid(13),9599,simonw,closed,0,,,,,11,2020-05-10T20:58:06Z,2020-12-19T07:44:49Z,2020-05-10T21:57:13Z,MEMBER,,"```
% python -i $(which photos-to-sqlite) apple-photos photos.db                     
Traceback (most recent call last):
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py"", line 611, in place
    return self._place  # pylint: disable=access-member-before-definition
AttributeError: 'PhotoInfo' object has no attribute '_place'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/bin/photos-to-sqlite"", line 11, in <module>
    load_entry_point('photos-to-sqlite', 'console_scripts', 'photos-to-sqlite')()
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/cli.py"", line 249, in apple_photos
    photo_row = osxphoto_to_row(sha256, photo)
  File ""/Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/utils.py"", line 91, in osxphoto_to_row
    place = photo.place
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py"", line 614, in place
    self._place = PlaceInfo5(self._info[""reverse_geolocation""])
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/placeinfo.py"", line 505, in __init__
    self._plrevgeoloc = archiver.unarchive(revgeoloc_bplist)
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 16, in unarchive
    return Unarchive(plist).top_object()
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 256, in top_object
    return self.decode_object(self.top_uid)
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 247, in decode_object
    obj = klass.decode_archive(ArchivedObject(raw_obj, self))
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/placeinfo.py"", line 126, in decode_archive
    mapItem = archive.decode(""mapItem"")
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 140, in decode
    return self._unarchiver.decode_key(self._object, key)
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 216, in decode_key
    return self.decode_object(val)
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 247, in decode_object
    obj = klass.decode_archive(ArchivedObject(raw_obj, self))
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/placeinfo.py"", line 180, in decode_archive
    sortedPlaceInfos = archive.decode(""sortedPlaceInfos"")
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 140, in decode
    return self._unarchiver.decode_key(self._object, key)
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 216, in decode_key
    return self.decode_object(val)
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 247, in decode_object
    obj = klass.decode_archive(ArchivedObject(raw_obj, self))
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 112, in decode_archive
    return [archive._decode_index(index) for index in uids]
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 112, in <listcomp>
    return [archive._decode_index(index) for index in uids]
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 137, in _decode_index
    return self._unarchiver.decode_object(index)
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 247, in decode_object
    obj = klass.decode_archive(ArchivedObject(raw_obj, self))
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/placeinfo.py"", line 217, in decode_archive
    placeType = archive.decode(""placeType"")
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 140, in decode
    return self._unarchiver.decode_key(self._object, key)
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 216, in decode_key
    return self.decode_object(val)
  File ""/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py"", line 227, in decode_object
    raise CircularReference(index)
bpylist.archiver.CircularReference: archive has a cycle with uid(13)
```
In the debugger I traced this back to:
```
178  	    @staticmethod
179  	    def decode_archive(archive):
180  ->	        sortedPlaceInfos = archive.decode(""sortedPlaceInfos"")
181  	        finalPlaceInfos = archive.decode(""finalPlaceInfos"")
182  	        return PLRevGeoMapItem(sortedPlaceInfos, finalPlaceInfos)
```",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
615626118,MDU6SXNzdWU2MTU2MjYxMTg=,22,Try out ExifReader,9599,simonw,open,0,,,,,4,2020-05-11T06:32:13Z,2020-05-14T05:59:53Z,,MEMBER,,"https://pypi.org/project/ExifReader/

New fork that should be able to handle EXIF in HEIC files.

Forked here: https://github.com/ianare/exif-py/issues/102#issuecomment-626376522

Refs #3 ",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/22/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
621280529,MDU6SXNzdWU2MjEyODA1Mjk=,23,create-subset command for creating a publishable subset of a photos database,9599,simonw,closed,0,,,,,1,2020-05-19T20:58:20Z,2020-05-19T22:32:48Z,2020-05-19T22:32:37Z,MEMBER,,"I want to share a subset of my photos, without sharing everything. Idea:

    $ photos-to-sqlite create-subset photos.db public.db ""select sha256 from ... where ...""

So the command takes a SQL query that returns sha256 hashes, then creates a new file called `public.db` containing just the data corresponding to those photos.",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/23/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
621323348,MDU6SXNzdWU2MjEzMjMzNDg=,24,Configurable URL for images,9599,simonw,open,0,,,,,1,2020-05-19T22:25:56Z,2020-05-20T06:00:29Z,,MEMBER,,"This is hard-coded at the moment, which is bad:
https://github.com/dogsheep/photos-to-sqlite/blob/d5d69b9019703c47bc251444838578dd752801e2/photos_to_sqlite/cli.py#L269-L272",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/24/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
621332242,MDU6SXNzdWU2MjEzMzIyNDI=,25,Create a public demo,9599,simonw,closed,0,,,,,5,2020-05-19T22:47:20Z,2020-05-21T22:26:16Z,2020-05-20T05:54:18Z,MEMBER,,"So I can show people what this does, using some of my photos.",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/25/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
621444763,MDU6SXNzdWU2MjE0NDQ3NjM=,26,Rename project to dogsheep-photos,9599,simonw,closed,0,,,,,8,2020-05-20T04:12:34Z,2020-05-20T04:31:02Z,2020-05-20T04:30:40Z,MEMBER,,`photos-to-sqlite` doesn't really capture the full scope of this project anymore.,256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/26/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
621486115,MDU6SXNzdWU2MjE0ODYxMTU=,27,photos_with_apple_metadata view should include labels,9599,simonw,open,0,,,,,0,2020-05-20T06:06:17Z,2020-05-20T06:06:17Z,,MEMBER,,"https://dogsheep-photos.dogsheep.net/public/photos_with_apple_metadata?place_city=New+Orleans&_facet=place_city&_facet_array=albums&_facet_array=persons

Here's one way to add that:
```sql
        select
          rowid,
          photo,
          (
            select
              json_group_array(
                json_object(
                  'label',
                  normalized_string,
                  'href',
                  '/photos/labelled?_hide_sql=1&label=' || normalized_string
                )
              )
            from
              labels
            where
              labels.uuid = photos_with_apple_metadata.uuid
          ) as labels,
          date,
```",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/27/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
624490929,MDU6SXNzdWU2MjQ0OTA5Mjk=,28,Invalid SQL no such table: main.uploads,41439,dmd,open,0,,,,,1,2020-05-25T21:25:39Z,2020-12-24T22:26:22Z,,NONE,,"http://127.0.0.1:8001/photos/photos_with_apple_metadata gives ""Invalid SQL no such table: main.uploads""",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/28/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
638375985,MDExOlB1bGxSZXF1ZXN0NDM0MTYyMzE2,29,Fixed bug in SQL query for photo scores,41546558,RhetTbull,closed,0,,,,,1,2020-06-14T15:39:22Z,2020-12-04T22:32:36Z,2020-12-04T22:32:27Z,CONTRIBUTOR,dogsheep/dogsheep-photos/pulls/29,"The join on ZCOMPUTEDASSETATTRIBUTES used the wrong columns.  In most of the Photos database tables, table.ZASSET joins with ZGENERICASSET.Z_PK",256834907,dogsheep-photos,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/29/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
602533481,MDU6SXNzdWU2MDI1MzM0ODE=,3,"Import EXIF data into SQLite - lens used, ISO, aperture etc",9599,simonw,open,0,,,5324096,Apple Photos online and securely browsable,2,2020-04-18T19:24:31Z,2021-10-05T12:38:24Z,,MEMBER,,,256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
655974395,MDExOlB1bGxSZXF1ZXN0NDQ4MzU1Njgw,30,Handle empty bucket on first upload. Allow specifying the endpoint_url for services other than S3 (like b2 and digitalocean spaces),110038,scanner,open,0,,,,,0,2020-07-13T16:15:26Z,2020-07-13T16:15:26Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/dogsheep-photos/pulls/30,"Finally got around to trying dogsheep-photos but I want to use backblaze's b2 service instead of AWS S3.
Had to add a way to optionally specify the endpoint_url to connect to. Then with the bucket being empty the initial key retrieval would fail. Probably a better way to see that the bucket is empty than doing a test inside the paginator loop.

Also probably a better way to specify the endpoint_url as we get and test for it twice using the same code in two different places but did not want to spend too much time worrying about it.",256834907,dogsheep-photos,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/30/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
771511344,MDExOlB1bGxSZXF1ZXN0NTQzMDE1ODI1,31,Update for Big Sur,41546558,RhetTbull,open,0,,,,,7,2020-12-20T04:36:45Z,2023-08-08T15:52:52Z,,CONTRIBUTOR,dogsheep/dogsheep-photos/pulls/31,Refactored out the SQL for extracting aesthetic scores to use osxphotos -- adds compatbility for Big Sur via osxphotos which has been updated for new table names in Big Sur.  Have not yet refactored the SQL for extracting  labels which is still compatible with Big Sur.,256834907,dogsheep-photos,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
803333769,MDU6SXNzdWU4MDMzMzM3Njk=,32,KeyError: 'Contents' on running upload,11855322,robmarkcole,open,0,,,,,3,2021-02-08T08:36:37Z,2021-07-22T06:40:25Z,,NONE,,"Following the readme, on big sur, and having entered my auth creds via `dogsheep-photos s3-auth`:

```
(venv) (base) Robins-MacBook:datasette robin$ dogsheep-photos upload photos.db     ~/Pictures/Photos\ /Users/robin/Pictures/Library.photoslibrary --dry-run

Fetching existing keys from S3...
Traceback (most recent call last):
  File ""/Users/robin/datasette/venv/bin/dogsheep-photos"", line 8, in <module>
    sys.exit(cli())
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/cli.py"", line 96, in upload
    key.split(""."")[0] for key in get_all_keys(client, creds[""photos_s3_bucket""])
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/utils.py"", line 46, in get_all_keys
    for row in page[""Contents""]:
KeyError: 'Contents'
```

Possibly since the bucket is in `EU (London) eu-west-2` and this into is not requested?",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
803338729,MDU6SXNzdWU4MDMzMzg3Mjk=,33,photo-to-sqlite: command not found,11855322,robmarkcole,open,0,,,,,4,2021-02-08T08:42:57Z,2021-02-12T15:00:44Z,,NONE,,"Having installed in a venv I get:
```
(venv) (base) Robins-MacBook:datasette robin$ photo-to-sqlite apple-photos photos.db

-bash: photo-to-sqlite: command not found
```",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
830283447,MDU6SXNzdWU4MzAyODM0NDc=,34,bucket name,6213,dsisnero,open,0,,,,,0,2021-03-12T16:40:57Z,2021-03-12T16:40:57Z,,NONE,,I followed the instructions to setup credentials but I am getting a invalid bucket name. Can you put a sample auth.json file in the base that shows the correct format for this?  Thanks,256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/34/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
842695374,MDU6SXNzdWU4NDI2OTUzNzQ=,35,Support to annotate photos on other than  macOS OSes,1151557,ligurio,open,0,,,,,1,2021-03-28T09:01:25Z,2021-04-05T07:37:57Z,,NONE,,dogsheep-photos allows to annotate photos using Apple Photo's db. It would be nice to have such ability on other OSes too. For example using trained local model or using Google Vision API (see #14).,256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/35/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
988493790,MDExOlB1bGxSZXF1ZXN0NzI3MzkwODM1,36,Correct naming of tool in readme,2129,badboy,open,0,,,,,1,2021-09-05T12:05:40Z,2022-01-06T16:04:46Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/dogsheep-photos/pulls/36,,256834907,dogsheep-photos,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/36/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1293698966,PR_kwDOD079W84600uh,37,Fix former command name in readme,578773,DanLipsitt,open,0,,,,,0,2022-07-05T02:09:13Z,2022-07-05T02:09:13Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/dogsheep-photos/pulls/37,Looks like a previous commit missed a `photo-to-sqlite`→ `dogsheep-photos` replacement.,256834907,dogsheep-photos,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/37/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1827427757,PR_kwDOD079W85WtUKG,38,photos-to-sql not found?,319473,coldclimate,closed,0,,,,,2,2023-07-29T09:59:42Z,2023-07-29T10:01:27Z,2023-07-29T10:01:23Z,FIRST_TIME_CONTRIBUTOR,dogsheep/dogsheep-photos/pulls/38,"I wonder if `photos-to-sql` is an old name for `dogsheep-photos`, because I can't find it anywhere.

I can't actually get this command to work (`sqlite3.OperationalError: no such table: attached.ZGENERICASSET` thrown) but I don't think that's related",256834907,dogsheep-photos,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/38/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1827436260,PR_kwDOD079W85WtVyk,39,Missing option in datasette instructions,319473,coldclimate,open,0,,,,,0,2023-07-29T10:34:48Z,2023-07-29T10:34:48Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/dogsheep-photos/pulls/39,Gotta tell it where to look,256834907,dogsheep-photos,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/39/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
602533539,MDU6SXNzdWU2MDI1MzM1Mzk=,4,Upload all my photos to a secure S3 bucket,9599,simonw,closed,0,,,5324096,Apple Photos online and securely browsable,14,2020-04-18T19:24:50Z,2020-04-18T21:58:11Z,2020-04-18T21:57:13Z,MEMBER,,"- [x] Create a bucket with bucket credentials
- [x] Programmatically upload some recent photos to it (from a notebook)
- [x] Turn this into a script",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1871935751,I_kwDOD079W85vk3kH,40, ImportError: cannot import name 'formatargspec' from 'inspect',36752421,hosslikw,closed,0,,,,,0,2023-08-29T15:36:31Z,2023-08-31T03:18:07Z,2023-08-31T03:18:06Z,NONE,,"I get the following error when running ""pip3 install dogsheep-photos""
""  from inspect import ismethod, isclass, formatargspec
      ImportError: cannot import name 'formatargspec' from 'inspect' (/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/inspect.py). Did you mean: 'formatargvalues'?""
    
Python 3.12.0rc1
sqlite 3.43.0
datasette, version 0.64.3",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/40/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
602551638,MDU6SXNzdWU2MDI1NTE2Mzg=,5,photos-to-sqlite s3-auth command,9599,simonw,closed,0,,,,,1,2020-04-18T21:05:25Z,2020-04-18T21:08:44Z,2020-04-18T21:08:44Z,MEMBER,,Modeled on `github-to-sqlite auth` - prompts the user for their S3 credentials and saves them to `auth.json`.,256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
602575575,MDU6SXNzdWU2MDI1NzU1NzU=,6,Add progress bar to upload command,9599,simonw,closed,0,,,,,2,2020-04-18T23:32:41Z,2020-04-19T00:15:24Z,2020-04-19T00:15:24Z,MEMBER,,Upload was added in #4 ,256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
602585497,MDU6SXNzdWU2MDI1ODU0OTc=,7,Integrate image content hashing,9599,simonw,open,0,,,,,2,2020-04-19T00:36:58Z,2021-08-26T02:01:01Z,,MEMBER,,To spot duplicate images (where the file content differs such that the sha256 is no longer a match) it would be useful to calculate and store perceptual hashes of some sort.,256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,
605147638,MDU6SXNzdWU2MDUxNDc2Mzg=,8,Should I have used MD5 instead of SHA256?,9599,simonw,closed,0,,,,,2,2020-04-23T00:02:08Z,2020-04-23T00:03:35Z,2020-04-23T00:03:35Z,MEMBER,,"https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html

> Objects created by the PUT Object, POST Object, or Copy operation, or through the AWS Management Console, and are encrypted by SSE-S3 or plaintext, have ETags that are an MD5 digest of their object data.
",256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
605938063,MDU6SXNzdWU2MDU5MzgwNjM=,9,"upload command should be resumable, should only upload photos not already uploaded",9599,simonw,closed,0,,,,,2,2020-04-23T23:31:08Z,2020-04-23T23:39:14Z,2020-04-23T23:39:14Z,MEMBER,,Follow on from #4. ,256834907,dogsheep-photos,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
519979091,MDExOlB1bGxSZXF1ZXN0MzM4NjQ3Mzc4,1,Add parkrun-to-sqlite,1101318,mrw34,closed,0,,,,,0,2019-11-08T12:05:32Z,2020-10-12T00:35:16Z,2020-10-12T00:35:16Z,CONTRIBUTOR,dogsheep/dogsheep.github.io/pulls/1,,214746582,dogsheep.github.io,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
541274681,MDU6SXNzdWU1NDEyNzQ2ODE=,2,Add linkedin-to-sqlite,881925,mnp,open,0,,,,,0,2019-12-21T03:13:40Z,2019-12-21T03:13:40Z,,NONE,,"There is an API available. https://developer.linkedin.com/docs/rest-api#

At the minimum, I would think contact list and messages would be of interest.",214746582,dogsheep.github.io,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
543717994,MDExOlB1bGxSZXF1ZXN0MzU3OTc0MzI2,3,Add todoist-to-sqlite,706257,bcongdon,closed,0,,,,,0,2019-12-30T04:02:59Z,2020-10-12T00:35:58Z,2020-10-12T00:35:57Z,CONTRIBUTOR,dogsheep/dogsheep.github.io/pulls/3,"Really enjoying getting into the dogsheep/datasette ecosystem. I made a downloader for Todoist, and I think/hope others might find this useful",214746582,dogsheep.github.io,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/3/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
558715564,MDExOlB1bGxSZXF1ZXN0MzcwMDI0Njk3,4,Add beeminder-to-sqlite,706257,bcongdon,closed,0,,,,,0,2020-02-02T15:51:36Z,2020-10-12T00:36:16Z,2020-10-12T00:36:16Z,CONTRIBUTOR,dogsheep/dogsheep.github.io/pulls/4,,214746582,dogsheep.github.io,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
723499985,MDExOlB1bGxSZXF1ZXN0NTA1MDc2NDE4,5,Add fitbit-to-sqlite,4632208,mrphil007,open,0,,,,,0,2020-10-16T20:04:05Z,2020-10-16T20:04:05Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/dogsheep.github.io/pulls/5,,214746582,dogsheep.github.io,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
842765105,MDExOlB1bGxSZXF1ZXN0NjAyMjYxMDky,6,Add testres-db tool,1151557,ligurio,closed,0,,,,,1,2021-03-28T15:43:23Z,2022-02-16T05:12:05Z,2022-02-16T05:12:05Z,NONE,dogsheep/dogsheep.github.io/pulls/6,,214746582,dogsheep.github.io,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
925384329,MDExOlB1bGxSZXF1ZXN0NjczODcyOTc0,7,Add instagram-to-sqlite,36654812,gavindsouza,open,0,,,,,0,2021-06-19T12:26:16Z,2021-07-28T07:58:59Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/dogsheep.github.io/pulls/7,"The tool covers only chat imports at the time of opening this PR but I'm planning to import everything else that I feel inquisitive about

ref: https://github.com/gavindsouza/instagram-to-sqlite",214746582,dogsheep.github.io,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/7/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
927385540,MDU6SXNzdWU5MjczODU1NDA=,8,any guidance / experience on imessage-to-sqlite ?,2675621,Casyfill,open,0,,,,,0,2021-06-22T15:46:16Z,2021-06-22T15:46:16Z,,NONE,,,214746582,dogsheep.github.io,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
718934942,MDU6SXNzdWU3MTg5MzQ5NDI=,1,Documentation on how to use this with Datasette,9599,simonw,open,0,,,,,1,2020-10-11T21:56:27Z,2020-10-11T22:14:00Z,,MEMBER,,In particular how to use `datasette-render-images` to see the images.,303218369,evernote-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
770712149,MDExOlB1bGxSZXF1ZXN0NTQyNDA2OTEw,10,BugFix for encoding and not update info.,1277270,riverzhou,closed,0,,,,,1,2020-12-18T08:58:54Z,2021-02-11T22:37:56Z,2021-02-11T22:37:56Z,NONE,dogsheep/evernote-to-sqlite/pulls/10,"Bugfix 1:

Traceback (most recent call last):
  File ""d:\anaconda3\lib\runpy.py"", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File ""d:\anaconda3\lib\runpy.py"", line 87, in _run_code
    exec(code, run_globals)
  File ""D:\Anaconda3\Scripts\evernote-to-sqlite.exe\__main__.py"", line 7, in <module>
  File ""d:\anaconda3\lib\site-packages\click\core.py"", line 829, in __call__
  File ""d:\anaconda3\lib\site-packages\click\core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""d:\anaconda3\lib\site-packages\click\core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
    return ctx.invoke(self.callback, **ctx.params)
  File ""d:\anaconda3\lib\site-packages\click\core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""d:\anaconda3\lib\site-packages\evernote_to_sqlite\cli.py"", line 30, in enex
    for tag, note in find_all_tags(fp, [""note""], progress_callback=bar.update):
  File ""d:\anaconda3\lib\site-packages\evernote_to_sqlite\utils.py"", line 11, in find_all_tags
    chunk = fp.read(1024 * 1024)
UnicodeDecodeError: 'gbk' codec can't decode byte 0xa4 in position 383: illegal multibyte sequence

Bugfix 2:

Traceback (most recent call last):
  File ""D:\Anaconda3\Scripts\evernote-to-sqlite-script.py"", line 33, in <module>
    sys.exit(load_entry_point('evernote-to-sqlite==0.3', 'console_scripts', 'evernote-to-sqlite')())
  File ""D:\Anaconda3\lib\site-packages\click\core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""D:\Anaconda3\lib\site-packages\click\core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""D:\Anaconda3\lib\site-packages\click\core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""D:\Anaconda3\lib\site-packages\click\core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""D:\Anaconda3\lib\site-packages\click\core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""D:\Anaconda3\lib\site-packages\evernote_to_sqlite-0.3-py3.8.egg\evernote_to_sqlite\cli.py"", line 31, in enex
  File ""D:\Anaconda3\lib\site-packages\evernote_to_sqlite-0.3-py3.8.egg\evernote_to_sqlite\utils.py"", line 28, in save_note
AttributeError: 'NoneType' object has no attribute 'text'",303218369,evernote-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
792851444,MDU6SXNzdWU3OTI4NTE0NDQ=,11,XML parse error,3613583,dskrad,closed,0,,,,,2,2021-01-24T17:38:54Z,2021-02-11T21:18:58Z,2021-02-11T21:18:48Z,NONE,,"I am on Windows 10 using Windows Subsystem for Linux, Python 3.8. I installed evernote-to-sqlite via pipx (in a venv). I tried using enex files from the latest version of Evernote for Windows (10.6.9 which only lets you export 50 notes at a time) and from Legacy Evernote (6.25.2.9198 which lets you export all your notes at once). The enex file from latest evernote gives this error:

    File ""/usr/lib/python3.8/xml/etree/ElementTree.py"", line 1320, in XML parser.feed(text)
    xml.etree.ElementTree.ParseError: XML or text declaration not at start of entity: line 2, column 6

The enex file from Legacy Evernote gives this error:

    File ""/home/david/.local/pipx/venvs/evernote-to-sqlite/lib/python3.8/site-packages/evernote_to_sqlite/utils.py"", line 28, in save_note
    updated = note.find(""updated"").text
    AttributeError: 'NoneType' object has no attribute 'text'",303218369,evernote-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
892383270,MDExOlB1bGxSZXF1ZXN0NjQ1MTAwODQ4,12,Recovering of malformed ENEX file,8431437,engdan77,open,0,,,,,0,2021-05-15T07:49:31Z,2021-05-15T19:57:50Z,,FIRST_TIMER,dogsheep/evernote-to-sqlite/pulls/12,"Hey .. Awesome work developing this project, that I found very useful to me and saved me some work.. Thanks.. :)

Some background to this PR... 
I've been searching around for a tool allowing me to transforming my personal collection of Evernote notes to a format easier to search and potentially easier import to future services. 

Now I discovered problem processing my large data ~5GB using the existing source using Pythons builtin xml-parser that unfortunately was unable to succeed without exception breaking the process. 

My first attempt I tried to adapt to more robust lxml package allowing huge data and with ""recover"", but even if it worked better it also failed processing the whole data. Even using the memory efficient etree.iterparse() it also unfortunately got into trouble.

And with no luck finding any other libraries successfully parsing this enormous file I instead chose to build a ""hugexmlparser"" module that allows parsing this huge file using yield (on a byte-to-byte-level) and allows you to set a maximum size for <note> to cater for potential malformed or undesirable large attachments to export, should succeed covering potential exceptions. Some cases found where the parses discover malformed XML within <content> so also in those cases try to save as much as possible by escaping (to be dealt at a later stage, better than nothing), and if a missing end </note> before new (malformed?) it would add this after encounter a new start-tag.

The code for the recovery process is a bit rough and for certain room for refactoring, but at the moment is seem to achieve what I wanted.

Now with the above we pass this a minor changed version of save_note_recovery() assure the existing works.
Also adding this as a new recover-enex command to click and kept the original options. 
A couple of new tests was added as well to check against using this command.

Now this currently works to me, but thought I might share a PR in such as you find use for this yourself or found useful to others finding this repository.

As a second step .. When the time allows it would have been nice to also be able to easily export from SQLite to formatted HTML/MD and attachments saved... but that might perhaps be better a separate project ... or if you or someone else have something that might shared to save some trouble, I would be interested ;-) ",303218369,evernote-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/12/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
978743426,MDU6SXNzdWU5Nzg3NDM0MjY=,13,xml.etree.ElementTree.ParseError: not well-formed (invalid token),9599,simonw,closed,0,,,,,4,2021-08-25T05:48:21Z,2021-08-26T18:45:13Z,2021-08-26T18:45:13Z,MEMBER,,"Got this error today:
```
(evernote-to-sqlite) /tmp % evernote-to-sqlite enex evernote.db simonwillison\'s\ notebook.enex 
Importing from ENEX  [######------------------------------]   17%
Traceback (most recent call last):
  File ""/Users/simon/.local/bin/evernote-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py"", line 1137, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py"", line 1062, in main
    rv = self.invoke(ctx)
  File ""/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py"", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py"", line 763, in invoke
    return __callback(*args, **kwargs)
  File ""/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/evernote_to_sqlite/cli.py"", line 31, in enex
    save_note(db, note)
  File ""/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/evernote_to_sqlite/utils.py"", line 36, in save_note
    content = ET.tostring(ET.fromstring(content_xml)).decode(""utf-8"")
  File ""/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py"", line 1347, in XML
    parser.feed(text)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 2, column 132
```",303218369,evernote-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
986829194,MDU6SXNzdWU5ODY4MjkxOTQ=,14,xml.etree.ElementTree.Parse Error - mismatched tag,46968,step21,open,0,,,,,1,2021-09-02T14:46:36Z,2021-09-02T14:53:11Z,,NONE,,"This is an error message I get upon parsing the enex file of my Inbox. Please find the full error message below. Any hints welcome.

```
Importing from ENEX  [##################------------------]   50%  00:00:50
Traceback (most recent call last):
  File ""/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/bin/evernote-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py"", line 1137, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py"", line 1062, in main
    rv = self.invoke(ctx)
  File ""/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py"", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py"", line 763, in invoke
    return __callback(*args, **kwargs)
  File ""/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/evernote_to_sqlite/cli.py"", line 30, in enex
    for tag, note in find_all_tags(fp, [""note""], progress_callback=bar.update):
  File ""/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/evernote_to_sqlite/utils.py"", line 17, in find_all_tags
    for event, el in parser.read_events():
  File ""/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py"", line 1329, in read_events
    raise event
  File ""/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py"", line 1301, in feed
    self._parser.feed(data)
xml.etree.ElementTree.ParseError: mismatched tag: line 6837961, column 2
```
",303218369,evernote-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/14/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1042759769,PR_kwDOEhK-wc4uAJb9,15,include note tags in the export,436138,d-rep,open,0,,,,,0,2021-11-02T20:04:31Z,2021-11-02T20:04:31Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/evernote-to-sqlite/pulls/15,"When parsing the Evernote `<note>` elements, the script will now also parse any nested `<tag>` elements, writing them out into a separate sqlite table.

Here is an example of how to query the data after the script has run:
```
select notes.*,
	(select group_concat(tag) from notes_tags where notes_tags.note_id=notes.id) as tags
from notes;
```

My .enex source file is 3+ years old so I am assuming the structure hasn't changed.  Interestingly, my _notebook names_ show up in the _tags_ list where the tag name is prefixed with `notebook_`, so this could maybe help work around the first limitation mentioned in the [evernote-to-sqlite blog post](https://simonwillison.net/2020/Oct/16/building-evernote-sqlite-exporter/).
",303218369,evernote-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/15/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1943259395,I_kwDOEhK-wc5z08kD,16, time data '2014-11-21T11:44:12.000Z' does not match format '%Y%m%dT%H%M%SZ',3746270,linonetwo,open,0,,,,,0,2023-10-14T13:24:39Z,2023-10-14T13:24:39Z,,NONE,,"
```
evernote-to-sqlite enex evernote.db ./我的笔记.enex
Importing from ENEX  [#####-------------------------------]   14%
Traceback (most recent call last):
  File ""/usr/local/bin/evernote-to-sqlite"", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/cli.py"", line 31, in enex
    save_note(db, note)
  File ""/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/utils.py"", line 46, in save_note
    ""created"": convert_datetime(created),
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/utils.py"", line 111, in convert_datetime
    return datetime.datetime.strptime(s, ""%Y%m%dT%H%M%SZ"").isoformat()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/_strptime.py"", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/_strptime.py"", line 349, in _strptime
    raise ValueError(""time data %r does not match format %r"" %
ValueError: time data '2014-11-21T11:44:12.000Z' does not match format '%Y%m%dT%H%M%SZ'
```

enex is exported by evernote mac client ",303218369,evernote-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/16/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
718938046,MDU6SXNzdWU3MTg5MzgwNDY=,2,Convert dates to a better format,9599,simonw,closed,0,,,,,0,2020-10-11T22:12:33Z,2020-10-11T23:15:03Z,2020-10-11T23:15:03Z,MEMBER,,"They currently look like this:

https://github.com/dogsheep/evernote-to-sqlite/blob/9d8efd17580f6ddf76745c145d1e69dd24e52b64/tests/test_evernote_to_sqlite.py#L35-L36",303218369,evernote-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
718938321,MDU6SXNzdWU3MTg5MzgzMjE=,3,Use a content hash for the note IDs,9599,simonw,closed,0,,,,,0,2020-10-11T22:13:46Z,2020-10-11T23:15:04Z,2020-10-11T23:15:04Z,MEMBER,,"Without a GUID note IDs are pretty ineffective, but using a hash of the contents will at least avoid creating identical duplicates in the future.

https://sqlite-utils.readthedocs.io/en/stable/python-api.html#setting-an-id-based-on-the-hash-of-the-row-contents",303218369,evernote-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/3/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
718938508,MDU6SXNzdWU3MTg5Mzg1MDg=,4,Configure FTS + add an index on the date columns,9599,simonw,closed,0,,,,,2,2020-10-11T22:14:40Z,2020-10-11T23:41:29Z,2020-10-11T23:41:29Z,MEMBER,,"Sort by date descending is likely the most common way of sorting, so that column should be indexed.

Also add FTS configuration for both notes and the OCR column on resources.",303218369,evernote-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
718938889,MDU6SXNzdWU3MTg5Mzg4ODk=,5,Figure out how to display images from <en-media> tags inline in Datasette,9599,simonw,open,0,,,,,6,2020-10-11T22:17:03Z,2020-10-16T20:16:28Z,,MEMBER,,"Relates to #1. Evernote XML looks like this:

```xml
<?xml version=""1.0""?>
<en-note>
  <div>This note includes two images.</div>
  <div>
    <b>The Python logo</b>
  </div>
  <div>
    <en-media hash=""61098c2c541de7f0a907c301dd6542da"" type=""image/svg+xml"" width=""125""/>
  </div>
  <div>
    <b>The Evernote logo</b>
  </div>
  <div>
    <en-media hash=""91bd26175acac0b2ffdb6efac199f8ca"" type=""image/svg+xml"" width=""125""/>
  </div>
</en-note>
```
That hash is the md5 we use to store resources. It should be possible to turn these into embedded image tags, especially if done in conjunction with the https://github.com/simonw/datasette-media plugin.",303218369,evernote-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
718949182,MDU6SXNzdWU3MTg5NDkxODI=,6,Better handling of OCR data,9599,simonw,closed,0,,,,,2,2020-10-11T23:20:52Z,2020-10-12T00:04:10Z,2020-10-12T00:04:10Z,MEMBER,,"> I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought.

_Originally posted by @simonw in https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706784028_",303218369,evernote-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
743297582,MDU6SXNzdWU3NDMyOTc1ODI=,7,evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace',42387931,martinvanwieringen,closed,0,,,,,1,2020-11-15T16:57:28Z,2021-02-11T22:13:17Z,2021-02-11T22:13:17Z,NONE,,"running evernote-to-sqlite 0.2 on windows 10. Command:

evernote-to-sqlite enex evernote.db MyNotes.enex

I get the followinng error:

File ""C:\Users\marti\AppData\Roaming\Python\Python38\site-packages\evernote_to_sqlite\utils.py"", line 46, in save_note
    note_id = db[""notes""].insert(row, hash_id=""id"", replace=True, alter=True).last_pk
TypeError: insert() got an unexpected keyword argument 'replace'

Removing replace=True,

Leads to below error:

  note_id = db[""notes""].insert(row, hash_id=""id"", alter=True).last_pk
  File ""C:\Users\marti\AppData\Roaming\Python\Python38\site-packages\sqlite_utils\db.py"", line 924, in insert
    return self.insert_all(
  File ""C:\Users\marti\AppData\Roaming\Python\Python38\site-packages\sqlite_utils\db.py"", line 1046, in insert_all
    result = self.db.conn.execute(sql, values)
sqlite3.IntegrityError: UNIQUE constraint failed: notes.id",303218369,evernote-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
748370021,MDExOlB1bGxSZXF1ZXN0NTI1MzcxMDI5,8,"fix import error if note has no ""updated"" element",4028322,mkorosec,closed,0,,,,,0,2020-11-22T22:51:05Z,2021-02-11T22:34:06Z,2021-02-11T22:34:06Z,CONTRIBUTOR,dogsheep/evernote-to-sqlite/pulls/8,"I got the following error when executing evernote-to-sqlite enex evernote.db evernote.enex
```   
... 
  File ""evernote_to_sqlite/cli.py"", line 31, in enex
    save_note(db, note)
  File ""evernote_to_sqlite/utils.py"", line 28, in save_note
    updated = note.find(""updated"").text
AttributeError: 'NoneType' object has no attribute 'text'
```    

Seems that in some cases the updated element is not added to the note, this is a part of the problematic note:

```    
<created>20201019T074518Z</created>
<note-attributes>
   <source>web.clip7</source>
   <source-application>webclipper.evernote</source-application>
</note-attributes>
```",303218369,evernote-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
748372469,MDU6SXNzdWU3NDgzNzI0Njk=,9,ParseError: undefined entity &scaron;,4028322,mkorosec,closed,0,,,,,1,2020-11-22T23:04:35Z,2021-02-11T22:10:55Z,2021-02-11T22:10:55Z,CONTRIBUTOR,,"I encountered a parse error if the enex file contained &scaron; or &nbsp; 

Run command: 
evernote-to-sqlite enex evernote.db evernote.enex

```
Traceback (most recent call last):
...
  File ""evernote_to_sqlite/cli.py"", line 31, in enex
    save_note(db, note)
  File ""evernote_to_sqlite/utils.py"", line 35, in save_note
    content = ET.tostring(ET.fromstring(content_xml)).decode(""utf-8"")
  File ""/usr/lib/python3.8/xml/etree/ElementTree.py"", line 1320, in XML
    parser.feed(text)
xml.etree.ElementTree.ParseError: undefined entity &scaron;: line 3, column 35
```

Workaround:
```
sed -i 's/&scaron;//g' evernote.enex
sed -i 's/&nbsp;//g' evernote.enex
```",303218369,evernote-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
496415321,MDU6SXNzdWU0OTY0MTUzMjE=,1,Figure out some interesting example SQL queries,9599,simonw,open,0,,,,,9,2019-09-20T15:28:07Z,2021-05-03T03:46:23Z,,MEMBER,,My knowledge of genetics has left me short here. I'd love to be able to provide some interesting example SELECT queries - maybe one that spots if you are [likely to have red hair?](https://www.snpedia.com/index.php/Rs1805007),209590345,genome-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
664793260,MDU6SXNzdWU2NjQ3OTMyNjA=,2,Yak shave,145425,ekg,open,0,,,,,0,2020-07-23T22:04:18Z,2020-07-23T22:04:18Z,,NONE,,"Just a quick note... The 23andme data is not exactly your genome, but a SNP chip of your genome. It's ""some of your genotypes."" Or about 0.1% of your genome. Nice work in any case! It deserves to be liberated!!!!!",209590345,genome-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
493599818,MDU6SXNzdWU0OTM1OTk4MTg=,1,Command for fetching starred repos,9599,simonw,closed,0,,,,,0,2019-09-14T08:36:29Z,2019-09-14T21:30:48Z,2019-09-14T21:30:48Z,MEMBER,,,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
516967682,MDU6SXNzdWU1MTY5Njc2ODI=,10,Add this repos_starred view,9599,simonw,closed,0,,,,,3,2019-11-04T05:44:38Z,2020-05-02T16:37:36Z,2020-05-02T16:37:36Z,MEMBER,,"```sql
create view repos_starred as select
  stars.starred_at,
  users.login,
  repos.*
from
  repos
  join stars on repos.id = stars.repo
  join users on repos.owner = users.id
order by
  starred_at desc;
```",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
520521843,MDU6SXNzdWU1MjA1MjE4NDM=,11,Command to fetch releases,9599,simonw,closed,0,,,,,0,2019-11-09T22:23:30Z,2019-11-09T22:57:00Z,2019-11-09T22:57:00Z,MEMBER,,"https://developer.github.com/v3/repos/releases/#list-releases-for-a-repository

`GET /repos/:owner/:repo/releases`",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/11/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
520756546,MDU6SXNzdWU1MjA3NTY1NDY=,12,Add this view for seeing new releases,9599,simonw,closed,0,,,,,5,2019-11-11T06:00:12Z,2020-05-02T18:58:18Z,2020-05-02T18:58:17Z,MEMBER,,"```sql
CREATE VIEW recent_releases AS select
  json_object(""label"", repos.full_name, ""href"", repos.html_url) as repo,
  json_object(
    ""href"",
    releases.html_url,
    ""label"",
    releases.name
  ) as release,
  substr(releases.published_at, 0, 11) as date,
  releases.body as body_markdown,
  releases.published_at
from
  releases
  join repos on repos.id = releases.repo
order by
  releases.published_at desc
```",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
521275281,MDU6SXNzdWU1MjEyNzUyODE=,13,Set up a live demo Datasette instance,9599,simonw,closed,0,,,5225818,1.0,9,2019-11-12T01:27:02Z,2020-03-24T00:03:26Z,2020-03-24T00:03:25Z,MEMBER,,"I deployed https://github-to-sqlite-releases-j7hipcg4aq-uc.a.run.app/ by running this:
```
#!/bin/bash
# Fetch repos for simonw and dogsheep
github-to-sqlite repos github.db simonw dogsheep -a auth.json

# Fetch releases for the repos tagged 'datasette-io'
sqlite-utils github.db ""
select full_name from repos where rowid in (
    select repos.rowid from repos, json_each(repos.topics) j
    where j.value = 'datasette-io'
)"" --csv --no-headers | while read repo;
    do github-to-sqlite releases \
            github.db $(echo $repo | tr -d '\r') \
            -a auth.json;
        sleep 2;
    done;
```
And then deploying using this:
```
$ datasette publish cloudrun github.db \
  --title ""github-to-sqlite releases demo"" \
  --about_url=""https://github.com/simonw/github-to-sqlite"" \
  --about='github-to-sqlite' \
  --install=datasette-render-markdown \
  --install=datasette-json-html \
  --service=github-to-sqlite-releases
```
This should happen automatically for every release. I can run it once a day in Circle CI to keep the demo database up-to-date.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
530491074,MDU6SXNzdWU1MzA0OTEwNzQ=,14,Command for importing events,9599,simonw,open,0,,,,,3,2019-11-29T21:28:58Z,2020-04-14T19:38:34Z,,MEMBER,,"Eg from https://api.github.com/users/simonw/events

Docs here: https://developer.github.com/v3/activity/events/#list-events-performed-by-a-user",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
544571092,MDU6SXNzdWU1NDQ1NzEwOTI=,15,Assets table with downloads,2029,garethr,closed,0,,,5225818,1.0,4,2020-01-02T13:05:28Z,2020-03-28T12:17:01Z,2020-03-23T19:17:32Z,NONE,,"The `releases` command extracts the releases table, but data about the individual assets are locked up in the JSON document in the `assets` field. My main interest is in individual and aggregate download counts. I was wondering if creating a new table with a record per asset may be useful?
If so I'm happy to send a PR when I get a moment. Do you have opinions about that simply being part of the `releases` command or would you prefer a separate command as well?",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
546051181,MDU6SXNzdWU1NDYwNTExODE=,16,Exception running first command: IndexError: list index out of range,15092,jayvdb,closed,0,,,,,4,2020-01-07T03:01:58Z,2020-04-14T18:37:21Z,2020-04-14T18:37:21Z,NONE,,"Exception running first command without an existing db or auth.

```py
> mkdir ~/.github/coala
> /usr/bin/github-to-sqlite repos ~/.github/coala coala
Traceback (most recent call last):
  File ""/usr/bin/github-to-sqlite"", line 11, in <module>
    load_entry_point('github-to-sqlite==0.6', 'console_scripts', 'github-to-sqlite')()
  File ""/usr/lib/python3.7/site-packages/click/core.py"", line 764, in __call__
    return self.main(*args, **kwargs)
  File ""/usr/lib/python3.7/site-packages/click/core.py"", line 717, in main
    rv = self.invoke(ctx)
  File ""/usr/lib/python3.7/site-packages/click/core.py"", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/usr/lib/python3.7/site-packages/click/core.py"", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/usr/lib/python3.7/site-packages/click/core.py"", line 555, in invoke
    return callback(*args, **kwargs)
  File ""/usr/lib/python3.7/site-packages/github_to_sqlite/cli.py"", line 163, in repos
    utils.save_repo(db, repo)
  File ""/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py"", line 120, in save_repo
    to_save[""owner""] = save_user(db, to_save[""owner""])
  File ""/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py"", line 61, in save_user
    return db[""users""].upsert(to_save, pk=""id"", alter=True).last_pk
  File ""/usr/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1135, in upsert
    extracts=extracts,
  File ""/usr/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1162, in upsert_all
    upsert=True,
  File ""/usr/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1105, in insert_all
    row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0]
IndexError: list index out of range
```",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
578883725,MDU6SXNzdWU1Nzg4ODM3MjU=,17,Command for importing commits,9599,simonw,closed,0,,,,,2,2020-03-10T21:55:12Z,2020-03-11T02:47:37Z,2020-03-11T02:47:37Z,MEMBER,,Using this API: https://api.github.com/repos/dogsheep/github-to-sqlite/commits,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
585411547,MDU6SXNzdWU1ODU0MTE1NDc=,18,Commits in GitHub API can have null author,9599,simonw,closed,0,,,5225818,1.0,8,2020-03-21T02:20:56Z,2020-03-23T20:44:49Z,2020-03-23T20:44:26Z,MEMBER,,"```
Traceback (most recent call last):
  File ""/home/ubuntu/datasette-venv/bin/github-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py"", line 764, in __call__
    return self.main(*args, **kwargs)
  File ""/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py"", line 717, in main
    rv = self.invoke(ctx)
  File ""/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py"", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py"", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py"", line 555, in invoke
    return callback(*args, **kwargs)
  File ""/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/cli.py"", line 235, in commits
    utils.save_commits(db, commits, repo_full[""id""])
  File ""/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/utils.py"", line 290, in save_commits
    commit_to_insert[""author""] = save_user(db, commit[""author""])
  File ""/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/utils.py"", line 54, in save_user
    for key, value in user.items()
AttributeError: 'NoneType' object has no attribute 'items'
```
Got this running the `commits` command from cron.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
585850715,MDU6SXNzdWU1ODU4NTA3MTU=,19,"Enable full-text search for more stuff (like commits, issues and issue_comments)",9599,simonw,closed,0,,,5225818,1.0,2,2020-03-23T00:19:56Z,2020-03-23T19:06:39Z,2020-03-23T19:06:39Z,MEMBER,,Currently FTS is only enabled for repos and releases.,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
493668862,MDU6SXNzdWU0OTM2Njg4NjI=,2,Extract licenses from repos into a separate table,9599,simonw,closed,0,,,,,0,2019-09-14T21:33:41Z,2019-09-14T21:46:58Z,2019-09-14T21:46:58Z,MEMBER,,"<img width=""588"" alt=""starred3__repos__800_rows"" src=""https://user-images.githubusercontent.com/9599/64913845-a2074980-d6fc-11e9-815a-f103ad0c6f56.png"">
",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
586454513,MDU6SXNzdWU1ODY0NTQ1MTM=,20,Upgrade to sqlite-utils 2.x,9599,simonw,closed,0,,,5225818,1.0,0,2020-03-23T19:17:58Z,2020-03-23T19:22:52Z,2020-03-23T19:22:52Z,MEMBER,,,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/20/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
586561727,MDU6SXNzdWU1ODY1NjE3Mjc=,21,Turn GitHub API errors into exceptions,9599,simonw,closed,0,,,5225818,1.0,2,2020-03-23T22:37:24Z,2020-03-23T23:48:23Z,2020-03-23T23:48:22Z,MEMBER,,"This would have really helped in debugging the mess in #13. Running with this `auth.json` is a useful demo:
```json
{""github_personal_token"": """"}
```",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
586567379,MDU6SXNzdWU1ODY1NjczNzk=,22,Handle empty git repositories,9599,simonw,closed,0,,,,,0,2020-03-23T22:49:48Z,2020-03-23T23:13:11Z,2020-03-23T23:13:11Z,MEMBER,,"Got this error:
```
github_to_sqlite.utils.GitHubError: {'message': 'Git Repository is empty.', 'documentation_url': 'https://developer.github.com/v3/repos/commits/#list-commits-on-a-repository'}
```
From https://api.github.com/repos/dogsheep/beta/commits",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/22/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
586595839,MDU6SXNzdWU1ODY1OTU4Mzk=,23,Release 1.0,9599,simonw,closed,0,,,5225818,1.0,1,2020-03-24T00:03:55Z,2020-03-24T00:15:50Z,2020-03-24T00:15:50Z,MEMBER,,Need to compile release notes.,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
599776345,MDU6SXNzdWU1OTk3NzYzNDU=,24,Feature idea: github-to-sqlite everything ...,9599,simonw,open,0,,,,,0,2020-04-14T18:34:00Z,2020-04-14T18:34:00Z,,MEMBER,,"At the moment if you want to pull all your repos, issues, issues comments etc you have to do it with a sequence of separate commands.

Consider adding a `everything` or `all` command which fetches everything that the tool knows how to fetch, and is designed to be run on a cron in a way that fetches just new stuff each time.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/24/reactions"", ""total_count"": 7, ""+1"": 7, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
601265023,MDU6SXNzdWU2MDEyNjUwMjM=,25,Improvements to demo instance,9599,simonw,closed,0,,,,,1,2020-04-16T17:26:55Z,2020-04-16T18:07:12Z,2020-04-16T18:07:12Z,MEMBER,,- [x] Demo should pull issue-comments as well,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/25/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
601271612,MDU6SXNzdWU2MDEyNzE2MTI=,26,Topics are missing from repositories,9599,simonw,closed,0,,,,,2,2020-04-16T17:36:32Z,2020-04-16T17:41:11Z,2020-04-16T17:41:11Z,MEMBER,,"I'm sure this used to work, but right now repositories are fetched without their topics.

https://developer.github.com/v3/repos/ says you need to send a custom `Accept` header of `application/vnd.github.mercy-preview+json` to get topics.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/26/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
601330277,MDU6SXNzdWU2MDEzMzAyNzc=,27,Repos have a big blob of JSON in the organization column,9599,simonw,closed,0,,,,,5,2020-04-16T18:43:14Z,2020-04-18T00:19:16Z,2020-04-18T00:18:52Z,MEMBER,,"e.g. https://github-to-sqlite.dogsheep.net/github/repos

![github__repos__11_rows_where_sorted_by_updated_at_descending](https://user-images.githubusercontent.com/9599/79494124-5640b980-7fd7-11ea-99a2-17ffbd82f9ce.png)

This appears to be obsolete because the `owner` column already links to that record, albeit in the `users` table with `type` set to `Organization`: https://github-to-sqlite.dogsheep.net/github/users/53015001",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/27/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
601333634,MDU6SXNzdWU2MDEzMzM2MzQ=,28,Pull repository contributors,9599,simonw,closed,0,,,,,3,2020-04-16T18:46:40Z,2020-04-18T15:05:10Z,2020-04-18T15:05:10Z,MEMBER,,"https://developer.github.com/v3/repos/#list-contributors

`GET /repos/:owner/:repo/contributors`

Not sure if this should be a separate command or should be part of the existing `repos` command. I'm leaning towards a new `contributors` command.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/28/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
603617013,MDU6SXNzdWU2MDM2MTcwMTM=,29,Milestones should have foreign key to creator and repo,9599,simonw,closed,0,,,,,1,2020-04-21T00:20:44Z,2020-04-21T00:43:58Z,2020-04-21T00:43:58Z,MEMBER,,"https://github-to-sqlite.dogsheep.net/github/milestones

Creator is an integer but not a foreign key to users

Repo is missing entirely!",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/29/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
493670426,MDU6SXNzdWU0OTM2NzA0MjY=,3,Command to fetch all repos belonging to a user or organization,9599,simonw,closed,0,,,,,2,2019-09-14T21:54:21Z,2019-09-17T00:17:53Z,2019-09-17T00:17:53Z,MEMBER,,"How about this:

    $ github-to-sqlite repos simonw",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
603618244,MDU6SXNzdWU2MDM2MTgyNDQ=,30,Issues milestone column is the wrong type,9599,simonw,closed,0,,,,,2,2020-04-21T00:24:34Z,2020-04-21T00:45:23Z,2020-04-21T00:36:22Z,MEMBER,,"https://github-to-sqlite.dogsheep.net/github/issues?milestone=2857392

![2A4C1185-2434-4F29-9EA0-3246E2F03F77](https://user-images.githubusercontent.com/9599/79811760-b7e08b00-832b-11ea-9ad7-684a6ae097a6.jpeg)

It is TEXT when it should be an INTEGER - which is why the foreign key label is not correctly displayed.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/30/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
603624862,MDU6SXNzdWU2MDM2MjQ4NjI=,31,Issue and milestone should have foreign key to repo,9599,simonw,closed,0,,,,,3,2020-04-21T00:46:24Z,2020-04-22T01:20:19Z,2020-04-22T01:20:19Z,MEMBER,,"Currently the `repo` column on those tables is a string `simonw/datasette` rather than an ID referencing a row in `repos`.

_Originally posted by @simonw in https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275_",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
604222295,MDU6SXNzdWU2MDQyMjIyOTU=,32,Issue comments don't appear to populate issues foreign key,9599,simonw,closed,0,,,,,3,2020-04-21T19:17:32Z,2020-04-22T01:17:44Z,2020-04-22T01:17:44Z,MEMBER,,"https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101

<img width=""774"" alt=""Screen Shot 2020-04-21 at 12 16 57 PM"" src=""https://user-images.githubusercontent.com/9599/79904752-0b091b00-83ca-11ea-9b15-b2893f05dfdc.png"">
",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
609950090,MDU6SXNzdWU2MDk5NTAwOTA=,33,Fall back to authentication via ENV,2029,garethr,closed,0,,,,,4,2020-04-30T12:58:14Z,2020-05-02T18:46:10Z,2020-05-02T18:45:37Z,NONE,,"Would you accept a PR that falls back to looking for an environment variable for the GitHub token? Specifically a change here:
https://github.com/dogsheep/github-to-sqlite/blob/c34d5a18bfc41fa08755ba3d5cf9fe09ff204238/github_to_sqlite/cli.py#L271

I'd like to use `github-to-sqlite` in a GitHub Action workflow and this would be simpler than trying to fill out the prompt or generate a file with sensitive content.

Wanted to check first, I'm happy to submit a PR with tests and updates to the docs. ",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/33/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
610408908,MDU6SXNzdWU2MTA0MDg5MDg=,34,Command for retrieving dependents for a repo,9599,simonw,closed,0,,,,,6,2020-04-30T21:47:51Z,2020-05-03T15:53:01Z,2020-05-03T15:53:01Z,MEMBER,,"I really, really want to start grabbing this data: https://github.com/simonw/datasette/network/dependents",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
610511450,MDU6SXNzdWU2MTA1MTE0NTA=,35,Create index on issue_comments(user) and other foreign keys,9599,simonw,closed,0,,,,,3,2020-05-01T02:06:56Z,2020-05-02T18:26:24Z,2020-05-02T18:26:24Z,MEMBER,,"```
create index issue_comments_user on issue_comments(user)
```
I'm sure there are other user columns that could benefit from an index.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
610842926,MDU6SXNzdWU2MTA4NDI5MjY=,36,Add view for better display of dependent repos,9599,simonw,closed,0,,,,,2,2020-05-01T16:33:44Z,2020-05-02T16:50:31Z,2020-05-02T16:30:11Z,MEMBER,,"```sql
select
  repos.full_name as repo,
  'https://github.com/' || repos2.full_name as dependent,
  repos2.created_at as dependent_repo_created,
  repos2.updated_at as dependent_repo_updated,
  repos2.stargazers_count as dependent_repo_stars,
  repos2.watchers_count as dependent_repo_watchers
from
  dependents
  join repos as repos2 on dependents.dependent = repos2.id
  join repos on dependents.repo = repos.id
order by
  repos2.created_at desc
```
https://dogsheep.simonwillison.net/github?sql=select%0D%0A++repos.full_name+as+repo%2C%0D%0A++%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos2.full_name+as+dependent%2C%0D%0A++repos2.created_at+as+dependent_repo_created%2C%0D%0A++repos2.updated_at+as+dependent_repo_updated%2C%0D%0A++repos2.stargazers_count+as+dependent_repo_stars%2C%0D%0A++repos2.watchers_count+as+dependent_repo_watchers%0D%0Afrom%0D%0A++dependents%0D%0A++join+repos+as+repos2+on+dependents.dependent+%3D+repos2.id%0D%0A++join+repos+on+dependents.repo+%3D+repos.id%0D%0Aorder+by%0D%0A++repos2.created_at+desc",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
610843136,MDU6SXNzdWU2MTA4NDMxMzY=,37,Mechanism for creating views if they don't yet exist,9599,simonw,closed,0,,,,,3,2020-05-01T16:34:10Z,2020-05-02T16:19:47Z,2020-05-02T16:19:31Z,MEMBER,,Needed for #36 #10 #12 ,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
611284481,MDU6SXNzdWU2MTEyODQ0ODE=,38,[Feature Request] Support Repo Name in Search 🥺,5779832,zzeleznick,closed,0,,,,,4,2020-05-02T22:08:51Z,2020-05-03T02:34:32Z,2020-05-02T23:15:11Z,NONE,,"## Description
Per your [v2.2 release tweet](https://twitter.com/simonw/status/1256700238099693568) I played with the demo, but the output did not match my expectations.

## Expected Behavior
Expected a search query for ""twitter"" contained within the `repo` column to return non-zero results.

## Actual Behavior
😭  [0 rows where repo contains ""twitter"" sorted by starred_at descending](https://github-to-sqlite.dogsheep.net/github/stars?repo__contains=twitter&_sort_desc=starred_at) 

## Best Explanation
Per the table schema (see appendix) `repo` is of type `INTEGER` which built from `repo_id` and does not expose the repo name in search.

## Desired Behavior
Given that searching for ""206156866"" is less intuitive than ""twitter"",  it would be great to support this via extending the search capabilities or by adding an additional column.

✅ 104 rows where repo contains ""twitter""
❌  [104 rows where repo contains ""206156866"" sorted by starred_at descending](https://github-to-sqlite.dogsheep.net/github/stars?repo__contains=206156866&_sort_desc=starred_at) 

## Appendix
```
CREATE TABLE [stars] (
   [user] INTEGER REFERENCES [users]([id]),
   [repo] INTEGER REFERENCES [repos]([id]),
   [starred_at] TEXT,
   PRIMARY KEY ([user], [repo])
);
CREATE INDEX [idx_stars_repo]
                ON [stars] ([repo]);
CREATE INDEX [idx_stars_user]
                ON [stars] ([user]);
```",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/38/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
613777056,MDU6SXNzdWU2MTM3NzcwNTY=,39,issues foreign key to repo isn't working,9599,simonw,closed,0,,,,,1,2020-05-07T05:11:48Z,2020-08-18T14:24:46Z,2020-08-18T14:23:56Z,MEMBER,,"https://github-to-sqlite.dogsheep.net/github/issues?_facet=repo

<img width=""725"" alt=""github__issues__2_303_rows_where_sorted_by_updated_at_descending"" src=""https://user-images.githubusercontent.com/9599/81256810-723be780-8fe6-11ea-9e1f-f37d40f4d3ae.png"">

If the foreign key was working those would be repository names.

From the schema at the bottom of the page:
```
   [repo] TEXT,
```
That's the wrong type and not a foreign key.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/39/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
493670730,MDU6SXNzdWU0OTM2NzA3MzA=,4,Command to fetch stargazers for one or more repos,9599,simonw,closed,0,,,,,8,2019-09-14T21:58:22Z,2020-05-02T21:30:27Z,2020-05-02T21:30:27Z,MEMBER,,"Maybe this:

    $ github-to-sqlite stargazers github.db simonw/datasette

It could accept more than one repos.

Maybe have options similar to `--sql` in [twitter-to-sqlite](https://github.com/dogsheep/twitter-to-sqlite) so you can e.g. fetch all stargazers for all of the repos you have fetched into the database already (or all of the repos belonging to owner X)",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
637899539,MDU6SXNzdWU2Mzc4OTk1Mzk=,40,Demo deploy is broken,9599,simonw,closed,0,,,,,2,2020-06-12T17:20:17Z,2020-06-12T18:06:48Z,2020-06-12T18:06:48Z,MEMBER,,"https://github.com/dogsheep/github-to-sqlite/runs/766180404?check_suite_focus=true

```
The following NEW packages will be installed:
  sqlite3
0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.
Need to get 752 kB of archives.
After this operation, 2482 kB of additional disk space will be used.
Ign:1 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3
Err:1 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3
  404  Not Found [IP: 52.177.174.250 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/s/sqlite3/sqlite3_3.22.0-1ubuntu0.3_amd64.deb  404  Not Found [IP: 52.177.174.250 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
##[error]Process completed with exit code 100.
```",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
651159727,MDU6SXNzdWU2NTExNTk3Mjc=,41,Demo is failing to deploy,9599,simonw,closed,0,,,,,7,2020-07-05T22:40:33Z,2020-07-06T01:07:03Z,2020-07-06T01:07:02Z,MEMBER,,"https://github.com/dogsheep/github-to-sqlite/runs/837714622?check_suite_focus=true
```
Creating Revision.........................................................................................................................................failed
Deployment failed
ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.
Traceback (most recent call last):
  File ""/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette"", line 8, in <module>
    sys.exit(cli())
  File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/cloudrun.py"", line 138, in cloudrun
    check_call(
  File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/subprocess.py"", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'gcloud run deploy --allow-unauthenticated --platform=managed --image gcr.io/datasette-222320/datasette github-to-sqlite' returned non-zero exit status 1.
##[error]Process completed with exit code 1.
```",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
654405302,MDU6SXNzdWU2NTQ0MDUzMDI=,42,Option for importing just specific repos,9599,simonw,closed,0,,,,,0,2020-07-09T23:20:15Z,2020-07-09T23:25:35Z,2020-07-09T23:25:35Z,MEMBER,,"For if you know which specific repos you care about, as opposed to loading everything owned by the authenticated user.

    github-to-sqlite repos specific.db -r simonw/datasette -r simonw/github-contents
",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/42/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
660355904,MDU6SXNzdWU2NjAzNTU5MDQ=,43,github-to-sqlite tags command for fetching tags,9599,simonw,closed,0,,,,,4,2020-07-18T20:14:12Z,2020-07-18T23:05:56Z,2020-07-18T21:52:15Z,MEMBER,,Fetches paginated data from https://api.github.com/repos/simonw/datasette/tags,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
660413281,MDU6SXNzdWU2NjA0MTMyODE=,44,Rename tags.repo_id column to tags.repo,9599,simonw,closed,0,,,,,0,2020-07-18T22:13:46Z,2020-07-18T22:15:12Z,2020-07-18T22:15:12Z,MEMBER,,"For improved consistency with other tables. https://observablehq.com/@simonw/datasette-table-diagram

![datasette-table-diagram(1)](https://user-images.githubusercontent.com/9599/87862843-3cca4900-c909-11ea-9c76-58b3f4aca43f.png)
",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/44/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
660429601,MDU6SXNzdWU2NjA0Mjk2MDE=,45,Fix the demo - it breaks because of the tags table change,9599,simonw,closed,0,,,,,5,2020-07-18T22:49:32Z,2020-07-18T23:03:14Z,2020-07-18T23:03:13Z,MEMBER,,"https://github.com/dogsheep/github-to-sqlite/runs/885773677
```
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 476, in save_tags
    db[""tags""].insert_all(
  File ""/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1145, in insert_all
    result = self.db.conn.execute(query, params)
sqlite3.OperationalError: table tags has no column named repo
```
That's because I changed the name in #44. I thought this would be safe since no-one else could possibly be using this yet (it hadn't shipped in a release) but turns out I broke my demo!",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
664485022,MDU6SXNzdWU2NjQ0ODUwMjI=,46,Feature: pull request reviews and comments,1326704,bhrutledge,open,0,,,,,6,2020-07-23T13:43:45Z,2022-12-20T14:40:15Z,,NONE,,"Hi there! I saw your [presentation at Boston Python](https://www.meetup.com/bostonpython/events/271887195). I'm already a light user of Datasette (thank you!), but wasn't aware of this project.

I've been working on a ""pull request dashboard"" to get a comprehensive view of the state of open PR's, esp. related to reviews (i.e., pending, approved, changes requested). Currently it's a CLI command, but I thought a Datasette UI might be fun.

I see that PR's are available from the `issues` command, but I don't see reviews anywhere. From the [API docs](https://docs.github.com/en/rest/reference/pulls#reviews), it looks like there are separate endpoints for those (as well as pull requests in general). What do you think about adding that? Would you accept a PR? Any sense of the level of effort?",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
681086659,MDU6SXNzdWU2ODEwODY2NTk=,47,emojis command,9599,simonw,closed,0,,,,,1,2020-08-18T14:26:26Z,2020-08-18T14:52:13Z,2020-08-18T14:52:13Z,MEMBER,,For fun - it can import https://api.github.com/emojis - maybe with an option to fetch the binary representations in addition to the URLs.,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/47/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
681228542,MDExOlB1bGxSZXF1ZXN0NDY5NjUxNzMy,48,Add pull requests,755825,adamjonas,closed,0,,,,,2,2020-08-18T17:58:44Z,2020-11-29T23:51:09Z,2020-11-29T23:51:09Z,CONTRIBUTOR,dogsheep/github-to-sqlite/pulls/48,"ref #46 

Issues don't have merge information on them, which means that PRs need to be pulled separately.

Did my best to mimic the API of issues.",207052882,github-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
703216044,MDU6SXNzdWU3MDMyMTYwNDQ=,49,Feature: gists and starred gists,9599,simonw,open,0,,,,,0,2020-09-17T02:30:52Z,2020-09-17T02:30:52Z,,MEMBER,,https://developer.github.com/v3/gists/#list-starred-gists,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/49/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
493671014,MDU6SXNzdWU0OTM2NzEwMTQ=,5,"Add ""incomplete"" boolean to users table for incomplete profiles",9599,simonw,closed,0,,,,,2,2019-09-14T22:01:50Z,2020-03-23T19:23:31Z,2020-03-23T19:23:30Z,MEMBER,,"User profiles that are fetched from e.g. stargazers (#4) are incomplete - they have a login but they don't have name, company etc. 

Add a `incomplete` boolean flag to the `users` table to record this. Then later I can add a `backfill-users` command which loops through and fetches missing data for those incomplete profiles.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
703218756,MDU6SXNzdWU3MDMyMTg3NTY=,50,Commands for making authenticated API calls,9599,simonw,open,0,,,,,7,2020-09-17T02:39:07Z,2020-10-19T05:01:29Z,,MEMBER,,"Similar to `twitter-to-sqlite fetch`, see https://github.com/dogsheep/twitter-to-sqlite/issues/51",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
703246031,MDU6SXNzdWU3MDMyNDYwMzE=,51,github-to-sqlite should handle rate limits better,9599,simonw,open,0,,,,,4,2020-09-17T04:01:50Z,2022-10-14T16:34:07Z,,MEMBER,,From #50 - right now it will crash with an error of it hits the rate limit. Since the rate limit information (including reset time) is available in the headers it could automatically sleep and try again instead.,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
724264574,MDU6SXNzdWU3MjQyNjQ1NzQ=,52,Option to fetch README and/or HTML-rendered README for repos,9599,simonw,closed,0,,,,,0,2020-10-19T05:10:24Z,2020-10-19T05:33:42Z,2020-10-19T05:33:42Z,MEMBER,,"I'm thinking:

    github-to-sqlite repos ... --readme # Populates readme column with raw text
    github-to-sqlite repos ... --readme-html # Populates readme_html column with raw HTML

https://developer.github.com/v3/repos/contents/#get-a-repository-readme",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/52/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
753000405,MDU6SXNzdWU3NTMwMDA0MDU=,53,Command for fetching file contents,9599,simonw,open,0,,,,,1,2020-11-29T20:31:04Z,2020-11-30T00:36:09Z,,MEMBER,,"Something like this:

    github-to-sqlite files github.db simonw/datasette

This would fetch all files from the `main` branch into a `files` table.

Additional options could handle things like pulling files from a branch or tag, or just pulling files that match a specific glob or that exist in a specific directory.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/53/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
753026003,MDU6SXNzdWU3NTMwMjYwMDM=,54,github-to-sqlite workflows command,9599,simonw,closed,0,,,,,3,2020-11-29T21:56:42Z,2020-11-29T22:08:46Z,2020-11-29T21:57:17Z,MEMBER,,"A command that fetches the YAML workflows for different repos, parses them and stores them in relational tables would be really useful for maintaining larger numbers of workflows.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
753026388,MDU6SXNzdWU3NTMwMjYzODg=,55,github-to-sqlite workflows does not correctly replace existing records,9599,simonw,closed,0,,,,,0,2020-11-29T21:58:43Z,2020-11-29T23:48:50Z,2020-11-29T23:48:50Z,MEMBER,,Following #54 - see this TODO: https://github.com/dogsheep/github-to-sqlite/blob/1b23ce11953f9f59c0161ea1f99188b55b5ea11c/github_to_sqlite/utils.py#L700,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/55/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
753122082,MDU6SXNzdWU3NTMxMjIwODI=,56,Link to example tables from the README,9599,simonw,closed,0,,,,,0,2020-11-30T04:01:51Z,2020-11-30T04:10:27Z,2020-11-30T04:10:27Z,MEMBER,,Would help demonstrate how the tool works.,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/56/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
758944006,MDU6SXNzdWU3NTg5NDQwMDY=,57,--readme throws 404 error if README does not exist in repo,9599,simonw,closed,0,,,,,0,2020-12-07T23:58:49Z,2020-12-16T18:17:54Z,2020-12-16T18:17:54Z,MEMBER,,It should fail silently (populate the column with a null) instead.,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/57/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
769150394,MDU6SXNzdWU3NjkxNTAzOTQ=,58,Readme HTML has broken internal links,9599,simonw,closed,0,,,,,2,2020-12-16T17:58:11Z,2020-12-16T19:20:14Z,2020-12-16T19:20:14Z,MEMBER,,"From https://github.com/simonw/datasette.io/issues/46
```html
<li><a href=""#filtering-tables"">Filtering tables</a></li>
...
<h3><a id=""user-content-filtering-tables"" class=""anchor"" aria-hidden=""true"" href=""#filtering-tables""><svg class=""octicon octicon-link"" viewBox=""0 0 16 16"" version=""1.1"" width=""16"" height=""16"" aria-hidden=""true""><path fill-rule=""evenodd"" d=""M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z""></path></svg></a>Filtering tables</h3>
```
So this is a bug in GitHub's API, but we need to work around it.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
771872303,MDExOlB1bGxSZXF1ZXN0NTQzMjQ2NTM1,59,Remove unneeded exists=True for -a/--auth flag.,631242,frosencrantz,closed,0,,,,,3,2020-12-21T06:03:55Z,2021-05-22T14:06:19Z,2021-05-19T16:08:12Z,CONTRIBUTOR,dogsheep/github-to-sqlite/pulls/59,The file does not need to exist when using an environment variable.,207052882,github-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
504238461,MDU6SXNzdWU1MDQyMzg0NjE=,6,sqlite3.OperationalError: table users has no column named bio,1055831,dazzag24,closed,0,,,,,2,2019-10-08T19:39:52Z,2019-10-13T05:31:28Z,2019-10-13T05:30:19Z,NONE,,"```
$ github-to-sqlite repos github.db
$ github-to-sqlite starred github.db dazzag24

Traceback (most recent call last):
  File ""/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/bin/github-to-sqlite"", line 10, in <module>
    sys.exit(cli())
  File ""/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py"", line 764, in __call__
    return self.main(*args, **kwargs)
  File ""/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py"", line 717, in main
    rv = self.invoke(ctx)
  File ""/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py"", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py"", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py"", line 555, in invoke
    return callback(*args, **kwargs)
  File ""/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/cli.py"", line 106, in starred
    utils.save_stars(db, user, stars)
  File ""/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py"", line 177, in save_stars
    user_id = save_user(db, user)
  File ""/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py"", line 61, in save_user
    return db[""users""].upsert(to_save, pk=""id"").last_pk
  File ""/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py"", line 1067, in upsert
    extracts=extracts,
  File ""/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py"", line 916, in insert
    extracts=extracts,
  File ""/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py"", line 1024, in insert_all
    result = self.db.conn.execute(sql, values)
sqlite3.OperationalError: table users has no column named bio

```

```
$ pipenv graph
github-to-sqlite==0.4
  - requests [required: Any, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
  - sqlite-utils [required: ~=1.11, installed: 1.11]
    - click [required: Any, installed: 7.0]
    - click-default-group [required: Any, installed: 1.2.2]
      - click [required: Any, installed: 7.0]
    - tabulate [required: Any, installed: 0.8.5]

Python 3.6.8
```",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
797097140,MDU6SXNzdWU3OTcwOTcxNDA=,60,Use Data from SQLite in other commands,22578954,daniel-butler,open,0,,,,,3,2021-01-29T18:35:52Z,2021-02-12T18:29:43Z,,CONTRIBUTOR,,"As a total beginner here how could you access data from the sqlite table to run other commands.

What I am thinking is I want to get all the repos in an organization then using the repo list pull all the commit messages for each repo. 

I love this project by the way!",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
797108702,MDExOlB1bGxSZXF1ZXN0NTY0MTcyMTQw,61,fixing typo in get cli help text,22578954,daniel-butler,closed,0,,,,,1,2021-01-29T18:57:04Z,2021-05-19T16:07:09Z,2021-05-19T16:07:09Z,CONTRIBUTOR,dogsheep/github-to-sqlite/pulls/61,,207052882,github-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/61/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
797784080,MDU6SXNzdWU3OTc3ODQwODA=,62,Stargazers and workflows commands always require an auth file when using GITHUB_TOKEN ,631242,frosencrantz,open,0,,,,,0,2021-01-31T18:56:05Z,2021-01-31T18:56:05Z,,CONTRIBUTOR,,"Requested fix in https://github.com/dogsheep/github-to-sqlite/pull/59

The stargazers and workflows commands always require an auth file, even when using a `GITHUB_TOKEN`.  Other commands don't require the auth file.

",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/62/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
897212458,MDU6SXNzdWU4OTcyMTI0NTg=,63,Ability to fetch commits from branches other than the default,9599,simonw,open,0,,,,,0,2021-05-20T17:58:08Z,2021-05-20T17:58:08Z,,MEMBER,,This tool is currently almost entirely ignorant of the concept of branches. One example: you can't retrieve commits from any branch other than the default (usually main).,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/63/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
920636216,MDU6SXNzdWU5MjA2MzYyMTY=,64,"feature: support ""events""",231498,khimaros,open,0,,,,,5,2021-06-14T17:42:49Z,2021-06-15T00:48:37Z,,NONE,,"the GitHub API provides the ability to fetch all events for a given user, organization, or repository: https://docs.github.com/en/rest/reference/activity#list-events-for-the-authenticated-user

this would allow users to export all of the issue comments, new issues, etc. that they created. something which is currently missing from the GitHub takeout exports.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
923270900,MDExOlB1bGxSZXF1ZXN0NjcyMDUzODEx,65,basic support for events,231498,khimaros,open,0,,,,,2,2021-06-17T00:51:30Z,2022-10-03T22:35:03Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/github-to-sqlite/pulls/65,"a quick first pass at implementing the feature requested in https://github.com/dogsheep/github-to-sqlite/issues/64

testing instructions:

```
$ github-to-sqlite events events.db user/khimaros
```

if the specified user is the authenticated user, it will also include private events.

caveat: pagination appears to be broken (i don't see `next` in the response JSON from GitHub)",207052882,github-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
975161924,MDExOlB1bGxSZXF1ZXN0NzE2MzU3OTgy,66,Add --merged-by flag to pull-requests sub command,30531572,sarcasticadmin,open,0,,,,,1,2021-08-20T00:57:55Z,2021-09-28T21:50:31Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/github-to-sqlite/pulls/66,"## Description

Proposing a solution to the API limitation for `merged_by` in pull_requests. Specifically the following called out in the readme:

```
Note that the merged_by column on the pull_requests table will only be populated for pull requests that are loaded using the --pull-request option - the GitHub API does not return this field for pull requests that are loaded in bulk.
```

This approach might cause larger repos to hit rate limits called out in https://github.com/dogsheep/github-to-sqlite/issues/51 but seems to work well in the repos I tested and included below.

## Old Behavior
- Had to list out the pull-requests individually via multiple `--pull-request` flags

## New Behavior

- `--merged-by` flag for getting 'merge_by' information out of pull-requests without having to specify individual PR numbers.

# Testing

Picking some repo that has more than one merger (datasette only has 1 😉 )

```
$ github-to-sqlite pull-requests ./github.db opnsense/tools --merged-by
$ echo ""select id, url, merged_by from pull_requests;"" | sqlite3 ./github.db 
83533612|https://github.com/opnsense/tools/pull/39|1915288
102632885|https://github.com/opnsense/tools/pull/43|1915288
149114810|https://github.com/opnsense/tools/pull/57|1915288
160394495|https://github.com/opnsense/tools/pull/64|1915288
163308408|https://github.com/opnsense/tools/pull/67|1915288
169723264|https://github.com/opnsense/tools/pull/69|1915288
171381422|https://github.com/opnsense/tools/pull/72|1915288
179938195|https://github.com/opnsense/tools/pull/77|1915288
196233824|https://github.com/opnsense/tools/pull/82|1915288
215289964|https://github.com/opnsense/tools/pull/93|
219696100|https://github.com/opnsense/tools/pull/97|1915288
223664843|https://github.com/opnsense/tools/pull/99|
228446172|https://github.com/opnsense/tools/pull/103|1915288
238930434|https://github.com/opnsense/tools/pull/110|1915288
255507110|https://github.com/opnsense/tools/pull/119|1915288
255980675|https://github.com/opnsense/tools/pull/120|1915288
261906770|https://github.com/opnsense/tools/pull/125|
263800503|https://github.com/opnsense/tools/pull/127|1915288
264038685|https://github.com/opnsense/tools/pull/128|1915288
264696704|https://github.com/opnsense/tools/pull/129|1915288
266660547|https://github.com/opnsense/tools/pull/130|1915288
273120409|https://github.com/opnsense/tools/pull/133|1915288
274370803|https://github.com/opnsense/tools/pull/135|
276600629|https://github.com/opnsense/tools/pull/139|
277303655|https://github.com/opnsense/tools/pull/141|1915288
293033714|https://github.com/opnsense/tools/pull/145|
294827649|https://github.com/opnsense/tools/pull/146|
295140008|https://github.com/opnsense/tools/pull/147|1915288
305690829|https://github.com/opnsense/tools/pull/150|9783985
307077931|https://github.com/opnsense/tools/pull/152|1915288
321782100|https://github.com/opnsense/tools/pull/155|
337265672|https://github.com/opnsense/tools/pull/160|
337267484|https://github.com/opnsense/tools/pull/161|1915288
368251763|https://github.com/opnsense/tools/pull/169|
428262505|https://github.com/opnsense/tools/pull/181|
437557011|https://github.com/opnsense/tools/pull/182|1915288
447079893|https://github.com/opnsense/tools/pull/185|
461822092|https://github.com/opnsense/tools/pull/191|
463290142|https://github.com/opnsense/tools/pull/193|1915288
470112962|https://github.com/opnsense/tools/pull/194|1915288
472644649|https://github.com/opnsense/tools/pull/195|1915288
488696898|https://github.com/opnsense/tools/pull/198|
513289902|https://github.com/opnsense/tools/pull/201|
522530265|https://github.com/opnsense/tools/pull/203|
564443347|https://github.com/opnsense/tools/pull/213|
597579516|https://github.com/opnsense/tools/pull/220|1915288
602860357|https://github.com/opnsense/tools/pull/221|1915288
608744738|https://github.com/opnsense/tools/pull/222|1915288
623279673|https://github.com/opnsense/tools/pull/228|1915288
664656182|https://github.com/opnsense/tools/pull/233|
664781786|https://github.com/opnsense/tools/pull/234|1915288
670683636|https://github.com/opnsense/tools/pull/235|1915288
683150764|https://github.com/opnsense/tools/pull/237|
685016233|https://github.com/opnsense/tools/pull/238|
687099825|https://github.com/opnsense/tools/pull/239|1915288
715705652|https://github.com/opnsense/tools/pull/244|1915288
715721248|https://github.com/opnsense/tools/pull/245|1915288
```
`userid` are now present for those PRs that were merged.

Without the flag the `merged_by` behavior remains missing as expected when get PRs bulk:

```
$ github-to-sqlite pull-requests ./github.db opnsense/tools
$ echo ""select id, url, merged_by from pull_requests;"" | sqlite3 ./github.db 
83533612|https://github.com/opnsense/tools/pull/39|
102632885|https://github.com/opnsense/tools/pull/43|
149114810|https://github.com/opnsense/tools/pull/57|
160394495|https://github.com/opnsense/tools/pull/64|
163308408|https://github.com/opnsense/tools/pull/67|
169723264|https://github.com/opnsense/tools/pull/69|
171381422|https://github.com/opnsense/tools/pull/72|
179938195|https://github.com/opnsense/tools/pull/77|
196233824|https://github.com/opnsense/tools/pull/82|
215289964|https://github.com/opnsense/tools/pull/93|
219696100|https://github.com/opnsense/tools/pull/97|
223664843|https://github.com/opnsense/tools/pull/99|
228446172|https://github.com/opnsense/tools/pull/103|
238930434|https://github.com/opnsense/tools/pull/110|
255507110|https://github.com/opnsense/tools/pull/119|
255980675|https://github.com/opnsense/tools/pull/120|
261906770|https://github.com/opnsense/tools/pull/125|
263800503|https://github.com/opnsense/tools/pull/127|
264038685|https://github.com/opnsense/tools/pull/128|
264696704|https://github.com/opnsense/tools/pull/129|
266660547|https://github.com/opnsense/tools/pull/130|
273120409|https://github.com/opnsense/tools/pull/133|
274370803|https://github.com/opnsense/tools/pull/135|
276600629|https://github.com/opnsense/tools/pull/139|
277303655|https://github.com/opnsense/tools/pull/141|
293033714|https://github.com/opnsense/tools/pull/145|
294827649|https://github.com/opnsense/tools/pull/146|
295140008|https://github.com/opnsense/tools/pull/147|
305690829|https://github.com/opnsense/tools/pull/150|
307077931|https://github.com/opnsense/tools/pull/152|
321782100|https://github.com/opnsense/tools/pull/155|
337265672|https://github.com/opnsense/tools/pull/160|
337267484|https://github.com/opnsense/tools/pull/161|
368251763|https://github.com/opnsense/tools/pull/169|
428262505|https://github.com/opnsense/tools/pull/181|
437557011|https://github.com/opnsense/tools/pull/182|
447079893|https://github.com/opnsense/tools/pull/185|
461822092|https://github.com/opnsense/tools/pull/191|
463290142|https://github.com/opnsense/tools/pull/193|
470112962|https://github.com/opnsense/tools/pull/194|
472644649|https://github.com/opnsense/tools/pull/195|
488696898|https://github.com/opnsense/tools/pull/198|
513289902|https://github.com/opnsense/tools/pull/201|
522530265|https://github.com/opnsense/tools/pull/203|
564443347|https://github.com/opnsense/tools/pull/213|
597579516|https://github.com/opnsense/tools/pull/220|
602860357|https://github.com/opnsense/tools/pull/221|
608744738|https://github.com/opnsense/tools/pull/222|
623279673|https://github.com/opnsense/tools/pull/228|
664656182|https://github.com/opnsense/tools/pull/233|
664781786|https://github.com/opnsense/tools/pull/234|
670683636|https://github.com/opnsense/tools/pull/235|
683150764|https://github.com/opnsense/tools/pull/237|
685016233|https://github.com/opnsense/tools/pull/238|
687099825|https://github.com/opnsense/tools/pull/239|
715705652|https://github.com/opnsense/tools/pull/244|
715721248|https://github.com/opnsense/tools/pull/245|
```

Individual PRs passed via `--pull-request` flag behaves as expected (unchanged):

```
$ github-to-sqlite pull-requests ./github.db opnsense/tools --pull-request 39 --pull-request 237
$ echo ""select id, url, merged_by from pull_requests;"" | sqlite3 ./github.db
83533612|https://github.com/opnsense/tools/pull/39|1915288
683150764|https://github.com/opnsense/tools/pull/237|
```
> Picking 1 PR that has a merged_by (39) and one that does not (237)",207052882,github-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/66/reactions"", ""total_count"": 3, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",0,
981690086,MDExOlB1bGxSZXF1ZXN0NzIxNjg2NzIx,67,Replacing step ID key with step_id,16374374,jshcmpbll,open,0,,,,,0,2021-08-28T01:26:41Z,2021-08-28T01:27:00Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/github-to-sqlite/pulls/67,"Workflows that have an `id` in any step result in the following error when running `workflows`:

e.g.`github-to-sqlite workflows github.db nixos/nixpkgs`

```Traceback (most recent call last):
  File ""/usr/local/bin/github-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1137, in __call__
    return self.main(*args, **kwargs)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1062, in main
    rv = self.invoke(ctx)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1668, in invoke```Traceback (most recent call last):
  File ""/usr/local/bin/github-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1137, in __call__
    return self.main(*args, **kwargs)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1062, in main
    rv = self.invoke(ctx)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 763, in invoke
    return __callback(*args, **kwargs)
  File ""/usr/local/lib/python3.8/dist-packages/github_to_sqlite/cli.py"", line 601, in workflows
    utils.save_workflow(db, repo_id, filename, content)
  File ""/usr/local/lib/python3.8/dist-packages/github_to_sqlite/utils.py"", line 865, in save_workflow
    db[""steps""].insert_all(
  File ""/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py"", line 2596, in insert_all
    self.insert_chunk(
  File ""/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py"", line 2378, in insert_chunk
    result = self.db.execute(query, params)
  File ""/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py"", line 419, in execute
    return self.conn.execute(sql, parameters)
sqlite3.IntegrityError: datatype mismatch
```

 - [Information about the ID key in a step for GHA](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepsid)
 - [An example workflow from a public repo](https://github.com/NixOS/nixpkgs/blob/b4cc66827745e525ce7bb54659845ac89788a597/.github/workflows/direct-push.yml#L16)

# Changes
I'm proposing that the key for `id` in step is replaced with `step_id` so that it no longer interferes with the table `id` for tracking the record.

Special thanks to @sarcasticadmin @egiffen and @ruebenramirez for helping a bit on this 😄 ",207052882,github-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/67/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1013506559,PR_kwDODFdgUs4skaNS,68,Add support for retrieving teams / members,68329,philwills,open,0,,,,,0,2021-10-01T15:55:02Z,2021-10-01T15:59:53Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/github-to-sqlite/pulls/68,Adds a method for retrieving all the teams within an organisation and all the members in those teams. The latter is stored as a join table `team_members` beteween `teams` and `users`.,207052882,github-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/68/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1071071397,I_kwDODFdgUs4_10Cl,69,View that combines issues and issue comments,9599,simonw,open,0,,,,,1,2021-12-04T00:34:33Z,2021-12-04T00:34:52Z,,MEMBER,,I want to see a reverse chronologically ordered interface onto both issues and comments - essentially a unified log of comments and issues opened across one or multiple projects.,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/69/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
506276893,MDU6SXNzdWU1MDYyNzY4OTM=,7,issue-comments command for importing issue comments,9599,simonw,closed,0,,,,,1,2019-10-13T05:23:58Z,2019-10-14T14:44:12Z,2019-10-13T05:24:30Z,MEMBER,,Using this API: https://developer.github.com/v3/issues/comments/,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/7/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1149402080,PR_kwDODFdgUs4zaUta,70,scrape-dependents: enable paging through package menu option if present,36061055,stanbiryukov,open,0,,,,,0,2022-02-24T15:07:25Z,2022-02-24T15:07:25Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/github-to-sqlite/pulls/70,Some repos organize network dependents by a Package toggle. This PR adds the ability to page through those options and scrape underlying dependents.,207052882,github-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/70/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1177059481,I_kwDODFdgUs5GKICZ,71,Store commit parents,64686,carltongibson,closed,0,,,,,0,2022-03-22T17:06:48Z,2022-04-22T12:44:04Z,2022-04-22T12:44:04Z,NONE,,"Hi @simonw 👋 

Currently, stored commit data doesn't quite give me the information I'm needing... 

Committer date and author date are not 100% reliable for dividing a commit history up by release or branch. 
A PR created before a release but merged after can have earlier dates… — this can be quite frustrating if you're trying to pin down commits for a release: _It should be there!_, but then isn't. (This gets worse using release branches.) 

Would you be open to adding the `sha` of a `parent` of a commit to the commit table? (As an FK? 🤔 — likely not feasible.)

It's part of the [response body](https://docs.github.com/en/rest/reference/commits#get-a-commit): 

```
  ""parents"": [
    {
      ""url"": ""https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e"",
      ""sha"": ""6dcb09b5b57875f334f61aebed695e2e4193db5e""
    }
  ],
```

I think this list should only have a single entry. (🤔 — not sure why it's a list then...)

With this it would be possible to build/reconstruct a chain of commits from the history, that I don't **think** is available as yet (unless you know a better way). 

It is certainly possible to get sequential lists of commits out of git directly, so the same would be possible combining tools, but wondering if a single tool could do it. 

What do you think? 
Thanks! 🏅 

",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/71/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1211283427,I_kwDODFdgUs5IMrfj,72,feature: display progress bar when downloading multi-page responses,9020979,hydrosquall,open,0,,,,,1,2022-04-21T16:37:12Z,2022-04-21T17:29:31Z,,NONE,,"## Motivation

For a long running command (longer than 1 minute) for a big table (like pull requests or commits), it can be tricky to know if the script is still running, or if a rate limit/error was encountered

We know how many pages there are, so it may be possible to indicate how many remain.

https://github.com/dogsheep/github-to-sqlite/blob/a6e237f75a4b86963d91dcb5c9582e3a1b3349d6/github_to_sqlite/utils.py#L367

## Resources

- Using the existing Click API: 
   - https://click.palletsprojects.com/en/5.x/utils/#showing-progress-bars
- Loading spinner: https://github.com/pavdmyt/yaspin
- Progress bar: https://github.com/tqdm/tqdm",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/72/reactions"", ""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1261884917,PR_kwDODFdgUs45K1L3,73,Fixing 'NoneType' object has no attribute 'items',1224205,empjustine,closed,0,,,,,1,2022-06-06T13:58:11Z,2022-07-18T19:40:12Z,2022-07-18T19:40:12Z,CONTRIBUTOR,dogsheep/github-to-sqlite/pulls/73,"Under some conditions, GitHub caches removed starred repositories and ends up leaving dangling `None` user references.

    Traceback (most recent call last):
      File ""/home/dogsheep/dogsheep/github-to-sqlite/bin/github-to-sqlite"", line 8, in <module>
        sys.exit(cli())
      File ""/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py"", line 1130, in __call__
        return self.main(*args, **kwargs)
      File ""/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py"", line 1055, in main
        rv = self.invoke(ctx)
      File ""/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py"", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File ""/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py"", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File ""/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py"", line 760, in invoke
        return __callback(*args, **kwargs)
      File ""/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/cli.py"", line 181, in starred
        utils.save_stars(db, user, stars)
      File ""/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py"", line 494, in save_stars
        repo_id = save_repo(db, repo)
      File ""/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py"", line 308, in save_repo
        to_save[""owner""] = save_user(db, to_save[""owner""])
      File ""/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py"", line 229, in save_user
        for key, value in user.items()
    AttributeError: 'NoneType' object has no attribute 'items'",207052882,github-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/73/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1308461063,I_kwDODFdgUs5N_YgH,74,500 error in github-to-sqlite demo,9599,simonw,closed,0,,,,,5,2022-07-18T19:39:32Z,2022-07-18T21:16:18Z,2022-07-18T21:14:22Z,MEMBER,,"https://github-to-sqlite.dogsheep.net/github/issue_comments throws a 500:

> `cannot import name 'etree' from 'markdown.util' (/usr/local/lib/python3.8/site-packages/markdown/util.py)`

https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/metrics?project=datasette-222320 suggests this started happening 3 days ago.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1363244199,I_kwDODFdgUs5RQXSn,75,Fetch repos doesn't support organisations,2757699,OverkillGuy,open,0,,,,,0,2022-09-06T12:55:06Z,2022-09-06T12:55:06Z,,NONE,,"Say I want to get all my Github Org's repos info, for data analysis. Not just the public repos, but also the private/internal repos.

The endpoints are different for organisation, and this tool doesn't take it into account:
https://github.com/dogsheep/github-to-sqlite/blob/ace13ec3d98090d99bd71871c286a4a612c96a50/github_to_sqlite/utils.py#L453
https://github.com/dogsheep/github-to-sqlite/blob/ace13ec3d98090d99bd71871c286a4a612c96a50/github_to_sqlite/utils.py#L455

The endpoints for organisation repos is instead ([source](https://docs.github.com/en/rest/repos/repos#list-organization-repositories)):
`url = ""https://api.github.com/orgs/{}/repos"".format(username)`

Let's add support for organisations repo scraping.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/75/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1363280254,PR_kwDODFdgUs4-cIa_,76,Add organization support to repos command,2757699,OverkillGuy,open,0,,,,,1,2022-09-06T13:21:42Z,2022-09-06T13:59:08Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/github-to-sqlite/pulls/76,"New --organization flag to signify all given ""usernames"" are private
orgs. Adapts API URL to the organization path instead.

Not the best implementation, but a first draft to talk around

Fixes #75 (badly, no tests, overly vague, untested)",207052882,github-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/76/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1410548368,I_kwDODFdgUs5UE0KQ,77,Feature: Support GitHub discussions,631242,frosencrantz,open,0,,,,,0,2022-10-16T16:53:38Z,2022-10-16T16:53:38Z,,CONTRIBUTOR,,"Hi @simonw I've been a happy user of this tool.  Thank you for writing it and sharing it.

I wanted to suggest a feature request to support  Discussions.  For example the VisiData project has discussions https://github.com/saulpw/visidata/discussions , and it would be useful if there was a way to pull that data into the database.

However, I'm not offering a pull request.",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/77/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1505411725,I_kwDODFdgUs5ZusKN,78,self-hosted or corp github enterprise,549431,ebdavison,open,0,,,,,0,2022-12-20T22:51:45Z,2022-12-20T22:51:45Z,,NONE,,"We use github enterprise at work and I would like to use this tool to pull info from that site rather than the public github.com instance.  Is there an option for this?  If not, can one be added for a custom repo URL?",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/78/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1570375808,I_kwDODFdgUs5dmgiA,79,Deploy demo job is failing due to rate limit,9599,simonw,open,0,,,,,2,2023-02-03T20:05:01Z,2023-12-08T14:50:15Z,,MEMBER,,https://github.com/dogsheep/github-to-sqlite/actions/runs/4080058087/jobs/7032116511,207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
516763727,MDExOlB1bGxSZXF1ZXN0MzM1OTgwMjQ2,8,"stargazers command, refs #4",9599,simonw,closed,0,,,,,5,2019-11-03T00:37:36Z,2020-05-02T20:00:27Z,2020-05-02T20:00:26Z,MEMBER,dogsheep/github-to-sqlite/pulls/8,Needs tests. Refs #4.,207052882,github-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
516769276,MDU6SXNzdWU1MTY3NjkyNzY=,9,Commands do not work without an auth.json file,9599,simonw,closed,0,,,,,0,2019-11-03T01:54:28Z,2019-11-11T05:30:48Z,2019-11-11T05:30:48Z,MEMBER,,"`auth.json` is meant to be optional. If it's not provided, the tool should make heavily rate-limited unauthenticated requests.

```
$ github-to-sqlite repos .data/repos.db simonw
Usage: github-to-sqlite repos [OPTIONS] DB_PATH [USERNAME]
Try ""github-to-sqlite repos --help"" for help.

Error: Invalid value for ""-a"" / ""--auth"": File ""auth.json"" does not exist.
```",207052882,github-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/issues/9/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
504720731,MDU6SXNzdWU1MDQ3MjA3MzE=,1,Add more details on how to request data from google takeout correctly.,1055831,dazzag24,open,0,,,,,0,2019-10-09T15:17:34Z,2019-10-09T15:17:34Z,,NONE,,"The default is to download everything.  This can result in an enormous amount of data when you only really need 2 types of data for now:

- My Activity
- Location History

In addition unless you specify that ""My Activity"" is downloaded in JSON format the default is HTML.  This then causes the 

`google-takeout-to-sqlite my-activity takeout.db takeout.zip`

command to fail as it only contains html files not json files.

Thanks",206649770,google-takeout-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1123393829,I_kwDODFE5qs5C9aEl,10,sqlite3.OperationalError: no such table: main.my_activity,69208826,glxblt14,open,0,,,,,1,2022-02-03T17:59:29Z,2022-03-20T02:38:07Z,,NONE,,"Hello,
When i run the command `google-takeout-to-sqlite my-activity db.db takeout-20220203T174446Z-001.zip`, i get this error :
```
Traceback (most recent call last):
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\runpy.py"", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\runpy.py"", line 87, in _run_code
    exec(code, run_globals)
  File ""C:\Users\julie\AppData\Local\Programs\Python\Python39-32\Scripts\google-takeout-to-sqlite.exe\__main__.py"", line 7, in <module>
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py"", line 1128, in __call__
    return self.main(*args, **kwargs)
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py"", line 1053, in main
    rv = self.invoke(ctx)
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py"", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py"", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py"", line 754, in invoke
    return __callback(*args, **kwargs)
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\google_takeout_to_sqlite\cli.py"", line 31, in my_activity
    utils.save_my_activity(db, zf)
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\google_takeout_to_sqlite\utils.py"", line 19, in save_my_activity
    db[""my_activity""].create_index([""time""])
  File ""c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\sqlite_utils\db.py"", line 629, in create_index
    self.db.conn.execute(sql)
sqlite3.OperationalError: no such table: main.my_activity
```
Thank you for your help
Sorry for my bad English
EDIT: i used the json format",206649770,google-takeout-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/10/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1250287607,PR_kwDODFE5qs44jvRV,11,Update README.md,11887,ashanan,open,0,,,,,0,2022-05-27T03:13:59Z,2022-05-27T03:13:59Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/google-takeout-to-sqlite/pulls/11,Fix typo,206649770,google-takeout-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/11/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1557599877,I_kwDODFE5qs5c1xaF,12,location history changes,14809320,gerardrbentley,open,0,,,,,0,2023-01-26T03:57:25Z,2023-01-26T03:57:25Z,,NONE,,"not sure if each download is unique, but I had to change some things to work with the takeout zip I made 2023-01-25

filename changed from ""Location History.json"" to ""Records.json""

`""timestampMs""` is not present, `""timestamp""` is roughly iso timestamp

```py
def get_timestamp_ms(raw_timestamp):
    try:
        return datetime.datetime.strptime(raw_timestamp, ""%Y-%m-%dT%H:%M:%SZ"").timestamp()
    except ValueError:
        return datetime.datetime.strptime(raw_timestamp, ""%Y-%m-%dT%H:%M:%S.%fZ"").timestamp()

def save_location_history(db, zf):
    location_history = json.load(
        zf.open(""Takeout/Location History/Records.json"")
    )
    db[""location_history""].upsert_all(
        (
            {
                ""id"": id_for_location_history(row),
                ""latitude"": row[""latitudeE7""] / 1e7,
                ""longitude"": row[""longitudeE7""] / 1e7,
                ""accuracy"": row[""accuracy""],
                ""timestampMs"": get_timestamp_ms(row[""timestamp""]),
                ""when"": row[""timestamp""],
            }
            for row in location_history[""locations""]
        ),
        pk=""id"",
    )


def id_for_location_history(row):
    # We want an ID that is unique but can be sorted by in
    # date order - so we use the isoformat date + the first
    # 6 characters of a hash of the JSON
    first_six = hashlib.sha1(
        json.dumps(row, separators=("","", "":""), sort_keys=True).encode(""utf8"")
    ).hexdigest()[:6]
    return ""{}-{}"".format(
        row['timestamp'],
        first_six,
    )
```

example locations from mine

```json
{
    ""latitudeE7"": 427220206,
    ""longitudeE7"": -923423972,
    ""accuracy"": 10,
    ""deviceTag"": -1312429967,
    ""deviceDesignation"": ""PRIMARY"",
    ""timestamp"": ""2019-01-08T23:31:50.867Z""
  }
```

```json
{
    ""latitudeE7"": 427011317,
    ""longitudeE7"": -923448300,
    ""accuracy"": 5,
    ""deviceTag"": -1312429967,
    ""deviceDesignation"": ""PRIMARY"",
    ""timestamp"": ""2019-01-08T23:33:53Z""
  }, 
```",206649770,google-takeout-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/12/reactions"", ""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 2}",,
1884499674,PR_kwDODFE5qs5ZtYMc,13,"use poetry for packages, asdf for versioning, and gh actions for ci",150855,iloveitaly,open,0,,,,,0,2023-09-06T17:59:16Z,2023-09-06T17:59:16Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/google-takeout-to-sqlite/pulls/13,"- build: use poetry for package management, asdf for python version
- build: cleanup poetry config, add keywords, ignore dist
- ci: migrate circleci to gh actions
- fix: dup method definition
",206649770,google-takeout-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/13/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
769376447,MDU6SXNzdWU3NjkzNzY0NDc=,2,killed by oomkiller on large location-history,231498,khimaros,open,0,,,,,2,2020-12-17T00:32:24Z,2020-12-17T00:48:32Z,,NONE,,"memory seems to grow unbounded and is oom-killed after about 20GB memory usage.

this is happening while loading a ~1GB uncompressed location history.",206649770,google-takeout-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
769397742,MDU6SXNzdWU3NjkzOTc3NDI=,3,sqlite-utils error on takeout import,231498,khimaros,open,0,,,,,0,2020-12-17T01:18:48Z,2020-12-17T01:19:04Z,,NONE,,"```
$ google-takeout-to-sqlite my-activity takeout.db /path/to/zip
...
sqlite3.OperationalError: no such table: main.my_activity
```

there is no table create in `utils.py`, unlike other importers such as github-to-sqlite

additionally, this package and hackernews-to-sqlite have conflicting `sqlite-utils` dep with datasette and dogsheep-beta",206649770,google-takeout-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/3/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
778380836,MDU6SXNzdWU3NzgzODA4MzY=,4,Feature Request: Gmail,203343,Btibert3,open,0,,,,,5,2021-01-04T21:31:09Z,2021-03-04T20:54:44Z,,NONE,,"From takeout, I only exported my Gmail account.  Ideally I could parse this into sqlite via this tool.",206649770,google-takeout-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
813880401,MDExOlB1bGxSZXF1ZXN0NTc3OTUzNzI3,5,WIP: Add Gmail takeout mbox import,306240,UtahDave,open,0,,,,,25,2021-02-22T21:30:40Z,2021-07-28T07:18:56Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/google-takeout-to-sqlite/pulls/5,"WIP

This PR adds the ability to import emails from a Gmail mbox export from Google Takeout.

This is my first PR to a datasette/dogsheep repo. I've tested this on my personal Google Takeout mbox with ~520,000 emails going back to 2004. This took around ~20 minutes to process.

To provide some feedback on the progress of the import I added the ""rich"" python module. I'm happy to remove that if adding a dependency is discouraged. However, I think it makes a nice addition to give feedback on the progress of a long import.

Do we want to log emails that have errors when trying to import them?

Dealing with encodings with emails is a bit tricky. I'm very open to feedback on how to deal with those better. As well as any other feedback for improvements.",206649770,google-takeout-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
821841046,MDU6SXNzdWU4MjE4NDEwNDY=,6,Upgrade to latest sqlite-utils,9599,simonw,open,0,,,,,1,2021-03-04T07:21:54Z,2021-03-04T07:22:51Z,,MEMBER,,This is pinned to v1 at the moment.,206649770,google-takeout-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
930946817,MDU6SXNzdWU5MzA5NDY4MTc=,7,KeyError: 'accuracy' when processing Location History,403152,davidwilemski,open,0,,,,,0,2021-06-27T14:39:43Z,2021-06-27T14:39:43Z,,NONE,,"I'm new to both the dogsheep tools and datasette but have been experimenting a bit the last few days and these are really cool tools!

I encountered a problem running my Google location history through this tool running the latest release in a docker container:

```
Traceback (most recent call last):
  File ""/usr/local/bin/google-takeout-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/cli.py"", line 49, in my_activity
    utils.save_location_history(db, zf)
  File ""/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py"", line 27, in save_location_history
    db[""location_history""].upsert_all(
  File ""/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1105, in upsert_all
    return self.insert_all(
  File ""/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py"", line 990, in insert_all
    chunk = list(chunk)
  File ""/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py"", line 33, in <genexpr>
    ""accuracy"": row[""accuracy""],
KeyError: 'accuracy'
```

It looks like the tool assumes the `accuracy` key will be in every location history entry. 

My first attempt at a local patch to get myself going was to convert accessing the `accuracy` key to a `.get` instead to hopefully make the row nullable but I wasn't quite sure what `sqlite_utils` would do there. That did work in that the import happened and so I was going to propose a patch that made that change but in updating the existing test to include an entry with a missing accuracy entry, I noticed the expected type of the field appeared to be changing to a string in the test (and from a quick scan through the sqlite_utils code, probably TEXT in the database). Given this change in column type, it seemed that opening an issue first before proposing a fix seemed warranted. It seems the schema would need to be explicitly specified if you wanted a nullable integer column.

Now that I've done a successful import run using my initial fix of calling `.get` on the row dict, I can see with datasette that I only have 7 data points (out of ~250k) that have a null accuracy column. They are all from 2011-2012 in an import that includes points spanning ~2010-2016 so perhaps another approach might be to filter those entries out during import if it really is that infrequent?

I'm happy to provide a PR for a fix but figured I'd ask about which direction is preferred first.",206649770,google-takeout-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/7/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
954546309,MDExOlB1bGxSZXF1ZXN0Njk4NDIzNjY3,8,Add Gmail takeout mbox import (v2),28565,maxhawkins,open,0,,,,,7,2021-07-28T07:05:32Z,2023-09-08T01:22:49Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/google-takeout-to-sqlite/pulls/8,"WIP

This PR builds on #5 to continue implementing gmail import support.

Building on @UtahDave's work, these commits add a few performance and bug fixes:

* Decreased memory overhead for import by manually parsing mbox headers.
* Fixed error where some messages in the mbox would yield a row with NULL in all columns.

I will send more commits to fix any errors I encounter as I run the importer on my personal takeout data.",206649770,google-takeout-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1046887492,PR_kwDODFE5qs4uMsMJ,9,Removed space from filename My Activity.json,91880982,widadmogral,open,0,,,,,0,2021-11-08T00:04:31Z,2021-11-08T00:04:31Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/google-takeout-to-sqlite/pulls/9,"File name from google takeout has no space. The code only runs without error if filename is ""MyActivity.json"" and not ""My Activity.json"". Is it a new change by Google?",206649770,google-takeout-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/9/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
585526292,MDU6SXNzdWU1ODU1MjYyOTI=,1,Set up full text search,9599,simonw,closed,0,,,,,1,2020-03-21T15:57:35Z,2020-03-21T19:47:46Z,2020-03-21T19:45:52Z,MEMBER,,"Should run against `title` and `text` in `items`, and `about` and `id` in `users`.",248903544,hacker-news-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
952179830,MDU6SXNzdWU5NTIxNzk4MzA=,2,Command for fetching Hacker News threads from the search API,9599,simonw,open,0,,,,,4,2021-07-25T02:00:45Z,2021-07-25T03:12:57Z,,MEMBER,,"I want to be able to fetch every item for a domain, e.g. https://news.ycombinator.com/from?site=simonwillison.net",248903544,hacker-news-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
952189173,MDU6SXNzdWU5NTIxODkxNzM=,3,Use HN algolia endpoint to retrieve trees,9599,simonw,open,0,,,,,3,2021-07-25T03:35:27Z,2021-07-25T18:41:17Z,,MEMBER,,"The `trees` command currently has to make a request for every single comment. Algolia have an endpoint that bundles the entire thread together into a single request.

`https://hn.algolia.com/api/v1/items/ID`

Here's an example that loads quickly, with about 50 comments: https://hn.algolia.com/api/v1/items/27941108

It doesn't appear to use pagination at all - if a thread is big then the response is big.

I ran this search to find some stories with more than 1000 comments: https://hn.algolia.com/api/v1/search?tags=story&numericFilters=num_comments%3E=1000

Here's one: https://news.ycombinator.com/item?id=25015967 with 4759 comments. Hitting the API takes 41s and returns 3.7 MB of JSON!
```
wget 'https://hn.algolia.com/api/v1/items/25015967'  0.03s user 0.04s system 0% cpu 41.368 total
/tmp % ls -lah 25015967 
-rw-r--r--  1 simon  wheel   3.7M Jul 24 20:31 25015967
```",248903544,hacker-news-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1205867842,I_kwDODtX3eM5H4BVC,4,Retrieve the top-level story for a comment,1755789,telotortium,open,0,,,,,0,2022-04-15T20:25:39Z,2022-04-15T20:25:39Z,,NONE,,"I think that each comment inserted into the database should include a column `onstory` that contains the ID of the story on which the comment was made. This is exactly equivalent to the link after ""on:"" at the top of an HN comment page ([example](https://news.ycombinator.com/item?id=18358028)). We could do this either by directly retrieving the HTML page and using Beautiful Soup to find that link, or alternatively recurse up the tree in the Firebase API using the `parent` field (probably using `functools.lru_cache` in case a person has commented a bunch of times on the same story).",248903544,hacker-news-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1353418822,PR_kwDODtX3eM497MOV,5,The program fails when the user has no submissions,2467,fernand0,open,0,,,,,0,2022-08-28T17:25:45Z,2022-08-28T17:25:45Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/hacker-news-to-sqlite/pulls/5,"Tested with:
    
     hacker-news-to-sqlite user hacker-news.db fernand0

Result:
`
Traceback (most recent call last):
  File ""/home/ftricas/.pyenv/versions/3.10.6/bin/hacker-news-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py"", line 1130, in __call__
    return self.main(*args, **kwargs)
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py"", line 1055, in main
    rv = self.invoke(ctx)
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py"", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py"", line 760, in invoke
    return __callback(*args, **kwargs)
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/hacker_news_to_sqlite/cli.py"", line 27, in user
    submitted = user.pop(""submitted"", None) or []
AttributeError: 'NoneType' object has no attribute 'pop'
`

There is a problem of style with the patch (but not sure what to do) because with the new inicialization ( submitted = []) the part 

     or []

is not needed. Maybe there is a more adequate way of doing this.",248903544,hacker-news-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1641117021,PR_kwDODtX3eM5M66op,6,Add permalink virtual field to items table,1231935,xavdid,open,0,,,,,1,2023-03-26T22:22:38Z,2023-03-29T18:38:52Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/hacker-news-to-sqlite/pulls/6,"I added a virtual column (no storage overhead) to the output that easily links back to the source. It works nicely out of the box with datasette:

![](https://cdn.zappy.app/faf43661d539ee0fee02c0421de22d65.png)

I got bit a bit by https://github.com/simonw/sqlite-utils/issues/411, so I went with a manual `table_xinfo` and creating the table via execute. Happy to adjust if that issue moves, but this seems like it works.

I also added my best-guess instructions for local development on this package. I'm shooting in the dark, so feel free to replace with how you work on it locally.",248903544,hacker-news-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
470637068,MDU6SXNzdWU0NzA2MzcwNjg=,1,Use XML Analyser to figure out the structure of the export XML,9599,simonw,closed,0,,,,,1,2019-07-20T05:19:02Z,2019-07-20T05:20:09Z,2019-07-20T05:20:09Z,MEMBER,,https://github.com/simonw/xml_analyser,197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
519038979,MDU6SXNzdWU1MTkwMzg5Nzk=,10,Failed to import workout points,9599,simonw,closed,0,,,,,4,2019-11-07T04:50:22Z,2019-11-08T01:18:37Z,2019-11-08T01:18:37Z,MEMBER,,"I just ran the script and it failed to import any `workout_points`, though it did import `workouts`.",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
723838331,MDU6SXNzdWU3MjM4MzgzMzE=,11,export.xml file name varies with different language settings,572,jarib,closed,0,,,,,7,2020-10-17T20:07:18Z,2020-10-17T21:39:15Z,2020-10-17T21:14:10Z,NONE,,"The XML file exported from my phone has a Norwegian file name  – `eksport.xml` 🙄  

I can work around this by unpacking the zip and using `--xml`, but then I lose the workout points. 

Perhaps this could be solved by `--localized-xml eksport.xml`? Alternatively just fall back to the first XML file in the root folder of the zip.

",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
727848625,MDU6SXNzdWU3Mjc4NDg2MjU=,12,"Some workout columns should be float, not text",9599,simonw,open,0,,,,,4,2020-10-23T02:47:02Z,2022-06-23T04:35:02Z,,MEMBER,,"Columns `duration`, `totalDistance` and `totalEnergyBurned` should be converted to float.

https://github.com/dogsheep/healthkit-to-sqlite/blob/71e36e1cf034b96de2a8e6652265d782d3fdf63b/healthkit_to_sqlite/utils.py#L50-L57",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
743071410,MDExOlB1bGxSZXF1ZXN0NTIxMDU0NjEy,13,SQLite does not have case sensitive columns,1689944,tomaskrehlik,open,0,,,,,1,2020-11-14T20:12:32Z,2021-08-24T13:28:26Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/healthkit-to-sqlite/pulls/13,"This solves a weird issue when there is record with metadata key
that is only different in letter cases.

See the test for details.",197882382,healthkit-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/13/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
771608692,MDU6SXNzdWU3NzE2MDg2OTI=,14,UNIQUE constraint failed: workouts.id,1234956,n8henrie,open,0,,,,,5,2020-12-20T15:11:20Z,2023-07-10T14:46:52Z,,NONE,,"I'm getting an error on my initial attempt to import data:

```console
$ healthkit-to-sqlite 20201119\ healthkit\ export.zip healthkit.db
Importing from HealthKit  [###################################-]   98%  00:00:01
Traceback (most recent call last):
  File ""venv/bin/healthkit-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""venv/lib/python3.9/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""venv/lib/python3.9/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""venv/lib/python3.9/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""venv/lib/python3.9/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""venv/lib/python3.9/site-packages/healthkit_to_sqlite/cli.py"", line 57, in cli
    convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)
  File ""venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py"", line 34, in convert_xml_to_sqlite
    workout_to_db(el, db, zipfile)
  File ""venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py"", line 57, in workout_to_db
    pk = db[""workouts""].insert(record, alter=True, hash_id=""id"").last_pk
  File ""venv/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1660, in insert
    return self.insert_all(
  File ""venv/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1778, in insert_all
    self.insert_chunk(
  File ""venv/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1588, in insert_chunk
    result = self.db.execute(query, params)
  File ""venv/lib/python3.9/site-packages/sqlite_utils/db.py"", line 213, in execute
    return self.conn.execute(sql, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: workouts.id
```",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
793907673,MDExOlB1bGxSZXF1ZXN0NTYxNTEyNTAz,15,added try / except to write_records ,9857779,ryancheley,open,0,,,,,0,2021-01-26T03:56:21Z,2021-01-26T03:56:21Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/healthkit-to-sqlite/pulls/15,"to keep the data write from failing if it came across an error during processing. In particular when trying to convert my HealthKit zip file (and that of my wife's) it would consistently error out with the following:

```
db.py 1709 insert_chunk
result = self.db.execute(query, params)

db.py 226 execute
return self.conn.execute(sql, parameters)

sqlite3.OperationalError:
too many SQL variables

---------------------------------------------------------------------------------------------------------------------------------------------------------------------
db.py 1709 insert_chunk
result = self.db.execute(query, params)

db.py 226 execute
return self.conn.execute(sql, parameters)

sqlite3.OperationalError:
too many SQL variables

---------------------------------------------------------------------------------------------------------------------------------------------------------------------
db.py 1709 insert_chunk
result = self.db.execute(query, params)

db.py 226 execute
return self.conn.execute(sql, parameters)

sqlite3.OperationalError:
table rBodyMass has no column named metadata_HKWasUserEntered

---------------------------------------------------------------------------------------------------------------------------------------------------------------------
healthkit-to-sqlite 8 <module>
sys.exit(cli())

core.py 829 __call__
return self.main(*args, **kwargs)

core.py 782 main
rv = self.invoke(ctx)

core.py 1066 invoke
return ctx.invoke(self.callback, **ctx.params)

core.py 610 invoke
return callback(*args, **kwargs)

cli.py 57 cli
convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)

utils.py 42 convert_xml_to_sqlite
write_records(records, db)

utils.py 143 write_records
db[table].insert_all(

db.py 1899 insert_all
self.insert_chunk(

db.py 1720 insert_chunk
self.insert_chunk(

db.py 1720 insert_chunk
self.insert_chunk(

db.py 1714 insert_chunk
result = self.db.execute(query, params)

db.py 226 execute
return self.conn.execute(sql, parameters)

sqlite3.OperationalError:
table rBodyMass has no column named metadata_HKWasUserEntered
```

Adding the try / except in the `write_records` seems to fix that issue. ",197882382,healthkit-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/15/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
830901133,MDExOlB1bGxSZXF1ZXN0NTkyMzY0MjU1,16,"Add a fallback ID, print if no ID found",1234956,n8henrie,open,0,,,,,0,2021-03-13T13:38:29Z,2021-03-13T14:44:04Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/healthkit-to-sqlite/pulls/16,"Fixes https://github.com/dogsheep/healthkit-to-sqlite/issues/14
",197882382,healthkit-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/16/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
836063389,MDU6SXNzdWU4MzYwNjMzODk=,17,Datetime columns are not properly formatted to be recognizes as datetime,1234956,n8henrie,open,0,,,,,0,2021-03-19T14:33:04Z,2021-03-19T14:33:04Z,,NONE,,"

Currently, the datetimes are formatted in a way that is not recognized by datasette-vega for plotting with a `Date/time` type for the axis. 

For example, if you have datasette running locally with `datasette-vega` installed and have a database that includes resting heart rate:

```
http://localhost:8001/healthkit/rRestingHeartRate#g.mark=line&g.x_column=startDate&g.x_type=temporal&g.y_column=value&g.y_type=quantitative
```

The plot is blank unless you choose `Label` as the type for the date data.

The `startDate` (and `creationDate` and `endDate`) columns appear like: `2019-11-14 18:22:18 -0700`

If instead the format for this column is changed slightly: `2019-11-14T18:22:18-07:00` they are recognized as proper dates and the charting works as expected.

I have a PR that addresses this issue, will submit shortly.",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/17/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
836064851,MDExOlB1bGxSZXF1ZXN0NTk2NjI3Nzgw,18,Add datetime parsing,1234956,n8henrie,open,0,,,,,0,2021-03-19T14:34:22Z,2021-03-19T14:34:22Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/healthkit-to-sqlite/pulls/18,"Parses the datetime columns so they are subsequently properly recognized as
datetime.

Fixes https://github.com/dogsheep/healthkit-to-sqlite/issues/17
",197882382,healthkit-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/18/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
975158266,MDU6SXNzdWU5NzUxNTgyNjY=,19,table activity_summary has no column named appleMoveTime,9599,simonw,closed,0,,,,,0,2021-08-20T00:46:44Z,2021-08-20T00:54:34Z,2021-08-20T00:54:34Z,MEMBER,,"Got this error today against a fresh export:

    table activity_summary has no column named appleMoveTime
",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/19/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
470637152,MDU6SXNzdWU0NzA2MzcxNTI=,2,Import workouts,9599,simonw,closed,0,,,,,1,2019-07-20T05:20:21Z,2019-07-20T06:21:41Z,2019-07-20T06:21:41Z,MEMBER,,From #1,197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
975166271,MDU6SXNzdWU5NzUxNjYyNzE=,20,Add index on workout_points.date,9599,simonw,open,0,,,,,2,2021-08-20T01:08:04Z,2021-08-20T01:12:48Z,,MEMBER,,"Sorting that by date makes sense for seeing most recent points, and my DB has 2.5m points in so it's an expensive sort!",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
977128935,MDU6SXNzdWU5NzcxMjg5MzU=,21,Duplicate Column,32016596,FabianHertwig,open,0,,,,,1,2021-08-23T15:00:44Z,2021-08-23T17:00:59Z,,NONE,,"Hey, thank you for this repo!

When I try to convert my export, I get a multiple column error. Here is the stack trace:

```sh
(.venv) (base) computer:bodyweight_app user$ healthkit-to-sqlite ./data/Health_export.zip ./data/healthkit.db
Importing from HealthKit  [###############################-----]   87%  00:00:22
Traceback (most recent call last):
  File ""/MyProject/.venv/bin/healthkit-to-sqlite"", line 10, in <module>
    sys.exit(cli())
  File ""/MyProject/.venv/lib/python3.7/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/MyProject/.venv/lib/python3.7/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/MyProject/.venv/lib/python3.7/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/MyProject/.venv/lib/python3.7/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/cli.py"", line 57, in cli
    convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)
  File ""/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py"", line 41, in convert_xml_to_sqlite
    write_records(records, db)
  File ""/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py"", line 146, in write_records
    batch_size=50,
  File ""/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py"", line 2579, in insert_all
    extracts=extracts,
  File ""/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1246, in create
    extracts=extracts,
  File ""/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py"", line 767, in create_table
    self.execute(sql)
  File ""/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py"", line 421, in execute
    return self.conn.execute(sql)
sqlite3.OperationalError: duplicate column name: metadata_Meal
```",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/21/reactions"", ""total_count"": 5, ""+1"": 5, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
978086284,MDExOlB1bGxSZXF1ZXN0NzE4NzM0MTkx,22,Make sure that case-insensitive column names are unique,32016596,FabianHertwig,open,0,,,,,1,2021-08-24T13:13:38Z,2021-08-24T13:26:20Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/healthkit-to-sqlite/pulls/22,"This closes #21.

When there are metadata entries with the same case insensitive string, then there is an error when trying to create a new column for that metadata entry in the database table, because a column with that case insensitive name already exists.

```xml
 <Record type=""HKQuantityTypeIdentifierDietaryCholesterol"" sourceName=""MyFitnessPal"" sourceVersion=""35120"" unit=""mg"" creationDate=""2021-07-04 20:55:27 +0200"" startDate=""2021-07-04 20:55:00 +0200"" endDate=""2021-07-04 20:55:00 +0200"" value=""124"">
  <MetadataEntry key=""meal"" value=""Dinner""/>
  <MetadataEntry key=""Meal"" value=""Dinner""/>
 </Record>
```

The code added in this PR checks if a key already exists in a record and if so adds a number at its end. The resulting column names look like the example below then. Interestingly, the column names viewed with Datasette are not case insensitive.

```text
startDate, endDate, value, unit, sourceName, sourceVersion, creationDate, metadata_meal, metadata_Meal_2, metadata_Mahlzeit
```
",197882382,healthkit-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/22/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1515717718,PR_kwDOC8tyDs5Gc-VH,23,Include workout statistics,2129,badboy,open,0,,,,,0,2023-01-01T17:29:57Z,2023-01-01T17:29:57Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/healthkit-to-sqlite/pulls/23,"Not sure when this changed (iOS 16 maybe?), but the `WorkoutStatistics` now has a whole bunch of information about workouts, e.g. for runs it contains the distance (as a `<WorkoutStatistics type=""HKQuantityTypeIdentifierDistanceWalkingRunning ...>` element).

Adding it as another column at leat allows me to pull these out (using SQLite's JSON support).
I'm running with this patch on my own data now.",197882382,healthkit-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/23/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1515883470,I_kwDOC8tyDs5aWovO,24,DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 ,6231413,mmngreco,open,0,,,,,2,2023-01-01T23:00:38Z,2023-03-30T10:17:31Z,,NONE,,"Hi @simonw 

I hope you find this issue ok, the idea is provide some documentation to other users like me about how to solve this problem and save some time.

Following the instructions from the `README.md` I've faced this error:

```bash
(venv) mgreco@pop-os apple-health master* (23:44|0s)
$ healthkit-to-sqlite apple_health_export/export.xml healthkit.db --xml
Importing from HealthKit  [------------------------------------]    0%
Traceback (most recent call last):
  File ""/home/mgreco/github/mmngreco/apple-health/venv/bin/healthkit-to-sqlite"", line 33, in <module>
    sys.exit(load_entry_point('healthkit-to-sqlite', 'console_scripts', 'healthkit-to-sqlite')())
  File ""/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py"", line 1130, in __call__
    return self.main(*args, **kwargs)
  File ""/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py"", line 1055, in main
    rv = self.invoke(ctx)
  File ""/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/site-packages/click/core.py"", line 760, in invoke
    return __callback(*args, **kwargs)
  File ""/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/cli.py"", line 57, in cli
    convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)
  File ""/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/utils.py"", line 25, in convert_xml_to_sqlite
    for tag, el in find_all_tags(
  File ""/home/mgreco/github/mmngreco/apple-health/.deps/healthkit-to-sqlite/healthkit_to_sqlite/utils.py"", line 12, in find_all_tags
    for event, el in parser.read_events():
  File ""/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/xml/etree/ElementTree.py"", line 1324, in read_events
    raise event
  File ""/home/mgreco/github/mmngreco/apple-health/venv/lib/python3.10/xml/etree/ElementTree.py"", line 1296, in feed
    self._parser.feed(data)
xml.etree.ElementTree.ParseError: syntax error: line 156, column 0
```

So, after debugging and searching on internet I found this useful link: https://discussions.apple.com/thread/254202523 (etresoft, the real hero). Which basically says that the xml given by the health app (healthkit version 12) has some bugs but  fortunately, they can be solved with a couple of commads:

1. Uncompress the zip and move the new folder where `export.xml` is.
1. Create a `patch.txt` with the following content

    ```diff
    --- export.xml	2022-09-18 15:17:09.000000000 -0400
    +++ export-fixed.xml	2022-09-18 16:37:08.000000000 -0400
    @@ -15,6 +15,7 @@
       HKCharacteristicTypeIdentifierBiologicalSex       CDATA #REQUIRED
       HKCharacteristicTypeIdentifierBloodType           CDATA #REQUIRED
       HKCharacteristicTypeIdentifierFitzpatrickSkinType CDATA #REQUIRED
    +  HKCharacteristicTypeIdentifierCardioFitnessMedicationsUse CDATA #IMPLIED
     >
     <!ELEMENT Record ((MetadataEntry|HeartRateVariabilityMetadataList)*)>
     <!ATTLIST Record
    @@ -39,7 +40,7 @@
       startDate     CDATA #REQUIRED
       endDate       CDATA #REQUIRED
     >
    -<!ELEMENT Workout ((MetadataEntry|WorkoutEvent|WorkoutRoute)*)>
    +<!ELEMENT Workout ((MetadataEntry|WorkoutEvent|WorkoutRoute|WorkoutStatistics)*)>
     <!ATTLIST Workout
       workoutActivityType   CDATA #REQUIRED
       duration              CDATA #IMPLIED
    @@ -63,7 +64,7 @@
       duration             CDATA #IMPLIED
       durationUnit         CDATA #IMPLIED
     >
    -<!ELEMENT WorkoutEvent EMPTY>
    +<!ELEMENT WorkoutEvent (MetadataEntry?)>
     <!ATTLIST WorkoutEvent
       type                 CDATA #REQUIRED
       date                 CDATA #REQUIRED
    @@ -79,6 +80,7 @@
       minimum              CDATA #IMPLIED
       maximum              CDATA #IMPLIED
       sum                  CDATA #IMPLIED
    +  unit                 CDATA #IMPLIED
     >
     <!ELEMENT WorkoutRoute ((MetadataEntry|FileReference)*)>
     <!ATTLIST WorkoutRoute
    @@ -153,6 +155,7 @@
       dateIssued       CDATA #REQUIRED
       expirationDate   CDATA #REQUIRED
       brand            CDATA #IMPLIED
    +>
     <!ELEMENT RightEye EMPTY>
     <!ATTLIST RightEye
       sphere           CDATA #IMPLIED
    @@ -203,13 +206,6 @@
       diameter         CDATA #IMPLIED
       diameterUnit     CDATA #IMPLIED
     >
    -  device           CDATA #IMPLIED
    -<!ELEMENT MetadataEntry EMPTY>
    -<!ATTLIST MetadataEntry
    -  key              CDATA #IMPLIED
    -  value            CDATA #IMPLIED
    ->
    ->
     ]>
     <HealthData>
      <ExportDate/>
    ```
1. Apply the path with the command: `patch < patch.txt`
1. Fix endDates with the command `sed 's/startDate/endDate/2' export.xml > export-fixed.xml`
1. Try again `healthkit-to-sqlite export-fixed.xml healthkit.db --xml`",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
470637206,MDU6SXNzdWU0NzA2MzcyMDY=,3,Import ActivitySummary,9599,simonw,closed,0,,,,,0,2019-07-20T05:21:00Z,2019-07-20T05:58:07Z,2019-07-20T05:58:07Z,MEMBER,,"From #1

```python
  'ActivitySummary': {'attr_counts': {'activeEnergyBurned': 980,
                                     'activeEnergyBurnedGoal': 980,
                                     'activeEnergyBurnedUnit': 980,
                                     'appleExerciseTime': 980,
                                     'appleExerciseTimeGoal': 980,
                                     'appleStandHours': 980,
                                     'appleStandHoursGoal': 980,
                                     'dateComponents': 980},
                     'child_counts': {},
                     'count': 980,
                     'parent_counts': {'HealthData': 980}},
```",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/3/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
470640505,MDU6SXNzdWU0NzA2NDA1MDU=,4,Import Records,9599,simonw,closed,0,,,,,1,2019-07-20T06:11:20Z,2019-07-20T06:21:41Z,2019-07-20T06:21:41Z,MEMBER,,"From #1:
```python
 'Record': {'attr_counts': {'creationDate': 2672233,
                            'device': 2665111,
                            'endDate': 2672233,
                            'sourceName': 2672233,
                            'sourceVersion': 2671779,
                            'startDate': 2672233,
                            'type': 2672233,
                            'unit': 2650012,
                            'value': 2672232},
            'child_counts': {'HeartRateVariabilityMetadataList': 2318,
                             'MetadataEntry': 287974},
            'count': 2672233,
            'parent_counts': {'Correlation': 2, 'HealthData': 2672231}},
```",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
470691622,MDU6SXNzdWU0NzA2OTE2MjI=,5,Add progress bar,9599,simonw,closed,0,,,,,2,2019-07-20T16:29:07Z,2019-07-22T03:30:13Z,2019-07-22T02:49:22Z,MEMBER,,"Showing a progress bar would be nice, using Click.

The easiest way to do this would probably be be to hook it up to the length of the compressed content, and update it as this code pushes more XML bytes through the parser:

https://github.com/dogsheep/healthkit-to-sqlite/blob/d64299765064501f4efdd9a0b21dbdba9ec4287f/healthkit_to_sqlite/utils.py#L6-L10",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
470856782,MDU6SXNzdWU0NzA4NTY3ODI=,6,Break up records into different tables for each type,9599,simonw,closed,0,,,,,1,2019-07-22T01:54:59Z,2019-07-22T03:28:55Z,2019-07-22T03:28:50Z,MEMBER,,"I don't think there's much benefit to having all of the different record types stored in the same enormous table. Here's what I get when I use `_facet=type`:

<img width=""358"" alt=""hello2__records__2_672_233_rows"" src=""https://user-images.githubusercontent.com/9599/61601118-e2f54d00-abe8-11e9-8bf6-3df2ef969112.png"">

I'm going to try splitting these up into separate tables - so `HKQuantityTypeIdentifierBodyMassIndex` becomes a table called `rBodyMassIndex` - and see if that's nicer to work with.",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
472097220,MDU6SXNzdWU0NzIwOTcyMjA=,7,Script uses a lot of RAM,9599,simonw,closed,0,,,,,3,2019-07-24T06:11:11Z,2019-07-24T06:35:52Z,2019-07-24T06:35:52Z,MEMBER,,"I'm using an XML pull parser which should avoid the need to slurp the whole XML file into memory, but it's not working - the script still uses over 1GB of RAM when it runs according to Activity Monitor.

I think this is because I'm still causing the full root element to be incrementally loaded into memory just in case I try and access it later.

http://effbot.org/elementtree/iterparse.htm says I should use `elem.clear()` as I go. It also says:

> The above pattern has one drawback; it does not clear the root element, so you will end up with a single element with lots of empty child elements. If your files are huge, rather than just large, this might be a problem. To work around this, you need to get your hands on the root element.

So I will try that recipe and see if it helps.",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
472104705,MDExOlB1bGxSZXF1ZXN0MzAwNTgwMjIx,8,Use less RAM,9599,simonw,closed,0,,,,,0,2019-07-24T06:35:01Z,2019-07-24T06:35:52Z,2019-07-24T06:35:52Z,MEMBER,dogsheep/healthkit-to-sqlite/pulls/8,Closes #7,197882382,healthkit-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
472429048,MDU6SXNzdWU0NzI0MjkwNDg=,9,Too many SQL variables,166463,tholo,closed,0,,,,,4,2019-07-24T18:24:17Z,2019-07-26T10:01:05Z,2019-07-26T10:01:05Z,NONE,,"Decided to try importing my data, and ran into this:

```
Traceback (most recent call last):
  File ""/Users/tholo/Source/health/bin/healthkit-to-sqlite"", line 10, in <module>
    sys.exit(cli())
  File ""/Users/tholo/Source/health/lib/python3.7/site-packages/click/core.py"", line 764, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/tholo/Source/health/lib/python3.7/site-packages/click/core.py"", line 717, in main
    rv = self.invoke(ctx)
  File ""/Users/tholo/Source/health/lib/python3.7/site-packages/click/core.py"", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/tholo/Source/health/lib/python3.7/site-packages/click/core.py"", line 555, in invoke
    return callback(*args, **kwargs)
  File ""/Users/tholo/Source/health/lib/python3.7/site-packages/healthkit_to_sqlite/cli.py"", line 50, in cli
    convert_xml_to_sqlite(fp, db, progress_callback=bar.update)
  File ""/Users/tholo/Source/health/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py"", line 41, in convert_xml_to_sqlite
    write_records(records, db)
  File ""/Users/tholo/Source/health/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py"", line 80, in write_records
    column_order=[""startDate"", ""endDate"", ""value"", ""unit""],
  File ""/Users/tholo/Source/health/lib/python3.7/site-packages/sqlite_utils/db.py"", line 911, in insert_all
    result = self.db.conn.execute(sql, values)
sqlite3.OperationalError: too many SQL variables
```

Added some debug output in sqlite_utils/db.py, which resulted in:

```
                INSERT INTO [rBodyMassIndex] ([creationDate], [endDate], [metadata_HKWasUserEntered], [metadata_Health Mate App Version], [metadata_Modified Date], [metadata_Withings Link], [metadata_Withings User Identifier], [sourceName], [sourceVersion], [startDate], [unit], [value]) VALUES
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ,
                    (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                ;
```

with the attached data:

```
['2019-06-27 22:55:10 -0700', '2011-06-22 21:05:53 -0700', '0', '4.4.2', '2011-06-23 04:05:53 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1308801953&type=1', '301293', 'Health Mate', '4040200', '2011-06-22 21:05:53 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2011-06-23 09:36:27 -0700', '0', '4.4.2', '2011-06-23 16:36:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1308846987&type=1', '301293', 'Health Mate', '4040200', '2011-06-23 09:36:27 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2011-06-23 23:54:07 -0700', '0', '4.4.2', '2011-06-24 06:55:19 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1308898447&type=1', '301293', 'Health Mate', '4040200', '2011-06-23 23:54:07 -0700', 'count', '30.679', '2019-06-27 22:55:10 -0700', '2011-06-24 09:13:40 -0700', '0', '4.4.2', '2011-06-24 16:14:35 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1308932020&type=1', '301293', 'Health Mate', '4040200', '2011-06-24 09:13:40 -0700', 'count', '30.3549', '2019-06-27 22:55:10 -0700', '2011-06-25 08:30:08 -0700', '0', '4.4.2', '2011-06-25 15:30:49 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309015808&type=1', '301293', 'Health Mate', '4040200', '2011-06-25 08:30:08 -0700', 'count', '30.3395', '2019-06-27 22:55:10 -0700', '2011-06-26 07:47:51 -0700', '0', '4.4.2', '2011-06-26 14:48:27 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309099671&type=1', '301293', 'Health Mate', '4040200', '2011-06-26 07:47:51 -0700', 'count', '30.2315', '2019-06-27 22:55:10 -0700', '2011-06-28 08:48:26 -0700', '0', '4.4.2', '2011-06-28 15:49:13 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309276106&type=1', '301293', 'Health Mate', '4040200', '2011-06-28 08:48:26 -0700', 'count', '30.0617', '2019-06-27 22:55:10 -0700', '2011-06-29 09:21:16 -0700', '0', '4.4.2', '2011-06-29 16:21:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309364476&type=1', '301293', 'Health Mate', '4040200', '2011-06-29 09:21:16 -0700', 'count', '29.9537', '2019-06-27 22:55:10 -0700', '2011-06-30 08:41:46 -0700', '0', '4.4.2', '2011-06-30 15:42:30 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309448506&type=1', '301293', 'Health Mate', '4040200', '2011-06-30 08:41:46 -0700', 'count', '29.8302', '2019-06-27 22:55:10 -0700', '2011-07-01 09:05:28 -0700', '0', '4.4.2', '2011-07-01 16:06:24 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309536328&type=1', '301293', 'Health Mate', '4040200', '2011-07-01 09:05:28 -0700', 'count', '29.8611', '2019-06-27 22:55:10 -0700', '2011-07-02 08:58:50 -0700', '0', '4.4.2', '2011-07-02 15:59:40 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309622330&type=1', '301293', 'Health Mate', '4040200', '2011-07-02 08:58:50 -0700', 'count', '29.8765', '2019-06-27 22:55:10 -0700', '2011-07-04 09:33:43 -0700', '0', '4.4.2', '2011-07-04 16:34:19 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309797223&type=1', '301293', 'Health Mate', '4040200', '2011-07-04 09:33:43 -0700', 'count', '30.0309', '2019-06-27 22:55:10 -0700', '2011-07-06 09:40:23 -0700', '0', '4.4.2', '2011-07-06 16:41:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309970423&type=1', '301293', 'Health Mate', '4040200', '2011-07-06 09:40:23 -0700', 'count', '30.1852', '2019-06-27 22:55:10 -0700', '2011-07-08 08:08:48 -0700', '0', '4.4.2', '2011-07-08 15:09:51 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310137728&type=1', '301293', 'Health Mate', '4040200', '2011-07-08 08:08:48 -0700', 'count', '30.0309', '2019-06-27 22:55:10 -0700', '2011-07-09 08:31:05 -0700', '0', '4.4.2', '2011-07-09 15:31:48 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310225465&type=1', '301293', 'Health Mate', '4040200', '2011-07-09 08:31:05 -0700', 'count', '29.9537', '2019-06-27 22:55:10 -0700', '2011-07-10 08:14:36 -0700', '0', '4.4.2', '2011-07-10 15:15:12 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310310876&type=1', '301293', 'Health Mate', '4040200', '2011-07-10 08:14:36 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2011-07-12 07:55:21 -0700', '0', '4.4.2', '2011-07-12 14:55:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310482521&type=1', '301293', 'Health Mate', '4040200', '2011-07-12 07:55:21 -0700', 'count', '30.108', '2019-06-27 22:55:10 -0700', '2011-07-13 08:48:05 -0700', '0', '4.4.2', '2011-07-13 15:48:42 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310572085&type=1', '301293', 'Health Mate', '4040200', '2011-07-13 08:48:05 -0700', 'count', '30', '2019-06-27 22:55:10 -0700', '2011-07-14 09:05:16 -0700', '0', '4.4.2', '2011-07-14 16:05:57 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310659516&type=1', '301293', 'Health Mate', '4040200', '2011-07-14 09:05:16 -0700', 'count', '29.9074', '2019-06-27 22:55:10 -0700', '2011-07-15 07:09:56 -0700', '0', '4.4.2', '2011-07-15 14:10:35 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310738996&type=1', '301293', 'Health Mate', '4040200', '2011-07-15 07:09:56 -0700', 'count', '29.9537', '2019-06-27 22:55:10 -0700', '2011-07-16 09:26:04 -0700', '0', '4.4.2', '2011-07-16 16:26:44 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310833564&type=1', '301293', 'Health Mate', '4040200', '2011-07-16 09:26:04 -0700', 'count', '29.7531', '2019-06-27 22:55:10 -0700', '2011-07-17 09:52:59 -0700', '0', '4.4.2', '2011-07-17 16:53:38 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310921579&type=1', '301293', 'Health Mate', '4040200', '2011-07-17 09:52:59 -0700', 'count', '29.8765', '2019-06-27 22:55:10 -0700', '2011-07-19 08:56:16 -0700', '0', '4.4.2', '2011-07-19 15:57:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311090976&type=1', '301293', 'Health Mate', '4040200', '2011-07-19 08:56:16 -0700', 'count', '29.7685', '2019-06-27 22:55:10 -0700', '2011-07-21 08:21:20 -0700', '0', '4.4.2', '2011-07-21 15:22:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311261680&type=1', '301293', 'Health Mate', '4040200', '2011-07-21 08:21:20 -0700', 'count', '29.7685', '2019-06-27 22:55:10 -0700', '2011-07-23 08:49:56 -0700', '0', '4.4.2', '2011-07-23 15:50:40 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311436196&type=1', '301293', 'Health Mate', '4040200', '2011-07-23 08:49:56 -0700', 'count', '29.7222', '2019-06-27 22:55:10 -0700', '2011-07-24 09:17:35 -0700', '0', '4.4.2', '2011-07-24 16:18:14 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311524255&type=1', '301293', 'Health Mate', '4040200', '2011-07-24 09:17:35 -0700', 'count', '29.5833', '2019-06-27 22:55:10 -0700', '2011-07-25 07:51:55 -0700', '0', '4.4.2', '2011-07-25 14:52:48 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311605515&type=1', '301293', 'Health Mate', '4040200', '2011-07-25 07:51:55 -0700', 'count', '29.5525', '2019-06-27 22:55:10 -0700', '2011-08-06 10:04:05 -0700', '0', '4.4.2', '2011-08-06 17:04:47 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1312650245&type=1', '301293', 'Health Mate', '4040200', '2011-08-06 10:04:05 -0700', 'count', '29.7377', '2019-06-27 22:55:10 -0700', '2011-08-08 07:52:22 -0700', '0', '4.4.2', '2011-08-08 14:53:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1312815142&type=1', '301293', 'Health Mate', '4040200', '2011-08-08 07:52:22 -0700', 'count', '29.6605', '2019-06-27 22:55:10 -0700', '2011-08-10 07:57:30 -0700', '0', '4.4.2', '2011-08-10 14:58:12 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1312988250&type=1', '301293', 'Health Mate', '4040200', '2011-08-10 07:57:30 -0700', 'count', '29.7531', '2019-06-27 22:55:10 -0700', '2011-08-12 07:51:14 -0700', '0', '4.4.2', '2011-08-12 14:51:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1313160674&type=1', '301293', 'Health Mate', '4040200', '2011-08-12 07:51:14 -0700', 'count', '29.6914', '2019-06-27 22:55:10 -0700', '2011-08-13 07:45:28 -0700', '0', '4.4.2', '2011-08-13 14:46:08 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1313246728&type=1', '301293', 'Health Mate', '4040200', '2011-08-13 07:45:28 -0700', 'count', '29.5833', '2019-06-27 22:55:10 -0700', '2011-08-17 09:06:20 -0700', '0', '4.4.2', '2011-08-17 16:07:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1313597180&type=1', '301293', 'Health Mate', '4040200', '2011-08-17 09:06:20 -0700', 'count', '29.5679', '2019-06-27 22:55:10 -0700', '2011-08-22 08:28:08 -0700', '0', '4.4.2', '2011-08-22 15:28:57 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1314026888&type=1', '301293', 'Health Mate', '4040200', '2011-08-22 08:28:08 -0700', 'count', '29.9846', '2019-06-27 22:55:10 -0700', '2011-08-25 08:59:30 -0700', '0', '4.4.2', '2011-08-25 16:00:15 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1314287970&type=1', '301293', 'Health Mate', '4040200', '2011-08-25 08:59:30 -0700', 'count', '29.9691', '2019-06-27 22:55:10 -0700', '2011-08-30 08:13:59 -0700', '0', '4.4.2', '2011-08-30 15:46:08 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1314717239&type=1', '301293', 'Health Mate', '4040200', '2011-08-30 08:13:59 -0700', 'count', '29.784', '2019-06-27 22:55:10 -0700', '2011-09-12 08:47:51 -0700', '0', '4.4.2', '2011-09-12 15:48:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1315842471&type=1', '301293', 'Health Mate', '4040200', '2011-09-12 08:47:51 -0700', 'count', '29.7377', '2019-06-27 22:55:10 -0700', '2011-09-13 09:17:27 -0700', '0', '4.4.2', '2011-09-13 16:48:30 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1315930647&type=1', '301293', 'Health Mate', '4040200', '2011-09-13 09:17:27 -0700', 'count', '29.7531', '2019-06-27 22:55:10 -0700', '2011-10-01 09:12:20 -0700', '0', '4.4.2', '2011-10-01 16:13:00 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1317485540&type=1', '301293', 'Health Mate', '4040200', '2011-10-01 09:12:20 -0700', 'count', '29.8148', '2019-06-27 22:55:10 -0700', '2011-10-11 11:14:11 -0700', '0', '4.4.2', '2011-10-11 18:15:14 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1318356851&type=1', '301293', 'Health Mate', '4040200', '2011-10-11 11:14:11 -0700', 'count', '29.7377', '2019-06-27 22:55:10 -0700', '2011-10-16 09:29:47 -0700', '0', '4.4.2', '2011-10-16 16:30:39 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1318782587&type=1', '301293', 'Health Mate', '4040200', '2011-10-16 09:29:47 -0700', 'count', '29.6914', '2019-06-27 22:55:10 -0700', '2011-10-19 09:21:44 -0700', '0', '4.4.2', '2011-10-19 16:22:25 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1319041304&type=1', '301293', 'Health Mate', '4040200', '2011-10-19 09:21:44 -0700', 'count', '29.7685', '2019-06-27 22:55:10 -0700', '2011-10-24 07:04:22 -0700', '0', '4.4.2', '2011-10-24 14:05:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1319465062&type=1', '301293', 'Health Mate', '4040200', '2011-10-24 07:04:22 -0700', 'count', '29.5988', '2019-06-27 22:55:10 -0700', '2011-11-07 09:33:17 -0700', '0', '4.4.2', '2011-11-07 16:33:58 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1320683597&type=1', '301293', 'Health Mate', '4040200', '2011-11-07 09:33:17 -0700', 'count', '29.8611', '2019-06-27 22:55:10 -0700', '2011-11-10 07:59:03 -0700', '0', '4.4.2', '2011-11-10 14:59:48 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1320937143&type=1', '301293', 'Health Mate', '4040200', '2011-11-10 07:59:03 -0700', 'count', '29.9383', '2019-06-27 22:55:10 -0700', '2011-11-13 09:28:31 -0700', '0', '4.4.2', '2011-11-13 16:29:20 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1321201711&type=1', '301293', 'Health Mate', '4040200', '2011-11-13 09:28:31 -0700', 'count', '29.7531', '2019-06-27 22:55:10 -0700', '2011-11-21 08:45:06 -0700', '0', '4.4.2', '2011-11-21 15:46:04 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1321890306&type=1', '301293', 'Health Mate', '4040200', '2011-11-21 08:45:06 -0700', 'count', '29.9691', '2019-06-27 22:55:10 -0700', '2011-11-23 09:55:44 -0700', '0', '4.4.2', '2011-11-23 16:56:18 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1322067344&type=1', '301293', 'Health Mate', '4040200', '2011-11-23 09:55:44 -0700', 'count', '29.8302', '2019-06-27 22:55:10 -0700', '2011-11-29 09:50:44 -0700', '0', '4.4.2', '2011-11-29 16:51:31 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1322585444&type=1', '301293', 'Health Mate', '4040200', '2011-11-29 09:50:44 -0700', 'count', '30.1698', '2019-06-27 22:55:10 -0700', '2011-11-30 11:13:21 -0700', '0', '4.4.2', '2011-11-30 18:14:14 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1322676801&type=1', '301293', 'Health Mate', '4040200', '2011-11-30 11:13:21 -0700', 'count', '30.0617', '2019-06-27 22:55:10 -0700', '2011-12-04 10:24:36 -0700', '0', '4.4.2', '2011-12-04 17:25:24 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1323019476&type=1', '301293', 'Health Mate', '4040200', '2011-12-04 10:24:36 -0700', 'count', '29.9691', '2019-06-27 22:55:10 -0700', '2011-12-10 09:22:18 -0700', '0', '4.4.2', '2011-12-10 16:23:07 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1323534138&type=1', '301293', 'Health Mate', '4040200', '2011-12-10 09:22:18 -0700', 'count', '29.9537', '2019-06-27 22:55:10 -0700', '2011-12-26 10:36:42 -0700', '0', '4.4.2', '2011-12-26 17:37:31 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1324921002&type=1', '301293', 'Health Mate', '4040200', '2011-12-26 10:36:42 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2012-01-11 11:24:13 -0700', '0', '4.4.2', '2012-01-11 18:25:04 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1326306253&type=1', '301293', 'Health Mate', '4040200', '2012-01-11 11:24:13 -0700', 'count', '29.8302', '2019-06-27 22:55:10 -0700', '2012-01-15 10:17:09 -0700', '0', '4.4.2', '2012-01-15 17:17:51 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1326647829&type=1', '301293', 'Health Mate', '4040200', '2012-01-15 10:17:09 -0700', 'count', '29.8302', '2019-06-27 22:55:10 -0700', '2012-01-19 09:24:32 -0700', '0', '4.4.2', '2012-01-19 16:25:21 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1326990272&type=1', '301293', 'Health Mate', '4040200', '2012-01-19 09:24:32 -0700', 'count', '29.7994', '2019-06-27 22:55:10 -0700', '2012-01-29 10:26:13 -0700', '0', '4.4.2', '2012-01-29 17:26:52 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1327857973&type=1', '301293', 'Health Mate', '4040200', '2012-01-29 10:26:13 -0700', 'count', '30.0154', '2019-06-27 22:55:10 -0700', '2012-02-03 10:13:28 -0700', '0', '4.4.2', '2012-02-03 17:15:01 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1328289208&type=1', '301293', 'Health Mate', '4040200', '2012-02-03 10:13:28 -0700', 'count', '29.8457', '2019-06-27 22:55:10 -0700', '2012-02-12 09:23:01 -0700', '0', '4.4.2', '2012-02-12 16:23:53 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1329063781&type=1', '301293', 'Health Mate', '4040200', '2012-02-12 09:23:01 -0700', 'count', '30.1235', '2019-06-27 22:55:10 -0700', '2012-03-03 09:26:06 -0700', '0', '4.4.2', '2012-03-03 16:26:54 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1330791966&type=1', '301293', 'Health Mate', '4040200', '2012-03-03 09:26:06 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2012-03-11 11:23:15 -0700', '0', '4.4.2', '2012-03-11 18:24:16 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1331490195&type=1', '301293', 'Health Mate', '4040200', '2012-03-11 11:23:15 -0700', 'count', '30.2161', '2019-06-27 22:55:10 -0700', '2012-03-16 09:39:36 -0700', '0', '4.4.2', '2012-03-16 16:40:20 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1331915976&type=1', '301293', 'Health Mate', '4040200', '2012-03-16 09:39:36 -0700', 'count', '30.2778', '2019-06-27 22:55:10 -0700', '2012-03-21 08:33:07 -0700', '0', '4.4.2', '2012-03-21 15:34:00 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1332343987&type=1', '301293', 'Health Mate', '4040200', '2012-03-21 08:33:07 -0700', 'count', '30.1389', '2019-06-27 22:55:10 -0700', '2012-04-11 08:49:34 -0700', '0', '4.4.2', '2012-04-11 15:50:18 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1334159374&type=1', '301293', 'Health Mate', '4040200', '2012-04-11 08:49:34 -0700', 'count', '30.0154', '2019-06-27 22:55:10 -0700', '2012-04-13 08:32:06 -0700', '0', '4.4.2', '2012-04-13 15:32:49 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1334331126&type=1', '301293', 'Health Mate', '4040200', '2012-04-13 08:32:06 -0700', 'count', '29.9383', '2019-06-27 22:55:10 -0700', '2012-04-20 08:21:38 -0700', '0', '4.4.2', '2012-04-20 15:52:45 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1334935298&type=1', '301293', 'Health Mate', '4040200', '2012-04-20 08:21:38 -0700', 'count', '30.2006', '2019-06-27 22:55:10 -0700', '2012-04-25 09:00:01 -0700', '0', '4.4.2', '2012-04-25 16:00:42 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1335369601&type=1', '301293', 'Health Mate', '4040200', '2012-04-25 09:00:01 -0700', 'count', '30.2006', '2019-06-27 22:55:10 -0700', '2012-05-04 11:10:18 -0700', '0', '4.4.2', '2012-05-04 18:10:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1336155018&type=1', '301293', 'Health Mate', '4040200', '2012-05-04 11:10:18 -0700', 'count', '30.4321', '2019-06-27 22:55:10 -0700', '2012-05-12 09:35:00 -0700', '0', '4.4.2', '2012-05-12 16:35:43 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1336840500&type=1', '301293', 'Health Mate', '4040200', '2012-05-12 09:35:00 -0700', 'count', '30.1235', '2019-06-27 22:55:10 -0700', '2012-05-22 09:27:53 -0700', '0', '4.4.2', '2012-05-22 16:28:37 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1337704073&type=1', '301293', 'Health Mate', '4040200', '2012-05-22 09:27:53 -0700', 'count', '30.4167', '2019-06-27 22:55:10 -0700', '2012-05-31 09:23:16 -0700', '0', '4.4.2', '2012-05-31 16:24:04 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1338481396&type=1', '301293', 'Health Mate', '4040200', '2012-05-31 09:23:16 -0700', 'count', '30.2006', '2019-06-27 22:55:10 -0700', '2012-06-08 09:29:07 -0700', '0', '4.4.2', '2012-06-08 16:29:52 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1339172947&type=1', '301293', 'Health Mate', '4040200', '2012-06-08 09:29:07 -0700', 'count', '30.5247', '2019-06-27 22:55:10 -0700', '2012-06-21 08:07:33 -0700', '0', '4.4.2', '2012-06-21 15:08:20 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1340291253&type=1', '301293', 'Health Mate', '4040200', '2012-06-21 08:07:33 -0700', 'count', '30.5864', '2019-06-27 22:55:10 -0700', '2012-08-08 10:02:22 -0700', '0', '4.4.2', '2012-08-08 17:03:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1344445342&type=1', '301293', 'Health Mate', '4040200', '2012-08-08 10:02:22 -0700', 'count', '30.6636', '2019-06-27 22:55:10 -0700', '2012-08-17 09:11:32 -0700', '0', '4.4.2', '2012-08-17 16:42:05 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1345219892&type=1', '301293', 'Health Mate', '4040200', '2012-08-17 09:11:32 -0700', 'count', '30.8796', '2019-06-27 22:55:10 -0700', '2012-09-10 08:27:21 -0700', '0', '4.4.2', '2012-09-10 15:28:07 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1347290841&type=1', '301293', 'Health Mate', '4040200', '2012-09-10 08:27:21 -0700', 'count', '31.034', '2019-06-27 22:55:10 -0700', '2012-09-17 08:35:33 -0700', '0', '4.4.2', '2012-09-17 15:35:33 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1347896133&type=1', '301293', 'Health Mate', '4040200', '2012-09-17 08:35:33 -0700', 'count', '30.7099', '2019-06-27 22:55:10 -0700', '2012-09-26 08:59:46 -0700', '0', '4.4.2', '2012-09-26 16:13:18 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1348675186&type=1', '301293', 'Health Mate', '4040200', '2012-09-26 08:59:46 -0700', 'count', '30.679', '2019-06-27 22:55:10 -0700', '2012-10-18 08:51:16 -0700', '0', '4.4.2', '2012-10-18 15:51:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1350575476&type=1', '301293', 'Health Mate', '4040200', '2012-10-18 08:51:16 -0700', 'count', '30.7716', '2019-06-27 22:55:10 -0700', '2012-11-15 08:54:57 -0700', '0', '4.4.2', '2012-11-15 15:55:58 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1352994897&type=1', '301293', 'Health Mate', '4040200', '2012-11-15 08:54:57 -0700', 'count', '31.0802', '2019-06-27 22:55:10 -0700', '2012-12-17 09:13:40 -0700', '0', '4.4.2', '2012-12-17 16:20:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1355760820&type=1', '301293', 'Health Mate', '4040200', '2012-12-17 09:13:40 -0700', 'count', '29.784', '2019-06-27 22:55:10 -0700', '2012-12-19 11:09:55 -0700', '0', '4.4.2', '2012-12-19 18:10:37 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1355940595&type=1', '301293', 'Health Mate', '4040200', '2012-12-19 11:09:55 -0700', 'count', '29.6914', '2019-06-27 22:55:10 -0700', '2012-12-25 10:37:41 -0700', '0', '4.4.2', '2012-12-25 17:38:25 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1356457061&type=1', '301293', 'Health Mate', '4040200', '2012-12-25 10:37:41 -0700', 'count', '29.8765', '2019-06-27 22:55:10 -0700', '2013-01-01 10:44:02 -0700', '0', '4.4.2', '2013-01-01 17:44:46 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1357062242&type=1', '301293', 'Health Mate', '4040200', '2013-01-01 10:44:02 -0700', 'count', '30.0772', '2019-06-27 22:55:10 -0700', '2013-01-15 09:10:46 -0700', '0', '4.4.2', '2013-01-15 16:11:28 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1358266246&type=1', '301293', 'Health Mate', '4040200', '2013-01-15 09:10:46 -0700', 'count', '29.9691', '2019-06-27 22:55:10 -0700', '2013-01-20 11:03:39 -0700', '0', '4.4.2', '2013-01-20 18:04:22 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1358705019&type=1', '301293', 'Health Mate', '4040200', '2013-01-20 11:03:39 -0700', 'count', '30.108', '2019-06-27 22:55:10 -0700', '2013-01-30 08:56:30 -0700', '0', '4.4.2', '2013-01-30 15:57:14 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1359561390&type=1', '301293', 'Health Mate', '4040200', '2013-01-30 08:56:30 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2013-02-04 11:02:35 -0700', '0', '4.4.2', '2013-02-04 18:03:25 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1360000955&type=1', '301293', 'Health Mate', '4040200', '2013-02-04 11:02:35 -0700', 'count', '29.8148', '2019-06-27 22:55:10 -0700', '2013-02-07 09:07:06 -0700', '0', '4.4.2', '2013-02-07 16:07:49 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1360253226&type=1', '301293', 'Health Mate', '4040200', '2013-02-07 09:07:06 -0700', 'count', '30.1389', '2019-06-27 22:55:10 -0700', '2013-02-19 08:49:57 -0700', '0', '4.4.2', '2013-02-19 15:50:39 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1361288997&type=1', '301293', 'Health Mate', '4040200', '2013-02-19 08:49:57 -0700', 'count', '30.1235', '2019-06-27 22:55:10 -0700', '2013-03-02 11:20:54 -0700', '0', '4.4.2', '2013-03-02 18:21:38 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1362248454&type=1', '301293', 'Health Mate', '4040200', '2013-03-02 11:20:54 -0700', 'count', '30', '2019-06-27 22:55:10 -0700', '2013-04-23 08:05:30 -0700', '0', '4.4.2', '2013-04-23 15:06:59 +0000', 'withings-bd2://timeline/measure?user                    """"""
id=301293&date=1366729530&type=1', '301293', 'Health Mate', '4040200', '2013-04-23 08:05:30 -0700', 'count', '30.5247', '2019-06-27 22:55:10 -0700', '2013-05-09 09:49:18 -0700', '0', '4.4.2', '2013-05-09 16:50:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1368118158&type=1', '301293', 'Health Mate', '4040200', '2013-05-09 09:49:18 -0700', 'count', '30.4167', '2019-06-27 22:55:10 -0700', '2013-06-09 09:28:47 -0700', '0', '4.4.2', '2013-06-09 16:29:30 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1370795327&type=1', '301293', 'Health Mate', '4040200', '2013-06-09 09:28:47 -0700', 'count', '30.8333', '2019-06-27 22:55:10 -0700', '2013-07-09 08:00:17 -0700', '0', '4.4.2', '2013-07-09 15:01:00 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1373382017&type=1', '301293', 'Health Mate', '4040200', '2013-07-09 08:00:17 -0700', 'count', '30.8179', '2019-06-27 22:55:10 -0700', '2013-07-28 09:16:55 -0700', '0', '4.4.2', '2013-07-28 16:17:39 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1375028215&type=1', '301293', 'Health Mate', '4040200', '2013-07-28 09:16:55 -0700', 'count', '30.5556', '2019-06-27 22:55:10 -0700', '2013-09-13 09:22:19 -0700', '0', '4.4.2', '2013-09-13 16:23:08 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1379089339&type=1', '301293', 'Health Mate', '4040200', '2013-09-13 09:22:19 -0700', 'count', '30.9568', '2019-06-27 22:55:10 -0700', '2013-09-24 08:08:23 -0700', '0', '4.4.2', '2013-09-24 15:09:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1380035303&type=1', '301293', 'Health Mate', '4040200', '2013-09-24 08:08:23 -0700', 'count', '31.4352', '2019-06-27 22:55:10 -0700', '2013-10-01 08:15:13 -0700', '0', '4.4.2', '2013-10-01 15:15:57 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1380640513&type=1', '301293', 'Health Mate', '4040200', '2013-10-01 08:15:13 -0700', 'count', '31.2037', '2019-06-27 22:55:10 -0700', '2013-10-23 09:31:25 -0700', '0', '4.4.2', '2013-10-23 16:32:13 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1382545885&type=1', '301293', 'Health Mate', '4040200', '2013-10-23 09:31:25 -0700', 'count', '31.8056']
```",197882382,healthkit-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1353411865,I_kwDODEpn8M5Qq20Z,1,Problem with my user,2467,fernand0,open,0,,,,,0,2022-08-28T16:59:37Z,2022-08-28T16:59:37Z,,NONE,,"If I call the program with:
    inaturalist-to-sqlite inaturalist.db ftricas
the program exits with an error:
 `Importing 36 observations
Traceback (most recent call last):
  File ""/home/ftricas/.pyenv/versions/3.10.6/bin/inaturalist-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py"", line 1130, in __call__
    return self.main(*args, **kwargs)
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py"", line 1055, in main
    rv = self.invoke(ctx)
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py"", line 760, in invoke
    return __callback(*args, **kwargs)
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/inaturalist_to_sqlite/cli.py"", line 51, in cli
    save_observation(observation, db)
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/inaturalist_to_sqlite/utils.py"", line 34, in save_observation
    db[""observations""]
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py"", line 2965, in insert
    return self.insert_all(
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py"", line 3068, in insert_all
    self.create(
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py"", line 1564, in create
    self.db.create_table(
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py"", line 951, in create_table
    sql = self.create_table_sql(
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py"", line 765, in create_table_sql
    foreign_keys = self.resolve_foreign_keys(name, foreign_keys or [])
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py"", line 702, in resolve_foreign_keys
    other_table = table.guess_foreign_table(column)
  File ""/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/sqlite_utils/db.py"", line 2061, in guess_foreign_table
    raise NoObviousTable(
sqlite_utils.db.NoObviousTable: No obvious foreign key table for column 'taxon' - tried ['taxon', 'taxons']
`
If I call the program with your user everything seems to go well and then, I can call the program with my own user without problems. Moreover, I can call the program again with my own user and everything goes well now.

Additional info, the command:
    
    sqlite-utils tables inaturalist.db

shows that the correct name can be 'taxons'.

There is another small problem with a warning:
     
   warnings.warn(""urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported ""

",206202864,inaturalist-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/inaturalist-to-sqlite/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1485017981,I_kwDODEpn8M5Yg5N9,2,table identifications has no column named previous_observation_taxon,520541,heaversm,open,0,,,,,0,2022-12-08T16:47:17Z,2022-12-08T16:47:17Z,,NONE,,"Installed successfully with pip and ran `inaturalist-to-sqlite inaturalist.db simonw` and got the error:

```
sqlite3.OperationalError: table identifications has no column named previous_observation_taxon
```",206202864,inaturalist-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/inaturalist-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
503233021,MDU6SXNzdWU1MDMyMzMwMjE=,1,Use better pagination (and implement progress bar),9599,simonw,closed,0,,,,,4,2019-10-07T04:58:11Z,2020-03-27T22:13:57Z,2020-03-27T22:13:57Z,MEMBER,,"Right now we attempt to load everything at once - which caps out at 5,000 items and is really slow.

We can do better by implementing pagination using count and offset.",213286752,pocket-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1246826792,I_kwDODLZ_YM5KUREo,10,"When running `auth` command, don't overwrite an existing auth.json file",11887,ashanan,closed,0,,,,,3,2022-05-24T16:42:20Z,2022-09-07T15:07:38Z,2022-08-22T16:17:19Z,NONE,,"Ran the `auth` command in the same directory I'd previously set up an auth.json file for `twitter-to-sqlite` and it was completely overwritten.  Not the biggest issue, but still unexpected.  Ideally, for me, the keys would just be added to the existing file, but getting a warning and a chance to back out would be a good solution as well.",213286752,pocket-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/10/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1345452427,I_kwDODLZ_YM5QMfmL,11,"-a option is used for ""--auth"" and for ""--all""",2467,fernand0,closed,0,,,,,3,2022-08-21T10:50:48Z,2022-08-21T21:11:57Z,2022-08-21T21:11:57Z,NONE,,"I'm not sure which option is best, instead of -a -all.",213286752,pocket-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1795187493,I_kwDODLZ_YM5rAGMl,12,Switch to pyproject.toml,9599,simonw,closed,0,,,,,2,2023-07-09T01:06:56Z,2023-07-09T01:19:43Z,2023-07-09T01:19:42Z,MEMBER,,First of my CLI tools to use https://til.simonwillison.net/python/pyproject,213286752,pocket-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
503234169,MDU6SXNzdWU1MDMyMzQxNjk=,2,Track and use the 'since' value,9599,simonw,closed,0,,,,,3,2019-10-07T05:02:59Z,2020-03-27T22:22:30Z,2020-03-27T22:22:30Z,MEMBER,,"Pocket says:

> Whenever possible, you should use the since parameter, or count and and offset parameters when retrieving a user's list. After retrieving the list, you should store the current time (which is provided along with the list response) and pass that in the next request for the list. This way the server only needs to return a small set (changes since that time) instead of the user's entire list every time.

At the bottom of https://getpocket.com/developer/docs/v3/retrieve",213286752,pocket-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
503243784,MDU6SXNzdWU1MDMyNDM3ODQ=,3,Extract images into separate tables,9599,simonw,open,0,,,,,1,2019-10-07T05:43:01Z,2020-09-01T06:17:45Z,,MEMBER,,"As already done with authors. Slightly harder because images do not have a universally unique ID. Also need to figure out what to do about there being columns for both `image` and `images`.

<img width=""1522"" alt=""memory__items"" src=""https://user-images.githubusercontent.com/9599/66287418-9ab20680-e88a-11e9-96bf-6c80d881eff0.png"">
",213286752,pocket-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
589402939,MDU6SXNzdWU1ODk0MDI5Mzk=,4,"Store authentication information as ""pocket_access_token"" etc",9599,simonw,closed,0,,,,,0,2020-03-27T20:43:22Z,2020-03-27T20:43:59Z,2020-03-27T20:43:59Z,MEMBER,,The `pocket_` prefix will mean that the same `auth.json` file can be used for other Dogsheep tools without Pocket over-riding a value set by some other tool.,213286752,pocket-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
629473827,MDU6SXNzdWU2Mjk0NzM4Mjc=,5,Set up a demo,26745575,harryvederci,open,0,,,,,1,2020-06-02T19:56:49Z,2020-09-01T06:18:43Z,,NONE,,"First off, thanks for open sourcing this application! This is a suggestion to increase the amount of people that would make use of it: an example in the readme file would help.

Currently, users have to clone the app, install it, authorize through pocket, run a command, an then find out if this application does what they hope it does.

Another possibility is to add a file `example-output.db`, containing one (mock) Pocket article.

Keep up the good work!",213286752,pocket-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
689848827,MDU6SXNzdWU2ODk4NDg4Mjc=,6,ISO timestamps,9599,simonw,open,0,,,,,0,2020-09-01T06:16:42Z,2020-09-01T06:16:42Z,,MEMBER,,"The `time_added`, `time_updated` and `time_read` columns currently store data like this:

    September 19, 2019 - 00:30:30 UTC

Should use ISO instead, e.g. `2020-07-26T01:05:24+00:00`",213286752,pocket-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
750141615,MDExOlB1bGxSZXF1ZXN0NTI2ODQ3ODIz,7,Fixed conflicting  CLI flags,8944,tlockney,closed,0,,,,,1,2020-11-24T23:25:12Z,2022-08-21T21:11:56Z,2022-08-21T21:11:56Z,CONTRIBUTOR,dogsheep/pocket-to-sqlite/pulls/7,"The `-a` used for the auth credentials and the shortened form of the `--all` flags were in conflict on the `fetch` command. To be consistent with other `-to-sqlite` libraries in the Dogsheep ecosystem, I removed the shortened form of the `--all` flag.",213286752,pocket-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/7/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
797728929,MDU6SXNzdWU3OTc3Mjg5Mjk=,8,QUESTION: extract full text,417363,darribas,open,0,,,,,0,2021-01-31T14:50:10Z,2021-01-31T14:50:10Z,,NONE,,"This may be solved or a feature already, but I couldn't figure it out, is it possible to extract and store also full text from the saved pages? The same way that Pocket parses the text, it'd be amazing to be able to store (and thus make searchable later) the text.

Thank you very much for the project, it's such an amazing idea! ",213286752,pocket-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
801780625,MDU6SXNzdWU4MDE3ODA2MjU=,9,SSL Error,12669260,jfeiwell,open,0,,,,,2,2021-02-05T02:12:56Z,2021-02-07T18:45:04Z,,NONE,,"Here's the error I get when running `pip install pocket-to-sqlite`: 

```
  Could not fetch URL https://pypi.python.org/simple/pocket-to-sqlite/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661) - skipping
  Could not find a version that satisfies the requirement pocket-to-sqlite (from versions: )
No matching distribution found for pocket-to-sqlite
```

Does this require python 3? ",213286752,pocket-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
487598042,MDU6SXNzdWU0ODc1OTgwNDI=,1,Implement code to pull checkins from the Foursquare API,9599,simonw,closed,0,,,,,0,2019-08-30T17:40:02Z,2019-08-30T18:23:24Z,2019-08-30T18:23:24Z,MEMBER,,"The tool currently only works with a pre-prepared JSON file of checkins.

When called without options, it should prompt the user to paste in a Foursquare OAuth token.

The `--token=` option should work too, and should be backed up by an optional environment variable.",205429375,swarm-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
719637258,MDExOlB1bGxSZXF1ZXN0NTAxNzkxNjYz,10,Update utils.py to fix sqlite3.OperationalError,29426418,mattiaborsoi,closed,0,,,,,1,2020-10-12T20:17:53Z,2020-10-12T20:25:10Z,2020-10-12T20:25:09Z,CONTRIBUTOR,dogsheep/swarm-to-sqlite/pulls/10,"Fixes the errors:
- sqlite3.OperationalError: table posts has no column named text
- sqlite3.OperationalError: table photos has no column named hasSticker

That will cause sqlite-utils to notice if there's a missing column and add it. As recommended by @simonw",205429375,swarm-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/10/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
743400216,MDU6SXNzdWU3NDM0MDAyMTY=,11,Error thrown: sqlite3.OperationalError: table users has no column named lastName,61791,beaugunderson,closed,0,,,,,2,2020-11-16T01:21:18Z,2021-01-18T04:35:22Z,2021-01-18T04:35:22Z,NONE,,"Just installed `swarm-to-sqlite-0.3.2` and tried according to the docs:

```
Traceback (most recent call last):
  File ""/usr/local/bin/swarm-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/usr/local/lib/python3.9/site-packages/swarm_to_sqlite/cli.py"", line 73, in cli
    save_checkin(checkin, db)
  File ""/usr/local/lib/python3.9/site-packages/swarm_to_sqlite/utils.py"", line 82, in save_checkin
    checkins_table.m2m(""users"", user, m2m_table=""likes"", pk=""id"")
  File ""/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1914, in m2m
    id = other_table.insert(record, pk=pk, replace=True).last_pk
  File ""/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1647, in insert
    return self.insert_all(
  File ""/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1765, in insert_all
    self.insert_chunk(
  File ""/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1575, in insert_chunk
    result = self.db.execute(query, params)
  File ""/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py"", line 200, in execute
    return self.conn.execute(sql, parameters)
sqlite3.OperationalError: table users has no column named lastName
```",205429375,swarm-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
951817328,MDU6SXNzdWU5NTE4MTczMjg=,12,403 when getting token,285352,treyhunner,open,0,,,,,1,2021-07-23T18:43:26Z,2021-10-12T18:31:57Z,,NONE,,"I tried to use https://your-foursquare-oauth-token.glitch.me/ to get my Swarm auth token and got a 403 after I clicked the Allow button:

![image](https://user-images.githubusercontent.com/285352/126826478-60e53614-263d-40bb-9f1d-c1a676644eb0.png)

I'm not sure if this is the right repo to report this in",205429375,swarm-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/12/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1373210675,I_kwDODD6af85R2Ygz,13,fails before generating views. ERR:  table sqlite_master may not be modified,116795,pax,open,0,,,,,4,2022-09-14T15:41:50Z,2023-04-11T03:46:17Z,,NONE,,"generates checkins.db but seems to fail before generating views 

note: it worked on an Ubuntu WSL but fails on macOS 12.5.1

later edit: I suspect this is a problem with my local set-up, `dogsheep-beta index` also throws the same error

full error:

    Importing 2591 checkins  [###################################-]   98%  00:00:00
    Traceback (most recent call last):
      File ""/Users/pax/devbox/envAll/bin/swarm-to-sqlite"", line 8, in <module>
        sys.exit(cli())
      File ""/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
        return self.main(*args, **kwargs)
      File ""/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py"", line 782, in main
        rv = self.invoke(ctx)
      File ""/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File ""/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
        return callback(*args, **kwargs)
      File ""/Users/pax/devbox/envAll/lib/python3.8/site-packages/swarm_to_sqlite/cli.py"", line 77, in cli
        ensure_foreign_keys(db)
      File ""/Users/pax/devbox/envAll/lib/python3.8/site-packages/swarm_to_sqlite/utils.py"", line 145, in ensure_foreign_keys
        db[fk.table].add_foreign_key(fk.column, fk.other_table, fk.other_column)
      File ""/Users/pax/devbox/envAll/lib/python3.8/site-packages/sqlite_utils/db.py"", line 2123, in add_foreign_key
        self.db.add_foreign_keys([(self.name, column, other_table, other_column)])
      File ""/Users/pax/devbox/envAll/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1086, in add_foreign_keys
        cursor.execute(
    sqlite3.OperationalError: table sqlite_master may not be modified",205429375,swarm-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1393330070,PR_kwDODD6af84__DNJ,14,Photo links,6782721,redmanmale,open,0,,,,,0,2022-10-01T09:44:15Z,2022-11-18T17:10:49Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/swarm-to-sqlite/pulls/14,"* add to `checkin_details` view new column for a calculated photo links
* supported multiple links split by newline
* create `events` table if there's no events in the history to avoid SQL errors

Fixes #9.",205429375,swarm-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/14/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1661617056,I_kwDODD6af85jCkOg,15,ambiguous column name: createdAt - on checkin_details view,9599,simonw,closed,0,,,,,0,2023-04-11T01:07:47Z,2023-04-11T03:16:37Z,2023-04-11T03:16:37Z,MEMBER,,"It looks like Swarm changed their schema and now both `venues` and `checkins` have `createdAt` fields.

Which breaks this view: https://github.com/dogsheep/swarm-to-sqlite/blob/719b6e96a016d0ca8b316d3bed9c2a7a0cb499ee/swarm_to_sqlite/utils.py#L171-L188",205429375,swarm-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/15/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
487598468,MDU6SXNzdWU0ODc1OTg0Njg=,2,--save option to dump checkins to a JSON file on disk,9599,simonw,closed,0,,,,,1,2019-08-30T17:41:06Z,2019-08-31T02:40:21Z,2019-08-31T02:40:21Z,MEMBER,,"This is a complement to the `--load` option - mainly useful for development purposes.

(I'll rename `--file` to `--load` as part of this issue).",205429375,swarm-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
487600595,MDU6SXNzdWU0ODc2MDA1OTU=,3,Option to fetch only checkins more recent than the current max checkin,9599,simonw,closed,0,,,,,4,2019-08-30T17:46:45Z,2019-10-16T20:41:23Z,2019-10-16T20:39:59Z,MEMBER,,"The Foursquare checkins API supports ""return every checkin occurring after this point"" - I can pass it the maximum createdAt date currently stored in the database. This will allow for quick incremental fetches via a cron.",205429375,swarm-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
487601121,MDU6SXNzdWU0ODc2MDExMjE=,4,Online tool for getting a Foursquare OAuth token,9599,simonw,closed,0,,,,,1,2019-08-30T17:48:14Z,2019-08-31T18:07:26Z,2019-08-31T18:07:26Z,MEMBER,,"I will link to this from the documentation. See also this conversation on Twitter: https://twitter.com/simonw/status/1166822603023011840

I've decided to go with ""copy and paste in a token"" rather than hooking up a local web server that can have tokens passed to it.",205429375,swarm-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
487721884,MDU6SXNzdWU0ODc3MjE4ODQ=,5,Treat Foursquare timestamps as UTC,9599,simonw,closed,0,,,,,0,2019-08-31T02:44:47Z,2019-08-31T02:50:41Z,2019-08-31T02:50:41Z,MEMBER,,"Current test failure is due to timezone differences between my laptop and Circle CI:

https://circleci.com/gh/dogsheep/swarm-to-sqlite/3

```
E         Full diff:
E         - [{'created': '2018-07-01T04:48:19',
E         ?                           ^
E         + [{'created': '2018-07-01T02:48:19',
E         ?                           ^
E         'createdAt': 1530413299,
```

The timestamps I store in `created` should always be UTC.",205429375,swarm-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
543355051,MDExOlB1bGxSZXF1ZXN0MzU3NjQwMTg2,6,don't break if source is missing,78035,mfa,closed,0,,,,,1,2019-12-29T10:46:47Z,2020-03-28T02:28:11Z,2020-03-28T02:28:11Z,CONTRIBUTOR,dogsheep/swarm-to-sqlite/pulls/6,broke for me. very old checkins in 2010 had no source set.,205429375,swarm-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
589491711,MDU6SXNzdWU1ODk0OTE3MTE=,7,Upgrade to sqlite-utils 2.x,9599,simonw,closed,0,,,,,0,2020-03-28T02:24:51Z,2020-03-28T02:25:03Z,2020-03-28T02:25:03Z,MEMBER,,,205429375,swarm-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/7/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
648245071,MDU6SXNzdWU2NDgyNDUwNzE=,8,Error thrown: table photos has no column named hasSticker,18504,harperreed,closed,0,,,,,2,2020-06-30T14:54:37Z,2020-10-12T20:35:06Z,2020-10-12T20:25:24Z,NONE,,"While running `swarm-to-sqlite` it throws an error:

    harper@:~/dogsheep/swarm$ swarm-to-sqlite checkins.db --save=checkins.json
    Please provide your Foursquare OAuth token:
    Importing 8127 checkins  [#################-------------------]   49%  00:01:52
    Traceback (most recent call last):
    File ""/home/harper/.local/bin/swarm-to-sqlite"", line 11, in <module>
        sys.exit(cli())
    File ""/home/harper/.local/lib/python3.6/site-packages/click/core.py"", line 829, in __call__
        return self.main(*args, **kwargs)
    File ""/home/harper/.local/lib/python3.6/site-packages/click/core.py"", line 782, in main
        rv = self.invoke(ctx)
    File ""/home/harper/.local/lib/python3.6/site-packages/click/core.py"", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
    File ""/home/harper/.local/lib/python3.6/site-packages/click/core.py"", line 610, in invoke
        return callback(*args, **kwargs)
    File ""/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/cli.py"", line 73, in cli
        save_checkin(checkin, db)
    File ""/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/utils.py"", line 94, in save_checkin
        photos_table.insert(photo, replace=True)
    File ""/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/db.py"", line 963, in insert
        alter = self.value_or_default(""alter"", alter)
    File ""/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/db.py"", line 1142, in insert_all
        def upsert_all(
    sqlite3.OperationalError: table photos has no column named hasSticker

Where should i dig in?",205429375,swarm-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
673602857,MDU6SXNzdWU2NzM2MDI4NTc=,9,Define a view that displays photos correctly,9599,simonw,open,0,,,,,0,2020-08-05T14:53:39Z,2020-08-05T14:53:39Z,,MEMBER,,"The `photos` table stores data like this:

id | createdAt | source | prefix | suffix | width | height | visibility | created ▲ | user
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
5e12c9708506bc000840262a | January 06, 2020 - 05:45:20 UTC | Swarm for iOS 1 | https://fastly.4sqi.net/img/general/ | /15889193_AXxGk4I1nbzUZuyYqObgbXdJNyEHiwj6AUDq0tPZWtw.jpg | 1920 | 1440 | public | 2020-01-06T05:45:20 | 15889193

The photo URL can be derived from those pieces - define a SQL view which does that (using `datasette-json-html` to display the pictures)",205429375,swarm-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/9/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
488833136,MDU6SXNzdWU0ODg4MzMxMzY=,1,"Imported followers should go in ""users"", relationships in ""following""",9599,simonw,closed,0,,,,,0,2019-09-03T21:27:37Z,2019-09-04T20:23:04Z,2019-09-04T20:23:04Z,MEMBER,,"Right now `twitter-to-sqlite followers` dumps everything in a `followers` table, and doesn't actually record which account they are following!

It should instead save them all in a global `users` table and then set up m2m relationships in a `following` table. This also means it should create a record for the specified user in order to record both sides of each relationship.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
492297930,MDU6SXNzdWU0OTIyOTc5MzA=,10,Rethink progress bars for various commands,9599,simonw,closed,0,,,,,5,2019-09-11T15:06:47Z,2020-04-01T03:45:48Z,2020-04-01T03:45:48Z,MEMBER,,"Progress bars and the `--silent` option are implemented inconsistently across commands at the moment.

This is made more challenging by the fact that for many operations the total length is not known.

https://click.palletsprojects.com/en/7.x/api/#click.progressbar",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
503045221,MDU6SXNzdWU1MDMwNDUyMjE=,11,Commands for recording real-time tweets from the streaming API,9599,simonw,closed,0,,,,,1,2019-10-06T03:09:30Z,2019-10-06T04:54:17Z,2019-10-06T04:48:31Z,MEMBER,,"https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter

We can support tracking keywords and following specific users.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/11/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
503053800,MDU6SXNzdWU1MDMwNTM4MDA=,12,"Extract ""source"" into a separate lookup table",9599,simonw,closed,0,,,,,3,2019-10-06T05:17:23Z,2019-10-17T15:49:24Z,2019-10-17T15:49:24Z,MEMBER,,"It's pretty bulky and ugly at the moment:

<img width=""334"" alt=""trump__tweets__1_820_rows"" src=""https://user-images.githubusercontent.com/9599/66264630-df23a080-e7bd-11e9-9154-403c2e69f841.png"">
",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
503085013,MDU6SXNzdWU1MDMwODUwMTM=,13,statuses-lookup command,9599,simonw,closed,0,,,,,1,2019-10-06T11:00:20Z,2019-10-07T00:33:49Z,2019-10-07T00:31:44Z,MEMBER,,"For bulk retrieving tweets by their ID.

https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-lookup

Rate limit is 900/15 minutes (1 call per second) but each call can pull up to 100 IDs, so we can pull 6,000 per minute.

Should support `--SQL` and `--attach` #8 ",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/13/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
503244410,MDU6SXNzdWU1MDMyNDQ0MTA=,14,"When importing favorites, record which user favorited them",9599,simonw,closed,0,,,,,0,2019-10-07T05:45:11Z,2019-10-14T03:30:25Z,2019-10-14T03:30:25Z,MEMBER,,"This code currently just dumps them into the `tweets` table without recording who it was who had favorited them.

https://github.com/dogsheep/twitter-to-sqlite/blob/436a170d74ec70903d1b4ca430c2c6b6435cdfcc/twitter_to_sqlite/cli.py#L152-L157",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/14/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
505666744,MDExOlB1bGxSZXF1ZXN0MzI3MDUxNjcz,15,"twitter-to-sqlite import command, refs #4",9599,simonw,closed,0,,,,,0,2019-10-11T06:37:14Z,2019-10-11T06:45:01Z,2019-10-11T06:45:01Z,MEMBER,dogsheep/twitter-to-sqlite/pulls/15,,206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/15/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
505673645,MDU6SXNzdWU1MDU2NzM2NDU=,16,Do a better job with archived direct message threads,9599,simonw,open,0,,,,,0,2019-10-11T06:55:21Z,2019-10-11T06:55:27Z,,MEMBER,,https://github.com/dogsheep/twitter-to-sqlite/blob/fb2698086d766e0333a55bb73435e7283feeb438/twitter_to_sqlite/archive.py#L98-L99,206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/16/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
505674949,MDU6SXNzdWU1MDU2NzQ5NDk=,17,import command should empty all archive-* tables first,9599,simonw,closed,0,,,,,2,2019-10-11T06:58:43Z,2019-10-11T15:40:08Z,2019-10-11T15:40:08Z,MEMBER,,Can have a CLI option for NOT doing that.,206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
505928530,MDU6SXNzdWU1MDU5Mjg1MzA=,18,Command to import home-timeline,9599,simonw,closed,0,,,,,4,2019-10-11T15:47:54Z,2019-10-11T16:51:33Z,2019-10-11T16:51:12Z,MEMBER,,"Feature request: https://twitter.com/johankj/status/1182563563136868352
> Would it be possible to save all tweets in my timeline from the last X days? I would love to see how big a percentage some users are of my daily timeline as a metric on whether I should unfollow them/move them to a list.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
506087267,MDU6SXNzdWU1MDYwODcyNjc=,19,since_id support for home-timeline,9599,simonw,closed,0,,,,,3,2019-10-11T22:48:24Z,2019-10-16T19:13:06Z,2019-10-16T19:12:46Z,MEMBER,,Currently every time you run `home-timeline` we pull all 800 available tweets. We should offer to support `since_id` (which can be provided or can be pulled directly from the database) in order to work more efficiently if this command is executed e.g. on a cron.,206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
488833698,MDU6SXNzdWU0ODg4MzM2OTg=,2,"""twitter-to-sqlite user-timeline"" command for pulling tweets by a specific user",9599,simonw,closed,0,,,,,3,2019-09-03T21:29:12Z,2019-09-04T20:02:11Z,2019-09-04T20:02:11Z,MEMBER,,"Twitter only allows up to 3,200 tweets to be retrieved from https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html

I'm going to do:

    $ twitter-to-sqlite tweets simonw

",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
506268945,MDU6SXNzdWU1MDYyNjg5NDU=,20,--since support for various commands for refresh-by-cron,9599,simonw,closed,0,,,,,3,2019-10-13T03:40:46Z,2019-10-21T03:32:04Z,2019-10-16T19:26:11Z,MEMBER,,"I want to run a cron that updates my Twitter database every X minutes.

It should be able to retrieve the following without needing to paginate through everything:

- [x] Tweets I have tweeted
- [x] My home timeline (see #19)
- [x] Tweets I have favourited

It would be nice if this could be standardized across all commands as a `--since` option.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
506432572,MDU6SXNzdWU1MDY0MzI1NzI=,21,Fix &amp; escapes in tweet text,9599,simonw,closed,0,,,,,1,2019-10-14T03:37:28Z,2019-10-15T18:48:16Z,2019-10-15T18:48:16Z,MEMBER,,"<img width=""1136"" alt=""twitter__tweets__21_773_rows_where_sorted_by_id_descending"" src=""https://user-images.githubusercontent.com/9599/66728360-38f91b80-edf9-11e9-95b5-ce6d097fe18e.png"">

Shouldn't be storing `&amp;` here.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/21/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
508024032,MDU6SXNzdWU1MDgwMjQwMzI=,22,Ability to import from uncompressed archive or from specific files,9599,simonw,closed,0,,,,,0,2019-10-16T18:31:57Z,2019-10-16T18:53:36Z,2019-10-16T18:53:36Z,MEMBER,,"Currently you can only import like this:

    $ twitter-to-sqlite import path-to-twitter.zip

It would be useful if you could import from a folder that was decompressed from that zip:

    $ twitter-to-sqlite import path-to-twitter/

AND from individual files within that folder - since that would allow you to e.g. selectively import certain files:

    $ twitter-to-sqlite import path-to-twitter/favorites.js path-to-twitter/tweets.js",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/22/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
508190730,MDU6SXNzdWU1MDgxOTA3MzA=,23,Extremely simple migration system,9599,simonw,closed,0,,,,,2,2019-10-17T02:13:57Z,2019-10-17T16:57:17Z,2019-10-17T16:57:17Z,MEMBER,,"Needed for #12. This is going to be an incredibly simple version of the Django migration system.

* A `migrations` table, keeping track of which migrations were applied (and when)
* A `migrate()` function which applies any pending migrations
* A `MIGRATIONS` constant which is a list of functions to be applied

The function names will be detected and used as the names of the migrations.

Every time you run the CLI tool it will call the `migrate()` function before doing anything else.

Needs to take into account that there might be no tables at all. As such, migration functions should sanity check that the tables they are going to work on actually exist.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
508553387,MDExOlB1bGxSZXF1ZXN0MzI5MzI0MzY4,24,Tweet source extraction and new migration system,9599,simonw,closed,0,,,,,0,2019-10-17T15:24:56Z,2019-10-17T15:49:29Z,2019-10-17T15:49:24Z,MEMBER,dogsheep/twitter-to-sqlite/pulls/24,Closes #12 and #23,206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/24/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
508578780,MDU6SXNzdWU1MDg1Nzg3ODA=,25,Ensure migrations don't accidentally create foreign key twice,9599,simonw,closed,0,,,,,2,2019-10-17T16:08:50Z,2019-10-17T16:56:47Z,2019-10-17T16:56:47Z,MEMBER,,"Is it possible for these lines to run against a database table that already has these foreign keys?

https://github.com/dogsheep/twitter-to-sqlite/blob/c9295233f219c446fa2085cace987067488a31b9/twitter_to_sqlite/migrations.py#L21-L22",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
513074501,MDU6SXNzdWU1MTMwNzQ1MDE=,26,Command for importing mentions timeline,9599,simonw,closed,0,,,,,1,2019-10-28T03:14:27Z,2019-10-30T02:36:13Z,2019-10-30T02:20:47Z,MEMBER,,"https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-mentions_timeline

Almost identical to home-timeline #18 but it uses `https://api.twitter.com/1.1/statuses/mentions_timeline.json` instead.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/26/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
514459062,MDU6SXNzdWU1MTQ0NTkwNjI=,27,retweets-of-me command,9599,simonw,closed,0,,,,,4,2019-10-30T07:43:01Z,2019-11-03T01:12:58Z,2019-11-03T01:12:58Z,MEMBER,,https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-retweets_of_me,206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
515658861,MDU6SXNzdWU1MTU2NTg4NjE=,28,Add indexes to followers table,9599,simonw,closed,0,,,,,1,2019-10-31T18:40:22Z,2019-11-09T20:15:42Z,2019-11-09T20:11:48Z,MEMBER,,`select follower_id from following where followed_id = 12497` takes over a second for me at the moment.,206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/28/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
518725064,MDU6SXNzdWU1MTg3MjUwNjQ=,29,`import` command fails on empty files,21148,jacobian,closed,0,,,,,4,2019-11-06T20:34:26Z,2019-11-09T20:33:38Z,2019-11-09T19:36:36Z,CONTRIBUTOR,,"If a file in the export is empty (in my case it was `account-suspensions.js`), `twitter-to-sqlite import` fails:

```
$ twitter-to-sqlite import twitter.db ~/Downloads/twitter-2019-11-06-926f4f3be4b3b1fcb1aa387c40cd14f7c8aaf9bbcdb2d78ac14d9989add501bb.zip
Traceback (most recent call last):
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/bin/twitter-to-sqlite"", line 10, in <module>
    sys.exit(cli())
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py"", line 764, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py"", line 717, in main
    rv = self.invoke(ctx)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py"", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py"", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py"", line 555, in invoke
    return callback(*args, **kwargs)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/cli.py"", line 627, in import_
    archive.import_from_file(db, filename, content)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/archive.py"", line 224, in import_from_file
    db[table_name].upsert_all(rows, hash_id=""pk"")
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1113, in upsert_all
    extracts=extracts,
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/sqlite_utils/db.py"", line 980, in insert_all
    first_record = next(records)
StopIteration
```

This appears to be because `db.upsert_all` is called with no rows -- I think? 

I hacked around this by modifying `import_from_file` to have an `if rows:` clause:

```
    for table, rows in to_insert.items():
        if rows:
            table_name = ""archive_{}"".format(table.replace(""-"", ""_""))
            ...
```

I'm happy to work up a real PR if that's the right approach, but I'm not sure it is.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
488833975,MDU6SXNzdWU0ODg4MzM5NzU=,3,Command for running a search and saving tweets for that search,9599,simonw,closed,0,,,,,6,2019-09-03T21:29:56Z,2019-11-04T05:31:56Z,2019-11-04T05:31:16Z,MEMBER,,    $ twitter-to-sqlite search dogsheep,206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
518739697,MDU6SXNzdWU1MTg3Mzk2OTc=,30,`followers` fails because `transform_user` is called twice,21148,jacobian,closed,0,,,,,2,2019-11-06T20:44:52Z,2019-11-09T20:15:28Z,2019-11-09T19:55:52Z,CONTRIBUTOR,,"Trying to run `twitter-to-sqlite followers` errors out:

```
Traceback (most recent call last):
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/bin/twitter-to-sqlite"", line 10, in <module>
    sys.exit(cli())
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py"", line 764, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py"", line 717, in main
    rv = self.invoke(ctx)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py"", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py"", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py"", line 555, in invoke
    return callback(*args, **kwargs)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/cli.py"", line 130, in followers
    go(bar.update)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/cli.py"", line 116, in go
    utils.save_users(db, [profile])
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/utils.py"", line 302, in save_users
    transform_user(user)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/utils.py"", line 181, in transform_user
    user[""created_at""] = parser.parse(user[""created_at""])
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py"", line 1374, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py"", line 646, in parse
    res, skipped_tokens = self._parse(timestr, **kwargs)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py"", line 725, in _parse
    l = _timelex.split(timestr)         # Splits the timestr into tokens
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py"", line 207, in split
    return list(cls(s))
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py"", line 76, in __init__
    '{itype}'.format(itype=instream.__class__.__name__))
TypeError: Parser must be a string or character stream, not datetime
```

This appears to be because https://github.com/dogsheep/twitter-to-sqlite/blob/master/twitter_to_sqlite/cli.py#L111 calls `transform_user`, and then https://github.com/dogsheep/twitter-to-sqlite/blob/master/twitter_to_sqlite/cli.py#L116 calls `transform_user` again, which fails because the user is already transformed.

I was able to work around this by commenting out https://github.com/dogsheep/twitter-to-sqlite/blob/master/twitter_to_sqlite/cli.py#L116. 

Shall I work up a patch for that, or is there a better approach?",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
520508502,MDU6SXNzdWU1MjA1MDg1MDI=,31,"""friends"" command (similar to ""followers"")",9599,simonw,closed,0,,,,,2,2019-11-09T20:20:20Z,2022-09-20T05:05:03Z,2020-02-07T07:03:28Z,MEMBER,,"Current list of commands:
```
  followers          Save followers for specified user (defaults to...
  followers-ids      Populate followers table with IDs of account followers
  friends-ids        Populate followers table with IDs of account friends
```
Obvious omission here is `friends`, which would be powered by `https://api.twitter.com/1.1/friends/list.json`: https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-friends-list",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
561454071,MDU6SXNzdWU1NjE0NTQwNzE=,32,"Documentation for "" favorites"" command",9599,simonw,closed,0,,,,,0,2020-02-07T06:50:11Z,2020-02-07T06:59:10Z,2020-02-07T06:59:10Z,MEMBER,,"It looks like I forgot to document this one in the README.

https://github.com/dogsheep/twitter-to-sqlite/blob/6ebd482619bd94180e54bb7b56549c413077d329/twitter_to_sqlite/cli.py#L183-L194",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/32/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
561469252,MDExOlB1bGxSZXF1ZXN0MzcyMjczNjA4,33,Upgrade to sqlite-utils 2.2.1,9599,simonw,closed,0,,,,,1,2020-02-07T07:32:12Z,2020-03-20T19:21:42Z,2020-03-20T19:21:41Z,MEMBER,dogsheep/twitter-to-sqlite/pulls/33,,206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/33/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
585266763,MDU6SXNzdWU1ODUyNjY3NjM=,34,IndexError running user-timeline command,9599,simonw,closed,0,,,,,2,2020-03-20T18:54:08Z,2020-03-20T19:20:52Z,2020-03-20T19:20:37Z,MEMBER,,"```
$ twitter-to-sqlite user-timeline data.db --screen_name Allen_Joines
Traceback (most recent call last):
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite"", line 11, in <module>
    load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')()
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 764, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 717, in main
    rv = self.invoke(ctx)
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 555, in invoke
    return callback(*args, **kwargs)
  File ""/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py"", line 256, in user_timeline
    utils.save_tweets(db, chunk)
  File ""/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py"", line 289, in save_tweets
    db[""users""].upsert(user, pk=""id"", alter=True)
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1128, in upsert
    conversions=conversions,
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1157, in upsert_all
    upsert=True,
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1096, in insert_all
    row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0]
IndexError: list index out of range
```",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
585282212,MDU6SXNzdWU1ODUyODIyMTI=,35,twitter-to-sqlite user-timeline [screen_names] --sql / --attach,9599,simonw,closed,0,,,,,5,2020-03-20T19:26:07Z,2020-03-20T20:17:00Z,2020-03-20T20:16:35Z,MEMBER,,Split from #8.,206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
585306847,MDU6SXNzdWU1ODUzMDY4NDc=,36,twitter-to-sqlite followers/friends --sql / --attach,9599,simonw,closed,0,,,,,0,2020-03-20T20:20:33Z,2020-03-20T23:12:38Z,2020-03-20T23:12:38Z,MEMBER,,"Split from #8. The `friends` and `followers` commands don't yet support `--sql` and `--attach`.

(`friends-ids` and `followers-ids` do though).",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/36/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
585353598,MDU6SXNzdWU1ODUzNTM1OTg=,37,"Handle ""User not found"" error",9599,simonw,closed,0,,,,,3,2020-03-20T22:14:32Z,2020-04-17T23:43:46Z,2020-04-17T23:43:46Z,MEMBER,,"While running `user-timeline` I got this bug (because a screen name I asked for didn't exist):
```
  File ""/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py"", line 185, in transform_user
    user[""created_at""] = parser.parse(user[""created_at""])
KeyError: 'created_at'
>>> import pdb
>>> pdb.pm()
> /Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py(185)transform_user()
-> user[""created_at""] = parser.parse(user[""created_at""])
(Pdb) user
{'errors': [{'code': 50, 'message': 'User not found.'}]}
```",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
585359363,MDU6SXNzdWU1ODUzNTkzNjM=,38,Screen name display for user-timeline is uneven,9599,simonw,closed,0,,,,,1,2020-03-20T22:30:23Z,2020-03-20T22:37:17Z,2020-03-20T22:37:17Z,MEMBER,,"```
CDPHE  [####################################]  67
CHFSKy  [####################################]  3216
DHSWI  [####################################]  41
DPHHSMT  [####################################]  742
Delaware_DHSS  [####################################]  3231
DhhsNevada  [####################################]  639
```
I could format them to match the length of the longest screen name instead.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/38/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
590666760,MDU6SXNzdWU1OTA2NjY3NjA=,39,--since feature can be confused by retweets,9599,simonw,closed,0,,,,,11,2020-03-30T23:25:33Z,2020-04-01T03:45:16Z,2020-04-01T03:45:16Z,MEMBER,,"If you run `twitter-to-sqlite user-timeline ... --since` it's supposed to fetch Tweets those specific users tweeted since last time the command was run.

It does this by seeking out the max ID of their previous tweets:

https://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/cli.py#L305-L311

BUT... this has a nasty flaw: if another account had retweeted one of their recent tweets the retweeted-tweet will have been loaded into the database - so we may treat that as the most recent since ID and miss a bunch of their tweets!",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
488835586,MDU6SXNzdWU0ODg4MzU1ODY=,4,Command for importing data from a Twitter Export file,9599,simonw,closed,0,,,,,2,2019-09-03T21:34:13Z,2019-10-11T06:45:02Z,2019-10-11T06:45:02Z,MEMBER,,"Twitter lets you export all of your data as an archive file: https://twitter.com/settings/your_twitter_data

A command for importing this data into SQLite would be extremely useful.

    $ twitter-to-sqlite import twitter.db path-to-archive.zip
",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
590669793,MDU6SXNzdWU1OTA2Njk3OTM=,40,Feature: record history of follower counts,9599,simonw,closed,0,,,,,5,2020-03-30T23:32:28Z,2020-04-01T04:13:05Z,2020-04-01T04:13:05Z,MEMBER,,"We currently over-write the follower count every time we import a tweet (when we import that user profile again):

https://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/utils.py#L293-L294

It would be neat if we noticed if that user's follower count (and maybe other counts?) had changed since we last saved them and recorded that change in a separate history table. This would be an inexpensive way of building up rough charts of follower count over time.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
591613579,MDU6SXNzdWU1OTE2MTM1Nzk=,41,"Bug: recorded a since_id for None, None",9599,simonw,closed,0,,,,,0,2020-04-01T04:29:43Z,2020-04-01T04:31:11Z,2020-04-01T04:31:11Z,MEMBER,,"This shouldn't happen in the `since_ids` table (relates to #39):

<img width=""422"" alt=""twitter__since_ids__2_rows"" src=""https://user-images.githubusercontent.com/9599/78099419-b0fed200-7396-11ea-87d8-992a171d6fab.png"">
",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/41/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
602173589,MDU6SXNzdWU2MDIxNzM1ODk=,42,Error running user-timeline with --sql and --ids together,9599,simonw,closed,0,,,,,0,2020-04-17T19:02:06Z,2020-04-17T23:34:40Z,2020-04-17T23:34:40Z,MEMBER,,"```
$ twitter-to-sqlite user-timeline tweets.db --sql='select id from users' --ids
Traceback (most recent call last):
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite"", line 11, in <module>
    load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')()
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 764, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 717, in main
    rv = self.invoke(ctx)
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py"", line 555, in invoke
    return callback(*args, **kwargs)
  File ""/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py"", line 284, in user_timeline
    ""@{:"" + str(max(len(identifier) for identifier in identifiers)) + ""}""
  File ""/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py"", line 284, in <genexpr>
    ""@{:"" + str(max(len(identifier) for identifier in identifiers)) + ""}""
TypeError: object of type 'int' has no len()
```
But this DID work - casting to strings:
```
$ twitter-to-sqlite user-timeline tweets.db --sql='select """" || id from users' --ids
... this worked ...
```",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/42/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
602176870,MDU6SXNzdWU2MDIxNzY4NzA=,43,"""twitter-to-sqlite lists"" command for retrieving a user's owned lists",9599,simonw,closed,0,,,,,1,2020-04-17T19:08:59Z,2020-04-17T23:48:28Z,2020-04-17T23:30:39Z,MEMBER,,"https://developer.twitter.com/en/docs/accounts-and-users/create-manage-lists/api-reference/get-lists-ownerships

`https://api.twitter.com/1.1/lists/ownerships.json `",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/43/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
602181581,MDU6SXNzdWU2MDIxODE1ODE=,44,"tweet[""source""] can be an empty string",9599,simonw,closed,0,,,,,0,2020-04-17T19:18:26Z,2020-04-17T22:01:44Z,2020-04-17T22:01:44Z,MEMBER,,"Got this excepion:
```
  File ""/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py"", line 641, in extract_and_save_source
    details = m.groupdict()
AttributeError: 'NoneType' object has no attribute 'groupdict'
```
I traced it back to this tweet: https://twitter.com/osder/status/578712651393576960
```
(Pdb) source_re
re.compile('<a href=""(?P<url>.*?)"".*?>(?P<name>.*?)</a>')
(Pdb) locals()['source']
''
(Pdb) u
> /Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py(393)save_tweets()
-> tweet[""source""] = extract_and_save_source(db, tweet[""source""])
(Pdb) tweet
{'created_at': '2015-03-20T00:20:22+00:00', 'id': 578712651393576960, 'full_text': '@osder', 'truncated': False, 'display_text_range': [0, 6], 'source': '', 'in_reply_to_status_id': 578712521382715392, 'in_reply_to_user_id': 1545741, 'in_reply_to_screen_name': 'osder', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 0, 'favorite_count': 0, 'favorited': False, 'retweeted': False, 'lang': 'und', 'user': 1545741}
```",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/44/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
602619330,MDU6SXNzdWU2MDI2MTkzMzA=,45,Use raise_for_status() everywhere,9599,simonw,open,0,,,,,1,2020-04-19T04:38:28Z,2020-04-19T04:39:22Z,,MEMBER,,"I keep seeing errors which I think are caused by authentication or rate limit problems but which appear to be unexpected JSON responses - presumably because they are actually an error message.

Recent example: https://github.com/simonw/jsk-fellows-on-twitter/runs/598892575

Using `response.raise_for_status()` everywhere will make these errors less confusing.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
610284471,MDU6SXNzdWU2MTAyODQ0NzE=,46,Error running 'search' for the first time,9599,simonw,closed,0,,,,,0,2020-04-30T18:11:20Z,2020-04-30T18:11:58Z,2020-04-30T18:11:58Z,MEMBER,,"```
% twitter-to-sqlite search infodemic.db '#infodemic'
Traceback (most recent call last):
  File ""/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/bin/twitter-to-sqlite"", line 11, in <module>
    load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')()
  File ""/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/Users/simon/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py"", line 867, in search
    for tweet in tweets:
  File ""/Users/simon/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py"", line 165, in fetch_timeline
    [since_type_id, since_key],
sqlite3.OperationalError: no such table: since_ids
```",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/46/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
639542974,MDU6SXNzdWU2Mzk1NDI5NzQ=,47,Fall back to FTS4 if FTS5 is not available,73579,hpk42,open,0,,,,,3,2020-06-16T10:11:23Z,2020-06-17T20:13:48Z,,NONE,,"got this with version 0.21.1 from pypi.  twitter-to-sqlite auth worked but then ""twitter-to-sqlite user-timeline USER.db"" produced a tracekback ending in ""no such module: FTS5"". ",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
663976976,MDU6SXNzdWU2NjM5NzY5NzY=,48,Add a table of contents to the README,9599,simonw,closed,0,,,,,3,2020-07-22T18:54:33Z,2020-07-23T17:46:07Z,2020-07-22T19:03:02Z,MEMBER,,Using https://github.com/jonschlinkert/markdown-toc,206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
681575714,MDExOlB1bGxSZXF1ZXN0NDY5OTQ0OTk5,49,"Document the use of --stop_after with favorites, refs #20",370930,mikepqr,closed,0,,,,,1,2020-08-19T06:10:52Z,2021-08-20T00:02:11Z,2021-08-20T00:02:11Z,CONTRIBUTOR,dogsheep/twitter-to-sqlite/pulls/49,(I discovered this trawling the issues for how to use --since with favorites),206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
488874815,MDU6SXNzdWU0ODg4NzQ4MTU=,5,Write tests that simulate the Twitter API,9599,simonw,open,0,,,,,1,2019-09-03T23:55:35Z,2019-09-03T23:56:28Z,,MEMBER,,I can use betamax for this: https://pypi.org/project/betamax/,206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
698791218,MDU6SXNzdWU2OTg3OTEyMTg=,50,"favorites --stop_after=N stops after min(N, 200)",370930,mikepqr,open,0,,,,,2,2020-09-11T03:38:14Z,2020-09-13T05:11:14Z,,CONTRIBUTOR,,"For any number greater than 200, `favorites --stop_after` stops after getting 200 tweets, e.g.
```
$ twitter-to-sqlite favorites tweets.db --stop_after=300
Importing favorites  [####################################]  199
$
```
I don't _think_ this is a limitation of the API (if you omit `--stop_after` you get some very large number, possibly all of them), so I _think_ this is a bug.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
703218448,MDU6SXNzdWU3MDMyMTg0NDg=,51,Documentation for twitter-to-sqlite fetch,9599,simonw,open,0,,,,,0,2020-09-17T02:38:10Z,2020-09-17T02:38:10Z,,MEMBER,,"It's mentioned in passing in the README but it deserves its own section:
```
$ twitter-to-sqlite fetch \
    ""https://api.twitter.com/1.1/account/verify_credentials.json"" \
    | grep '""id""' | head -n 1
```",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/51/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
745393298,MDU6SXNzdWU3NDUzOTMyOTg=,52,Discussion: Adding support for fetching only fresh tweets,4169772,fatihky,closed,0,,,,,1,2020-11-18T07:01:48Z,2020-11-18T07:12:45Z,2020-11-18T07:12:45Z,NONE,,I think it'd be very useful if this tool has an option like `--incremental` to fetch only newer tweets. This way operations could complete very fast in sequential runs. I'd want to try to implement this feature if it seems OK for this tool's purpose. ,206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/52/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
771324837,MDU6SXNzdWU3NzEzMjQ4Mzc=,53,--since support for favorites,27,anotherjesse,closed,0,,,,,1,2020-12-19T07:08:23Z,2020-12-19T07:47:11Z,2020-12-19T07:47:11Z,NONE,,"Having support for `--since` for updating your favorites would be ideal as the api is both slow and it only returns ~3k most recent favorites.

https://twittercommunity.com/t/cant-get-all-favorite-tweets-by-rest-api/22007/3

The api seems to take an optional `since_id` parameter - https://developer.twitter.com/en/docs/twitter-api/v1/tweets/post-and-engage/api-reference/get-favorites-list",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/53/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
779088071,MDU6SXNzdWU3NzkwODgwNzE=,54,Archive import appears to be broken on recent exports,21148,jacobian,open,0,,,,,5,2021-01-05T14:18:01Z,2023-01-04T11:06:55Z,,CONTRIBUTOR,,"I requested a Twitter export yesterday, and unfortunately they seem to have changed it such that `twitter-to-sqlite import` can't handle it anymore 😢 

So far I've ran into two issues. The first was easy to work around, but the second will take more investigation. If I can find the time I'll keep working on it and update this issue accordingly.

The issues (so far):

### 1. Data seems to have moved to a `data/` subdirectory

Running `twitter-to-sqlite import` on the raw zip file reports a bunch of ""not yet implemented"" errors, and then exits without actually importing anything:

```
❯ twitter-to-sqlite import tarchive.db twitter.zip
...
data/manifest: not yet implemented
data/account-creation-ip: not yet implemented
data/account-suspension: not yet implemented
... (dozens of more lines like this, including critical stuff like data/tweets) ...
```

(`tarchive.db` now exists, but is empty)

Workaround: unpack the zip file, and run `twitter-to-sqlite import tarchive.db path/to/archive/data`

That gets further, but:

### 2. Some schema(s?) have changed

At least, the `blocks` schema seems different now:

```
❯ twitter-to-sqlite import tarchive.db archive/data
direct-messages-group: not yet implemented
branch-links: not yet implemented
periscope-expired-broadcasts: not yet implemented
direct-messages: not yet implemented
mute: not yet implemented
Traceback (most recent call last):
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/bin/twitter-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/cli.py"", line 772, in import_
    archive.import_from_file(db, filepath.name, open(filepath, ""rb"").read())
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py"", line 215, in import_from_file
    to_insert = transformer(data)
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py"", line 115, in lists_member
    return {""lists-member"": _list_from_common(data)}
  File ""/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py"", line 200, in _list_from_common
    for url in block[""userListInfo""][""urls""]:
KeyError: 'urls'
```

That's as far as I got before I needed to work on something else. I'll report back if I get further!",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
779211940,MDExOlB1bGxSZXF1ZXN0NTQ5MjA0MDYz,55,Fix archive imports,21148,jacobian,closed,0,,,,,2,2021-01-05T15:54:48Z,2021-08-20T00:02:49Z,2021-08-20T00:02:49Z,CONTRIBUTOR,dogsheep/twitter-to-sqlite/pulls/55,This fixes the issues discussed in #54,206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
796736607,MDU6SXNzdWU3OTY3MzY2MDc=,56,Not all quoted statuses get fetched?,42315895,gsajko,closed,0,,,,,3,2021-01-29T09:48:44Z,2021-02-03T10:36:36Z,2021-02-03T10:36:36Z,NONE,,"
![image](https://user-images.githubusercontent.com/42315895/106259325-5f75dc80-621f-11eb-8311-db8f2fe2a257.png)

In my database I have 13300 quote tweets, but eta 3600 have `quoted_status` empty.

I fetched some of them using `https://api.twitter.com/1.1/statuses/show.json?id=xx` and they did have ids of quoted tweets.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
907645813,MDU6SXNzdWU5MDc2NDU4MTM=,57,"Error: Use either --since or --since_id, not both",42904,rubenv,closed,0,,,,,6,2021-05-31T18:11:04Z,2021-08-20T00:01:31Z,2021-08-20T00:01:31Z,CONTRIBUTOR,,"I'm using the following command:

```
twitter-to-sqlite user-timeline -a twitter-auth.json twitter/tweets.db --since
```

Which gives the following error:
```
Error: Use either --since or --since_id, not both
```

Running without `--since`.

```
Traceback (most recent call last):
  File ""/usr/local/bin/twitter-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1137, in __call__
    return self.main(*args, **kwargs)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1062, in main
    rv = self.invoke(ctx)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 763, in invoke
    return __callback(*args, **kwargs)
  File ""/usr/local/lib/python3.9/site-packages/twitter_to_sqlite/cli.py"", line 317, in user_timeline
    for tweet in bar:
  File ""/usr/local/lib/python3.9/site-packages/click/_termui_impl.py"", line 328, in generator
    for rv in self.iter:
  File ""/usr/local/lib/python3.9/site-packages/twitter_to_sqlite/utils.py"", line 234, in fetch_user_timeline
    yield from fetch_timeline(
  File ""/usr/local/lib/python3.9/site-packages/twitter_to_sqlite/utils.py"", line 202, in fetch_timeline
    raise Exception(str(tweets[""errors""]))
Exception: [{'code': 44, 'message': 'since_id parameter is invalid.'}]
```

```
Python 3.9.5
twitter-to-sqlite, version 0.21.3
```",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57/reactions"", ""total_count"": 4, ""+1"": 4, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
984939366,MDU6SXNzdWU5ODQ5MzkzNjY=,58,"Error: Use either --since or --since_id, not both - still broken",42904,rubenv,closed,0,,,,,1,2021-09-01T09:45:28Z,2021-09-21T17:37:41Z,2021-09-21T17:37:41Z,CONTRIBUTOR,,"Hi Simon,

It appears the fix for #57 doesn't fix things for me:

```
$ twitter-to-sqlite --version
twitter-to-sqlite, version 0.21.4
$ python --version
Python 3.9.6
```

```
$ twitter-to-sqlite home-timeline -a twitter-auth.json twitter/timeline.db --since
Importing tweets
Error: Use either --since or --since_id, not both
```

Is there any way I can help debug this?",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/58/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
984942782,MDExOlB1bGxSZXF1ZXN0NzI0MzE3NjUw,59,"Fix for since_id bug, closes #58",42904,rubenv,closed,0,,,,,1,2021-09-01T09:49:09Z,2021-09-21T17:37:40Z,2021-09-21T17:37:40Z,CONTRIBUTOR,dogsheep/twitter-to-sqlite/pulls/59,Fixes remaining instances of this bug,206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/59/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
489419782,MDU6SXNzdWU0ODk0MTk3ODI=,6,Extract extended_entities into a media table,9599,simonw,closed,0,,,,,0,2019-09-04T21:59:10Z,2019-09-04T22:08:01Z,2019-09-04T22:08:01Z,MEMBER,,"<img width=""747"" alt=""cleo__select___from_tweets_order_by_id_limit_101"" src=""https://user-images.githubusercontent.com/9599/64294799-880f7f00-cf24-11e9-83b8-75c1ee54182c.png"">
",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1063982712,I_kwDODEm0Qs4_axZ4,60,Execution on Windows,1733616,bernard01,open,0,,,,,1,2021-11-26T00:24:34Z,2022-10-14T16:58:27Z,,NONE,,"My installation on Windows using pip has been successful. I have Python 3.6.

How do I run twitter-to-sqlite? I cannot even figure out how ""auth"" is a command. I have python on my path:
C:\prog\python\Python36;C:\prog\python\Python36\Scripts

Where should the commands be executed, and where are the files created?

Could some basics please be added to the documentation to get beginners started?",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/60/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1077560091,I_kwDODEm0Qs5AOkMb,61,"Data Pull fails for ""Essential"" level access to the Twitter API (for Documentation)",57161638,jmnickerson05,open,0,,,,,1,2021-12-11T14:59:41Z,2022-10-31T14:47:58Z,,NONE,,"Per Twitter documentation:
https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api#v2-access-leve

This isn't any fault of twitter-to-sqlite of course, but it should probably be documented as a side-note.

![image](https://user-images.githubusercontent.com/57161638/145681272-8c85b3b9-be95-44ff-9760-1bafa4917ce2.png)

And this is how I'm surfacing the message from utils.py:
![image](https://user-images.githubusercontent.com/57161638/145681005-2776c0ad-9822-4461-b43a-450ab2e828eb.png)
",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/61/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1088816961,I_kwDODEm0Qs5A5gdB,62,KeyError: 'created_at' for private accounts?,6764957,swyxio,closed,0,,,,,2,2021-12-26T17:51:51Z,2022-03-12T02:36:32Z,2022-02-24T18:10:18Z,NONE,,"hey Simon!

i was running `twitter-to-sqlite user-timeline twitter.db` for [my private alt](https://twitter.com/swyxio) and ran into this error:

<details>
<summary>

![image](https://user-images.githubusercontent.com/6764957/147416165-46b69c30-100a-406f-8534-8612b75547ae.png)

</summary>



```bash
Traceback (most recent call last):
  File ""/Users/swyx/Work/datasette/env/bin/twitter-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py"", line 1128, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py"", line 1053, in main
    rv = self.invoke(ctx)
  File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py"", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py"", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py"", line 754, in invoke
    return __callback(*args, **kwargs)
  File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/cli.py"", line 291, in user_timeline
    profile = utils.get_profile(db, session, **kwargs)
  File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py"", line 133, in get_profile
    save_users(db, [profile])
  File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py"", line 453, in save_users
    transform_user(user)
  File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py"", line 285, in transform_user
    user[""created_at""] = parser.parse(user[""created_at""])
KeyError: 'created_at'
```

</details>


this looks awfully like #37  but it can't be, because i'm authed into my account and obviously i have perms to read my own account. wonder if there's any diagnostic methods i should apply here? just filing an issue for others to find while i investigate.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/62/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1091850530,I_kwDODEm0Qs5BFFEi,63,Import archive error 'withheld_in_countries',521097,pauloxnet,open,0,,,,,0,2022-01-01T16:58:59Z,2022-01-01T16:58:59Z,,NONE,,"Importing the twitter archive  I received this error:
```bash
$ twitter-to-sqlite import archive.db twitter-2021-12-31-<hash>.zip 
birdwatch-note-rating: not yet implemented
birdwatch-note: not yet implemented
branch-links: not yet implemented
community-tweet: not yet implemented
contact: not yet implemented
device-token: not yet implemented
direct-message-mute: not yet implemented
mute: not yet implemented
periscope-account-information: not yet implemented
periscope-ban-information: not yet implemented
periscope-broadcast-metadata: not yet implemented
periscope-comments-made-by-user: not yet implemented
periscope-expired-broadcasts: not yet implemented
periscope-followers: not yet implemented
periscope-profile-description: not yet implemented
professional-data: not yet implemented
protected-history: not yet implemented
reply-prompt: not yet implemented
screen-name-change: not yet implemented
smartblock: not yet implemented
spaces-metadata: not yet implemented
sso: not yet implemented
Traceback (most recent call last):
  File ""/home/paulox/.virtualenvs/dogsheep/bin/twitter-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py"", line 1128, in __call__
    return self.main(*args, **kwargs)
  File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py"", line 1053, in main
    rv = self.invoke(ctx)
  File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py"", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py"", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py"", line 754, in invoke
    return __callback(*args, **kwargs)
  File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/cli.py"", line 759, in import_
    archive.import_from_file(db, filename, content)
  File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/archive.py"", line 246, in import_from_file
    db[table_name].insert_all(rows, pk=pk, replace=True)
  File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2625, in insert_all
    self.insert_chunk(
  File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2406, in insert_chunk
    result = self.db.execute(query, params)
  File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py"", line 422, in execute
    return self.conn.execute(sql, parameters)
sqlite3.OperationalError: table archive_tweet has no column named withheld_in_countries
```

I found only a single tweet with the key `withheld_in_countries` in `tweet.js` that seems the problems:
```JSON
[
{
    ""tweet"" : {
      ""retweeted"" : false,
      ""source"" : ""<a href=\""http://twitter.com/download/android\"" rel=\""nofollow\"">Twitter for Android</a>"",
      ""entities"" : {
        ""hashtags"" : [
          {
            ""text"" : ""NowOnAndroid"",
            ""indices"" : [
              ""64"",
              ""77""
            ]
          }
        ],
        ""symbols"" : [ ],
        ""user_mentions"" : [
          {
            ""name"" : ""Periscope"",
            ""screen_name"" : ""PeriscopeCo"",
            ""indices"" : [
              ""3"",
              ""15""
            ],
            ""id_str"" : ""1111111111"",
            ""id"" : ""222222222""
          }
        ],
        ""urls"" : [
          {
            ""url"" : ""https://t.co/xxxxxxxxx"",
            ""expanded_url"" : ""https://vine.co/v/xxxxxxxxx"",
            ""display_url"" : ""vine.co/v/xxxxxxxxxx"",
            ""indices"" : [
              ""78"",
              ""101""
            ]
          }
        ]
      },
      ""display_text_range"" : [
        ""0"",
        ""101""
      ],
      ""favorite_count"" : ""0"",
      ""id_str"" : ""1111111111111111111111"",
      ""truncated"" : false,
      ""retweet_count"" : ""0"",
      ""withheld_in_countries"" : [
        ""TR""
      ],
      ""id"" : ""000000000000000000"",
      ""possibly_sensitive"" : false,
      ""created_at"" : ""Fri Aug 14 06:04:03 +0000 2015"",
      ""favorited"" : false,
      ""full_text"" : ""RT @periscopeco: Travel the world. LIVE. The Global Map is here #NowOnAndroid https://t.co/NZXdsPWROk"",
      ""lang"" : ""en""
    }
  }
  ]
```

I solved the error removing the key from the `tweet.js` but I'm reporting this error to improve the project.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/63/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1097332098,I_kwDODEm0Qs5BZ_WC,64,Include all entities for tweets,111631,max,open,0,,,,,0,2022-01-09T23:35:28Z,2022-01-09T23:35:28Z,,NONE,,"Per our conversation [on Twitter](https://twitter.com/mschoening/status/1480312477246054401):

It would be neat if all entities (including URLs) were captured. This way you can ensure, that URLs are parsed out exactly the same way Twitter parses URLs – we all know parsing URLs with a regex ain't fun.

Right now, I believe the tool filters out all entities that are not of type `media`.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/64/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1160327106,PR_kwDODEm0Qs4z_V3w,65,"Update Twitter dev link, clarify apps vs projects",2657547,rixx,open,0,,,,,0,2022-03-05T11:56:08Z,2022-03-05T11:56:08Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/twitter-to-sqlite/pulls/65,"Twitter pushes you heavily towards v2 projects instead of v1 apps – I know the README mentions v1 API compatibility at the top, but I still nearly got turned around here.",206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/65/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1244082183,PR_kwDODEm0Qs44PPLy,66,Ageinfo workaround,11887,ashanan,open,0,,,,,0,2022-05-21T21:08:29Z,2022-05-21T21:09:16Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/twitter-to-sqlite/pulls/66,"I'm not sure if this is due to a new format or just because my ageinfo file is blank, but trying to import an archive would crash when it got to that file.  This PR adds a guard clause in the `ageinfo` transformer and sets a default value that doesn't throw an exception.  Seems likely to be the same issue mentioned by danp in https://github.com/dogsheep/twitter-to-sqlite/issues/54, my ageinfo file looks the same.  Added that same ageinfo file to the test archive as well to help confirm my workaround didn't break anything.

Let me know if you want any changes!",206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/66/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1513237712,PR_kwDODEm0Qs5GUoG_,67,Add support for app-only bearer tokens,26161409,sometimes-i-send-pull-requests,open,0,,,,,0,2022-12-28T23:31:20Z,2022-12-28T23:31:20Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/twitter-to-sqlite/pulls/67,"Previously, twitter-to-sqlite only supported OAuth1 authentication, and the token must be on behalf of a user.  However, Twitter also supports application-only bearer tokens, documented here:
https://developer.twitter.com/en/docs/authentication/oauth-2-0/bearer-tokens This PR adds support to twitter-to-sqlite for using application-only bearer tokens.  To use, the auth.json file just needs to contain a ""bearer_token"" key instead of ""api_key"", ""api_secret_key"", etc.",206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/67/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1513237982,PR_kwDODEm0Qs5GUoKL,68,Archive: Import mute table,26161409,sometimes-i-send-pull-requests,open,0,,,,,0,2022-12-28T23:32:06Z,2022-12-28T23:32:06Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/twitter-to-sqlite/pulls/68,,206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/68/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1513238152,PR_kwDODEm0Qs5GUoMM,69,Archive: Import new tweets table name,26161409,sometimes-i-send-pull-requests,open,0,,,,,0,2022-12-28T23:32:44Z,2022-12-28T23:32:44Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/twitter-to-sqlite/pulls/69,"Given the code here, it seems like in the past this file was named ""tweet.js"".  In recent exports, it's named ""tweets.js"".  The archive importer needs to be modified to take this into account.  Existing logic is reused for importing this table.  (However, the resulting table name will be different, matching the different file name -- archive_tweets, rather than archive_tweet).",206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/69/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
490798130,MDU6SXNzdWU0OTA3OTgxMzA=,7,users-lookup command for fetching users,9599,simonw,closed,0,,,,,0,2019-09-08T19:47:59Z,2019-09-08T20:32:13Z,2019-09-08T20:32:13Z,MEMBER,,"https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-users-lookup
```
https://api.twitter.com/1.1/users/lookup.json?user_id=783214,6253282
https://api.twitter.com/1.1/users/lookup.json?screen_name=simonw,cleopaws
```
CLI design:
```
$ twitter-to-sqlite users-lookup simonw cleopaws
$ twitter-to-sqlite users-lookup 783214 6253282 --ids
```",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/7/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1513238314,PR_kwDODEm0Qs5GUoN6,70,Archive: Import Twitter Circle data,26161409,sometimes-i-send-pull-requests,open,0,,,,,0,2022-12-28T23:33:09Z,2022-12-28T23:33:09Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/twitter-to-sqlite/pulls/70,,206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/70/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1513238455,PR_kwDODEm0Qs5GUoPm,71,"Archive: Fix ""ni devices"" typo in importer",26161409,sometimes-i-send-pull-requests,open,0,,,,,0,2022-12-28T23:33:31Z,2022-12-28T23:33:31Z,,FIRST_TIME_CONTRIBUTOR,dogsheep/twitter-to-sqlite/pulls/71,,206156866,twitter-to-sqlite,pull,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/71/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1524431805,I_kwDODEm0Qs5a3Pu9,72,"Import thread, including self- and others' replies",601708,mcint,open,0,,,,,0,2023-01-08T09:51:06Z,2023-01-08T09:51:06Z,,NONE,,"statuses-lookup, home-timeline, mentions (only for auth'ed user) don't cover this.

`twitter-to-sqlite fetch-thread tw-group1.db 1234123412341234`

twitter-to-sqlite focuses on archiving users, but does not easily support archiving conversations or community activity.

For reference, this is [implemented in twarc](https://sourcegraph.com/github.com/DocNow/twarc/-/blob/twarc/client.py?L708-766&subtree=true), using a search, optionally recursively.

Other research suggests that this formerly, or currently, requires a [search query](https://stackoverflow.com/a/30480103/1020467), use of [undocumented `related_results` api](https://stackoverflow.com/a/9419346/1020467), or with requested inclusion of [newer conversation_id](https://stackoverflow.com/a/68115718/1020467) with subsequent query.

",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/72/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1816830546,I_kwDODEm0Qs5sSqJS,73,Twitter v1 API shutdown,6341745,david-perez,open,0,,,,,0,2023-07-22T16:57:41Z,2023-07-22T16:57:41Z,,NONE,,"I've been using this project reliably over the past two years to periodically download my liked tweets, but unfortunately since 19th July I get:

```
[2023-07-19 21:00:04.937536]   File ""/home/pi/code/liked-tweets/lib/python3.7/site-packages/twitter_to_sqlite/utils.py"", line 202, in fetch_timeline
[2023-07-19 21:00:04.937606]     raise Exception(str(tweets[""errors""]))
[2023-07-19 21:00:04.937678] Exception: [{'message': 'You currently have access to a subset of Twitter API v2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only. If you need access to this endpoint, you may need a different access level. You can learn more here: https://developer.twitter.com/en/portal/product', 'code': 453}]
```

It appears like Twitter has now shut down their v1 endpoints, which is rather gracious of them, considering they [announced they'd be deprecated on 29th April](https://twittercommunity.com/t/reminder-to-migrate-to-the-new-free-basic-or-enterprise-plans-of-the-twitter-api/189737).

Unfortunately [retrieving likes using the v2 API](https://developer.twitter.com/en/docs/twitter-api/tweets/likes/introduction) is not part of their [free plan](https://developer.twitter.com/en/portal/products). In fact, with the free plan one can only post and delete tweets and retrieve information about oneself.

So I'm afraid this is the end of this very nice project. It was very useful, thank you!
",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/73/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",,
490803176,MDU6SXNzdWU0OTA4MDMxNzY=,8,--sql and --attach options for feeding commands from SQL queries,9599,simonw,closed,0,,,,,4,2019-09-08T20:35:49Z,2020-03-20T23:13:01Z,2020-03-20T23:13:01Z,MEMBER,,"Say you want to fetch Twitter profiles for a list of accounts that are stored in another database:

    $ twitter-to-sqlite users-lookup users.db --attach attending.db \
        --sql ""select Twitter from attending.attendes where Twitter is not null""

The SQL query you feed in is expected to return a list of screen names suitable for processing further by the command.

Should be supported by all three of:

- [x] `twitter-to-sqlite users-lookup`
- [x] `twitter-to-sqlite user-timeline`
- [x] `twitter-to-sqlite followers` and `friends`

The `--attach` option allows other SQLite databases to be attached to the connection. Without it the SQL query will have to read from the single attached database.",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
491791152,MDU6SXNzdWU0OTE3OTExNTI=,9,followers-ids and friends-ids subcommands,9599,simonw,closed,0,,,,,1,2019-09-10T16:58:15Z,2019-09-10T17:36:55Z,2019-09-10T17:36:55Z,MEMBER,,"These will import follower and friendship IDs into the following tables, using these APIs:

https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-followers-ids
https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-friends-ids",206156866,twitter-to-sqlite,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/9/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
267513424,MDU6SXNzdWUyNjc1MTM0MjQ=,1,Addressable pages for every row in a table,9599,simonw,closed,0,,,2857392,Ship first public release,6,2017-10-23T00:44:16Z,2017-10-24T14:11:04Z,2017-10-24T14:11:03Z,OWNER,,"    /database-name-7sha256/table-name/compound-pk
    /database-name-7sha256/table-name/compound-pk.json

Tricky part will be figuring out what the private key is - especially since it could be a compound primary key and it might involve different data types.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
267517381,MDU6SXNzdWUyNjc1MTczODE=,10,Set up Travis,9599,simonw,closed,0,,,2859414,v1 stretch goals,1,2017-10-23T01:29:07Z,2017-11-04T23:48:57Z,2017-11-04T23:48:57Z,OWNER,,,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/10/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
274160723,MDU6SXNzdWUyNzQxNjA3MjM=,100,TemplateAssertionError: no filter named 'tojson',13304454,coisnepe,closed,0,,,,,2,2017-11-15T13:43:41Z,2017-11-16T09:25:10Z,2017-11-16T00:14:13Z,NONE,,"A 500 error is raised upon clicking on the name of a table on the homepage, say _http://0.0.0.0:8001/_ to _http://0.0.0.0:8001/test_check-c1f4771/users_ The API part seems to function as intended, though...

```
2017-11-15 14:33:57 - (sanic)[ERROR]: Traceback (most recent call last):
  File ""/usr/local/lib/python3.5/dist-packages/sanic/app.py"", line 503, in handle_request
    response = await response
  File ""/usr/local/lib/python3.5/dist-packages/datasette/app.py"", line 155, in get
    return await self.view_get(request, name, hash, **kwargs)
  File ""/usr/local/lib/python3.5/dist-packages/datasette/app.py"", line 219, in view_get
    **context,
  File ""/usr/local/lib/python3.5/dist-packages/sanic_jinja2/__init__.py"", line 84, in render
    return html(self.render_string(template, request, **context))
  File ""/usr/local/lib/python3.5/dist-packages/sanic_jinja2/__init__.py"", line 81, in render_string
    return self.env.get_template(template).render(**context)
  File ""/usr/lib/python3/dist-packages/jinja2/environment.py"", line 812, in get_template
    return self._load_template(name, self.make_globals(globals))
  File ""/usr/lib/python3/dist-packages/jinja2/environment.py"", line 786, in _load_template
    template = self.loader.load(self, name, globals)
  File ""/usr/lib/python3/dist-packages/jinja2/loaders.py"", line 125, in load
    code = environment.compile(source, name, filename)
  File ""/usr/lib/python3/dist-packages/jinja2/environment.py"", line 565, in compile
    self.handle_exception(exc_info, source_hint=source_hint)
  File ""/usr/lib/python3/dist-packages/jinja2/environment.py"", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File ""/usr/lib/python3/dist-packages/jinja2/_compat.py"", line 37, in reraise
    raise value.with_traceback(tb)
  File ""/usr/local/lib/python3.5/dist-packages/datasette/templates/table.html"", line 29, in template
    <pre>params = {{ query.params|tojson(4) }}</pre>
  File ""/usr/lib/python3/dist-packages/jinja2/environment.py"", line 515, in _generate
    return generate(source, self, name, filename, defer_init=defer_init)
  File ""/usr/lib/python3/dist-packages/jinja2/compiler.py"", line 62, in generate
    generator.visit(node)
  File ""/usr/lib/python3/dist-packages/jinja2/visitor.py"", line 38, in visit
    return f(node, *args, **kwargs)
  File ""/usr/lib/python3/dist-packages/jinja2/compiler.py"", line 849, in visit_Template
    self.blockvisit(block.body, block_frame)
  File ""/usr/lib/python3/dist-packages/jinja2/compiler.py"", line 492, in blockvisit
    self.visit(node, frame)
  File ""/usr/lib/python3/dist-packages/jinja2/visitor.py"", line 38, in visit
    return f(node, *args, **kwargs)
  File ""/usr/lib/python3/dist-packages/jinja2/compiler.py"", line 1172, in visit_If
    self.blockvisit(node.body, if_frame)
  File ""/usr/lib/python3/dist-packages/jinja2/compiler.py"", line 492, in blockvisit
    self.visit(node, frame)
  File ""/usr/lib/python3/dist-packages/jinja2/visitor.py"", line 38, in visit
    return f(node, *args, **kwargs)
  File ""/usr/lib/python3/dist-packages/jinja2/compiler.py"", line 1353, in visit_Output
    self.visit(argument, frame)
  File ""/usr/lib/python3/dist-packages/jinja2/visitor.py"", line 38, in visit
    return f(node, *args, **kwargs)
  File ""/usr/lib/python3/dist-packages/jinja2/compiler.py"", line 1565, in visit_Filter
    self.fail('no filter named %r' % node.name, node.lineno)
  File ""/usr/lib/python3/dist-packages/jinja2/compiler.py"", line 427, in fail
    raise TemplateAssertionError(msg, lineno, self.name, self.filename)
jinja2.exceptions.TemplateAssertionError: no filter named 'tojson'

2017-11-15 14:33:57 - (network)[INFO][127.0.0.1:41316]: GET http://0.0.0.0:8001/test_check-c1f4771/users  500 144
2017-11-15 14:33:57 - (network)[INFO][127.0.0.1:41316]: GET http://0.0.0.0:8001/favicon.ico  200 0
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/100/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
717746043,MDExOlB1bGxSZXF1ZXN0NTAwMjU2NDg1,1000,datasette.client internal requests mechanism,9599,simonw,closed,0,9599,simonw,5971510,Datasette 0.50,18,2020-10-08T23:58:25Z,2020-10-09T16:11:26Z,2020-10-09T16:11:25Z,OWNER,simonw/datasette/pulls/1000,Refs #943,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1000/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
717768441,MDU6SXNzdWU3MTc3Njg0NDE=,1001,OPTIONS requests return a 500 error,9599,simonw,closed,0,,,5971510,Datasette 0.50,8,2020-10-09T00:57:13Z,2020-10-09T01:44:41Z,2020-10-09T01:43:58Z,OWNER,,"```
% curl -vv -XOPTIONS https://latest.datasette.io/
*   Trying 216.58.195.83:443...
> OPTIONS / HTTP/1.1
> Host: latest.datasette.io
> User-Agent: curl/7.70.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1001/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
717783692,MDU6SXNzdWU3MTc3ODM2OTI=,1002,Release notes for Datasette 0.50,9599,simonw,closed,0,,,5971510,Datasette 0.50,1,2020-10-09T01:45:00Z,2020-10-09T17:52:54Z,2020-10-09T17:52:53Z,OWNER,,https://github.com/simonw/datasette/compare/0.49.1...c12b7a5def7028845a54a9fdac4052a87a0a8bb8,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1002/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
718238967,MDU6SXNzdWU3MTgyMzg5Njc=,1003,from_json jinja2 filter,649467,mhalle,open,0,,,,,4,2020-10-09T15:30:58Z,2020-10-09T17:17:07Z,,NONE,,"When JSON fields are rendered in a jinja2 template, it is handy to be able to manipulate them as data (e.g., iterate over an array of values). 

Ansible has a ""from_json"" function, which just called json.loads. It's a trivial as a datasette plugin, but it seems generally useful. Does it makes sense to add it directly into the app?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1003/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
718255803,MDU6SXNzdWU3MTgyNTU4MDM=,1004,Replace MockRequest with Request.fake(),9599,simonw,closed,0,,,5971510,Datasette 0.50,0,2020-10-09T15:55:28Z,2020-10-09T16:26:24Z,2020-10-09T16:26:24Z,OWNER,,"This code: https://github.com/simonw/datasette/blob/7249ac5ca04b5ddc6517750326ee7e522cc49145/tests/utils.py#L1-L8

Predates the introduction of this class method: https://github.com/simonw/datasette/blob/7249ac5ca04b5ddc6517750326ee7e522cc49145/datasette/utils/asgi.py#L108-L121",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1004/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
718259202,MDU6SXNzdWU3MTgyNTkyMDI=,1005,Remove xfail tests when new httpx is released,9599,simonw,closed,0,,,3268330,Datasette 1.0,3,2020-10-09T16:00:19Z,2021-02-28T22:41:08Z,2021-02-28T22:41:08Z,OWNER,,"> My `httpx` pull request adding `raw_path` support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet.
>
> I'm going to mark these tests as `xfail` so I can land this change - I'll remove that once an `httpx` release comes out that I can use to get the tests passing.
>
_Originally posted by @simonw in https://github.com/simonw/datasette/pull/1000#issuecomment-706263157_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1005/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
718264811,MDU6SXNzdWU3MTgyNjQ4MTE=,1006,Documentation for datasette.client,9599,simonw,closed,0,,,5971510,Datasette 0.50,2,2020-10-09T16:09:02Z,2020-10-09T17:22:31Z,2020-10-09T17:20:37Z,OWNER,,"> I'm going to document this in a separate issue.

_Originally posted by @simonw in https://github.com/simonw/datasette/pull/1000#issuecomment-706269271_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1006/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
718272593,MDU6SXNzdWU3MTgyNzI1OTM=,1007,set-env and add-path commands have been deprecated,9599,simonw,open,0,,,,,1,2020-10-09T16:21:18Z,2020-10-09T16:23:51Z,,OWNER,,"https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/

> Starting today runner version 2.273.5 will begin to warn you if you use the `add-path` or `set-env` commands. We are monitoring telemetry for the usage of these commands and plan to fully disable them in the future.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1007/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
718395987,MDExOlB1bGxSZXF1ZXN0NTAwNzk4MDkx,1008,Add json_loads and json_dumps jinja2 filters,649467,mhalle,open,0,,,,,1,2020-10-09T20:11:34Z,2020-12-15T02:30:28Z,,FIRST_TIME_CONTRIBUTOR,simonw/datasette/pulls/1008,,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1008/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
274161964,MDU6SXNzdWUyNzQxNjE5NjQ=,101,TemplateAssertionError: no filter named 'tojson',450244,eaubin,closed,0,,,,,1,2017-11-15T13:47:32Z,2017-11-15T13:48:55Z,2017-11-15T13:48:55Z,NONE,,"I get an exception clicking on the table link: 

```
2017-11-15 08:40:10 - (sanic)[ERROR]: Traceback (most recent call last):
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/sanic/app.py"", line 503, in handle_request
    response = await response
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/datasette/app.py"", line 155, in get
    return await self.view_get(request, name, hash, **kwargs)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/datasette/app.py"", line 219, in view_get
    **context,
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/sanic_jinja2/__init__.py"", line 84, in render
    return html(self.render_string(template, request, **context))
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/sanic_jinja2/__init__.py"", line 81, in render_string
    return self.env.get_template(template).render(**context)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py"", line 812, in get_template
    return self._load_template(name, self.make_globals(globals))
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py"", line 786, in _load_template
    template = self.loader.load(self, name, globals)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/loaders.py"", line 125, in load
    code = environment.compile(source, name, filename)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py"", line 565, in compile
    self.handle_exception(exc_info, source_hint=source_hint)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py"", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/_compat.py"", line 37, in reraise
    raise value.with_traceback(tb)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/datasette/templates/table.html"", line 29, in template
    <pre>params = {{ query.params|tojson(4) }}</pre>
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/environment.py"", line 515, in _generate
    return generate(source, self, name, filename, defer_init=defer_init)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py"", line 62, in generate
    generator.visit(node)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/visitor.py"", line 38, in visit
    return f(node, *args, **kwargs)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py"", line 849, in visit_Template
    self.blockvisit(block.body, block_frame)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py"", line 492, in blockvisit
    self.visit(node, frame)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/visitor.py"", line 38, in visit
    return f(node, *args, **kwargs)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py"", line 1172, in visit_If
    self.blockvisit(node.body, if_frame)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py"", line 492, in blockvisit
    self.visit(node, frame)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/visitor.py"", line 38, in visit
    return f(node, *args, **kwargs)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py"", line 1353, in visit_Output
    self.visit(argument, frame)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/visitor.py"", line 38, in visit
    return f(node, *args, **kwargs)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py"", line 1565, in visit_Filter
    self.fail('no filter named %r' % node.name, node.lineno)
  File ""/Users/e/anaconda3-4.2.0/lib/python3.5/site-packages/jinja2/compiler.py"", line 427, in fail
    raise TemplateAssertionError(msg, lineno, self.name, self.filename)
jinja2.exceptions.TemplateAssertionError: no filter named 'tojson'
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/101/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
718484082,MDU6SXNzdWU3MTg0ODQwODI=,1010,json / CSV links are broken in Datasette 0.50,9599,simonw,closed,0,,,,,3,2020-10-10T00:07:42Z,2020-10-10T02:32:03Z,2020-10-10T02:32:03Z,OWNER,,"e.g. on https://latest.datasette.io/fixtures/sortable

That export link block is  broken. The HTML is:

```html
<p class=""export-links"">
  This data as <a href=""//fixtures/sortable.json"">json</a>,
  <a href=""//fixtures/sortable.csv?_size=max"">CSV</a> (<a href=""#export"">advanced</a>)
</p>
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1010/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
718521469,MDU6SXNzdWU3MTg1MjE0Njk=,1011,column name links broken in 0.50.1,649467,mhalle,closed,0,,,,,4,2020-10-10T03:37:51Z,2020-10-10T04:09:32Z,2020-10-10T03:52:07Z,NONE,,"I just upgraded from 0.49 to 0.50.1 and found that the links on column headers are broken. 

If I inspect the source, they have a leading ""//"" (without host or port) rather than including base_url like other links on the page do. The links in the ""gears"" menu for each column do work.

 I don't have custom templates for my project. 
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1011/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
718540751,MDU6SXNzdWU3MTg1NDA3NTE=,1012,For 1.0 update trove classifier in setup.py,9599,simonw,open,0,,,3268330,Datasette 1.0,5,2020-10-10T05:52:08Z,2021-11-16T13:18:36Z,,OWNER,,    Development Status :: 5 - Production/Stable,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1012/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
718723543,MDU6SXNzdWU3MTg3MjM1NDM=,1014,Add Link: pagination HTTP headers,9599,simonw,closed,0,,,6026070,0.51,6,2020-10-10T23:42:40Z,2020-10-23T19:44:05Z,2020-10-11T00:18:51Z,OWNER,,Spun off from #782. These can go on all of the JSON endpoints that support pagination.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1014/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
718910318,MDU6SXNzdWU3MTg5MTAzMTg=,1015,Research: could Datasette install its own plugins?,9599,simonw,open,0,,,,,1,2020-10-11T19:33:06Z,2020-10-11T19:35:04Z,,OWNER,,"It would be cool if Datasette could offer a plugin browsing interface where users could install plugins by clicking ""Install"" on them - similar to how VS Code extensions work.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1015/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
718953669,MDU6SXNzdWU3MTg5NTM2Njk=,1016,"Add a ""delete"" icon next to filters (in addition to ""remove filter"")",9599,simonw,closed,0,,,6026070,0.51,3,2020-10-11T23:49:53Z,2020-10-23T19:44:06Z,2020-10-12T03:01:58Z,OWNER,,"The ""remove filter"" option in the select box is not very discoverable.

<img width=""549"" alt=""evernote__notes__85_rows_where_where_source-url_is_not_blank_sorted_by_id"" src=""https://user-images.githubusercontent.com/9599/95693532-b8265880-0be1-11eb-971e-038dede14cdb.png"">

It would be good to have an additional remove icon, pointed to by the pink arrow, which removes a specific selected filter.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1016/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
719381863,MDExOlB1bGxSZXF1ZXN0NTAxNTc5MDg4,1017,"Update janus requirement from <0.6,>=0.4 to >=0.4,<0.7",27856297,dependabot-preview[bot],closed,0,,,,,1,2020-10-12T13:29:46Z,2020-10-14T21:52:08Z,2020-10-14T21:52:07Z,CONTRIBUTOR,simonw/datasette/pulls/1017,"Updates the requirements on [janus](https://github.com/aio-libs/janus) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/aio-libs/janus/blob/master/CHANGES.rst"">janus's changelog</a>.</em></p>
<blockquote>
<h1>Changes</h1>
<h2>0.5.0 (2020-04-23)</h2>
<ul>
<li>Remove explicit loop arguments and forbid creating queues outside event loops <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/246"">#246</a></li>
</ul>
<h2>0.4.0 (2018-07-28)</h2>
<ul>
<li>Add <code>py.typed</code> macro <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/89"">#89</a></li>
<li>Drop python 3.4 support and fix minimal version python3.5.3 <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/88"">#88</a></li>
<li>Add property with that indicates if queue is closed <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/86"">#86</a></li>
</ul>
<h2>0.3.2 (2018-07-06)</h2>
<ul>
<li>Fixed python 3.7 support <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/97"">#97</a></li>
</ul>
<h2>0.3.1 (2018-01-30)</h2>
<ul>
<li>Fixed bug with join() in case tasks are added by sync_q.put() <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/75"">#75</a></li>
</ul>
<h2>0.3.0 (2017-02-21)</h2>
<ul>
<li>Expose unfinished_tasks property <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/34"">#34</a></li>
</ul>
<h2>0.2.4 (2016-12-05)</h2>
<ul>
<li>Restore tarball deploying</li>
</ul>
<h2>0.2.3 (2016-07-12)</h2>
<ul>
<li>Fix exception type</li>
</ul>
<h2>0.2.2 (2016-07-11)</h2>
<ul>
<li>Update asyncio.async() to use asyncio.ensure_future() <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/6"">#6</a></li>
</ul>
<h2>0.2.1 (2016-03-24)</h2>
<ul>
<li>Fix python setup.py test command <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/4"">#4</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/aio-libs/janus/commit/d1867245cc803c2b9540b252928343415c16a1d7""><code>d186724</code></a> Fix yaml</li>
<li><a href=""https://github.com/aio-libs/janus/commit/dbb2d7bcd5a3b916c12ade55a9af3a27ff44ec50""><code>dbb2d7b</code></a> Fix deploy script</li>
<li><a href=""https://github.com/aio-libs/janus/commit/18df625c1c8ce0876db57370dea43c623657b8ba""><code>18df625</code></a> Bump to 0.6.0</li>
<li><a href=""https://github.com/aio-libs/janus/commit/a50b7ec0b9bf04d5c86a55fd255969984f10d93c""><code>a50b7ec</code></a> Test on ubuntu only, the library has no platform specific dependencies</li>
<li><a href=""https://github.com/aio-libs/janus/commit/b599d94fbfd17c179feea025b7c068f7454b968a""><code>b599d94</code></a> Fix workflow</li>
<li><a href=""https://github.com/aio-libs/janus/commit/9897fca5e2196c8036324c313f486616850c3eb2""><code>9897fca</code></a> Setup github workflows</li>
<li><a href=""https://github.com/aio-libs/janus/commit/cde69186efea8ae76f522ea0655070f110bffacf""><code>cde6918</code></a> Drop Python 3.5, test on Python 3.9, format with black/isort</li>
<li><a href=""https://github.com/aio-libs/janus/commit/5f04d79a6a51bd285bba10494f5e0ed2196339d7""><code>5f04d79</code></a> Support Python 3.9 officially</li>
<li><a href=""https://github.com/aio-libs/janus/commit/ac23eb74d2a15286449de6cafaa854086550394a""><code>ac23eb7</code></a> janus: remove unused type ignores (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/287"">#287</a>)</li>
<li><a href=""https://github.com/aio-libs/janus/commit/0da8f954ad87634093588163c2a40d7400882afb""><code>0da8f95</code></a> Make all tests non-skipped again</li>
<li>Additional commits viewable in <a href=""https://github.com/aio-libs/janus/compare/v0.4.0...v0.6.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a ""Dependabot enabled"" badge to your readme

Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)



</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1017/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
719382156,MDExOlB1bGxSZXF1ZXN0NTAxNTc5MzE1,1018,"Update asgiref requirement from ~=3.2.10 to >=3.2.10,<3.4.0",27856297,dependabot-preview[bot],closed,0,,,,,1,2020-10-12T13:30:09Z,2020-10-14T21:51:36Z,2020-10-14T21:51:35Z,CONTRIBUTOR,simonw/datasette/pulls/1018,"Updates the requirements on [asgiref](https://github.com/django/asgiref) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/django/asgiref/blob/master/CHANGELOG.txt"">asgiref's changelog</a>.</em></p>
<blockquote>
<h2>3.3.0 (2020-10-09)</h2>
<ul>
<li>sync_to_async now defaults to thread-sensitive mode being on</li>
<li>async_to_sync now works inside of forked processes</li>
<li>WsgiToAsgi now correctly clamps its response body when Content-Length is set</li>
</ul>
<h2>3.2.10 (2020-08-18)</h2>
<ul>
<li>Fixed bugs due to bad WeakRef handling introduced in 3.2.8</li>
</ul>
<h2>3.2.9 (2020-06-16)</h2>
<ul>
<li>Fixed regression with exception handling in 3.2.8 related to the contextvars fix.</li>
</ul>
<h2>3.2.8 (2020-06-15)</h2>
<ul>
<li>Fixed small memory leak in local.Local</li>
<li>contextvars are now persisted through AsyncToSync</li>
</ul>
<h2>3.2.7 (2020-03-24)</h2>
<ul>
<li>Bug fixed in local.Local where deleted Locals would occasionally inherit
their storage into new Locals due to memory reuse.</li>
</ul>
<h2>3.2.6 (2020-03-23)</h2>
<ul>
<li>local.Local now works in all threading situations, no longer requires
periodic garbage collection, and works with libraries that monkeypatch
threading (like gevent)</li>
</ul>
<h2>3.2.5 (2020-03-11)</h2>
<ul>
<li><strong>self</strong> is now preserved on methods by async_to_sync</li>
</ul>
<h2>3.2.4 (2020-03-10)</h2>
<!-- raw HTML omitted -->
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/django/asgiref/commit/7dba5ff61313dfd1535de200590b9155abde20f0""><code>7dba5ff</code></a> Releasing 3.3.0</li>
<li><a href=""https://github.com/django/asgiref/commit/e1e0dd9a900966b29345e6069c0fdbc9a449e97f""><code>e1e0dd9</code></a> Added ZeroCopy extension</li>
<li><a href=""https://github.com/django/asgiref/commit/3834d13a8f5da921bb5f8ca16ee66d353fdbdadb""><code>3834d13</code></a> Added rpc.py to Implementations (<a href=""https://github-redirect.dependabot.com/django/asgiref/issues/198"">#198</a>)</li>
<li><a href=""https://github.com/django/asgiref/commit/03b0dbb518e3bbf6c4083ae1e51bd6e2062fa503""><code>03b0dbb</code></a> Clamped WsgiToAsgi response body using Content-Length value</li>
<li><a href=""https://github.com/django/asgiref/commit/cfd82e48a2059ff93ce20b94e1755c455e3c3d29""><code>cfd82e4</code></a> Fix linting with unused import removal</li>
<li><a href=""https://github.com/django/asgiref/commit/cc1877efc8bcf5b6e606eb5c50cfdcca1213c83a""><code>cc1877e</code></a> Fix import sorting in previous commit.</li>
<li><a href=""https://github.com/django/asgiref/commit/7becc9daca2628c46af1cb7e46b4c47c1ea27adf""><code>7becc9d</code></a> Making thread_sensitive=True the default</li>
<li><a href=""https://github.com/django/asgiref/commit/66a6e68af258d626f5004ef305255b3154e552d4""><code>66a6e68</code></a> Fixed <a href=""https://github-redirect.dependabot.com/django/asgiref/issues/194"">#194</a>: Made async_to_sync work inside a fork</li>
<li><a href=""https://github.com/django/asgiref/commit/4ab9d8ea79bc202aa52ab4e71f315a65d025a377""><code>4ab9d8e</code></a> Fixed <a href=""https://github-redirect.dependabot.com/django/asgiref/issues/193"">#193</a>: Bumped docs version to 3.0</li>
<li><a href=""https://github.com/django/asgiref/commit/1c9d06329dbd679a7c564b0652147855a721edb4""><code>1c9d063</code></a> Clarified &quot;Optional&quot; meaning (<a href=""https://github-redirect.dependabot.com/django/asgiref/issues/190"">#190</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/django/asgiref/compare/3.2.10...3.3.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a ""Dependabot enabled"" badge to your readme

Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)



</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1018/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
721050815,MDU6SXNzdWU3MjEwNTA4MTU=,1019,"""Edit SQL"" button on canned queries",639012,jsfenfen,closed,0,,,6026070,0.51,7,2020-10-14T00:51:39Z,2020-10-23T19:44:06Z,2020-10-14T03:44:23Z,CONTRIBUTOR,,"Feature request: Would it be possible to add an ""edit this query"" button on canned queries? Clicking it would open the canned query as an editable sql query. I think the intent is to have named parameters to allow this, but sometimes you just gotta rewrite it? ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1019/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
274264175,MDU6SXNzdWUyNzQyNjQxNzU=,102,datasette publish elasticbeanstalk,9599,simonw,closed,0,,,,,1,2017-11-15T18:48:31Z,2021-01-04T20:13:20Z,2021-01-04T20:13:19Z,OWNER,,"It looks like Elastic Beanstalk is the most convenient way to deploy a docker container to AWS without first deploying a cluster.

https://aws.amazon.com/blogs/devops/dockerizing-a-python-web-app/ looks helpful. We would need to automate the deployment with Boto: http://boto3.readthedocs.io/en/latest/reference/services/elasticbeanstalk.html",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/102/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
721068929,MDU6SXNzdWU3MjEwNjg5Mjk=,1020,Method for datasette.client() to forward on authentication,9599,simonw,open,0,,,,,6,2020-10-14T01:47:49Z,2020-10-19T22:45:01Z,,OWNER,,"I stumbled into this while working on Dogsheep Beta: the requests it re-dispatched through `TableView` did not carry authentication cookies, and since this was against a private instance they were thus denied.

https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L223-L231

This made me think that `datasette.client.get()` (which Dogsheep Beta will start using shortly) could benefit from some kind of utility mechanism for passing through the cookies and general authenticated state from the current request.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1020/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
721830990,MDExOlB1bGxSZXF1ZXN0NTAzNjg1MDc3,1022,Fix table name in spatialite example command,639012,jsfenfen,closed,0,,,,,2,2020-10-14T22:19:34Z,2020-10-14T23:46:46Z,2020-10-14T23:46:46Z,CONTRIBUTOR,simonw/datasette/pulls/1022,The example query for creating a new point geometry seems to be using a table called 'museums' but at one point it instead uses 'events'. I *believe* it is intended to be museums (the example makes more sense if so). ,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1022/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
722673818,MDU6SXNzdWU3MjI2NzM4MTg=,1023,Fix issues relating to base_url,9599,simonw,closed,0,,,6026070,0.51,3,2020-10-15T21:02:06Z,2020-11-24T19:51:44Z,2020-10-31T20:51:01Z,OWNER,,Lots of `base_url` bugs that I'd like to solve at once.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1023/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
722674708,MDU6SXNzdWU3MjI2NzQ3MDg=,1024,Figure out how to run an environment that exercises the base_url proxy setting,9599,simonw,closed,0,,,6026070,0.51,9,2020-10-15T21:03:39Z,2020-10-23T19:44:06Z,2020-10-15T22:34:04Z,OWNER,,Refs #1023.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1024/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
722724086,MDU6SXNzdWU3MjI3MjQwODY=,1025,"Fix last remaining links to ""/"" that do not respect base_url",9599,simonw,closed,0,,,6026070,0.51,7,2020-10-15T22:46:38Z,2020-10-23T19:44:06Z,2020-10-20T05:21:29Z,OWNER,,"Refs #1023
```
datasette % git grep '""/""' -- '*.html'
datasette/templates/error.html:        <a href=""/"">home</a>
datasette/templates/patterns.html:        <a href=""/"">home</a> /
datasette/templates/query.html:        <a href=""/"">home</a> /
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1025/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
722738988,MDU6SXNzdWU3MjI3Mzg5ODg=,1026,How should datasette.client interact with base_url,9599,simonw,closed,0,,,6026070,0.51,5,2020-10-15T23:07:11Z,2020-10-31T19:29:52Z,2020-10-31T19:29:51Z,OWNER,,"Refs #1023. If Datasette is running with a `base_url` setting and a plugin calls e.g. `datasette.client.get(""/-/plugins.json"")` what should happen?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1026/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
722758132,MDU6SXNzdWU3MjI3NTgxMzI=,1027,Add documentation on serving Datasette behind a proxy using base_url,9599,simonw,closed,0,,,6026070,0.51,5,2020-10-15T23:46:29Z,2020-10-31T21:14:05Z,2020-10-31T21:14:05Z,OWNER,,"This can go on this page: https://docs.datasette.io/en/stable/deploying.html

Refs #1023, #865",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1027/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
723803777,MDU6SXNzdWU3MjM4MDM3Nzc=,1028,--load-extension=spatialite shortcut,9599,simonw,closed,0,,,6026070,0.51,1,2020-10-17T17:02:08Z,2022-01-20T21:29:41Z,2020-10-19T22:37:55Z,OWNER,,I added this to `sqlite-utils` in https://github.com/simonw/sqlite-utils/issues/136 and I really like it: pass a special value of `spatialite` and Datasette should attempt to load it from known likely installation locations.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1028/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
723837704,MDExOlB1bGxSZXF1ZXN0NTA1MzM5NTE1,1029,fix(docs): broken link,17075617,jthodge,closed,0,,,,,1,2020-10-17T20:03:20Z,2020-10-17T20:05:04Z,2020-10-17T20:05:04Z,CONTRIBUTOR,simonw/datasette/pulls/1029,This PR fixes a broken markdown link in the `Publish` docs page.,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1029/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
274265878,MDU6SXNzdWUyNzQyNjU4Nzg=,103,datasette publish appengine,9599,simonw,closed,0,,,,,1,2017-11-15T18:54:18Z,2021-01-04T20:05:14Z,2021-01-04T20:05:14Z,OWNER,,"Similar approach to Heroku, discussed in #90 

Looks like this could be pretty easy:  https://cloud.google.com/appengine/docs/flexible/python/quickstart",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/103/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
723982480,MDExOlB1bGxSZXF1ZXN0NTA1NDUzOTAw,1030,Make `package` command deal with a configuration directory argument,299380,frankier,open,0,,,,,1,2020-10-18T11:07:02Z,2020-10-19T08:01:51Z,,FIRST_TIME_CONTRIBUTOR,simonw/datasette/pulls/1030,Currently if we run `datasette package` on a configuration directory we'll get an exception when we try to hard link to the directory. This PR copies the tree and makes the Dockerfile run inspect on all *.db files.,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1030/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
724369025,MDExOlB1bGxSZXF1ZXN0NTA1NzY5NDYy,1031,Fallback to databases in inspect-data.json when no -i options are passed,299380,frankier,closed,0,,,,,6,2020-10-19T07:51:06Z,2021-03-29T01:46:45Z,2021-03-29T00:23:41Z,FIRST_TIME_CONTRIBUTOR,simonw/datasette/pulls/1031,Currenlty `Datasette.__init__` checks immutables against None to decide whether to fallback to inspect-data.json. This patch modifies the serve command to pass None when no -i options are passed so this fallback works correctly.,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1031/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
724878151,MDU6SXNzdWU3MjQ4NzgxNTE=,1032,Bring date parsing into Datasette core,9599,simonw,open,0,,,,,8,2020-10-19T18:30:45Z,2020-10-19T19:37:55Z,,OWNER,,"Currently this is mainly handled by a plugin - https://github.com/simonw/datasette-dateutil - but I realise now that this really needs to be core functionality.

See also Twitter thread: https://twitter.com/simonw/status/1318234808653213696",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1032/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
725099777,MDU6SXNzdWU3MjUwOTk3Nzc=,1033,datasette.urls.static_plugins(...) method,9599,simonw,closed,0,,,6026070,0.51,5,2020-10-20T01:16:32Z,2020-10-24T22:58:33Z,2020-10-24T20:03:52Z,OWNER,,"Follow-on from #904. For constructing URLs like this:

    /-/static-plugins/NAME_OF_PLUGIN_PACKAGE/yourfile.js

Should be documented on https://docs.datasette.io/en/latest/writing_plugins.html#static-assets and https://docs.datasette.io/en/latest/internals.html#datasette-urls",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1033/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
725184645,MDU6SXNzdWU3MjUxODQ2NDU=,1034,Better way of representing binary data in .csv output,9599,simonw,closed,0,,,6026070,0.51,19,2020-10-20T04:28:58Z,2021-06-17T18:13:21Z,2020-10-29T22:47:46Z,OWNER,,"I just noticed this: https://latest.datasette.io/fixtures/binary_data.csv

```csv
rowid,data
1,b'\x15\x1c\x02\xc7\xad\x05\xfe'
2,b'\x15\x1c\x03\xc7\xad\x05\xfe'
```
There's no good way to represent binary data in a CSV file, but this seems like one of the more-bad options.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1034/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
725743755,MDU6SXNzdWU3MjU3NDM3NTU=,1035,"datasette.urls.table(..., format=""json"") argument",9599,simonw,closed,0,,,6026070,0.51,3,2020-10-20T16:09:34Z,2020-10-31T18:16:43Z,2020-10-31T18:16:43Z,OWNER,,"> That `datasette.urls.table(""db"", ""table"") + "".json""` example is bad because if the table name contains a `.` it should be `?_format=json` instead.
> 
> Maybe `.table()` should have a `format=""json""` option that knows how to do this.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1026#issuecomment-712962517_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1035/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
725996507,MDU6SXNzdWU3MjU5OTY1MDc=,1036,Make it possible to download BLOB data from the Datasette UI,9599,simonw,closed,0,,,6026070,0.51,16,2020-10-20T22:47:56Z,2021-01-18T17:45:00Z,2020-10-25T00:14:52Z,OWNER,,"Currently you can only extract binary BLOB data as base64-encoded JSON, which is not user friendly at all. It should always be possible for end-users to get the binary data out.

I'm worried about XSS vulnerabilities here, but hopefully sending `Content-Type: application/octet-stream` helps there? Need to research that.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1036/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
726094754,MDU6SXNzdWU3MjYwOTQ3NTQ=,1037,Add horizontal scrollbar to tables,9599,simonw,closed,0,,,6026070,0.51,3,2020-10-21T03:13:34Z,2020-10-27T20:52:04Z,2020-10-21T03:16:36Z,OWNER,,Currently you have to scroll the entire page sideways if a table is wide. Make the table `overflow-x: auto` instead.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1037/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
726154220,MDExOlB1bGxSZXF1ZXN0NTA3MjY3MDg3,1038,DOC: Fix syntax error,194147,gerrymanoim,closed,0,,,,,2,2020-10-21T05:45:38Z,2020-10-21T22:57:21Z,2020-10-21T22:44:17Z,CONTRIBUTOR,simonw/datasette/pulls/1038,"If I understand https://docs.datasette.io/en/stable/plugin_hooks.html#register-routes correctly, `register_routes` should return a `List[Tuple[str, Callable]]`. I believe the current code in documentation has a syntax error (extra `)`). ",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1038/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
726687572,MDU6SXNzdWU3MjY2ODc1NzI=,1039,Add an animation to the column actions menu,9599,simonw,closed,0,,,6026070,0.51,1,2020-10-21T16:56:28Z,2020-10-23T19:44:07Z,2020-10-21T17:02:32Z,OWNER,,"Inspired by the animation on some of GitHub's dropdown menus:

https://github.com/primer/css/blob/da8ee54248e6d76c15c18e53684a15a6516b5b7f/src/utilities/animations.scss#L114-L131

```css
/* Fade in an element and scale it fast */
.anim-scale-in {
  animation-name: scale-in;
  animation-duration: 0.15s;
  animation-timing-function: cubic-bezier(0.2, 0, 0.13, 1.5);
}

@keyframes scale-in {
  0% {
    opacity: 0;
    transform: scale(0.5);
  }

  100% {
    opacity: 1;
    transform: scale(1);
  }
}
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1039/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
274284246,MDExOlB1bGxSZXF1ZXN0MTUyODcwMDMw,104,[WIP] Add publish to heroku support,21148,jacobian,closed,0,,,,,6,2017-11-15T19:56:22Z,2017-11-21T20:55:05Z,2017-11-21T20:55:05Z,CONTRIBUTOR,simonw/datasette/pulls/104,"

Refs #90 ",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/104/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
726910999,MDExOlB1bGxSZXF1ZXN0NTA3OTAzMzky,1040,/db/table/-/blob/pk/column.blob download URL,9599,simonw,closed,0,,,6026070,0.51,3,2020-10-21T22:39:15Z,2020-10-24T23:09:20Z,2020-10-24T23:09:19Z,OWNER,simonw/datasette/pulls/1040,"Refs #1036. Still needs:

- [x] Comprehensive tests across all of the code branches, plus permissions
- [x] A bit more refactoring to share logic cleanly with `RowView`
- ~~A configuration option to disable this feature (probably)~~",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1040/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
727627923,MDU6SXNzdWU3Mjc2Mjc5MjM=,1041,extra_js_urls and extra_css_urls should respect base_url setting,9599,simonw,closed,0,,,6026070,0.51,4,2020-10-22T18:34:33Z,2020-10-31T20:49:28Z,2020-10-31T20:48:58Z,OWNER,,"_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1033#issuecomment-714681365_

Refs #1023",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1041/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
727802081,MDU6SXNzdWU3Mjc4MDIwODE=,1042,Plugin hook for loading templates,9599,simonw,closed,0,,,6026070,0.51,14,2020-10-23T00:18:39Z,2020-10-30T17:47:21Z,2020-10-30T17:47:20Z,OWNER,,This can work with the Jinja template loaders. It would unlock things like storing templates in SQLite.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1042/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
727915394,MDExOlB1bGxSZXF1ZXN0NTA4NzE5NTY3,1043,Include LICENSE in sdist,45380,bollwyvl,closed,0,,,,,4,2020-10-23T05:04:12Z,2020-10-26T00:14:57Z,2020-10-23T20:54:35Z,CONTRIBUTOR,simonw/datasette/pulls/1043,"Hi, thanks for `datasette`! 

This PR adds the `LICENSE` to source distributions, which seems the norm for Apache-2.0 stuff.

I noticed the [0.50.2 sdist](https://files.pythonhosted.org/packages/f2/ba/1b5f182c3f1769c0863bcaa77406bdcb81c92e31bb579959c01b1d8951c0/datasette-0.50.2.tar.gz) doesn't ship `LICENSE`, but the 0.5.2 `whl` does, so I'm assuming the intent _is_ to ship... and it's a one-liner!

Motivation: 

It might be a bit of a slog, but I'm looking to see about getting `datasette` (and friends!) available on conda-forge. There are a few missing upstreams (`asgi-csrf`, `python-basecov`, `mergedeep`) and some of the plugins don't even appear to _have_ tarballs (just `whl`!), but the little stuff like licenses are nice to get out handled upstream vs separately grabbing them.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1043/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
727916744,MDExOlB1bGxSZXF1ZXN0NTA4NzIwNjYw,1044,Add minimum supported python,45380,bollwyvl,closed,0,,,,,2,2020-10-23T05:08:03Z,2020-10-23T20:53:08Z,2020-10-23T20:53:08Z,CONTRIBUTOR,simonw/datasette/pulls/1044,"Thanks for `datasette`!

This PR adds `python_requires` to formally signal the [minimum supported python version](https://packaging.python.org/guides/dropping-older-python-versions/#specify-the-version-ranges-for-supported-python-distributions) (which is pointed out with classifiers, so seems pretty straightforward).",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1044/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
728600048,MDU6SXNzdWU3Mjg2MDAwNDg=,1045,"Document that datasette.render_template(template, ...) also accepts a list of templates",9599,simonw,closed,0,,,6026070,0.51,1,2020-10-23T23:37:12Z,2020-10-24T00:22:10Z,2020-10-24T00:22:09Z,OWNER,,"https://docs.datasette.io/en/stable/internals.html#await-render-template-template-context-none-request-none

`await .render_template(template, context=None, request=None)`

This currently only accepts a single template. If it accepted a list of templates (where the first available template gets rendered) it could be more widely used by Datasette internally.

Spotted this while researching #1042.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1045/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
728895193,MDU6SXNzdWU3Mjg4OTUxOTM=,1046,Link to blob downloads in the right places,9599,simonw,closed,0,,,6026070,0.51,2,2020-10-24T23:00:41Z,2020-10-25T00:13:21Z,2020-10-25T00:13:21Z,OWNER,,"Split from #1040, refs #1036.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1046/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
728895233,MDU6SXNzdWU3Mjg4OTUyMzM=,1047,A new section in the docs about how Datasette handles BLOB columns,9599,simonw,closed,0,,,6026070,0.51,1,2020-10-24T23:01:02Z,2020-10-31T22:11:25Z,2020-10-31T21:38:05Z,OWNER,,"Split from #1040, refs #1036.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1047/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
728905098,MDU6SXNzdWU3Mjg5MDUwOTg=,1048,Documentation and unit tests for urls.row() urls.row_blob() methods,9599,simonw,open,0,,,,,7,2020-10-25T00:13:53Z,2022-07-10T16:23:57Z,,OWNER,,https://github.com/simonw/datasette/blob/5db7ae3ce165ded57c7fb1cfbdb3258b1cf06c10/datasette/app.py#L1307-L1313,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1048/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
729017519,MDExOlB1bGxSZXF1ZXN0NTA5NTkwMjA1,1049,Add template block prior to extra URL loaders,82988,psychemedia,closed,0,,,,,4,2020-10-25T13:08:55Z,2020-10-29T09:20:52Z,2020-10-29T09:20:34Z,CONTRIBUTOR,simonw/datasette/pulls/1049,"To handle packages that require Javascript state setting prior to loading a package (eg [`thebelab`](https://thebelab.readthedocs.io/en/latest/examples/minimal_example.html), provide a template block before the URLs are loaded.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1049/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
274314940,MDU6SXNzdWUyNzQzMTQ5NDA=,105,Consider data-package as a format for metadata,9599,simonw,closed,0,,,,,4,2017-11-15T21:43:34Z,2017-11-20T19:50:53Z,2017-11-20T19:50:53Z,OWNER,,http://frictionlessdata.io/specs/data-package/,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/105/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
729057388,MDU6SXNzdWU3MjkwNTczODg=,1050,Switch to .blob render extension for BLOB downloads,9599,simonw,closed,0,,,6026070,0.51,10,2020-10-25T16:26:21Z,2020-10-29T22:01:39Z,2020-10-29T22:01:39Z,OWNER,,This may require a complete rethink of the `/db/table/-/blob/row/column.blob` mechanism I just built for #1036.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1050/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
729096595,MDU6SXNzdWU3MjkwOTY1OTU=,1051,Better display of binary data on arbitrary query results page,9599,simonw,closed,0,,,,,6,2020-10-25T19:38:06Z,2020-10-29T22:12:16Z,2020-10-29T22:01:39Z,OWNER,,"https://latest.datasette.io/fixtures?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+101

<img width=""718"" alt=""fixtures__select_rowid__data_from_binary_data_order_by_rowid_limit_101_and_Switch_to__blob_render_extension_for_BLOB_downloads_·_Issue__1050_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/97117089-c2a32080-16be-11eb-8e4b-63821715b98e.png"">

Problem: if these were larger fields that HTML page could have multiple megabytes of Python binary string representations on it.

It should behave more like the regular table view does:

<img width=""593"" alt=""fixtures__binary_data__2_rows"" src=""https://user-images.githubusercontent.com/9599/97117113-ea928400-16be-11eb-8730-3c1c8d6c3491.png"">
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1051/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
729183332,MDU6SXNzdWU3MjkxODMzMzI=,1052,Column action menu overlapped by Leaflet maps,9599,simonw,closed,0,,,6026070,0.51,1,2020-10-26T02:17:29Z,2020-10-27T20:52:04Z,2020-10-26T02:19:36Z,OWNER,,"Using `datasette-leaflet-geojson`:

<img width=""913"" alt=""Cursor_and_benches__benches__17_387_rows"" src=""https://user-images.githubusercontent.com/9599/97127361-a4f1ad80-16f6-11eb-9c81-91ba7c990982.png"">",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1052/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
729604838,MDU6SXNzdWU3Mjk2MDQ4Mzg=,1053,Document recommendations for plugin authors to design URLs,9599,simonw,closed,0,,,6026070,0.51,1,2020-10-26T14:19:21Z,2020-10-29T19:37:58Z,2020-10-29T19:35:40Z,OWNER,,"See thread: https://twitter.com/kanedr/status/1320653434895347713

> The process and API for making a plugin is great btw. One question I had was the best format for URLs. I've created an url like /<database>/<table>/reconcile as a json endpoint, but that could conflict with the row-level URLs. Is there a recommended pattern to use?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1053/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
730199464,MDU6SXNzdWU3MzAxOTk0NjQ=,1054,Switch from versioneer to concrete version in setup.py,9599,simonw,closed,0,,,6026070,0.51,2,2020-10-27T07:38:08Z,2020-10-29T03:38:18Z,2020-10-29T03:38:17Z,OWNER,,"The new PyPI resolver keeps on showing me warnings like this one when I install Datasette directly from GitHub using `pip install https://github.com/simonw/datasette/archive/main.zip`:
```
Successfully built datasette
Installing collected packages: datasette
  Attempting uninstall: datasette
    Found existing installation: datasette 0.50.2
    Uninstalling datasette-0.50.2:
      Successfully uninstalled datasette-0.50.2
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

datasette-upload-csvs 0.5 requires datasette>=0.47, but you'll have datasette 0+unknown which is incompatible.
datasette-publish-vercel 0.8 requires datasette>=0.44, but you'll have datasette 0+unknown which is incompatible.
datasette-psutil 0.2 requires datasette>=0.44, but you'll have datasette 0+unknown which is incompatible.
datasette-leaflet-geojson 0.6 requires datasette>=0.48, but you'll have datasette 0+unknown which is incompatible.
datasette-edit-schema 0.3 requires datasette>=0.44, but you'll have datasette 0+unknown which is incompatible.
datasette-cluster-map 0.13 requires datasette>=0.48, but you'll have datasette 0+unknown which is incompatible.
Successfully installed datasette-0+unknown
```
This is because we use versioneer. I'm going to drop that in favour of embedding the version directly in `setup.py`, like I do in other projects such as `sqlite-utils`.

I'll use a `.dev` suffix in the development version, as suggested by https://www.python.org/dev/peps/pep-0440/#developmental-releases",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1054/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
730210880,MDU6SXNzdWU3MzAyMTA4ODA=,1055,query.html and table.html should share the same table implementation,9599,simonw,open,0,,,3268330,Datasette 1.0,0,2020-10-27T07:58:21Z,2020-10-27T07:58:29Z,,OWNER,,In #998 I made a change that affected the table page but didn't affect the query page because I incorrectly assumed they shared rendering logic.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1055/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
730752399,MDExOlB1bGxSZXF1ZXN0NTExMDA1NTQy,1056,"Radical new colour scheme and base styles, courtesy of @natbat",9599,simonw,closed,0,,,6026070,0.51,1,2020-10-27T19:31:49Z,2020-10-27T19:39:57Z,2020-10-27T19:39:56Z,OWNER,simonw/datasette/pulls/1056,,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1056/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
730797787,MDU6SXNzdWU3MzA3OTc3ODc=,1057,--cors should enable /fixtures.db CORS access,9599,simonw,closed,0,,,6026070,0.51,1,2020-10-27T20:38:34Z,2020-10-27T20:52:05Z,2020-10-27T20:51:09Z,OWNER,,So Datasette can work with `SQL.js` as seen in https://observablehq.com/@mbostock/sqlite,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1057/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
730802994,MDU6SXNzdWU3MzA4MDI5OTQ=,1058,Database download should implement cascading permissions,9599,simonw,closed,0,,,6026070,0.51,1,2020-10-27T20:43:27Z,2020-10-28T03:15:47Z,2020-10-28T03:15:47Z,OWNER,,"https://github.com/simonw/datasette/blob/5a1519796037105bc20bcf2f91a76e022926c204/datasette/views/database.py#L130-L136

Should be updated for #832 cascading permissions. Example commit: https://github.com/simonw/datasette/commit/d6e03b04302a0852e7133dc030eab50177c37be7",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1058/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
731445447,MDExOlB1bGxSZXF1ZXN0NTExNTQ5Mzc0,1059,"Update aiofiles requirement from <0.6,>=0.4 to >=0.4,<0.7",27856297,dependabot-preview[bot],closed,0,,,,,2,2020-10-28T13:32:40Z,2020-10-28T17:08:29Z,2020-10-28T17:08:28Z,CONTRIBUTOR,simonw/datasette/pulls/1059,"Updates the requirements on [aiofiles](https://github.com/Tinche/aiofiles) to permit the latest version.
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/Tinche/aiofiles/commit/b4482aeef5312d4bf0c1abda5892df80957fa01f""><code>b4482ae</code></a> 0.6.0</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/eaccd3e43013b843f606665a68d3db395437e537""><code>eaccd3e</code></a> Update Changelog</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/2de90cf95e48db9178ef5bb246b5866892e8ef8e""><code>2de90cf</code></a> Test on 3.9</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/62b4c27096a5437f0608886e98cbe2681992ca57""><code>62b4c27</code></a> Update .travis.yml</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/4083c7371a2e667a316efa55613cfe6b826304b2""><code>4083c73</code></a> Removed pypy3 for ppc64le</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/26e795b5dafe7d227147f1663ffeca6a76efe453""><code>26e795b</code></a> Update .travis.yml</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/bd154cf3bf60f03aa22b3b264578a987d136be85""><code>bd154cf</code></a> Update .travis.yml</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/7ae0e2ace9193e4e2417fc76ea51caf84ba0e656""><code>7ae0e2a</code></a> Update .travis.yml</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/84eeb6c686ca6ba0ac700e93a078547cbaac625d""><code>84eeb6c</code></a> Update .travis.yml</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/c48f5b0a825f8c68c2238f098585c1f1c6ba2c16""><code>c48f5b0</code></a> Proxy name and mode properties</li>
<li>Additional commits viewable in <a href=""https://github.com/Tinche/aiofiles/compare/v0.4.0...v0.6.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a ""Dependabot enabled"" badge to your readme

Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)



</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1059/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
274315193,MDU6SXNzdWUyNzQzMTUxOTM=,106,Document how pagination works,9599,simonw,closed,0,,,,,1,2017-11-15T21:44:32Z,2019-06-24T06:42:33Z,2019-06-24T06:42:33Z,OWNER,,I made a start at that in this comment: https://news.ycombinator.com/item?id=15691926,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/106/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
731827081,MDExOlB1bGxSZXF1ZXN0NTExODY4MTUz,1060,New explicit versioning mechanism,9599,simonw,closed,0,,,6026070,0.51,1,2020-10-28T22:14:55Z,2020-10-29T03:38:17Z,2020-10-29T03:38:16Z,OWNER,simonw/datasette/pulls/1060,"- Remove all references to versioneer
- Re-implement versioning to use a static string baked into the repo
- Ensure that string is output by `datasette --version` and `/-/versions`

Refs #1054",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1060/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
732634375,MDExOlB1bGxSZXF1ZXN0NTEyNTQ1MzY0,1061,.blob output renderer,9599,simonw,closed,0,,,6026070,0.51,4,2020-10-29T20:25:08Z,2020-10-29T22:01:40Z,2020-10-29T22:01:39Z,OWNER,simonw/datasette/pulls/1061,"- [x] Remove the `/-/...blob/...` route I added in #1040 in place of the new `.blob` renderer URLs
- [x] Link to new `.blob` download links on the arbitrary query page (using `_blob_hash=...`) - plus tests for this

Closes #1050, Closes #1051",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1061/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
732674148,MDU6SXNzdWU3MzI2NzQxNDg=,1062,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,9599,simonw,open,0,,,3268330,Datasette 1.0,5,2020-10-29T21:25:02Z,2022-09-28T14:09:54Z,,OWNER,,This can drive the upgrade of the `register_output_renderer` hook to be able to handle streaming all rows in a large query.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1062/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
732685643,MDU6SXNzdWU3MzI2ODU2NDM=,1063,.csv should link to .blob downloads,9599,simonw,closed,0,,,6026070,0.51,3,2020-10-29T21:45:58Z,2021-06-17T18:12:30Z,2020-10-29T22:47:45Z,OWNER,,"- [x] Update `.csv` output to link to these things (and get that `xfail` test to pass)
- ~~Add a `.csv?_blob_base64=1` argument that causes them to be output in base64 in the CSV~~

> Moving the CSV work to a separate ticket.
_Originally posted by @simonw in https://github.com/simonw/datasette/pull/1061#issuecomment-719042601_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1063/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
732798913,MDU6SXNzdWU3MzI3OTg5MTM=,1064,Navigation menu plus plugin hook,9599,simonw,closed,0,,,6026070,0.51,10,2020-10-30T00:49:36Z,2020-10-30T03:45:16Z,2020-10-30T03:45:16Z,OWNER,,Needed for #690. Prototype in https://github.com/simonw/datasette/commit/0d7ac764861d84be24d661cf4104ce61ea11a82a,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1064/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
732856937,MDExOlB1bGxSZXF1ZXN0NTEyNzM2NzA1,1065,Nav menu plus menu_links() hook,9599,simonw,closed,0,,,6026070,0.51,1,2020-10-30T03:40:18Z,2020-10-30T03:45:17Z,2020-10-30T03:45:16Z,OWNER,simonw/datasette/pulls/1065,"Closes #1064, refs #690.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1065/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
732859030,MDU6SXNzdWU3MzI4NTkwMzA=,1066,Table actions menu plus plugin hook,9599,simonw,closed,0,,,6026070,0.51,3,2020-10-30T03:46:54Z,2020-10-30T05:18:36Z,2020-10-30T05:16:50Z,OWNER,,"> For the table actions: attaching it to a cog icon next to the table name could make sense.
>
> <img width=""503"" alt=""data__faa-wildlife-strikes__26_rows_where_where_AIRPORT____OIL_RIG__sorted_by_rowid_and_Fix_table_name_in_spatialite_example_command_by_jsfenfen_·_Pull_Request__1022_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/96168896-45192c80-0ed6-11eb-9d90-6172a8346468.png"">
>
> This is the column action icon at twice the size, color `#666`.
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/690#issuecomment-709497595_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1066/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
732905360,MDU6SXNzdWU3MzI5MDUzNjA=,1067,"Table actions menu on view pages, not on query pages",9599,simonw,closed,0,,,6026070,0.51,6,2020-10-30T05:56:39Z,2020-10-31T17:51:31Z,2020-10-31T17:40:14Z,OWNER,,Follow-on from #1066.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1067/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
732939921,MDU6SXNzdWU3MzI5Mzk5MjE=,1068,Default menu links should check a real permission ,9599,simonw,closed,0,,,6026070,0.51,5,2020-10-30T07:08:34Z,2020-10-30T15:44:13Z,2020-10-30T15:42:11Z,OWNER,,"https://github.com/simonw/datasette/blob/18a64fbb29271ce607937110bbdb55488c43f4e0/datasette/default_menu_links.py#L4-L6

This should check a named permission so that it can be customized by permission plugins.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1068/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
733303548,MDExOlB1bGxSZXF1ZXN0NTEzMTA2MDI2,1069,load_template() plugin hook,9599,simonw,closed,0,,,6026070,0.51,6,2020-10-30T15:59:45Z,2020-10-30T17:47:20Z,2020-10-30T17:47:19Z,OWNER,simonw/datasette/pulls/1069,Refs #1042,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1069/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
274343647,MDExOlB1bGxSZXF1ZXN0MTUyOTE0NDgw,107,add support for ?field__isnull=1,3433657,raynae,closed,0,,,,,4,2017-11-15T23:36:36Z,2017-11-17T15:12:29Z,2017-11-17T13:29:22Z,CONTRIBUTOR,simonw/datasette/pulls/107,Is this what you had in mind for [this issue](https://github.com/simonw/datasette/issues/64)?,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/107/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
733390884,MDU6SXNzdWU3MzMzOTA4ODQ=,1070,load_template() example in documentation showing loading from a database,9599,simonw,closed,0,,,6026070,0.51,1,2020-10-30T17:45:03Z,2020-10-31T16:22:51Z,2020-10-31T16:22:45Z,OWNER,,"> I should include an example in the documentation that shows loading templates from a database table.
_Originally posted by @simonw in https://github.com/simonw/datasette/pull/1069#issuecomment-719664530_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1070/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
733485423,MDU6SXNzdWU3MzM0ODU0MjM=,1071,Messages should be displayed full width,9599,simonw,closed,0,,,6026070,0.51,1,2020-10-30T20:11:35Z,2020-10-30T20:20:02Z,2020-10-30T20:13:05Z,OWNER,,"In the pattern portfolio:

<img width=""907"" alt=""Datasette__Pattern_Portfolio_and_datasette_—_pipenv_shell_▸_Python_—_142×48"" src=""https://user-images.githubusercontent.com/9599/97752509-5a36b380-1ab1-11eb-9a26-700cb295fe11.png"">

But they're currently showing like this:

<img width=""901"" alt=""Datasette___memory_"" src=""https://user-images.githubusercontent.com/9599/97752544-69b5fc80-1ab1-11eb-80c2-c88aabe6a832.png"">
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1071/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
733499930,MDU6SXNzdWU3MzM0OTk5MzA=,1072,load_template hook doesn't work for include/extends,9599,simonw,closed,0,,,6026070,0.51,20,2020-10-30T20:33:44Z,2020-10-31T20:48:18Z,2020-10-30T22:50:57Z,OWNER,,"Includes like this one always go to disk, without hitting the `load_template` plugin hook:
```html+jinja
<footer class=""ft"">{% block footer %}{% include ""_footer.html"" %}{% endblock %}</footer>
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1072/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
733560417,MDU6SXNzdWU3MzM1NjA0MTc=,1073,Remove load_template plugin hook,9599,simonw,closed,0,,,6026070,0.51,1,2020-10-30T22:51:52Z,2020-10-31T16:22:00Z,2020-10-31T16:22:00Z,OWNER,,"I couldn't get it working correctly with async (necessary for include/extend to function), and on deeper investigation it appears that I can build something equivalent to what I wanted using the existing `prepare_jinja2_environment` hook.

> I'm going to remove the `load_template` plugin hook and see if it's possible to build the edit templates extension against `prepare_jinja2_environment` instead.
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1072#issuecomment-719833744_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1073/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
733768037,MDU6SXNzdWU3MzM3NjgwMzc=,1074,latest.datasette.io should include plugins from fixtures,9599,simonw,closed,0,,,6026070,0.51,3,2020-10-31T17:23:23Z,2020-10-31T19:47:47Z,2020-10-31T19:47:47Z,OWNER,,"> It bothers me that these aren't visible in any public demos. Maybe `latest.datasette.io` should include the `my_plugins.py` and `my_plugins2.py` plugins?
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1067#issuecomment-719961701_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1074/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
733796942,MDU6SXNzdWU3MzM3OTY5NDI=,1075,PrefixedUrlString mechanism broke everything,9599,simonw,closed,0,,,6026070,0.51,6,2020-10-31T19:58:05Z,2020-10-31T20:48:51Z,2020-10-31T20:48:51Z,OWNER,,Added in 7a67bc7a569509d65b3a8661e0ad2c65f0b09166 refs #1026. Lots of tests are failing now.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1075/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
733805089,MDU6SXNzdWU3MzM4MDUwODk=,1076,Release notes for 0.51,9599,simonw,closed,0,,,6026070,0.51,0,2020-10-31T20:51:21Z,2020-10-31T22:27:00Z,2020-10-31T22:27:00Z,OWNER,,Start by combining release notes from https://github.com/simonw/datasette/releases/tag/0.51a0 and https://github.com/simonw/datasette/releases/tag/0.51a1 and https://github.com/simonw/datasette/releases/tag/0.51a2,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1076/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
733829385,MDU6SXNzdWU3MzM4MjkzODU=,1077,database_actions plugin hook,9599,simonw,closed,0,,,6055094,Datasette 0.52,3,2020-10-31T23:48:12Z,2020-11-02T18:43:25Z,2020-11-02T18:29:50Z,OWNER,,Like `column_actions` but adds a cog menu to the database page.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1077/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
733999615,MDU6SXNzdWU3MzM5OTk2MTU=,1079,Handle long breadcrumbs better with new menu,9599,simonw,open,0,,,,,1,2020-11-01T15:57:41Z,2022-01-13T22:21:29Z,,OWNER,,"On this page when signed in as root: https://latest.datasette.io/fixtures/roadside_attraction_characteristics/1

![EF921CB1-625F-4D04-A850-490B812A72B3](https://user-images.githubusercontent.com/9599/97807807-db0fbf80-1c17-11eb-9c77-ae5169b12c3d.jpeg)

![A49D8B76-5ACF-4F71-A8B4-21A44F5C8D51](https://user-images.githubusercontent.com/9599/97807809-dea34680-1c17-11eb-9511-a49af56a4bd2.jpeg)
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1079/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
274374317,MDU6SXNzdWUyNzQzNzQzMTc=,108,"Include version in python code, output in template",9599,simonw,closed,0,,,,,0,2017-11-16T02:32:40Z,2017-11-16T15:30:04Z,2017-11-16T15:30:04Z,OWNER,,It would be useful if I could tell which version of datasette was running on a site. Embed version number and include it in maybe a tooltip on the “powered by datasette” link,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/108/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
734777631,MDU6SXNzdWU3MzQ3Nzc2MzE=,1080,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",9599,simonw,open,0,,,3268330,Datasette 1.0,7,2020-11-02T19:55:06Z,2022-02-04T06:25:18Z,,OWNER,,Can use `/database/-/...` namespace from #296,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1080/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
735644513,MDU6SXNzdWU3MzU2NDQ1MTM=,1081,"Fixtures should use FTS4 or FTS5, not FTS3",9599,simonw,closed,0,,,6055094,Datasette 0.52,0,2020-11-03T21:24:13Z,2020-11-12T00:03:00Z,2020-11-12T00:02:59Z,OWNER,,"Just spotted that `fixtures.db` uses FTS3, which is pretty much obsolete these days.

https://github.com/simonw/datasette/blob/13d1228d80c91d382a05b1a9549ed02c300ef851/tests/fixtures.py#L488-L489",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1081/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
735852274,MDU6SXNzdWU3MzU4NTIyNzQ=,1082,DigitalOcean buildpack memory errors for large sqlite db?,39538958,justmars,open,0,,,,,3,2020-11-04T06:35:32Z,2020-11-04T19:35:44Z,,NONE,,"1. Have a sqlite db stored in Dropbox
2. Previously tried the Digital Ocean build pack minimal approach (e.g. Procfile, requirements.txt, bin/post_compile)
3. bin/post_compile with wget from Dropbox
4. download of large sqlite db is successful 
5. log reveals that when building Docker container, Digital Ocean runs out of memory for 5gb+ sqlite db but works fine for 2gb+ sqlite db",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1082/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
736365306,MDU6SXNzdWU3MzYzNjUzMDY=,1083,Advanced CSV export for arbitrary queries,9599,simonw,open,0,,,,,2,2020-11-04T19:23:05Z,2021-06-17T18:12:31Z,,OWNER,,"There's no link to download the CSV file - the table page has that as an advanced export option, but this is missing from the query page.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1083/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
737394470,MDU6SXNzdWU3MzczOTQ0NzA=,1084,Table/database action menu cut off if too short ,9599,simonw,closed,0,,,6055094,Datasette 0.52,4,2020-11-06T01:55:23Z,2020-11-21T23:45:59Z,2020-11-21T23:45:59Z,OWNER,,"![3CC0C181-959E-4B20-BE39-806ED93E833E](https://user-images.githubusercontent.com/9599/98316836-03891800-1f90-11eb-9e52-5266baf33296.jpeg)
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1084/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
740512882,MDExOlB1bGxSZXF1ZXN0NTE4OTg4ODc5,1085,Use FTS4 in fixtures,9599,simonw,closed,0,,,,,1,2020-11-11T06:44:30Z,2020-11-12T00:02:59Z,2020-11-12T00:02:58Z,OWNER,simonw/datasette/pulls/1085,Refs #1081,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1085/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
741021342,MDU6SXNzdWU3NDEwMjEzNDI=,1086,Foreign keys with blank titles result in non-clickable links,9599,simonw,closed,0,,,6055094,Datasette 0.52,3,2020-11-11T19:41:09Z,2020-11-28T23:28:29Z,2020-11-11T23:46:20Z,OWNER,,"<img width=""851"" alt=""index__core_snapshot_tags__3_rows_and_swarm__select_venues_name__checkins_id__createdAt__type__timeZoneOffset__editableUntil___like___isMayor__source__venue__createdBy__event__sticker__created__comments_count__entities__shout__checkins_priv"" src=""https://user-images.githubusercontent.com/9599/98856656-bafbaf80-2412-11eb-8070-4f3a47511973.png"">

The HTML looks like this:

```html
<td class=""col-tag_id type-int""><a href=""/index/core_tag/1""></a>&nbsp;<em>1</em></td>
```
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1086/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
741231849,MDU6SXNzdWU3NDEyMzE4NDk=,1087,Idea: ?_extra=urls for getting back URLs to useful things,9599,simonw,open,0,,,,,0,2020-11-12T02:55:41Z,2021-12-15T18:06:16Z,,OWNER,,"Working on https://github.com/simonw/datasette-search-all/issues/10 made me realize that sometimes it can be difficult to calculate the URL for a database, table or row within Datasette.

It would be useful to have an optional extra JSON extension (using `?_extra=` from #262) that can help with this.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1087/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
741268956,MDU6SXNzdWU3NDEyNjg5NTY=,1088,OperationalError('interrupted') can 500 on row page,9599,simonw,closed,0,,,6055094,Datasette 0.52,3,2020-11-12T04:29:55Z,2020-11-28T23:28:35Z,2020-11-12T04:36:52Z,OWNER,,"I got this on my (private) https://dogsheep.simonwillison.net/twitter/tweets/1188612004572880896 page:

<img width=""1059"" alt=""Error_500"" src=""https://user-images.githubusercontent.com/9599/98895899-a6440980-245c-11eb-98e8-fb97d10ec2fd.png"">
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1088/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
741665726,MDU6SXNzdWU3NDE2NjU3MjY=,1089,Sweep documentation for words that minimize involved difficulty,9599,simonw,closed,0,,,6055094,Datasette 0.52,1,2020-11-12T14:53:05Z,2020-11-28T23:28:43Z,2020-11-12T20:07:26Z,OWNER,,Inspired by https://github.com/django/django/pull/11482,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1089/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
274378301,MDU6SXNzdWUyNzQzNzgzMDE=,109,Set up readthedocs,9599,simonw,closed,0,,,,,1,2017-11-16T02:58:01Z,2017-11-16T16:53:26Z,2017-11-16T16:13:56Z,OWNER,,,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/109/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
741862364,MDU6SXNzdWU3NDE4NjIzNjQ=,1090,Custom widgets for canned query forms,9599,simonw,open,0,,,,,3,2020-11-12T19:21:07Z,2021-03-27T16:25:25Z,,OWNER,,"This is an idea that was cut from the first version of writable canned queries:

> I really want the option to use a `<textarea>` for a specific value.
> 
> Idea: metadata syntax like this:

> 
> ```json
> {
>     ""databases"": {
>        ""my-database"": {
>            ""queries"": {
>                ""add_twitter_handle"": {
>                    ""sql"": ""insert into twitter_handles (username) values (:username)"",
>                    ""write"": true,
>                    ""params"": {
>                        ""username"": {
>                             ""widget"": ""textarea""
>                        }
>                    }
>                }
>            }
>        }
>     }
> }
> ```
> 
> I can ship with some default widgets and provide a plugin hook for registering extra widgets.
>
> This opens up some really exciting possibilities for things like map widgets that let you draw polygons.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/698#issuecomment-608125928_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1090/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
742011049,MDU6SXNzdWU3NDIwMTEwNDk=,1091,.json and .csv exports fail to apply base_url,9599,simonw,closed,0,,,6346396,Datasette 0.54,22,2020-11-12T23:45:16Z,2021-01-24T21:20:24Z,2021-01-09T22:19:29Z,OWNER,,"> Just tested with the latest Docker image, and it works pretty much everywhere!  THANK YOU!
> 
> I did notice that if I try to export json or csv, the base is not applied.  Not sure if I should reopen this issue or open a new one.
> 
> To see this, go here: https://corpora.tika.apache.org/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES
> 
> Click/hover over json or CSV and you'll see that the 'datasette' base is not included.

_Originally posted by @tballison in https://github.com/simonw/datasette/issues/865#issuecomment-726385422_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1091/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
742041667,MDU6SXNzdWU3NDIwNDE2Njc=,1092,Make cascading permission checks available to plugins,9599,simonw,closed,0,,,,,1,2020-11-13T01:02:55Z,2023-08-30T22:17:42Z,2023-08-30T22:17:41Z,OWNER,,"The `BaseView` class has a method for cascading permission checks, but it's not easily accessible to plugins.

https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/views/base.py#L75-L99

This leaves plugins like `datasette-graphql` having to implement their own versions of this logic, which is bad: https://github.com/simonw/datasette-graphql/issues/65

> First check `view-database` - if that says `False` then disallow access, if it says `True` then allow access. If it says `None` check `view-instance`.

This should become a supported API that plugins are encouraged to use.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1092/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
743011397,MDU6SXNzdWU3NDMwMTEzOTc=,1094,import EX_CANTCREAT means datasette fails to work on Windows,1049910,drkane,closed,0,,,,,1,2020-11-14T14:17:11Z,2020-12-05T19:35:04Z,2020-12-05T19:35:04Z,NONE,,"Trying to use datasette 0.51.1 gives the following error:

```
ImportError: cannot import name 'EX_CANTCREAT' from 'os' (C:\Users\drkan\AppData\Local\Programs\Python\Python39\lib\os.py)
```

Looks like that code is only available on unix: https://docs.python.org/3/library/os.html#os.EX_CANTCREAT

Removing the line makes it work fine (`EX_CANTCREAT` doesn't seem to be used anywhere?)",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1094/reactions"", ""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
743359646,MDU6SXNzdWU3NDMzNTk2NDY=,1096,TSV should be a default export option,9599,simonw,open,0,,,,,1,2020-11-15T22:24:02Z,2021-06-17T18:12:31Z,,OWNER,,Refs #1095,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1096/reactions"", ""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
743369188,MDExOlB1bGxSZXF1ZXN0NTIxMjc2Mjk2,1097,Use f-strings,9599,simonw,closed,0,,,,,1,2020-11-15T23:12:36Z,2020-11-15T23:24:24Z,2020-11-15T23:24:23Z,OWNER,simonw/datasette/pulls/1097,"Since Datasette now requires Python 3.6, how about some f-strings?

I ran this in the `datasette` root checkout:
```
pip install flynt
flynt .
black .
```",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1097/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
743370900,MDU6SXNzdWU3NDMzNzA5MDA=,1098,Foreign key links break for compound foreign keys,9599,simonw,closed,0,,,,,5,2020-11-15T23:22:14Z,2020-11-29T19:50:31Z,2020-11-29T19:30:23Z,OWNER,,"Reported on Twitter here: https://twitter.com/ZaneSelvans/status/1328093641395548161

> Maybe I'm doing something wrong here but the automatically generated links based foreign key relationships seem to be working here for utility_id_eia, but not for plant_id_eia & generator_id which seems odd: https://pudl-datasette-xl7xwcpe2a-uc.a.run.app/pudl/generators_eia860
>
> Right now it seems like they're trying to, but with only one of the two keys, so it gives ""Error 500. You did not supply a value for binding 2."" Maybe only create the links when it's a simple foreign key?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1098/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
743371103,MDU6SXNzdWU3NDMzNzExMDM=,1099,Support linking to compound foreign keys,9599,simonw,open,0,,,,,6,2020-11-15T23:23:17Z,2023-01-25T00:58:26Z,,OWNER,,Reported as a bug in #1098 because they caused 500 errors - but it would be even better if Datasette could hyperlink to related rows via compound foreign keys.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1099/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
267522549,MDU6SXNzdWUyNjc1MjI1NDk=,11,Code that generates compile-time properties about the database ,9599,simonw,closed,0,,,2857392,Ship first public release,1,2017-10-23T02:18:24Z,2017-10-23T16:04:23Z,2017-10-23T16:04:23Z,OWNER,,"At a minimum this will include:

* sha hash of each database file
* list of tables with row counts for each database file",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/11/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
274578142,MDU6SXNzdWUyNzQ1NzgxNDI=,110,Add --load-extension option to datasette for loading extra SQLite extensions,9599,simonw,closed,0,,,,,2,2017-11-16T16:26:19Z,2017-11-16T18:38:30Z,2017-11-16T16:58:50Z,OWNER,,"This would allow users with extra SQLite extensions installed (like spatialite) to load them at runtime.

Inspired by this comment: https://github.com/simonw/datasette/issues/46#issuecomment-344810525",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/110/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
747702144,MDU6SXNzdWU3NDc3MDIxNDQ=,1100,Error on OPTIONS request to database,1319404,akehrer,closed,0,,,,,2,2020-11-20T18:16:43Z,2020-12-03T00:57:35Z,2020-12-03T00:50:17Z,NONE,,"When I perform an OPTIONS request against a database or table datasette fails with an internal error.

All these tests result in the traceback below.

```
curl -XOPTIONS http://127.0.0.1:8001/test-db
curl -XOPTIONS http://127.0.0.1:8001/test-db/table1
curl -XOPTIONS http://127.0.0.1:8001/test-db/table1\?_search\=test
```

```
Traceback (most recent call last):
  File ""[path-to-python]/site-packages/datasette/app.py"", line 1033, in route_path
    response = await view(request, send)
  File ""[path-to-python]/site-packages/datasette/views/base.py"", line 146, in view
    request, **request.scope[""url_route""][""kwargs""]
  File ""[path-to-python]/site-packages/datasette/views/base.py"", line 118, in dispatch_request
    return await handler(request, *args, **kwargs)
TypeError: object Response can't be used in 'await' expression
```

Making the `options` function in the `DataView` class async fixed it for me.

```python
    async def options(self, request, *args, **kwargs):
        r = Response.text(""ok"")
        if self.ds.cors:
            r.headers[""Access-Control-Allow-Origin""] = ""*""
        return r
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1100/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
749283032,MDU6SXNzdWU3NDkyODMwMzI=,1101,register_output_renderer() should support streaming data,9599,simonw,open,0,,,3268330,Datasette 1.0,13,2020-11-24T02:17:09Z,2023-01-21T22:07:19Z,,OWNER,,"> I'd like to implement this by first extending the `register_output_renderer()` hook to support streaming huge responses, then switching CSV to use the plugin hook in addition to TSV using it.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1096#issuecomment-732542285_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1101/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
749289611,MDU6SXNzdWU3NDkyODk2MTE=,1102,Plugin testing docs should show datasette.client,9599,simonw,closed,0,,,,,1,2020-11-24T02:34:46Z,2020-11-29T07:47:22Z,2020-11-29T07:44:58Z,OWNER,,https://docs.datasette.io/en/stable/testing_plugins.html currently shows how to use HTTPX directly.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1102/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
749979454,MDU6SXNzdWU3NDk5Nzk0NTQ=,1103,Rename /-/config to /-/settings,9599,simonw,closed,0,,,6055094,Datasette 0.52,2,2020-11-24T19:31:00Z,2020-11-24T20:19:20Z,2020-11-24T20:19:19Z,OWNER,,"As part of rebranding config to settings, see also #992.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1103/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
749981663,MDU6SXNzdWU3NDk5ODE2NjM=,1104,config.json in directory config mode should be settings.json,9599,simonw,closed,0,,,6055094,Datasette 0.52,2,2020-11-24T19:34:38Z,2020-11-24T20:37:42Z,2020-11-24T20:37:41Z,OWNER,,"Another knock-on effect of #992.

https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/docs/config.rst#L51-L55",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1104/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
749982022,MDU6SXNzdWU3NDk5ODIwMjI=,1105,Rebrand config as settings,9599,simonw,closed,0,,,6055094,Datasette 0.52,2,2020-11-24T19:35:12Z,2020-11-24T21:40:28Z,2020-11-24T21:40:28Z,OWNER,,"I realized I need a tracking ticket for this.

I want to start splitting things like plugin configuration and default facets / sort order out of `metadata.json` - so I want to start calling those things configuration. But the term configuration is already used for the `--config` family of global settings. So I'm rebranding that type of configuration as settings to free up the name ""configuration"" for more run-time concerns (default sort order) and plugin configuration.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1105/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
749983857,MDU6SXNzdWU3NDk5ODM4NTc=,1106,Rebrand and redirect config.rst as settings.rst,9599,simonw,closed,0,,,6055094,Datasette 0.52,4,2020-11-24T19:38:17Z,2020-11-24T21:39:58Z,2020-11-24T21:39:58Z,OWNER,,"> I'd like to redirect https://docs.datasette.io/en/stable/config.html to a new https://docs.datasette.io/en/stable/settings.html page too. I can use https://docs.readthedocs.io/en/stable/user-defined-redirects.html for that.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1105#issuecomment-733190827_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1106/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
750079085,MDU6SXNzdWU3NTAwNzkwODU=,1107,Rename datasette.config() method to datasette.setting(),9599,simonw,closed,0,,,6055094,Datasette 0.52,5,2020-11-24T21:24:11Z,2020-11-24T22:09:11Z,2020-11-24T22:06:38Z,OWNER,,Part of #1105. Thankfully this isn't yet part of the documented public API on https://docs.datasette.io/en/stable/internals.html,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1107/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
750087350,MDU6SXNzdWU3NTAwODczNTA=,1108,Configure /en/stable/config.html redirect when I ship 0.52,9599,simonw,closed,0,,,6055094,Datasette 0.52,1,2020-11-24T21:39:19Z,2020-11-29T02:42:42Z,2020-11-29T02:42:42Z,OWNER,,"Like this:

<img width=""654"" alt=""Edit_Redirects___Read_the_Docs"" src=""https://user-images.githubusercontent.com/9599/100154418-60bf0d80-2e5a-11eb-92bf-527a14edb5c5.png"">

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1106#issuecomment-733248437_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1108/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
750089847,MDU6SXNzdWU3NTAwODk4NDc=,1109,Deprecate --config in Datasette 1.0 (in favour of --setting),9599,simonw,open,0,,,3268330,Datasette 1.0,0,2020-11-24T21:43:57Z,2020-12-17T22:07:49Z,,OWNER,,I added a deprecation warning to this in #992.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1109/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
274615452,MDU6SXNzdWUyNzQ2MTU0NTI=,111,Add “updated” to metadata,9599,simonw,open,0,,,,,12,2017-11-16T18:22:20Z,2021-09-21T22:48:27Z,,OWNER,,"To give an indication as to when the data was last updated.

This should be a field in the metadata that is then shown on the index page and in the footer, if it is set.

Also support setting it using an option to “datasette publish” and “datasette package” - which can either be a string or can be the magic string “today” to set it to today’s date:

    datasette publish file.db --updated=today",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/111/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
750330029,MDU6SXNzdWU3NTAzMzAwMjk=,1110,datasette publish option for installing extra apt-get packages,9599,simonw,closed,0,,,6055094,Datasette 0.52,2,2020-11-25T03:03:43Z,2020-11-28T23:28:56Z,2020-11-25T03:05:41Z,OWNER,,I ran into a need for this while playing with https://github.com/simonw/datasette-ripgrep - I need to install the `ripgrep` Ubuntu package when I deploy the plugin using Cloud Run.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1110/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
751195017,MDU6SXNzdWU3NTExOTUwMTc=,1111,Accessing a database's `.json` is slow for very large SQLite files,15178711,asg017,open,0,,,,,3,2020-11-26T00:27:27Z,2021-01-04T19:57:53Z,,CONTRIBUTOR,,"I have a SQLite DB that's pretty large, 23GB and something like 300 million rows. I expect that most queries I run on it will be slow, which is fine, but there are some things that Datasette does that makes working with the DB very slow. Specifically, when I access the `.json` metadata for a table (which I believe it comes from `datasette/views/database.py`, it takes 43 seconds for the request to come in:

```bash
$ time curl localhost:9999/out.json
{""database"": ""out"", ""size"": 24291454976, ""tables"": [{""name"": ""PageviewsHour"", ""columns"": [""file"", ""code"", ""page"", ""pageviews""], ""primary_keys"": [], ""count"": null, ""hidden"": false, ""fts_table"": null, ""foreign_keys"": {""incoming"": [], ""outgoing"": [{""other_table"": ""PageviewsHourFiles"", ""column"": ""file"", ""other_column"": ""file_id""}]}, ""private"": false}, {""name"": ""PageviewsHourFiles"", ""columns"": [""file_id"", ""filename"", ""sha256"", ""size"", ""day"", ""hour""], ""primary_keys"": [""file_id""], ""count"": null, ""hidden"": false, ""fts_table"": null, ""foreign_keys"": {""incoming"": [{""other_table"": ""PageviewsHour"", ""column"": ""file_id"", ""other_column"": ""file""}], ""outgoing"": []}, ""private"": false}, {""name"": ""sqlite_sequence"", ""columns"": [""name"", ""seq""], ""primary_keys"": [], ""count"": 1, ""hidden"": false, ""fts_table"": null, ""foreign_keys"": {""incoming"": [], ""outgoing"": []}, ""private"": false}], ""hidden_count"": 0, ""views"": [], ""queries"": [], ""private"": false, ""allow_execute_sql"": true, ""query_ms"": 43340.23213386536}
real	0m43.417s
user	0m0.006s
sys	0m0.016s
```

I suspect this is because a `COUNT(*)` is happening under the hood, which, when I run it through sqlite directly, does take around the same time:

```bash
$ time sqlite3 out.db < <(echo ""select count(*) from PageviewsHour;"")
362794272

real	0m44.523s
user	0m2.497s
sys	0m6.703s
```

I'm using the `.json` request in the [Observable Datasette Client](https://observablehq.com/@asg017/datasette-client) to 1) verify that a link passed in is a reachable Datasette instance, and 2) a quick way to look at metadata for a db. A few different solutions I can think of:

1. Have some other endpoint, like `/-/datasette.json` that the Observable Datasette client can fetch from to verify that the passed in URL is a valid Datasette (doesnt solve the slow problem, feel free to split this issue into 2)
2. Have a way to turn off table counts when accessing a database's `.json` view, like `?no_count=1` or something
3. Maybe have a timeout on the `table_counts()` function if it takes too long. which is odd, because it seems like it already does that (I think?), I can debug a little more if that's the case

More than happy to debug further, or send a PR if you like one of the proposals above!",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1111/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
752749485,MDExOlB1bGxSZXF1ZXN0NTI4OTk3NjE0,1112,Fix --metadata doc usage,50527,jefftriplett,closed,0,,,6055094,Datasette 0.52,3,2020-11-28T19:19:51Z,2020-11-28T23:28:21Z,2020-11-28T19:53:48Z,CONTRIBUTOR,simonw/datasette/pulls/1112,"I stumbled on this while trying to figure out how to configure datasette-ripgrep via https://github.com/simonw/datasette-ripgrep/issues/15

You may not want to update the changelog (those are annoying) so I added two commits in case that's easier. ",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1112/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
752789159,MDU6SXNzdWU3NTI3ODkxNTk=,1113,500 error on row page if query against foreign keys hits time limit,9599,simonw,closed,0,,,,,1,2020-11-28T23:20:08Z,2020-11-29T02:40:01Z,2020-11-28T23:23:31Z,OWNER,,"This page exhibited the following error: https://data.catalyst.coop/ferc1/f1_respondent_id/145

`(OperationalError('interrupted'), 'select (select count(*) from f1_acb_epda where respondent_id=:id), (select count(*) from f1_accumdepr_prvsn where respondent_id=:id), (select count(*) from f1_accumdfrrdtaxcr where respondent_id=:id), (select count(*) from f1_adit_190_detail where respondent_id=:id), (select count(*) from f1_adit_190_notes where respondent_id=:id), (select count(*) from f1_adit_amrt_prop where respondent_id=:id), (select count(*) from f1_adit_other where respondent_id=:id), (select count(*) from f1_adit_other_prop where respondent_id=:id), (select count(*) from f1_allowances where respondent_id=:id), (select count(*) from f1_bal_sheet_cr where respondent_id=:id), (select count(*) from f1_capital_stock where respondent_id=:id), (select count(*) from f1_cash_flow where respondent_id=:id), (select count(*) from f1_cmmn_utlty_p_e where respondent_id=:id), (select count(*) from f1_comp_balance_db where respondent_id=:id), (select count(*) from f1_construction where respondent_id=:id), (select count(*) from f1_control_respdnt where respondent_id=:id), (select count(*) from f1_co_directors where respondent_id=:id), (select count(*) from f1_cptl_stk_expns where respondent_id=:id), (select count(*) from f1_csscslc_pcsircs where respondent_id=:id), (select count(*) from f1_dacs_epda where respondent_id=:id), (select count(*) from f1_dscnt_cptl_stk where respondent_id=:id), (select count(*) from f1_edcfu_epda where respondent_id=:id), (select count(*) from f1_elctrc_erg_acct where respondent_id=:id), (select count(*) from f1_elctrc_oper_rev where respondent_id=:id), (select count(*) from f1_elc_oper_rev_nb where respondent_id=:id), (select count(*) from f1_elc_op_mnt_expn where respondent_id=:id), (select count(*) from f1_electric where respondent_id=:id), (select count(*) from f1_envrnmntl_expns where respondent_id=:id), (select count(*) from f1_envrnmntl_fclty where respondent_id=:id), (select count(*) from f1_fuel where respondent_id=:id), (select count(*) from f1_general_info where respondent_id=:id), (select count(*) from f1_gnrt_plant where respondent_id=:id), (select count(*) from f1_important_chg where respondent_id=:id), (select count(*) from f1_incm_stmnt_2 where respondent_id=:id), (select count(*) from f1_income_stmnt where respondent_id=:id), (select count(*) from f1_miscgen_expnelc where respondent_id=:id), (select count(*) from f1_misc_dfrrd_dr where respondent_id=:id), (select count(*) from f1_mthly_peak_otpt where respondent_id=:id), (select count(*) from f1_mtrl_spply where respondent_id=:id), (select count(*) from f1_nbr_elc_deptemp where respondent_id=:id), (select count(*) from f1_nonutility_prop where respondent_id=:id), (select count(*) from f1_nuclear_fuel where respondent_id=:id), (select count(*) from f1_officers_co where respondent_id=:id), (select count(*) from f1_othr_dfrrd_cr where respondent_id=:id), (select count(*) from f1_othr_pd_in_cptl where respondent_id=:id), (select count(*) from f1_othr_reg_assets where respondent_id=:id), (select count(*) from f1_othr_reg_liab where respondent_id=:id), (select count(*) from f1_overhead where respondent_id=:id), (select count(*) from f1_pccidica where respondent_id=:id), (select count(*) from f1_plant_in_srvce where respondent_id=:id), (select count(*) from f1_pumped_storage where respondent_id=:id), (select count(*) from f1_purchased_pwr where respondent_id=:id), (select count(*) from f1_reconrpt_netinc where respondent_id=:id), (select count(*) from f1_reg_comm_expn where respondent_id=:id), (select count(*) from f1_respdnt_control where respondent_id=:id), (select count(*) from f1_retained_erng where respondent_id=:id), (select count(*) from f1_r_d_demo_actvty where respondent_id=:id), (select count(*) from f1_sales_by_sched where respondent_id=:id), (select count(*) from f1_sale_for_resale where respondent_id=:id), (select count(*) from f1_sbsdry_totals where respondent_id=:id), (select count(*) from f1_schedules_list where respondent_id=:id), (select count(*) from f1_security_holder where respondent_id=:id), (select count(*) from f1_slry_wg_dstrbtn where respondent_id=:id), (select count(*) from f1_substations where respondent_id=:id), (select count(*) from f1_taxacc_ppchrgyr where respondent_id=:id), (select count(*) from f1_unrcvrd_cost where respondent_id=:id), (select count(*) from f1_utltyplnt_smmry where respondent_id=:id), (select count(*) from f1_work where respondent_id=:id), (select count(*) from f1_xmssn_adds where respondent_id=:id), (select count(*) from f1_xmssn_elc_bothr where respondent_id=:id), (select count(*) from f1_xmssn_elc_fothr where respondent_id=:id), (select count(*) from f1_xmssn_line where respondent_id=:id), (select count(*) from f1_xtraordnry_loss where respondent_id=:id), (select count(*) from f1_audit_log where respondent_id=:id), (select count(*) from f1_load_file_names where respondent_id=:id), (select count(*) from f1_privilege where respondent_id=:id), (select count(*) from f1_sys_error_log where respondent_id=:id), (select count(*) from f1_hydro where respondent_id=:id), (select count(*) from f1_ident_attsttn where respondent_id=:id), (select count(*) from f1_steam where respondent_id=:id), (select count(*) from f1_leased where respondent_id=:id), (select count(*) from f1_sbsdry_detail where respondent_id=:id), (select count(*) from f1_plant where respondent_id=:id), (select count(*) from f1_long_term_debt where respondent_id=:id), (select count(*) from f1_106_2009 where respondent_id=:id), (select count(*) from f1_106a_2009 where respondent_id=:id), (select count(*) from f1_106b_2009 where respondent_id=:id), (select count(*) from f1_208_elc_dep where respondent_id=:id), (select count(*) from f1_231_trn_stdycst where respondent_id=:id), (select count(*) from f1_324_elc_expns where respondent_id=:id), (select count(*) from f1_325_elc_cust where respondent_id=:id), (select count(*) from f1_331_transiso where respondent_id=:id), (select count(*) from f1_338_dep_depl where respondent_id=:id), (select count(*) from f1_397_isorto_stl where respondent_id=:id), (select count(*) from f1_398_ancl_ps where respondent_id=:id), (select count(*) from f1_399_mth_peak where respondent_id=:id), (select count(*) from f1_400_sys_peak where respondent_id=:id), (select count(*) from f1_400a_iso_peak where respondent_id=:id), (select count(*) from f1_429_trans_aff where respondent_id=:id), (select count(*) from f1_allowances_nox where respondent_id=:id), (select count(*) from f1_cmpinc_hedge_a where respondent_id=:id), (select count(*) from f1_cmpinc_hedge where respondent_id=:id), (select count(*) from f1_email where respondent_id=:id), (select count(*) from f1_rg_trn_srv_rev where respondent_id=:id), (select count(*) from f1_s0_filing_log where respondent_id=:id), (select count(*) from f1_security where respondent_id=:id)', {'id': '145'})`",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1113/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
752966476,MDU6SXNzdWU3NTI5NjY0NzY=,1114,--load-extension=spatialite not working with datasetteproject/datasette docker image,2182,danp,closed,0,,,,,4,2020-11-29T17:35:20Z,2022-01-20T21:29:42Z,2020-11-29T17:37:45Z,CONTRIBUTOR,,"https://github.com/simonw/datasette/commit/6aa5886379dd9017215904fb28567b80018902f9 added the `--load-extension=spatialite` shortcut looking for the extension in these places:

https://github.com/simonw/datasette/blob/12877d7a48e2aa28bb5e780f929a218f7265d849/datasette/utils/__init__.py#L56-L60

However, in the datasetteproject/datasette docker image the file is at `/usr/local/lib/mod_spatialite.so`.

This results in the example command [here](https://docs.datasette.io/en/stable/installation.html#loading-spatialite) failing:

```
% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=spatialite
Error: Could not find SpatiaLite extension
```

But it does work when given an explicit path:

```
% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=/usr/local/lib/mod_spatialite.so
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
...
```

Perhaps `SPATIALITE_PATHS` should include `/usr/local/lib/mod_spatialite.so`?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1114/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
752995227,MDU6SXNzdWU3NTI5OTUyMjc=,1115,SpatiaLite error could suggest --load-extension=spatialite,9599,simonw,closed,0,,,,,1,2020-11-29T20:05:07Z,2022-01-20T21:29:42Z,2020-11-29T20:13:22Z,OWNER,,"https://github.com/simonw/datasette/blob/242bc89fdf2e775e340d69a4e851b3a9accb31c6/datasette/cli.py#L533-L548

This could use the `find_spatialite()` function and, if it finds something, suggest the user use `--load-extension=spatialite`

https://github.com/simonw/datasette/blob/242bc89fdf2e775e340d69a4e851b3a9accb31c6/datasette/utils/__init__.py#L1015-L1019",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1115/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
753668177,MDU6SXNzdWU3NTM2NjgxNzc=,1116,GENERATED column support,2789593,nattaylor,closed,0,,,,,9,2020-11-30T17:33:47Z,2020-11-30T21:29:59Z,2020-11-30T21:29:59Z,NONE,,"I think this is a feature request... perhaps I should just try to contribute it myself, but thought I'd check in case support is planned already.

For a table with the following schema, datasette 0.51.1 doesn't pick up the GENERATED columns and the column list only contains `(rowid, body)`  If I edit the SQL and select the generated columns, it will happily show them.

At first glance it appears that [`def table_column_details(conn, table):`](https://github.com/simonw/datasette/blob/4777362bf2692bc72b221ec47c3e6216151d1b89/datasette/utils/__init__.py#L575) would have to be refactored to use a different methodology to get the columns, since `PRAGMA table_info(deeds);` returns just `0|body|TEXT|0||0` so maybe it wouldn't be worth it.

```
CREATE TABLE deeds (
    body TEXT,
    id INT GENERATED ALWAYS AS (json_extract(body, '$.id')) STORED,
    consideration INT GENERATED ALWAYS AS (json_extract(body, '$.consideration')) STORED
);
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1116/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
753767911,MDExOlB1bGxSZXF1ZXN0NTI5NzgzMjc1,1117,Support for generated columns,9599,simonw,closed,0,,,,,6,2020-11-30T20:10:46Z,2020-11-30T22:23:19Z,2020-11-30T21:29:58Z,OWNER,simonw/datasette/pulls/1117,"Refs #1116. My first attempt at this worked on my laptop but broke in CI, so I'm going to iterate on it in a pull request instead.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1117/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
753788261,MDU6SXNzdWU3NTM3ODgyNjE=,1118,messagge_is_html typo,9599,simonw,closed,0,,,,,0,2020-11-30T20:43:22Z,2020-11-30T21:24:28Z,2020-11-30T21:24:28Z,OWNER,,https://ripgrep.datasette.io/-/ripgrep?pattern=messagge_is_html,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1118/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
753876808,MDU6SXNzdWU3NTM4NzY4MDg=,1119,"Include generated columns in fixtures.db, if SQLite version supports it",9599,simonw,closed,0,,,,,1,2020-11-30T23:25:31Z,2020-12-01T00:41:14Z,2020-12-01T00:28:04Z,OWNER,,"In #1117 I added a one-off test that creates a DB with generated columns in it:

https://github.com/simonw/datasette/blob/461670a0b87efa953141b449a9a261919864ceb3/tests/test_api.py#L1933-L1964

If this table was conditionally added to `fixtures.db` (if SQLite supports the feature) it would be available in the demo at https://latest.datasette.io/",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1119/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
274617240,MDU6SXNzdWUyNzQ2MTcyNDA=,112,Allow --load-extension to be set via environment variables,9599,simonw,closed,0,,,,,1,2017-11-16T18:28:31Z,2017-11-17T14:19:23Z,2017-11-17T14:17:27Z,OWNER,,"This will make it easier to package up datasette in a Docker container with a bunch of pre-compiled extensions without the user having to remember to include all of the options every time.

Click has a mechanism for this: http://click.pocoo.org/5/options/#multiple-values-from-environment-values",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/112/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
753898359,MDExOlB1bGxSZXF1ZXN0NTI5ODg3ODYx,1120,generated_columns table in fixtures.py,9599,simonw,closed,0,,,,,1,2020-12-01T00:17:19Z,2020-12-01T00:28:03Z,2020-12-01T00:28:02Z,OWNER,simonw/datasette/pulls/1120,Refs #1119,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1120/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
754178780,MDU6SXNzdWU3NTQxNzg3ODA=,1121,Table actions cog is misaligned,3243482,abdusco,closed,0,,,,,1,2020-12-01T08:41:25Z,2020-12-03T01:03:19Z,2020-12-03T00:33:36Z,CONTRIBUTOR,,"At the moment it looks like this
https://datasette-graphql-demo.datasette.io/github/repos

![image](https://user-images.githubusercontent.com/3243482/100716533-e6e2d300-33c9-11eb-866e-1e83ba228bf5.png)

Adding a few flex statements fixes the alignment and centers `h1` text and the cog icon vertically.
![image](https://user-images.githubusercontent.com/3243482/100716605-f8c47600-33c9-11eb-8d69-0e37499cf641.png)

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1121/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
754179035,MDExOlB1bGxSZXF1ZXN0NTMwMTI1Njk1,1122,Fix misaligned table actions cog,3243482,abdusco,closed,0,,,,,2,2020-12-01T08:41:46Z,2020-12-03T10:56:40Z,2020-12-03T00:33:37Z,CONTRIBUTOR,simonw/datasette/pulls/1122,Fixes https://github.com/simonw/datasette/issues/1121,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1122/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
755721275,MDU6SXNzdWU3NTU3MjEyNzU=,1123,"Table actions hook are order dependent, should not be",9599,simonw,closed,0,,,,,1,2020-12-03T00:47:12Z,2020-12-03T00:50:17Z,2020-12-03T00:50:17Z,OWNER,,"Got this error: https://github.com/simonw/datasette/runs/1489770800
```
>       assert get_table_actions_links(response_2.text) == [
            {""label"": ""From async"", ""href"": ""/""},
            {""label"": ""Database: fixtures"", ""href"": ""/""},
            {""label"": f""Table: {table_or_view}"", ""href"": ""/""},
        ]
E       AssertionError: assert [{'href': '/'...'From async'}] == [{'href': '/'...: facetable'}]
E         At index 0 diff: {'label': 'Database: fixtures', 'href': '/'} != {'label': 'From async', 'href': '/'}
E         Use -v to get the full diff
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1123/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
756439516,MDU6SXNzdWU3NTY0Mzk1MTY=,1124,Datasette on Amazon Linux on ARM returns 404 for static assets,9599,simonw,closed,0,,,,,9,2020-12-03T18:20:37Z,2020-12-03T21:42:02Z,2020-12-03T21:10:54Z,OWNER,,"Very weird bug this one. Steps to reproduce:
```
# I started a amzn2-ami-hvm-2.0.20201126.0-arm64-gp2 t4g.micro instance
ec2 % ssh -i simonw-ec2.pem ec2-user@ec2-18-219-238-192.us-east-2.compute.amazonaws.com
[ec2-user@ip-172-31-30-7 ~]$ sudo yum install python3
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
...
Is this ok [y/d/N]: y
Downloading packages:
(1/4): python3-3.7.9-1.amzn2.0.1.aarch64.rpm                                                                                        |  72 kB  00:00:00     
(2/4): python3-pip-9.0.3-1.amzn2.0.2.noarch.rpm                                                                                     | 1.9 MB  00:00:00     
(3/4): python3-setuptools-38.4.0-3.amzn2.0.6.noarch.rpm                                                                             | 617 kB  00:00:00     
(4/4): python3-libs-3.7.9-1.amzn2.0.1.aarch64.rpm                                                                                   | 9.1 MB  00:00:00     
-----------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                       68 MB/s |  12 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : python3-pip-9.0.3-1.amzn2.0.2.noarch                                                                                                    1/4 
  Installing : python3-3.7.9-1.amzn2.0.1.aarch64                                                                                                       2/4 
  Installing : python3-setuptools-38.4.0-3.amzn2.0.6.noarch                                                                                            3/4 
  Installing : python3-libs-3.7.9-1.amzn2.0.1.aarch64                                                                                                  4/4 
  Verifying  : python3-setuptools-38.4.0-3.amzn2.0.6.noarch                                                                                            1/4 
  Verifying  : python3-pip-9.0.3-1.amzn2.0.2.noarch                                                                                                    2/4 
  Verifying  : python3-3.7.9-1.amzn2.0.1.aarch64                                                                                                       3/4 
  Verifying  : python3-libs-3.7.9-1.amzn2.0.1.aarch64                                                                                                  4/4 

Installed:
  python3.aarch64 0:3.7.9-1.amzn2.0.1                                                                                                                      

Dependency Installed:
  python3-libs.aarch64 0:3.7.9-1.amzn2.0.1          python3-pip.noarch 0:9.0.3-1.amzn2.0.2          python3-setuptools.noarch 0:38.4.0-3.amzn2.0.6         

Complete!
[ec2-user@ip-172-31-30-7 ~]$ python3 -m pip install --user pipx
Collecting pipx
  Downloading https://files.pythonhosted.org/packages/42/8a/8447ec14562c5d97afbee54ec390864718bccce9dfb0506c8c77852489d3/pipx-0.15.6.0-py3-none-any.whl (43kB)
    100% |████████████████████████████████| 51kB 3.8MB/s 
Collecting packaging>=20.0 (from pipx)
  Downloading https://files.pythonhosted.org/packages/28/87/8edcf555adaf60d053ead881bc056079e29319b643ca710339ce84413136/packaging-20.7-py2.py3-none-any.whl
Collecting argcomplete<2.0,>=1.9.4 (from pipx)
  Downloading https://files.pythonhosted.org/packages/e3/d0/ee7fc6ceac8957196def9bfa3b955d02163058defd3edd51ef7b1ff2769e/argcomplete-1.12.2-py2.py3-none-any.whl
Collecting userpath>=1.4.1 (from pipx)
  Downloading https://files.pythonhosted.org/packages/fb/f1/faca8a3cc86bd2223aaf72edd222bc31d6ae2eea5feaf17a144634a92e6d/userpath-1.4.1-py2.py3-none-any.whl
Collecting pyparsing>=2.0.2 (from packaging>=20.0->pipx)
  Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67kB)
    100% |████████████████████████████████| 71kB 8.8MB/s 
Collecting importlib-metadata<4,>=0.23; python_version == ""3.7"" (from argcomplete<2.0,>=1.9.4->pipx)
  Downloading https://files.pythonhosted.org/packages/99/c7/4ccf2baa455613aa9e61372365aba8594ff2806c82189c31a6c65e7c679e/importlib_metadata-3.1.1-py3-none-any.whl
Collecting click (from userpath>=1.4.1->pipx)
  Downloading https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl (82kB)
    100% |████████████████████████████████| 92kB 10.5MB/s 
Collecting distro; platform_system == ""Linux"" (from userpath>=1.4.1->pipx)
  Downloading https://files.pythonhosted.org/packages/25/b7/b3c4270a11414cb22c6352ebc7a83aaa3712043be29daa05018fd5a5c956/distro-1.5.0-py2.py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata<4,>=0.23; python_version == ""3.7""->argcomplete<2.0,>=1.9.4->pipx)
  Downloading https://files.pythonhosted.org/packages/41/ad/6a4f1a124b325618a7fb758b885b68ff7b058eec47d9220a12ab38d90b1f/zipp-3.4.0-py3-none-any.whl
Installing collected packages: pyparsing, packaging, zipp, importlib-metadata, argcomplete, click, distro, userpath, pipx
Successfully installed argcomplete-1.12.2 click-7.1.2 distro-1.5.0 importlib-metadata-3.1.1 packaging-20.7 pipx-0.15.6.0 pyparsing-2.4.7 userpath-1.4.1 zipp-3.4.0
[ec2-user@ip-172-31-30-7 ~]$ python3 -m pipx ensurepath
/home/ec2-user/.local/bin is already in PATH.
/home/ec2-user/.local/bin is already in PATH.

All pipx binary directories have been added to PATH. If you are sure
you want to proceed, try again with the '--force' flag.

Otherwise pipx is ready to go! ✨ 🌟 ✨
[ec2-user@ip-172-31-30-7 ~]$ pipx install datasette
  installed package datasette 0.52.2, Python 3.7.9
  These apps are now globally available
    - datasette
done! ✨ 🌟 ✨
[ec2-user@ip-172-31-30-7 ~]$ datasette --version
datasette, version 0.52.2
[ec2-user@ip-172-31-30-7 ~]$ datasette --get /-/static/app.css
404
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1124/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
756622648,MDU6SXNzdWU3NTY2MjI2NDg=,1125,Show pysqlite3 version on /-/versions,9599,simonw,closed,0,,,,,5,2020-12-03T21:57:23Z,2020-12-04T04:16:57Z,2020-12-04T04:16:57Z,OWNER,,"This code can use `pysqlite3` if available. The version should be exposed on `/-/versions`.

https://github.com/simonw/datasette/blob/4cce5516661b24afeddaf35bee84b00fbf5c7f89/datasette/utils/sqlite.py#L1-L4",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1125/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
756761963,MDU6SXNzdWU3NTY3NjE5NjM=,1126,Switch to google-github-actions/setup-gcloud for demo deploy,9599,simonw,closed,0,,,,,1,2020-12-04T03:11:24Z,2020-12-04T03:51:38Z,2020-12-04T03:51:38Z,OWNER,,"That workflow is showing warnings: https://github.com/simonw/datasette/actions/runs/399400077

> Thank you for using setup-gcloud Action. GoogleCloudPlatform/github-actions/setup-gcloud has been deprecated, please switch to google-github-actions/setup-gcloud.

https://github.com/google-github-actions/setup-gcloud has a Cloud Run example here: https://github.com/google-github-actions/setup-gcloud/blob/master/example-workflows/cloud-run/.github/workflows/cloud-run.yml",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1126/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
756818250,MDU6SXNzdWU3NTY4MTgyNTA=,1127,Make the custom SQL query text box larger or resizable,596279,zaneselvans,closed,0,,,,,1,2020-12-04T05:37:11Z,2021-06-02T04:29:06Z,2021-06-02T04:28:55Z,NONE,,"The text entry field for custom SQL queries is too small to display a moderately complex query, especially when it's been formatted. Would it be easy to make the textbox resizable by the user rather than having a fixed height?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1127/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
756867924,MDExOlB1bGxSZXF1ZXN0NTMyMzQyMDI1,1128,Fix startup error on windows,3243482,abdusco,closed,0,,,,,2,2020-12-04T07:12:26Z,2020-12-06T08:41:45Z,2020-12-05T19:35:04Z,CONTRIBUTOR,simonw/datasette/pulls/1128,"Fixes https://github.com/simonw/datasette/issues/1094

This import isn't used at all, and causes error on startup on Windows.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1128/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
756875827,MDU6SXNzdWU3NTY4NzU4Mjc=,1129,Fix footer to the bottom of the page,3243482,abdusco,open,0,,,,,0,2020-12-04T07:28:07Z,2020-12-04T16:04:29Z,,CONTRIBUTOR,,"Footer doesn't stick to the bottom if the body content isn't long enough to reach the end of viewport.

![before & after](https://user-images.githubusercontent.com/3243482/101134785-f6595a80-361b-11eb-81ce-b8b5cb9c5bc2.png)


This can be fixed using flexbox.

```css
body {
  min-height: 100vh;
  display: flex;
  flex-direction: column;
}

.content {
  flex-grow: 1;
}
```
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1129/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
274662378,MDU6SXNzdWUyNzQ2NjIzNzg=,113,Fix the &nbsp; bug on the database custom SQL query view,9599,simonw,closed,0,,,2919870,Foreign key edition,0,2017-11-16T21:01:26Z,2017-11-17T15:40:52Z,2017-11-17T15:40:52Z,OWNER,,"https://sf-film-locations.now.sh/sf-film-locations-57704b7?sql=select+*+from+Film_Locations_in_San_Francisco

<img width=""852"" alt=""sf-film-locations"" src=""https://user-images.githubusercontent.com/9599/32915540-19bad984-cace-11e7-9ed3-bd7294891dc0.png"">

This is the bug I fixed in 01e0c3fa18cd0dd7970e208790ffd683a420c924 - but I only fixed it in one place.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/113/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
756876238,MDExOlB1bGxSZXF1ZXN0NTMyMzQ4OTE5,1130,Fix footer not sticking to bottom in short pages,3243482,abdusco,open,0,,,,,4,2020-12-04T07:29:01Z,2021-06-15T13:27:48Z,,CONTRIBUTOR,simonw/datasette/pulls/1130,Fixes https://github.com/simonw/datasette/issues/1129,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1130/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
757481949,MDU6SXNzdWU3NTc0ODE5NDk=,1131,"""datasette inspect"" outputs invalid JSON if an error is logged",9599,simonw,closed,0,,,,,3,2020-12-05T00:00:45Z,2020-12-05T20:48:34Z,2020-12-05T05:21:19Z,OWNER,,"See https://github.com/simonw/register-of-members-interests/issues/6:
```
% datasette inspect regmem.db                       
ERROR: conn=<sqlite3.Connection object at 0x10d2a9c60>, sql = 'select count(*) from [items_fts]', params = None: SQL logic error
{
  ""regmem"": {
    ""hash"": ""6fde27e3dea80d6b65f2ac7f89cd8448980fee8c91b505ba29c311ba0393317f"",
    ""size"": 936198144,
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1131/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
758899581,MDU6SXNzdWU3NTg4OTk1ODE=,1132,New filter: array does not contain,9599,simonw,closed,0,,,,,1,2020-12-07T22:28:20Z,2020-12-07T22:50:36Z,2020-12-07T22:41:09Z,OWNER,,"I want to see all of my GitHub repos that are tagged `datasette-plugin` but are NOT tagged `datasette-io`:

https://github-to-sqlite.dogsheep.net/github/repos?_facet=owner&owner=9599&_facet_array=topics&topics__arraycontains=datasette-plugin&topics__arraycontains=datasette-io#facet-topics",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1132/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
759695780,MDU6SXNzdWU3NTk2OTU3ODA=,1133,Option to omit header row in CSV export,9599,simonw,closed,0,,,,,2,2020-12-08T18:54:46Z,2021-06-17T18:12:31Z,2020-12-10T23:28:51Z,OWNER,,`?_header=off` - for symmetry with existing option `?_nl=on`.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1133/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
760312579,MDU6SXNzdWU3NjAzMTI1Nzk=,1134,"""_searchmode=raw"" throws an index out of range error when combined with ""_search_COLUMN""",2181410,clausjuhl,closed,0,,,,,4,2020-12-09T13:05:37Z,2020-12-10T05:57:17Z,2020-12-09T19:56:55Z,NONE,,"Hi Simon!
Maybe it's just me, but when [using _searchmode=raw (trying to enable wildcard-searching) in combination with the ""_search_COLUMN""-table argument](https://byraadsarkivet.aarhus.dk/db/cases?_searchmode=raw&_search_title=sundhedsfrem*), I get a list index out of range error. [When combining with the simpler ""_search""-argument everything works, including wildcard-seaches.](https://byraadsarkivet.aarhus.dk/db/cases?_search=sundhedsfrem*&_searchmode=raw). Here's the traceback:

```
Traceback (most recent call last):
  File ""/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/utils/asgi.py"", line 122, in route_path
    return await view(new_scope, receive, send)
  File ""/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/utils/asgi.py"", line 196, in view
    request, **scope[""url_route""][""kwargs""]
  File ""/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/views/base.py"", line 204, in get
    request, database, hash, correct_hash_provided, **kwargs
  File ""/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/views/base.py"", line 342, in view_get
    request, database, hash, **kwargs
  File ""/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/views/table.py"", line 393, in data
    search_col = key.split(""_search_"", 1)[1]
IndexError: list index out of range

```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1134/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
760605882,MDU6SXNzdWU3NjA2MDU4ODI=,1135,Feature: --create option to create database file if it does not yet exist,9599,simonw,closed,0,,,,,0,2020-12-09T19:23:58Z,2021-01-24T21:19:39Z,2020-12-09T19:45:52Z,OWNER,,"I'd like to be able to tell people to run the following in the Datasette documentation to get started:

    brew install datasette
    datasette install datasette-upload-csvs
    datasette data.db --create --root --open

This would give them a local Datasette instance with the ability to drag-and-drop CSV files directly into it.

Just one catch: I don't want to have to talk them through creating an empty SQLite database file. So I want to add a new `--create` option which means ""If any of the database files passed on the command-line do not yet exist, create them"".",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1135/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
760621356,MDU6SXNzdWU3NjA2MjEzNTY=,1136,Establish pattern for release branches to support bug fixes,9599,simonw,closed,0,,,,,8,2020-12-09T19:48:18Z,2020-12-09T20:17:02Z,2020-12-09T20:14:41Z,OWNER,,"I want to fix the bug in #1134 and ship it as Datasette 0.52.5 - but the `main` branch now has a feature in it (4c25b035b2370983c8dd5e0c8762e9154e379774 added `arraynotcontains`, #1132).

I'm not ready for a feature release, so instead I want to release 0.52.5 with just that bug fix.

This is the first time I will have shipped a release from a branch. I need to establish that pattern and add it to the documentation in https://docs.datasette.io/en/stable/contributing.html",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1136/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
761706858,MDU6SXNzdWU3NjE3MDY4NTg=,1137,Update README to reflect new datasette.io site,9599,simonw,closed,0,,,,,0,2020-12-10T23:22:06Z,2020-12-10T23:28:50Z,2020-12-10T23:28:50Z,OWNER,,Can finally close #659.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1137/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
761713079,MDU6SXNzdWU3NjE3MTMwNzk=,1138,"""Powered by Datasette"" should link to new datasette.io site",9599,simonw,closed,0,,,,,0,2020-12-10T23:33:41Z,2020-12-15T02:28:10Z,2020-12-10T23:37:14Z,OWNER,,https://datasette.io/,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1138/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
274733145,MDExOlB1bGxSZXF1ZXN0MTUzMjAxOTQ1,114,"Add spatialite, switch to debian and local build",54999,ingenieroariel,closed,0,,,,,1,2017-11-17T02:37:09Z,2017-11-17T03:50:52Z,2017-11-17T03:50:52Z,CONTRIBUTOR,simonw/datasette/pulls/114,"Improves the Dockerfile to support spatial datasets, work with the local datasette code (Friendly with git tags and Dockerhub) and moves to slim debian, a small image easy to extend via apt packages for sqlite.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/114/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
763207948,MDU6SXNzdWU3NjMyMDc5NDg=,1141,Default styling for bullet point lists,9599,simonw,closed,0,,,,,0,2020-12-12T02:49:33Z,2021-03-29T00:14:05Z,2021-03-29T00:14:05Z,OWNER,,"I just noticed that https://datasette.io/content/recent_releases (which uses `datasette-render-markdown`) is missing its bullet points:

<img width=""864"" alt=""content__recent_releases__399_rows"" src=""https://user-images.githubusercontent.com/9599/101970823-95b0bb80-3be1-11eb-9512-9274cef64cf3.png"">
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1141/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
763361458,MDU6SXNzdWU3NjMzNjE0NTg=,1142,"""Stream all rows"" is not at all obvious",9599,simonw,open,0,,,,,9,2020-12-12T06:24:57Z,2021-06-17T18:12:31Z,,OWNER,,"Got a question about how to download all rows - the current option isn't at all clear.

<img width=""668"" alt=""loans__ppp_loans__9_511_rows_where_where_search_matches__tech__sorted_by_rowid"" src=""https://user-images.githubusercontent.com/9599/101977057-ac660b00-3bff-11eb-88f4-c93ffd03d3e0.png"">
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1142/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
764059235,MDU6SXNzdWU3NjQwNTkyMzU=,1143,"More flexible CORS support in core, to encourage good security practices",114388,yurivish,open,0,,,3268330,Datasette 1.0,6,2020-12-12T17:06:35Z,2022-02-13T17:41:17Z,,NONE,,"It would be nice if the `--cors` option accepted an origin regex to more securely allow secure local development.

As an example, Observable notebooks namespace every user's notebooks by their username and user content is served from username.observableusercontent.com, so you would set `--cors-origin username.observableusercontent.com` to restrict access to a local development Datasette instance to only your own notebooks, rather than exposing the data to any website that makes a request.

Thank you for all of your work on Datasette!",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1143/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
765637324,MDU6SXNzdWU3NjU2MzczMjQ=,1144,JavaScript to help plugins interact with the fragment part of the URL,9599,simonw,open,0,,,,,1,2020-12-13T20:36:06Z,2020-12-14T14:47:11Z,,OWNER,,"Suggested by Markus Holtermann on Twitter, who is building https://github.com/MarkusH/datasette-chartjs

> I've been looking at datasette-vega for how you persist chart settings between form submissions. I've adopted that for datasette-chartjs. Any thoughts on adding a public JS API to #datasette itself, that plugins can rely on?
>
> I'm talking about functions like onFragmentChange, serialize, unserialize, ... That turn an object into a URL encoded string and put it into the location's hash. And also updating all links/forms automatically.
>
> Essentially, a plugins could do something like `document.datasette.setConfigValue('prefix', 'foo', 'bar')` and `.getConfigValue('prefix', 'foo')`. And the functions would take care of updating document.location.hash, all (necessary) a.href and form.action

https://twitter.com/m_holtermann/status/1338183973311295492",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1144/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
766494367,MDExOlB1bGxSZXF1ZXN0NTM5NDg5NTI1,1145,"Update pytest requirement from <6.2.0,>=5.2.2 to >=5.2.2,<6.3.0",27856297,dependabot-preview[bot],closed,0,,,6346396,Datasette 0.54,1,2020-12-14T14:22:16Z,2021-01-24T21:20:29Z,2020-12-16T21:44:39Z,CONTRIBUTOR,simonw/datasette/pulls/1145,"Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/pytest-dev/pytest/releases"">pytest's releases</a>.</em></p>
<blockquote>
<h2>6.2.0</h2>
<h1>pytest 6.2.0 (2020-12-12)</h1>
<h2>Breaking Changes</h2>
<ul>
<li><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7808"">#7808</a>: pytest now supports python3.6+ only.</li>
</ul>
<h2>Deprecations</h2>
<ul>
<li>
<p><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7469"">#7469</a>: Directly constructing/calling the following classes/functions is now deprecated:</p>
<ul>
<li><code>_pytest.cacheprovider.Cache</code></li>
<li><code>_pytest.cacheprovider.Cache.for_config()</code></li>
<li><code>_pytest.cacheprovider.Cache.clear_cache()</code></li>
<li><code>_pytest.cacheprovider.Cache.cache_dir_from_config()</code></li>
<li><code>_pytest.capture.CaptureFixture</code></li>
<li><code>_pytest.fixtures.FixtureRequest</code></li>
<li><code>_pytest.fixtures.SubRequest</code></li>
<li><code>_pytest.logging.LogCaptureFixture</code></li>
<li><code>_pytest.pytester.Pytester</code></li>
<li><code>_pytest.pytester.Testdir</code></li>
<li><code>_pytest.recwarn.WarningsRecorder</code></li>
<li><code>_pytest.recwarn.WarningsChecker</code></li>
<li><code>_pytest.tmpdir.TempPathFactory</code></li>
<li><code>_pytest.tmpdir.TempdirFactory</code></li>
</ul>
<p>These have always been considered private, but now issue a deprecation warning, which may become a hard error in pytest 7.0.0.</p>
</li>
<li>
<p><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7530"">#7530</a>: The <code>--strict</code> command-line option has been deprecated, use <code>--strict-markers</code> instead.</p>
<p>We have plans to maybe in the future to reintroduce <code>--strict</code> and make it an encompassing flag for all strictness
related options (<code>--strict-markers</code> and <code>--strict-config</code> at the moment, more might be introduced in the future).</p>
</li>
<li>
<p><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7988"">#7988</a>: The <code>@pytest.yield_fixture</code> decorator/function is now deprecated. Use pytest.fixture instead.</p>
<p><code>yield_fixture</code> has been an alias for <code>fixture</code> for a very long time, so can be search/replaced safely.</p>
</li>
</ul>
<h2>Features</h2>
<ul>
<li>
<p><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/5299"">#5299</a>: pytest now warns about unraisable exceptions and unhandled thread exceptions that occur in tests on Python&gt;=3.8.
See unraisable for more information.</p>
</li>
<li>
<p><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7425"">#7425</a>: New pytester fixture, which is identical to testdir but its methods return pathlib.Path when appropriate instead of <code>py.path.local</code>.</p>
<p>This is part of the movement to use pathlib.Path objects internally, in order to remove the dependency to <code>py</code> in the future.</p>
<p>Internally, the old Testdir &lt;_pytest.pytester.Testdir&gt; is now a thin wrapper around Pytester &lt;_pytest.pytester.Pytester&gt;, preserving the old interface.</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst"">pytest's changelog</a>.</em></p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pytest-dev/pytest/commit/e7073afe6e2373175130511135020af8b4e3a670""><code>e7073af</code></a> Prepare release version 6.2.0</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/683f29f84df61e9dab92970a57190d20ca58385e""><code>683f29f</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/8129"">#8129</a> from bluetech/docs-pygments-workaround</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/0feeddf8edb87052402fafe690d019e3eb75dfa4""><code>0feeddf</code></a> doc: temporary workaround for pytest-pygments lexing error</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/b4782757772373a2ea00802606407e9399b98031""><code>b478275</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/8128"">#8128</a> from bluetech/skip-reason-empty</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/3302ff994959964c3ec2ab35ec1fe98c038684c0""><code>3302ff9</code></a> terminal: when the skip/xfail is empty, don't show it as &quot;()&quot;</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/59bd0f6912fd9ec2b4c3e3bbc989636bb5f54dbf""><code>59bd0f6</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/8126"">#8126</a> from bluetech/tox-regen-pretend-scm2</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/6298ff1f4e21307627b59aa86437145ddcf2bafe""><code>6298ff1</code></a> tox: use pip legacy resolver for regen job</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/d51ecbd44d7e2f4f8c21f246fe9ed034bc8bae5b""><code>d51ecbd</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/8125"">#8125</a> from bluetech/tox-rm-pip-req</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/f237b077fc83fe834589e619be1e898f28026eb2""><code>f237b07</code></a> tox: remove requires: pip&gt;=20.3.1</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/95e0e19b8d5b55b0462d420af7454f3bdcbe7fbc""><code>95e0e19</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/8124"">#8124</a> from bluetech/s0undt3ch-feature/skip-context-hook</li>
<li>Additional commits viewable in <a href=""https://github.com/pytest-dev/pytest/compare/5.2.2...6.2.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a ""Dependabot enabled"" badge to your readme

Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)



</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1145/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
767561886,MDU6SXNzdWU3Njc1NjE4ODY=,1148,Syntax error with + symbol when deployed to Vercel,765871,kaihendry,closed,0,,,,,2,2020-12-15T12:53:12Z,2020-12-16T21:57:42Z,2020-12-16T21:51:54Z,NONE,,"Works locally:
```
(ins)[hendry@t14s aws-partners]$ sqlite3 partners.db < test.sql
5|{""href"":""https://partners.amazonaws.com/partners/001E000000NaBI0IAN"",""label"":""Slalom""}
6|{""href"":""https://partners.amazonaws.com/partners/001E000000VHBQIIA5"",""label"":""Accenture""}
7|{""href"":""https://partners.amazonaws.com/partners/001E000000Rp588IAB"",""label"":""Druva""}
8|{""href"":""https://partners.amazonaws.com/partners/001E0000013FeQXIA0"",""label"":""Palo Alto Networks""}
9|{""href"":""https://partners.amazonaws.com/partners/001E000000Rl12lIAB"",""label"":""New Relic""}
10|{""href"":""https://partners.amazonaws.com/partners/001E000000NaBHWIA3"",""label"":""Deloitte""}
11|{""href"":""https://partners.amazonaws.com/partners/001E000000Rp5GDIAZ"",""label"":""MegazoneCloud""}
12|{""href"":""https://partners.amazonaws.com/partners/001E000000NaBHMIA3"",""label"":""iret, Inc.""}
(ins)[hendry@t14s aws-partners]$ cat test.sql
select A.launch_rank, A.partner_info from summary A INNER JOIN summary B
    ON A.launch_rank>=B.launch_rank-3 AND A.launch_rank<=B.launch_rank+4
   WHERE B.""partner_info"" LIKE '%Palo Alto%'
```

Copy the SQL into https://aws-partners-singapore.vercel.app/partners and get syntax error:

![image](https://user-images.githubusercontent.com/765871/102217393-78e4f280-3f17-11eb-9e13-ca79ed62ec34.png)
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1148/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
769520939,MDU6SXNzdWU3Njk1MjA5Mzk=,1149,Make it easier to theme Datasette with CSS,9599,simonw,open,0,,,3268330,Datasette 1.0,3,2020-12-17T05:01:26Z,2021-03-22T21:43:16Z,,OWNER,,"I want to theme https://datasette.io/ so that when you visit https://datasette.io/content (the Datasette UI part of it) the navigation from the parent site is used.

I tried dropping in a `base.html` template like this:

```html
{% extends ""page_base.html"" %}

{% block base_extra_head %}

<meta name=""viewport"" content=""width=device-width, initial-scale=1, shrink-to-fit=no"">
{% for url in extra_css_urls %}
    <link rel=""stylesheet"" href=""{{ url.url }}""{% if url.sri %} integrity=""{{ url.sri }}"" crossorigin=""anonymous""{% endif %}>
{% endfor %}
{% for url in extra_js_urls %}
    <script src=""{{ url.url }}""{% if url.sri %} integrity=""{{ url.sri }}"" crossorigin=""anonymous""{% endif %}></script>
{% endfor %}
{% block extra_head %}{% endblock %}
{% endblock %}

{% block extra_body_end %}
{% include ""_close_open_menus.html"" %}

{% for body_script in body_scripts %}
    <script>{{ body_script }}</script>
{% endfor %}
{% endblock %}
```
But this resulted in pages looking like this:

<img width=""1067"" alt=""content__categories__3_rows"" src=""https://user-images.githubusercontent.com/9599/102446045-c168e280-3fe1-11eb-94d6-e7350798eb96.png"">

Note that the cog menu is broken and the filter UI is unstyled. To get these working correctly I would need to copy over a whole lot of Datasette's default CSS - and that means that when Datasette changes in the future those pages could break in subtle ways.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1149/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
274877366,MDExOlB1bGxSZXF1ZXN0MTUzMzA2ODgy,115,Add keyboard shortcut to execute SQL query,198537,rgieseke,closed,0,,,,,1,2017-11-17T14:13:33Z,2017-11-17T15:16:34Z,2017-11-17T14:22:56Z,CONTRIBUTOR,simonw/datasette/pulls/115,"Very cool tool, thanks a lot!

This PR adds a `Shift-Enter` short cut to execute the SQL query. I used CodeMirrors keyboard handling.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/115/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
770436876,MDU6SXNzdWU3NzA0MzY4NzY=,1150,Maintain an in-memory SQLite table of connected databases and their tables,9599,simonw,closed,0,,,3268330,Datasette 1.0,32,2020-12-17T23:02:13Z,2020-12-27T14:51:39Z,2020-12-18T22:34:12Z,OWNER,,"I want Datasette to have its own internal metadata about connected tables, to power features like a paginated searchable homepage in #461. I want this to be a SQLite table.

This could also be part of the directory scanning mechanism prototyped in #672 - where Datasette can be set to continually scan a directory for new database files that it can serve.

Also relevant to the Datasette Library concept in #417.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1150/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
770448622,MDU6SXNzdWU3NzA0NDg2MjI=,1151,Database class mechanism for cross-connection in-memory databases,9599,simonw,closed,0,,,6346396,Datasette 0.54,11,2020-12-17T23:25:43Z,2021-01-26T19:07:44Z,2020-12-18T01:01:26Z,OWNER,,"> Next challenge: figure out how to use the `Database` class from https://github.com/simonw/datasette/blob/0.53/datasette/database.py for an in-memory database which persists data for the duration of the lifetime of the server, and allows access to that in-memory database from multiple threads in a way that lets them see each other's changes.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1150#issuecomment-747768112_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1151/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
770598024,MDU6SXNzdWU3NzA1OTgwMjQ=,1152,Efficiently calculate list of databases/tables a user can view,9599,simonw,open,0,,,,,12,2020-12-18T06:13:01Z,2021-12-27T23:04:31Z,,OWNER,,"> The homepage currently performs a massive flurry of permission checks - one for each, database, table and view: https://github.com/simonw/datasette/blob/0.53/datasette/views/index.py#L21-L75
> 
> A paginated version of this is a little daunting as the permission checks would have to be carried out in every single table just to calculate the count that will be paginated.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1150#issuecomment-747864831_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1152/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
771202454,MDU6SXNzdWU3NzEyMDI0NTQ=,1153,"Use YAML examples in documentation by default, not JSON",9599,simonw,closed,0,,,,,22,2020-12-18T22:20:15Z,2023-07-08T20:09:48Z,2023-07-08T20:08:13Z,OWNER,,"YAML configuration is much better for multi-line strings, and I'm increasingly adding configuration options to Datasette that benefit from that - fragments of HTML in `description_html` or SQL queries used to configure things like https://github.com/simonw/datasette-atom for example.

Rather than confusing things by showing both in the documentation, I should switch all of the default examples to use YAML instead.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1153/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
771208009,MDU6SXNzdWU3NzEyMDgwMDk=,1154,Documentation for new _internal database and tables,9599,simonw,closed,0,,,6346396,Datasette 0.54,2,2020-12-18T22:34:52Z,2021-01-25T00:09:22Z,2021-01-25T00:08:41Z,OWNER,,"> Needs documentation, but I can wait to write that until I've tested out the feature a bit more.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1150#issuecomment-748352106_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1154/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
771216293,MDU6SXNzdWU3NzEyMTYyOTM=,1155,Better internal database_name for _internal database,9599,simonw,closed,0,,,,,9,2020-12-18T22:47:27Z,2020-12-22T20:04:35Z,2020-12-22T20:04:35Z,OWNER,,"https://latest.datasette.io/login-as-root then https://latest.datasette.io/_internal

<img width=""1627"" alt=""_schemas__columns__132_rows"" src=""https://user-images.githubusercontent.com/9599/102668086-e70bfe00-413f-11eb-927d-e37ec9238e12.png"">

That `database_name` is ugly. It should just be `_internal`.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1155/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
772408750,MDU6SXNzdWU3NzI0MDg3NTA=,1156,Rename _schemas to _internal,9599,simonw,closed,0,,,6346396,Datasette 0.54,1,2020-12-21T19:27:58Z,2021-01-24T21:20:39Z,2020-12-21T19:51:18Z,OWNER,,"I like `_internal` as the name for the in-memory Datasette schema database better.

Refs #1154 #1150 #1155",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1156/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
772438273,MDU6SXNzdWU3NzI0MzgyNzM=,1157,Use time.perf_counter() instead of time.time() to measure performance,9599,simonw,closed,0,,,6346396,Datasette 0.54,1,2020-12-21T20:21:41Z,2021-01-24T21:20:42Z,2020-12-21T21:49:20Z,OWNER,,"I do that in a bunch of places: https://ripgrep.datasette.io/-/ripgrep?pattern=time%28%29&literal=on&glob=datasette%2F%2A%2A

https://docs.python.org/3/library/time.html#time.perf_counter

> `time.``perf_counter`() → float
> 
> Return the value (in fractional seconds) of a performance counter, i.e. a clock with the highest available resolution to measure a short duration. It does include time elapsed during sleep and is system-wide. The reference point of the returned value is undefined, so that only the difference between the results of consecutive calls is valid.
> 
> _New in version 3.3._",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1157/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
773913793,MDExOlB1bGxSZXF1ZXN0NTQ0OTIzNDM3,1158,Modernize code to Python 3.6+,6774676,eumiro,closed,0,,,6346396,Datasette 0.54,4,2020-12-23T16:21:38Z,2021-01-24T21:20:50Z,2020-12-23T17:04:32Z,CONTRIBUTOR,simonw/datasette/pulls/1158,"- compact dict and set building
- remove redundant parentheses
- simplify chained conditions
- change method name to lowercase
- use triple double quotes for docstrings

please feel free to accept/reject any of these independent commits",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1158/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
774332247,MDExOlB1bGxSZXF1ZXN0NTQ1MjY0NDM2,1159,Improve the display of facets information,552629,lovasoa,open,0,,,3268330,Datasette 1.0,9,2020-12-24T11:01:47Z,2023-07-31T18:57:59Z,,FIRST_TIME_CONTRIBUTOR,simonw/datasette/pulls/1159,"This PR changes the display of facets to hopefully make them more readable.

Before | After
---|---
![image](https://user-images.githubusercontent.com/552629/103084609-b1ec2980-45df-11eb-85bc-68ab8df3e8d9.png) | ![image](https://user-images.githubusercontent.com/552629/103085220-620e6200-45e1-11eb-8189-5dd5d3e2569e.png)
",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1159/reactions"", ""total_count"": 4, ""+1"": 4, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
274884209,MDU6SXNzdWUyNzQ4ODQyMDk=,116,Add documentation section about SQLite extensions,9599,simonw,closed,0,,,,,1,2017-11-17T14:36:30Z,2018-05-28T17:23:42Z,2018-05-28T17:23:41Z,OWNER,,,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/116/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
775666296,MDU6SXNzdWU3NzU2NjYyOTY=,1160,"""datasette insert"" command and plugin hook",9599,simonw,open,0,,,,,23,2020-12-29T02:37:03Z,2021-06-17T18:12:32Z,,OWNER,,"Tools for loading data into Datasette currently mostly exist as separate utilities - `yaml-to-sqlite` and `csvs-to-sqlite` and suchlike.

Bringing these into Datasette could have some interesting properties:

- A `datasette insert` command could be extended with plugins to handle more formats
- Any format that can be inserted on the command-line could also be inserted using a web UI or web API - which would benefit from new format plugin hooks
- If Datasette ever grows beyond SQLite (see #670) a built-in import mechanism could work for those other databases as well - without me needing to write `yaml-to-postgresql` and suchlike",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1160/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
776101101,MDU6SXNzdWU3NzYxMDExMDE=,1161,Update a whole bunch of links to datasette.io instead of datasette.readthedocs.io,9599,simonw,open,0,,,,,1,2020-12-29T21:47:31Z,2020-12-29T21:49:57Z,,OWNER,,https://ripgrep.datasette.io/-/ripgrep?pattern=%28datasette%7Csqlite-utils%29%5C.readthedocs%5C.io,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1161/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
776128269,MDU6SXNzdWU3NzYxMjgyNjk=,1162,"First working version of ""datasette insert data.db file.csv""",9599,simonw,open,0,,,,,0,2020-12-29T23:20:11Z,2021-06-17T18:12:32Z,,OWNER,,Refs #1160,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1162/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
776128565,MDU6SXNzdWU3NzYxMjg1NjU=,1163,"""datasette insert data.db url-to-csv""",9599,simonw,open,0,,,,,1,2020-12-29T23:21:21Z,2021-06-17T18:12:32Z,,OWNER,,"Refs #1160 - get filesystem imports working first for #1162, then add import-from-URL.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1163/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
776634318,MDU6SXNzdWU3NzY2MzQzMTg=,1164,Mechanism for minifying JavaScript that ships with Datasette,9599,simonw,open,0,,,,,9,2020-12-30T20:59:06Z,2022-01-13T22:21:29Z,,OWNER,,"> If I'm going to minify it I'll need to figure out a build step in Datasette itself so that I can easily work on that minified version.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/983#issuecomment-752748496_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1164/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
776635426,MDU6SXNzdWU3NzY2MzU0MjY=,1165,Mechanism for executing JavaScript unit tests,9599,simonw,open,0,,,,,9,2020-12-30T21:02:34Z,2022-01-13T22:21:29Z,,OWNER,,"> I'm going to need to add JavaScript unit tests for this new plugin system.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/983#issuecomment-752757289_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1165/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
777140799,MDU6SXNzdWU3NzcxNDA3OTk=,1166,Adopt Prettier for JavaScript code formatting,9599,simonw,open,0,,,,,10,2020-12-31T21:25:27Z,2022-01-13T22:22:18Z,,OWNER,,https://prettier.io/ - I'm going to go with 2 spaces.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1166/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
777145954,MDU6SXNzdWU3NzcxNDU5NTQ=,1167,Add Prettier to contributing documentation,9599,simonw,closed,0,,,6346396,Datasette 0.54,3,2020-12-31T22:00:55Z,2021-01-25T02:01:19Z,2021-01-25T01:58:28Z,OWNER,,"Following #1166 - the docs at https://docs.datasette.io/en/stable/contributing.html should include a section about JavaScript, and it should document how to run Prettier.

I run it in VS Code but it can be run on the command-line too:

    npx prettier 'datasette/static/*[!.min].js' --write
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1167/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
777333388,MDU6SXNzdWU3NzczMzMzODg=,1168,Mechanism for storing metadata in _metadata tables,9599,simonw,open,0,,,,,21,2021-01-01T18:47:27Z,2023-09-28T18:29:05Z,,OWNER,,"_Original title: Perhaps metadata should all live in a `_metadata` in-memory database_

Inspired by #1150 - metadata should be exposed as an API, and for large Datasette instances that API may need to be paginated. So why not expose it through an in-memory database table?

One catch to this: plugins. #860 aims to add a plugin hook for metadata. But if the metadata comes from an in-memory table, how do the plugins interact with it?

The need to paginate over metadata does make a plugin hook that returns metadata for an individual table seem less wise, since we don't want to have to do 10,000 plugin hook invocations to show a list of all metadata.

If those plugins write directly to the in-memory table how can their contributions survive the server restarting?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1168/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
777677671,MDU6SXNzdWU3Nzc2Nzc2NzE=,1169,Prettier package not actually being cached,3637,benpickles,closed,0,,,,,4,2021-01-03T17:04:41Z,2021-01-04T19:52:34Z,2021-01-04T19:52:33Z,CONTRIBUTOR,,"With the current configuration Prettier seems to be installed on every run - which can been [seen from the output](https://github.com/simonw/datasette/runs/1631686028?check_suite_focus=true#step:4:4):

```
npx: installed 1 in 5.166s
```

Prettier isn't explicitly being installed (it's surprising that actually installing the dependencies isn't included in the [actions/cache docs](https://github.com/actions/cache/blob/main/examples.md#macos-and-ubuntu)) but it turns out that `npx` will automatically install the package for the specified command (it actually _guesses_ the package name from the name of the command). I'm not sure where Prettier ends up being installed but it doesn't appear to be in `~/.npm` according to the [post-cache output](https://github.com/simonw/datasette/runs/1631686028#step:7:2) (or `./node_modules` when I tested locally):

```
Cache hit occurred on the primary key Linux-npm-565329898f77080e58b14d45cf816ab94877e6f2ece9d395c369c533548a7ee7, not saving cache.
```

I think there are a couple of approaches to tackling this, you could manually install/cache Prettier within the action, or add a `package.json` with Prettier. I would go with the latter because it's a more standard and maintainable approach and it will also ensure that, along with CI, anyone working on the project will run the same version of Prettier (you'll also get Dependabot JavaScript updates).

I've tested the [`package.json` approach on a branch](https://github.com/simonw/datasette/compare/main...benpickles:cache-prettier) and am happy to turn it into a pull request if you fancy.
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1169/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
274900388,MDExOlB1bGxSZXF1ZXN0MTUzMzI0MzAx,117,Don't prevent tabbing to `Run SQL` button,198537,rgieseke,closed,0,,,,,1,2017-11-17T15:27:50Z,2017-11-19T20:30:24Z,2017-11-18T00:53:43Z,CONTRIBUTOR,simonw/datasette/pulls/117,"Mentioned in #115 

Here you go!",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/117/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
778126516,MDExOlB1bGxSZXF1ZXN0NTQ4MjcxNDcy,1170,Install Prettier via package.json,3637,benpickles,closed,0,,,6346396,Datasette 0.54,3,2021-01-04T14:18:03Z,2021-01-24T21:21:01Z,2021-01-04T19:52:34Z,CONTRIBUTOR,simonw/datasette/pulls/1170,This adds a package.json with Prettier and means that developers/CI will use the same version. It also ensures that NPM packages are cached on GitHub Actions which fixes #1169.,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1170/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
778450486,MDU6SXNzdWU3Nzg0NTA0ODY=,1171,GitHub Actions workflow to build and sign macOS binary executables,9599,simonw,open,0,,,,,8,2021-01-04T23:36:59Z,2021-01-07T19:36:00Z,,OWNER,,"Using PyInstaller, as explored in #93 and https://til.simonwillison.net/python/packaging-pyinstaller

The bigger challenge will be the code signing bit. I'll need a Apple Developer account ($99/year) and some extensive CI fiddling.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1171/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
778530523,MDU6SXNzdWU3Nzg1MzA1MjM=,1172,/-/static should be excluded from auth and permission checks,9599,simonw,open,0,,,,,0,2021-01-05T02:53:41Z,2021-01-05T02:53:41Z,,OWNER,,"I want to set far future / immutable cache headers on everything served from `/-/static` and `/-/static-plugins`

This has security implications since it will be possible to see what plugins are installed by checking for known static URLs. I'm fine with that - performance is more important here.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1172/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
778682317,MDU6SXNzdWU3Nzg2ODIzMTc=,1173,GitHub Actions workflow to build manylinux binary,9599,simonw,open,0,,,,,1,2021-01-05T07:41:11Z,2021-01-05T07:41:43Z,,OWNER,,Refs #1171 and #93,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1173/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
779156520,MDU6SXNzdWU3NzkxNTY1MjA=,1175,Use structlog for logging,9599,simonw,open,0,,,,,4,2021-01-05T15:11:36Z,2022-07-26T12:52:10Z,,OWNER,,To solve #241 JSON logging.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1175/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
779691739,MDU6SXNzdWU3Nzk2OTE3Mzk=,1176,"Policy on documenting ""public"" datasette.utils functions",9599,simonw,closed,0,,,3268330,Datasette 1.0,13,2021-01-05T22:55:25Z,2022-02-07T06:43:32Z,2022-02-07T06:42:58Z,OWNER,,"https://github.com/simonw/datasette-css-properties starts [like this](https://github.com/simonw/datasette-css-properties/blob/0.1/datasette_css_properties/__init__.py#L1-L3):

```python
from datasette import hookimpl
from datasette.utils.asgi import Response
from datasette.utils import escape_css_string, to_css_class
```
`escape_css_string` and `to_css_class` are not documented, which means relying on them is risky since there's no promise that they won't change.

Would be good to figure out a policy on this, and maybe promote some of them to ""documented"" status.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1176/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
780153562,MDU6SXNzdWU3ODAxNTM1NjI=,1177,Ability to stream all rows as newline-delimited JSON,9599,simonw,open,0,,,3268330,Datasette 1.0,1,2021-01-06T07:10:48Z,2022-03-21T15:08:52Z,,OWNER,,"> Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:
> 
>     pandas.read_json(""https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on"", lines=True)
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1101#issuecomment-755128038_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1177/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
780267857,MDU6SXNzdWU3ODAyNjc4NTc=,1178,Use force_https_urls on when deploying with Cloud Run,9599,simonw,closed,0,,,6346396,Datasette 0.54,9,2021-01-06T08:20:55Z,2021-01-24T21:21:05Z,2021-01-06T18:24:47Z,OWNER,,"_Original title: datasette.absolute_url() should return https:// not http:// on Cloud Run_

https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on currently provides `http://` links, which break in Observable since it won't load `http://` content.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1178/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
780278550,MDU6SXNzdWU3ODAyNzg1NTA=,1179,Make original path available to render hooks,9599,simonw,open,0,,,,,8,2021-01-06T08:31:45Z,2021-01-25T04:44:33Z,,OWNER,,"https://github.com/simonw/datasette-export-notebook/blob/0.1/datasette_export_notebook/__init__.py

```python
async def render_notebook(datasette, request):
    return Response.html(
        await datasette.render_template(
            ""export_notebook.html"",
            {
                ""csv_stream_url"": datasette.absolute_url(
                    request,
                    path_with_format(
                        request=request, format=""csv"", extra_qs={""_stream"": ""on""}
                    ),
                ),
                ""json_url"": datasette.absolute_url(
                    request,
                    path_with_format(
                        request=request, format=""json"", extra_qs={""_shape"": ""array""}
                    ),
                ),
                ""json"": json,
            },
        )
    )
```
This results in https://latest-with-plugins.datasette.io/github/issue_comments.Notebook showing `http://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_format=json&_shape=array`",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1179/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
275048699,MDExOlB1bGxSZXF1ZXN0MTUzNDMyMDQ1,118,Foreign key information on row and table pages,9599,simonw,closed,0,,,,,0,2017-11-18T03:13:27Z,2017-11-18T03:15:57Z,2017-11-18T03:15:50Z,OWNER,simonw/datasette/pulls/118,,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/118/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
780767542,MDU6SXNzdWU3ODA3Njc1NDI=,1180,Lazily evaluated arguments for call_with_supported_arguments,9599,simonw,open,0,,,,,2,2021-01-06T18:43:34Z,2021-01-07T18:56:24Z,,OWNER,,"While building https://github.com/simonw/datasette-export-notebook I thought it would be nice to be able to show a count of exported records on the page ""This will stream 10,422 records to your notebook"".

None of the documented arguments on https://docs.datasette.io/en/0.53/plugin_hooks.html#register-output-renderer-datasette expose the count. The closest is `sql` which could be executed as `select count(*) from ({sql})` but that's a bit inelegant.

So, idea: if your defined render function takes a `total` argument, the caller runs a `count(*)` query and passes you that total.

To implement this I would need to teach the `call_with_supported_arguments` that some arguments are lazy - they should execute a function (or an async function) but only if they are needed.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1180/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
781262510,MDU6SXNzdWU3ODEyNjI1MTA=,1181,"Certain database names results in 404: ""Database not found: None""",1470389,jieter,closed,0,,,6346396,Datasette 0.54,4,2021-01-07T12:01:16Z,2021-12-21T18:25:15Z,2021-01-25T05:13:19Z,NONE,,"I have a file named `test-database (1).sqlite`. When requesting the home route `/`, I see datasette is able to read it correctly:

<img width=""409"" alt=""Screenshot 2021-01-07 at 12 54 46"" src=""https://user-images.githubusercontent.com/1470389/103889964-9442b800-50e7-11eb-9548-fbf10ed3b1ea.png"">

However, if I click any of the links, datasette replies with: `Error 404 Database not found: None`

It seems the hash is crucial, as renaming the file to `database (1).sqlite` makes the error go away.

This lines checks for a single dash:
https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/views/base.py#L184

```
$ datasette test-database\ \(1\).sqlite 
INFO:     Started server process [68314]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
INFO:     127.0.0.1:54043 - ""GET /favicon.ico HTTP/1.1"" 200 OK
INFO:     127.0.0.1:54043 - ""GET / HTTP/1.1"" 200 OK
...
INFO:     127.0.0.1:54044 - ""GET /favicon.ico HTTP/1.1"" 200 OK
INFO:     127.0.0.1:54044 - ""GET /test-database (1) HTTP/1.1"" 404 Not Found

```
Version:
```
$ datasette --version
datasette, version 0.53
```
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1181/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
782692159,MDU6SXNzdWU3ODI2OTIxNTk=,1182,"Retire ""Ecosystem"" page in favour of datasette.io/plugins and /tools",9599,simonw,closed,0,,,6346396,Datasette 0.54,3,2021-01-09T21:54:47Z,2021-01-24T21:21:09Z,2021-01-09T22:17:28Z,OWNER,,https://docs.datasette.io/en/stable/ecosystem.html is no longer needed.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1182/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
782708469,MDU6SXNzdWU3ODI3MDg0Njk=,1183,"Take advantage of sqlite-utils cached table counts, if available",9599,simonw,open,0,,,,,2,2021-01-09T23:51:48Z,2021-01-12T02:42:08Z,,OWNER,,"sqlite-utils 3.2 now has a mechanism for creating a `_counts` table with triggers to maintain counts for individual tables. Datasette could look for this table and use it for counts, dramatically speeding up places that currently suffer from slow counts. Refs #859.

https://sqlite-utils.datasette.io/en/stable/python-api.html#cached-table-counts-using-triggers",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1183/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
783714076,MDU6SXNzdWU3ODM3MTQwNzY=,1184,request.full_path property,9599,simonw,closed,0,,,6346396,Datasette 0.54,0,2021-01-11T21:21:58Z,2021-01-24T21:21:16Z,2021-01-11T21:34:47Z,OWNER,,"> I'll also add `request.full_path` for consistency with these: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/asgi.py#L77-L90

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1179#issuecomment-755495387_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1184/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
784628163,MDU6SXNzdWU3ODQ2MjgxNjM=,1185,"""Statement may not contain PRAGMA"" error is not strictly true",9599,simonw,closed,0,,,6346396,Datasette 0.54,3,2021-01-12T22:07:10Z,2021-01-24T21:21:37Z,2021-01-12T22:26:26Z,OWNER,,"Consider https://latest.datasette.io/fixtures?sql=select+%27select%0D%0A%27+%7C%7C+group_concat%28%27++++case+when+%5B%27+%7C%7C+name+%7C%7C+%27%5D+is+not+null+then+%27+%7C%7C+quote%28name+%7C%7C+%27%2C+%27%29+%7C%7C+%27+else+%27%27%27%27+end%27%2C+%27+%7C%7C%0D%0A%27%29+%7C%7C+%27%0D%0A++as+columns%2C%0D%0A++count%28*%29+as+num_rows%0D%0Afrom%0D%0A++%5B%27+%7C%7C+%3Atable+%7C%7C+%27%5D%0D%0Agroup+by%0D%0A++columns%0D%0Aorder+by%0D%0A++num_rows+desc%27+as+query+from+pragma_ytable_info%28%3Atable%29&table=facetable

It says ""Statement may not contain PRAGMA"" - but that's not actually true. Datasette has an allow-list of PRAGMA that are OK - in this case there was a typo in `pragma_ytable_info` which caused the error, but pragma_table_info` would have been OK.

So the error message is misleading.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1185/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
785573793,MDU6SXNzdWU3ODU1NzM3OTM=,1186,"script type=""module"" support",9599,simonw,closed,0,,,6346396,Datasette 0.54,1,2021-01-14T01:17:47Z,2021-01-24T21:21:41Z,2021-01-14T01:50:58Z,OWNER,,"Custom JavaScript can be loaded in `metadata.json` like this:
```json
{
    ""extra_js_urls"": [
        {
            ""url"": ""https://code.jquery.com/jquery-3.2.1.slim.min.js"",
            ""sri"": ""sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g=""
        }
    ]
}
```
Add a `""module"": true` option which causes the resulting script element to use `<script type=""module"">`",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1186/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
785588942,MDU6SXNzdWU3ODU1ODg5NDI=,1187,"extra_body_script() support for script type=""module""",9599,simonw,closed,0,,,6346396,Datasette 0.54,1,2021-01-14T02:01:47Z,2021-01-24T21:21:44Z,2021-01-14T02:14:39Z,OWNER,,"Follows #1186. The `extra_body_script()` plugin hook should provide a mechanism for specifying that the script should use `<script type=""module"">`.

Relevant docs: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1187/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275082158,MDU6SXNzdWUyNzUwODIxNTg=,119,"Build an ""export this data to google sheets"" plugin",9599,simonw,closed,0,,,,,1,2017-11-18T14:14:51Z,2020-06-04T18:46:40Z,2020-06-04T18:46:39Z,OWNER,,"Inspired by https://github.com/kren1/tosheets

It should be a plug-in because I'd like to keep all interactions with proprietary / non-open-source software encapsulated in plugins rather than shipped as part of core.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/119/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
787098146,MDU6SXNzdWU3ODcwOTgxNDY=,1190,`datasette publish upload` mechanism for uploading databases to an existing Datasette instance,1024355,tomershvueli,closed,0,,,,,3,2021-01-15T18:18:42Z,2023-08-30T22:16:39Z,2023-08-30T22:16:38Z,NONE,,"If I have a self-hosted instance of Datasette up and running, I'd like to be able to the use the CLI to publish databases to that instance, not only Google or Heroku. Ideally there'd be a `url` parameter or something similar to which one could point the publish command to their instance. ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1190/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
787098345,MDU6SXNzdWU3ODcwOTgzNDU=,1191,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,9599,simonw,open,0,,,3268330,Datasette 1.0,12,2021-01-15T18:18:51Z,2023-09-18T06:55:52Z,,OWNER,,"Sometimes a plugin may want to add content to an existing default template - for example `datasette-search-all` adds a new search box at the top of `index.html`. I also want `datasette-upload-csvs` to add a CTA on the `database.html` page: https://github.com/simonw/datasette-upload-csvs/issues/18

Currently plugins can do this by providing a new version of the `index.html` template - but if multiple plugins try to do that only one of them will succeed.

It would be better if there were known areas of those templates which plugins could add additional content to, such that multiple plugins can use the same spot.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1191/reactions"", ""total_count"": 4, ""+1"": 4, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
787104850,MDU6SXNzdWU3ODcxMDQ4NTA=,1192,Form Plugin for in-depth Datasette Querying,1024355,tomershvueli,open,0,,,,,0,2021-01-15T18:24:50Z,2021-01-15T18:24:50Z,,NONE,,I envision a sort of easy-to-build form plugin that would be able to map a user's inputs to different fields/columns in a Datasette database. ,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1192/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
787173276,MDU6SXNzdWU3ODcxNzMyNzY=,1193,Research plugin hook for alternative database backends,9599,simonw,open,0,,,,,1,2021-01-15T20:27:50Z,2021-03-12T01:01:54Z,,OWNER,,"I started exploring what Datasette would like running against PostgreSQL in #670 and @dazzag24 did some work on Parquet described in #657.

I had initially thought this was WAY too much additional complexity, but I'm beginning to think that the `Database` class may be small enough that having it abstract away the details of running queries against alternative database backends could be feasible.

A bigger issue is SQL generation, but I realized that most of Datasette's SQL generation code exists just in the `TableView` class that runs the table page. If this was abstracted into some kind of SQL builder that could be then customized per-database it might be reasonable to get it working.

Very unlikely for this to make it into Datasette 1.0, but maybe this would be the defining feature of Datasette 2.0?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1193/reactions"", ""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
788447787,MDU6SXNzdWU3ODg0NDc3ODc=,1194,?_size= argument is not persisted by hidden form fields in the table filters,9599,simonw,closed,0,,,6346396,Datasette 0.54,3,2021-01-18T17:41:52Z,2021-01-25T03:10:23Z,2021-01-25T03:10:23Z,OWNER,,"Click ""Apply"" on https://covid-19.datasettes.com/covid/ny_times_us_counties?_size=1000&county__exact=San+Francisco&state__exact=California&_sort_desc=date#g.mark=line&g.x_column=date&g.x_type=temporal&g.y_column=cases&g.y_type=quantitative and the `?_size=1000` parameter from the URL will no longer apply on the reloaded page.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1194/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
789336592,MDU6SXNzdWU3ODkzMzY1OTI=,1195,"view_name = ""query"" for the query page",9599,simonw,open,0,,,,,4,2021-01-19T20:21:36Z,2021-01-25T04:40:08Z,,OWNER,,It uses `view_name` of `database` at the moment which isn't as useful.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1195/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
791237799,MDU6SXNzdWU3OTEyMzc3OTk=,1196,Access Denied Error in Windows,2826376,QAInsights,open,0,,,,,2,2021-01-21T15:40:40Z,2021-04-14T19:28:38Z,,NONE,,"I am trying to publish a db to vercel. But while issuing the below command throwing `Access Denied` error which is leading to `RecursionError: maximum recursion depth exceeded while calling a Python object`.

I am using PyCharm and Python 3.9. I have reinstalled both and launched PyCharm as Admin in Windows 10. But still the issue persists.

Issued command `datasette publish vercel jmeter.db --project jmeter --install datasette-vega`

PS: localhost is working fine.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1196/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
791381623,MDU6SXNzdWU3OTEzODE2MjM=,1197,DB size limit for publishing with Heroku,1186275,mtdukes,closed,0,,,,,1,2021-01-21T18:08:43Z,2021-01-24T20:53:44Z,2021-01-24T20:53:44Z,NONE,,"Hello,
I tried searching for this, but can't seem to get a great answer: Does anybody know the size limit for databases deploying to Heroku? The files I'm working with are pretty large, but I might be able to pare them down if I have a limit in mind.

 I'm getting the following error when running `datasette heroku publish`:

`RangeError [ERR_INVALID_OPT_VALUE]: The value ""14504095744"" is invalid for option ""size""`",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1197/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
792625812,MDU6SXNzdWU3OTI2MjU4MTI=,1198,Plugin testing documentation on using pytest-httpx,9599,simonw,closed,0,,,,,1,2021-01-23T18:46:16Z,2021-01-24T20:40:38Z,2021-01-24T20:38:43Z,OWNER,,"I keep on having to figure this out: if you use the https://pypi.org/project/pytest-httpx/ fixture to write tests against mocked external APIs, they will fail because that module will break Datasette's own `datasette.client` testing mechanism.

You can fix this using:

```python
@pytest.fixture
def non_mocked_hosts():
    return [""localhost""]
```

See https://github.com/simonw/datasette-indieauth/blob/1.2/tests/test_indieauth.py

I can add this tip to the https://docs.datasette.io/en/stable/testing_plugins.html page.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1198/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
792652391,MDU6SXNzdWU3OTI2NTIzOTE=,1199,Experiment with PRAGMA mmap_size=N,9599,simonw,open,0,,,,,2,2021-01-23T21:24:09Z,2021-07-17T17:39:17Z,,OWNER,,"https://sqlite.org/mmap.html - SQLite supports memory-mapped I/O but it's disabled by default. The `PRAGMA mmap_size=N` option can be used to enable it.

It would be very interesting to understand the impact this could have on Datasette performance for various different shapes of data.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1199/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
267523511,MDU6SXNzdWUyNjc1MjM1MTE=,12,Make it so you can override templates,9599,simonw,closed,0,,,2949431,Custom templates edition,1,2017-10-23T02:25:35Z,2017-11-30T16:42:46Z,2017-11-30T16:38:34Z,OWNER,,"The app will ship with default templates but, just like with the Django admin, you will be able to override them using either explicit configuration settings or just by dropping in templates with certain file names.

Template inheritance should work here, both allowing you to override just the base template and allowing you to customize tiny bits of others.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/12/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275087397,MDU6SXNzdWUyNzUwODczOTc=,120,Plugin that adds an authentication layer of some sort,9599,simonw,closed,0,,,,,4,2017-11-18T15:39:13Z,2020-03-16T18:48:06Z,2020-03-16T18:48:06Z,OWNER,,"Would allow people who want to host private data to do so.
.sh ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/120/reactions"", ""total_count"": 7, ""+1"": 5, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 2, ""rocket"": 0, ""eyes"": 0}",,completed
792890765,MDU6SXNzdWU3OTI4OTA3NjU=,1200,?_size=10 option for the arbitrary query page would be useful,9599,simonw,open,0,,,,,2,2021-01-24T20:55:35Z,2021-02-11T03:13:59Z,,OWNER,,"https://latest.datasette.io/fixtures?sql=select+*+from+compound_three_primary_keys&_size=10 - `_size=10` does not do anything at the moment. It would be useful if it did.

Would also be good if it persisted in a hidden form field.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1200/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
792904595,MDU6SXNzdWU3OTI5MDQ1OTU=,1201,Release notes for Datasette 0.54,9599,simonw,closed,0,,,6346396,Datasette 0.54,5,2021-01-24T21:22:28Z,2021-01-25T17:42:21Z,2021-01-25T17:42:21Z,OWNER,,"These will incorporate the release notes from the alpha, much expanded: https://github.com/simonw/datasette/releases/tag/0.54a0",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1201/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
792931244,MDU6SXNzdWU3OTI5MzEyNDQ=,1202,Documentation convention for marking unstable APIs.,9599,simonw,closed,0,,,6346396,Datasette 0.54,2,2021-01-24T23:47:18Z,2021-01-25T00:01:02Z,2021-01-25T00:01:02Z,OWNER,,"> I'm going to document this but mark it as unstable, using a new documentation convention for marking unstable APIs.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1154#issuecomment-766462197_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1202/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
792958773,MDExOlB1bGxSZXF1ZXN0NTYwNzI1NzE0,1203,Easier way to run Prettier locally,9599,simonw,closed,0,,,,,0,2021-01-25T01:39:06Z,2021-01-25T01:41:46Z,2021-01-25T01:41:46Z,OWNER,simonw/datasette/pulls/1203,Refs #1167,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1203/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
793002853,MDExOlB1bGxSZXF1ZXN0NTYwNzYwMTQ1,1204,WIP: Plugin includes,9599,simonw,open,0,,,,,3,2021-01-25T03:59:06Z,2021-12-17T07:10:49Z,,OWNER,simonw/datasette/pulls/1204,"Refs #1191

Next steps:

- [ ] Get comfortable that this pattern is the right way to go
- [ ] Implement it for all of the other pages, not just the table page
- [ ] Add a new set of plugin tests that exercise ALL of these new hook locations
- [ ] Document, then ship",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1204/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1,
793027837,MDU6SXNzdWU3OTMwMjc4Mzc=,1205,Rename /:memory: to /_memory,9599,simonw,closed,0,,,3268330,Datasette 1.0,3,2021-01-25T05:04:56Z,2021-01-28T22:55:02Z,2021-01-28T22:51:42Z,OWNER,,"For consistency with `/_internal` - and because then we don't need to escape the `:` characters.

This change would need to be in before Datasette 1.0. I could land it earlier and set up redirects from the old URLs though.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1205/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
793086333,MDExOlB1bGxSZXF1ZXN0NTYwODMxNjM4,1206,Release 0.54,9599,simonw,closed,0,,,,,3,2021-01-25T06:45:47Z,2021-01-25T17:33:30Z,2021-01-25T17:33:29Z,OWNER,simonw/datasette/pulls/1206,Refs #1201,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1206/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
793881756,MDU6SXNzdWU3OTM4ODE3NTY=,1207,"Document the Datasette(..., pdb=True) testing pattern",9599,simonw,closed,0,,,,,1,2021-01-26T02:48:10Z,2021-01-29T02:37:19Z,2021-01-29T02:12:34Z,OWNER,,"If you're writing tests for a Datasette plugin and you get a 500 error from inside Datasette, you can cause Datasette to open a PDB session within the application server code by doing this:

```python
ds = Datasette([db_path], pdb=True)
response = await ds.client.get(""/"")
```

You'll need to run `pytest -s` to interact with the debugger, otherwise you'll get an error.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1207/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
794554881,MDU6SXNzdWU3OTQ1NTQ4ODE=,1208,A lot of open(file) functions are used without a context manager thus producing ResourceWarning: unclosed file <_io.TextIOWrapper,4488943,kbaikov,closed,0,,,,,2,2021-01-26T20:56:28Z,2021-03-11T16:15:49Z,2021-03-11T16:15:49Z,CONTRIBUTOR,,"Your code is full of open files that are never closed, especially when you deal with reading/writing json/yaml files.

If you run python with warnings enabled this problem becomes evident.
This probably contributes to some memory leaks in long running datasettes if the GC will not 'collect' those resources properly.

This is easily fixed by using a context manager instead of just using open:
```python
with open('some_file', 'w') as opened_file:
    opened_file.write('string')
```

In some newer parts of the code you use Path objects 'read_text' and 'write_text' functions which close the file properly and are prefered in some cases.


If you want I can create a PR for all places i found this pattern in.


Bellow is a fraction of places where i found a ResourceWarning:
```python

update-docs-help.py:
  20          actual = actual.replace(""Usage: cli "", ""Usage: datasette "")
  21:         open(docs_path / filename, ""w"").write(actual)
  22  

datasette\app.py:
  210          ):
  211:             inspect_data = json.load((config_dir / ""inspect-data.json"").open())
  212              if immutables is None:

  266          if config_dir and (config_dir / ""settings.json"").exists() and not config:
  267:             config = json.load((config_dir / ""settings.json"").open())
  268          self._settings = dict(DEFAULT_SETTINGS, **(config or {}))

  445              self._app_css_hash = hashlib.sha1(
  446:                 open(os.path.join(str(app_root), ""datasette/static/app.css""))
  447                  .read()

datasette\cli.py:
  130      else:
  131:         out = open(inspect_file, ""w"")
  132      loop = asyncio.get_event_loop()

  459      if inspect_file:
  460:         inspect_data = json.load(open(inspect_file))
  461  

```

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1208/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275089535,MDU6SXNzdWUyNzUwODk1MzU=,121,?_json=foo&_json=bar query string argument ,9599,simonw,closed,0,,,,,4,2017-11-18T16:09:55Z,2018-05-31T13:48:12Z,2018-05-28T18:11:51Z,OWNER,,"Causes the specified columns in the output to be treated as JSON, and returned deserialized in the .json or .jsono response.

This will be particularly powerful when combined with https://sqlite.org/json1.html",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/121/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
796234313,MDU6SXNzdWU3OTYyMzQzMTM=,1210,Immutable Database w/ Canned Queries,525780,heyarne,closed,0,,,,,2,2021-01-28T18:08:29Z,2021-02-05T11:30:34Z,2021-02-05T11:30:34Z,NONE,,"I have a database that I only want to read from; when instructing datasette to treat the database as immutable my defined canned queries disappear. Are these two features incompatible or have I hit an unintended bug? Thanks for datasette in any way, it's a joy to use!",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1210/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
797649915,MDExOlB1bGxSZXF1ZXN0NTY0NjA4MjY0,1211,Use context manager instead of plain open,4488943,kbaikov,closed,0,,,,,3,2021-01-31T07:58:10Z,2021-03-11T16:15:50Z,2021-03-11T16:15:50Z,CONTRIBUTOR,simonw/datasette/pulls/1211,"Context manager with open closes the files after usage. Fixes: https://github.com/simonw/datasette/issues/1208

When the object is already a pathlib.Path i used read_text
write_text functions

In some cases pathlib.Path.open were used in context manager,
it is basically the same as builtin open.

Tests are passing: 850 passed, 5 xfailed, 10 xpassed",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1211/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
797651831,MDU6SXNzdWU3OTc2NTE4MzE=,1212,Tests are very slow. ,4488943,kbaikov,closed,0,,,,,4,2021-01-31T08:06:16Z,2021-02-19T22:54:13Z,2021-02-19T22:54:13Z,CONTRIBUTOR,,"Working on my PR i noticed that tests are very slow.

The plain pytest run took about 37 minutes for me.
However i could shave of about 10 minutes from that if i used pytest-xdist to parallelize execution.
`pytest -n 8` is run only in 28 minutes on my machine.

I can create a PR to mention that in your documentation.
This will be a simple change to add pytest-xdist to requirements and change a command to run pytest in documentation.

Does that make sense to you?

After a bit more investigation it looks like python-xdist is not an answer. It creates a race condition for tests that try to clead temp dir before run.

Profiling shows that most time is spent on conn.executescript(TABLES) in make_app_client function. Which makes sense.

Perhaps the better approach would be look at the app_client fixture which is already session scoped, but not used by all test cases.
And/or use conn = sqlite3.connect("":memory:"") which is much faster.
And/or truncate tables after each TC instead of deleting the file and re-creating them.

I can take a look which is the best approach if you give the go-ahead. ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1212/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
799663959,MDU6SXNzdWU3OTk2NjM5NTk=,1213,gzip support for HTML (and JSON) responses,9599,simonw,open,0,,,,,3,2021-02-02T20:36:28Z,2021-02-02T20:41:55Z,,OWNER,,"This page https://datasette-tiles-demo.datasette.io/San_Francisco/tiles is 2MB because of all of the base64 images. Gzipped it's 1.5MB.

Since Datasette is usually deployed without a frontend gzipping proxy, Datasette itself needs to solve for this.

Gzipping everything won't work because some endpoints - the all-rows CSV endpoint and the download-database endpoint - are streaming and hence can't be buffered-and-gzipped.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1213/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
799693777,MDU6SXNzdWU3OTk2OTM3Nzc=,1214,Re-submitting filter form duplicates _x querystring arguments,9599,simonw,closed,0,,,,,3,2021-02-02T21:13:35Z,2021-02-02T21:28:53Z,2021-02-02T21:21:13Z,OWNER,,"Really nasty bug, caused by #1194 fix in 07e163561592c743e4117f72102fcd350a600909

Navigate to this page: https://github-to-sqlite.dogsheep.net/github/labels?_search=help&_sort=id

Click ""Apply"" to submit the form and the resulting URL is https://github-to-sqlite.dogsheep.net/github/labels?_search=help&_sort=id&_search=help&_sort=id

That's because the (truncated) HTML for the form looks like this:

```html
    ... <input id=""_search"" type=""search"" name=""_search"" value=""help"">
    ...
            <div class=""select-wrapper small-screen-only"">
                <select name=""_sort"" id=""sort_by"">
                    <option value="""">Sort...</option>
                            <option value=""id"" selected>Sort by id</option>
                            <option value=""node_id"">Sort by node_id</option>
                            ...
                </select>
            </div>
            ...
            <input type=""hidden"" name=""_search"" value=""help"">
            <input type=""hidden"" name=""_sort"" value=""id"">
        <input type=""submit"" value=""Apply"">
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1214/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
800669347,MDU6SXNzdWU4MDA2NjkzNDc=,1216,"/-/databases should reflect connection order, not alphabetical order",9599,simonw,closed,0,,,,,1,2021-02-03T20:20:23Z,2021-06-02T03:10:19Z,2021-06-02T03:10:19Z,OWNER,,"The order in which databases are attached to Datasette matters - it affects the homepage, and it's beginning to influence how certain plugins work (see https://github.com/simonw/datasette-tiles/issues/8).

Two years ago in cccea85be6aaaeadb31f3b588ec7f732628815f5 I made `/-/databases` return things in alphabetical order, to fix a test failure in Python 3.5.

Python 3.5 is no longer supported, so this is no longer necessary - and this behaviour should now be treated as a bug.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1216/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
802513359,MDU6SXNzdWU4MDI1MTMzNTk=,1217,Possible to deploy as a python app (for Rstudio connect server)?,6165713,plpxsk,open,0,,,,,4,2021-02-05T22:21:24Z,2022-11-04T11:37:52Z,,NONE,,"Is it possible to deploy a `datasette` application as a python web app?

In my enterprise, I have option to deploy python apps via [Rstudio Connect](https://github.com/rstudio/rsconnect-python), and I would like to publish a `datasette` dashboard for sharing.

I welcome any pointers to converting `datasette serve` into a python app that can be run as something like `python datasette.py --my_data.db`",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1217/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
803356942,MDU6SXNzdWU4MDMzNTY5NDI=,1218, /usr/local/opt/python3/bin/python3.6: bad interpreter: No such file or directory,11855322,robmarkcole,open,0,,,,,1,2021-02-08T09:07:00Z,2021-02-23T12:12:17Z,,NONE,,"Error as above, however I do have python3.8 and the readme indicates this is supported.

```
(venv) (base) Robins-MacBook:datasette robin$ ls /usr/local/opt/python3/bin/

.. pip3 python3 python3.8
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1218/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
803929694,MDU6SXNzdWU4MDM5Mjk2OTQ=,1219,Try profiling Datasette using scalene,9599,simonw,open,0,,,,,2,2021-02-08T20:37:06Z,2021-02-08T22:13:00Z,,OWNER,,https://github.com/emeryberger/scalene looks like an interesting profiling tool.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1219/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
275092453,MDU6SXNzdWUyNzUwOTI0NTM=,122,"Redesign JSON output, ditch jsono, offer variants controlled by parameter instead",9599,simonw,closed,0,,,,,5,2017-11-18T16:52:28Z,2018-04-08T14:54:09Z,2018-04-08T14:54:09Z,OWNER,,"I want to support three variants for the rows output:

* a list of lists, with a columns key saying what they are
* a list of dictionaries
* a single dictionary where the keys are the primary keys of the rows and the values are the row dictionaries themselves

I also want to make the various bits of metadata opt-in - so you don't get the SQL statement unless you ask for it.

These output options should be controlled by query string arguments.

I will set the .jsono URL to redirect to .json with the corresponding options. ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/122/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
806743116,MDU6SXNzdWU4MDY3NDMxMTY=,1220,Installing datasette via docker: Path 'fixtures.db' does not exist,30607,aborruso,closed,0,,,,,4,2021-02-11T21:09:14Z,2021-02-12T21:35:17Z,2021-02-12T21:35:17Z,NONE,,"Hi,
If I run

```
docker run -p 8001:8001 -v `pwd`:/mnt \                                                                         1 ↵
    datasetteproject/datasette \
    datasette -p 8001 -h 0.0.0.0 fixtures.db
```

I have 

```
Error: Invalid value for '[FILES]...': Path 'fixtures.db' does not exist.
```

If I run `test -f fixtures.db && echo ""it exists.""` I have `it exists.`.

What's my error?

Thank you",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1220/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
806849424,MDU6SXNzdWU4MDY4NDk0MjQ=,1221,Support SSL/TLS directly,9599,simonw,closed,0,,,,,4,2021-02-12T00:18:29Z,2022-12-18T02:39:04Z,2021-02-12T00:52:18Z,OWNER,,This should be pretty easy because Uvicorn supports them already. Need a good mechanism for testing it - https://pypi.org/project/trustme/ looks ideal.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1221/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
806861312,MDExOlB1bGxSZXF1ZXN0NTcyMjA5MjQz,1222,"--ssl-keyfile and --ssl-certfile, refs #1221",9599,simonw,closed,0,,,,,0,2021-02-12T00:45:58Z,2021-02-12T00:52:18Z,2021-02-12T00:52:17Z,OWNER,simonw/datasette/pulls/1222,,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1222/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
806918878,MDExOlB1bGxSZXF1ZXN0NTcyMjU0MTAz,1223,Add compile option to Dockerfile to fix failing test (fixes #696),7476523,bobwhitelock,closed,0,,,,,2,2021-02-12T03:38:05Z,2021-03-07T12:01:12Z,2021-03-07T07:41:17Z,CONTRIBUTOR,simonw/datasette/pulls/1223,"This test was failing when run inside the Docker container: `test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3]`,

with this error:

```
    def test_searchable(app_client, path, expected_rows):
        response = app_client.get(path)
>       assert expected_rows == response.json[""rows""]
E       AssertionError: assert [[1, 'barry c...sel', 'puma']] == []
E         Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']
E         Full diff:
E         + []
E         - [[1, 'barry cat', 'terry dog', 'panther'],
E         -  [2, 'terry dog', 'sara weasel', 'puma']]
```

The issue was that the version of sqlite3 built inside the Docker container was built with FTS3 and FTS4 enabled, but without the
`SQLITE_ENABLE_FTS3_PARENTHESIS` compile option passed, which adds support for using `AND` and `NOT` within `match` expressions (see https://sqlite.org/fts3.html#compiling_and_enabling_fts3_and_fts4 and https://www.sqlite.org/compile.html).

Without this, the `AND` used in the search in this test was being interpreted as a literal string, and so no matches were found. Adding this compile option fixes this.

---

I actually ran into this issue because the same test was failing when I ran the test suite on my own machine, outside of Docker, and so I eventually tracked this down to my system sqlite3 also being compiled without this option.

I wonder if this is a sign of a slightly deeper issue, that Datasette can silently behave differently based on the version and compilation of sqlite3 it is being used with. On my own system I fixed the test suite by running `pip install pysqlite3-binary`, so that this would be picked up instead of the `sqlite` package, as this seems to be compiled using this option, . Maybe using `pysqlite3-binary` could be installed/recommended by default so a more deterministic version of sqlite is used? Or there could be some feature detection done on the available sqlite version, to know what features are available and can be used/tested?",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1223/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
807433181,MDU6SXNzdWU4MDc0MzMxODE=,1224,can't start immutable databases from configuration dir mode,295329,camallen,closed,0,,,,,0,2021-02-12T17:50:13Z,2021-03-29T00:17:31Z,2021-03-29T00:17:31Z,CONTRIBUTOR,,"Say I have a `/databases/` directory with multiple sqlite db files in that dir (`1.db` & `2.db`) and an `inspect-data.json` file.

If I start datasette via `datasette -h 0.0.0.0 /databases/` then the resulting databases are set to `is_mutable: true` as inspected via http://127.0.0.1:8001/-/databases.json

I don't want to have to list out the databases by name, e.g. `datasette -i /databases/1.db -i /databases/2.db` as i want the system to autodetect the sqlite dbs i have in the configuration directory 

According to the docs outlined in https://docs.datasette.io/en/latest/settings.html?highlight=immutable#configuration-directory-mode this should be possible
> `inspect-data.json` the result of running datasette inspect - any database files listed here will be treated as immutable, so they should not be changed while Datasette is running
 
I believe that if the `inspect-json.json` file present, then in theory the databases will be automatically set to immutable via this code https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/app.py#L211-L216

However it appears the Click Multiple Options will return a tuple via https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L311-L317

The resulting tuple is passed to the Datasette app via `kwargs` and overrides the behaviour to set the databases to immutable via this arg https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/app.py#L182

If you think this is a bug and needs fixing, I am willing to make a PR to check for the empty `immutable` tuple before calling the Datasette class initializer as I think leaving that class interface alone is the best path here.

Thoughts?

Also - i'm loving Datasette, it truly is a wonderful tool, thank you :)",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1224/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
808771690,MDU6SXNzdWU4MDg3NzE2OTA=,1225,More flexible formatting of records with CSS grid,649467,mhalle,open,0,,,,,0,2021-02-15T19:28:17Z,2021-02-15T19:28:35Z,,NONE,,"In several applications I've been experimenting with alternate formatting of datasette query results. Lately I've found that CSS grids work very well and seem quite general for formatting rows. In CSS I use grid templates to define the layout of each record and the regions for each field, hiding the fields I don't want. It's pretty flexible and looks good. It's also a great basis for highly responsive layout.

I initially thought I'd only use this feature for record detail views, but now I use it for index views as well. 

However, there are some limitations:
* With the existing table templates, it seems that you can change the `display` property on the enclosing `table`, `tbody`, and `tr` to make them be grid-like, but that seems hacky (convert `table` and `tbody` to be `display: block` and `tr` to be `display: grid`).
* More significantly, it's very nice to have the column name available when rendering each record to display headers/field labels. The existing templates don't do that, so a custom `_table` template is necessary.
* I don't know if any plugins are sensitive to whether data is rendered as a table or not since I'm not completely clear how plugins get their data. 
* Regardless, you need custom CSS to take full advantage of grids. I don't have a proposal on how to integrate them more deeply.

It would be helpful to at least have an official example or test that used a grid layout for records to make sure nothing in datasette breaks with it. ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1225/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
808843401,MDU6SXNzdWU4MDg4NDM0MDE=,1226,--port option should validate port is between 0 and 65535,9599,simonw,closed,0,,,,,4,2021-02-15T22:01:33Z,2021-02-18T18:41:27Z,2021-02-18T18:41:27Z,OWNER,,"Currently throws an ugly error message:
```
(datasette-graphql) datasette-graphql % datasette fivethirtyeight.db -p 80094
INFO:     Started server process [45497]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
Traceback (most recent call last):
  File ""/Users/simon/.local/share/virtualenvs/datasette-graphql-n1OSJCS8/bin/datasette"", line 8, in <module>
    sys.exit(cli())
...
    server = await loop.create_server(
  File ""/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/asyncio/base_events.py"", line 1461, in create_server
    sock.bind(sa)
OverflowError: bind(): port must be 0-65535.
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1226/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
810394616,MDU6SXNzdWU4MTAzOTQ2MTY=,1227,Configure sphinx.ext.extlinks for issues,9599,simonw,closed,0,,,,,1,2021-02-17T17:38:02Z,2021-08-02T21:38:39Z,2021-02-18T01:20:33Z,OWNER,,"Spotted this in the aspw documentation: https://github.com/rogerbinns/apsw/blob/3.34.0-r1/doc/conf.py#L29-L36

```python
extlinks={
    'cvstrac': ('https://sqlite.org/cvstrac/tktview?tn=%s',
                'SQLite ticket #'),
    'sqliteapi': ('https://sqlite.org/c3ref/%s.html', 'XXYouShouldNotSeeThisXX'),
    'issue': ('https://github.com/rogerbinns/apsw/issues/%s',
              'APSW issue '),
    'source': ('https://github.com/rogerbinns/apsw/blob/master/%s',
               ''),
    }
```
Which lets you link to issues like this:

    :issue:`268`",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1227/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
810397025,MDU6SXNzdWU4MTAzOTcwMjU=,1228,500 error caused  by faceting if a column called `n` exists,7107523,Kabouik,closed,0,,,,,5,2021-02-17T17:41:20Z,2022-03-19T06:44:40Z,2022-03-19T01:38:04Z,NONE,,"I recently discovered `datasette` thanks to your great talk at FOSDEM and would like to use it for some projects. However, when trying to use it on databases created from some csv ot tsv files, I am sometimes getting this issue when going to http://127.0.0.1:8001/databasetest/databasetest and I don't exactly understand what it refers to.

So far, I couldn't find anything relevant when reviewing the raw text files that could explain this issue, nor could I find something obvious between the files that generate this issue and those that don't. Does the error ring a bell and, if so, could you please point me to the right direction?

```
$ datasette databasetest.db 
INFO:     Started server process [1408482]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
INFO:     127.0.0.1:56394 - ""GET / HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56394 - ""GET /-/static/app.css?4e362c HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56396 - ""GET /-/static-plugins/datasette_vega/main.2acbb312.css HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56398 - ""GET /-/static-plugins/datasette_vega/main.08f5d3d8.js HTTP/1.1"" 200 OK
Traceback (most recent call last):
  File ""/home/kabouik/.local/lib/python3.7/site-packages/datasette/app.py"", line 1099, in route_path
    response = await view(request, send)
  File ""/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/base.py"", line 147, in view
    request, **request.scope[""url_route""][""kwargs""]
  File ""/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/base.py"", line 121, in dispatch_request
    return await handler(request, *args, **kwargs)
  File ""/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/base.py"", line 260, in get
    request, database, hash, correct_hash_provided, **kwargs
  File ""/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/base.py"", line 434, in view_get
    request, database, hash, **kwargs
  File ""/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/table.py"", line 782, in data
    suggested_facets.extend(await facet.suggest())
  File ""/home/kabouik/.local/lib/python3.7/site-packages/datasette/facets.py"", line 168, in suggest
    and any(r[""n""] > 1 for r in distinct_values)
  File ""/home/kabouik/.local/lib/python3.7/site-packages/datasette/facets.py"", line 168, in <genexpr>
    and any(r[""n""] > 1 for r in distinct_values)
TypeError: '>' not supported between instances of 'str' and 'int'
INFO:     127.0.0.1:56402 - ""GET /databasetest/databasetest HTTP/1.1"" 500 Internal Server Error
INFO:     127.0.0.1:56402 - ""GET /-/static/app.css?4e362c HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56404 - ""GET / HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56404 - ""GET /-/static/app.css?4e362c HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56406 - ""GET /-/static-plugins/datasette_vega/main.2acbb312.css HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56408 - ""GET /-/static-plugins/datasette_vega/main.08f5d3d8.js HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56408 - ""GET /databasetest HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56408 - ""GET /-/static/app.css?4e362c HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56404 - ""GET /-/static-plugins/datasette_vega/main.2acbb312.css HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56406 - ""GET /-/static/codemirror-5.57.0.min.css HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56410 - ""GET /-/static-plugins/datasette_vega/main.08f5d3d8.js HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56414 - ""GET /-/static/codemirror-5.57.0-sql.min.js HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56412 - ""GET /-/static/codemirror-5.57.0.min.js HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56408 - ""GET /-/static/sql-formatter-2.3.3.min.js HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56408 - ""GET /databasetest?sql=select+*+from+databasetest HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56410 - ""GET /-/static/app.css?4e362c HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56408 - ""GET /-/static-plugins/datasette_vega/main.2acbb312.css HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56412 - ""GET /-/static/codemirror-5.57.0.min.css HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56404 - ""GET /-/static/sql-formatter-2.3.3.min.js HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56406 - ""GET /-/static/codemirror-5.57.0.min.js HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56414 - ""GET /-/static-plugins/datasette_vega/main.08f5d3d8.js HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56408 - ""GET /-/static/codemirror-5.57.0-sql.min.js HTTP/1.1"" 200 OK
INFO:     127.0.0.1:56410 - ""GET /databasetest.json?sql=select+*+from+databasetest&_shape=array&_shape=array HTTP/1.1"" 200 OK
^CINFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [1408482]
```

Note that there is no error if I go to http://127.0.0.1:8001/databasetest and then click on `Run SQL`.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1228/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
810507413,MDExOlB1bGxSZXF1ZXN0NTc1MTg3NDU3,1229,ensure immutable databses when starting in configuration directory mode with,295329,camallen,closed,0,,,,,3,2021-02-17T20:18:26Z,2022-04-22T13:16:36Z,2021-03-29T00:17:32Z,CONTRIBUTOR,simonw/datasette/pulls/1229,"fixes #1224 

This PR ensures all databases found in a configuration directory that match the files in `inspect-data.json` will be set to `immutable` as outlined in https://docs.datasette.io/en/latest/settings.html#configuration-directory-mode

specifically on building the `datasette` instance it checks:
- if `immutables` is an empty tuple - as passed by the cli code
- if `immutables` is the default function value `None` - when it's not explicitly set

And correctly builds the immutable database list from the `inspect-data[file]` keys.

Note for this to work the `inspect-data.json` file must contain `file` paths which are relative to the configuration directory otherwise the file paths won't match and the dbs won't be set to immutable. 

I couldn't find an easy way to test this due to the way `make_app_client` works, happy to take directions on adding a test for this. 

I've updated the relevant docs as well, i.e. use the `inspect` cli cmd from the config directory path to create the relevant file
```
cd $config_dir
datasette inspect *.db --inspect-file=inspect-data.json
```
https://docs.datasette.io/en/latest/performance.html#using-datasette-inspect",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1229/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
275125561,MDU6SXNzdWUyNzUxMjU1NjE=,123,Datasette serve should accept paths/URLs to CSVs and other file formats,9599,simonw,open,0,,,,,9,2017-11-19T02:05:48Z,2021-07-19T00:04:32Z,,OWNER,,"This would remove the csvs-to-sqlite step which I end up using for almost everything.

I'm hesitant to introduce pandas as a required dependency though since it require compiling numpy. Could build it so this option is only available if you have pandas installed.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/123/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,
811367257,MDU6SXNzdWU4MTEzNjcyNTc=,1231,Race condition errors in new refresh_schemas() mechanism,9599,simonw,closed,0,,,,,11,2021-02-18T18:49:54Z,2021-07-16T19:44:59Z,2021-07-16T19:44:59Z,OWNER,,I tried running a Locust load test against Datasette and hit an error message about a failure to create tables because they already existed. I think this means there are race conditions in the new `refresh_schemas()` mechanism added in #1150.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1231/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
811407131,MDExOlB1bGxSZXF1ZXN0NTc1OTQwMTkz,1232,--crossdb option for joining across databases,9599,simonw,closed,0,,,,,8,2021-02-18T19:48:50Z,2021-02-18T22:09:13Z,2021-02-18T22:09:12Z,OWNER,simonw/datasette/pulls/1232,"Refs #283. Still needs:

- [x] Unit test for --crossdb queries
- [x] Show warning on console if it truncates at ten databases (or on web interface)
- [x] Show connected databases on the `/_memory` database page
- [x] Documentation
- [x] https://latest.datasette.io/ demo should demonstrate this feature",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1232/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
811458446,MDU6SXNzdWU4MTE0NTg0NDY=,1233,"""datasette publish cloudrun"" cannot publish files with spaces in their name",9599,simonw,open,0,,,,,1,2021-02-18T21:08:31Z,2021-02-18T21:10:08Z,,OWNER,,"Got this error:
```
Step 6/9 : RUN datasette inspect fixtures.db extra database.db --inspect-file inspect-data.json
 ---> Running in db9da0068592
Usage: datasette inspect [OPTIONS] [FILES]...
Try 'datasette inspect --help' for help.

Error: Invalid value for '[FILES]...': Path 'extra' does not exist.
The command '/bin/sh -c datasette inspect fixtures.db extra database.db --inspect-file inspect-data.json' returned a non-zero code: 2
ERROR
ERROR: build step 0 ""gcr.io/cloud-builders/docker"" failed: step exited with non-zero status: 2
```
While working on the demo for #1232, using this deploy command:
```
GITHUB_SHA=crossdb datasette publish cloudrun fixtures.db 'extra database.db' \
            -m fixtures.json \
            --plugins-dir=plugins \
            --branch=$GITHUB_SHA \
            --version-note=$GITHUB_SHA \
            --extra-options=""--setting template_debug 1 --crossdb"" \
            --install=pysqlite3-binary \
            --service=datasette-latest-crossdb
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1233/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
811505638,MDU6SXNzdWU4MTE1MDU2Mzg=,1234,Runtime support for ATTACHing multiple databases,9599,simonw,open,0,,,,,1,2021-02-18T22:06:47Z,2021-02-22T21:06:28Z,,OWNER,,"> The implementation in #1232 is ready to land. It's the simplest-thing-that-could-possibly-work: you can run `datasette one.db two.db three.db --crossdb` and then use the `/_memory` page to run joins across tables from multiple databases.
>
> It only works on the first 10 databases that were passed to the command-line. This means that if you have a Datasette instance with hundreds of attached databases (see [Datasette Library](https://github.com/simonw/datasette/issues/417)) this won't be particularly useful for you.
>
> So... a better, future version of this feature would be one that lets you join across databases on command - maybe by hitting `/_memory?attach=db1&attach=db2` to get a special connection.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/283#issuecomment-781665560_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1234/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
811589344,MDU6SXNzdWU4MTE1ODkzNDQ=,1235,Upgrade Python version used by official Datasette Docker image,9599,simonw,closed,0,,,,,2,2021-02-19T00:47:40Z,2021-02-19T01:48:31Z,2021-02-19T01:48:30Z,OWNER,,"Currently uses 3.7.2:

https://github.com/simonw/datasette/blob/73bed175631a79e13a521eee82f8451dd0477eb3/Dockerfile#L1

There's a security fix for Python which it would be good to ship in this image (even though I'm reasonably confident it doesn't affect Datasette): https://bugs.python.org/issue42938",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1235/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
812228314,MDU6SXNzdWU4MTIyMjgzMTQ=,1236,Ability to increase size of the SQL editor window,9599,simonw,closed,0,,,,,9,2021-02-19T18:09:27Z,2021-05-18T03:28:25Z,2021-02-22T21:05:21Z,OWNER,,,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1236/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
812704869,MDU6SXNzdWU4MTI3MDQ4Njk=,1237,?_pretty=1 option for pretty-printing JSON output,9599,simonw,open,0,,,3268330,Datasette 1.0,1,2021-02-20T20:54:40Z,2021-11-16T18:28:33Z,,OWNER,,Suggested by @frankieroberto in https://github.com/simonw/datasette/issues/782#issuecomment-782746755,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1237/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
813899472,MDU6SXNzdWU4MTM4OTk0NzI=,1238,Custom pages don't work with base_url setting,79913,tsibley,closed,0,,,,,9,2021-02-22T21:58:58Z,2021-06-05T18:59:55Z,2021-06-05T18:59:55Z,NONE,,"It seems that custom pages aren't routing properly when the `base_url` setting is used.

To reproduce, with Datasette 0.55.

Create a `templates/pages/custom.html` with some text.
```
mkdir -p templates/pages/
echo ""Hello, world!"" > templates/pages/custom.html
```

Start Datasette.

```
datasette --template-dir templates/
```

Visit http://localhost:8001/custom and see ""Hello, world!"".

Start Datasette with a `base_url`.

```
datasette --template-dir templates/ --setting base_url /prefix/
```

Visit http://localhost:8001/prefix/custom and see a ""Database not found: custom"" 404.

Note that like all routes, http://localhost:8001/custom still works when run with `base_url`.
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1238/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
813978858,MDU6SXNzdWU4MTM5Nzg4NTg=,1239,JSON filter fails if column contains spaces,9599,simonw,closed,0,,,,,1,2021-02-23T00:18:07Z,2021-02-23T00:22:53Z,2021-02-23T00:22:53Z,OWNER,,"Got this exception:

`ERROR: conn=<sqlite3.Connection object at 0x10ea68e40>, sql = 'select Address, Affiliation, County, [Has Report], [Latest report notes], [Latest report yes], Latitude, [Location Type], Longitude, Name, id, [Appointment scheduling instructions], [Availability Info], [Latest report] from locations where rowid in (\n            select locations.rowid from locations, json_each(locations.Availability Info) j\n            where j.value = :p0\n        ) and ""Latest report yes"" = :p1 order by id limit 101', params = {'p0': 'Yes: appointment required', 'p1': '1'}: near ""Info"": syntax error`",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1239/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275125805,MDU6SXNzdWUyNzUxMjU4MDU=,124,Option to open readonly but not immutable,9599,simonw,closed,0,,,,,5,2017-11-19T02:11:03Z,2019-06-24T06:43:46Z,2019-06-24T06:43:46Z,OWNER,,Immutable assumes no other process can modify the file. An option to open reqdonly instead would enable other processes to update the file in place.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/124/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
814591962,MDU6SXNzdWU4MTQ1OTE5NjI=,1240,Allow facetting on custom queries,7107523,Kabouik,closed,0,,,,,3,2021-02-23T15:52:19Z,2021-02-26T18:19:46Z,2021-02-26T18:18:18Z,NONE,,"Facets are a tremendously useful feature, especially for people peeking at the database for the first time and still having little knowledge about the details of the data. It is of great assistance to discover interesting features to explore futher in advanced queries.

Yet, it seems it's impossible to use facets when running a custom SQL query, be it from the little gear icons in column names, the facet suggestions at the top (hidden when performing a custom query), or by appending a facet code to the URL. 

Is there a technical limitation, or is this something that could be unlocked easily?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1240/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
814595021,MDU6SXNzdWU4MTQ1OTUwMjE=,1241,Share button for copying current URL,7107523,Kabouik,open,0,,,,,6,2021-02-23T15:55:40Z,2023-08-24T20:09:52Z,,NONE,,"I use datasette in an `iframe` inside another HTML file that contains other ways to represent my data (mostly leaflets maps built with R on summarized data), and the datasette `iframe` is a tab in that page. 

This particular use prevents users to access the full URLs of their datasette views and queries, which is a shame because the way datasette handles URLs to make every view or query easy to share is awesome. I know how to get the URL from the context menu of my browser, but I don't think many visitors would do it or even notice that datasette uses permalinks for pretty much every action they do. Would it be possible to add a ""Share link"" button to the interface, either in datasette itself or in a plugin?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1241/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
815955014,MDExOlB1bGxSZXF1ZXN0NTc5Njk3ODMz,1243,fix small typo,306240,UtahDave,closed,0,,,,,2,2021-02-25T00:22:34Z,2021-03-04T05:46:10Z,2021-03-04T05:46:10Z,CONTRIBUTOR,simonw/datasette/pulls/1243,,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1243/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
817528452,MDU6SXNzdWU4MTc1Mjg0NTI=,1244,Plugin tip: look at the examples linked from the hooks page,9599,simonw,closed,0,,,,,1,2021-02-26T17:18:27Z,2021-02-26T17:30:38Z,2021-02-26T17:27:15Z,OWNER,,"Someone asked ""what are good example plugins I can look at?"" and I realized that the answer is to look through the example links on https://docs.datasette.io/en/stable/plugin_hooks.html - but that tip should be written down somewhere on the https://docs.datasette.io/en/stable/writing_plugins.html page.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1244/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
817544251,MDU6SXNzdWU4MTc1NDQyNTE=,1245,"Sticky table column headers would be useful, especially on the query page",9599,simonw,open,0,,,,,1,2021-02-26T17:42:51Z,2021-04-02T20:53:35Z,,OWNER,,Suggestion from office hours.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1245/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
817597268,MDU6SXNzdWU4MTc1OTcyNjg=,1246,Suggest for ArrayFacet possibly confused by blank values,9599,simonw,closed,0,,,,,3,2021-02-26T19:11:52Z,2021-03-01T03:46:11Z,2021-03-01T03:46:11Z,OWNER,,I sometimes don't get the suggestion for facet-by-array for columns that contain arrays. I think it may be because they have empty spaces in them - or perhaps it's because the null detection doesn't actually work.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1246/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
818430405,MDU6SXNzdWU4MTg0MzA0MDU=,1247,datasette.add_memory_database() method,9599,simonw,closed,0,,,,,2,2021-03-01T03:48:38Z,2021-03-01T04:02:26Z,2021-03-01T04:02:26Z,OWNER,,"I just wrote this code:

https://github.com/simonw/datasette/blob/47eb885cc2c3aafa03645c330c6f597bee9b3b25/tests/test_facets.py#L334-L335

It would be nice if you didn't have to separately instantiate a database object here.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1247/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
823035080,MDU6SXNzdWU4MjMwMzUwODA=,1248,duckdb database (very low performance in SQLite),15836677,verajosemanuel,closed,0,,,,,1,2021-03-05T12:20:29Z,2021-03-08T00:25:27Z,2021-03-08T00:25:27Z,NONE,,"My sqlite is getting too big to be processed by datasette (more than 10 minutes waiting to load) so I am working with duckdb and is waaaaay faster. I think the fastest embeddable database actually.

https://duckdb.org/

Taking into account DuckDb is SQLite based it would be GREAT to use it with datasette.

is that possible?

Regards and thanks for a superb job",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1248/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
824064069,MDU6SXNzdWU4MjQwNjQwNjk=,1249,Updated Dockerfile with SpatiaLite version 5.0,9599,simonw,closed,0,,,,,45,2021-03-08T00:17:36Z,2022-01-20T21:29:43Z,2021-03-29T00:57:13Z,OWNER,,"The version bundled in Datasette's Docker image right now is 4.4.0-RC0

https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/Dockerfile#L16-L17

5 has been out for a couple of months and has a bunch of big improvements, most notable stable KNN support.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1249/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275135393,MDU6SXNzdWUyNzUxMzUzOTM=,125,Plot rows on a map with Leaflet and Leaflet.markercluster,9599,simonw,closed,0,,,,,2,2017-11-19T06:05:05Z,2018-04-26T15:14:31Z,2018-04-26T15:14:31Z,OWNER,,"https://github.com/Leaflet/Leaflet.markercluster would allow us to paginate-load in an enormous set of rows with latitude/longitude points, e.g. https://australian-dunnies.now.sh/

Here's a demo of it loading 50,000 markers: https://leaflet.github.io/Leaflet.markercluster/example/marker-clustering-realworld.50000.html - and it looks like it's easy to support progress bars for if we were iteratively loading 1,000 markers at a time using datasette pagination.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/125/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
824067604,MDU6SXNzdWU4MjQwNjc2MDQ=,1250,Research: Plugin hook for alternative database connections,9599,simonw,closed,0,,,,,2,2021-03-08T00:28:15Z,2021-03-12T01:01:25Z,2021-03-12T01:01:17Z,OWNER,,"The `Database` class is a natural looking fit for a plugin hook to load custom database connections... potentially even databases other than SQLite. DuckDB (refs #968) could make for a great starting point, since it looks very compatible with the existing SQLite code.

The real win would be if this could lead to running Datasette against PostgreSQL. I made some initial explorations in that direction a while ago in #670.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1250/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
824750134,MDU6SXNzdWU4MjQ3NTAxMzQ=,1251,facet option not appearing when table is big,15836677,verajosemanuel,open,0,,,,,0,2021-03-08T16:54:04Z,2021-03-08T16:54:16Z,,NONE,,"I have a big table with more than 500.000 rows.
Trying to facet by one of my columns, the options are not available as for the other smaller tables.

I have tried to set it in URL as:

`&_facet=city_id`

to no avail.

is there any limit?  how can I force the option ""facet"" to appear for big tables?
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1251/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
825217564,MDExOlB1bGxSZXF1ZXN0NTg3MzMyNDcz,1252,Add back styling to lists within table cells (fixes #1141),7476523,bobwhitelock,closed,0,,,,,2,2021-03-09T03:00:57Z,2021-03-29T00:14:04Z,2021-03-29T00:14:04Z,CONTRIBUTOR,simonw/datasette/pulls/1252,"This overrides the Datasette reset - see https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/static/app.css#L35-L38 - to add back the default styling of list items displayed within Datasette table cells.

Following this change, the same content as in the original issue looks like this:

![2021-03-09_02:57:32](https://user-images.githubusercontent.com/7476523/110411982-63e5ae80-8083-11eb-9b5c-e5dc825073e2.png)
",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1252/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
826064552,MDU6SXNzdWU4MjYwNjQ1NTI=,1253,"Capture ""Ctrl + Enter"" or ""⌘ + Enter"" to send SQL query?",9308268,rayvoelker,open,0,,,,,1,2021-03-09T15:00:50Z,2021-10-30T16:00:42Z,,NONE,,"It appears as though ""Shift + Enter"" triggers the form submit action to submit SQL, but could that action be bound to the ""Ctrl + Enter"" or ""⌘ + Enter"" action?

I feel like that pattern already exists in a number of similar tools and could improve usability of the editor.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1253/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
826613352,MDExOlB1bGxSZXF1ZXN0NTg4NjAxNjI3,1254,Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions,3200608,durkie,closed,0,,,,,6,2021-03-09T20:49:08Z,2021-03-10T18:27:45Z,2021-03-09T22:04:23Z,NONE,simonw/datasette/pulls/1254,"This requires adding the RT Topology library (Spatialite changed to RT Topology from LWGEOM between 4.4 and 5.0), as well as upgrading the GEOS version (which is the reason for switching to `python:3.7.10-slim-buster` as the base image.)

`autoconf` and `libtool` are added to build RT Topology, and Spatialite is now built with `--disable-minizip` (minizip wasn't an option in 4.4 and I didn't want to add another dependency) and `--disable-dependency-tracking` which, according to Spatialite, ""speeds up one-time builds""",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1254/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
826700095,MDU6SXNzdWU4MjY3MDAwOTU=,1255,Facets timing out but work when filtering,1219001,robroc,open,0,,,,,2,2021-03-09T22:01:39Z,2021-04-02T20:50:08Z,,NONE,,"System info:

Windows 10
Datasette 0.55 installed via pip
Python 3.8.5 in a conda environment

I'm getting the message `These facets timed out` on any faceting operation. However, when I apply a filter, the facets appear in the filtered view. The error returns when the filter is removed. My data only has 38,450 rows.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1255/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
827341657,MDExOlB1bGxSZXF1ZXN0NTg5MjYzMjk3,1256,Minor type in IP adress,6371750,JBPressac,closed,0,,,,,3,2021-03-10T08:28:22Z,2021-03-10T18:26:46Z,2021-03-10T18:26:40Z,CONTRIBUTOR,simonw/datasette/pulls/1256,127.0.01 replaced by 127.0.0.1,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1256/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
828811618,MDU6SXNzdWU4Mjg4MTE2MTg=,1257,Table names containing single quotes break things,9599,simonw,closed,0,,,,,2,2021-03-11T06:29:38Z,2021-06-02T03:28:29Z,2021-06-02T03:28:29Z,OWNER,,"e.g. I found a table called `Yesterday's ELRs by County`

It threw an error inside the `detect_fts()` function attempting to run this SQL query:

```sql
        select name from sqlite_master
            where rootpage = 0
            and (
                sql like '%VIRTUAL TABLE%USING FTS%content=""Yesterday's ELRs by County""%'
                or sql like '%VIRTUAL TABLE%USING FTS%content=[Yesterday's ELRs by County]%'
                or (
                    tbl_name = ""Yesterday's ELRs by County""
                    and sql like '%VIRTUAL TABLE%USING FTS%'
                )
            )
```
Here's the code at fault: https://github.com/simonw/datasette/blob/640ac7071b73111ba4423812cd683756e0e1936b/datasette/utils/__init__.py#L534-L548",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1257/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
828858421,MDU6SXNzdWU4Mjg4NTg0MjE=,1258,Allow canned query params to specify default values,1385831,wdccdw,open,0,,,,,5,2021-03-11T07:19:02Z,2023-02-20T23:39:58Z,,NONE,,"If I call a canned query that includes named parameters, without passing any parameters, datasette runs the query anyway, resulting in an HTTP status code 400, and a visible error in the browser, with only a link back to home.  This means that one of the default links on https://site/database/ will lead to a broken page with no apparent way out.

![image](https://user-images.githubusercontent.com/1385831/110748683-13e72300-820e-11eb-855c-32e03dfef5bf.png)

Is there any way to skip performing the query when parameters aren't supplied, but otherwise render the usual canned query page?  Alternatively, can I supply default values for my parameters, either when defining my canned queries or when linking to the canned query page from the default database template.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1258/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
830567275,MDU6SXNzdWU4MzA1NjcyNzU=,1259,Research using CTEs for faster facet counts,9599,simonw,open,0,,,,,5,2021-03-12T22:19:49Z,2021-03-21T22:55:31Z,,OWNER,,"https://www.sqlite.org/changes.html#version_3_35_0

> Add support for the [MATERIALIZED](https://www.sqlite.org/lang_with.html#mathint) and [NOT MATERIALIZED](https://www.sqlite.org/lang_with.html#mathint) hints when specifying [common table expressions](https://www.sqlite.org/lang_with.html). The default behavior was formerly NOT MATERIALIZED, but is now changed to MATERIALIZED for CTEs that are used more than once.

If a CTE creates a table that is used multiple time in that query, SQLite will now default to creating a materialized table for the duration of that query.

This could be a big performance boost when applying faceting multiple times against the same query. Consider this example query:
```sql
WITH data as (
  select
    *
  from
    [global-power-plants]
),
country_long as (select 
  'country_long' as col, country_long as value, count(*) as c from data group by country_long
  order by c desc limit 10
),
primary_fuel as (
select
  'primary_fuel' as col, primary_fuel as value, count(*) as c from data group by primary_fuel
  order by c desc limit 10
)
select * from primary_fuel union select * from country_long order by col, c desc
```
https://global-power-plants.datasettes.com/global-power-plants?sql=WITH+data+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Acountry_long+as+%28select+%0D%0A++%27country_long%27+as+col%2C+country_long+as+value%2C+count%28*%29+as+c+from+data+group+by+country_long%0D%0A++order+by+c+desc+limit+10%0D%0A%29%2C%0D%0Aprimary_fuel+as+%28%0D%0Aselect%0D%0A++%27primary_fuel%27+as+col%2C+primary_fuel+as+value%2C+count%28*%29+as+c+from+data+group+by+primary_fuel%0D%0A++order+by+c+desc+limit+10%0D%0A%29%0D%0Aselect+*+from+primary_fuel+union+select+*+from+country_long+order+by+col%2C+c+desc

Outputs:

col | value | c
-- | -- | --
country_long | United States of America | 8688
country_long | China | 4235
country_long | United Kingdom | 2603
country_long | Brazil | 2360
country_long | France | 2155
country_long | India | 1590
country_long | Germany | 1309
country_long | Canada | 1159
country_long | Spain | 829
country_long | Russia | 545
primary_fuel | Solar | 9662
primary_fuel | Hydro | 7155
primary_fuel | Wind | 5188
primary_fuel | Gas | 3922
primary_fuel | Coal | 2390
primary_fuel | Oil | 2290
primary_fuel | Biomass | 1396
primary_fuel | Waste | 1087
primary_fuel | Nuclear | 198
primary_fuel | Geothermal | 189
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1259/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
275135535,MDU6SXNzdWUyNzUxMzU1MzU=,126,Blog entry announcing foreign key support,9599,simonw,closed,0,,,2919870,Foreign key edition,1,2017-11-19T06:09:06Z,2017-11-30T16:49:24Z,2017-11-30T16:49:24Z,OWNER,,,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/126/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
831163537,MDExOlB1bGxSZXF1ZXN0NTkyNTQ4MTAz,1260,Fix: code quality issues,25361949,withshubh,closed,0,,,,,2,2021-03-14T13:56:10Z,2021-03-29T00:22:41Z,2021-03-29T00:22:41Z,NONE,simonw/datasette/pulls/1260,"### Description
Hi :wave: I work at [DeepSource](https://deepsource.io), I ran DeepSource analysis on the forked copy of this repo and found some interesting [code quality issues](https://deepsource.io/gh/withshubh/datasette/issues/?category=recommended) in the codebase, opening this PR so you can assess if our platform is right and helpful for you.

### Summary of changes

- Replaced ternary syntax with if expression
- Removed redundant `None` default
- Used `is` to compare type of objects
- Iterated dictionary directly
- Removed unnecessary lambda expression
- Refactored unnecessary `else` / `elif` when `if` block has a `return` statement
- Refactored unnecessary `else` / `elif` when `if` block has a `raise` statement
- Added .deepsource.toml to continuously analyze and detect code quality issues",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1260/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
832092321,MDU6SXNzdWU4MzIwOTIzMjE=,1261,Some links aren't properly URL encoded.,812795,brimstone,closed,0,,,,,3,2021-03-15T18:43:59Z,2021-03-21T02:06:44Z,2021-03-20T21:36:06Z,NONE,,"It seems like a percent sign in the query causes some links to end invalid.

The json and CSV links on this page don't behave like expected: https://honeypot-brimston3.vercel.app/honeypot?sql=select+time%2C+count%28time%29+as+count+from+%28select+strftime%28%22%25Y-%25m-%25d%22%2C+_etime%29+as+time+from+ssh+%29+group+by+time+order+by+time%3B

I can take a swing at trying to fix this, but my python isn't strong and I need a pointer at the right approach and files to change.

Thanks!",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1261/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
834602299,MDU6SXNzdWU4MzQ2MDIyOTk=,1262,Plugin hook that could support 'order by random()' for table view,19328961,henry501,open,0,,,,,3,2021-03-18T10:02:01Z,2021-03-18T17:55:01Z,,NONE,,"I am frequently using Datasette to quickly get a visual impression for a table without reviewing it in its entirety. Because I have some groups of similar records, the default sorting options mean that each page is very similar and not representative of the full dataset. The current interface allows sorting by columns, but random sorting is only available via custom SQL.

Maybe this could be a button or link.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1262/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
836123030,MDU6SXNzdWU4MzYxMjMwMzA=,1265,Support for HTTP Basic Authentication,468612,yunzheng,closed,0,,,,,3,2021-03-19T15:31:09Z,2021-03-19T22:05:12Z,2021-03-19T21:03:09Z,NONE,,"It would be nice if datasette could support [HTTP Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication).

For now I could ofcourse leverage Nginx for basic authentication, but it would be nice to have support for this in datasette by default or via a plugin like datasette-auth-github.

My main usecase is to put the whole datasette instance behind a username/password prompt via Basic Auth and not specific urls.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1265/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
836273891,MDU6SXNzdWU4MzYyNzM4OTE=,1266,Documentation for Response.asgi_send(send) method,9599,simonw,closed,0,,,,,1,2021-03-19T18:52:49Z,2021-03-20T21:35:00Z,2021-03-20T21:32:28Z,OWNER,,"I found myself wanting to use this method for https://github.com/simonw/datasette-auth-passwords/issues/15 - but it's not documented. It should be documented.

https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/asgi.py#L320-L340",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1266/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
837208901,MDU6SXNzdWU4MzcyMDg5MDE=,1267,Update Datasette alternativeto listening with details,921217,RayBB,closed,0,,,,,1,2021-03-21T23:20:20Z,2021-03-22T04:37:26Z,2021-03-22T04:37:26Z,NONE,,"Hello,

I recently learned about Datasette from an old hackernews post. It seems like an awesome project and I actually have use case I might be trying out in the coming months.

Alas, to get a better understanding of your project I looked it up on alternativeto to see what it is similar too. I promise it's not spam, it's reputable enough to have a [Wikipedia](https://en.wikipedia.org/wiki/AlternativeTo) page. There was no listing on the website so I went ahead and created a listing that is now approved.

I encourage anyone who likes this project and hopes to spread the word to help update the listing by:
1. Adding to the list of software it compares to
2. Uploading screenshots
3. Writing a review
4. Adding ""features""

I know this may seem spammy but I promise I have no affiliation with alternativeto I'm just a happy user and know it's a popular site for discovering software.

Here is the listing for datasette:
https://alternativeto.net/software/datasette/about/

Cheers",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1267/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
837308703,MDU6SXNzdWU4MzczMDg3MDM=,1268,Figure out why SpatiaLite 5.0 hangs the database page on Linux,9599,simonw,closed,0,,,,,18,2021-03-22T04:44:16Z,2022-01-20T21:29:43Z,2021-03-22T17:41:12Z,OWNER,,"See detailed notes in https://github.com/simonw/datasette/issues/1249 - for some reason SpatiaLite 5.0 hangs the `/dbname` page on Linux (inside Docker containers, both with a custom compiled SpatiaLite and one installed from the Ubuntu 20.10 package repository). This doesn't happen on macOS with SpatiaLite 5 installed using Homebrew.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1268/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
837348479,MDU6SXNzdWU4MzczNDg0Nzk=,1269,Don't attempt to run count(*) against virtual tables,9599,simonw,closed,0,,,,,2,2021-03-22T05:57:43Z,2021-03-22T17:40:42Z,2021-03-22T17:40:41Z,OWNER,,"Counting the rows in a virtual table doesn't seem very interesting to me, and it's the cause of at least one crashing bug with SpatiaLite 5.0 on Linux, see https://github.com/simonw/datasette/issues/1268",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1269/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275135719,MDU6SXNzdWUyNzUxMzU3MTk=,127,"Filtered tables should show count of all matching rows, if fast enough",9599,simonw,closed,0,,,2919870,Foreign key edition,2,2017-11-19T06:13:29Z,2017-11-24T22:02:01Z,2017-11-24T22:02:01Z,OWNER,,"Relates to #86. If you are viewing a filtered page e.g. https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9/bob-ross%2Felements-by-episode?CLOUDS=1 we should show the count of matching rows.

Since this could be an expensive operation, we will run it with a strict time limit (maybe 50ms). If the time limit is exceeded we will display ""many"" instead, perhaps? Maybe even link to a count(*) query that would get the full 1000ms time limit which the user can click on if they like (that could even Ajax-in the result).",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/127/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
837350092,MDU6SXNzdWU4MzczNTAwOTI=,1270,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),9599,simonw,open,0,,,,,3,2021-03-22T06:00:17Z,2021-03-23T16:45:39Z,,OWNER,,"> Maybe I could implement SQLite query timeouts using the `interrupt()` method instead of the progress handler hack I'm currently using?
>
> https://stackoverflow.com/questions/43240496/python-sqlite3-how-to-quickly-and-cleanly-interrupt-long-running-query-with-e has some tips.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1268#issuecomment-803764919_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1270/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
837956424,MDExOlB1bGxSZXF1ZXN0NTk4MjEzNTY1,1271,Use SQLite conn.interrupt() instead of sqlite_timelimit(),9599,simonw,open,0,,,,,3,2021-03-22T17:34:20Z,2021-03-22T21:49:27Z,,OWNER,simonw/datasette/pulls/1271,"Refs #1270, #1268, #1249

Before merging this I need to do some more testing (to make sure that expensive queries really are properly cancelled). I also need to delete a bunch of code relating to the old mechanism of cancelling queries.

[See comment below: this doesn't actually cancel the query due to a thread-local confusion]",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1271/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1,
838245338,MDU6SXNzdWU4MzgyNDUzMzg=,1272,Unit tests for the Dockerfile,9599,simonw,open,0,,,,,3,2021-03-23T01:36:29Z,2022-07-29T10:22:59Z,,OWNER,,"Working on the Dockerfile in #1249 made me wish for automated tests - to confirm that it boots up correctly, can run SpatiaLite and doesn't have weird bugs like the `/db` page hanging.

These could run in CI too, but maybe only if the `Dockerfile` is updated.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1272/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
838382890,MDU6SXNzdWU4MzgzODI4OTA=,1273,Refresh SpatiaLite documentation,9599,simonw,open,0,,,,,4,2021-03-23T06:05:55Z,2022-01-20T21:28:50Z,,OWNER,,https://docs.datasette.io/en/0.55/spatialite.html was written before I had tools like [geojson-to-sqlite](https://datasette.io/tools/geojson-to-sqlite) and [shapefile-to-sqlite](https://datasette.io/tools/shapefile-to-sqlite).,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1273/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
839008371,MDU6SXNzdWU4MzkwMDgzNzE=,1274,Might there be some way to comment metadata.json?,192568,mroswell,closed,0,,,,,2,2021-03-23T18:33:00Z,2021-03-23T20:14:54Z,2021-03-23T20:14:54Z,CONTRIBUTOR,,"I don't know what license to use... Would be nice to be able to add a comment regarding that uncertainty in my metadata.json file

I like laktak's little video comment in favor of Human json (Hjson)
https://stackoverflow.com/questions/244777/can-comments-be-used-in-json

Hmmm... one of the commenters there said comments are allowed in yaml... so that's a good argument for yaml.

Anyhow, just came to mind, and thought I'd mention it here. Looks like https://hjson.github.io/ has the details.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1274/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
839367451,MDU6SXNzdWU4MzkzNjc0NTE=,1275,Idea: long-running query mode,9599,simonw,open,0,,,,,0,2021-03-24T05:23:20Z,2021-03-24T05:23:20Z,,OWNER,,"It would be cool if you could run Datasette in a long-running query mode, for use with trusted users - something like this:

    datasette --unlimited my.db

This would disable the query limit, but would also enable a feature where if a query takes longer than e.g. 1s to return Datasette returns an HTML page to the browser with a progress indicator and polls the server until the query is complete.... but also provides the user with a ""cancel"" button. This relates to the `.interrupt()` research in #1270.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1275/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
841456306,MDU6SXNzdWU4NDE0NTYzMDY=,1276,"Invalid SQL: ""no such table: pragma_database_list"" on database page",1314318,justinallen,closed,0,,,,,7,2021-03-26T00:03:53Z,2021-03-31T16:27:27Z,2021-03-28T23:52:31Z,NONE,,"Don't think this has been covered here yet. I'm a little stumped with this one and can't tell if it's a bug or I have something misconfigured. 

Oddly, when running locally the usual list of tables populates (i.e. at /charts a list of tables in charts.db). But when on the web server it throws an Invalid SQL error and ""no such table: pragma_database_list"" below. 

All the url endpoints seem to work fine aside from this - individual tables (/charts/chart_one), as well as stored queries (/charts/query_one). 

Not sure if this has anything to do with upgrading to Datasette 0.55, or something to do with our setup, which uses a metadata build script similar to [the one for the 538 server](https://github.com/simonw/fivethirtyeight-datasette/blob/main/make_metadata.py), or something else. 
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1276/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
842212586,MDU6SXNzdWU4NDIyMTI1ODY=,1277,Facet by array breaks if table name contains a space,9599,simonw,closed,0,,,,,1,2021-03-26T18:38:19Z,2021-03-27T03:49:38Z,2021-03-27T03:49:34Z,OWNER,,It breaks when you try to select a filtered item.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1277/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
842416110,MDU6SXNzdWU4NDI0MTYxMTA=,1278,SpatiaLite timezones demo is broken,9599,simonw,closed,0,,,,,2,2021-03-27T04:45:27Z,2022-01-20T21:29:43Z,2021-03-27T16:17:13Z,OWNER,,https://github.com/simonw/datasette/blob/5fd02890650db790b2ffdb90eb9f78f8e0639c37/docs/spatialite.rst#L96,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1278/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
842556944,MDExOlB1bGxSZXF1ZXN0NjAyMTA3OTM1,1279,Minor Docs Update. Added `--app` to fly install command.,1019791,koaning,closed,0,,,,,2,2021-03-27T16:58:08Z,2021-03-29T00:11:55Z,2021-03-29T00:11:55Z,CONTRIBUTOR,simonw/datasette/pulls/1279,"Without this flag, there's an error locally. 

```
> datasette publish fly bigmac.db

Usage: datasette publish fly [OPTIONS] [FILES]...
Try 'datasette publish fly --help' for help.

Error: Missing option '-a' / '--app'.
```

I also got an error message which later turned out to be because I hadn't added my credit card information yet to `fly`. I wasn't sure if I should add that mention to the docs here, or to submit a bug-report over at https://github.com/simonw/datasette-publish-fly. ",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1279/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
275159710,MDU6SXNzdWUyNzUxNTk3MTA=,128,"Every visualization should have an ""embed"" button",9599,simonw,open,0,,,,,0,2017-11-19T13:38:13Z,2019-05-13T18:33:51Z,,OWNER,,"At least for the first round of visualizations, any time you construct one using the UI the result should include an ""embed this"" button that returns source code to copy and paste

These examples should use unpkg.com (or similarl) urls with SRI hashes, eg https://www.srihash.org - and should load data from the datasette JSON API.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/128/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
842862708,MDU6SXNzdWU4NDI4NjI3MDg=,1280,Ability to run CI against multiple SQLite versions,9599,simonw,open,0,,,,,2,2021-03-28T23:54:50Z,2021-05-10T19:07:46Z,,OWNER,,"Issue #1276 happened because I didn't run tests against a SQLite version prior to 3.16.0 (released 2017-01-02).

Glitch is a deployment target and runs SQLite 3.11.0 from 2016-02-15.

If CI ran against that version of SQLite this bug could have been avoided.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1280/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
842881221,MDU6SXNzdWU4NDI4ODEyMjE=,1281,Latest Datasette tags missing from Docker Hub,9599,simonw,closed,0,,,,,7,2021-03-29T00:58:30Z,2021-03-29T01:41:48Z,2021-03-29T01:41:48Z,OWNER,,"Spotted this while testing https://github.com/simonw/datasette/issues/1249#issuecomment-808998719_

https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated isn't showing the tags for any version more recent than 0.54.1 - we are up to 0.56 now.

But the `:latest` tag is for the new 0.56 release.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1281/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
843739658,MDExOlB1bGxSZXF1ZXN0NjAzMDgyMjgw,1282,Fix little typo,192568,mroswell,closed,0,,,,,2,2021-03-29T19:45:28Z,2021-03-29T19:57:34Z,2021-03-29T19:57:34Z,CONTRIBUTOR,simonw/datasette/pulls/1282,,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1282/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
843884745,MDU6SXNzdWU4NDM4ODQ3NDU=,1283,advanced #export causes unexpected scrolling,192568,mroswell,open,0,,,,,0,2021-03-29T22:46:57Z,2021-03-29T22:46:57Z,,CONTRIBUTOR,,"1. Visit a datasette table page
2. Click on the ""(advanced)"" link. This adds a fragment identifier ""#export"" to the URL, and scrolls down to the ""Advanced export"" div with the ""export"" id.
3. Manually scroll back up, and click on a suggested facet. The fragment identifier is still present, and the app scrolls back down to the ""Advanced export"" div. I think this is unwanted behavior.

The user remedy seems to be to manually remove the ""#export"" from the URL.

This behavior happens in my project, and in:
https://covid-19.datasettes.com/covid/economist_excess_deaths  (for instance) 
but not in this table: 
https://global-power-plants.datasettes.com/global-power-plants/global-power-plants",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1283/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
845794436,MDU6SXNzdWU4NDU3OTQ0MzY=,1284,Feature or Documentation Request: Individual table as home page template,192568,mroswell,open,0,,,,,4,2021-03-31T03:56:17Z,2021-11-04T03:15:01Z,,CONTRIBUTOR,,"It would be great to have a sample showing how to move a single database that has a single table, to the index page. I'm trying it now, and find there is a real depth of Datasette and Python understanding that's required to be successful. 

I've got all the basic jinja concepts down... variables, template control structures, template inheritance, template overrides, css, html, the --template-dir and --static arguments, etc. 

But copying the table.html file to index.html doesn't work. There are undocumented functions and filters... I can figure some of them out (yay, url_builder.py and utils/__init__.py!) but it's a slog better handled by a much stronger Python developer. 

One sample would make a world of difference. The ideal form of this documentation would be a diff between the default table.html and how that would look if essentially moved to index.html. The use case is for everyone who wants to create a public-facing website to explore a single table at the root directory. (Maybe a second bit of documentation for people who have a single database with multiple tables.)

(Hmm... might be cool to have a setting for that, where it happens automagically! If only one table, then home page is at the table level. if only one database, then home page is at the database level.... as an option.)

I suppose I could ignore this, and somehow do this in the DNS settings once I hook up Vercel to a domain name, maybe.. and remove the breadcrumbs in table.html... but for now, a documentation request in the form of a diff... for viewing a single table (or a single database) at the root.

(Actually, there's probably room for a whole expanded section on templates. Noticed some nice table metadata in one of the datasette examples, for instance...  Hmm... maybe a whole library of solutions in one place... maybe a documentation hackathon! If that's of interest, of course it's a separate issue. )

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1284/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
847700726,MDU6SXNzdWU4NDc3MDA3MjY=,1285,Feature Request or Plugin Request: Numeric Range Facets,192568,mroswell,open,0,,,,,0,2021-04-01T01:50:20Z,2021-04-01T02:28:19Z,,CONTRIBUTOR,,"It would be great to offer facets for numeric data ranges. 

The ranges could pull from typical GIS methods of creating choropleth maps. 
https://gisgeography.com/choropleth-maps-data-classification/
Of the following, for mapping, I've always preferred a Jenks Natural Breaks, or a cross between Jenks and Pretty breaks.

- Equal Intervals 
- Quantile (equal count) 
- Standard Deviation
- Natural Breaks (Jenks) Classification 
- Pretty Breaks
- Some sort of Aggregate Jenks Classification (this isn't standard, but it would be nice to be able to set classification ranges that work across tables.)

Here are some links for Natural Breaks, in case this method is unfamiliar.

- https://en.wikipedia.org/wiki/Jenks_natural_breaks_optimization
- http://wiki.gis.com/wiki/index.php/Jenks_Natural_Breaks_Classification
- https://medium.com/analytics-vidhya/jenks-natural-breaks-best-range-finder-algorithm-8d1907192051

Per that last link, there is a Jenks Python module...  They also describe it as data-intensive for larger datasets. Maybe this is a good plugin idea.

An example of equal Intervals would be 
0 – < 10
10 – < 20
20 – < 30
30 – < 40

It's kind of confusing to have that less-than sign in there. it could also be displayed as:
0 – 10
10 – 20
20 – 30
30 – 40

But then it's not completely clear which category 10 is in, for instance.

(Best to right-justify.. and use an ""en dash"" between numbers.)

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1285/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
849220154,MDU6SXNzdWU4NDkyMjAxNTQ=,1286,Better default display of arrays of items,192568,mroswell,open,0,,,,,5,2021-04-02T13:31:40Z,2021-06-12T12:36:15Z,,CONTRIBUTOR,,"Would be great to have template filters that convert array fields to bullets and/or delimited lists upon table display:
```
|to_bullets
|to_comma_delimited
|to_semicolon_delimited
```
or maybe: 
```
|join_array(""bullet"")
|join_array(""bullet"",""square"")
|join_array("";"")
|join_array("","")
```
Keeping in mind that bullets show up in html as \<li\> while other delimiting characters appear after the value.

Of course, the fields themselves would remain as facetable arrays. ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1286/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
849396758,MDU6SXNzdWU4NDkzOTY3NTg=,1287,Upgrade to Python 3.9.4,9599,simonw,open,0,,,,,5,2021-04-02T18:43:15Z,2021-04-03T22:38:39Z,,OWNER,,Has some security fixes https://pythoninsider.blogspot.com/2021/04/python-393-and-389-are-now-available.html,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1287/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
849512840,MDU6SXNzdWU4NDk1MTI4NDA=,1288,Facets: show counts for null,1111743,jungle-boogie,open,0,,,,,0,2021-04-02T22:33:44Z,2021-04-02T22:33:44Z,,NONE,,"Hi,

Thank you for Datasette and being a fan of SQLite!

Not all rows in a record will always contain data.
So when using a facet on a column where some records have data and others don't, you don't get an accurate count of the results.

Please consider also counting and showing null records with facets.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1288/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
849543502,MDU6SXNzdWU4NDk1NDM1MDI=,1289,Speed up tests with pytest-xdist,9599,simonw,closed,0,,,,,3,2021-04-03T00:47:39Z,2021-04-03T03:42:28Z,2021-04-03T03:42:28Z,OWNER,,"I think I can get this working for almost every test, then use the pattern in https://github.com/pytest-dev/pytest-xdist/issues/385#issuecomment-444545641 to opt specific tests out of being run in parallel.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1289/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275164558,MDU6SXNzdWUyNzUxNjQ1NTg=,129,Hide FTS-created tables by default on the database index page,9599,simonw,closed,0,,,,,2,2017-11-19T14:50:42Z,2017-11-22T20:22:02Z,2017-11-22T20:19:04Z,OWNER,,"SQLite databases that use FTS include a number of automatically generated tables, e.g.:

https://sf-trees-search.now.sh/sf-trees-search-a899b92

<img width=""730"" alt=""sf-trees-search_and_sf-trees-search"" src=""https://user-images.githubusercontent.com/9599/32991960-bf05abee-ccf9-11e7-8bd1-54dcde4ca491.png"">

Of these, only the `Street_Tree_List` table is actually relevant to the user.

We can detect which tables are FTS tables by first finding the virtual tables:

    sqlite> .headers on
    sqlite> select * from sqlite_master where rootpage = 0;
    type|name|tbl_name|rootpage|sql
    table|Search|Search|0|CREATE VIRTUAL TABLE ""Street_Tree_List_fts"" USING FTS4 (""qAddress"", ""qCaretaker"", ""qSpecies"")

Then parsing the above to figure out which ones are USING FTS? - then assume that any table which starts with that `Street_Tree_List_fts` prefix was created to support search:

    sqlite> select * from sqlite_master where type='table' and tbl_name like 'Street_Tree_List_fts%';
    type|name|tbl_name|rootpage|sql
    table|Search_content|Search_content|10355|CREATE TABLE 'Street_Tree_List_fts_content'(docid INTEGER PRIMARY KEY, 'c0qAddress', 'c1qCaretaker', 'c2qSpecies')
    table|Search_segments|Search_segments|10356|CREATE TABLE 'Street_Tree_List_fts_segments'(blockid INTEGER PRIMARY KEY, block BLOB)
    table|Search_segdir|Search_segdir|10357|CREATE TABLE 'Street_Tree_List_fts_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx))
    table|Search_docsize|Search_docsize|10359|CREATE TABLE 'Street_Tree_List_fts_docsize'(docid INTEGER PRIMARY KEY, size BLOB)
    table|Search_stat|Search_stat|10360|CREATE TABLE 'Street_Tree_List_fts_stat'(id INTEGER PRIMARY KEY, value BLOB)

We won't hide these completely - instead, we'll default the database index view to not showing them with a message that says ""5 hidden tables"" and support ?_hidden=1 to display them.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/129/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
849568079,MDExOlB1bGxSZXF1ZXN0NjA4MzIzMDI4,1290,Use pytest-xdist to speed up tests,9599,simonw,closed,0,,,,,1,2021-04-03T03:34:36Z,2021-04-03T03:42:29Z,2021-04-03T03:42:28Z,OWNER,simonw/datasette/pulls/1290,"Closes #1289, refs #1212.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1290/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
849582643,MDExOlB1bGxSZXF1ZXN0NjA4MzM0MDk2,1291,Update docs: explain allow_download setting,5413548,louispotok,closed,0,,,,,2,2021-04-03T05:28:33Z,2021-06-05T19:48:51Z,2021-06-05T19:48:51Z,CONTRIBUTOR,simonw/datasette/pulls/1291,"This fixes one possible source of confusion seen in #502 and clarifies
when database downloads will be shown and allowed.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1291/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
849975810,MDU6SXNzdWU4NDk5NzU4MTA=,1292,Research ctypes.util.find_library('spatialite'),9599,simonw,open,0,,,,,1,2021-04-04T22:36:59Z,2022-01-20T21:28:50Z,,OWNER,,"Spotted this in the Django SpatiaLite backend: https://github.com/django/django/blob/8f6a7a0e9e7c5404af6520ae606927e32415eb00/django/contrib/gis/db/backends/spatialite/base.py#L24-L36

```python
ctypes.util.find_library('spatialite')
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1292/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
849978964,MDU6SXNzdWU4NDk5Nzg5NjQ=,1293,Show column metadata plus links for foreign keys on arbitrary query results,9599,simonw,open,0,,,,,51,2021-04-04T22:59:42Z,2022-09-02T17:34:09Z,,OWNER,,"Related to #620. It would be _really_ cool if Datasette could magically detect the source of the data displayed in an arbitrary query and, if that data represents a foreign key, display it as a hyperlink.

Compare https://latest.datasette.io/fixtures/facetable

<img width=""1202"" alt=""fixtures__facetable__15_rows"" src=""https://user-images.githubusercontent.com/9599/113523748-a7d2b300-955e-11eb-904d-8cf2639b0b5f.png"">

To https://latest.datasette.io/fixtures?sql=select+pk%2C+created%2C+planet_int%2C+on_earth%2C+state%2C+city_id%2C+neighborhood%2C+tags%2C+complex_array%2C+distinct_some_null+from+facetable+order+by+pk+limit+101

<img width=""998"" alt=""fixtures__select_pk__created__planet_int__on_earth__state__city_id__neighborhood__tags__complex_array__distinct_some_null_from_facetable_order_by_pk_limit_101"" src=""https://user-images.githubusercontent.com/9599/113523761-be790a00-955e-11eb-8c82-36b9d0226bc8.png"">
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1293/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,reopened
853672224,MDU6SXNzdWU4NTM2NzIyMjQ=,1294,"""You can check out any time you like. But you can never leave!""",192568,mroswell,open,0,,,,,0,2021-04-08T17:02:15Z,2021-04-08T18:35:50Z,,CONTRIBUTOR,,"(Feel free to rename this one.)

- The column gear lets you ""Show not-blank rows."" Then it places a parameter in the URL, which a web developer would notice, but a lot of users won't notice, or know to delete it. Would be good to toggle ""Show not-blank rows"" with ""Show all rows."" (Also would be quite helpful to have a ""Show blank rows | Show all rows"" option)
- The column gear lets you ""Sort ascending"" and ""Sort descending"" but then you're stuck with some sort of sorted version thereafter, unless you know to sort the ID column, or to remove the full _sort parameter and its value in the URL. Would be good to offer a ""Remove sort"" option in the gear.
- These requests are in the same camp as: https://github.com/simonw/datasette-vega/issues/36
- I suspect there are other url parameter instances where similar analysis would be helpful, but the three above are the use cases I've run across. 

UPDATE:
- It would be helpful to have a ""Previous page"" available for all but the first table page.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1294/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
855296937,MDU6SXNzdWU4NTUyOTY5Mzc=,1295,Errors should have links to further information,9599,simonw,open,0,,,,,2,2021-04-11T12:39:12Z,2022-12-14T23:28:49Z,,OWNER,,"Inspired by this tweet:
https://twitter.com/willmcgugan/status/1381186384510255104

> While I am thinking about faqs. I’d also like to add short URLs to Rich exceptions.
>
> I loath cryptic error messages, and I’ve created a fair few myself. In Rich I’ve tried to make them as plain English as possible. But...
>
>  would be great if every error message linked to a page that explains the error in detail and offers fixes.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1295/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
855446829,MDExOlB1bGxSZXF1ZXN0NjEzMTc4OTY4,1296,Dockerfile: use Ubuntu 20.10 as base,82332573,tmcl-it,open,0,,,,,4,2021-04-12T00:23:32Z,2021-07-20T08:52:13Z,,FIRST_TIME_CONTRIBUTOR,simonw/datasette/pulls/1296,"This PR changes the main Dockerfile to use ubuntu:20.10 as base image instead of python:3.9.2-slim-buster (itself based on debian:buster-slim).

The Dockerfile is essentially the one from https://github.com/simonw/datasette/issues/1249#issuecomment-803698983 with some additional cleanups to slim it down.

This fixes a couple of issues:
1. The SQLite version in Debian Buster (2.6.0) doesn't support generated columns
2. Installing SpatiaLite from the Debian sid repositories has the side effect of also installing updates to libc and libstdc++ from sid.

As a bonus, the Docker image becomes smaller:


```
$ docker image ls
REPOSITORY                   TAG           IMAGE ID       CREATED       SIZE
datasette                    0.56-ubuntu   f7aca255140a   5 hours ago   212MB
datasetteproject/datasette   0.56          efb3b282f390   13 days ago   258MB
```

### Reproduction of the first issue

```
$ curl -O https://latest.datasette.io/fixtures.db
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  260k    0  260k    0     0   489k      0 --:--:-- --:--:-- --:--:--  489k

$ docker run -v `pwd`:/mnt datasetteproject/datasette:0.56 datasette /mnt/fixtures.db
Traceback (most recent call last):
  File ""/usr/local/bin/datasette"", line 8, in <module>
    sys.exit(cli())
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/usr/local/lib/python3.9/site-packages/datasette/cli.py"", line 544, in serve
    asyncio.get_event_loop().run_until_complete(check_databases(ds))
  File ""/usr/local/lib/python3.9/asyncio/base_events.py"", line 642, in run_until_complete
    return future.result()
  File ""/usr/local/lib/python3.9/site-packages/datasette/cli.py"", line 584, in check_databases
    await database.execute_fn(check_connection)
  File ""/usr/local/lib/python3.9/site-packages/datasette/database.py"", line 155, in execute_fn
    return await asyncio.get_event_loop().run_in_executor(
  File ""/usr/local/lib/python3.9/concurrent/futures/thread.py"", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File ""/usr/local/lib/python3.9/site-packages/datasette/database.py"", line 153, in in_thread
    return fn(conn)
  File ""/usr/local/lib/python3.9/site-packages/datasette/utils/__init__.py"", line 892, in check_connection
    for r in conn.execute(
sqlite3.DatabaseError: malformed database schema (generated_columns) - near ""AS"": syntax error
```

Here is the SQLite version:

```
$ docker run -v `pwd`:/mnt -it datasetteproject/datasette:0.56 /bin/bash
root@d9220d3b95dd:/# python3
Python 3.9.2 (default, Mar 27 2021, 02:50:26) 
[GCC 8.3.0] on linux
Type ""help"", ""copyright"", ""credits"" or ""license"" for more information.
>>> import sqlite3
>>> sqlite3.version
'2.6.0'
```

### Reproduction of the second issue

```
$ docker build . -t datasette --build-arg VERSION=0.55
[...snip...]
The following packages will be upgraded:
  libc-bin libc6 libstdc++6
[...snip...]
Unpacking libc6:amd64 (2.31-11) over (2.28-10) ...
[...snip...]
Unpacking libstdc++6:amd64 (10.2.1-6) over (8.3.0-6) ...
[...snip...]
```

Both libc and libstdc++ are backwards compatible, so the image still works, but it will result in a combination of libraries and Python versions that exists only in the Datasette image, so it's likely untested. In addition, since Debian sid is an always-changing rolling-release, the versions of libc, libstdc++, Spatialite, and their dependencies change frequently, so the library versions in the Datasette image will depend on the day when it was built.
",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1296/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
855451460,MDU6SXNzdWU4NTU0NTE0NjA=,1297,"Documentation: json1, and introspection endpoints",192568,mroswell,open,0,,,,,0,2021-04-12T00:38:00Z,2021-04-12T01:29:33Z,,CONTRIBUTOR,,"https://docs.datasette.io/en/stable/facets.html notes that:
> If your SQLite installation provides the json1 extension (you can check using /-/versions) Datasette will automatically detect columns that contain JSON arrays...

When I check -/versions I see two sections relevant to json1:
```
        ""extensions"": {
            ""json1"": null
        },
        ""compile_options"": [
            ...
            ""ENABLE_JSON1"",
```
        
   The ENABLE_JSON1 makes me think json1 is likely available. But the  `""json1"": null` made me think it wasn't available (because of the `null`). It would help if the documentation provided clarity about how to know if json1 is installed. It would also be helpful if the `/-/versions` information signalled somehow that that is to be appended to the hostname or domain name (or whatever you want to call it, or simply show it, using `example.com/-/versions` instead of `/-/versions`. Likewise on that last point, for https://docs.datasette.io/en/stable/introspection.html#introspection , at least at some point on that page detailing where those introspection endpoints go. (Sometimes documentation can be so abbreviated that it's hard for new users to figure out what's going on.)
   ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1297/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
855476501,MDU6SXNzdWU4NTU0NzY1MDE=,1298,improve table horizontal scroll experience,192568,mroswell,open,0,,,,,4,2021-04-12T01:55:16Z,2022-08-30T21:11:49Z,,CONTRIBUTOR,,"Wide tables aren't a huge problem if you know to click and drag right. But it's not at all obvious to do that. (it also tends to blue-select any content as it's dragging.) Depending on column widths, public users might entirely miss all the columns to the right. 

There is a scrollbar at the bottom of the table, but I'm displaying ALL my records because it's the only way for datasette-vega to make accurate charts. So that bottom scrollbar is likely to be missed. I wonder if some sort of javascript-y mouseover to an arrow might help, similar to those seen in image carousels. Ah: here's a perfect example:

1. Visit http://google.com
2. Search for: animals endangered
3. Note the 'g-right-button' (in the code) that looks like a right-facing caret in a circle. 
4. Click on that and the carousel scrolls right (and 'g-left-button' appears on the left).

Might be tricky to do that on a table, rather than a one-row carousel, but it's worth experimenting with.

Another option is just to put the scrollbars at the top of the table, too. 

Meantime, I'm trying to build a button like the ""View/hide all columns on https://salaries.news.baltimoresun.com/salaries-be494cf/2019+Maryland+state+salaries
Might be nice to have that available by default, with settings in the metadata showing which are on by default.

(I saw some other closed issues related to horizontal scrolling, and admit I don't entirely understand them. For instance, the animated gif at https://github.com/simonw/datasette/issues/998#issuecomment-714117534 confuses me. )
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1298/reactions"", ""total_count"": 4, ""+1"": 4, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
856895291,MDU6SXNzdWU4NTY4OTUyOTE=,1299,Design better empty states,9599,simonw,open,0,,,,,0,2021-04-13T12:06:12Z,2021-04-13T12:06:12Z,,OWNER,,Inspiration here: https://emptystat.es/,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1299/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
267542338,MDU6SXNzdWUyNjc1NDIzMzg=,13,Add a syntax highlighting SQL editor,9599,simonw,closed,0,,,,,1,2017-10-23T05:03:33Z,2017-11-15T02:04:51Z,2017-11-15T02:04:51Z,OWNER,,https://ace.c9.io/#nav=embedding looks like a good option,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/13/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275166078,MDU6SXNzdWUyNzUxNjYwNzg=,130,"Rename ""datasette build"" to ""datasette inspect""",9599,simonw,closed,0,,,,,0,2017-11-19T15:08:02Z,2017-12-07T16:57:58Z,2017-12-07T16:57:58Z,OWNER,,"This command introspects the databases and writes out a JSON summary.

I think I'd like to use `datasette build` for something more interesting, potentially duplicating functionality from https://github.com/simonw/csvs-to-sqlite

Since the internal method that does this is called `ds.inspect()` that seems like a reasonable replacement name for the command.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/130/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
860625833,MDU6SXNzdWU4NjA2MjU4MzM=,1300,Make row available to `render_cell` plugin hook,3243482,abdusco,closed,0,,,,,5,2021-04-18T10:14:37Z,2022-07-07T16:34:05Z,2022-07-07T16:31:22Z,CONTRIBUTOR,,"*Original title: **Generating URL for a row inside `render_cell` hook***

Hey,
I am using Datasette to view a database that contains video metadata. It has BLOB columns that contain video thumbnails in JPG format (around 100-500KB per row). 

I've registered an output formatter that extends `datasette.blob_renderer.render_blob` function and serves the column with `image/jpeg` content type.

```python
from datasette.blob_renderer import render_blob

async def render_jpg(datasette, database, rows, columns, request, table, view_name):
    response = await render_blob(datasette, database, rows, columns, request, table, view_name)
    response.content_type = ""image/jpeg""
    response.headers[""Content-Disposition""] = f'inline; filename=""image.jpg""'
    return response


@hookimpl
def register_output_renderer():
    return {
        ""extension"": ""jpg"",
        ""render"": render_jpg,
        ""can_render"": lambda: True,
    }
```

This works well. I can visit `http://localhost:8001/mydb/videos/1.jpg?_blob_column=thumbnail` and view the image.

I want to display the image directly with an `<img>` tag (lazy-loaded of course). So, I need a URL, because embedding base64 would increase the page size too much (each image > 100KB). 

Datasette generates a link with `.blob` extension for blob columns. It does this by calling `datasette.urls.row_blob`

https://github.com/simonw/datasette/blob/7a2ed9f8a119e220b66d67c7b9e07cbab47b1196/datasette/views/table.py#L169-L179

But I have no way of getting the row inside the `render_cell` hook. 

```python
@hookimpl
def render_cell(value, column, table, database, datasette):
    if isinstance(value, bytes) and imghdr.what(None, value):
        # generate url
        return '$renderedLink'
```

Any pointers?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1300/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
860722711,MDU6SXNzdWU4NjA3MjI3MTE=,1301,Publishing to cloudrun with immutable mode?,5413548,louispotok,open,0,,,,,1,2021-04-18T17:51:46Z,2022-10-07T02:38:04Z,,CONTRIBUTOR,,"I'm a bit confused about immutable mode and publishing to cloudrun. (I want to publish with immutable mode so that I can support database downloads.)

Running `datasette publish cloudrun --extra-options=""-i example.db""` leads to an error:
> Error: Invalid value for '-i' / '--immutable': Path 'example.db' does not exist. 

However, running `datasette publish cloudrun example.db` not only works but seems to publish in immutable mode anyway! I'm seeing this both with `/-/databases.json` and the fact that downloads are working.

When I just `datasette serve` locally, this succeeds both ways and works as expected.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1301/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
860734722,MDU6SXNzdWU4NjA3MzQ3MjI=,1302,Fix disappearing facets,192568,mroswell,open,0,,,,,0,2021-04-18T18:42:33Z,2021-04-20T07:40:15Z,,CONTRIBUTOR,,"1. Clone https://github.com/mroswell/list-N
2. Run `datasette disinfectants.db -o`
3. Select the `Safer_or_Toxic` facet.
4. Select `Toxic`.
5. Close out the `Safer_or_Toxic` facet.
6. Examine `Suggested facets` list. `Safer_or_Toxic` is GONE.
7. Try some other facets. When you select an element, and then close the list, in some cases, the facet properly returns to the `Suggested facet` list... Arrays and dates properly return to the list, but fields with strings don't return to the list. 

Since my site is devoted to whether disinfectants are Safer or Toxic, having the suggested facet disappear from the suggested facet list is very confusing* to end-users. This, along with a few other issues, unfortunately proved beyond my own programming ability to address. So I hired a Senior-level developer to address a number of issues, including this disappearing act.

8. Open a new terminal. Run `datasette disinfectants.db -m metadata.json --static static:static/  --template-dir templates/ --plugins-dir plugins/ -p 8001 -o`
9. Repeat steps 3-6, but this time, the Safer_or_Toxic facet returns to the list (and the related URL parameters are removed).

I'm not sure how to do a pull request for this, because the plugin contains other functionality that goes beyond this bug. I wanted the facets sorted in a certain order (both in the suggested facet list, and the detail lists) (... the detail lists were hopping around all over the place before...) I wanted the duplicate facets removed (leaving only the one where you can facet by individual item in an array.) I wanted the arrays to be presented in a prettier fashion (I did that in the template... That could be moved over to the plugin at some point)

I'm thinking it'll be very helpful if applicable parts of my project's plugin (sort_suggested_facets_plugin.py) will be able to be incorporated back into datasette, but I leave that to you to consider.

(* The disappearing facet bug was especially confusing because I'm removing the filters and sql from the table page, at the request of the organization. The filters and sql detail created a lot of confusion for end users who try to find disinfectants used by Hospitals, for instance, as an '=' won't find them, since they are part of the Use_site array.) My disappearing-facet confusion was documented in my own issue: https://github.com/mroswell/list-N/issues/57 (addressed by the plugin). Other facet-related issues here: https://github.com/mroswell/list-N/issues/54 (addressed by the plugin); https://github.com/mroswell/list-N/issues/15 (addressed by template); https://github.com/mroswell/list-N/issues/53 (not yet addressed). 
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1302/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
861331159,MDExOlB1bGxSZXF1ZXN0NjE4MDExOTc3,1303,"Update pytest-asyncio requirement from <0.15,>=0.10 to >=0.10,<0.16",27856297,dependabot-preview[bot],closed,0,,,,,1,2021-04-19T13:49:12Z,2021-04-19T18:18:17Z,2021-04-19T18:18:17Z,CONTRIBUTOR,simonw/datasette/pulls/1303,"Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version.
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/7aa574af1830665be23b983407fdff8f5fcf3ccc""><code>7aa574a</code></a> 0.15.0</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/dadff91937e03c9cc08b290d14c8bc72aa793f20""><code>dadff91</code></a> Remove poetry</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/d4b79ea6481cb498a2fd3feb8a05f48249559d55""><code>d4b79ea</code></a> Lint</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/22d91b2a6df1b3557cfc0370736d7cad4ba8bef4""><code>22d91b2</code></a> Black linting</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/5c47b78ebfab7a42a9450b88e4a13bf5b135e228""><code>5c47b78</code></a> Covecov</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/22e338a4dbf67a08146757cc3a2320b7e88c7eae""><code>22e338a</code></a> Fix CI</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/70cbdeff9f9b04635714b2347ebdbea71acd93a6""><code>70cbdef</code></a> Travis -&gt; GitHub Actions</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/740af1843f1dafb47ba6db236c1e354731993949""><code>740af18</code></a> Close event loops when replacing them</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/a5161343442fb64b7fcb91dfc5841b9f79698989""><code>a516134</code></a> plugin: Set unused_tcp_port_factory scope to 'session'</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/1bb7f300f4f5c31d47675ed018e9a46ecd4d8496""><code>1bb7f30</code></a> refactor: Removed the &quot;_35&quot; and &quot;_36&quot; suffixes from test modules.</li>
<li>Additional commits viewable in <a href=""https://github.com/pytest-dev/pytest-asyncio/compare/v0.10.0...v0.15.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a ""Dependabot enabled"" badge to your readme

Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)



</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1303/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
863884805,MDU6SXNzdWU4NjM4ODQ4MDU=,1304,"Document how to send multiple values for ""Named parameters"" ",9308268,rayvoelker,open,0,,,,,4,2021-04-21T13:19:06Z,2021-12-08T03:23:14Z,,NONE,,"https://docs.datasette.io/en/stable/sql_queries.html#named-parameters

I thought that I had seen an example of how to do this example below, but I can't seem to find it

```sql
select
  *
from
  bib
where
  bib.bib_record_num in (1008088,1008092)
```

```sql
select
  *
from
  bib
where
  bib.bib_record_num in (:bib_record_numbers)
```
![image](https://user-images.githubusercontent.com/9308268/115558839-2333a480-a281-11eb-85e6-ce3bada79140.png)

https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-204d100?sql=select%0D%0A++*%0D%0Afrom%0D%0A++bib%0D%0Awhere%0D%0A++bib.bib_record_num+in+%28%3Abib_record_numbers%29&bib_record_numbers=1008088%2C1008092

Or, maybe this isn't a fully supported feature.

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1304/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
864969683,MDU6SXNzdWU4NjQ5Njk2ODM=,1305,Index view crashes when any database table is not accessible to actor,416374,gfrmin,closed,0,,,,,0,2021-04-22T13:44:22Z,2021-06-02T04:26:29Z,2021-06-02T04:26:29Z,CONTRIBUTOR,,"Because of https://github.com/simonw/datasette/blob/main/datasette/views/index.py#L63, the ```tables``` dict built does not include invisible tables; however, if https://github.com/simonw/datasette/blob/main/datasette/views/index.py#L80 is reached (because table_counts was not successfully initialized, e.g. due to a very large database) then as db.get_all_foreign_keys() returns ALL tables, a KeyError will be raised.

This error can be recreated with the fixtures.db if any table is hidden, e.g. by adding something like                 ```""foreign_key_references"": {
                    ""allow"": {}
                }``` to fixtures-metadata.json and deleting ```or not table_counts``` from https://github.com/simonw/datasette/blob/main/datasette/views/index.py#L77.

I'm not sure how to fix this error; perhaps by testing if the table is in the aforementions ```tables``` dict.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1305/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
864979486,MDExOlB1bGxSZXF1ZXN0NjIxMTE3OTc4,1306,Avoid error sorting by relationships if related tables are not allowed,416374,gfrmin,closed,0,,,,,4,2021-04-22T13:53:17Z,2021-06-02T04:27:00Z,2021-06-02T04:25:28Z,CONTRIBUTOR,simonw/datasette/pulls/1306,Refs #1305,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1306/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
866668415,MDU6SXNzdWU4NjY2Njg0MTU=,1308,"Columns named ""link"" display in bold",9599,simonw,closed,0,,,,,3,2021-04-24T05:58:11Z,2021-04-24T06:07:49Z,2021-04-24T06:07:49Z,OWNER,,Reported in office hours today.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1308/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
869237023,MDExOlB1bGxSZXF1ZXN0NjI0NjM1NDQw,1309,Bump black from 20.8b1 to 21.4b0,27856297,dependabot-preview[bot],closed,0,,,,,2,2021-04-27T20:28:11Z,2021-04-28T18:26:06Z,2021-04-28T18:26:04Z,CONTRIBUTOR,simonw/datasette/pulls/1309,"Bumps [black](https://github.com/psf/black) from 20.8b1 to 21.4b0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.4b0</h2>
<h4><em>Black</em></h4>
<ul>
<li>
<p>Fixed a rare but annoying formatting instability created by the combination of
optional trailing commas inserted by <code>Black</code> and optional parentheses looking at
pre-existing &quot;magic&quot; trailing commas. This fixes issue <a href=""https://github-redirect.dependabot.com/psf/black/issues/1629"">#1629</a> and all of its many many
duplicates. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2126"">#2126</a>)</p>
</li>
<li>
<p><code>Black</code> now processes one-line docstrings by stripping leading and trailing spaces,
and adding a padding space when needed to break up &quot;&quot;&quot;&quot;. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1740"">#1740</a>)</p>
</li>
<li>
<p><code>Black</code> now cleans up leading non-breaking spaces in comments (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2092"">#2092</a>)</p>
</li>
<li>
<p><code>Black</code> now respects <code>--skip-string-normalization</code> when normalizing multiline
docstring quotes (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1637"">#1637</a>)</p>
</li>
<li>
<p><code>Black</code> no longer removes all empty lines between non-function code and decorators
when formatting typing stubs. Now <code>Black</code> enforces a single empty line. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1646"">#1646</a>)</p>
</li>
<li>
<p><code>Black</code> no longer adds an incorrect space after a parenthesized assignment expression
in if/while statements (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1655"">#1655</a>)</p>
</li>
<li>
<p>Added <code>--skip-magic-trailing-comma</code> / <code>-C</code> to avoid using trailing commas as a reason
to split lines (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1824"">#1824</a>)</p>
</li>
<li>
<p>fixed a crash when PWD=/ on POSIX (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1631"">#1631</a>)</p>
</li>
<li>
<p>fixed &quot;I/O operation on closed file&quot; when using --diff (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1664"">#1664</a>)</p>
</li>
<li>
<p>Prevent coloured diff output being interleaved with multiple files (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1673"">#1673</a>)</p>
</li>
<li>
<p>Added support for PEP 614 relaxed decorator syntax on python 3.9 (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1711"">#1711</a>)</p>
</li>
<li>
<p>Added parsing support for unparenthesized tuples and yield expressions in annotated
assignments (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1835"">#1835</a>)</p>
</li>
<li>
<p>use lowercase hex strings (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1692"">#1692</a>)</p>
</li>
<li>
<p>added <code>--extend-exclude</code> argument (PR <a href=""https://github-redirect.dependabot.com/psf/black/issues/2005"">#2005</a>)</p>
</li>
<li>
<p>speed up caching by avoiding pathlib (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1950"">#1950</a>)</p>
</li>
<li>
<p><code>--diff</code> correctly indicates when a file doesn't end in a newline (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1662"">#1662</a>)</p>
</li>
<li>
<p>Added <code>--stdin-filename</code> argument to allow stdin to respect <code>--force-exclude</code> rules
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/1780"">#1780</a>)</p>
</li>
<li>
<p>Lines ending with <code>fmt: skip</code> will now be not formatted (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1800"">#1800</a>)</p>
</li>
<li>
<p>PR <a href=""https://github-redirect.dependabot.com/psf/black/issues/2053"">#2053</a>: Black no longer relies on typed-ast for Python 3.8 and higher</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/master/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h3>21.4b0</h3>
<h4><em>Black</em></h4>
<ul>
<li>
<p>Fixed a rare but annoying formatting instability created by the combination of
optional trailing commas inserted by <code>Black</code> and optional parentheses looking at
pre-existing &quot;magic&quot; trailing commas. This fixes issue <a href=""https://github-redirect.dependabot.com/psf/black/issues/1629"">#1629</a> and all of its many many
duplicates. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2126"">#2126</a>)</p>
</li>
<li>
<p><code>Black</code> now processes one-line docstrings by stripping leading and trailing spaces,
and adding a padding space when needed to break up &quot;&quot;&quot;&quot;. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1740"">#1740</a>)</p>
</li>
<li>
<p><code>Black</code> now cleans up leading non-breaking spaces in comments (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2092"">#2092</a>)</p>
</li>
<li>
<p><code>Black</code> now respects <code>--skip-string-normalization</code> when normalizing multiline
docstring quotes (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1637"">#1637</a>)</p>
</li>
<li>
<p><code>Black</code> no longer removes all empty lines between non-function code and decorators
when formatting typing stubs. Now <code>Black</code> enforces a single empty line. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1646"">#1646</a>)</p>
</li>
<li>
<p><code>Black</code> no longer adds an incorrect space after a parenthesized assignment expression
in if/while statements (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1655"">#1655</a>)</p>
</li>
<li>
<p>Added <code>--skip-magic-trailing-comma</code> / <code>-C</code> to avoid using trailing commas as a reason
to split lines (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1824"">#1824</a>)</p>
</li>
<li>
<p>fixed a crash when PWD=/ on POSIX (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1631"">#1631</a>)</p>
</li>
<li>
<p>fixed &quot;I/O operation on closed file&quot; when using --diff (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1664"">#1664</a>)</p>
</li>
<li>
<p>Prevent coloured diff output being interleaved with multiple files (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1673"">#1673</a>)</p>
</li>
<li>
<p>Added support for PEP 614 relaxed decorator syntax on python 3.9 (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1711"">#1711</a>)</p>
</li>
<li>
<p>Added parsing support for unparenthesized tuples and yield expressions in annotated
assignments (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1835"">#1835</a>)</p>
</li>
<li>
<p>added <code>--extend-exclude</code> argument (PR <a href=""https://github-redirect.dependabot.com/psf/black/issues/2005"">#2005</a>)</p>
</li>
<li>
<p>speed up caching by avoiding pathlib (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1950"">#1950</a>)</p>
</li>
<li>
<p><code>--diff</code> correctly indicates when a file doesn't end in a newline (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1662"">#1662</a>)</p>
</li>
<li>
<p>Added <code>--stdin-filename</code> argument to allow stdin to respect <code>--force-exclude</code> rules
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/1780"">#1780</a>)</p>
</li>
<li>
<p>Lines ending with <code>fmt: skip</code> will now be not formatted (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1800"">#1800</a>)</p>
</li>
<li>
<p>PR <a href=""https://github-redirect.dependabot.com/psf/black/issues/2053"">#2053</a>: Black no longer relies on typed-ast for Python 3.8 and higher</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://api.dependabot.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=20.8b1&new-version=21.4b0)](https://dependabot.com/compatibility-score/?dependency-name=black&package-manager=pip&previous-version=20.8b1&new-version=21.4b0)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a ""Dependabot enabled"" badge to your readme

Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)



</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1309/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
275166669,MDU6SXNzdWUyNzUxNjY2Njk=,131,UI support for running FTS searches,9599,simonw,closed,0,,,,,3,2017-11-19T15:16:20Z,2017-11-19T17:18:05Z,2017-11-19T17:00:12Z,OWNER,,"Here's an example query that searches all FTS indexed columns in a table: https://sf-trees-search.now.sh/sf-trees-search-a899b92?sql=select+*+from+Street_Tree_List+where+rowid+in+%28select+rowid+from+Street_Tree_List_fts+where+Street_Tree_List_fts+match+%27grove+london+dpw%27%29%0D%0A

And here's a query that searches a specific column: https://sf-trees-search.now.sh/sf-trees-search-a899b92?sql=select+*+from+Street_Tree_List+where+rowid+in+%28select+rowid+from+Street_Tree_List_fts+where+qSpecies+match+%27london%27%29%0D%0A

If we detect that a table has FTS enabled (which we can do by looking for it as a content table reference in another FTS table's create definition) we should add a search box to the table page which constructs this query - maybe using `?_search=XXX` in the query string?

<s>To support search against specified columns, we can do `?_search__ qSpecies=London`.</s> - not necessary, see comment below.

- [x] Detect if a table has a FTS index defined against it as a content= parameter
- [x] Decide what to do if there is more than one FTS index (maybe just pick the first one?)
- [x] Add the `?_search=` query string argument
- [x] Add the UI",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/131/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
870125126,MDU6SXNzdWU4NzAxMjUxMjY=,1310,I'm creating a plugin to export a spreadsheet file (.ods or .xlsx),3747136,ColinMaudry,closed,0,,,,,2,2021-04-28T16:20:11Z,2021-04-30T07:26:11Z,2021-04-30T06:58:46Z,NONE,,"Hi,

I have started developing a plugin to export records as a spreadsheet file. It could be ods or xlsx, whatever is easier.

I have spotted the following packages:

- ods files: https://pypi.org/project/odswriter/
- xlsx files: https://openpyxl.readthedocs.io/en/stable/index.html (quite powerful) or https://xlsxwriter.readthedocs.io/ (faster)

This is the code I have so far, I test it with the `--plugins-dir` option:

```python
from datasette import hookimpl
from datasette.utils.asgi import Response
import odswriter as ods

def render_spreadsheet(rows):
    with ods.writer(open(""test.ods"",""wb"")) as odsfile:
        for row in rows:
            odsfile.writerow([""String"", ""ABCDEF123456"", ""123456""])
        return Response(odsfile, content_type=""application/vnd.oasis.opendocument.spreadsheet"", status=200)


@hookimpl
def register_output_renderer():
    return {""extension"": ""ods"", ""render"": render_spreadsheet}

``` 

I get the following error:

```
Traceback (most recent call last):
  File ""/home/colin/.local/lib/python3.8/site-packages/datasette/app.py"", line 1128, in route_path
    await response.asgi_send(send)
  File ""/home/colin/.local/lib/python3.8/site-packages/datasette/utils/asgi.py"", line 339, in asgi_send
    body = body.encode(""utf-8"")
AttributeError: 'ODSWriter' object has no attribute 'encode'
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File ""/home/colin/.local/lib/python3.8/site-packages/datasette/app.py"", line 1128, in route_path
    await response.asgi_send(send)
  File ""/home/colin/.local/lib/python3.8/site-packages/datasette/utils/asgi.py"", line 339, in asgi_send
    body = body.encode(""utf-8"")
AttributeError: 'ODSWriter' object has no attribute 'encode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ""/home/colin/.local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py"", line 396, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File ""/home/colin/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py"", line 45, in __call__
    return await self.app(scope, receive, send)
  File ""/home/colin/.local/lib/python3.8/site-packages/datasette/utils/asgi.py"", line 161, in __call__
    await self.app(scope, receive, send)
  File ""/home/colin/.local/lib/python3.8/site-packages/datasette/tracer.py"", line 75, in __call__
    await self.app(scope, receive, send)
  File ""/home/colin/.local/lib/python3.8/site-packages/asgi_csrf.py"", line 107, in app_wrapped_with_csrf
    await app(scope, receive, wrapped_send)
  File ""/home/colin/.local/lib/python3.8/site-packages/datasette/app.py"", line 1086, in __call__
    return await self.route_path(scope, receive, send, path)
  File ""/home/colin/.local/lib/python3.8/site-packages/datasette/app.py"", line 1133, in route_path
    return await self.handle_500(request, send, exception)
  File ""/home/colin/.local/lib/python3.8/site-packages/datasette/app.py"", line 1267, in handle_500
    await asgi_send_html(
  File ""/home/colin/.local/lib/python3.8/site-packages/datasette/utils/asgi.py"", line 217, in asgi_send_html
    await asgi_send(
  File ""/home/colin/.local/lib/python3.8/site-packages/datasette/utils/asgi.py"", line 237, in asgi_send
    await asgi_start(send, status, headers, content_type)
  File ""/home/colin/.local/lib/python3.8/site-packages/datasette/utils/asgi.py"", line 246, in asgi_start
    await send(
  File ""/home/colin/.local/lib/python3.8/site-packages/asgi_csrf.py"", line 103, in wrapped_send
    await send(event)
  File ""/home/colin/.local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py"", line 482, in send
    raise RuntimeError(msg % message_type)
RuntimeError: Expected ASGI message 'http.response.body', but got 'http.response.start'.
```

I tried with `AsgiFileDownload` like in [DatabaseDownload](https://github.com/simonw/datasette/blob/main/datasette/views/database.py#L150) to deal with the binary nature of the ods file, but the renderer expects a Response:

> <datasette.utils.asgi.AsgiFileDownload object at 0x7f5b9bd503d0> should be dict or Response

However, the `Response` class only supports the following methods, not binary:

- html
- text
- json
- redirect

How would you suggest me to proceed to have my ods file downloaded?

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1310/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
870227815,MDExOlB1bGxSZXF1ZXN0NjI1NDU3NTc5,1311,Bump black from 20.8b1 to 21.4b1,27856297,dependabot-preview[bot],closed,0,,,,,2,2021-04-28T18:25:58Z,2021-04-29T13:58:11Z,2021-04-29T13:58:09Z,CONTRIBUTOR,simonw/datasette/pulls/1311,"Bumps [black](https://github.com/psf/black) from 20.8b1 to 21.4b1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.4b1</h2>
<h4><em>Black</em></h4>
<ul>
<li>
<p>Fix crash on docstrings ending with &quot;\ &quot;. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2142"">#2142</a>)</p>
</li>
<li>
<p>Fix crash when atypical whitespace is cleaned out of dostrings (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2120"">#2120</a>)</p>
</li>
<li>
<p>Reflect the <code>--skip-magic-trailing-comma</code> and <code>--experimental-string-processing</code> flags
in the name of the cache file. Without this fix, changes in these flags would not take
effect if the cache had already been populated. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2131"">#2131</a>)</p>
</li>
<li>
<p>Don't remove necessary parentheses from assignment expression containing assert /
return statements. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2143"">#2143</a>)</p>
</li>
</ul>
<h4><em>Packaging</em></h4>
<ul>
<li>Bump pathspec to &gt;= 0.8.1 to solve invalid .gitignore exclusion handling</li>
</ul>
<h2>21.4b0</h2>
<h4><em>Black</em></h4>
<ul>
<li>
<p>Fixed a rare but annoying formatting instability created by the combination of
optional trailing commas inserted by <code>Black</code> and optional parentheses looking at
pre-existing &quot;magic&quot; trailing commas. This fixes issue <a href=""https://github-redirect.dependabot.com/psf/black/issues/1629"">#1629</a> and all of its many many
duplicates. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2126"">#2126</a>)</p>
</li>
<li>
<p><code>Black</code> now processes one-line docstrings by stripping leading and trailing spaces,
and adding a padding space when needed to break up &quot;&quot;&quot;&quot;. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1740"">#1740</a>)</p>
</li>
<li>
<p><code>Black</code> now cleans up leading non-breaking spaces in comments (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2092"">#2092</a>)</p>
</li>
<li>
<p><code>Black</code> now respects <code>--skip-string-normalization</code> when normalizing multiline
docstring quotes (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1637"">#1637</a>)</p>
</li>
<li>
<p><code>Black</code> no longer removes all empty lines between non-function code and decorators
when formatting typing stubs. Now <code>Black</code> enforces a single empty line. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1646"">#1646</a>)</p>
</li>
<li>
<p><code>Black</code> no longer adds an incorrect space after a parenthesized assignment expression
in if/while statements (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1655"">#1655</a>)</p>
</li>
<li>
<p>Added <code>--skip-magic-trailing-comma</code> / <code>-C</code> to avoid using trailing commas as a reason
to split lines (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1824"">#1824</a>)</p>
</li>
<li>
<p>fixed a crash when PWD=/ on POSIX (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1631"">#1631</a>)</p>
</li>
<li>
<p>fixed &quot;I/O operation on closed file&quot; when using --diff (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1664"">#1664</a>)</p>
</li>
<li>
<p>Prevent coloured diff output being interleaved with multiple files (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1673"">#1673</a>)</p>
</li>
<li>
<p>Added support for PEP 614 relaxed decorator syntax on python 3.9 (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1711"">#1711</a>)</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/master/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h3>21.4b1</h3>
<h4><em>Black</em></h4>
<ul>
<li>
<p>Fix crash on docstrings ending with &quot;\ &quot;. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2142"">#2142</a>)</p>
</li>
<li>
<p>Fix crash when atypical whitespace is cleaned out of dostrings (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2120"">#2120</a>)</p>
</li>
<li>
<p>Reflect the <code>--skip-magic-trailing-comma</code> and <code>--experimental-string-processing</code> flags
in the name of the cache file. Without this fix, changes in these flags would not take
effect if the cache had already been populated. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2131"">#2131</a>)</p>
</li>
<li>
<p>Don't remove necessary parentheses from assignment expression containing assert /
return statements. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2143"">#2143</a>)</p>
</li>
</ul>
<h4><em>Packaging</em></h4>
<ul>
<li>Bump pathspec to &gt;= 0.8.1 to solve invalid .gitignore exclusion handling</li>
</ul>
<h3>21.4b0</h3>
<h4><em>Black</em></h4>
<ul>
<li>
<p>Fixed a rare but annoying formatting instability created by the combination of
optional trailing commas inserted by <code>Black</code> and optional parentheses looking at
pre-existing &quot;magic&quot; trailing commas. This fixes issue <a href=""https://github-redirect.dependabot.com/psf/black/issues/1629"">#1629</a> and all of its many many
duplicates. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2126"">#2126</a>)</p>
</li>
<li>
<p><code>Black</code> now processes one-line docstrings by stripping leading and trailing spaces,
and adding a padding space when needed to break up &quot;&quot;&quot;&quot;. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1740"">#1740</a>)</p>
</li>
<li>
<p><code>Black</code> now cleans up leading non-breaking spaces in comments (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2092"">#2092</a>)</p>
</li>
<li>
<p><code>Black</code> now respects <code>--skip-string-normalization</code> when normalizing multiline
docstring quotes (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1637"">#1637</a>)</p>
</li>
<li>
<p><code>Black</code> no longer removes all empty lines between non-function code and decorators
when formatting typing stubs. Now <code>Black</code> enforces a single empty line. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1646"">#1646</a>)</p>
</li>
<li>
<p><code>Black</code> no longer adds an incorrect space after a parenthesized assignment expression
in if/while statements (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1655"">#1655</a>)</p>
</li>
<li>
<p>Added <code>--skip-magic-trailing-comma</code> / <code>-C</code> to avoid using trailing commas as a reason
to split lines (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1824"">#1824</a>)</p>
</li>
<li>
<p>fixed a crash when PWD=/ on POSIX (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1631"">#1631</a>)</p>
</li>
<li>
<p>fixed &quot;I/O operation on closed file&quot; when using --diff (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1664"">#1664</a>)</p>
</li>
<li>
<p>Prevent coloured diff output being interleaved with multiple files (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1673"">#1673</a>)</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://api.dependabot.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=20.8b1&new-version=21.4b1)](https://dependabot.com/compatibility-score/?dependency-name=black&package-manager=pip&previous-version=20.8b1&new-version=21.4b1)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a ""Dependabot enabled"" badge to your readme

Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)



</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1311/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
870946764,MDU6SXNzdWU4NzA5NDY3NjQ=,1312,how to query many-to-many relationship via json API?,5268174,bram2000,open,0,,,,,0,2021-04-29T12:09:49Z,2021-04-29T12:09:49Z,,NONE,,"Hi,

Firstly thanks for Datasette, it's great!

I'm trying to use the JSON API to query data from a Datasette instance. I have a simple 3 table many-to-many relationship, like so:

`category` - list of categories
`document` - list of documents
`document_category` - join table (a category contains many documents, and a document can be a member of multiple categories)

the `document_category` table foreign keys to the other two using their respective row_ids.

Now I want to return ""all documents within category X"" but I cannot see a way to do this without executing two queries; the first to lookup the row_id of category X, and the second to join `document` with `document_category` where category ID is <id>.

I could easily write this in SQL, but this makes programmatic handling of pagination much more difficult (we'd have to dynamically modify the SQL to select the row_id and include the correct where and limit clauses).

Is there a way to achieve this using the JSON API?

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1312/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
871046111,MDExOlB1bGxSZXF1ZXN0NjI2MTMwMTM1,1313,Bump black from 20.8b1 to 21.4b2,27856297,dependabot-preview[bot],closed,0,,,,,2,2021-04-29T13:58:06Z,2021-04-29T15:47:50Z,2021-04-29T15:47:49Z,CONTRIBUTOR,simonw/datasette/pulls/1313,"Bumps [black](https://github.com/psf/black) from 20.8b1 to 21.4b2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.4b2</h2>
<h4><em>Black</em></h4>
<ul>
<li>
<p>Fix crash if the user configuration directory is inaccessible. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2158"">#2158</a>)</p>
</li>
<li>
<p>Clarify
<a href=""https://github.com/psf/black/blob/master/docs/the_black_code_style.md#pragmatism"">circumstances</a>
in which <em>Black</em> may change the AST (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2159"">#2159</a>)</p>
</li>
</ul>
<h4><em>Packaging</em></h4>
<ul>
<li>Install <code>primer.json</code> (used by <code>black-primer</code> by default) with black. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2154"">#2154</a>)</li>
</ul>
<h2>21.4b1</h2>
<h4><em>Black</em></h4>
<ul>
<li>
<p>Fix crash on docstrings ending with &quot;\ &quot;. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2142"">#2142</a>)</p>
</li>
<li>
<p>Fix crash when atypical whitespace is cleaned out of dostrings (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2120"">#2120</a>)</p>
</li>
<li>
<p>Reflect the <code>--skip-magic-trailing-comma</code> and <code>--experimental-string-processing</code> flags
in the name of the cache file. Without this fix, changes in these flags would not take
effect if the cache had already been populated. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2131"">#2131</a>)</p>
</li>
<li>
<p>Don't remove necessary parentheses from assignment expression containing assert /
return statements. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2143"">#2143</a>)</p>
</li>
</ul>
<h4><em>Packaging</em></h4>
<ul>
<li>Bump pathspec to &gt;= 0.8.1 to solve invalid .gitignore exclusion handling</li>
</ul>
<h2>21.4b0</h2>
<h4><em>Black</em></h4>
<ul>
<li>
<p>Fixed a rare but annoying formatting instability created by the combination of
optional trailing commas inserted by <code>Black</code> and optional parentheses looking at
pre-existing &quot;magic&quot; trailing commas. This fixes issue <a href=""https://github-redirect.dependabot.com/psf/black/issues/1629"">#1629</a> and all of its many many
duplicates. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2126"">#2126</a>)</p>
</li>
<li>
<p><code>Black</code> now processes one-line docstrings by stripping leading and trailing spaces,
and adding a padding space when needed to break up &quot;&quot;&quot;&quot;. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1740"">#1740</a>)</p>
</li>
<li>
<p><code>Black</code> now cleans up leading non-breaking spaces in comments (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2092"">#2092</a>)</p>
</li>
<li>
<p><code>Black</code> now respects <code>--skip-string-normalization</code> when normalizing multiline
docstring quotes (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1637"">#1637</a>)</p>
</li>
<li>
<p><code>Black</code> no longer removes all empty lines between non-function code and decorators
when formatting typing stubs. Now <code>Black</code> enforces a single empty line. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1646"">#1646</a>)</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/master/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h3>21.4b2</h3>
<h4><em>Black</em></h4>
<ul>
<li>
<p>Fix crash if the user configuration directory is inaccessible. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2158"">#2158</a>)</p>
</li>
<li>
<p>Clarify
<a href=""https://github.com/psf/black/blob/master/docs/the_black_code_style.md#pragmatism"">circumstances</a>
in which <em>Black</em> may change the AST (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2159"">#2159</a>)</p>
</li>
</ul>
<h4><em>Packaging</em></h4>
<ul>
<li>Install <code>primer.json</code> (used by <code>black-primer</code> by default) with black. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2154"">#2154</a>)</li>
</ul>
<h3>21.4b1</h3>
<h4><em>Black</em></h4>
<ul>
<li>
<p>Fix crash on docstrings ending with &quot;\ &quot;. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2142"">#2142</a>)</p>
</li>
<li>
<p>Fix crash when atypical whitespace is cleaned out of dostrings (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2120"">#2120</a>)</p>
</li>
<li>
<p>Reflect the <code>--skip-magic-trailing-comma</code> and <code>--experimental-string-processing</code> flags
in the name of the cache file. Without this fix, changes in these flags would not take
effect if the cache had already been populated. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2131"">#2131</a>)</p>
</li>
<li>
<p>Don't remove necessary parentheses from assignment expression containing assert /
return statements. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2143"">#2143</a>)</p>
</li>
</ul>
<h4><em>Packaging</em></h4>
<ul>
<li>Bump pathspec to &gt;= 0.8.1 to solve invalid .gitignore exclusion handling</li>
</ul>
<h3>21.4b0</h3>
<h4><em>Black</em></h4>
<ul>
<li>
<p>Fixed a rare but annoying formatting instability created by the combination of
optional trailing commas inserted by <code>Black</code> and optional parentheses looking at
pre-existing &quot;magic&quot; trailing commas. This fixes issue <a href=""https://github-redirect.dependabot.com/psf/black/issues/1629"">#1629</a> and all of its many many
duplicates. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2126"">#2126</a>)</p>
</li>
<li>
<p><code>Black</code> now processes one-line docstrings by stripping leading and trailing spaces,
and adding a padding space when needed to break up &quot;&quot;&quot;&quot;. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1740"">#1740</a>)</p>
</li>
<li>
<p><code>Black</code> now cleans up leading non-breaking spaces in comments (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2092"">#2092</a>)</p>
</li>
<li>
<p><code>Black</code> now respects <code>--skip-string-normalization</code> when normalizing multiline
docstring quotes (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1637"">#1637</a>)</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://api.dependabot.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=20.8b1&new-version=21.4b2)](https://dependabot.com/compatibility-score/?dependency-name=black&package-manager=pip&previous-version=20.8b1&new-version=21.4b2)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a ""Dependabot enabled"" badge to your readme

Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)



</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1313/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
871157602,MDExOlB1bGxSZXF1ZXN0NjI2MjIyNjc2,1314,Upgrade to GitHub-native Dependabot,27856297,dependabot-preview[bot],closed,0,,,,,1,2021-04-29T15:36:41Z,2021-04-29T15:47:22Z,2021-04-29T15:47:21Z,CONTRIBUTOR,simonw/datasette/pulls/1314,"_Dependabot Preview will be shut down on August 3rd, 2021. In order to keep getting Dependabot updates, please merge this PR and migrate to GitHub-native Dependabot before then._

Dependabot has been fully integrated into GitHub, so you no longer have to install and manage a separate app. This pull request migrates your configuration from Dependabot.com to a config file, using the [new syntax][new_syntax]. When merged, we'll swap out `dependabot-preview` (me) for a new `dependabot` app, and you'll be all set!

With this change, you'll now use the [Dependabot page in GitHub][dependabot_page], rather than the [Dependabot dashboard][dashboard], to monitor your version updates, and you'll configure Dependabot through the new config file rather than a UI.







If you've got any questions or feedback for us, please let us know by creating an issue in the [dependabot/dependabot-core][issues] repository.

[Learn more about migrating to GitHub-native Dependabot][learn]

Please note that regular `@dependabot` commands do not work on this pull request.

[dashboard]: https://app.dependabot.com/
[dependabot_page]: https://github.com/simonw/datasette/network/updates
[issues]: https://github.com/dependabot/dependabot-core/issues/new?assignees=%40dependabot%2Fpreview-migration-reviewers&labels=E%3A+preview-migration&template=migration-issue.md
[learn]: http://docs.github.com/code-security/supply-chain-security/upgrading-from-dependabotcom-to-github-native-dependabot
[new_syntax]: https://help.github.com/en/github/administering-a-repository/configuration-options-for-dependency-updates
[org_secrets_url]: https://github.com/settings/secrets/dependabot
[repo_secrets_url]: https://github.com/simonw/datasette/settings/secrets/dependabot
",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1314/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
871304967,MDU6SXNzdWU4NzEzMDQ5Njc=,1315,"settings.json should be picked up by ""datasette publish cloudrun""",9599,simonw,open,0,,,,,0,2021-04-29T18:16:41Z,2021-04-29T18:16:41Z,,OWNER,,,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1315/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
876431852,MDExOlB1bGxSZXF1ZXN0NjMwNTc4NzM1,1318,Bump black from 21.4b2 to 21.5b0,49699333,dependabot[bot],closed,0,,,,,2,2021-05-05T13:07:51Z,2021-05-11T13:12:32Z,2021-05-11T13:12:31Z,CONTRIBUTOR,simonw/datasette/pulls/1318,"Bumps [black](https://github.com/psf/black) from 21.4b2 to 21.5b0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.5b0</h2>
<h4><em>Black</em></h4>
<ul>
<li>Set <code>--pyi</code> mode if <code>--stdin-filename</code> ends in <code>.pyi</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2169"">#2169</a>)</li>
<li>Stop detecting target version as Python 3.9+ with pre-PEP-614 decorators that are
being called but with no arguments (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2182"">#2182</a>)</li>
</ul>
<h4><em>Black-Primer</em></h4>
<ul>
<li>Add <code>--no-diff</code> to black-primer to suppress formatting changes (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2187"">#2187</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/master/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h3>21.5b0</h3>
<h4><em>Black</em></h4>
<ul>
<li>Set <code>--pyi</code> mode if <code>--stdin-filename</code> ends in <code>.pyi</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2169"">#2169</a>)</li>
<li>Stop detecting target version as Python 3.9+ with pre-PEP-614 decorators that are
being called but with no arguments (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2182"">#2182</a>)</li>
</ul>
<h4><em>Black-Primer</em></h4>
<ul>
<li>Add <code>--no-diff</code> to black-primer to suppress formatting changes (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2187"">#2187</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.4b2&new-version=21.5b0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1318/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
881219362,MDExOlB1bGxSZXF1ZXN0NjM0ODIxMDY1,1319,Add Docker multi-arch support with Buildx,10801138,blairdrummond,closed,0,,,,,5,2021-05-08T19:35:03Z,2021-05-27T16:49:24Z,2021-05-27T16:49:24Z,CONTRIBUTOR,simonw/datasette/pulls/1319,"This adds Docker support to extra CPU architectures (like arm) using [Docker's Buildx action](https://github.com/marketplace/actions/docker-setup-buildx)

You can see [what that looks like on Dockerhub](https://hub.docker.com/r/blairdrummond/datasette/tags?page=1&ordering=last_updated)

And how it lets Datasette run on a Raspberry Pi (top is my dockerhub, bottom is upstream)

![Screenshot from 2021-05-08 15-32-25](https://user-images.githubusercontent.com/10801138/117551210-a17a9f80-b012-11eb-966b-10e1590dd4a9.png)

The workflow log [here](https://github.com/blairdrummond/datasette/runs/2535743398?check_suite_focus=true) (I subbed `blairdrummond` for datasetteproject in my branch) ",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1319/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
275175929,MDU6SXNzdWUyNzUxNzU5Mjk=,132,Row view is not currently expanding foreign keys,9599,simonw,closed,0,,,2919870,Foreign key edition,1,2017-11-19T17:24:25Z,2017-11-23T21:51:51Z,2017-11-23T21:51:30Z,OWNER,,Eg https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List/1,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/132/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
884952179,MDU6SXNzdWU4ODQ5NTIxNzk=,1320,Can't use apt-get in Dockerfile when using datasetteproj/datasette as base,2670795,brandonrobertz,closed,0,,,,,4,2021-05-10T19:37:27Z,2021-05-24T18:15:56Z,2021-05-24T18:07:08Z,CONTRIBUTOR,,"The datasette base Docker image is super convenient, but there's one problem: if any of the plugins you install require additional system dependencies (e.g., xz, git, curl) then any attempt to use apt in said Dockerfile results in an explosion:

```
$ docker-compose build
Building server
[+] Building 9.9s (7/9)
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                     0.0s
 => => transferring dockerfile: 666B                                                                                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                        0.0s
 => => transferring context: 34B                                                                                                                                                                                                                         0.0s
 => [internal] load metadata for docker.io/datasetteproject/datasette:latest                                                                                                                                                                             0.6s
 => [base 1/4] FROM docker.io/datasetteproject/datasette@sha256:2250d0fbe57b1d615a8d6df0c9d43deb9533532e00bac68854773d8ff8dcf00a                                                                                                                         0.0s
 => [internal] load build context                                                                                                                                                                                                                        1.8s
 => => transferring context: 2.44MB                                                                                                                                                                                                                      1.8s
 => CACHED [base 2/4] WORKDIR /datasette                                                                                                                                                                                                                 0.0s
 => ERROR [base 3/4] RUN apt-get update     && apt-get install --no-install-recommends -y git ssh curl xz-utils                                                                                                                                          9.2s
------
 > [base 3/4] RUN apt-get update     && apt-get install --no-install-recommends -y git ssh curl xz-utils:
#6 0.446 Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
#6 0.449 Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
#6 0.459 Get:3 http://httpredir.debian.org/debian sid InRelease [157 kB]
#6 0.784 Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
#6 0.790 Get:5 http://httpredir.debian.org/debian sid/main amd64 Packages [8626 kB]
#6 1.003 Get:6 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
#6 1.180 Get:7 http://security.debian.org/debian-security buster/updates/main amd64 Packages [286 kB]
#6 7.095 Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [10.9 kB]
#6 8.058 Fetched 17.2 MB in 8s (2243 kB/s)
#6 8.058 Reading package lists...
#6 9.166 E: flAbsPath on /var/lib/dpkg/status failed - realpath (2: No such file or directory)
#6 9.166 E: Could not open file  - open (2: No such file or directory)
#6 9.166 E: Problem opening
#6 9.166 E: The package lists or status file could not be parsed or opened.
```

The problem seems to be from completely wiping out `/var/lib/dpkg` in the upstream Dockerfile:

https://github.com/simonw/datasette/blob/1b697539f5b53cec3fe13c0f4ada13ba655c88c7/Dockerfile#L18

I've tested without removing the directory and apt works as expected.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1320/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
887241681,MDExOlB1bGxSZXF1ZXN0NjQwNDg0OTY2,1321,Bump black from 21.4b2 to 21.5b1,49699333,dependabot[bot],closed,0,,,,,1,2021-05-11T13:12:28Z,2021-05-22T23:55:39Z,2021-05-22T23:55:39Z,CONTRIBUTOR,simonw/datasette/pulls/1321,"Bumps [black](https://github.com/psf/black) from 21.4b2 to 21.5b1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.5b1</h2>
<h3><em>Black</em></h3>
<ul>
<li>Refactor <code>src/black/__init__.py</code> into many files (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2206"">#2206</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>Replaced all remaining references to the <a href=""https://github.com/psf/black/tree/main""><code>master</code></a> branch with the <a href=""https://github.com/psf/black/tree/main""><code>main</code></a> branch. Some additional changes in the source code were also made. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2210"">#2210</a>)</li>
<li>Sigificantly reorganized the documentation to make much more sense. Check them out by heading over to <a href=""https://black.readthedocs.io/en/stable/"">the stable docs on RTD</a>. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2174"">#2174</a>)</li>
</ul>
<h2>21.5b0</h2>
<h4><em>Black</em></h4>
<ul>
<li>Set <code>--pyi</code> mode if <code>--stdin-filename</code> ends in <code>.pyi</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2169"">#2169</a>)</li>
<li>Stop detecting target version as Python 3.9+ with pre-PEP-614 decorators that are
being called but with no arguments (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2182"">#2182</a>)</li>
</ul>
<h4><em>Black-Primer</em></h4>
<ul>
<li>Add <code>--no-diff</code> to black-primer to suppress formatting changes (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2187"">#2187</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>21.5b1</h2>
<h3><em>Black</em></h3>
<ul>
<li>Refactor <code>src/black/__init__.py</code> into many files (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2206"">#2206</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>Replaced all remaining references to the
<a href=""https://github.com/psf/black/tree/main""><code>master</code></a> branch with the
<a href=""https://github.com/psf/black/tree/main""><code>main</code></a> branch. Some additional changes in
the source code were also made. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2210"">#2210</a>)</li>
<li>Sigificantly reorganized the documentation to make much more sense. Check them out by
heading over to <a href=""https://black.readthedocs.io/en/stable/"">the stable docs on RTD</a>.
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/2174"">#2174</a>)</li>
</ul>
<h2>21.5b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Set <code>--pyi</code> mode if <code>--stdin-filename</code> ends in <code>.pyi</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2169"">#2169</a>)</li>
<li>Stop detecting target version as Python 3.9+ with pre-PEP-614 decorators that are
being called but with no arguments (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2182"">#2182</a>)</li>
</ul>
<h3><em>Black-Primer</em></h3>
<ul>
<li>Add <code>--no-diff</code> to black-primer to suppress formatting changes (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2187"">#2187</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.4b2&new-version=21.5b1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1321/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
890073888,MDExOlB1bGxSZXF1ZXN0NjQzMTQ5Mjcz,1323,"Update click requirement from ~=7.1.1 to >=7.1.1,<8.1.0",49699333,dependabot[bot],closed,0,,,,,1,2021-05-12T13:08:56Z,2021-05-22T23:54:48Z,2021-05-22T23:54:48Z,CONTRIBUTOR,simonw/datasette/pulls/1323,"Updates the requirements on [click](https://github.com/pallets/click) to permit the latest version.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/pallets/click/releases"">click's releases</a>.</em></p>
<blockquote>
<h2>8.0.0</h2>
<p>New major versions of all the core Pallets libraries, including Click 8.0, have been released! :tada:</p>
<ul>
<li>Read the announcement on our blog: <a href=""https://palletsprojects.com/blog/flask-2-0-released/"">https://palletsprojects.com/blog/flask-2-0-released/</a></li>
<li>Read the full list of changes: <a href=""https://click.palletsprojects.com/changes/#version-8-0-0"">https://click.palletsprojects.com/changes/#version-8-0-0</a></li>
<li>Retweet the announcement on Twitter: <a href=""https://twitter.com/PalletsTeam/status/1392266507296514048"">https://twitter.com/PalletsTeam/status/1392266507296514048</a></li>
<li>Follow our blog, Twitter, or GitHub to see future announcements.</li>
</ul>
<p>This represents a significant amount of work, and there are quite a few changes. Be sure to carefully read the changelog, and use tools such as pip-compile and Dependabot to pin your dependencies and control your updates.</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pallets/click/blob/main/CHANGES.rst"">click's changelog</a>.</em></p>
<blockquote>
<h2>Version 8.0.0</h2>
<p>Released 2021-05-11</p>
<ul>
<li>Drop support for Python 2 and 3.5.</li>
<li>Colorama is always installed on Windows in order to provide style
and color support. :pr:<code>1784</code></li>
<li>Adds a repr to Command, showing the command name for friendlier
debugging. :issue:<code>1267</code>, :pr:<code>1295</code></li>
<li>Add support for distinguishing the source of a command line
parameter. :issue:<code>1264</code>, :pr:<code>1329</code></li>
<li>Add an optional parameter to <code>ProgressBar.update</code> to set the
<code>current_item</code>. :issue:<code>1226</code>, :pr:<code>1332</code></li>
<li><code>version_option</code> uses <code>importlib.metadata</code> (or the
<code>importlib_metadata</code> backport) instead of <code>pkg_resources</code>.
:issue:<code>1582</code></li>
<li>If validation fails for a prompt with <code>hide_input=True</code>, the value
is not shown in the error message. :issue:<code>1460</code></li>
<li>An <code>IntRange</code> or <code>FloatRange</code> option shows the accepted range in
its help text. :issue:<code>1525</code>, :pr:<code>1303</code></li>
<li><code>IntRange</code> and <code>FloatRange</code> bounds can be open (<code>&lt;</code>) instead
of closed (<code>&lt;=</code>) by setting <code>min_open</code> and <code>max_open</code>. Error
messages have changed to reflect this. :issue:<code>1100</code></li>
<li>An option defined with duplicate flag names (<code>&quot;--foo/--foo&quot;</code>)
raises a <code>ValueError</code>. :issue:<code>1465</code></li>
<li><code>echo()</code> will not fail when using pytest's <code>capsys</code> fixture on
Windows. :issue:<code>1590</code></li>
<li>Resolving commands returns the canonical command name instead of the
matched name. This makes behavior such as help text and
<code>Context.invoked_subcommand</code> consistent when using patterns like
<code>AliasedGroup</code>. :issue:<code>1422</code></li>
<li>The <code>BOOL</code> type accepts the values &quot;on&quot; and &quot;off&quot;. :issue:<code>1629</code></li>
<li>A <code>Group</code> with <code>invoke_without_command=True</code> will always invoke
its result callback. :issue:<code>1178</code></li>
<li><code>nargs == -1</code> and <code>nargs &gt; 1</code> is parsed and validated for
values from environment variables and defaults. :issue:<code>729</code></li>
<li>Detect the program name when executing a module or package with
<code>python -m name</code>. :issue:<code>1603</code></li>
<li>Include required parent arguments in help synopsis of subcommands.
:issue:<code>1475</code></li>
<li>Help for boolean flags with <code>show_default=True</code> shows the flag
name instead of <code>True</code> or <code>False</code>. :issue:<code>1538</code></li>
<li>Non-string objects passed to <code>style()</code> and <code>secho()</code> will be
converted to string. :pr:<code>1146</code></li>
<li><code>edit(require_save=True)</code> will detect saves for editors that exit
very fast on filesystems with 1 second resolution. :pr:<code>1050</code></li>
<li>New class attributes make it easier to use custom core objects
throughout an entire application. :pr:<code>938</code></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pallets/click/commit/9da166957f5848b641231d485467f6140bca2bc0""><code>9da1669</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/click/issues/1877"">#1877</a> from pallets/release-8.0.0</li>
<li><a href=""https://github.com/pallets/click/commit/dfa63691631e733712d8a7d706e154f3d7b7cd5d""><code>dfa6369</code></a> release version 8.0.0</li>
<li><a href=""https://github.com/pallets/click/commit/b862cb17cb99e748bf0160583d00874aaabe2f04""><code>b862cb1</code></a> update requirements</li>
<li><a href=""https://github.com/pallets/click/commit/f51584cf506bec6624435b408ce3cf2f7a10baf6""><code>f51584c</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/click/issues/1876"">#1876</a> from pallets/pre-commit-ci-schedule</li>
<li><a href=""https://github.com/pallets/click/commit/804c71c4dbf22cb5fa5bda6c2a5fb902b790f229""><code>804c71c</code></a> update pre-commit monthly</li>
<li><a href=""https://github.com/pallets/click/commit/ac655f8ac8a13cd45ce96bdc296bd857654088f0""><code>ac655f8</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/click/issues/1872"">#1872</a> from janLuke/fix/formatter_write_text</li>
<li><a href=""https://github.com/pallets/click/commit/dcd991d986d43c4f12cc838c4008dfafc015c21d""><code>dcd991d</code></a> HelpFormatter.write_text uses full width</li>
<li><a href=""https://github.com/pallets/click/commit/5215fc1030ed911c0c67b18e2fcad95fc173dd0e""><code>5215fc1</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/click/issues/1870"">#1870</a> from AdrienPensart/allow_colors_in_metavar</li>
<li><a href=""https://github.com/pallets/click/commit/e3e1691620c4791e662e4d8846b18295c1a018a2""><code>e3e1691</code></a> repr is erasing ANSI escapes codes</li>
<li><a href=""https://github.com/pallets/click/commit/482e6e6ae7e4da1712c16baf28c2cb239ee35b8a""><code>482e6e6</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/click/issues/1875"">#1875</a> from pallets/pre-commit-ci-update-config</li>
<li>Additional commits viewable in <a href=""https://github.com/pallets/click/compare/7.1.1...8.0.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1323/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
890073940,MDExOlB1bGxSZXF1ZXN0NjQzMTQ5MzIw,1324,"Update jinja2 requirement from <2.12.0,>=2.10.3 to >=2.10.3,<3.1.0",49699333,dependabot[bot],closed,0,,,,,2,2021-05-12T13:08:59Z,2021-05-17T17:19:41Z,2021-05-17T17:19:40Z,CONTRIBUTOR,simonw/datasette/pulls/1324,"Updates the requirements on [jinja2](https://github.com/pallets/jinja) to permit the latest version.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/pallets/jinja/releases"">jinja2's releases</a>.</em></p>
<blockquote>
<h2>3.0.0</h2>
<p>New major versions of all the core Pallets libraries, including Jinja 3.0, have been released! :tada:</p>
<ul>
<li>Read the announcement on our blog: <a href=""https://palletsprojects.com/blog/flask-2-0-released/"">https://palletsprojects.com/blog/flask-2-0-released/</a></li>
<li>Read the full list of changes: <a href=""https://jinja.palletsprojects.com/changes/#version-3-0-0"">https://jinja.palletsprojects.com/changes/#version-3-0-0</a></li>
<li>Retweet the announcement on Twitter: <a href=""https://twitter.com/PalletsTeam/status/1392266507296514048"">https://twitter.com/PalletsTeam/status/1392266507296514048</a></li>
<li>Follow our blog, Twitter, or GitHub to see future announcements.</li>
</ul>
<p>This represents a significant amount of work, and there are quite a few changes. Be sure to carefully read the changelog, and use tools such as pip-compile and Dependabot to pin your dependencies and control your updates.</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pallets/jinja/blob/main/CHANGES.rst"">jinja2's changelog</a>.</em></p>
<blockquote>
<h2>Version 3.0.0</h2>
<p>Released 2021-05-11</p>
<ul>
<li>Drop support for Python 2.7 and 3.5.</li>
<li>Bump MarkupSafe dependency to &gt;=1.1.</li>
<li>Bump Babel optional dependency to &gt;=2.1.</li>
<li>Remove code that was marked deprecated.</li>
<li>Add type hinting. :pr:<code>1412</code></li>
<li>Use :pep:<code>451</code> API to load templates with
:class:<code>~loaders.PackageLoader</code>. :issue:<code>1168</code></li>
<li>Fix a bug that caused imported macros to not have access to the
current template's globals. :issue:<code>688</code></li>
<li>Add ability to ignore <code>trim_blocks</code> using <code>+%}</code>. :issue:<code>1036</code></li>
<li>Fix a bug that caused custom async-only filters to fail with
constant input. :issue:<code>1279</code></li>
<li>Fix UndefinedError incorrectly being thrown on an undefined variable
instead of <code>Undefined</code> being returned on
<code>NativeEnvironment</code> on Python 3.10. :issue:<code>1335</code></li>
<li>Blocks can be marked as <code>required</code>. They must be overridden at
some point, but not necessarily by the direct child. :issue:<code>1147</code></li>
<li>Deprecate the <code>autoescape</code> and <code>with</code> extensions, they are
built-in to the compiler. :issue:<code>1203</code></li>
<li>The <code>urlize</code> filter recognizes <code>mailto:</code> links and takes
<code>extra_schemes</code> (or <code>env.policies[&quot;urlize.extra_schemes&quot;]</code>) to
recognize other schemes. It tries to balance parentheses within a
URL instead of ignoring trailing characters. The parsing in general
has been updated to be more efficient and match more cases. URLs
without a scheme are linked as <code>https://</code> instead of <code>http://</code>.
:issue:<code>522, 827, 1172</code>, :pr:<code>1195</code></li>
<li>Filters that get attributes, such as <code>map</code> and <code>groupby</code>, can
use a false or empty value as a default. :issue:<code>1331</code></li>
<li>Fix a bug that prevented variables set in blocks or loops from
being accessed in custom context functions. :issue:<code>768</code></li>
<li>Fix a bug that caused scoped blocks from accessing special loop
variables. :issue:<code>1088</code></li>
<li>Update the template globals when calling
<code>Environment.get_template(globals=...)</code> even if the template was
already loaded. :issue:<code>295</code></li>
<li>Do not raise an error for undefined filters in unexecuted
if-statements and conditional expressions. :issue:<code>842</code></li>
<li>Add <code>is filter</code> and <code>is test</code> tests to test if a name is a
registered filter or test. This allows checking if a filter is
available in a template before using it. Test functions can be
decorated with <code>@pass_environment</code>, <code>@pass_eval_context</code>,
or <code>@pass_context</code>. :issue:<code>842</code>, :pr:<code>1248</code></li>
<li>Support <code>pgettext</code> and <code>npgettext</code> (message contexts) in i18n
extension. :issue:<code>441</code></li>
<li>The <code>|indent</code> filter's <code>width</code> argument can be a string to</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pallets/jinja/commit/417f822196f66155e8c121e5229cc12a6b02ce14""><code>417f822</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/jinja/issues/1417"">#1417</a> from pallets/release-3.0.0</li>
<li><a href=""https://github.com/pallets/jinja/commit/b15fd1c94c696c0e17e2a2668ce70571d613d721""><code>b15fd1c</code></a> release version 3.0.0</li>
<li><a href=""https://github.com/pallets/jinja/commit/eed1546e195338458784c242e2aace220d0559e8""><code>eed1546</code></a> update requirements</li>
<li><a href=""https://github.com/pallets/jinja/commit/9627e73aa7024ba8823eac05e69038c5a1c9dda3""><code>9627e73</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/jinja/issues/1416"">#1416</a> from pallets/pre-commit-ci-schedule</li>
<li><a href=""https://github.com/pallets/jinja/commit/6630044b574577d86e26aa0823d3548acb948743""><code>6630044</code></a> update pre-commit monthly</li>
<li><a href=""https://github.com/pallets/jinja/commit/e1bf3bc3625c5e74f080594eead801ac3b0341af""><code>e1bf3bc</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/jinja/issues/1415"">#1415</a> from pallets/pre-commit-ci-update-config</li>
<li><a href=""https://github.com/pallets/jinja/commit/0798a6616e0de2bb848c8e5e9582a7b6dafbac39""><code>0798a66</code></a> [pre-commit.ci] pre-commit autoupdate</li>
<li><a href=""https://github.com/pallets/jinja/commit/53fea337d1b817aa12f5833df745987427e40e6a""><code>53fea33</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/jinja/issues/1414"">#1414</a> from pallets/typing-marker</li>
<li><a href=""https://github.com/pallets/jinja/commit/ba1f27e11b55cb6f6a0fd64a8f3d078369831691""><code>ba1f27e</code></a> fix py.typed in manifest</li>
<li><a href=""https://github.com/pallets/jinja/commit/1a3342b27ec09b547cd238ab0fb35c658116f822""><code>1a3342b</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/jinja/issues/1412"">#1412</a> from pallets/typing</li>
<li>Additional commits viewable in <a href=""https://github.com/pallets/jinja/compare/2.10.3...3.0.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1324/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
890073989,MDExOlB1bGxSZXF1ZXN0NjQzMTQ5MzY0,1325,"Update itsdangerous requirement from ~=1.1 to >=1.1,<3.0",49699333,dependabot[bot],closed,0,,,,,2,2021-05-12T13:09:03Z,2021-05-22T23:54:25Z,2021-05-22T23:54:25Z,CONTRIBUTOR,simonw/datasette/pulls/1325,"Updates the requirements on [itsdangerous](https://github.com/pallets/itsdangerous) to permit the latest version.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/pallets/itsdangerous/releases"">itsdangerous's releases</a>.</em></p>
<blockquote>
<h2>2.0.0</h2>
<p>New major versions of all the core Pallets libraries, including ItsDangerous 2.0, have been released! :tada:</p>
<ul>
<li>Read the announcement on our blog: <a href=""https://palletsprojects.com/blog/flask-2-0-released/"">https://palletsprojects.com/blog/flask-2-0-released/</a></li>
<li>Read the full list of changes: <a href=""https://itsdangerous.palletsprojects.com/changes/#version-2-0-0"">https://itsdangerous.palletsprojects.com/changes/#version-2-0-0</a></li>
<li>Retweet the announcement on Twitter: <a href=""https://twitter.com/PalletsTeam/status/1392266507296514048"">https://twitter.com/PalletsTeam/status/1392266507296514048</a></li>
<li>Follow our blog, Twitter, or GitHub to see future announcements.</li>
</ul>
<p>This represents a significant amount of work, and there are quite a few changes. Be sure to carefully read the changelog, and use tools such as pip-compile and Dependabot to pin your dependencies and control your updates.</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pallets/itsdangerous/blob/main/CHANGES.rst"">itsdangerous's changelog</a>.</em></p>
<blockquote>
<h2>Version 2.0.0</h2>
<p>Released 2021-05-11</p>
<ul>
<li>Drop support for Python 2 and 3.5.</li>
<li>JWS support (<code>JSONWebSignatureSerializer</code>,
<code>TimedJSONWebSignatureSerializer</code>) is deprecated. Use a dedicated
JWS/JWT library such as authlib instead. :issue:<code>129</code></li>
<li>Importing <code>itsdangerous.json</code> is deprecated. Import Python's
<code>json</code> module instead. :pr:<code>152</code></li>
<li>Simplejson is no longer used if it is installed. To use a different
library, pass it as <code>Serializer(serializer=...)</code>. :issue:<code>146</code></li>
<li><code>datetime</code> values are timezone-aware with <code>timezone.utc</code>. Code
using <code>TimestampSigner.unsign(return_timestamp=True)</code> or
<code>BadTimeSignature.date_signed</code> may need to change. :issue:<code>150</code></li>
<li>If a signature has an age less than 0, it will raise
<code>SignatureExpired</code> rather than appearing valid. This can happen if
the timestamp offset is changed. :issue:<code>126</code></li>
<li><code>BadTimeSignature.date_signed</code> is always a <code>datetime</code> object
rather than an <code>int</code> in some cases. :issue:<code>124</code></li>
<li>Added support for key rotation. A list of keys can be passed as
<code>secret_key</code>, oldest to newest. The newest key is used for
signing, all keys are tried for unsigning. :pr:<code>141</code></li>
<li>Removed the default SHA-512 fallback signer from
<code>default_fallback_signers</code>. :issue:<code>155</code></li>
<li>Add type information for static typing tools. :pr:<code>186</code></li>
</ul>
<h2>Version 1.1.0</h2>
<p>Released 2018-10-26</p>
<ul>
<li>Change default signing algorithm back to SHA-1. :pr:<code>113</code></li>
<li>Added a default SHA-512 fallback for users who used the yanked 1.0.0
release which defaulted to SHA-512. :pr:<code>114</code></li>
<li>Add support for fallback algorithms during deserialization to
support changing the default in the future without breaking existing
signatures. :pr:<code>113</code></li>
<li>Changed capitalization of packages back to lowercase as the change
in capitalization broke some tooling. :pr:<code>113</code></li>
</ul>
<h2>Version 1.0.0</h2>
<p>Released 2018-10-18</p>
<p>YANKED</p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pallets/itsdangerous/commit/d101100c395958d67368b8c37d95a9c404598c2e""><code>d101100</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/itsdangerous/issues/235"">#235</a> from pallets/release-2.0.0</li>
<li><a href=""https://github.com/pallets/itsdangerous/commit/ca0f59ac73507014729a5857e985229604e5e83b""><code>ca0f59a</code></a> release version 2.0.0</li>
<li><a href=""https://github.com/pallets/itsdangerous/commit/d1ed89f6ac50d58fa43ce37f92db371bebc20bc5""><code>d1ed89f</code></a> update requirements</li>
<li><a href=""https://github.com/pallets/itsdangerous/commit/d1722ea35b4239b6d08e8d418edc74d6594eebd6""><code>d1722ea</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/itsdangerous/issues/234"">#234</a> from pallets/pre-commit-ci-schedule</li>
<li><a href=""https://github.com/pallets/itsdangerous/commit/d1eb7aa76756268061f2d31551f79da7882b45a9""><code>d1eb7aa</code></a> update pre-commit monthly</li>
<li><a href=""https://github.com/pallets/itsdangerous/commit/acbc456c91d4a9e9f63b9230be5a1cf3d5a1d767""><code>acbc456</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/itsdangerous/issues/233"">#233</a> from pallets/pre-commit-ci-update-config</li>
<li><a href=""https://github.com/pallets/itsdangerous/commit/04e485a0b0b00d8b2dc73a287852d642d3366c72""><code>04e485a</code></a> [pre-commit.ci] pre-commit autoupdate</li>
<li><a href=""https://github.com/pallets/itsdangerous/commit/c0e6b484a6cdc92c38bec74ffcf350bc42f20e2b""><code>c0e6b48</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/itsdangerous/issues/232"">#232</a> from pallets/pre-commit-ci-update-config</li>
<li><a href=""https://github.com/pallets/itsdangerous/commit/6a9df8338ce39b780cebd8a7f54b8a7cb75eeab4""><code>6a9df83</code></a> [pre-commit.ci] pre-commit autoupdate</li>
<li><a href=""https://github.com/pallets/itsdangerous/commit/477f42c3e74f1a823deb3d7c3ab94a5b534d545b""><code>477f42c</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pallets/itsdangerous/issues/231"">#231</a> from pallets/dependabot/pip/pre-commit-2.12.1</li>
<li>Additional commits viewable in <a href=""https://github.com/pallets/itsdangerous/compare/1.1.0...2.0.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1325/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
891969037,MDU6SXNzdWU4OTE5NjkwMzc=,1326,How to limit fields returned from the JSON API?,5268174,bram2000,closed,0,,,,,1,2021-05-14T14:27:41Z,2021-05-23T02:55:06Z,2021-05-23T02:55:00Z,NONE,,"Hi,

I have quite wide tables, and in many cases only want a subset of the data (to save on network bandwidth). I need to use the JSON API as handling pagination is so much easier, but I can't see a way to select specific columns.

Is there a way to do this, or is it a feature request?

Thanks!",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1326/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
892457208,MDU6SXNzdWU4OTI0NTcyMDg=,1327,Support Unicode characters in metadata.json,20846286,GmGniap,closed,0,,,,,2,2021-05-15T14:33:58Z,2021-05-24T19:10:21Z,2021-05-24T19:10:21Z,NONE,,"Hello , when I used Burmese (Unicode) characters in metadata.json like below - 
![image](https://user-images.githubusercontent.com/20846286/118364978-cba70100-b5c0-11eb-967c-7dc3b62478f2.png)

It gave wrong results when I run datasette - 
![image](https://user-images.githubusercontent.com/20846286/118365025-fc873600-b5c0-11eb-97ce-19541b8cc6d8.png)

It would be great & helpful for us if metadata.json can support in Unicode supported Asian Languages. 
Thanks & Regards. ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1327/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275176006,MDU6SXNzdWUyNzUxNzYwMDY=,133,"If view is filtered, search should apply within those filtered rows",9599,simonw,closed,0,,,2919870,Foreign key edition,3,2017-11-19T17:25:36Z,2017-11-24T22:30:32Z,2017-11-24T22:30:15Z,OWNER,,Eg on https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List?qSpecies=1,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/133/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
893314402,MDExOlB1bGxSZXF1ZXN0NjQ1ODQ5MDI3,1330,"Update aiofiles requirement from <0.7,>=0.4 to >=0.4,<0.8",49699333,dependabot[bot],closed,0,,,,,1,2021-05-17T13:07:31Z,2021-05-22T23:53:57Z,2021-05-22T23:53:56Z,CONTRIBUTOR,simonw/datasette/pulls/1330,"Updates the requirements on [aiofiles](https://github.com/Tinche/aiofiles) to permit the latest version.
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/Tinche/aiofiles/commit/469c1a8b94ccccf25516706c88dde2e69ea45f3c""><code>469c1a8</code></a> v0.7.0</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/26decd33a86f1fa93599a1fad117cd8dead28c5a""><code>26decd3</code></a> Add PyPy</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/734d182bb97e1185eeb71a0f9dd92a4645779795""><code>734d182</code></a> README update</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/b0bb11077d9cedf2527422b855dc4f11acbb07cf""><code>b0bb110</code></a> More coverage</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/e28714d2c74cee0f5b0cd54e659467f9e8bf6f70""><code>e28714d</code></a> Fix CI</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/6765e3de2bfa5939548f478ce186269771174d91""><code>6765e3d</code></a> Switch to poetry some more</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/165ba880eaed010a15b590327cd8a7f396fc6f5b""><code>165ba88</code></a> Switch to poetry</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/aba1c10453f174391b1a40fc8c769a8e813c2907""><code>aba1c10</code></a> Try isolated_build</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/f8b0e2710e53f57368be5579b4826b51a2c02846""><code>f8b0e27</code></a> passenv</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/d291390e8b93b3585236233f49ad2a3b91744523""><code>d291390</code></a> Tweak Tox</li>
<li>Additional commits viewable in <a href=""https://github.com/Tinche/aiofiles/compare/v0.4.0...v0.7.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1330/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
893537744,MDU6SXNzdWU4OTM1Mzc3NDQ=,1331,Add support for Jinja2 version 3.0,475613,MarkusH,closed,0,,,,,10,2021-05-17T17:14:36Z,2021-05-23T00:57:39Z,2021-05-23T00:57:39Z,NONE,,"A week ago, [The Pallets Project](https://github.com/pallets) released [new major versions of several of its projects](https://palletsprojects.com/blog/flask-2-0-released/). Among those updates is one for Jinja2, which bumps it to version 3.0.0.

I'd like for datasette to support Jinaj2 version 3.0.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1331/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
893890496,MDU6SXNzdWU4OTM4OTA0OTY=,1332,?_facet_size=X to increase number of facets results on the page,192568,mroswell,closed,0,,,,,5,2021-05-18T02:40:16Z,2021-05-27T16:13:07Z,2021-05-23T00:34:37Z,CONTRIBUTOR,,"Is there a way to add a parameter to the URL to modify default_facet_size?

LIkewise, a way to produce a link on the three dots to expand to all items (or match previous number of items, or add x more)?

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1332/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
895315478,MDExOlB1bGxSZXF1ZXN0NjQ3NTUyMTQx,1335,Fix small typo,3243482,abdusco,closed,0,,,,,1,2021-05-19T11:17:04Z,2021-05-22T23:53:34Z,2021-05-22T23:53:34Z,CONTRIBUTOR,simonw/datasette/pulls/1335,,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1335/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
895686039,MDU6SXNzdWU4OTU2ODYwMzk=,1336,Document turning on WAL for live served SQLite databases,9599,simonw,open,0,,,,,1,2021-05-19T17:08:58Z,2022-01-13T21:55:59Z,,OWNER,,"Datasette docs don't talk about WAL yet, which allows you to safely serve reads from a database file while it is accepting writes.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1336/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
898904402,MDU6SXNzdWU4OTg5MDQ0MDI=,1337,"""More"" link for facets that shows _facet_size=max results",9599,simonw,closed,0,,,,,7,2021-05-23T00:08:51Z,2021-05-27T16:14:14Z,2021-05-27T16:01:03Z,OWNER,,"_Original title:  ""More"" link for facets that shows the full set of results_

The simplest way to do this will be to have it link to a generated SQL query.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1332#issuecomment-846479062_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1337/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
899169307,MDU6SXNzdWU4OTkxNjkzMDc=,1338,Fix jinja2 warnings,9599,simonw,closed,0,,,,,0,2021-05-24T01:38:23Z,2021-05-24T01:41:55Z,2021-05-24T01:41:55Z,OWNER,,"Lots of these in the test suite now, after the Jinja upgrade in #1331:
```
tests/test_plugins.py::test_hook_render_cell_link_from_json
  datasette/tests/plugins/my_plugin_2.py:45: DeprecationWarning: 'jinja2.escape' is deprecated and will be removed in Jinja 3.1. Import 'markupsafe.escape' instead.
    label=jinja2.escape(data[""label""] or """") or ""&nbsp;"",

tests/test_plugins.py::test_hook_render_cell_link_from_json
  datasette/tests/plugins/my_plugin_2.py:41: DeprecationWarning: 'jinja2.Markup' is deprecated and will be removed in Jinja 3.1. Import 'markupsafe.Markup' instead.
    return jinja2.Markup(
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1338/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
899851083,MDExOlB1bGxSZXF1ZXN0NjUxNDkyODg4,1339,?_col=/?_nocol= to show/hide columns on the table page,9599,simonw,closed,0,,,,,1,2021-05-24T17:15:20Z,2021-05-27T04:17:44Z,2021-05-27T04:17:43Z,OWNER,simonw/datasette/pulls/1339,"See #615. Still to do:

- [x] Allow combination of `?_col=` and `?_nocol=` (`_nocol` wins)
- [x] Deduplicate same column if passed in `?_col=` multiple times
- [x] Validate that user did not try to remove a primary key
- [x] Add tests
- [x] Ensure this works correctly for SQL views
- [x] Add documentation
",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1339/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
275176094,MDU6SXNzdWUyNzUxNzYwOTQ=,134,Filtered table view should show a count,9599,simonw,closed,0,,,2919870,Foreign key edition,1,2017-11-19T17:26:53Z,2017-11-19T18:10:49Z,2017-11-19T18:10:49Z,OWNER,,Let's do the thing where we attempt to show an accurate count if it can be done in less than 50ms,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/134/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
901009787,MDU6SXNzdWU5MDEwMDk3ODc=,1340,Research: Cell action menu (like column action but for individual cells),9599,simonw,open,0,,,,,1,2021-05-25T15:49:16Z,2021-05-26T18:59:58Z,,OWNER,,"Had an idea today that it might be useful to select an individual cell and say things like ""show me all other rows with the same value"" - maybe even a set of other menu options against cells as well.

Mocked up a show-on-hover ellipses demo using the CSS inspector:

![idea](https://user-images.githubusercontent.com/9599/119528316-f0744480-bd35-11eb-8eb4-1deea6d60cce.gif)
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1340/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
903200328,MDU6SXNzdWU5MDMyMDAzMjg=,1341,"""Show all columns"" cog menu item should show if ?_col= is used",9599,simonw,closed,0,,,,,1,2021-05-27T04:28:17Z,2021-05-27T04:31:16Z,2021-05-27T04:31:16Z,OWNER,,"On https://latest.datasette.io/fixtures/sortable?_col=sortable the ""Show all columns"" item (from #615) is not shown (it should be):

<img width=""630"" alt=""fixtures__sortable__201_rows"" src=""https://user-images.githubusercontent.com/9599/119766097-44337a80-be69-11eb-8674-8031e72cb17d.png"">
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1341/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
903902495,MDU6SXNzdWU5MDM5MDI0OTU=,1342,Improve `path_with_replaced_args()` and friends and document them,9599,simonw,open,0,,,,,3,2021-05-27T15:18:28Z,2021-05-27T15:23:02Z,,OWNER,,"> In order to cleanly implement this I need to expose the `path_with_replaced_args` utility function to Datasette's template engine. This is the first time this will become an exposed (and hence should-by-documented) API and I don't like its shape much.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1337#issuecomment-849721280_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1342/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
903978133,MDU6SXNzdWU5MDM5NzgxMzM=,1343,Figure out how to publish alpha/beta releases to Docker Hub,9599,simonw,closed,0,,,,,4,2021-05-27T16:42:17Z,2021-05-27T16:46:37Z,2021-05-27T16:45:41Z,OWNER,,"> It looks like all I need to do to ship an alpha version to Docker Hub is NOT point the `latest` tag at it after it goes live: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/publish.yml#L75-L77

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1319#issuecomment-849780481_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1343/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
903986178,MDU6SXNzdWU5MDM5ODYxNzg=,1344,Test Datasette Docker images built for different architectures,9599,simonw,open,0,,,,,10,2021-05-27T16:52:29Z,2022-09-06T00:07:58Z,,OWNER,,"Continuing on from #1319 - now that we have the ability to build Datasette's Docker image against multiple architectures we should test that it works.

We can do this with QEMU emulation, see https://twitter.com/nevali/status/1397958044571602945",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1344/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
904071938,MDU6SXNzdWU5MDQwNzE5Mzg=,1345,?_nocol= does not interact well with default facets,9599,simonw,closed,0,,,,,7,2021-05-27T18:39:55Z,2021-05-31T02:40:44Z,2021-05-31T02:31:21Z,OWNER,,"Clicking ""Hide this column"" on `fips` on https://covid-19.datasettes.com/covid/ny_times_us_counties shows this error:

https://covid-19.datasettes.com/covid/ny_times_us_counties?_nocol=fips

> ## Invalid SQL
> no such column: fips

The reason is that https://covid-19.datasettes.com/-/metadata sets up the following:

```json
  ""ny_times_us_counties"": {
      ""sort_desc"": ""date"",
      ""facets"": [
          ""state"",
          ""county"",
          ""fips""
      ],
```
It's setting `fips` as a default facet, which breaks if you attempt to remove the column using `?_nocol`.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1345/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
904537568,MDExOlB1bGxSZXF1ZXN0NjU1Njg0NDc3,1346,Re-display user's query with an error message if an error occurs,9599,simonw,closed,0,,,,,3,2021-05-28T02:04:20Z,2021-06-02T03:46:21Z,2021-06-02T03:46:21Z,OWNER,simonw/datasette/pulls/1346,Refs #619,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1346/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
904582277,MDExOlB1bGxSZXF1ZXN0NjU1NzI2Mzg3,1347,Test docker platform blair only,10801138,blairdrummond,closed,0,,,,,0,2021-05-28T02:47:09Z,2021-05-28T02:47:28Z,2021-05-28T02:47:28Z,CONTRIBUTOR,simonw/datasette/pulls/1347,,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1347/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
904598267,MDExOlB1bGxSZXF1ZXN0NjU1NzQxNDI4,1348,DRAFT: add test and scan for docker images,10801138,blairdrummond,open,0,,,,,2,2021-05-28T03:02:12Z,2021-05-28T03:06:16Z,,CONTRIBUTOR,simonw/datasette/pulls/1348,"**NOTE: I don't think this PR is ready, since the arm/v6 and arm/v7 images are failing pytest due to missing dependencies (gcc and friends). But it's pretty close.**

Closes https://github.com/simonw/datasette/issues/1344 . Using a build-matrix for the platforms and [this test](https://github.com/simonw/datasette/issues/1344#issuecomment-849820019), we test all the platforms in parallel. I also threw in container scanning.

### Switch `pip install` to use either tags or commit shas

Notably! This also [changes the Dockerfile](https://github.com/blairdrummond/datasette/blob/7fe5315d68e04fce64b5bebf4e2d7feec44f8546/Dockerfile#L20) so that it accepts tags or commit-shas.

```
# It's backwards compatible with tags, but also lets you use shas
root@712071df17af:/# pip install git+git://github.com/simonw/datasette.git@0.56                                                                                                                                            
Collecting git+git://github.com/simonw/datasette.git@0.56                                                                                                                                                                  
  Cloning git://github.com/simonw/datasette.git (to revision 0.56) to /tmp/pip-req-build-u6dhm945                                                                                                                          
  Running command git clone -q git://github.com/simonw/datasette.git /tmp/pip-req-build-u6dhm945                                                                                                                           
  Running command git checkout -q af5a7f1c09f6a902bb2a25e8edf39c7034d2e5de                                                                                                                                                 
Collecting Jinja2<2.12.0,>=2.10.3                                                                            
  Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB) 
```

This lets you build the containers in CI every push for testing, which maybe resolves [this problem](https://github.com/simonw/datasette/issues/1272#issuecomment-808648974)?

# Workflow run example

You can see the results in my workflow [here](https://github.com/blairdrummond/datasette/pull/2/checks?check_run_id=2690570717). The commit history is different because I squashed this branch, also in the testing branch I had to change `github.com/simonw` to `github.com/blairdrummond` for the CI to pick up my git_sha.

## Why did the builds fail?

**NOTE:** The results of all the tests fail, but for different reasons! A few fail to install Rust, the amd64 passes the tests (phew!) but has critical CVEs which fail the container scan, the Arm/v6 and Arm/v7 seem to fail to install the test dependencies due to missing programs like `gcc`. (`gcc` is not sufficient though, as [this run](https://github.com/blairdrummond/datasette/pull/3/checks?check_run_id=2690672982) indicates) ",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1348/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
906385991,MDU6SXNzdWU5MDYzODU5OTE=,1349,CSV ?_stream=on redundantly calculates facets for every page,9599,simonw,closed,0,,,,,9,2021-05-29T06:11:23Z,2021-06-17T18:12:32Z,2021-06-01T15:52:53Z,OWNER,,"I'm trying to figure out why a full CSV export from https://covid-19.datasettes.com/covid/ny_times_us_counties runs unbearably slowly.

It's because the streaming endpoint works by scrolling through every page, and it turns out every page calculates facets and suggested facets!",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1349/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275179724,MDU6SXNzdWUyNzUxNzk3MjQ=,135,?_search=x should work if used directly against a FTS virtual table,9599,simonw,closed,0,,,2949431,Custom templates edition,3,2017-11-19T18:17:53Z,2017-12-07T04:54:41Z,2017-12-07T04:54:41Z,OWNER,,e.g. https://sf-trees.now.sh/sf-trees-ebc2ad9/Street_Tree_List_fts?_search=grove should work,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/135/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
906977719,MDU6SXNzdWU5MDY5Nzc3MTk=,1350,?_nofacets=1 query string argument for disabling facets and suggested facets,9599,simonw,closed,0,,,,,2,2021-05-31T02:22:29Z,2021-06-01T16:19:38Z,2021-05-31T02:39:18Z,OWNER,,"This is needed as an internal option for #1349. `datasette-graphql` can benefit from this too - maybe can even use it so that if you pass `?_shape=array` it gets automatically added, fixing #263.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1350/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
906993731,MDU6SXNzdWU5MDY5OTM3MzE=,1351,Get `?_trace=1` working with CSV and streaming CSVs,9599,simonw,closed,0,,,,,1,2021-05-31T03:02:15Z,2021-06-17T18:12:32Z,2021-06-01T15:50:09Z,OWNER,,"> I think it's worth getting `?_trace=1` to work with streaming CSV - this would have helped me spot this issue a long time ago.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1349#issuecomment-851133125_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1351/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
908276134,MDExOlB1bGxSZXF1ZXN0NjU4OTkxNDA0,1352,Bump black from 21.5b1 to 21.5b2,49699333,dependabot[bot],closed,0,,,,,1,2021-06-01T13:08:52Z,2021-06-02T02:56:45Z,2021-06-02T02:56:44Z,CONTRIBUTOR,simonw/datasette/pulls/1352,"Bumps [black](https://github.com/psf/black) from 21.5b1 to 21.5b2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.5b2</h2>
<h3><em>Black</em></h3>
<ul>
<li>A space is no longer inserted into empty docstrings (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2249"">#2249</a>)</li>
<li>Fix handling of .gitignore files containing non-ASCII characters on Windows (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2229"">#2229</a>)</li>
<li>Respect <code>.gitignore</code> files in all levels, not only <code>root/.gitignore</code> file (apply
<code>.gitignore</code> rules like <code>git</code> does) (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2225"">#2225</a>)</li>
<li>Restored compatibility with Click 8.0 on Python 3.6 when LANG=C used (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2227"">#2227</a>)</li>
<li>Add extra uvloop install + import support if in python env (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2258"">#2258</a>)</li>
<li>Fix --experimental-string-processing crash when matching parens are not found (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2283"">#2283</a>)</li>
<li>Make sure to split lines that start with a string operator (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2286"">#2286</a>)</li>
<li>Fix regular expression that black uses to identify f-expressions (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2287"">#2287</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Add a lower bound for the <code>aiohttp-cors</code> dependency. Only 0.4.0 or higher is
supported. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2231"">#2231</a>)</li>
</ul>
<h3><em>Packaging</em></h3>
<ul>
<li>Release self-contained x86_64 MacOS binaries as part of the GitHub release pipeline
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/2198"">#2198</a>)</li>
<li>Always build binaries with the latest available Python (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2260"">#2260</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>Add discussion of magic comments to FAQ page (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2272"">#2272</a>)</li>
<li><code>--experimental-string-processing</code> will be enabled by default in the future (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2273"">#2273</a>)</li>
<li>Fix typos discovered by codespell (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2228"">#2228</a>)</li>
<li>Fix Vim plugin installation instructions. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2235"">#2235</a>)</li>
<li>Add new Frequently Asked Questions page (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2247"">#2247</a>)</li>
<li>Fix encoding + symlink issues preventing proper build on Windows (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2262"">#2262</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>21.5b2</h2>
<h3><em>Black</em></h3>
<ul>
<li>A space is no longer inserted into empty docstrings (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2249"">#2249</a>)</li>
<li>Fix handling of .gitignore files containing non-ASCII characters on Windows (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2229"">#2229</a>)</li>
<li>Respect <code>.gitignore</code> files in all levels, not only <code>root/.gitignore</code> file (apply
<code>.gitignore</code> rules like <code>git</code> does) (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2225"">#2225</a>)</li>
<li>Restored compatibility with Click 8.0 on Python 3.6 when LANG=C used (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2227"">#2227</a>)</li>
<li>Add extra uvloop install + import support if in python env (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2258"">#2258</a>)</li>
<li>Fix --experimental-string-processing crash when matching parens are not found (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2283"">#2283</a>)</li>
<li>Make sure to split lines that start with a string operator (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2286"">#2286</a>)</li>
<li>Fix regular expression that black uses to identify f-expressions (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2287"">#2287</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Add a lower bound for the <code>aiohttp-cors</code> dependency. Only 0.4.0 or higher is
supported. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2231"">#2231</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>The official Black action now supports choosing what version to use, and supports the
major 3 OSes. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1940"">#1940</a>)</li>
</ul>
<h3>Packaging</h3>
<ul>
<li>Release self-contained x86_64 MacOS binaries as part of the GitHub release pipeline
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/2198"">#2198</a>)</li>
<li>Always build binaries with the latest available Python (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2260"">#2260</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>Add discussion of magic comments to FAQ page (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2272"">#2272</a>)</li>
<li><code>--experimental-string-processing</code> will be enabled by default in the future (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2273"">#2273</a>)</li>
<li>Fix typos discovered by codespell (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2228"">#2228</a>)</li>
<li>Fix Vim plugin installation instructions. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2235"">#2235</a>)</li>
<li>Add new Frequently Asked Questions page (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2247"">#2247</a>)</li>
<li>Fix encoding + symlink issues preventing proper build on Windows (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2262"">#2262</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.5b1&new-version=21.5b2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1352/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
908446997,MDU6SXNzdWU5MDg0NDY5OTc=,1353,?_nocount=1 for opting out of table counts,9599,simonw,closed,0,,,,,2,2021-06-01T15:53:27Z,2021-06-01T16:18:54Z,2021-06-01T16:17:04Z,OWNER,,"Running a trace against a CSV streaming export with the new `_trace=1` feature from #1351 shows that the following code is executing a `select count(*) from table` for every page of results returned: https://github.com/simonw/datasette/blob/d1d06ace49606da790a765689b4fbffa4c6deecb/datasette/views/table.py#L700-L705

This is inefficient - a new `?_nocount=1` option would let us disable this count in the same way as #1349: https://github.com/simonw/datasette/blob/d1d06ace49606da790a765689b4fbffa4c6deecb/datasette/views/base.py#L264-L276

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1353/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
908465747,MDU6SXNzdWU5MDg0NjU3NDc=,1354,Update help in tests for latest Click,9599,simonw,closed,0,,,,,1,2021-06-01T16:14:31Z,2021-06-01T16:17:04Z,2021-06-01T16:17:04Z,OWNER,,"Now that Uvicorn 0.14 is out with an unpinned Click dependency - https://github.com/encode/uvicorn/pull/1033 - our test suite runs against Click 8.0 - which subtly changes the output of `--help` causing test failures: https://github.com/simonw/datasette/runs/2720383031?check_suite_focus=true
```
    def test_help_includes(name, filename):
        expected = (docs_path / filename).read_text()
        runner = CliRunner()
        result = runner.invoke(cli, name.split() + [""--help""], terminal_width=88)
        actual = f""$ datasette {name} --help\n\n{result.output}""
        # actual has ""Usage: cli package [OPTIONS] FILES""
        # because it doesn't know that cli will be aliased to datasette
        expected = expected.replace(""Usage: datasette"", ""Usage: cli"")
>       assert expected == actual
E       AssertionError: assert '$ datasette ...e and exit.\n' == '$ datasette ...e and exit.\n'
E         Skipping 848 identical leading characters in diff, use -v to show
E           nt_id xxx
E         + 
E             --version-note TEXT             Additional note to show on /-/versions
E             --secret TEXT                   Secret used for signing secure values, such as signed
E                                             cookies
E         + 
E             --title TEXT                    Title for metadata
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1354/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
910088936,MDU6SXNzdWU5MTAwODg5MzY=,1355,datasette --get should efficiently handle streaming CSV,9599,simonw,open,0,,,,,2,2021-06-03T04:40:40Z,2022-03-20T22:38:53Z,,OWNER,,"It would be great if you could use `datasette --get` to run queries that return streaming CSV data without running out of RAM.

Current implementation looks like it loads the entire result into memory first: https://github.com/simonw/datasette/blob/f78ebdc04537a6102316d6dbbf6c887565806078/datasette/cli.py#L546-L552",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1355/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
910092577,MDU6SXNzdWU5MTAwOTI1Nzc=,1356,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",9599,simonw,open,0,,,,,10,2021-06-03T04:49:42Z,2022-01-20T01:06:37Z,,OWNER,,"Inspired by https://github.com/simonw/sqlite-utils/issues/264 - in particular this example:
```
datasette covid.db --get='/covid.yaml?sql=select * from ny_times_us_counties limit 1' 
- date: '2020-01-21'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
```
Having to construct that URL - including potentially URL escaping the SQL query - isn't a great developer experience.

Imagine if you could do this instead:

    datasette covid.db --query ""select * from ny_times_us_counties limit 1"" --format yaml
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1356/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
912394511,MDExOlB1bGxSZXF1ZXN0NjYyNTU3MjQw,1357,Make custom pages compatible with base_url setting,9599,simonw,closed,0,,,,,1,2021-06-05T18:54:39Z,2021-06-05T18:59:54Z,2021-06-05T18:59:54Z,OWNER,simonw/datasette/pulls/1357,Refs #1238.,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1357/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
912418094,MDU6SXNzdWU5MTI0MTgwOTQ=,1358,Release Datasette 0.57,9599,simonw,closed,0,,,,,3,2021-06-05T19:56:13Z,2021-06-05T22:20:07Z,2021-06-05T22:20:07Z,OWNER,,"Need release notes. Changes are here: https://github.com/simonw/datasette/compare/0.56...368aa5f1b16ca35f82d90ff747023b9a2bfa27c1

Partial release notes already exist for the two alphas, https://github.com/simonw/datasette/releases/tag/0.57a0 and https://github.com/simonw/datasette/releases/tag/0.57a1",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1358/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
912419349,MDU6SXNzdWU5MTI0MTkzNDk=,1359,`?_trace=1` should only be available with a new `trace_debug` setting,9599,simonw,closed,0,,,,,0,2021-06-05T19:59:27Z,2021-06-05T20:18:46Z,2021-06-05T20:18:46Z,OWNER,,Just like template debug mode is controlled by this off-by-default setting: https://github.com/simonw/datasette/blob/368aa5f1b16ca35f82d90ff747023b9a2bfa27c1/datasette/app.py#L160-L164,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1359/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275228834,MDU6SXNzdWUyNzUyMjg4MzQ=,136,"""Reformat SQL"" button next to SQL editor textarea",9599,simonw,closed,0,,,,,0,2017-11-20T03:42:19Z,2019-10-14T03:46:13Z,2019-10-14T03:46:13Z,OWNER,,"Can use this:

https://github.com/zeroturnaround/sql-formatter
https://zeroturnaround.github.io/sql-formatter/
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/136/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
912464443,MDU6SXNzdWU5MTI0NjQ0NDM=,1360,"Security flaw, to be fixed in 0.56.1 and 0.57",9599,simonw,closed,0,,,,,2,2021-06-05T21:53:51Z,2021-06-05T22:23:23Z,2021-06-05T22:22:06Z,OWNER,,"See security advisory here for details: https://github.com/simonw/datasette/security/advisories/GHSA-xw7c-jx9m-xh5g - the `?_trace=1` debugging option was not correctly escaping its JSON output, resulting in a [reflected cross-site scripting](https://owasp.org/www-community/attacks/xss/#reflected-xss-attacks) vulnerability.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1360/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
912485040,MDU6SXNzdWU5MTI0ODUwNDA=,1361,Intermittent CI failure: restore_working_directory FileNotFoundError,9599,simonw,closed,0,,,,,4,2021-06-05T22:48:13Z,2021-06-05T23:16:24Z,2021-06-05T23:16:24Z,OWNER,,"e.g. in https://github.com/simonw/datasette/runs/2754772233 - this is an intermittent error:
```
__________ ERROR at setup of test_hook_register_routes_render_message __________
[gw0] linux -- Python 3.8.10 /opt/hostedtoolcache/Python/3.8.10/x64/bin/python

tmpdir = local('/tmp/pytest-of-runner/pytest-0/popen-gw0/test_hook_register_routes_rend0')
request = <SubRequest 'restore_working_directory' for <Function test_hook_register_routes_render_message>>

    @pytest.fixture
    def restore_working_directory(tmpdir, request):
>       previous_cwd = os.getcwd()
E       FileNotFoundError: [Errno 2] No such file or directory
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1361/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
912864936,MDU6SXNzdWU5MTI4NjQ5MzY=,1362,Consider using CSP to protect against future XSS,9599,simonw,open,0,,,,,17,2021-06-06T15:32:20Z,2022-10-08T18:42:09Z,,OWNER,,The XSS in #1360 would have been a lot less damaging if Datasette used CSP to protect against such vulnerabilities: https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1362/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
912959264,MDU6SXNzdWU5MTI5NTkyNjQ=,1364,Don't truncate columns on the list of databases,9599,simonw,closed,0,,,,,0,2021-06-06T22:01:56Z,2021-06-06T22:07:50Z,2021-06-06T22:07:50Z,OWNER,,"https://covid-19.datasettes.com/covid currently truncates at 9 database columns:

<img width=""860"" alt=""covid"" src=""https://user-images.githubusercontent.com/9599/120941536-11467d80-c6d8-11eb-970a-ce469623f92c.png"">

Django SQL Dashboard showed me that this is a bad idea - having the full list of columns is actually really useful documentation for crafting custom SQL queries.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1364/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
913017577,MDU6SXNzdWU5MTMwMTc1Nzc=,1365,pathlib.Path breaks internal schema,25778,eyeseast,closed,0,,,,,1,2021-06-07T01:40:37Z,2021-06-21T15:57:39Z,2021-06-21T15:57:39Z,CONTRIBUTOR,,"Ran into an issue while trying to build a plugin to render GeoJSON. I'm using pytest's `tmp_path` fixture, which is a `pathlib.Path`, to get a temporary database path. I was getting a weird error involving writes, but I was doing reads. Turns out it's the internal database trying to insert a `Path` where it wants a string.

My test looked like this:

```python
@pytest.mark.asyncio
async def test_render_feature_collection(tmp_path):
    database = tmp_path / ""test.db""
    datasette = Datasette([database])

    # this will break with a path
    await datasette.refresh_schemas()

    # build a url
    url = datasette.urls.table(database.stem, TABLE_NAME, format=""geojson"")

    response = await datasette.client.get(url)
    fc = response.json()

    assert 200 == response.status_code
```

I only ran into this while running tests, because passing in database paths from the CLI uses strings, but it's a weird error and probably something other people have run into.

The fix is easy enough: Convert the path to a string and everything works. So this:

```python
@pytest.mark.asyncio
async def test_render_feature_collection(tmp_path):
    database = tmp_path / ""test.db""
    datasette = Datasette([str(database)])

    # this is fine now
    await datasette.refresh_schemas()
```

This could (probably, haven't tested) be fixed [here](https://github.com/simonw/datasette/blob/03ec71193b9545536898a4bc7493274fec48bdd7/datasette/app.py#L357) by calling `str(db.path)` or by doing that conversion earlier.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1365/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
913809802,MDU6SXNzdWU5MTM4MDk4MDI=,1366,Get rid of this `restore_working_directory` hack entirely,9599,simonw,open,0,,,,,2,2021-06-07T18:01:21Z,2021-06-07T18:03:03Z,,OWNER,,"> That seems to have fixed it. I'd love to get rid of this `restore_working_directory` hack entirely.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1361#issuecomment-855308811_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1366/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
913823889,MDU6SXNzdWU5MTM4MjM4ODk=,1367,Navigation menu display bug,9599,simonw,closed,0,,,,,1,2021-06-07T18:18:08Z,2021-06-07T18:24:19Z,2021-06-07T18:24:19Z,OWNER,,"With Datasette 0.57 the navigation menu looks like this:

<img width=""991"" alt=""Datasette_Fixtures___memory___internal__fixtures__extra_database"" src=""https://user-images.githubusercontent.com/9599/121069137-ffb9b000-c781-11eb-962a-b250a2772590.png"">
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1367/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
913865304,MDExOlB1bGxSZXF1ZXN0NjYzODM2OTY1,1368,DRAFT: A new plugin hook for dynamic metadata,2670795,brandonrobertz,closed,0,,,,,5,2021-06-07T18:56:00Z,2021-06-26T22:24:54Z,2021-06-26T22:24:54Z,CONTRIBUTOR,simonw/datasette/pulls/1368,"Note that this is a WORK IN PROGRESS!

This PR adds the following plugin hook:

    get_metadata(
      datasette=self, key=key, database=database, table=table,
      fallback=fallback
    )

This gets called when we're building our metdata for the rest of the system to use. Datasette merges whatever the plugins return with any local metadata (from metadata.yml/yaml/json) allowing for a live-editable dynamic Datasette. __A major design consideration is this: should Datasette perform the metadata merge? Or should Datasette allow plugins to perform any modifications themselves?__

As a security precation, local meta is *not* overwritable by plugin hooks. The workflow for transitioning to live-meta would be to load the plugin with the full metadata.yaml and save. Then remove the parts of the metadata that you want to be able to change from the file.

I have a WIP dynamic configuration plugin here, for reference: https://github.com/next-LI/datasette-live-config/",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1368/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
913900374,MDU6SXNzdWU5MTM5MDAzNzQ=,1369,Don't show foreign key IDs twice if no label,9599,simonw,open,0,,,,,1,2021-06-07T19:47:02Z,2021-06-07T19:47:24Z,,OWNER,,"![B5B54D94-A768-4544-A88D-CDCAB417CD3C](https://user-images.githubusercontent.com/9599/121078979-6e9d0600-c78e-11eb-8b70-20e6d29b48b1.jpeg)
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1369/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
275415799,MDU6SXNzdWUyNzU0MTU3OTk=,137,Ability to combine multiple SQL queries on a single graph,9599,simonw,open,0,,,,,1,2017-11-20T16:26:57Z,2019-05-13T18:33:51Z,,OWNER,,This would make visualizations significantly more powerful. The interesting challenge will be around the URL design. It would be useful to be able to combine either multiple explicit SQL queries or multiple queries based on the filter string parameters passed to one or more table views.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/137/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
914130834,MDExOlB1bGxSZXF1ZXN0NjY0MDcyMDQ2,1370,Ensure db.path is a string before trying to insert into internal database,25778,eyeseast,closed,0,,,,,2,2021-06-08T01:16:48Z,2021-06-21T15:57:39Z,2021-06-21T15:57:39Z,CONTRIBUTOR,simonw/datasette/pulls/1370,"Fixes #1365 

This is the simplest possible fix, with a test that will fail without it. There are a bunch of places where `db.path` is getting converted to and from a `Path` type, so this fix errs on the side of calling `str(db.path)` right before it's inserted.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1370/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
915455228,MDU6SXNzdWU5MTU0NTUyMjg=,1371,Menu plugin hooks should include the request,9599,simonw,closed,0,,,,,1,2021-06-08T20:23:35Z,2021-06-10T04:46:01Z,2021-06-10T04:46:01Z,OWNER,,"https://docs.datasette.io/en/stable/plugin_hooks.html#menu-links-datasette-actor

- `menu_links(datasette, actor)`
- `table_actions(datasette, actor, database, table)`
- `database_actions(datasette, actor, database)`

All three of these should optionally also accept the `request` object. This would allow them to take into account additional cookies, `Authorization` headers or the current request URL (including the domain/subdomain) - or even access `request.scope` for extra context that might have been passed down from ASGI middleware.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1371/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
915488244,MDU6SXNzdWU5MTU0ODgyNDQ=,1372,"Add section to ""writing plugins"" about security, e.g. avoiding XSS",9599,simonw,open,0,,,,,0,2021-06-08T20:49:33Z,2021-06-08T20:49:46Z,,OWNER,,https://docs.datasette.io/en/stable/writing_plugins.html should have tips on writing secure plugins.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1372/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
916183914,MDExOlB1bGxSZXF1ZXN0NjY1ODkyMzEz,1373,"Update trustme requirement from <0.8,>=0.7 to >=0.7,<0.9",49699333,dependabot[bot],closed,0,,,,,1,2021-06-09T13:09:44Z,2021-06-13T15:38:47Z,2021-06-13T15:38:47Z,CONTRIBUTOR,simonw/datasette/pulls/1373,"Updates the requirements on [trustme](https://github.com/python-trio/trustme) to permit the latest version.
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/python-trio/trustme/commit/f9e13e03963260b32cab03f4e4c25cc515ddfad3""><code>f9e13e0</code></a> Release 0.8.0</li>
<li><a href=""https://github.com/python-trio/trustme/commit/4ae4435f4477dc8433f8f6d51cdb891652a5a228""><code>4ae4435</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/python-trio/trustme/issues/304"">#304</a> from python-trio/dependabot/add-v2-config-file</li>
<li><a href=""https://github.com/python-trio/trustme/commit/87679026732cc441ac87c4415cf1a2bd6547116c""><code>8767902</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/python-trio/trustme/issues/327"">#327</a> from graingert/test-on-py310</li>
<li><a href=""https://github.com/python-trio/trustme/commit/6abfddd5f3139bde9ee4ba0f848fb7c84c9ac54c""><code>6abfddd</code></a> Merge branch 'master' of github.com:python-trio/trustme into test-on-py310</li>
<li><a href=""https://github.com/python-trio/trustme/commit/51d3bdfb0d874ba52d59a84e10f2c12741de1046""><code>51d3bdf</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/python-trio/trustme/issues/328"">#328</a> from tiran/correct_ku_eku</li>
<li><a href=""https://github.com/python-trio/trustme/commit/034fb3aad7890ac6e4cb24d821413bd790a84b35""><code>034fb3a</code></a> retry codecov more</li>
<li><a href=""https://github.com/python-trio/trustme/commit/53e121d4f2746333cedb412cabd510549e677ba6""><code>53e121d</code></a> try codecov harder</li>
<li><a href=""https://github.com/python-trio/trustme/commit/c1e7923784e3282aa63c28a1afe2c7960d80390f""><code>c1e7923</code></a> require codecov in ci</li>
<li><a href=""https://github.com/python-trio/trustme/commit/e3ac2d61a5ab902cdeac1c95dfc0a49c81c5d290""><code>e3ac2d6</code></a> Update tests/test_trustme.py</li>
<li><a href=""https://github.com/python-trio/trustme/commit/496dca6f2146268058b300a7fc24d4bdb90c5c33""><code>496dca6</code></a> close the wrapped sockets to prevent Unraisable ResourceWarnings</li>
<li>Additional commits viewable in <a href=""https://github.com/python-trio/trustme/compare/v0.7.0...v0.8.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1373/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
918730335,MDExOlB1bGxSZXF1ZXN0NjY4MTI5NDQx,1374,Bump black from 21.5b2 to 21.6b0,49699333,dependabot[bot],closed,0,,,,,1,2021-06-11T13:07:39Z,2021-06-13T15:33:23Z,2021-06-13T15:33:22Z,CONTRIBUTOR,simonw/datasette/pulls/1374,"Bumps [black](https://github.com/psf/black) from 21.5b2 to 21.6b0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.6b0</h2>
<h2><em>Black</em></h2>
<ul>
<li>Fix failure caused by <code>fmt: skip</code> and indentation (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2281"">#2281</a>)</li>
<li>Account for += assignment when deciding whether to split string (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2312"">#2312</a>)</li>
<li>Correct max string length calculation when there are string operators (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2292"">#2292</a>)</li>
<li>Fixed option usage when using the <code>--code</code> flag (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2259"">#2259</a>)</li>
<li>Do not call <code>uvloop.install()</code> when <em>Black</em> is used as a library (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2303"">#2303</a>)</li>
<li>Added <code>--required-version</code> option to require a specific version to be running (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2300"">#2300</a>)</li>
<li>Fix incorrect custom breakpoint indices when string group contains fake f-strings
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/2311"">#2311</a>)</li>
<li>Fix regression where <code>R</code> prefixes would be lowercased for docstrings (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2285"">#2285</a>)</li>
<li>Fix handling of named escapes (<code>\N{...}</code>) when <code>--experimental-string-processing</code> is
used (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2319"">#2319</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>21.6b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Fix failure caused by <code>fmt: skip</code> and indentation (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2281"">#2281</a>)</li>
<li>Account for += assignment when deciding whether to split string (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2312"">#2312</a>)</li>
<li>Correct max string length calculation when there are string operators (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2292"">#2292</a>)</li>
<li>Fixed option usage when using the <code>--code</code> flag (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2259"">#2259</a>)</li>
<li>Do not call <code>uvloop.install()</code> when <em>Black</em> is used as a library (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2303"">#2303</a>)</li>
<li>Added <code>--required-version</code> option to require a specific version to be running (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2300"">#2300</a>)</li>
<li>Fix incorrect custom breakpoint indices when string group contains fake f-strings
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/2311"">#2311</a>)</li>
<li>Fix regression where <code>R</code> prefixes would be lowercased for docstrings (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2285"">#2285</a>)</li>
<li>Fix handling of named escapes (<code>\N{...}</code>) when <code>--experimental-string-processing</code> is
used (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2319"">#2319</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.5b2&new-version=21.6b0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1374/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
919508498,MDU6SXNzdWU5MTk1MDg0OTg=,1375,JSON export dumps JSON fields as TEXT,4068,frafra,closed,0,,,,,2,2021-06-12T09:45:08Z,2021-06-14T09:41:59Z,2021-06-13T15:37:58Z,NONE,,"Hi!
When a user tries to export data as JSON, I would expect to see the value of JSON columns represented as JSON instead of being rendered as a string. What do you think?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1375/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
919822817,MDU6SXNzdWU5MTk4MjI4MTc=,1376,Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns),1726460,jcgregorio,open,0,,,,,3,2021-06-13T15:25:51Z,2021-06-13T15:39:37Z,,NONE,,"Trying to run datasette via the Docker container doesn't seem to work:

```
$ docker run -p 8001:8001 -v `pwd`:/mnt     datasetteproject/datasette     datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
Traceback (most recent call last):
  File ""/usr/local/bin/datasette"", line 8, in <module>
    sys.exit(cli())
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/usr/local/lib/python3.9/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/usr/local/lib/python3.9/site-packages/datasette/cli.py"", line 544, in serve
    asyncio.get_event_loop().run_until_complete(check_databases(ds))
  File ""/usr/local/lib/python3.9/asyncio/base_events.py"", line 642, in run_until_complete
    return future.result()
  File ""/usr/local/lib/python3.9/site-packages/datasette/cli.py"", line 584, in check_databases
    await database.execute_fn(check_connection)
  File ""/usr/local/lib/python3.9/site-packages/datasette/database.py"", line 155, in execute_fn
    return await asyncio.get_event_loop().run_in_executor(
  File ""/usr/local/lib/python3.9/concurrent/futures/thread.py"", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File ""/usr/local/lib/python3.9/site-packages/datasette/database.py"", line 153, in in_thread
    return fn(conn)
  File ""/usr/local/lib/python3.9/site-packages/datasette/utils/__init__.py"", line 892, in check_connection
    for r in conn.execute(
sqlite3.DatabaseError: malformed database schema (generated_columns) - near ""AS"": syntax error
```

I have confirmed that the downloaded `fixtures.db` database is fine:

```
[skia-public] jcgregorio@jcgregorio840 ~/Downloads 
$ sqlite3 fixtures.db 
SQLite version 3.34.1 2021-01-20 14:10:07
Enter "".help"" for usage hints.
sqlite> pragma integrity_check;
ok
sqlite> 

```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1376/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
920884085,MDU6SXNzdWU5MjA4ODQwODU=,1377,Mechanism for plugins to exclude certain paths from CSRF checks,9599,simonw,closed,0,,,,,3,2021-06-15T00:48:20Z,2021-06-23T22:51:33Z,2021-06-23T22:51:33Z,OWNER,,I need this for a plugin I'm building that offers a POST API.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1377/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
923910375,MDExOlB1bGxSZXF1ZXN0NjcyNjIwMTgw,1378,"Update pytest-xdist requirement from <2.3,>=2.2.1 to >=2.2.1,<2.4",49699333,dependabot[bot],closed,0,,,,,1,2021-06-17T13:11:56Z,2021-06-20T00:17:07Z,2021-06-20T00:17:06Z,CONTRIBUTOR,simonw/datasette/pulls/1378,"Updates the requirements on [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst"">pytest-xdist's changelog</a>.</em></p>
<blockquote>
<h1>pytest-xdist 2.3.0 (2021-06-16)</h1>
<h2>Deprecations and Removals</h2>
<ul>
<li><code>[#654](https://github.com/pytest-dev/pytest-xdist/issues/654) &lt;https://github.com/pytest-dev/pytest-xdist/issues/654&gt;</code>_: Python 3.5 is no longer supported.</li>
</ul>
<h2>Features</h2>
<ul>
<li>
<p><code>[#646](https://github.com/pytest-dev/pytest-xdist/issues/646) &lt;https://github.com/pytest-dev/pytest-xdist/issues/646&gt;</code>_: Add <code>--numprocesses=logical</code> flag, which automatically uses the number of logical CPUs available, instead of physical CPUs with <code>auto</code>.</p>
<p>This is very useful for test suites which are not CPU-bound.</p>
</li>
<li>
<p><code>[#650](https://github.com/pytest-dev/pytest-xdist/issues/650) &lt;https://github.com/pytest-dev/pytest-xdist/issues/650&gt;</code>_: Added new <code>pytest_handlecrashitem</code> hook to allow handling and rescheduling crashed items.</p>
</li>
</ul>
<h2>Bug Fixes</h2>
<ul>
<li>
<p><code>[#421](https://github.com/pytest-dev/pytest-xdist/issues/421) &lt;https://github.com/pytest-dev/pytest-xdist/issues/421&gt;</code>_: Copy the parent process sys.path into local workers, to work around execnet's python -c adding the current directory to sys.path.</p>
</li>
<li>
<p><code>[#638](https://github.com/pytest-dev/pytest-xdist/issues/638) &lt;https://github.com/pytest-dev/pytest-xdist/issues/638&gt;</code>_: Fix issue caused by changing the branch name of the pytest repository.</p>
</li>
</ul>
<h2>Trivial Changes</h2>
<ul>
<li>
<p><code>[#592](https://github.com/pytest-dev/pytest-xdist/issues/592) &lt;https://github.com/pytest-dev/pytest-xdist/issues/592&gt;</code>_: Replace master with controller where ever possible.</p>
</li>
<li>
<p><code>[#643](https://github.com/pytest-dev/pytest-xdist/issues/643) &lt;https://github.com/pytest-dev/pytest-xdist/issues/643&gt;</code>_: Use 'main' to refer to pytest default branch in tox env names.</p>
</li>
</ul>
<h1>pytest-xdist 2.2.1 (2021-02-09)</h1>
<h2>Bug Fixes</h2>
<ul>
<li><code>[#623](https://github.com/pytest-dev/pytest-xdist/issues/623) &lt;https://github.com/pytest-dev/pytest-xdist/issues/623&gt;</code>_: Gracefully handle the pending deprecation of Node.fspath by using config.rootpath for topdir.</li>
</ul>
<h1>pytest-xdist 2.2.0 (2020-12-14)</h1>
<h2>Features</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/fe57b39563aa80be7605302baa090069a172037a""><code>fe57b39</code></a> fixup: add release title underline for 2.3.0</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/26e7d953f41c38c6aca209c63e2f82140def9255""><code>26e7d95</code></a> prepare release 2.3.0</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/b02a6db771ef2ca2e6df2f6c94143cbeeefc7f1b""><code>b02a6db</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/667"">#667</a> from graingert/fix-sys-path</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/b0722675e033b346382475b1ab2e0e32cdb07ae6""><code>b072267</code></a> add newsfile</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/881cc4827171e382cf2254687bc63df0ecb8c189""><code>881cc48</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/672"">#672</a> from pytest-dev/pre-commit-ci-update-config</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/958679e9a717fe4cea93aac989f072deb4962f08""><code>958679e</code></a> [pre-commit.ci] pre-commit autoupdate</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/7f07d50808eec48febcae8308a8194cc247eb40f""><code>7f07d50</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/646"">#646</a> from kroeschl/numprocesses-logical</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/fb518def5523b4ae0ae81f1dcea31aee8f6b3dea""><code>fb518de</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/669"">#669</a> from pytest-dev/pre-commit-ci-update-config</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/0b14d9289482cdbfe85d73c47b58ecc89ecf1c06""><code>0b14d92</code></a> [pre-commit.ci] pre-commit autoupdate</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/02f971d45ff40313d4b31c58396bbd546a120a2b""><code>02f971d</code></a> swap docstring</li>
<li>Additional commits viewable in <a href=""https://github.com/pytest-dev/pytest-xdist/compare/v2.2.1...v2.3.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1378/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
924203783,MDU6SXNzdWU5MjQyMDM3ODM=,1379,Idea: ?_end=1 option for streaming CSV responses,9599,simonw,open,0,,,,,0,2021-06-17T18:11:21Z,2021-06-17T18:11:30Z,,OWNER,,"As discussed in this thread: https://twitter.com/simonw/status/1405554676993433605 - one of the disadvantages of Datasette's streaming CSV feature is that it's hard to tell if you got the whole file or if the connection ended early - or if an error occurred.

Idea: offer an optional `?_end=1` parameter which, if enabled, adds a single row to the end of the CSV file that looks like this:

`END,,,,,,,,,`

For however many columns the CSV file usually has.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1379/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
275476839,MDU6SXNzdWUyNzU0NzY4Mzk=,138,"Per-database and per-table metadata, probably using data-package",9599,simonw,closed,0,,,,,1,2017-11-20T19:50:10Z,2017-12-10T03:08:36Z,2017-12-10T03:08:26Z,OWNER,,"Ability to annotate databases and tables with extra metadata describing their purpose, providing source and licensing information and describing individual columns.

http://frictionlessdata.io/specs/data-package/ looks like a great format for this, see #105 ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/138/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
924748955,MDU6SXNzdWU5MjQ3NDg5NTU=,1380,Serve all db files in a folder,193463,stratosgear,open,0,,,,,5,2021-06-18T10:03:32Z,2021-11-13T08:09:11Z,,NONE,,"I tried to get the `serve` command to serve all the .db files in the `/mnt` folder but is seems that the server does not refresh the list of files.

In more detail:

* Starting datasette as a docker container with: 
```
docker run -p 8001:8001 -v `pwd`:/mnt \
    datasetteproject/datasette \
    datasette -p 8001 -h 0.0.0.0 /mnt
```

* Datasette correctly serves all the *.db files found in the /mnt folder 
* When the server is running, if I copy a new file in the $PWD folder, Datasette does not seem to see the new files, forcing me to restart Docker.

Is there an option/setting that I overlooked, or is this something missing?

BTW, the `--reload` setting, although at first glance is what you think you need, does not seem to do anything in regards of seeing all *.db files.

Thanks!",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1380/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
925406964,MDU6SXNzdWU5MjU0MDY5NjQ=,1382,Datasette with Glitch - is it possible to use CSV with ISO-8859-1 encoding?,23701514,reichaves,closed,0,,,,,1,2021-06-19T14:37:20Z,2021-06-20T00:21:02Z,2021-06-20T00:20:06Z,NONE,,"Hi
Please, I used Remix on Glitch to create a project on Glitch and uploaded a CSV
But it's a CSV with ISO-8859-1 encoding (https://en.wikipedia.org/wiki/ISO/IEC_8859-1)
Is it possible for me to change the encoding to correctly visualize the data?
Example: https://emphasized-carpal-pillow.glitch.me/data/Emendas
Best",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1382/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
925491857,MDU6SXNzdWU5MjU0OTE4NTc=,1383,Improve test coverage for `inspect.py`,9599,simonw,open,0,,,,,0,2021-06-20T00:22:43Z,2021-06-20T00:22:49Z,,OWNER,,https://codecov.io/gh/simonw/datasette/src/main/datasette/inspect.py shows only 36% coverage for that module at the moment.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1383/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
930807135,MDU6SXNzdWU5MzA4MDcxMzU=,1384,Plugin hook for dynamic metadata,9599,simonw,open,0,,,,,22,2021-06-26T22:36:03Z,2022-03-14T00:36:42Z,,OWNER,,"@brandonrobertz contributed an implementation of this in PR #1368, which I just merged. Opening this ticket to track further work on this before it goes out in a Datasette release (likely preceded by an alpha).",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1384/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
930855052,MDExOlB1bGxSZXF1ZXN0Njc4NDU5NTU0,1385,Fix + improve get_metadata plugin hook docs,2670795,brandonrobertz,closed,0,,,,,1,2021-06-27T05:43:20Z,2021-09-13T18:53:11Z,2021-09-13T18:53:11Z,CONTRIBUTOR,simonw/datasette/pulls/1385,"This fixes documentation inaccuracies and adds a disclaimer about the signature of the `get_metadata` hook.

Addresses the following comments:
- https://github.com/simonw/datasette/issues/1384#issuecomment-869069926
- https://github.com/simonw/datasette/issues/1384#issuecomment-869075368",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1385/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
931557895,MDExOlB1bGxSZXF1ZXN0Njc5MDM1ODQ3,1386,"Update asgiref requirement from <3.4.0,>=3.2.10 to >=3.2.10,<3.5.0",49699333,dependabot[bot],closed,0,,,,,1,2021-06-28T13:13:07Z,2021-07-11T01:36:19Z,2021-07-11T01:36:18Z,CONTRIBUTOR,simonw/datasette/pulls/1386,"Updates the requirements on [asgiref](https://github.com/django/asgiref) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/django/asgiref/blob/main/CHANGELOG.txt"">asgiref's changelog</a>.</em></p>
<blockquote>
<h2>3.4.0 (2021-06-27)</h2>
<ul>
<li>
<p>Calling sync_to_async directly from inside itself (which causes a deadlock
when in the default, thread-sensitive mode) now has deadlock detection.</p>
</li>
<li>
<p>asyncio usage has been updated to use the new versions of get_event_loop,
ensure_future, wait and gather, avoiding deprecation warnings in Python 3.10.
Python 3.6 installs continue to use the old versions; this is only for 3.7+</p>
</li>
<li>
<p>sync_to_async and async_to_sync now have improved type hints that pass
through the underlying function type correctly.</p>
</li>
<li>
<p>All Websocket* types are now spelled WebSocket, to match our specs and the
official spelling. The old names will work until release 3.5.0, but will
raise deprecation warnings.</p>
</li>
<li>
<p>The typing for WebSocketScope and HTTPScope's <code>extensions</code> key has been
fixed.</p>
</li>
</ul>
<h2>3.3.4 (2021-04-06)</h2>
<ul>
<li>The async_to_sync type error is now a warning due the high false negative
rate when trying to detect coroutine-returning callables in Python.</li>
</ul>
<h2>3.3.3 (2021-04-06)</h2>
<ul>
<li>The sync conversion functions now correctly detect functools.partial and other
wrappers around async functions on earlier Python releases.</li>
</ul>
<h2>3.3.2 (2021-04-05)</h2>
<ul>
<li>
<p>SyncToAsync now takes an optional &quot;executor&quot; argument if you want to supply
your own executor rather than using the built-in one.</p>
</li>
<li>
<p>async_to_sync and sync_to_async now check their arguments are functions of
the correct type.</p>
</li>
<li>
<p>Raising CancelledError inside a SyncToAsync function no longer stops a future
call from functioning.</p>
</li>
<li>
<p>ThreadSensitive now provides context hooks/override options so it can be
made to be sensitive in a unit smaller than threads (e.g. per request)</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/django/asgiref/commits"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1386/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
935930820,MDU6SXNzdWU5MzU5MzA4MjA=,1387,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,9599,simonw,closed,0,,,,,8,2021-07-02T16:58:25Z,2021-07-02T17:58:23Z,2021-07-02T17:33:05Z,OWNER,,"Reported in the wild on https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-3d4a4b7/bib?_facet=bib_level_callnumber - the ""next page"" link links to https://127.0.0.1:8010/collection-analysis/current_collection-3d4a4b7/bib?_facet=bib_level_callnumber&_next=100

That installation uses `""base_url"": ""/collection-analysis/""`

Weirdly all of the other links on that page - to facet results, sort orders, row permalinks etc - work fine. It's JUST the `next_url` one that is broken.

Also broken in their JSON: https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-3d4a4b7/bib.json?_size=1 returns

```json
  ""suggested_facets"": [],
  ""next"": ""1"",
  ""next_url"": ""https://127.0.0.1:8010/collection-analysis/current_collection-3d4a4b7/bib.json?_size=1&_next=1"",
  ""private"": false,
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1387/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
939051549,MDU6SXNzdWU5MzkwNTE1NDk=,1388,Serve using UNIX domain socket,80737,aslakr,closed,0,,,,,13,2021-07-07T16:13:37Z,2021-07-11T01:18:38Z,2021-07-10T23:38:32Z,CONTRIBUTOR,,Would it be possible to make datasette serve using UNIX domain socket similar to Uvicorn's ``--uds``?,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1388/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
940077168,MDU6SXNzdWU5NDAwNzcxNjg=,1389,"""searchmode"": ""raw"" in table metadata",9599,simonw,closed,0,,,,,6,2021-07-08T17:32:10Z,2021-07-10T18:33:13Z,2021-07-10T18:33:13Z,OWNER,,"> http://localhost:8001/index/summary?_search=language%3Aeng&_sort=title&_searchmode=raw
>
> But I'm not able to manage it in the metadata file. Here is mine (note that the sort column is taken into account)
> Here it is:
>
> ```
> {
>     ""databases"": {
>       ""index"": {
>         ""tables"": {
>             ""summary"": {
>                 ""sort"": ""title"",
>                 ""searchmode"": ""raw""
>             }
>         }
>       }
>     }
>   }

_Originally posted by @Krazybug in https://github.com/simonw/datasette/issues/759#issuecomment-624860451_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1389/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275493851,MDU6SXNzdWUyNzU0OTM4NTE=,139,Build a visualization plugin for Vega,9599,simonw,closed,0,,,,,2,2017-11-20T20:47:41Z,2018-07-10T17:48:18Z,2018-07-10T17:48:18Z,OWNER,,"https://vega.github.io/vega/examples/population-pyramid/ for example looks pretty easy to hook up to Datasette.

Depends on #14 ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/139/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
940891698,MDU6SXNzdWU5NDA4OTE2OTg=,1390,Mention restarting systemd in documentation,9599,simonw,closed,0,,,,,2,2021-07-09T16:05:15Z,2021-07-09T16:32:57Z,2021-07-09T16:32:33Z,OWNER,,"https://docs.datasette.io/en/stable/deploying.html#running-datasette-using-systemd

Need to clarify that if you add a new database or change metadata you need to restart systemd.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1390/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
941300946,MDU6SXNzdWU5NDEzMDA5NDY=,1391,Stop using generated columns in fixtures.db,9599,simonw,closed,0,,,,,5,2021-07-10T18:26:11Z,2021-07-10T19:26:58Z,2021-07-10T19:26:00Z,OWNER,,"Refs #1376 - but I also keep running into this myself, where I try to run something against `fixtures.db` and get this confusing error:

    sqlite3.DatabaseError: malformed database schema (generated_columns) - near ""AS"": syntax error

I'm going to stop using generated columns in `fixtures.db` and instead dynamically generate the generated column table for the duration of the relevant test.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1391/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
941403676,MDExOlB1bGxSZXF1ZXN0Njg3Mjk4MTEy,1392,Update deploying.rst,80737,aslakr,closed,0,,,,,1,2021-07-11T08:43:19Z,2021-07-13T17:42:31Z,2021-07-13T17:42:27Z,CONTRIBUTOR,simonw/datasette/pulls/1392,Use same base url for Apache as in the example,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1392/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
941412189,MDExOlB1bGxSZXF1ZXN0Njg3MzA0MjQy,1393,Update deploying.rst,80737,aslakr,closed,0,,,,,1,2021-07-11T09:32:16Z,2021-07-13T18:32:49Z,2021-07-13T18:32:49Z,CONTRIBUTOR,simonw/datasette/pulls/1393,"Example on how to use Unix domain socket option on Apache. Not testet.

(Usually I would have used [`ProxyPassReverse`](https://httpd.apache.org/docs/current/mod/mod_proxy.html#proxypassreverse) in combination with `ProxyPass` , i.e.

```apache
ProxyPass /my-datasette/ http://127.0.0.1:8009/my-datasette/
ProxyPassReverse /my-datasette/ http://127.0.0.1:8009/my-datasette/
```

and

```apache
ProxyPass /my-datasette/ unix:/tmp/datasette.sock|http://localhost/my-datasette/
ProxyPassReverse /my-datasette/ unix:/tmp/datasette.sock|http://localhost/my-datasette/
```
)",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1393/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
944870799,MDU6SXNzdWU5NDQ4NzA3OTk=,1394,Big performance boost on faceting: skip the inner order by,9599,simonw,closed,0,,,,,4,2021-07-14T23:32:29Z,2021-07-16T02:23:32Z,2021-07-15T00:05:50Z,OWNER,,"I just noticed something that could make for a huge performance improvement in faceting.

The default query used by Datasette when faceting looks like this:
```sql
select
  country_long,
  count(*)
from (
  select * from [global-power-plants] order by rowid
)
where
  country_long is not null
group by
  country_long
order by
  count(*) desc
```
Here it takes 53ms: https://global-power-plants.datasettes.com/global-power-plants?sql=select%0D%0A++country_long%2C%0D%0A++count%28*%29%0D%0Afrom+%28%0D%0A++select+*+from+%5Bglobal-power-plants%5D+order+by+rowid%0D%0A%29%0D%0Awhere%0D%0A++country_long+is+not+null%0D%0Agroup+by%0D%0A++country_long%0D%0Aorder+by%0D%0A++count%28*%29+desc

Note that there's a `order by rowid` in there which isn't necessary - the order on that inner query doesn't matter since we're grouping and counting.

I had assumed SQLite would optimize this away - but it turns out it doesn't! Consider this version of the query, with that pointless order by removed:
```
select
  country_long,
  count(*)
from (
  select * from [global-power-plants]
)
where
  country_long is not null
group by
  country_long
order by
  count(*) desc
```
https://global-power-plants.datasettes.com/global-power-plants?sql=select%0D%0A++country_long%2C%0D%0A++count%28*%29%0D%0Afrom+%28%0D%0A++select+*+from+%5Bglobal-power-plants%5D%0D%0A%29%0D%0Awhere%0D%0A++country_long+is+not+null%0D%0Agroup+by%0D%0A++country_long%0D%0Aorder+by%0D%0A++count%28*%29+desc runs in 7.2ms!

I tried this optimization on a table with 2.5m rows in it - without the optimization it took 5 seconds, with the optimization it took 450ms. So this is a very significant improvement!",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1394/reactions"", ""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
944903881,MDU6SXNzdWU5NDQ5MDM4ODE=,1396,"""invalid reference format"" publishing Docker image",9599,simonw,closed,0,,,,,9,2021-07-15T01:02:07Z,2021-10-19T08:10:26Z,2021-07-15T19:47:25Z,OWNER,,"Error ocurred at the end of the publish flow for Datasette 0.58: https://github.com/simonw/datasette/runs/3072216421
```
Removing intermediate container cf32b9440907
 ---> dfd6985b2afc
Successfully built dfd6985b2afc
Successfully tagged ***/datasette:0.58
invalid reference format
Error: Process completed with exit code 1.
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1396/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
946553953,MDExOlB1bGxSZXF1ZXN0NjkxNzA3NDA5,1397,"Fix for race condition in refresh_schemas(), closes #1231",9599,simonw,closed,0,,,,,0,2021-07-16T19:44:43Z,2021-07-16T19:45:00Z,2021-07-16T19:44:58Z,OWNER,simonw/datasette/pulls/1397,,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1397/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
947044667,MDU6SXNzdWU5NDcwNDQ2Njc=,1398,Documentation on using Datasette as a library,9599,simonw,open,0,,,,,1,2021-07-18T14:15:27Z,2021-07-30T03:21:49Z,,OWNER,,"Instantiating `Datasette()` directly is an increasingly interesting pattern. I do it in tests all the time, but thanks to `datasette.client` there are plenty of neat things you can do with it in a library context.

Maybe support `from datasette import Datasette` for this.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1398/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
947596222,MDExOlB1bGxSZXF1ZXN0NjkyNTU3Mzgx,1399,Multiple sort,87192257,jgryko5,open,0,,,,,0,2021-07-19T12:20:14Z,2021-07-19T12:20:14Z,,FIRST_TIME_CONTRIBUTOR,simonw/datasette/pulls/1399,"Closes #197.
I have added support for sorting by multiple parameters as mentioned in the issue above, and together with that, a suggestion on how to implement such sorting in the user interface.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1399/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
267707940,MDU6SXNzdWUyNjc3MDc5NDA=,14,Datasette Plugins,9599,simonw,closed,0,,,,,22,2017-10-23T15:15:28Z,2019-05-13T18:58:20Z,2019-05-13T18:58:19Z,OWNER,,"It would be neat if additional functionality could be opted-in to the system in the form of easy-to-add plugins, hosted as separate packages. First example: a Google Analytics plugin, which adds GA tracking code with your tracking ID to the web interface for your dataset.

This may be an opportunity to experiment with entry points: http://amir.rachum.com/blog/2017/07/28/python-entry-points/",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/14/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275755475,MDU6SXNzdWUyNzU3NTU0NzU=,140,Heatmap visualization plugin,9599,simonw,open,0,,,,,2,2017-11-21T15:34:23Z,2019-05-13T18:33:51Z,,OWNER,,Could use https://github.com/scottbedard/svelte-heatmap,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/140/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
947640902,MDExOlB1bGxSZXF1ZXN0NjkyNTk2MDA2,1400,Bump black from 21.6b0 to 21.7b0,49699333,dependabot[bot],closed,0,,,,,1,2021-07-19T13:13:41Z,2021-08-25T01:29:56Z,2021-08-25T01:29:55Z,CONTRIBUTOR,simonw/datasette/pulls/1400,"Bumps [black](https://github.com/psf/black) from 21.6b0 to 21.7b0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.7b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Configuration files using TOML features higher than spec v0.5.0 are now supported
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/2301"">#2301</a>)</li>
<li>Add primer support and test for code piped into black via STDIN (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2315"">#2315</a>)</li>
<li>Fix internal error when <code>FORCE_OPTIONAL_PARENTHESES</code> feature is enabled (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2332"">#2332</a>)</li>
<li>Accept empty stdin (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2346"">#2346</a>)</li>
<li>Provide a more useful error when parsing fails during AST safety checks (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2304"">#2304</a>)</li>
</ul>
<h3>Docker</h3>
<ul>
<li>Add new <code>latest_release</code> tag automation to follow latest black release on docker
images (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2374"">#2374</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>The vim plugin now searches upwards from the directory containing the current buffer
instead of the current working directory for pyproject.toml. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1871"">#1871</a>)</li>
<li>The vim plugin now reads the correct string normalization option in pyproject.toml
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/1869"">#1869</a>)</li>
<li>The vim plugin no longer crashes Black when there's boolean values in pyproject.toml
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/1869"">#1869</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>21.7b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Configuration files using TOML features higher than spec v0.5.0 are now supported
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/2301"">#2301</a>)</li>
<li>Add primer support and test for code piped into black via STDIN (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2315"">#2315</a>)</li>
<li>Fix internal error when <code>FORCE_OPTIONAL_PARENTHESES</code> feature is enabled (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2332"">#2332</a>)</li>
<li>Accept empty stdin (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2346"">#2346</a>)</li>
<li>Provide a more useful error when parsing fails during AST safety checks (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2304"">#2304</a>)</li>
</ul>
<h3>Docker</h3>
<ul>
<li>Add new <code>latest_release</code> tag automation to follow latest black release on docker
images (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2374"">#2374</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>The vim plugin now searches upwards from the directory containing the current buffer
instead of the current working directory for pyproject.toml. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1871"">#1871</a>)</li>
<li>The vim plugin now reads the correct string normalization option in pyproject.toml
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/1869"">#1869</a>)</li>
<li>The vim plugin no longer crashes Black when there's boolean values in pyproject.toml
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/1869"">#1869</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.6b0&new-version=21.7b0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1400/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
950664971,MDU6SXNzdWU5NTA2NjQ5NzE=,1401,unordered list is not rendering bullet points in description_html on database page,536941,fgregg,open,0,,,,,2,2021-07-22T13:24:18Z,2021-10-23T13:09:10Z,,CONTRIBUTOR,,"Thanks for this tremendous package, @simonw!

In the `description_html` for a database, I [have an unordered list](https://github.com/labordata/warehouse/blob/fcea4502e5b615b0eb3e0bdcb45ec634abe20bb6/warehouse_metadata.yml#L19-L22).

However, on the database page on the deployed site, it is not rendering this as a bulleted list.

![Screenshot 2021-07-22 at 09-21-51 nlrb](https://user-images.githubusercontent.com/536941/126645923-2777b7f1-fd4c-4d2d-af70-a35e49a07675.png)

Page here: https://labordata-warehouse.herokuapp.com/nlrb-9da4ae5

The documentation gives an [example of using an unordered list](https://docs.datasette.io/en/stable/metadata.html#using-yaml-for-metadata) in a `description_html`, so I expected this will work.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1401/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
951185411,MDU6SXNzdWU5NTExODU0MTE=,1402,feature request: social meta tags,536941,fgregg,open,0,,,,,2,2021-07-23T01:57:23Z,2021-07-26T19:31:41Z,,CONTRIBUTOR,,"it would be very nice if the twitter, slack, and other social media could make rich cards when people post a link to a datasette instance

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1402/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
953218043,MDU6SXNzdWU5NTMyMTgwNDM=,1403,Labels explaining what hidden tables are for,9599,simonw,open,0,,,,,0,2021-07-26T19:29:22Z,2022-03-21T22:20:37Z,,OWNER,,"A reasonable question: ""What are those hidden tables for?""

This could be answered by adding a small piece of explanatory text to each table - based on if it's related to FTS or to SpatiaLite or configured to be hidden for some other reason.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1403/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
953352015,MDU6SXNzdWU5NTMzNTIwMTU=,1404,`register_routes()` hook should take `datasette` argument,9599,simonw,closed,0,,,,,1,2021-07-26T23:00:33Z,2021-07-26T23:27:07Z,2021-07-26T23:26:00Z,OWNER,,Currently that plugin hook takes no arguments at all. This means it's not possible to conditionally register routes based on Datasette plugin configuration.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1404/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
955316250,MDU6SXNzdWU5NTUzMTYyNTA=,1405,utils.parse_metadata() should be a documented internal function,9599,simonw,closed,0,,,,,3,2021-07-28T23:51:39Z,2021-07-29T23:33:30Z,2021-07-29T23:30:24Z,OWNER,,Because it's used by this plugin: https://github.com/simonw/datasette-remote-metadata,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1405/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
956303470,MDU6SXNzdWU5NTYzMDM0NzA=,1406,Tests failing with FileNotFoundError in runner.isolated_filesystem,9599,simonw,closed,0,,,,,8,2021-07-30T00:39:00Z,2021-07-31T18:56:35Z,2021-07-31T18:56:35Z,OWNER,,"e.g. https://github.com/simonw/datasette/runs/3197141955

I've seen this error before, but I don't yet have a good workaround for it.

```
    @contextlib.contextmanager
    def isolated_filesystem(
        self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None
    ) -> t.Iterator[str]:
        """"""A context manager that creates a temporary directory and
        changes the current working directory to it. This isolates tests
        that affect the contents of the CWD to prevent them from
        interfering with each other.
    
        :param temp_dir: Create the temporary directory under this
            directory. If given, the created directory is not removed
            when exiting.
    
        .. versionchanged:: 8.0
            Added the ``temp_dir`` parameter.
        """"""
>       cwd = os.getcwd()
E       FileNotFoundError: [Errno 2] No such file or directory

/opt/hostedtoolcache/Python/3.6.14/x64/lib/python3.6/site-packages/click/testing.py:466: FileNotFoundError
=========================== short test summary info ============================
FAILED tests/test_publish_cloudrun.py::test_publish_cloudrun_apt_get_install
FAILED tests/test_publish_cloudrun.py::test_publish_cloudrun_extra_options[---setting force_https_urls on]
FAILED tests/test_publish_cloudrun.py::test_publish_cloudrun_extra_options[--setting base_url /foo---setting base_url /foo --setting force_https_urls on]
FAILED tests/test_publish_cloudrun.py::test_publish_cloudrun_extra_options[--setting force_https_urls off---setting force_https_urls off]
FAILED tests/test_publish_heroku.py::test_publish_heroku_requires_heroku - Fi...
FAILED tests/test_publish_heroku.py::test_publish_heroku_installs_plugin - Fi...
FAILED tests/test_publish_heroku.py::test_publish_heroku - FileNotFoundError:...
FAILED tests/test_publish_heroku.py::test_publish_heroku_plugin_secrets - Fil...
================== 8 failed, 920 passed in 188.22s (0:03:08) ===================
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1406/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
957298475,MDU6SXNzdWU5NTcyOTg0NzU=,1407,OSError: AF_UNIX path too long in ds_unix_domain_socket_server,9599,simonw,closed,0,,,,,2,2021-07-31T18:36:06Z,2021-07-31T19:03:44Z,2021-07-31T19:03:44Z,OWNER,,"Got this exception while working on #1406.

```
    @pytest.fixture(scope=""session"")
    def ds_unix_domain_socket_server(tmp_path_factory):
        socket_folder = tmp_path_factory.mktemp(""uds"")
        uds = str(socket_folder / ""datasette.sock"")
        ds_proc = subprocess.Popen(
            [""datasette"", ""--memory"", ""--uds"", uds],
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            cwd=tempfile.gettempdir(),
        )
        # Give the server time to start
        time.sleep(1.5)
        # Check it started successfully
>       assert not ds_proc.poll(), ds_proc.stdout.read().decode(""utf-8"")
E       AssertionError: INFO:     Started server process [48453]
E         INFO:     Waiting for application startup.
E         INFO:     Application startup complete.
E         Traceback (most recent call last):
E           File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette"", line 33, in <module>
E             sys.exit(load_entry_point('datasette', 'console_scripts', 'datasette')())
E           File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 1137, in __call__
E             return self.main(*args, **kwargs)
E           File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 1062, in main
E             rv = self.invoke(ctx)
E           File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 1668, in invoke
E             return _process_result(sub_ctx.command.invoke(sub_ctx))
E           File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 1404, in invoke
E             return ctx.invoke(self.callback, **ctx.params)
E           File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 763, in invoke
E             return __callback(*args, **kwargs)
E           File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 583, in serve
E             uvicorn.run(ds.app(), **uvicorn_kwargs)
E           File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/uvicorn/main.py"", line 393, in run
E             server.run()
E           File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/uvicorn/server.py"", line 50, in run
E             loop.run_until_complete(self.serve(sockets=sockets))
E           File ""/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/asyncio/base_events.py"", line 616, in run_until_complete
E             return future.result()
E           File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/uvicorn/server.py"", line 67, in serve
E             await self.startup(sockets=sockets)
E           File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/uvicorn/server.py"", line 133, in startup
E             server = await asyncio.start_unix_server(
E           File ""/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/asyncio/streams.py"", line 132, in start_unix_server
E             return await loop.create_unix_server(factory, path, **kwds)
E           File ""/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/asyncio/unix_events.py"", line 296, in create_unix_server
E             sock.bind(path)
E         OSError: AF_UNIX path too long
E         
E       assert not 1
E        +  where 1 = <bound method Popen.poll of <subprocess.Popen object at 0x106924af0>>()
E        +    where <bound method Popen.poll of <subprocess.Popen object at 0x106924af0>> = <subprocess.Popen object at 0x106924af0>.poll
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1407/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
957302085,MDU6SXNzdWU5NTczMDIwODU=,1408,"Review places in codebase that use os.chdir(), in particularly relating to tests",9599,simonw,open,0,,,,,2,2021-07-31T18:57:06Z,2021-07-31T19:00:32Z,,OWNER,,"> To clarify: the core problem here is that an error is thrown any time you call `os.getcwd()` but the directory you are currently in has been deleted.
>
> `runner.isolated_filesystem()` assumes that the current directory in has not been deleted. But the various temporary directory utilities in `pytest` work by creating directories and then deleting them.
>
> Maybe there's a larger problem here that I play a bit fast and loose with `os.chdir()` in both the test suite and in various lines of code in Datasette itself (in particular in the publish commands)?

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1406#issuecomment-890390198_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1408/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
957310278,MDU6SXNzdWU5NTczMTAyNzg=,1409,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting),9599,simonw,closed,0,,,3268330,Datasette 1.0,10,2021-07-31T19:48:56Z,2023-01-07T18:06:01Z,2023-01-05T00:51:31Z,OWNER,,"In 49d6d2f7b0f6cb02e25022e1c9403811f1fa0a7c as part of #813 I removed the `allow_sql` setting - on the basis that users could disable the ability to execute custom SQL queries using the new permission system instead.

I don't think this was the right decision. Disabling custom SQL is an important security capability, and explaining how to do it using permissions is significantly  more complex than letting people know they can add `--setting allow_sql off`.

So I want to bring that setting back - maybe with a different, better name - and have it modify the default for that option if the permissions system doesn't have an opinion.

That way people can still use the setting but then use permissions to allow specific signed-in users access to execute SQL.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1409/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
275814941,MDU6SXNzdWUyNzU4MTQ5NDE=,141,datasette publish can fail if /tmp is on a different device,21148,jacobian,closed,0,,,2949431,Custom templates edition,5,2017-11-21T18:28:05Z,2020-04-29T03:27:54Z,2017-12-08T16:06:36Z,CONTRIBUTOR,,"`datasette publish` uses hard links to avoid copying the db into a tmp directory. This can fail if `/tmp` is on another device, because hardlinks can't cross devices. You'll see something like this:

```
$ datasette publish heroku whatever.db
...
OSError: [Errno 18] Invalid cross-device link: '/mnt/c/Users/jacob/c/datasette/whatever.db' -> '/tmp/tmpvxq2yof6/whatever.db'
```
[In my case this is failing because I'm on a Windows machine, using WSL, so my code's on a different virtual filesystem from the Linux subsystem, Because Reasons.]

I'm not sure if it's possible to detect this (can you figure out which device `/tmp` is on?), or what the fallback should be (soft link? copy?).",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/141/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
957315684,MDU6SXNzdWU5NTczMTU2ODQ=,1410,Rename settings to `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`,9599,simonw,open,0,,,3268330,Datasette 1.0,0,2021-07-31T20:27:12Z,2021-07-31T20:27:49Z,,OWNER,,"> If I was prone to over-thinking (which I am) I'd note that `allow_facet` and `allow_download` and `allow_csv_stream` are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent.
>
> So maybe there's a future task where I introduce those as both permissions and metadata `""allow_x""` blocks, then rename the settings themselves to be called `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`.
>
> If I was going to do that I should get it in before Datasette 1.0.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1409#issuecomment-890400425_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1410/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
957345476,MDU6SXNzdWU5NTczNDU0NzY=,1411,Canned query ?sql= is pointlessly echoed in query string starting from hidden mode,9599,simonw,closed,0,,,,,1,2021-08-01T00:17:13Z,2021-08-01T03:27:30Z,2021-08-01T00:58:17Z,OWNER,,"Example: https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1

Submitting that form again results in this:

https://latest.datasette.io/fixtures/neighborhood_search?sql=%0D%0Aselect+neighborhood%2C+facet_cities.name%2C+state%0D%0Afrom+facetable%0D%0A++++join+facet_cities%0D%0A++++++++on+facetable.city_id+%3D+facet_cities.id%0D%0Awhere+neighborhood+like+%27%25%27+%7C%7C+%3Atext+%7C%7C+%27%25%27%0D%0Aorder+by+neighborhood%3B%0D%0A&_hide_sql=1&text=cork

Because the HTML on https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1 includes this:

```html
    <h3>Custom SQL query returning 1 row <span class=""show-hide-sql"">(<a href=""/fixtures/neighborhood_search?text=cork"">show</a>)</span></h3>
    
    
        <input type=""hidden"" name=""sql"" value=""
select neighborhood, facet_cities.name, state
from facetable
    join facet_cities
        on facetable.city_id = facet_cities.id
where neighborhood like &#39;%&#39; || :text || &#39;%&#39;
order by neighborhood;
"">
        <input type=""hidden"" name=""_hide_sql"" value=""1"">
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1411/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
957347860,MDU6SXNzdWU5NTczNDc4NjA=,1412,Mention WAL mode in the documentation (plus backup tips),9599,simonw,open,0,,,,,0,2021-08-01T00:27:11Z,2021-08-01T00:27:11Z,,OWNER,,"This is useful for people who are deploying Datasette with any write functionality, especially if they might be using something like EFS.

I can add a section about this to the bottom of https://docs.datasette.io/en/stable/deploying.html and then link to that section from both https://docs.datasette.io/en/stable/sql_queries.html#writable-canned-queries and https://docs.datasette.io/en/stable/internals.html#await-db-execute-write-sql-params-none-block-false

Also useful: mention that just copying a SQLite database file while it is being written to may not get a consistent file, so tell people to use one of the SQLite backup mechanisms instead.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1412/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
959137143,MDU6SXNzdWU5NTkxMzcxNDM=,1415,feature request: document minimum permissions for service account for cloudrun,536941,fgregg,open,0,,,,,4,2021-08-03T13:48:43Z,2023-11-05T16:46:59Z,,CONTRIBUTOR,,"Thanks again for such a powerful project.

For deploying to cloudrun from github actions, I'd like to create a service account with minimal permissions.

It would be great to document what those minimum permission that need to be set in the IAM.

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1415/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
959276629,MDU6SXNzdWU5NTkyNzY2Mjk=,1416,"Use rich to render tracebacks on errors, if available",9599,simonw,closed,0,,,,,0,2021-08-03T16:12:08Z,2021-08-03T16:12:51Z,2021-08-03T16:12:51Z,OWNER,,"> Now thinking I should try adding Rich as an optional dependency to Datasette - if it's there, show tracebacks using it. Could be really handy for development
> https://twitter.com/simonw/status/1422576091055616003",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1416/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
959278472,MDU6SXNzdWU5NTkyNzg0NzI=,1417,Use codespell in CI to spot spelling errors,9599,simonw,closed,0,,,,,1,2021-08-03T16:14:15Z,2021-08-03T16:36:40Z,2021-08-03T16:36:40Z,OWNER,,"I noticed Rich is using this: https://github.com/willmcgugan/rich/commit/9c12a4537499797c43725fff5276ef0da62423ef#diff-ce84a1b2c9eb4ab3ea22f610cad7111cb9a2f66365c3b24679901376a2a73ab2

Ran it against the Datasette docs and found a bunch of obvious fixes, surprisingly with no false positives.

```
datasette % codespell docs/*.rst
docs/authentication.rst:63: perfom ==> perform
docs/authentication.rst:76: perfom ==> perform
docs/changelog.rst:429: repsonse ==> response
docs/changelog.rst:503: permissons ==> permissions
docs/changelog.rst:717: compatibilty ==> compatibility
docs/changelog.rst:1172: browseable ==> browsable
docs/deploying.rst:191: similiar ==> similar
docs/internals.rst:434: Respons ==> Response, respond
docs/internals.rst:440: Respons ==> Response, respond
docs/internals.rst:717: tha ==> than, that, the
docs/performance.rst:42: databse ==> database
docs/plugin_hooks.rst:667: utilites ==> utilities
docs/publish.rst:168: countainer ==> container
docs/settings.rst:352: inalid ==> invalid
docs/sql_queries.rst:406: preceeded ==> preceded, proceeded
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1417/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
959284434,MDExOlB1bGxSZXF1ZXN0NzAyNDIyMjYz,1418,Spelling corrections plus CI job for codespell,9599,simonw,closed,0,,,,,2,2021-08-03T16:21:19Z,2021-08-03T16:36:39Z,2021-08-03T16:36:38Z,OWNER,simonw/datasette/pulls/1418,Refs #1417.,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1418/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
959710008,MDU6SXNzdWU5NTk3MTAwMDg=,1419,`publish cloudrun` should deploy a more recent SQLite version,536941,fgregg,open,0,,,,,3,2021-08-04T00:45:55Z,2021-08-05T03:23:24Z,,CONTRIBUTOR,,"I recently changed from deploying a datasette using `datasette publish heroku` to `datasette publish cloudrun`. [A query that ran on the heroku site](https://odpr.bunkum.us/odpr-6c2f4fc?sql=with+pivot_members+as+%28%0D%0A++select%0D%0A++++f_num%2C%0D%0A++++max%28union_name%29+as+union_name%2C%0D%0A++++max%28aff_abbr%29+as+abbreviation%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2007%0D%0A++++%29+as+%222007%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2008%0D%0A++++%29+as+%222008%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2009%0D%0A++++%29+as+%222009%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2010%0D%0A++++%29+as+%222010%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2011%0D%0A++++%29+as+%222011%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2012%0D%0A++++%29+as+%222012%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2013%0D%0A++++%29+as+%222013%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2014%0D%0A++++%29+as+%222014%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2015%0D%0A++++%29+as+%222015%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2016%0D%0A++++%29+as+%222016%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2017%0D%0A++++%29+as+%222017%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2018%0D%0A++++%29+as+%222018%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2019%0D%0A++++%29+as+%222019%22%0D%0A++from%0D%0A++++lm_data%0D%0A++++left+join+ar_membership+using+%28rpt_id%29%0D%0A++where%0D%0A++++members+%21%3D+%27%27%0D%0A++++and+trim%28desig_name%29+%3D+%27NHQ%27%0D%0A++++and+union_name+not+in+%28%0D%0A++++++%27AFL-CIO%27%2C%0D%0A++++++%27CHANGE+TO+WIN%27%2C%0D%0A++++++%27FOOD+ALLIED+SVC+TRADES+DEPT+AFL-CIO%27%0D%0A++++%29%0D%0A++++and+f_num+not+in+%28387%2C+296%2C+123%2C+347%2C+531897%2C+30410%2C+49%29%0D%0A++++and+lower%28ar_membership.category%29+IN+%28%0D%0A++++++%27active+members%27%2C%0D%0A++++++%27active+professional%27%2C%0D%0A++++++%27regular+members%27%2C%0D%0A++++++%27full+time+member%27%2C%0D%0A++++++%27full+time+members%27%2C%0D%0A++++++%27active+education+support+professional%27%2C%0D%0A++++++%27active%27%2C%0D%0A++++++%27active+member%27%2C%0D%0A++++++%27members%27%2C%0D%0A++++++%27see+item+69%27%2C%0D%0A++++++%27regular%27%2C%0D%0A++++++%27dues+paying+members%27%2C%0D%0A++++++%27building+trades+journeyman%27%2C%0D%0A++++++%27full+per+capita+tax+payers%27%2C%0D%0A++++++%27active+postal%27%2C%0D%0A++++++%27active+membership%27%2C%0D%0A++++++%27full-time+member%27%2C%0D%0A++++++%27regular+active+member%27%2C%0D%0A++++++%27journeyman%27%2C%0D%0A++++++%27member%27%2C%0D%0A++++++%27journeymen%27%2C%0D%0A++++++%27one+half+per+capita+tax+payers%27%2C%0D%0A++++++%27schedule+1%27%2C%0D%0A++++++%27active+memebers%27%2C%0D%0A++++++%27active+members+-+us%27%2C%0D%0A++++++%27dues+paying+membership%27%0D%0A++++%29%0D%0A++GROUP+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++pivot_members%0D%0Aorder+by%0D%0A++%222019%22+desc%3B), now throws a syntax error on the [cloudrun site](https://labordata.bunkum.us/odpr-6c2f4fc?sql=with+pivot_members+as+%28%0D%0A++select%0D%0A++++f_num%2C%0D%0A++++max%28union_name%29+as+union_name%2C%0D%0A++++max%28aff_abbr%29+as+abbreviation%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2007%0D%0A++++%29+as+%222007%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2008%0D%0A++++%29+as+%222008%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2009%0D%0A++++%29+as+%222009%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2010%0D%0A++++%29+as+%222010%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2011%0D%0A++++%29+as+%222011%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2012%0D%0A++++%29+as+%222012%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2013%0D%0A++++%29+as+%222013%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2014%0D%0A++++%29+as+%222014%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2015%0D%0A++++%29+as+%222015%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2016%0D%0A++++%29+as+%222016%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2017%0D%0A++++%29+as+%222017%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2018%0D%0A++++%29+as+%222018%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2019%0D%0A++++%29+as+%222019%22%0D%0A++from%0D%0A++++lm_data%0D%0A++++left+join+ar_membership+using+%28rpt_id%29%0D%0A++where%0D%0A++++members+%21%3D+%27%27%0D%0A++++and+trim%28desig_name%29+%3D+%27NHQ%27%0D%0A++++and+union_name+not+in+%28%0D%0A++++++%27AFL-CIO%27%2C%0D%0A++++++%27CHANGE+TO+WIN%27%2C%0D%0A++++++%27FOOD+ALLIED+SVC+TRADES+DEPT+AFL-CIO%27%0D%0A++++%29%0D%0A++++and+f_num+not+in+%28387%2C+296%2C+123%2C+347%2C+531897%2C+30410%2C+49%29%0D%0A++++and+lower%28ar_membership.category%29+IN+%28%0D%0A++++++%27active+members%27%2C%0D%0A++++++%27active+professional%27%2C%0D%0A++++++%27regular+members%27%2C%0D%0A++++++%27full+time+member%27%2C%0D%0A++++++%27full+time+members%27%2C%0D%0A++++++%27active+education+support+professional%27%2C%0D%0A++++++%27active%27%2C%0D%0A++++++%27active+member%27%2C%0D%0A++++++%27members%27%2C%0D%0A++++++%27see+item+69%27%2C%0D%0A++++++%27regular%27%2C%0D%0A++++++%27dues+paying+members%27%2C%0D%0A++++++%27building+trades+journeyman%27%2C%0D%0A++++++%27full+per+capita+tax+payers%27%2C%0D%0A++++++%27active+postal%27%2C%0D%0A++++++%27active+membership%27%2C%0D%0A++++++%27full-time+member%27%2C%0D%0A++++++%27regular+active+member%27%2C%0D%0A++++++%27journeyman%27%2C%0D%0A++++++%27member%27%2C%0D%0A++++++%27journeymen%27%2C%0D%0A++++++%27one+half+per+capita+tax+payers%27%2C%0D%0A++++++%27schedule+1%27%2C%0D%0A++++++%27active+memebers%27%2C%0D%0A++++++%27active+members+-+us%27%2C%0D%0A++++++%27dues+paying+membership%27%0D%0A++++%29%0D%0A++GROUP+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++pivot_members%0D%0Aorder+by%0D%0A++%222019%22+desc%3B).

I suspect this is because they are running different versions of sqlite3.

- Heroku: sqlite3 3.31.1 ([-/versions](https://odpr.bunkum.us/-/versions))
- Cloudrun: sqlite3 3.27.2 ([-/versions](https://labordata.bunkum.us/-/versions))

If so, it would be great to

1. harmonize the sqlite3 versions across platforms
2. update the docker files so as to update the sqlite3 version for cloudrun",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1419/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
275917760,MDU6SXNzdWUyNzU5MTc3NjA=,142,Show extra instructions with the interrupted,9599,simonw,closed,0,,,,,3,2017-11-22T01:44:29Z,2018-05-28T21:25:06Z,2018-05-28T21:24:35Z,OWNER,,"When you are using Datasette locally for ad-hoc analysis it can be frustrating to hit the time limit.

 If you start it with the correct command line arguments you can disable that time limit.  So how about we tell you how to do that anytime you hit the interrupted error provided you are accessing it from localhost.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/142/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
959898166,MDU6SXNzdWU5NTk4OTgxNjY=,1420,`datasette publish cloudrun --cpu X` option,9599,simonw,closed,0,,,,,5,2021-08-04T05:04:31Z,2021-08-05T00:54:59Z,2021-08-04T05:33:48Z,OWNER,,"For setting the number of vCPUs - current valid values are 1, 2 or 4: https://cloud.google.com/run/docs/configuring/cpu

Pass that through to `gcloud run deploy --image IMAGE_URL --cpu CPU`",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1420/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
959999095,MDU6SXNzdWU5NTk5OTkwOTU=,1421,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times",6988,j4mie,closed,0,,,,,11,2021-08-04T07:29:04Z,2021-08-09T03:41:07Z,2021-08-09T03:33:02Z,NONE,,"Datasette version `0.58.1`.

I'm guessing this is a bug in the code that looks for `:param`-style query parameters..

<img width=""543"" alt=""image"" src=""https://user-images.githubusercontent.com/6988/128139832-ef9c5291-f3d7-4402-8625-b45d26b4e5bc.png"">
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1421/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
961367843,MDU6SXNzdWU5NjEzNjc4NDM=,1422,Ability to default to hiding the SQL for a canned query,9599,simonw,closed,0,,,,,4,2021-08-05T02:51:39Z,2021-08-07T05:32:29Z,2021-08-07T05:32:29Z,OWNER,,"I'm working on a project with some HUGE (400+ lines of SQL) canned queries right now.

Any time you land on the canned query page you have to scroll down a long distance to get to the results!

Would be useful to be able to default to https://latest.datasette.io/fixtures/magic_parameters?_hide_sql=1 without needing the parameter.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1422/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
962391325,MDU6SXNzdWU5NjIzOTEzMjU=,1423,Show count of facet values if ?_facet_size=max,9599,simonw,closed,0,,,,,9,2021-08-06T04:42:20Z,2021-12-15T17:48:40Z,2021-08-16T18:56:43Z,OWNER,,"I sometimes want to get a count of the values in a facet - if it's a facet of US states for example I want to know if all 50 are represented.

Idea: if `?_facet_size=max` is present, add a count to the facet heading. So on:

https://latest.datasette.io/fixtures/compound_three_primary_keys?_facet=content&_facet_size=max&_facet=pk1&_facet=pk2#facet-pk2

It could have something like this:

<img width=""658"" alt=""fixtures__compound_three_primary_keys__1_001_rows_and_NY_629_755_•_Untitled-1_—_datasette"" src=""https://user-images.githubusercontent.com/9599/128456981-1f3d5348-90f1-45cc-9287-0f7bb0015257.png"">

Note that the first column shows >1000 - because in that case we've truncated the facet calculation since the maximum allowed returned rows is 1000.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1423/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
963527045,MDU6SXNzdWU5NjM1MjcwNDU=,1424,Document exceptions that can be raised by db.execute() and friends,9599,simonw,open,0,,,,,4,2021-08-08T22:23:25Z,2021-08-08T22:27:31Z,,OWNER,,Not currently covered here: https://docs.datasette.io/en/stable/internals.html#await-db-execute-sql,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1424/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
963528457,MDU6SXNzdWU5NjM1Mjg0NTc=,1425,render_cell() hook should support returning an awaitable,9599,simonw,closed,0,,,,,11,2021-08-08T22:32:29Z,2021-08-09T07:14:35Z,2021-08-09T03:00:37Z,OWNER,,"Many of the plugin hooks can return an awaitable - e.g. https://docs.datasette.io/en/stable/plugin_hooks.html#plugin-hook-extra-template-vars - but `render_cell()` doesn't support this.

I recently found myself wanting to execute an additional SQL query from that hook, but it wasn't possible to do that since I couldn't use `await`.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1425/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
964322136,MDU6SXNzdWU5NjQzMjIxMzY=,1426,"Manage /robots.txt in Datasette core, block robots by default",9599,simonw,open,0,,,,,9,2021-08-09T19:56:56Z,2021-12-04T07:11:29Z,,OWNER,,"See accompanying Twitter thread: https://twitter.com/simonw/status/1424820203603431439

> Datasette currently has a plugin for configuring robots.txt, but I'm beginning to think it should be part of core and crawlers should be blocked by default - having people explicitly opt-in to having their sites crawled and indexed feels a lot safer https://datasette.io/plugins/datasette-block-robots

I have a lot of Datasettes deployed now, and tailing logs shows that they are being *hammered* by search engine crawlers even though many of them are not interesting enough to warrant indexing.

I'm starting to think blocking crawlers would actually be a better default for most people, provided it was well documented and easy to understand how to allow them.

Default-deny is usually a better policy than default-allow!",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1426/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
969548935,MDU6SXNzdWU5Njk1NDg5MzU=,1429,UI for setting `?_size=max` on table page,9599,simonw,open,0,,,,,2,2021-08-12T20:52:09Z,2021-08-13T04:37:41Z,,OWNER,,"It defaults to 100 per page, but you can increase that to 1000 per page using `?_size=max` (or higher if `max_returned_rows` is set higher than that).

But... that's only available to people who know how to hack URLs.

Solution: add a link that sets that option to the pagination block at the bottom of the table:

<img width=""565"" alt=""data__states__28_999_rows"" src=""https://user-images.githubusercontent.com/9599/129268119-989dd835-3e04-4baa-8e9e-28e56ad59154.png"">
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1429/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
275939188,MDU6SXNzdWUyNzU5MzkxODg=,143,"Mechanism for ""suggested visualizations""",9599,simonw,closed,0,,,,,1,2017-11-22T04:10:25Z,2018-07-10T17:48:34Z,2018-07-10T17:48:34Z,OWNER,," Each visualization should have a way of deciding if it might be appropriate for the current view of data. 

 We can then offer a ""suggested visualizations"" prompt which shows previews.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/143/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
969758038,MDExOlB1bGxSZXF1ZXN0NzExNzgzNjE2,1430,Column metadata,9599,simonw,closed,0,,,,,1,2021-08-12T23:34:39Z,2021-08-12T23:53:23Z,2021-08-12T23:53:23Z,OWNER,simonw/datasette/pulls/1430,"Refs #942

Still needs:

- [x] Tests
- [x] Documentation",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1430/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
969840302,MDU6SXNzdWU5Njk4NDAzMDI=,1431,`--help-config` should be called `--help-settings`,9599,simonw,closed,0,,,,,1,2021-08-13T00:46:48Z,2021-08-13T01:01:58Z,2021-08-13T01:01:58Z,OWNER,,Follow-on from #1105 rebranding exercise.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1431/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
969855774,MDU6SXNzdWU5Njk4NTU3NzQ=,1432,Rename Datasette.__init__(config=) parameter to settings=,9599,simonw,open,0,,,,,8,2021-08-13T01:00:27Z,2021-10-19T01:16:41Z,,OWNER,,"> While I'm doing this I should rename this internal variable to avoid confusion in the future:
>
> https://github.com/simonw/datasette/blob/e837095ef35ae155b4c78cc9a8b7133a48c94f03/datasette/app.py#L203

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1431#issuecomment-898072940_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1432/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
970386262,MDExOlB1bGxSZXF1ZXN0NzEyMzQ2MTk5,1433,"Update trustme requirement from <0.9,>=0.7 to >=0.7,<0.10",49699333,dependabot[bot],closed,0,,,,,1,2021-08-13T13:10:24Z,2021-08-25T01:29:27Z,2021-08-25T01:29:26Z,CONTRIBUTOR,simonw/datasette/pulls/1433,"Updates the requirements on [trustme](https://github.com/python-trio/trustme) to permit the latest version.
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/python-trio/trustme/commit/8fc5bf9c5afaf40843287e6eccca344713eddb50""><code>8fc5bf9</code></a> Bump version to 0.9.0</li>
<li><a href=""https://github.com/python-trio/trustme/commit/913e21dd94fc16a8b41742cc60338dcf638ae792""><code>913e21d</code></a> Bump types-cryptography from 3.3.3 to 3.3.5 (<a href=""https://github-redirect.dependabot.com/python-trio/trustme/issues/342"">#342</a>)</li>
<li><a href=""https://github.com/python-trio/trustme/commit/c66709d48bffc0d3a70d260313b752037ea71f8d""><code>c66709d</code></a> Bump types-pyopenssl from 20.0.4 to 20.0.5 (<a href=""https://github-redirect.dependabot.com/python-trio/trustme/issues/343"">#343</a>)</li>
<li><a href=""https://github.com/python-trio/trustme/commit/5131f79e9b7d73d993f901d8e48e6d744b0606ef""><code>5131f79</code></a> Add type annotations (<a href=""https://github-redirect.dependabot.com/python-trio/trustme/issues/341"">#341</a>)</li>
<li><a href=""https://github.com/python-trio/trustme/commit/a411dad9614f3fc2e746324d9a226e4947d55835""><code>a411dad</code></a> Bump charset-normalizer from 2.0.3 to 2.0.4 (<a href=""https://github-redirect.dependabot.com/python-trio/trustme/issues/340"">#340</a>)</li>
<li><a href=""https://github.com/python-trio/trustme/commit/be5ec8a3892ea218291ac93da8b382229b07f044""><code>be5ec8a</code></a> Bump sphinx from 4.1.1 to 4.1.2</li>
<li><a href=""https://github.com/python-trio/trustme/commit/d3b8865576a82189e06c735c64685fbad7ad02d0""><code>d3b8865</code></a> Bump charset-normalizer from 2.0.2 to 2.0.3</li>
<li><a href=""https://github.com/python-trio/trustme/commit/4503bef325bf0b23f4a0289654ee4c72d0c79fd1""><code>4503bef</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/python-trio/trustme/issues/334"">#334</a> from python-trio/dependabot/pip/charset-normalizer-2.0.2</li>
<li><a href=""https://github.com/python-trio/trustme/commit/ce8099d9e2b59d610cf9858d095b4d7213341ba1""><code>ce8099d</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/python-trio/trustme/issues/335"">#335</a> from python-trio/dependabot/pip/sphinx-4.1.1</li>
<li><a href=""https://github.com/python-trio/trustme/commit/8b6d3c660f4389b9d97f048f6eb2c11d201dda44""><code>8b6d3c6</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/python-trio/trustme/issues/336"">#336</a> from python-trio/dependabot/pip/idna-3.2</li>
<li>Additional commits viewable in <a href=""https://github.com/python-trio/trustme/compare/v0.7.0...v0.9.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1433/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
970463436,MDExOlB1bGxSZXF1ZXN0NzEyNDEyODgz,1434,Enrich arbitrary query results with foreign key links and column descriptions,9599,simonw,open,0,,,,,1,2021-08-13T14:43:01Z,2021-08-19T21:18:58Z,,OWNER,simonw/datasette/pulls/1434,"Refs #1293, follows #942.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1434/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
970626625,MDU6SXNzdWU5NzA2MjY2MjU=,1435,Turn off suggest facets on tables with large numbers of columns,9599,simonw,open,0,,,,,0,2021-08-13T18:30:48Z,2021-08-13T18:30:48Z,,OWNER,,If a table has 200 columns it will take multiple seconds to try and suggest facets. I should either quit after the first 20 or not suggest facets at all.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1435/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
972918533,MDU6SXNzdWU5NzI5MTg1MzM=,1438,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions,9599,simonw,open,0,,,,,7,2021-08-17T17:35:36Z,2021-08-18T00:22:23Z,,OWNER,,"> Confirmed: https://thesession.vercel.app/thesession?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A is a page where the URL correctly encoded `%` as `%25` - but then in the HTML on that page that links to the CSV and JSON versions we get this:
>
> ```html
> <p class=""export-links"">This data as
>   <a href=""/thesession.json?sql=select * from tunes where name like &#34;%wise maid%&#34;"">json</a>,
>   <a href=""/thesession.csv?sql=select * from tunes where name like &#34;%wise maid%&#34;&amp;_size=max"">CSV</a>
> </p>
> ```
Those CSV and JSON links are incorrect.

_Originally posted by @simonw in https://github.com/simonw/datasette-publish-vercel/issues/48#issuecomment-900497579_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1438/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
973139047,MDU6SXNzdWU5NzMxMzkwNDc=,1439,Rethink how .ext formats (v.s. ?_format=) works before 1.0,9599,simonw,closed,0,,,3268330,Datasette 1.0,48,2021-08-17T23:32:51Z,2022-03-15T20:51:26Z,2022-03-15T20:51:26Z,OWNER,,"Datasette currently has surprising special behaviour for if a table name ends in `.csv` - which can happen when a tool like `csvs-to-sqlite` creates tables that match the filename that they were imported from.

https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv illustrates this behaviour: it links to `.csv` and `.json` that look like this:

- https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=json
- https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=csv&_size=max

Where normally Datasette would add the `.csv` or `.json` extension to the path component of the URL (as seen on other pages such as https://latest.datasette.io/fixtures/facet_cities) here the [path_with_format() function](https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L710) notices that there is already a `.` in the path and instead adds `?_format=csv` to the query string instead.

The problem with this mechanism is that it's pretty surprising. Anyone writing external code to Datasette who wants to get back the `.csv` or `.json` version giving the URL to a table page will need to know about and implement this behaviour themselves. That's likely to cause all kinds of bugs in the future.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1439/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
276091279,MDU6SXNzdWUyNzYwOTEyNzk=,144,apsw as alternative sqlite3 binding (for full text search),649467,mhalle,closed,0,,,,,3,2017-11-22T14:40:39Z,2018-05-28T21:29:42Z,2018-05-28T21:29:42Z,NONE,,"Hey there,

Have you considered providing apsw support as an alternative to stock python sqlite3?  I use apsw because it keeps up with sqlite3 and is straightforward to bring in extensions like FTS5.  FTS really accelerates the kind of searching often done by web clients.

I may be able to help (it shouldn't be much code), but there are a couple of stylistic questions that come up when supporting an optional package.  Also, apsw is tricky in that it doesn't have a pypi package (author says limitations in providing options to setup.py). ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/144/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
974987856,MDU6SXNzdWU5NzQ5ODc4NTY=,1442,Mechanism to cause specific branches to deploy their own demos,9599,simonw,closed,0,,,,,6,2021-08-19T19:41:39Z,2021-08-19T21:11:45Z,2021-08-19T21:09:40Z,OWNER,,"A useful capability would be if it was super-easy to say ""any pushes to branch X should be deployed to `latest-X.datasette.io`"".

I'd like to use this for the column query information work in #1434",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1442/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
974995592,MDU6SXNzdWU5NzQ5OTU1OTI=,1443,datasette.databases should be a documented property,9599,simonw,closed,0,,,,,1,2021-08-19T19:53:04Z,2021-08-19T21:25:07Z,2021-08-19T21:23:47Z,OWNER,,"https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/app.py#L231

I want to use it in https://github.com/simonw/datasette-block-robots/issues/5",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1443/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
975049826,MDExOlB1bGxSZXF1ZXN0NzE2MjYyODI5,1444,Ability to deploy demos of branches,9599,simonw,closed,0,,,,,0,2021-08-19T21:08:04Z,2021-08-19T21:09:44Z,2021-08-19T21:09:39Z,OWNER,simonw/datasette/pulls/1444,See #1442.,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1444/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
977323133,MDU6SXNzdWU5NzczMjMxMzM=,1445,Ability to search for text across all columns in a table,9599,simonw,open,0,,,,,5,2021-08-23T18:50:48Z,2021-08-23T19:10:17Z,,OWNER,,"When I'm working with new data I often find myself wanting to run a search for text embedded in ANY of the columns of a table, without having to even fully understand the schema first.

I figured out a trick for doing that using a SQL-generated SQL query here: https://til.simonwillison.net/datasette/search-all-columns-trick

But maybe this should be a core Datasette feature? Or a plugin?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1445/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
978357984,MDU6SXNzdWU5NzgzNTc5ODQ=,1446,Modify base.html template to support optional sticky footer,9599,simonw,closed,0,,,,,3,2021-08-24T18:11:12Z,2021-08-31T01:54:59Z,2021-08-24T20:32:47Z,OWNER,,"The neatest way to have the footer stick to the bottom of the browser window that I've found is to use the flexbox pattern from https://css-tricks.com/couple-takes-sticky-footer/

```html
<body>
  <div class=""content"">
    content
  </div>
  <footer class=""footer""></footer>
</body>
```
```css
html, body {
  height: 100%;
}
body {
  display: flex;
  flex-direction: column;
}
.content {
  flex: 1 0 auto;
}
.footer {
  flex-shrink: 0;
}
```
I tried this in a custom plugin but it ended up having to duplicate the entire `base.html` template just to get a wrapper around the not-footer content.

I think Datasette's own `base.html` template should have this wrapper element instead.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1446/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
978614898,MDExOlB1bGxSZXF1ZXN0NzE5MTc1NTkz,1447,Remove underscore from search mode parameter name,127565,wragge,closed,0,,,,,1,2021-08-25T01:28:04Z,2021-08-25T01:28:58Z,2021-08-25T01:28:58Z,CONTRIBUTOR,simonw/datasette/pulls/1447,The fulltext search documentation refers to the parameter as `searchmode` but the `metadata.json` example uses `search_mode`. The latter doesn't actually seem to work.,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1447/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
980228553,MDExOlB1bGxSZXF1ZXN0NzIwNTA2MTM1,1448,"Update pluggy requirement from ~=0.13.0 to >=0.13,<1.1",49699333,dependabot[bot],closed,0,,,,,1,2021-08-26T13:09:52Z,2021-10-13T21:11:01Z,2021-10-13T21:11:00Z,CONTRIBUTOR,simonw/datasette/pulls/1448,"Updates the requirements on [pluggy](https://github.com/pytest-dev/pluggy) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst"">pluggy's changelog</a>.</em></p>
<blockquote>
<h1>pluggy 1.0.0 (2021-08-25)</h1>
<h2>Deprecations and Removals</h2>
<ul>
<li>
<p><code>[#116](https://github.com/pytest-dev/pluggy/issues/116) &lt;https://github.com/pytest-dev/pluggy/issues/116&gt;</code>_: Remove deprecated <code>implprefix</code> support.
Decorate hook implementations using an instance of HookimplMarker instead.
The deprecation was announced in release <code>0.7.0</code>.</p>
</li>
<li>
<p><code>[#120](https://github.com/pytest-dev/pluggy/issues/120) &lt;https://github.com/pytest-dev/pluggy/issues/120&gt;</code>_: Remove the deprecated <code>proc</code> argument to <code>call_historic</code>.
Use <code>result_callback</code> instead, which has the same behavior.
The deprecation was announced in release <code>0.7.0</code>.</p>
</li>
<li>
<p><code>[#265](https://github.com/pytest-dev/pluggy/issues/265) &lt;https://github.com/pytest-dev/pluggy/issues/265&gt;</code>_: Remove the <code>_Result.result</code> property. Use <code>_Result.get_result()</code> instead.
Note that unlike <code>result</code>, <code>get_result()</code> raises the exception if the hook raised.
The deprecation was announced in release <code>0.6.0</code>.</p>
</li>
<li>
<p><code>[#267](https://github.com/pytest-dev/pluggy/issues/267) &lt;https://github.com/pytest-dev/pluggy/issues/267&gt;</code>_: Remove official support for Python 3.4.</p>
</li>
<li>
<p><code>[#272](https://github.com/pytest-dev/pluggy/issues/272) &lt;https://github.com/pytest-dev/pluggy/issues/272&gt;</code>_: Dropped support for Python 2.
Continue to use pluggy 0.13.x for Python 2 support.</p>
</li>
<li>
<p><code>[#308](https://github.com/pytest-dev/pluggy/issues/308) &lt;https://github.com/pytest-dev/pluggy/issues/308&gt;</code>_: Remove official support for Python 3.5.</p>
</li>
<li>
<p><code>[#313](https://github.com/pytest-dev/pluggy/issues/313) &lt;https://github.com/pytest-dev/pluggy/issues/313&gt;</code>_: The internal <code>pluggy.callers</code>, <code>pluggy.manager</code> and <code>pluggy.hooks</code> are now explicitly marked private by a <code>_</code> prefix (e.g. <code>pluggy._callers</code>).
Only API exported by the top-level <code>pluggy</code> module is considered public.</p>
</li>
<li>
<p><code>[#59](https://github.com/pytest-dev/pluggy/issues/59) &lt;https://github.com/pytest-dev/pluggy/issues/59&gt;</code>_: Remove legacy <code>__multicall__</code> recursive hook calling system.
The deprecation was announced in release <code>0.5.0</code>.</p>
</li>
</ul>
<h2>Features</h2>
<ul>
<li>
<p><code>[#282](https://github.com/pytest-dev/pluggy/issues/282) &lt;https://github.com/pytest-dev/pluggy/issues/282&gt;</code>_: When registering a hookimpl which is declared as <code>hookwrapper=True</code> but whose
function is not a generator function, a <code>PluggyValidationError</code> exception is
now raised.</p>
<p>Previously this problem would cause an error only later, when calling the hook.</p>
<p>In the unlikely case that you have a hookwrapper that <em>returns</em> a generator</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pytest-dev/pluggy/commit/4259fdd799d64f953631e47ddcb69c3074a15c13""><code>4259fdd</code></a> Fix CHANGELOG title manually</li>
<li><a href=""https://github.com/pytest-dev/pluggy/commit/906abca9bb403729f4df44d02142698ddd54d5b5""><code>906abca</code></a> Preparing release 1.0.0</li>
<li><a href=""https://github.com/pytest-dev/pluggy/commit/56eb23c89aa0a43fa7b139894fbf6142684c2969""><code>56eb23c</code></a> Rename HOWTORELEASE to RELEASING to follow pytest</li>
<li><a href=""https://github.com/pytest-dev/pluggy/commit/fc6395ca27d5deb008db250da57a23dd40bc93a4""><code>fc6395c</code></a> Fix scripts/release.py to use main instead of master</li>
<li><a href=""https://github.com/pytest-dev/pluggy/commit/e04816f71b3d40bc959550b93f1d69098fc045fd""><code>e04816f</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pluggy/issues/324"">#324</a> from RonnyPfannschmidt/benchmarks</li>
<li><a href=""https://github.com/pytest-dev/pluggy/commit/1424ab0a6a54a878b171c806a39c79d54817c531""><code>1424ab0</code></a> add micro benchmarks for hook calling playing with a the number of callers, w...</li>
<li><a href=""https://github.com/pytest-dev/pluggy/commit/5e518648987a3afced3c2b014c73abab78f8306f""><code>5e51864</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pluggy/issues/323"">#323</a> from RonnyPfannschmidt/switch-to-main</li>
<li><a href=""https://github.com/pytest-dev/pluggy/commit/05c3bbd07c7186d3c124f1009aa6c8999ba70c2c""><code>05c3bbd</code></a> switch to main as primary branch</li>
<li><a href=""https://github.com/pytest-dev/pluggy/commit/6b344fbebc598a623bb8db6b58f6f04b27916515""><code>6b344fb</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pluggy/issues/319"">#319</a> from RonnyPfannschmidt/pre-commit-update</li>
<li><a href=""https://github.com/pytest-dev/pluggy/commit/71f2d6b4a21381314f730dd5b5f93519882f79da""><code>71f2d6b</code></a> introduce pyupgrade and update black</li>
<li>Additional commits viewable in <a href=""https://github.com/pytest-dev/pluggy/compare/0.13.0...1.0.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1448/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
981676832,MDU6SXNzdWU5ODE2NzY4MzI=,1449,`register_commands()` plugin hook to register extra CLI commands,9599,simonw,closed,0,,,,,13,2021-08-28T00:26:21Z,2021-08-28T01:58:35Z,2021-08-28T01:43:11Z,OWNER,,"The `datasette` CLI tool currently has 7 subcommands: `serve`, `inspect`, `install`, `package`, `plugins`, `publish` and `uninstall`.

A plugin hook could allow plugins to register extra subcommands.

I've avoided this for quite a while because I didn't have good use-cases for it - but the existence of the `datasette install xxx` command for installing packages into the correct virtual environment means that actually there's a good reason to do this: it would allow plugins to provide additional command-line mechanisms without the user having to understand how virtual environments work in order to install those commands into the same environment as the rest of Datasette.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1449/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
276192732,MDExOlB1bGxSZXF1ZXN0MTU0MjQ2ODE2,145,Fix pytest version conflict,9599,simonw,closed,0,,,,,0,2017-11-22T20:15:34Z,2017-11-22T20:17:54Z,2017-11-22T20:17:52Z,OWNER,simonw/datasette/pulls/145,"https://travis-ci.org/simonw/datasette/jobs/305929426

    pkg_resources.VersionConflict: (pytest 3.2.1 (/home/travis/virtualenv/python3.5.3/lib/python3.5/site-packages), 
    Requirement.parse('pytest==3.2.3'))",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/145/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
981681138,MDU6SXNzdWU5ODE2ODExMzg=,1450,"Datasette --help should show something more useful than ""Datasette!""",9599,simonw,closed,0,,,,,1,2021-08-28T00:44:51Z,2021-08-28T00:49:07Z,2021-08-28T00:49:07Z,OWNER,,"https://github.com/simonw/datasette/blob/a1a33bb5822214be1cebd98cd858b2058d91a4aa/datasette/cli.py#L122-L127

_Originally spotted by @simonw in https://github.com/simonw/datasette/issues/1449#issuecomment-907539668_

```
~ % datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1450/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
982780906,MDExOlB1bGxSZXF1ZXN0NzIyNDgwNTQy,1453,Bump black from 21.7b0 to 21.8b0,49699333,dependabot[bot],closed,0,,,,,2,2021-08-30T13:13:39Z,2021-09-14T13:10:40Z,2021-09-14T13:10:38Z,CONTRIBUTOR,simonw/datasette/pulls/1453,"Bumps [black](https://github.com/psf/black) from 21.7b0 to 21.8b0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.8b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Add support for formatting Jupyter Notebook files (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2357"">#2357</a>)</li>
<li>Move from <code>appdirs</code> dependency to <code>platformdirs</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2375"">#2375</a>)</li>
<li>Present a more user-friendly error if .gitignore is invalid (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2414"">#2414</a>)</li>
<li>The failsafe for accidentally added backslashes in f-string expressions has been hardened to handle more edge cases during quote normalization (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2437"">#2437</a>)</li>
<li>Avoid changing a function return type annotation's type to a tuple by adding a trailing comma (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2384"">#2384</a>)</li>
<li>Parsing support has been added for unparenthesized walruses in set literals, set comprehensions, and indices (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2447"">#2447</a>).</li>
<li>Pin <code>setuptools-scm</code> build-time dependency version (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2457"">#2457</a>)</li>
<li>Exclude typing-extensions version 3.10.0.1 due to it being broken on Python 3.10 (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2460"">#2460</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Replace sys.exit(-1) with raise ImportError as it plays more nicely with tools that scan installed packages (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2440"">#2440</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>The provided pre-commit hooks no longer specify <code>language_version</code> to avoid overriding <code>default_language_version</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2430"">#2430</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>21.8b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Add support for formatting Jupyter Notebook files (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2357"">#2357</a>)</li>
<li>Move from <code>appdirs</code> dependency to <code>platformdirs</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2375"">#2375</a>)</li>
<li>Present a more user-friendly error if .gitignore is invalid (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2414"">#2414</a>)</li>
<li>The failsafe for accidentally added backslashes in f-string expressions has been
hardened to handle more edge cases during quote normalization (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2437"">#2437</a>)</li>
<li>Avoid changing a function return type annotation's type to a tuple by adding a
trailing comma (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2384"">#2384</a>)</li>
<li>Parsing support has been added for unparenthesized walruses in set literals, set
comprehensions, and indices (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2447"">#2447</a>).</li>
<li>Pin <code>setuptools-scm</code> build-time dependency version (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2457"">#2457</a>)</li>
<li>Exclude typing-extensions version 3.10.0.1 due to it being broken on Python 3.10
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/2460"">#2460</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Replace sys.exit(-1) with raise ImportError as it plays more nicely with tools that
scan installed packages (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2440"">#2440</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>The provided pre-commit hooks no longer specify <code>language_version</code> to avoid overriding
<code>default_language_version</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2430"">#2430</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.7b0&new-version=21.8b0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1453/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
982803408,MDU6SXNzdWU5ODI4MDM0MDg=,1454,Feature Request: Publish to IPFS,1560788,blitmap,open,0,,,,,0,2021-08-30T13:36:18Z,2021-08-30T13:36:18Z,,NONE,,"Hello,

I am a huge fan of this being used for exploring data.  I think it has a lot of flexibility not found in other tools.

I'm not sure if what I'm asking for is possible:  Can this be extended to publish to IPFS?

IPFS is an attractive hosting option for decentralized journalism.

Food for thought ~",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1454/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
988325628,MDExOlB1bGxSZXF1ZXN0NzI3MjY1MDI1,1455,Add scientists to target groups,198537,rgieseke,closed,0,,,,,3,2021-09-04T16:28:58Z,2021-09-04T16:32:21Z,2021-09-04T16:31:38Z,CONTRIBUTOR,simonw/datasette/pulls/1455,"Not sure if you want them mentioned explicitly (it's already a long list), but following up on https://twitter.com/simonw/status/1434176989565382656",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1455/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
988552851,MDU6SXNzdWU5ODg1NTI4NTE=,1456,conda install results in non-functioning `datasette serve` due to out-of-date asgiref,51016,ctb,open,0,,,,,0,2021-09-05T16:59:55Z,2021-09-05T16:59:55Z,,CONTRIBUTOR,,"Over in https://github.com/ctb/2021-sourmash-datasette, I discovered that the following commands fail:
```
conda create -n datasette4 -y datasette=0.58.1
conda activate datasette4
datasette gathertax.db
```
with `ImportError: cannot import name 'WebSocketScope' from 'asgiref.typing'`.

This appears to be because asgiref 3.3.4 doesn't have WebSocketScope, but later versions do - a simple

```
pip install asgiref==3.4.1
```

fixes the problem for me, at least to the point where I can run datasette and poke around as usual.

I note that over in the conda-forge recipe, https://github.com/conda-forge/datasette-feedstock/blob/master/recipe/meta.yaml pins asgiref to < 3.4.0, but I'm not sure why - so I'm not sure how to best resolve this issue :).",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1456/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
988553806,MDU6SXNzdWU5ODg1NTM4MDY=,1457,suggestion: distinguish names in `--static` documentation,51016,ctb,closed,0,,,,,0,2021-09-05T17:04:27Z,2021-10-14T18:39:55Z,2021-10-14T18:39:55Z,CONTRIBUTOR,,"Over in https://docs.datasette.io/en/stable/custom_templates.html#serving-static-files, there is the slightly comical example command -

```
datasette -m metadata.json --static static:static/ --memory
```

(now, with MORE STATIC!)

It took me a while to sort out all the URLs and paths involved because I wasn't being very clever. But in the interests of simplification and distinction, I might suggest something like

```
datasette -m metadata.json --static loc:static-files/ --memory
```

I will submit a PR for your consideration.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1457/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
988555009,MDExOlB1bGxSZXF1ZXN0NzI3NDM2NDA0,1458,Rework the `--static` documentation a bit,51016,ctb,closed,0,,,,,1,2021-09-05T17:08:48Z,2021-10-15T13:24:29Z,2021-10-14T18:39:55Z,CONTRIBUTOR,simonw/datasette/pulls/1458,"Per https://github.com/simonw/datasette/issues/1457, I was confused by the current docs and took a few minutes to sort out what the right combination of locations was.

This is a PR that differentiates the docs to split out  `/static/` in URL from `--static` option and `./static/` path.

Not wedded to the details in any way, happy to change to suit.

Fixes #1457.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1458/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
988556488,MDU6SXNzdWU5ODg1NTY0ODg=,1459,suggestion: allow `datasette --open` to take a relative URL,51016,ctb,open,0,,,,,1,2021-09-05T17:17:07Z,2021-09-05T19:59:15Z,,CONTRIBUTOR,,"(soft suggestion because I'm not sure I'm using datasette right yet)

Over at https://github.com/ctb/2021-sourmash-datasette, I'm playing around with datasette, and I'm creating some static pages to send people to the right facets. There may well be better ways of achieving this end goal, and I will find out if so, I'm sure!

But regardless I think it might be neat to support an option to allow `-o/--open` to take a relative URL, that then gets appended to the hostname and port. This would let me improve my documentation. I don't see any downsides, either, but 🤷 there may well be some :)

Happy to dig in and provide a PR if it's of interest. I'm not sure off the top of my head how to support an optional value to a parameter in argparse - the current `-o` behavior is kinda nice so it'd be suboptimal to require a url for `-o`. Maybe `--open-url=` or something would work?

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1459/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
276455748,MDU6SXNzdWUyNzY0NTU3NDg=,146,datasette publish gcloud,9599,simonw,closed,0,,,,,2,2017-11-23T18:55:03Z,2019-06-24T06:48:20Z,2019-06-24T06:48:20Z,OWNER,,"See also #103 

It looks like you can start a Google Cloud VM with a ""docker container"" option - and the Google Cloud Registry is easy to push containers to. So it would be feasible to have `datasette publish gcloud ...` automatically build a container, push it to GCR, then start a new VM instance with it:

https://cloud.google.com/container-registry/docs/pushing-and-pulling
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/146/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
989109888,MDU6SXNzdWU5ODkxMDk4ODg=,1460,Override column metadata with metadata from another column,72577720,MichaelTiemannOSC,open,0,,,,,0,2021-09-06T12:13:33Z,2021-09-06T12:13:33Z,,CONTRIBUTOR,,"I have a table from the PUDL project (https://github.com/catalyst-cooperative/pudl) that looks like this:

```
CREATE TABLE fuel_ferc1 (
	id INTEGER NOT NULL, 
	record_id TEXT, 
	utility_id_ferc1 INTEGER, 
	report_year INTEGER, 
	plant_name_ferc1 TEXT, 
	fuel_type_code_pudl VARCHAR(7), 
	fuel_unit VARCHAR(7),
	fuel_qty_burned FLOAT,
	fuel_mmbtu_per_unit FLOAT, 
	fuel_cost_per_unit_burned FLOAT, 
	fuel_cost_per_unit_delivered FLOAT, 
	fuel_cost_per_mmbtu FLOAT, 
	PRIMARY KEY (id), 
	FOREIGN KEY(plant_name_ferc1, utility_id_ferc1) REFERENCES plants_ferc1 (plant_name_ferc1, utility_id_ferc1), 
	CONSTRAINT fuel_ferc1_fuel_type_code_pudl_enum CHECK (fuel_type_code_pudl IN ('coal', 'oil', 'gas', 'solar', 'wind', 'hydro', 'nuclear', 'waste', 'unknown')), 
	CONSTRAINT fuel_ferc1_fuel_unit_enum CHECK (fuel_unit IN ('ton', 'mcf', 'bbl', 'gal', 'kgal', 'gramsU', 'kgU', 'klbs', 'btu', 'mmbtu', 'mwdth', 'mwhth', 'unknown'))
);
```

Note that `fuel_unit` is a unit that **pint** can understand, and that `fuel_qty_burned` is a column of data that could be expressed in terms of actual units, not merely as a dimensionless number.  Ditto the `fuel_cost_per_unit_...` columns.  Is there a way to give a column a default metadata unit (such as *tons* or *USD/ton*) and then let that be overridden when the metadata in another column says *barrels* or *USD/gramsU*?

@catalyst-cooperative",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1460/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
989986586,MDU6SXNzdWU5ODk5ODY1ODY=,1461,Try blacken-docs,9599,simonw,closed,0,,,,,3,2021-09-07T13:28:50Z,2021-09-07T16:13:59Z,2021-09-07T16:13:59Z,OWNER,,https://github.com/asottile/blacken-docs,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1461/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
990367646,MDU6SXNzdWU5OTAzNjc2NDY=,1462,"Separate out ""debug"" options from ""root"" options",9599,simonw,open,0,,,,,1,2021-09-07T21:27:34Z,2021-09-07T21:34:33Z,,OWNER,,"> I ditched ""root"" for ""admin"" because root by default gives you a whole bunch of stuff which I think could be confusing:
> 
> <img width=""816"" alt=""Datasette___internal__temporary_and_datasette-app_—_Electron_Helper__Renderer__◂_npm_start_TMPDIR__var_folders_wr_hn3206rs1yzgq3r49bz8nvnh0000gn_T__XPC_FLAGS_0x0_—_180×58_and_datasette-app-support_—_pipenv_shell_▸_zsh_—_147×76"" src=""https://user-images.githubusercontent.com/9599/132412648-b768495b-ead2-4abe-875a-fd090fe3d71d.png"">
>
> Maybe the real problem here is that I'm conflating ""root"" permissions with ""debug"" options. Perhaps there should be an extra Datasette mode that unlocks debug tools for the root user?

_Originally posted by @simonw in https://github.com/simonw/datasette-app-support/issues/8#issuecomment-914638998_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1462/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
991121619,MDExOlB1bGxSZXF1ZXN0NzI5NjMyNjQz,1463,"Update beautifulsoup4 requirement from <4.10.0,>=4.8.1 to >=4.8.1,<4.11.0",49699333,dependabot[bot],closed,0,,,,,1,2021-09-08T13:09:38Z,2021-10-13T22:35:37Z,2021-10-13T22:35:36Z,CONTRIBUTOR,simonw/datasette/pulls/1463,"Updates the requirements on [beautifulsoup4](http://www.crummy.com/software/BeautifulSoup/bs4/) to permit the latest version.


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1463/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
991191951,MDU6SXNzdWU5OTExOTE5NTE=,1464,clean checkout & clean environment has test failures,51016,ctb,open,0,,,,,6,2021-09-08T14:16:23Z,2021-09-13T22:17:17Z,,CONTRIBUTOR,,"I followed the instructions [here](https://docs.datasette.io/en/stable/contributing.html#setting-up-a-development-environment), and even after running `python update-docs-help.py` I get the following failed tests -- any thoughts?

```
FAILED tests/test_api.py::test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3]
FAILED tests/test_api.py::test_searchmode[table_metadata1-_search=te*+AND+do*-expected_rows1]
FAILED tests/test_api.py::test_searchmode[table_metadata2-_search=te*+AND+do*&_searchmode=raw-expected_rows2]
```

This is with python 3.9.7 and lots of other packages, as in attached environment listing from `conda list`.
[conda-installed.txt](https://github.com/simonw/datasette/files/7129487/conda-installed.txt)
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1464/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
991206402,MDExOlB1bGxSZXF1ZXN0NzI5NzA0NTM3,1465,add support for -o --get /path,51016,ctb,open,0,,,,,0,2021-09-08T14:30:42Z,2021-09-08T14:31:45Z,,CONTRIBUTOR,simonw/datasette/pulls/1465,"Fixes https://github.com/simonw/datasette/issues/1459

Adds support for `--open --get /path` to be used in combination.

If `--open` is provided alone, datasette will open a web page to a default URL.
If `--get <url>` is provided alone, datasette will output the result of doing a GET to that URL and then exit.
If `--open --get <url>` are provided together, datasette will open a web page to that URL.

TODO items:
- [ ] update documentation
- [ ] print out error message when `--root --open --get <url>` is used
- [ ] adjust code to require that `<url>` start with a `/` when `-o --get <url>` is used
- [ ] add test(s)

note, '@CTB' is used in this PR to flag code that needs revisiting.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1465/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1,
991467558,MDU6SXNzdWU5OTE0Njc1NTg=,1466,Add Datasette Desktop to installation documentation,9599,simonw,closed,0,,,7571612,Datasette 0.60,2,2021-09-08T19:41:27Z,2022-01-13T22:28:28Z,2022-01-13T21:55:18Z,OWNER,,See https://datasette.io/desktop and https://simonwillison.net/2021/Sep/8/datasette-desktop/,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1466/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
991575770,MDExOlB1bGxSZXF1ZXN0NzMwMDIwODY3,1467,Add Authorization header when CORS flag is set,3058200,jameslittle230,closed,0,,,,,3,2021-09-08T22:14:41Z,2021-10-17T02:29:07Z,2021-10-14T18:54:18Z,NONE,simonw/datasette/pulls/1467,"This PR adds the [`Access-Control-Allow-Headers`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers) flag when CORS mode is enabled.

This would fix https://github.com/simonw/datasette-auth-tokens/issues/4. When making cross-origin requests, the server must respond with all allowable HTTP headers. A Datasette instance using auth tokens must accept the `Authorization` HTTP header in order for cross-origin authenticated requests to take place.

Please let me know if there's a better way of doing this! I couldn't figure out a way to change the app's response from the plugin itself, so I'm starting here. If you'd rather this logic live in the plugin, I'd love any guidance you're able to give.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1467/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
994390593,MDU6SXNzdWU5OTQzOTA1OTM=,1468,Faceting for custom SQL queries,72577720,MichaelTiemannOSC,closed,0,,,,,2,2021-09-13T02:52:16Z,2021-09-13T04:54:22Z,2021-09-13T04:54:17Z,CONTRIBUTOR,,"Facets are awesome.  But not when I need to join to tidy tables together.  Or even just running explicitly the default SQL query that simply lists all the rows and columns of a table (up to SIZE).  That is to say, when I browse a table, I see facets:

https://latest.datasette.io/fixtures/compound_three_primary_keys

But when I run a custom query, I don't:

https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101

Is there an idiom to cause custom SQL to come back with facet suggestions?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1468/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
994450961,MDU6SXNzdWU5OTQ0NTA5NjE=,1469,"Column cog shows ""facet by this"" when already default faceted",9599,simonw,closed,0,,,,,2,2021-09-13T04:51:26Z,2021-10-13T21:20:07Z,2021-10-13T21:20:07Z,OWNER,,"e.g. on https://covid-19.datasettes.com/covid/economist_excess_deaths

<img width=""782"" alt=""covid__economist_excess_deaths__17_576_rows_where_sorted_by_end_date_descending_and_New_Issue_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/133025772-beaf055e-374f-4180-aad3-bb0215e8aaf3.png"">

But if you add `?_facet=country` to the URL that goes away: https://covid-19.datasettes.com/covid/economist_excess_deaths?_facet_size=5&_facet=country

The logic that decides if the ""Facet by this"" item is shown does not take default `metadata.json` facets into account.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1469/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
276476670,MDU6SXNzdWUyNzY0NzY2NzA=,147,Tidy up design of the header of the table page,9599,simonw,closed,0,,,2919870,Foreign key edition,1,2017-11-23T21:52:58Z,2017-11-24T22:02:46Z,2017-11-24T22:02:46Z,OWNER,,"This is a bit messy:

<img width=""883"" alt=""fatal-police-shootings-data__fatal-police-shootings-data"" src=""https://user-images.githubusercontent.com/9599/33189214-9379807c-d055-11e7-96c4-58a70529cef0.png"">

Depends on #127 ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/147/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
995098231,MDU6SXNzdWU5OTUwOTgyMzE=,1470,?_sort=rowid with _next= returns error,19851673,eigenfoo,closed,0,,,,,4,2021-09-13T16:36:15Z,2021-10-18T19:30:15Z,2021-10-10T01:15:03Z,NONE,,"For example:

- Go to https://cryptics.eigenfoo.xyz/clues/clues?_next=100 (this is the second page of results in a Datasette site)
- Search anything using the FTS search bar. For example, searching for `hello` will take you to https://cryptics.eigenfoo.xyz/clues/clues?_search=hello&_sort=rowid&_next=100
- A `500 Error: list index out of range` is raised.

This is because the search URL includes the `&_next=100` UTM parameter, carried over from where the FTS search was run. However, there isn't a second page in the search results, so a `list index out of range` error is raised. You can confirm that removing this UTM parameter from the URL returns the appropriate search results.

The FTS search request should strip any `_next` UTM parameter.

---

```bash
datasette, version 0.58.1
sqlite-utils, version 3.17
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1470/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
996002181,PR_kwDOBm6k_c4rutdp,1471,Bump black from 21.7b0 to 21.9b0,49699333,dependabot[bot],closed,0,,,,,1,2021-09-14T13:10:35Z,2021-10-13T21:47:42Z,2021-10-13T21:47:42Z,CONTRIBUTOR,simonw/datasette/pulls/1471,"Bumps [black](https://github.com/psf/black) from 21.7b0 to 21.9b0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.9b0</h2>
<h3>Packaging</h3>
<ul>
<li>Fix missing modules in self-contained binaries (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2466"">#2466</a>)</li>
<li>Fix missing toml extra used during installation (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2475"">#2475</a>)</li>
</ul>
<h2>21.8b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Add support for formatting Jupyter Notebook files (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2357"">#2357</a>)</li>
<li>Move from <code>appdirs</code> dependency to <code>platformdirs</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2375"">#2375</a>)</li>
<li>Present a more user-friendly error if .gitignore is invalid (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2414"">#2414</a>)</li>
<li>The failsafe for accidentally added backslashes in f-string expressions has been hardened to handle more edge cases during quote normalization (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2437"">#2437</a>)</li>
<li>Avoid changing a function return type annotation's type to a tuple by adding a trailing comma (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2384"">#2384</a>)</li>
<li>Parsing support has been added for unparenthesized walruses in set literals, set comprehensions, and indices (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2447"">#2447</a>).</li>
<li>Pin <code>setuptools-scm</code> build-time dependency version (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2457"">#2457</a>)</li>
<li>Exclude typing-extensions version 3.10.0.1 due to it being broken on Python 3.10 (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2460"">#2460</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Replace sys.exit(-1) with raise ImportError as it plays more nicely with tools that scan installed packages (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2440"">#2440</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>The provided pre-commit hooks no longer specify <code>language_version</code> to avoid overriding <code>default_language_version</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2430"">#2430</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>21.9b0</h2>
<h3>Packaging</h3>
<ul>
<li>Fix missing modules in self-contained binaries (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2466"">#2466</a>)</li>
<li>Fix missing toml extra used during installation (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2475"">#2475</a>)</li>
</ul>
<h2>21.8b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Add support for formatting Jupyter Notebook files (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2357"">#2357</a>)</li>
<li>Move from <code>appdirs</code> dependency to <code>platformdirs</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2375"">#2375</a>)</li>
<li>Present a more user-friendly error if .gitignore is invalid (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2414"">#2414</a>)</li>
<li>The failsafe for accidentally added backslashes in f-string expressions has been
hardened to handle more edge cases during quote normalization (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2437"">#2437</a>)</li>
<li>Avoid changing a function return type annotation's type to a tuple by adding a
trailing comma (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2384"">#2384</a>)</li>
<li>Parsing support has been added for unparenthesized walruses in set literals, set
comprehensions, and indices (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2447"">#2447</a>).</li>
<li>Pin <code>setuptools-scm</code> build-time dependency version (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2457"">#2457</a>)</li>
<li>Exclude typing-extensions version 3.10.0.1 due to it being broken on Python 3.10
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/2460"">#2460</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Replace sys.exit(-1) with raise ImportError as it plays more nicely with tools that
scan installed packages (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2440"">#2440</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>The provided pre-commit hooks no longer specify <code>language_version</code> to avoid overriding
<code>default_language_version</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2430"">#2430</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.7b0&new-version=21.9b0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1471/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
999902754,I_kwDOBm6k_c47mU4i,1473,base logo link visits `undefined` rather than href url,192568,mroswell,open,0,,,,,2,2021-09-18T04:17:04Z,2021-09-19T00:45:32Z,,CONTRIBUTOR,,"I have two connected sites:
http://www.SaferOrToxic.org 
(a Hugo website)
and:
http://disinfectants.SaferOrToxic.org/disinfectants/listN
(a datasette table page)

The latter is linked as ""The List"" in the former's menu.
(I'd love a prettier URL, but that's what I've got.)

On:
http://disinfectants.SaferOrToxic.org/disinfectants/listN
... all the other menu links should point back to:
https://www.SaferOrToxic.org 
And they do!

But the logo, for some reason--though it has an href pointing to:
https://www.SaferOrToxic.org
Keeps going to this instead:
https://disinfectants.saferortoxic.org/disinfectants/undefined

What is causing that? How can I fix it?

In #1284 back in March, I was doing battle with the index.html template, in a still unresolved issue. (I wanted only a single table page at the root.)

But I thought, well, if I can't resolve that, at least I could just point the main website to the datasette page (""The List,"") and then have the List point back to the home website.

The menu hrefs to https://www.SaferOrToxic.org work just fine, exactly as they should, from the datasette page. Even the Home link works properly.

But the logo link keeps rewriting to: https://disinfectants.saferortoxic.org/disinfectants/undefined

This is the HTML:
```
<a class=""text-3xl font-bold leading-none"" href=""https://www.saferortoxic.org""><img src=""https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp"" class=""custom-logo"" alt=""Logo: Safer or Toxic?"" width=""300px""></a>
```


Is this somehow related to cloudflare?
Or something in the datasette code?

I'm starting to think it's a cloudflare issue.
<img width=""1046"" alt=""Screen Shot 2021-09-18 at 12 09 57 AM"" src=""https://user-images.githubusercontent.com/192568/133872043-5a008e14-88cb-43fd-b921-e63ff50636fb.png"">

Can I at least rule out it being a datasette issue?

My repository is here:
https://github.com/mroswell/list-N

(BTW, I couldn't figure out how to reference a local image, either, on the datasette side, which is why I'm using the image from the www home page.)
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1473/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1000779422,PR_kwDOBm6k_c4r9CTw,1474,Update full_text_search.rst,72577720,MichaelTiemannOSC,closed,0,,,,,0,2021-09-20T09:59:45Z,2021-10-13T21:10:23Z,2021-10-13T21:10:23Z,CONTRIBUTOR,simonw/datasette/pulls/1474,"Change ""above"" to ""below"" to correct correspondence of reference to example.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1474/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1001104942,PR_kwDOBm6k_c4r-EVH,1475,feat: allow joins using _through in both directions,5268174,bram2000,open,0,,,,,0,2021-09-20T15:28:20Z,2021-09-20T15:28:20Z,,FIRST_TIME_CONTRIBUTOR,simonw/datasette/pulls/1475,"Currently the `_through` clause can only work if the FK relationship is defined in a specific direction. I don't think there is any reason for this limitation, as an FK allows joining in both directions.

This is an admittedly hacky change to implement bidirectional joins using `_through`. It does work for our use-case, but I don't know if there are other implications that I haven't thought of. Also if this change is desirable we probably want to make the code a little nicer.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1475/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1002459220,PR_kwDOBm6k_c4sCfBT,1476,"Update pytest-xdist requirement from <2.4,>=2.2.1 to >=2.2.1,<2.5",49699333,dependabot[bot],closed,0,,,,,1,2021-09-21T13:13:01Z,2021-10-13T21:10:03Z,2021-10-13T21:10:03Z,CONTRIBUTOR,simonw/datasette/pulls/1476,"Updates the requirements on [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst"">pytest-xdist's changelog</a>.</em></p>
<blockquote>
<h1>pytest-xdist 2.4.0 (2021-09-20)</h1>
<h2>Features</h2>
<ul>
<li>
<p><code>[#696](https://github.com/pytest-dev/pytest-xdist/issues/696) &lt;https://github.com/pytest-dev/pytest-xdist/issues/696&gt;</code>_: On Linux, the process title now changes to indicate the current worker state (running/idle).</p>
<p>Depends on the <code>setproctitle &lt;https://pypi.org/project/setproctitle/&gt;</code>__ package, which can be installed with <code>pip install pytest-xdist[setproctitle]</code>.</p>
</li>
<li>
<p><code>[#704](https://github.com/pytest-dev/pytest-xdist/issues/704) &lt;https://github.com/pytest-dev/pytest-xdist/issues/704&gt;</code>_: Add support for Python 3.10.</p>
</li>
</ul>
<h1>pytest-xdist 2.3.0 (2021-06-16)</h1>
<h2>Deprecations and Removals</h2>
<ul>
<li><code>[#654](https://github.com/pytest-dev/pytest-xdist/issues/654) &lt;https://github.com/pytest-dev/pytest-xdist/issues/654&gt;</code>_: Python 3.5 is no longer supported.</li>
</ul>
<h2>Features</h2>
<ul>
<li>
<p><code>[#646](https://github.com/pytest-dev/pytest-xdist/issues/646) &lt;https://github.com/pytest-dev/pytest-xdist/issues/646&gt;</code>_: Add <code>--numprocesses=logical</code> flag, which automatically uses the number of logical CPUs available, instead of physical CPUs with <code>auto</code>.</p>
<p>This is very useful for test suites which are not CPU-bound.</p>
</li>
<li>
<p><code>[#650](https://github.com/pytest-dev/pytest-xdist/issues/650) &lt;https://github.com/pytest-dev/pytest-xdist/issues/650&gt;</code>_: Added new <code>pytest_handlecrashitem</code> hook to allow handling and rescheduling crashed items.</p>
</li>
</ul>
<h2>Bug Fixes</h2>
<ul>
<li>
<p><code>[#421](https://github.com/pytest-dev/pytest-xdist/issues/421) &lt;https://github.com/pytest-dev/pytest-xdist/issues/421&gt;</code>_: Copy the parent process sys.path into local workers, to work around execnet's python -c adding the current directory to sys.path.</p>
</li>
<li>
<p><code>[#638](https://github.com/pytest-dev/pytest-xdist/issues/638) &lt;https://github.com/pytest-dev/pytest-xdist/issues/638&gt;</code>_: Fix issue caused by changing the branch name of the pytest repository.</p>
</li>
</ul>
<h2>Trivial Changes</h2>
<ul>
<li>
<p><code>[#592](https://github.com/pytest-dev/pytest-xdist/issues/592) &lt;https://github.com/pytest-dev/pytest-xdist/issues/592&gt;</code>_: Replace master with controller where ever possible.</p>
</li>
<li>
<p><code>[#643](https://github.com/pytest-dev/pytest-xdist/issues/643) &lt;https://github.com/pytest-dev/pytest-xdist/issues/643&gt;</code>_: Use 'main' to refer to pytest default branch in tox env names.</p>
</li>
</ul>
<h1>pytest-xdist 2.2.1 (2021-02-09)</h1>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/4b487ed5e1932992bb2fd92cb0d04df5fac4c843""><code>4b487ed</code></a> Manually fix changelog title</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/ecf4d3be31a71365026f0bbb8d67c157f75d695d""><code>ecf4d3b</code></a> Release 2.4.0</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/87d89795e36748e4d0112423570264fcf3c21763""><code>87d8979</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/704"">#704</a> from hugovk/add-3.10</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/b4544c8849eb25135113031f6eea277aa7216939""><code>b4544c8</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/706"">#706</a> from pytest-dev/pre-commit-ci-update-config</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/66dc3904b4a0bd1ac491af9eb4e59750f92668f5""><code>66dc390</code></a> [pre-commit.ci] pre-commit autoupdate</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/e0ce1b74186043da5e347d63a47a8d34ee4cd36c""><code>e0ce1b7</code></a> Add news file to add support for Python 3.10</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/ed47f0e01a38af95100f806d0cfc98f96723b721""><code>ed47f0e</code></a> Add support for Python 3.10</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/1c8178abc5b0acdb95938d4ea99344fe78383618""><code>1c8178a</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/703"">#703</a> from pytest-dev/pre-commit-ci-update-config</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/9807064f66d28bbb5aee1bdae0a3f47919c39206""><code>9807064</code></a> [pre-commit.ci] pre-commit autoupdate</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/766e67ce526c88327670e9bcd4804c335df51015""><code>766e67c</code></a> Use setproctitle if available to show state (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/696"">#696</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/pytest-dev/pytest-xdist/compare/v2.2.1...v2.4.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1476/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1006016302,I_kwDOBm6k_c479pcu,1477,Consider adding request to the documented default template context,9599,simonw,open,0,,,,,0,2021-09-24T02:34:09Z,2021-09-24T02:34:09Z,,OWNER,,I made a plugin for this today but I think perhaps it should be a default thing instead: https://datasette.io/plugins/datasette-template-request,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1477/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1006781949,I_kwDOBm6k_c48AkX9,1478,Documentation Request: Feature alternative ID instead of default ID,192568,mroswell,open,0,,,,,0,2021-09-24T19:56:13Z,2021-09-25T16:18:54Z,,CONTRIBUTOR,,"My data already has an ID that comes from a federal agency.
Would love to have documentation on how to modify the template to:
- Remove the generated ID from the table
- Link the federal ID to the detail page
- and to ensure that the JSON file uses that as the ID. I'd be happy to include the database ID in the export, but not as a key.

I don't want to remove the ID from the database, though, because my experience with the federal agency is that data often has anomalies. I don't want all hell to break loose if they end up applying the same ID to multiple rows (which they haven't done yet). I just don't want it to display in the table or the data exports.

Perhaps this isn't a template issue, maybe more of a db manipulation...

Margie",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1478/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1010112818,I_kwDOBm6k_c48NRky,1479,"Win32 ""used by another process"" error with datasette publish",76450761,kirajano,open,0,,,,,7,2021-09-28T19:12:00Z,2023-09-07T02:14:16Z,,NONE,,"I unfortunately was not successful to deploy to fly.io. Please see the details above of the three scenarios that I took. I am also new to datasette.

Failed to deploy. Attaching logs:
1. Tried with an app created via `flyctl apps create frosty-fog-8565` and the ran `datasette publish fly covid.db --app frosty-fog-8565` 
``` 
Deploying frosty-fog-8565
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080

Error error connecting to docker: An unknown error occured.

Traceback (most recent call last):
  File ""c:\users\grott\anaconda3\lib\runpy.py"", line 193, in _run_module_as_main
    ""__main__"", mod_spec)
  File ""c:\users\grott\anaconda3\lib\runpy.py"", line 85, in _run_code
    exec(code, run_globals)
  File ""C:\Users\grott\Anaconda3\Scripts\datasette.exe\__main__.py"", line 7, in <module>
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""c:\users\grott\anaconda3\lib\site-packages\datasette_publish_fly\__init__.py"", line 156, in fly
    ""--remote-only"",
  File ""c:\users\grott\anaconda3\lib\contextlib.py"", line 119, in __exit__
    next(self.gen)
  File ""c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py"", line 451, in temporary_docker_directory
    tmp.cleanup()
  File ""c:\users\grott\anaconda3\lib\tempfile.py"", line 811, in cleanup
    _shutil.rmtree(self.name)
  File ""c:\users\grott\anaconda3\lib\shutil.py"", line 516, in rmtree
    return _rmtree_unsafe(path, onerror)
  File ""c:\users\grott\anaconda3\lib\shutil.py"", line 395, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File ""c:\users\grott\anaconda3\lib\shutil.py"", line 404, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File ""c:\users\grott\anaconda3\lib\shutil.py"", line 402, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\grott\\AppData\\Local\\Temp\\tmpgcm8cz66\\frosty-fog-8565'
```

2. Tried also with an app that gets autogenerate when running `flyctl launch`. This also generates the .toml file. Ran then `datasette publish fly covid.db --app dark-feather-168` **but different error now**
```Deploying dark-feather-168
==> Validating app configuration

Error not possible to validate configuration: server returned Post ""https://api.fly.io/graphql"": unexpected EOF

Traceback (most recent call last):
  File ""c:\users\grott\anaconda3\lib\runpy.py"", line 193, in _run_module_as_main        
    ""__main__"", mod_spec)
    exec(code, run_globals)
  File ""C:\Users\grott\Anaconda3\Scripts\datasette.exe\__main__.py"", line 7, in <module>
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""c:\users\grott\anaconda3\lib\site-packages\datasette_publish_fly\__init__.py"", line 156, in fly
    ""--remote-only"",
  File ""c:\users\grott\anaconda3\lib\contextlib.py"", line 119, in __exit__
    next(self.gen)
  File ""c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py"", line 451, in temporary_docker_directory
    tmp.cleanup()
  File ""c:\users\grott\anaconda3\lib\tempfile.py"", line 811, in cleanup
    _shutil.rmtree(self.name)
  File ""c:\users\grott\anaconda3\lib\shutil.py"", line 516, in rmtree
    return _rmtree_unsafe(path, onerror)
  File ""c:\users\grott\anaconda3\lib\shutil.py"", line 395, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File ""c:\users\grott\anaconda3\lib\shutil.py"", line 404, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File ""c:\users\grott\anaconda3\lib\shutil.py"", line 402, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\grott\\AppData\\Local\\Temp\\tmpnoyewcre\\dark-feather-168'
```

These are also the contents of the generated **.toml file** in 2 scenario:

```
# fly.toml file generated for dark-feather-168 on 2021-09-28T20:35:44+02:00

app = ""dark-feather-168""

kill_signal = ""SIGINT""
kill_timeout = 5
processes = []

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = [""app""]
  protocol = ""tcp""
  script_checks = []

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = ""connections""

  [[services.ports]]
    handlers = [""http""]
    port = 80

  [[services.ports]]
    handlers = [""tls"", ""http""]
    port = 443

  [[services.tcp_checks]]
    grace_period = ""1s""
    interval = ""15s""
    restart_limit = 6
    timeout = ""2s""
```

3. But also trying `datasette package covid.db` to create a local DOCKERFILE to later try to push it via `flyctl deploy` fails as well.

```[+] Building 147.3s (11/11) FINISHED
 => [internal] load build definition from Dockerfile                                                             0.2s 
 => => transferring dockerfile: 396B                                                                             0.0s 
 => [internal] load .dockerignore                                                                                0.1s 
 => => transferring context: 2B                                                                                  0.0s 
 => [internal] load metadata for docker.io/library/python:3.8                                                    4.7s 
 => [auth] library/python:pull token for registry-1.docker.io                                                    0.0s 
 => [internal] load build context                                                                                0.1s 
 => => transferring context: 82.37kB                                                                             0.0s 
 => [1/5] FROM docker.io/library/python:3.8@sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3  108.3s 
 => => resolve docker.io/library/python:3.8@sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3b5  0.0s 
 => => sha256:56182bcdf4d4283aa1f46944b4ef7ac881e28b4d5526720a4e9ba03a4730846a 2.22kB / 2.22kB                   0.0s 
 => => sha256:955615a668ce169f8a1443fc6b6e6215f43fe0babfb4790712a2d3171f34d366 54.93MB / 54.93MB                21.6s 
 => => sha256:911ea9f2bd51e53a455297e0631e18a72a86d7e2c8e1807176e80f991bde5d64 10.87MB / 10.87MB                15.5s 
 => => sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3b5c5b6 1.86kB / 1.86kB                   0.0s 
 => => sha256:ff08f08727e50193dcf499afc30594c47e70cc96f6fcfd1a01240524624264d0 8.65kB / 8.65kB                   0.0s 
 => => sha256:2756ef5f69a5190f4308619e0f446d95f5515eef4a814dbad0bcebbbbc7b25a8 5.15MB / 5.15MB                   6.4s 
 => => sha256:27b0a22ee906271a6ce9ddd1754fdd7d3b59078e0b57b6cc054c7ed7ac301587 54.57MB / 54.57MB                37.7s 
 => => sha256:8584d51a9262f9a3a436dea09ba40fa50f85802018f9bd299eee1bf538481077 196.45MB / 196.45MB              82.3s 
 => => sha256:524774b7d3638702fe9ae0ea3fcfb81b027dfd75cc2fc14f0119e764b9543d58 6.29MB / 6.29MB                  26.6s 
 => => extracting sha256:955615a668ce169f8a1443fc6b6e6215f43fe0babfb4790712a2d3171f34d366                        5.4s 
 => => sha256:9460f6b75036e38367e2f27bb15e85777c5d6cd52ad168741c9566186415aa26 16.81MB / 16.81MB                40.5s 
 => => extracting sha256:2756ef5f69a5190f4308619e0f446d95f5515eef4a814dbad0bcebbbbc7b25a8                        0.6s 
 => => extracting sha256:911ea9f2bd51e53a455297e0631e18a72a86d7e2c8e1807176e80f991bde5d64                        0.6s 
 => => sha256:9bc548096c181514aa1253966a330134d939496027f92f57ab376cd236eb280b 232B / 232B                      40.1s 
 => => extracting sha256:27b0a22ee906271a6ce9ddd1754fdd7d3b59078e0b57b6cc054c7ed7ac301587                        5.8s 
 => => sha256:1d87379b86b89fd3b8bb1621128f00c8f962756e6aaaed264ec38db733273543 2.35MB / 2.35MB                  41.8s 
 => => extracting sha256:8584d51a9262f9a3a436dea09ba40fa50f85802018f9bd299eee1bf538481077                       18.8s 
 => => extracting sha256:524774b7d3638702fe9ae0ea3fcfb81b027dfd75cc2fc14f0119e764b9543d58                        1.2s 
 => => extracting sha256:9460f6b75036e38367e2f27bb15e85777c5d6cd52ad168741c9566186415aa26                        2.9s 
 => => extracting sha256:9bc548096c181514aa1253966a330134d939496027f92f57ab376cd236eb280b                        0.0s 
 => => extracting sha256:1d87379b86b89fd3b8bb1621128f00c8f962756e6aaaed264ec38db733273543                        0.8s 
 => [2/5] COPY . /app                                                                                            2.3s 
 => [3/5] WORKDIR /app                                                                                           0.2s 
 => [4/5] RUN pip install -U datasette                                                                          26.9s 
 => [5/5] RUN datasette inspect covid.db --inspect-file inspect-data.json                                        3.1s
 => exporting to image                                                                                           1.2s 
 => => exporting layers                                                                                          1.2s 
 => => writing image sha256:b5db0c205cd3454c21fbb00ecf6043f261540bcf91c2dfc36d418f1a23a75d7a                     0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Traceback (most recent call last):
    ""__main__"", mod_spec)
  File ""c:\users\grott\anaconda3\lib\runpy.py"", line 85, in _run_code
    exec(code, run_globals)
  File ""C:\Users\grott\Anaconda3\Scripts\datasette.exe\__main__.py"", line 7, in <module>
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""c:\users\grott\anaconda3\lib\site-packages\datasette\cli.py"", line 283, in package
    call(args)
  File ""c:\users\grott\anaconda3\lib\contextlib.py"", line 119, in __exit__
    next(self.gen)
  File ""c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py"", line 451, in temporary_docker_directory
    tmp.cleanup()
  File ""c:\users\grott\anaconda3\lib\tempfile.py"", line 811, in cleanup
    _shutil.rmtree(self.name)
  File ""c:\users\grott\anaconda3\lib\shutil.py"", line 516, in rmtree
    return _rmtree_unsafe(path, onerror)
  File ""c:\users\grott\anaconda3\lib\shutil.py"", line 395, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File ""c:\users\grott\anaconda3\lib\shutil.py"", line 404, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File ""c:\users\grott\anaconda3\lib\shutil.py"", line 402, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\grott\\AppData\\Local\\Temp\\tmpkb27qid3\\datasette'```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1479/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
276477888,MDU6SXNzdWUyNzY0Nzc4ODg=,148,Need a != filter,9599,simonw,closed,0,,,2919870,Foreign key edition,0,2017-11-23T22:05:22Z,2017-11-23T22:10:02Z,2017-11-23T22:10:01Z,OWNER,,https://datasette-demos.now.sh/sf-trees-ebc2ad9/Street_Tree_List?qCareAssistant=1 shows trees managed by FUF - but how about trees that are NOT managed by FUF?,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/148/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1015646369,I_kwDOBm6k_c48iYih,1480,Exceeding Cloud Run memory limits when deploying a 4.8G database,110420,ghing,open,0,,,,,9,2021-10-04T21:20:24Z,2022-10-07T04:39:10Z,,CONTRIBUTOR,,"When I try to deploy a 4.8G SQLite database to Google Cloud Run, I get this error message:

> Memory limit of 8192M exceeded with 8826M used. Consider increasing the memory limit, see https://cloud.google.com/run/docs/configuring/memory-limits

Unfortunately, the maximum amount of memory that can be allocated to an instance is 8192M.

Naively profiling the memory usage of running Datasette with this database locally on my MacBook shows the following memory usage (using Activity Monitor) when I just start up Datasette locally:

- Real Memory Size: 70.6 MB
- Virtual Memory Size: 4.51 GB
- Shared Memory Size: 2.5 MB
- Private Memory Size: 57.4 MB

I'm trying to understand if there's a query or other operation that gets run during container deployment that causes memory use to be so large and if this can be avoided somehow.

This is somewhat related to #1082, but on a different platform, so I decided to open a new issue.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1480/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1020436713,PR_kwDOBm6k_c4s6bJm,1481,Fix compatibility with Python 3.10,9599,simonw,closed,0,,,,,13,2021-10-07T20:34:23Z,2021-10-24T22:19:55Z,2021-10-24T22:19:54Z,OWNER,simonw/datasette/pulls/1481,,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1481/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1021550542,I_kwDOBm6k_c4845_O,1482,Support Python 3.10,9599,simonw,closed,0,,,,,2,2021-10-09T00:30:52Z,2021-10-24T22:21:40Z,2021-10-24T22:19:55Z,OWNER,,"I started work on this in #1481 where I found a Python 3.10 bug that needs a workaround in Janus, see:

- https://github.com/aio-libs/janus/issues/358

This is a tracking issue for anything else that shows up.

This is also needed for the Homebrew package to upgrade to 3.10:

- https://github.com/Homebrew/homebrew-core/pull/86932",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1482/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1021849766,I_kwDOBm6k_c486DCm,1483,Running a search on page 2 of results should not preserve ?_next=,9599,simonw,closed,0,,,,,0,2021-10-10T01:18:12Z,2021-10-13T21:08:10Z,2021-10-13T21:08:10Z,OWNER,,Reported by @eigenfoo in https://github.com/simonw/datasette/issues/1470,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1483/reactions"", ""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1022294524,PR_kwDOBm6k_c4s_4Cw,1484,GitHub Actions: Add Python 3.10 to the tests,3709715,cclauss,closed,0,,,,,0,2021-10-11T06:03:03Z,2021-10-11T06:03:31Z,2021-10-11T06:03:28Z,FIRST_TIME_CONTRIBUTOR,simonw/datasette/pulls/1484,,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1484/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1022688960,PR_kwDOBm6k_c4tBIEE,1485,"Update pytest-timeout requirement from <1.5,>=1.4.2 to >=1.4.2,<2.1",49699333,dependabot[bot],closed,0,,,,,1,2021-10-11T13:10:51Z,2021-10-13T21:09:23Z,2021-10-13T21:09:23Z,CONTRIBUTOR,simonw/datasette/pulls/1485,"Updates the requirements on [pytest-timeout](https://github.com/pytest-dev/pytest-timeout) to permit the latest version.
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/pytest-dev/pytest-timeout/commits"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1485/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1023243105,I_kwDOBm6k_c48_XNh,1486,pipx installation instructions for plugins don't reference pipx inject,41546558,RhetTbull,closed,0,,,,,0,2021-10-12T00:43:42Z,2021-10-13T21:09:11Z,2021-10-13T21:09:11Z,CONTRIBUTOR,,"The datasette [installation instructions](https://github.com/simonw/datasette/blob/main/docs/installation.rst) discuss how to install with pipx, how to upgrade with pipx, and how to upgrade plugins with pipx but do not mention how to install a plugin with pipx.  You discussed this on your [blog](https://til.simonwillison.net/python/installing-upgrading-plugins-with-pipx) but looks like this didn't make it in when you updated the docs for pipx (#756). 

I'll submit a PR shortly to fix this.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1486/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1023245060,PR_kwDOBm6k_c4tC4Lx,1487,"Added instructions for installing plugins via pipx, #1486",41546558,RhetTbull,closed,0,,,,,1,2021-10-12T00:48:30Z,2021-10-13T21:09:11Z,2021-10-13T21:09:10Z,CONTRIBUTOR,simonw/datasette/pulls/1487,Adds missing instructions for installing plugins via pipx,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1487/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1025754125,I_kwDOBm6k_c49I8QN,1488,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'),9599,simonw,closed,0,,,,,5,2021-10-13T22:37:22Z,2021-10-14T18:03:45Z,2021-10-14T18:03:45Z,OWNER,,This is caused by a change made to `httpx` in https://github.com/encode/httpx/releases/tag/0.20.0,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1488/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1026379132,PR_kwDOBm6k_c4tM0JV,1489,"Update pyyaml requirement from ~=5.3 to >=5.3,<7.0",49699333,dependabot[bot],closed,0,,,,,3,2021-10-14T13:09:33Z,2021-10-14T18:10:43Z,2021-10-14T18:10:42Z,CONTRIBUTOR,simonw/datasette/pulls/1489,"Updates the requirements on [pyyaml](https://github.com/yaml/pyyaml) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/yaml/pyyaml/blob/master/CHANGES"">pyyaml's changelog</a>.</em></p>
<blockquote>
<p>6.0 (2021-10-13)</p>
<ul>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/327"">yaml/pyyaml#327</a> -- Change README format to Markdown</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/483"">yaml/pyyaml#483</a> -- Add a test for YAML 1.1 types</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/497"">yaml/pyyaml#497</a> -- fix float resolver to ignore <code>.</code> and <code>._</code></li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/550"">yaml/pyyaml#550</a> -- drop Python 2.7</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/553"">yaml/pyyaml#553</a> -- Fix spelling of “hexadecimal”</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/556"">yaml/pyyaml#556</a> -- fix representation of Enum subclasses</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/557"">yaml/pyyaml#557</a> -- fix libyaml extension compiler warnings</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/560"">yaml/pyyaml#560</a> -- fix ResourceWarning on leaked file descriptors</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/561"">yaml/pyyaml#561</a> -- always require <code>Loader</code> arg to <code>yaml.load()</code></li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/564"">yaml/pyyaml#564</a> -- remove remaining direct distutils usage</li>
</ul>
<p>5.4.1 (2021-01-20)</p>
<ul>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/480"">yaml/pyyaml#480</a> -- Fix stub compat with older pyyaml versions that may unwittingly load it</li>
</ul>
<p>5.4 (2021-01-19)</p>
<ul>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/407"">yaml/pyyaml#407</a> -- Build modernization, remove distutils, fix metadata, build wheels, CI to GHA</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/472"">yaml/pyyaml#472</a> -- Fix for CVE-2020-14343, moves arbitrary python tags to UnsafeLoader</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/441"">yaml/pyyaml#441</a> -- Fix memory leak in implicit resolver setup</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/392"">yaml/pyyaml#392</a> -- Fix py2 copy support for timezone objects</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/378"">yaml/pyyaml#378</a> -- Fix compatibility with Jython</li>
</ul>
<p>5.3.1 (2020-03-18)</p>
<ul>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/386"">yaml/pyyaml#386</a> -- Prevents arbitrary code execution during python/object/new constructor</li>
</ul>
<p>5.3 (2020-01-06)</p>
<ul>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/290"">yaml/pyyaml#290</a> -- Use <code>is</code> instead of equality for comparing with <code>None</code></li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/270"">yaml/pyyaml#270</a> -- Fix typos and stylistic nit</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/309"">yaml/pyyaml#309</a> -- Fix up small typo</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/161"">yaml/pyyaml#161</a> -- Fix handling of <strong>slots</strong></li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/358"">yaml/pyyaml#358</a> -- Allow calling add_multi_constructor with None</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/285"">yaml/pyyaml#285</a> -- Add use of safe_load() function in README</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/351"">yaml/pyyaml#351</a> -- Fix reader for Unicode code points over 0xFFFF</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/360"">yaml/pyyaml#360</a> -- Enable certain unicode tests when maxunicode not &gt; 0xffff</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/359"">yaml/pyyaml#359</a> -- Use full_load in yaml-highlight example</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/244"">yaml/pyyaml#244</a> -- Document that PyYAML is implemented with Cython</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/329"">yaml/pyyaml#329</a> -- Fix for Python 3.10</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/310"">yaml/pyyaml#310</a> -- Increase size of index, line, and column fields</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/260"">yaml/pyyaml#260</a> -- Remove some unused imports</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/163"">yaml/pyyaml#163</a> -- Create timezone-aware datetimes when parsed as such</li>
<li><a href=""https://github-redirect.dependabot.com/yaml/pyyaml/pull/363"">yaml/pyyaml#363</a> -- Add tests for timezone</li>
</ul>
<h2>5.2 (2019-12-02)</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/yaml/pyyaml/commit/8cdff2c80573b8be8e8ad28929264a913a63aa33""><code>8cdff2c</code></a> 6.0 release</li>
<li><a href=""https://github.com/yaml/pyyaml/commit/a4fb55e943abf25055725245b0779cf527069a78""><code>a4fb55e</code></a> Update Python 3.10 versions for Windows build</li>
<li><a href=""https://github.com/yaml/pyyaml/commit/e45b9643de3f7332e874ac67357ffd6dd5200211""><code>e45b964</code></a> Add Python 3.10 to tox.ini</li>
<li><a href=""https://github.com/yaml/pyyaml/commit/4808fba18e067a93a4fcc25ddda1aae9b976ceb8""><code>4808fba</code></a> 6.0b1 release</li>
<li><a href=""https://github.com/yaml/pyyaml/commit/d5aba406edc9d2618fc9d6ffccd257fe593e4283""><code>d5aba40</code></a> Omnibus CI/artifact build update</li>
<li><a href=""https://github.com/yaml/pyyaml/commit/a6d384c52ea54ded80df27032c7b6d3a98b0c2fa""><code>a6d384c</code></a> Various setup fixes</li>
<li><a href=""https://github.com/yaml/pyyaml/commit/8f3f9795b4ccc7087c439aad8342eaf916f273d0""><code>8f3f979</code></a> No longer using appveyor</li>
<li><a href=""https://github.com/yaml/pyyaml/commit/c2743653bc89e42679ba097b4f9888db47c61d63""><code>c274365</code></a> The yaml.load{,_all} functions require Loader= now</li>
<li><a href=""https://github.com/yaml/pyyaml/commit/2f87ac4838840a2e3cb48504dcd74041921ab184""><code>2f87ac4</code></a> Add a basic test file for yaml.load and yaml.dump</li>
<li><a href=""https://github.com/yaml/pyyaml/commit/7bd92df5435030b9a32f8843369ad6673d0d9294""><code>7bd92df</code></a> Makefile tweaks</li>
<li>Additional commits viewable in <a href=""https://github.com/yaml/pyyaml/compare/5.3...6.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1489/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
276704127,MDU6SXNzdWUyNzY3MDQxMjc=,149,Update custom SQL results to match new table view header,9599,simonw,closed,0,,,2919870,Foreign key edition,1,2017-11-24T22:03:59Z,2017-11-24T22:42:10Z,2017-11-24T22:42:09Z,OWNER,,"Follow-on from #147 - the custom SQL results page should more closely match the design of the table view, which now looks like this:

<img width=""798"" alt=""conventional_power_plants_eu__conventional_power_plants_eu__14_rows_where_company____nuon_"" src=""https://user-images.githubusercontent.com/9599/33224621-4d547f18-d120-11e7-9b33-44ee738a5ea2.png"">
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/149/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1026664511,PR_kwDOBm6k_c4tNtoe,1490,Upgrade to httpx 0.20,9599,simonw,closed,0,,,,,0,2021-10-14T17:51:05Z,2021-10-14T18:03:45Z,2021-10-14T18:03:44Z,OWNER,simonw/datasette/pulls/1490,Refs #1488 ,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1490/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1028115674,I_kwDOBm6k_c49R8za,1493,`--get '/:memory:.json?sql=select+3*5'` error with datasette 0.59,1580956,chenrui333,closed,0,,,,,1,2021-10-16T18:22:22Z,2021-10-19T04:39:11Z,2021-10-19T04:39:11Z,NONE,,"👋  trying to upgrade the formula to use the latest release, but runs into some regression test issue with `--get` command.

My QQ is does this `datasette --get '/:memory:.json?sql=select+3*5'` supposed to return 15? Thanks!

relates to https://github.com/Homebrew/homebrew-core/pull/87369",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1493/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1029100823,PR_kwDOBm6k_c4tU5cz,1494,"Update pytest-asyncio requirement from <0.16,>=0.10 to >=0.10,<0.17",49699333,dependabot[bot],closed,0,,,,,1,2021-10-18T13:14:17Z,2021-10-24T22:22:40Z,2021-10-24T22:22:39Z,CONTRIBUTOR,simonw/datasette/pulls/1494,"Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version.
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/f2fe98e5cbc4a25e31c24f5932d66f9c903f0fe5""><code>f2fe98e</code></a> 0.16.0</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/4e1df3191aee1a902c873901b35d82363e13b08c""><code>4e1df31</code></a> Remove obsolete test, add make test</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/6ec76477061ea14394cadbf2cef673b04971ef4d""><code>6ec7647</code></a> feat: Add support for Python 3.10.</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/42ff5d176b2be5aa77a43f586179d563ae2196b5""><code>42ff5d1</code></a> ci: Include Python 3.10 in the CI test run.</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/be3b32777b32774c288b604f0d95eacb32a4d1a2""><code>be3b327</code></a> build: Include Python 3.10 in Tox test runs.</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/1c283bd821249ee3c695ae59bb022d31a1473ae2""><code>1c283bd</code></a> refactor: test_async_fixtures_with_finalizer no longer trigger a DeprecationW...</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/2751982f3a84d55aad103ea039cc779537e90720""><code>2751982</code></a> refactor: Replaced tests asserting that the event loop is properly closed.</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/70989fd7b626a86db4f75acc40057f982f2c76dd""><code>70989fd</code></a> refactor: Grouped test cases together that are related to the use of the asyn...</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/b27abe8d138998a75cc4eb519be810439d3ed0d9""><code>b27abe8</code></a> refactor: Removed TestUnexistingLoop.remove_loop fixture, because it has no e...</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/e3ec312c1d55b3688e4ce115df7791a0478582d1""><code>e3ec312</code></a> Adjusted Hypothesis integration test to use the same event loop initializatio...</li>
<li>Additional commits viewable in <a href=""https://github.com/pytest-dev/pytest-asyncio/compare/v0.10.0...v0.16.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1494/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1033678984,PR_kwDOBm6k_c4tjgJ8,1495,Allow routes to have extra options,536941,fgregg,open,0,,,,,5,2021-10-22T15:00:45Z,2021-11-19T15:36:27Z,,CONTRIBUTOR,simonw/datasette/pulls/1495,"Right now, datasette routes can only be a 2-tuple of `(regex, view_fn)`. 

If it was possible for datasette to handle extra options, like [standard Django does](https://docs.djangoproject.com/en/3.2/topics/http/urls/#passing-extra-options-to-view-functions), it would add flexibility for plugin authors.

For example, if extra options were enabled, then it would be easy to make a single table the home page (#1284). This plugin would accomplish it.

```python
from datasette import hookimpl
from datasette.views.table import TableView

@hookimpl
def register_routes(datasette):
        return [
        (r""^/$"", TableView.as_view(datasette), {'db_name': 'DB_NAME',
                                                'table': 'TABLE_NAME'})
    ]
```
",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1495/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1033864602,I_kwDOBm6k_c49n4Wa,1496,Named parameters docs should include an example of a cast,9599,simonw,closed,0,,,,,1,2021-10-22T18:56:04Z,2021-10-22T19:38:23Z,2021-10-22T19:34:27Z,OWNER,,"https://docs.datasette.io/en/stable/sql_queries.html#named-parameters

It's not obvious that the values from parameters are always SQLite strings, which means that you can't do e.g. integer comparisons on them without casting them first. The documentation here should include an example of this.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1496/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1034535001,I_kwDOBm6k_c49qcBZ,1497,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",9599,simonw,closed,0,,,,,18,2021-10-24T22:57:07Z,2023-01-18T17:13:45Z,2021-10-24T23:36:55Z,OWNER,,"This means the Datasette 0.59.1 release has not been published to Docker Hub.

Here's where that failed: https://github.com/simonw/datasette/runs/3991043374?check_suite_focus=true

```
Preparing to unpack .../libc6_2.32-4_amd64.deb ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.28.1 /usr/local/share/perl/5.28.1 /usr/lib/x86_64-linux-gnu/perl5/5.28 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.28 /usr/share/perl/5.28 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Checking for services that may need to be restarted...
Checking init scripts...
Unpacking libc6:amd64 (2.32-4) over (2.28-10) ...
Setting up libc6:amd64 (2.32-4) ...
/usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
dpkg: error processing package libc6:amd64 (--configure):
 installed libc6:amd64 package post-installation script subprocess returned error exit status 127
Errors were encountered while processing:
 libc6:amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)
The command '/bin/sh -c apt-get update &&     apt-get -y --no-install-recommends install software-properties-common &&     add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" &&     apt-get update &&     apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite &&     apt-get remove -y software-properties-common &&     apt clean &&     rm -rf /var/lib/apt &&     rm -rf /var/lib/dpkg/info/*' returned a non-zero code: 100
```
Same problem when I attempted to publish using the ""Push specific Docker tag"" workflow: https://github.com/simonw/datasette/runs/3991059912?check_suite_focus=true",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1497/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
267713226,MDU6SXNzdWUyNjc3MTMyMjY=,15,Support multiple databases,9599,simonw,closed,0,,,2857392,Ship first public release,0,2017-10-23T15:29:51Z,2017-10-24T02:01:38Z,2017-10-24T02:01:38Z,OWNER,,"I'm going to loop through every database file in the app root directory and bundle all of them.

Each one will be accessible at /databasename

Note this is without the file extension, and we will disallow multiple files with the same name but different extensions.

Supported extensions to start with will be `.db` and `.sqlite` and `.sqlite3`",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/15/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
276704327,MDU6SXNzdWUyNzY3MDQzMjc=,150,_group_count= feature improvements,9599,simonw,closed,0,,,,,3,2017-11-24T22:06:18Z,2018-05-28T16:41:28Z,2018-05-28T16:41:28Z,OWNER,,"- [ ] The ""apply filters"" form should keep you on the _group_count= page
- [ ] Foreign key references should be expand
- [ ] Page title should reflect the view you are on",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/150/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1041158024,PR_kwDOBm6k_c4t7RKr,1500,Bump black from 21.9b0 to 21.10b0,49699333,dependabot[bot],closed,0,,,,,2,2021-11-01T13:11:23Z,2021-11-17T13:14:00Z,2021-11-17T13:13:58Z,CONTRIBUTOR,simonw/datasette/pulls/1500,"Bumps [black](https://github.com/psf/black) from 21.9b0 to 21.10b0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.10b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Document stability policy, that will apply for non-beta releases (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2529"">#2529</a>)</li>
<li>Add new <code>--workers</code> parameter (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2514"">#2514</a>)</li>
<li>Fixed feature detection for positional-only arguments in lambdas (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2532"">#2532</a>)</li>
<li>Bumped typed-ast version minimum to 1.4.3 for 3.10 compatiblity (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2519"">#2519</a>)</li>
<li>Fixed a Python 3.10 compatibility issue where the loop argument was still being passed
even though it has been removed (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2580"">#2580</a>)</li>
<li>Deprecate Python 2 formatting support (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2523"">#2523</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Remove dependency on aiohttp-cors (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2500"">#2500</a>)</li>
<li>Bump required aiohttp version to 3.7.4 (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2509"">#2509</a>)</li>
</ul>
<h3><em>Black-Primer</em></h3>
<ul>
<li>Add primer support for --projects (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2555"">#2555</a>)</li>
<li>Print primer summary after individual failures (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2570"">#2570</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Allow to pass <code>target_version</code> in the vim plugin (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1319"">#1319</a>)</li>
<li>Install build tools in docker file and use multi-stage build to keep the image size
down (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2582"">#2582</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>21.10b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Document stability policy, that will apply for non-beta releases (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2529"">#2529</a>)</li>
<li>Add new <code>--workers</code> parameter (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2514"">#2514</a>)</li>
<li>Fixed feature detection for positional-only arguments in lambdas (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2532"">#2532</a>)</li>
<li>Bumped typed-ast version minimum to 1.4.3 for 3.10 compatiblity (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2519"">#2519</a>)</li>
<li>Fixed a Python 3.10 compatibility issue where the loop argument was still being passed
even though it has been removed (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2580"">#2580</a>)</li>
<li>Deprecate Python 2 formatting support (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2523"">#2523</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Remove dependency on aiohttp-cors (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2500"">#2500</a>)</li>
<li>Bump required aiohttp version to 3.7.4 (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2509"">#2509</a>)</li>
</ul>
<h3><em>Black-Primer</em></h3>
<ul>
<li>Add primer support for --projects (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2555"">#2555</a>)</li>
<li>Print primer summary after individual failures (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2570"">#2570</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Allow to pass <code>target_version</code> in the vim plugin (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1319"">#1319</a>)</li>
<li>Install build tools in docker file and use multi-stage build to keep the image size
down (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2582"">#2582</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.9b0&new-version=21.10b0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1500/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1049946823,I_kwDOBm6k_c4-lOrH,1502,"Full-text search: No support to unary ""-"" operator",516827,gustavorps,open,0,,,,,0,2021-11-10T15:11:19Z,2021-11-10T15:11:19Z,,NONE,,"Reference: https://www.sqlite.org/fts3.html#set_operations_using_the_standard_query_syntax

Test: https://fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort+-freedman&_sort=rowid",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1502/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1050163432,I_kwDOBm6k_c4-mDjo,1503,`?_nocol=` removes that column from the filter interface,9599,simonw,closed,0,,,,,1,2021-11-10T18:22:50Z,2021-11-14T05:08:27Z,2021-11-14T04:53:07Z,OWNER,,"e.g. on https://latest.datasette.io/fixtures/sortable?_nocol=sortable

<img width=""723"" alt=""fixtures__sortable__201_rows"" src=""https://user-images.githubusercontent.com/9599/141170777-733b1ab2-c153-4276-8ba2-d975ed84156e.png"">

This causes weird behaviour when you e.g. facet by a hidden column, since selecting facets and then re-submitting the form will clear the selected filter.

![nocol-bug](https://user-images.githubusercontent.com/9599/141171135-aded71d1-a4cb-4b7f-a4ea-26828fa98906.gif)
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1503/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1051277222,I_kwDOBm6k_c4-qTem,1504,Link to ?_size=max at bottom of table page,9599,simonw,open,0,,,,,0,2021-11-11T19:06:33Z,2021-11-11T19:06:33Z,,OWNER,,"This can have text such as ""Show 1,000 rows per page"", based on the max size limit setting. Would make it easier for people to see more data at once without having to know how to hack the URL, similar to the `...` for facet sizes I added in #1337.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1504/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1052247023,I_kwDOBm6k_c4-uAPv,1505,Datasette should have an option to output CSV with semicolons,9599,simonw,open,0,,,,,1,2021-11-12T18:02:21Z,2021-11-16T11:40:52Z,,OWNER,,,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1505/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1052826038,I_kwDOBm6k_c4-wNm2,1506,Columns beginning with an underscore do not facet correctly,9599,simonw,closed,0,,,,,1,2021-11-14T02:20:32Z,2021-11-14T04:45:21Z,2021-11-14T04:45:21Z,OWNER,,"Datasette treats columns that start with an underscore as querystring parameters it should ignore!

<img width=""914"" alt=""bchydro__item_versions__99_918_rows"" src=""https://user-images.githubusercontent.com/9599/141664864-119990d5-42b2-47f7-91bd-09a74a288af9.png"">

Discovered in https://github.com/simonw/git-history/issues/14#issuecomment-968192464",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1506/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1052851176,I_kwDOBm6k_c4-wTvo,1507,ReadTheDocs build failed for 0.59.2 release,9599,simonw,closed,0,,,,,6,2021-11-14T05:24:34Z,2021-11-14T05:41:55Z,2021-11-14T05:41:55Z,OWNER,,"I had to cancel the 0.59.2 release because ReadTheDocs was failing to build the documentation.

https://readthedocs.org/projects/datasette/builds/15268454/

```
 /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/bin/python -m sphinx -T -b html -d _build/doctrees -D language=en . _build/html
Running Sphinx v1.8.5
loading translations [en]... done
making output directory...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 27 source files that are out of date
updating environment: 27 added, 0 changed, 0 removed
reading sources... [  3%] authentication

Traceback (most recent call last):
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/cmd/build.py"", line 304, in build_main
    app.build(args.force_all, filenames)
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/application.py"", line 341, in build
    self.builder.build_update()
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/builders/__init__.py"", line 347, in build_update
    len(to_build))
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/builders/__init__.py"", line 360, in build
    updated_docnames = set(self.read())
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/builders/__init__.py"", line 468, in read
    self._read_serial(docnames)
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/builders/__init__.py"", line 490, in _read_serial
    self.read_doc(docname)
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/builders/__init__.py"", line 534, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/io.py"", line 318, in read_doc
    pub.publish()
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/docutils/core.py"", line 219, in publish
    self.apply_transforms()
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/docutils/core.py"", line 200, in apply_transforms
    self.document.transformer.apply_transforms()
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/transforms/__init__.py"", line 90, in apply_transforms
    Transformer.apply_transforms(self)
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/docutils/transforms/__init__.py"", line 171, in apply_transforms
    transform.apply(**kwargs)
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/transforms/__init__.py"", line 245, in apply
    apply_source_workaround(n)
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/util/nodes.py"", line 94, in apply_source_workaround
    for classifier in reversed(node.parent.traverse(nodes.classifier)):
TypeError: argument to reversed() must be a sequence

Exception occurred:
  File ""/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/util/nodes.py"", line 94, in apply_source_workaround
    for classifier in reversed(node.parent.traverse(nodes.classifier)):
TypeError: argument to reversed() must be a sequence
The full traceback has been saved in /tmp/sphinx-err-vkl0oE.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks! 
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1507/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1053655062,PR_kwDOBm6k_c4uiE0n,1508,Update docutils requirement from <0.18 to <0.19,49699333,dependabot[bot],closed,0,,,,,1,2021-11-15T13:15:47Z,2021-11-30T02:35:19Z,2021-11-30T02:35:19Z,CONTRIBUTOR,simonw/datasette/pulls/1508,"Updates the requirements on [docutils](http://docutils.sourceforge.net/) to permit the latest version.


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1508/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1054243511,I_kwDOBm6k_c4-1nq3,1509,Datasette 1.0 JSON API (and documentation),9599,simonw,open,0,,,3268330,Datasette 1.0,3,2021-11-15T23:22:45Z,2022-03-15T20:38:56Z,,OWNER,,"The new JSON API in a stable, documented form.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1509/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
276718605,MDU6SXNzdWUyNzY3MTg2MDU=,151,Set up a pattern portfolio,9599,simonw,closed,0,,,,,2,2017-11-25T02:09:49Z,2020-07-02T00:13:24Z,2020-05-03T03:13:16Z,OWNER,,"https://www.slideshare.net/nataliedowne/practical-maintainable-css/75

This will be a single page that demonstrates all of the different CSS styles and classes available to Datasette.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/151/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1054244712,I_kwDOBm6k_c4-1n9o,1510,Datasette 1.0 documented template context (maybe via API docs),9599,simonw,open,0,,,3268330,Datasette 1.0,3,2021-11-15T23:23:58Z,2023-06-28T02:05:21Z,,OWNER,,Documented context plus protective unit tests. Goal is that custom templates built for 1.x will not break without a 2.x release.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1510/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1054246919,I_kwDOBm6k_c4-1ogH,1511,Review plugin hooks for Datasette 1.0,9599,simonw,open,0,,,3268330,Datasette 1.0,1,2021-11-15T23:26:05Z,2021-11-16T01:20:14Z,,OWNER,,I need to perform a detailed review of the plugin interface - especially the plugin hooks like [register_facet_classes()](https://docs.datasette.io/en/stable/plugin_hooks.html#register-facet-classes) which I don't yet have complete confidence in.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1511/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1055402144,PR_kwDOBm6k_c4unfnq,1512,New pattern for async view classes,9599,simonw,closed,0,,,,,7,2021-11-16T21:55:44Z,2021-11-17T01:39:54Z,2021-11-17T01:39:44Z,OWNER,simonw/datasette/pulls/1512,Refs #878 - starting out with the new `AsyncBase` class implementing a pytest-inspired `asyncio` parallel execution mechanism.,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1512/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1,
1055469073,I_kwDOBm6k_c4-6S4R,1513,Research: CTEs and union all to calculate facets AND query at the same time,9599,simonw,closed,0,,,,,12,2021-11-16T22:26:45Z,2021-11-16T23:41:46Z,2021-11-16T23:41:46Z,OWNER,,"Consider this page: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_search=plant&_facet=owner&_facet=country_long&_facet=primary_fuel

Datasette needs to run the main query for the rows on that page, a count query for the total query, then a separate query for each of those three specified facets.

This is a `_search=` query, so it needs to execute the FTS code once for the rows, again for the count, and then three more times for each of the facets.

Could running that query as a CTE and doing the other queries as part of the same large query produce significant speed improvements?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1513/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1056117435,PR_kwDOBm6k_c4up0R0,1514,Bump black from 21.9b0 to 21.11b0,49699333,dependabot[bot],closed,0,,,,,2,2021-11-17T13:13:55Z,2021-11-18T13:11:17Z,2021-11-18T13:11:15Z,CONTRIBUTOR,simonw/datasette/pulls/1514,"Bumps [black](https://github.com/psf/black) from 21.9b0 to 21.11b0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.11b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Warn about Python 2 deprecation in more cases by improving Python 2 only syntax
detection (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2592"">#2592</a>)</li>
<li>Add experimental PyPy support (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2559"">#2559</a>)</li>
<li>Add partial support for the match statement. As it's experimental, it's only enabled
when <code>--target-version py310</code> is explicitly specified (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2586"">#2586</a>)</li>
<li>Add support for parenthesized with (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2586"">#2586</a>)</li>
<li>Declare support for Python 3.10 for running Black (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2562"">#2562</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Fixed vim plugin with Python 3.10 by removing deprecated distutils import (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2610"">#2610</a>)</li>
<li>The vim plugin now parses <code>skip_magic_trailing_comma</code> from pyproject.toml (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2613"">#2613</a>)</li>
</ul>
<h2>21.10b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Document stability policy, that will apply for non-beta releases (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2529"">#2529</a>)</li>
<li>Add new <code>--workers</code> parameter (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2514"">#2514</a>)</li>
<li>Fixed feature detection for positional-only arguments in lambdas (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2532"">#2532</a>)</li>
<li>Bumped typed-ast version minimum to 1.4.3 for 3.10 compatiblity (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2519"">#2519</a>)</li>
<li>Fixed a Python 3.10 compatibility issue where the loop argument was still being passed
even though it has been removed (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2580"">#2580</a>)</li>
<li>Deprecate Python 2 formatting support (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2523"">#2523</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Remove dependency on aiohttp-cors (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2500"">#2500</a>)</li>
<li>Bump required aiohttp version to 3.7.4 (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2509"">#2509</a>)</li>
</ul>
<h3><em>Black-Primer</em></h3>
<ul>
<li>Add primer support for --projects (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2555"">#2555</a>)</li>
<li>Print primer summary after individual failures (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2570"">#2570</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Allow to pass <code>target_version</code> in the vim plugin (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1319"">#1319</a>)</li>
<li>Install build tools in docker file and use multi-stage build to keep the image size
down (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2582"">#2582</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>21.11b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Warn about Python 2 deprecation in more cases by improving Python 2 only syntax
detection (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2592"">#2592</a>)</li>
<li>Add experimental PyPy support (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2559"">#2559</a>)</li>
<li>Add partial support for the match statement. As it's experimental, it's only enabled
when <code>--target-version py310</code> is explicitly specified (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2586"">#2586</a>)</li>
<li>Add support for parenthesized with (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2586"">#2586</a>)</li>
<li>Declare support for Python 3.10 for running Black (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2562"">#2562</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Fixed vim plugin with Python 3.10 by removing deprecated distutils import (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2610"">#2610</a>)</li>
<li>The vim plugin now parses <code>skip_magic_trailing_comma</code> from pyproject.toml (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2613"">#2613</a>)</li>
</ul>
<h2>21.10b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Document stability policy, that will apply for non-beta releases (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2529"">#2529</a>)</li>
<li>Add new <code>--workers</code> parameter (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2514"">#2514</a>)</li>
<li>Fixed feature detection for positional-only arguments in lambdas (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2532"">#2532</a>)</li>
<li>Bumped typed-ast version minimum to 1.4.3 for 3.10 compatibility (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2519"">#2519</a>)</li>
<li>Fixed a Python 3.10 compatibility issue where the loop argument was still being passed
even though it has been removed (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2580"">#2580</a>)</li>
<li>Deprecate Python 2 formatting support (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2523"">#2523</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Remove dependency on aiohttp-cors (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2500"">#2500</a>)</li>
<li>Bump required aiohttp version to 3.7.4 (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2509"">#2509</a>)</li>
</ul>
<h3><em>Black-Primer</em></h3>
<ul>
<li>Add primer support for --projects (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2555"">#2555</a>)</li>
<li>Print primer summary after individual failures (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2570"">#2570</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Allow to pass <code>target_version</code> in the vim plugin (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1319"">#1319</a>)</li>
<li>Install build tools in docker file and use multi-stage build to keep the image size
down (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2582"">#2582</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.9b0&new-version=21.11b0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1514/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1056746091,I_kwDOBm6k_c4-_Kpr,1515,Handle foreign keys that point to a non-existent table,9599,simonw,open,0,,,,,0,2021-11-17T23:40:13Z,2021-11-18T01:31:56Z,,OWNER,,"Spotted in https://github.com/simonw/datasette-graphql/issues/79

Demo: https://datasette-graphql-demo.datasette.io/fixtures/bad_foreign_key

The foreign key links to a 404 page.

![B87009C7-CFCA-4DF9-8FBA-FA3E6CA28EC2](https://user-images.githubusercontent.com/9599/142334788-4d1a4acd-bc87-4426-b333-d46b221afcec.jpeg)
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1515/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1057340779,PR_kwDOBm6k_c4utsKs,1516,Bump black from 21.9b0 to 21.11b1,49699333,dependabot[bot],closed,0,,,,,1,2021-11-18T13:11:12Z,2021-11-30T02:35:29Z,2021-11-30T02:35:29Z,CONTRIBUTOR,simonw/datasette/pulls/1516,"Bumps [black](https://github.com/psf/black) from 21.9b0 to 21.11b1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.11b1</h2>
<h3><em>Black</em></h3>
<ul>
<li>Bumped regex version minimum to 2021.4.4 to fix Pattern class usage (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2621"">#2621</a>)</li>
</ul>
<h2>21.11b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Warn about Python 2 deprecation in more cases by improving Python 2 only syntax
detection (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2592"">#2592</a>)</li>
<li>Add experimental PyPy support (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2559"">#2559</a>)</li>
<li>Add partial support for the match statement. As it's experimental, it's only enabled
when <code>--target-version py310</code> is explicitly specified (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2586"">#2586</a>)</li>
<li>Add support for parenthesized with (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2586"">#2586</a>)</li>
<li>Declare support for Python 3.10 for running Black (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2562"">#2562</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Fixed vim plugin with Python 3.10 by removing deprecated distutils import (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2610"">#2610</a>)</li>
<li>The vim plugin now parses <code>skip_magic_trailing_comma</code> from pyproject.toml (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2613"">#2613</a>)</li>
</ul>
<h2>21.10b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Document stability policy, that will apply for non-beta releases (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2529"">#2529</a>)</li>
<li>Add new <code>--workers</code> parameter (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2514"">#2514</a>)</li>
<li>Fixed feature detection for positional-only arguments in lambdas (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2532"">#2532</a>)</li>
<li>Bumped typed-ast version minimum to 1.4.3 for 3.10 compatiblity (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2519"">#2519</a>)</li>
<li>Fixed a Python 3.10 compatibility issue where the loop argument was still being passed
even though it has been removed (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2580"">#2580</a>)</li>
<li>Deprecate Python 2 formatting support (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2523"">#2523</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Remove dependency on aiohttp-cors (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2500"">#2500</a>)</li>
<li>Bump required aiohttp version to 3.7.4 (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2509"">#2509</a>)</li>
</ul>
<h3><em>Black-Primer</em></h3>
<ul>
<li>Add primer support for --projects (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2555"">#2555</a>)</li>
<li>Print primer summary after individual failures (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2570"">#2570</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Allow to pass <code>target_version</code> in the vim plugin (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1319"">#1319</a>)</li>
<li>Install build tools in docker file and use multi-stage build to keep the image size
down (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2582"">#2582</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>21.11b1</h2>
<h3><em>Black</em></h3>
<ul>
<li>Bumped regex version minimum to 2021.4.4 to fix Pattern class usage (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2621"">#2621</a>)</li>
</ul>
<h2>21.11b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Warn about Python 2 deprecation in more cases by improving Python 2 only syntax
detection (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2592"">#2592</a>)</li>
<li>Add experimental PyPy support (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2559"">#2559</a>)</li>
<li>Add partial support for the match statement. As it's experimental, it's only enabled
when <code>--target-version py310</code> is explicitly specified (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2586"">#2586</a>)</li>
<li>Add support for parenthesized with (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2586"">#2586</a>)</li>
<li>Declare support for Python 3.10 for running Black (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2562"">#2562</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Fixed vim plugin with Python 3.10 by removing deprecated distutils import (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2610"">#2610</a>)</li>
<li>The vim plugin now parses <code>skip_magic_trailing_comma</code> from pyproject.toml (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2613"">#2613</a>)</li>
</ul>
<h2>21.10b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Document stability policy, that will apply for non-beta releases (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2529"">#2529</a>)</li>
<li>Add new <code>--workers</code> parameter (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2514"">#2514</a>)</li>
<li>Fixed feature detection for positional-only arguments in lambdas (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2532"">#2532</a>)</li>
<li>Bumped typed-ast version minimum to 1.4.3 for 3.10 compatibility (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2519"">#2519</a>)</li>
<li>Fixed a Python 3.10 compatibility issue where the loop argument was still being passed
even though it has been removed (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2580"">#2580</a>)</li>
<li>Deprecate Python 2 formatting support (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2523"">#2523</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Remove dependency on aiohttp-cors (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2500"">#2500</a>)</li>
<li>Bump required aiohttp version to 3.7.4 (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2509"">#2509</a>)</li>
</ul>
<h3><em>Black-Primer</em></h3>
<ul>
<li>Add primer support for --projects (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2555"">#2555</a>)</li>
<li>Print primer summary after individual failures (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2570"">#2570</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Allow to pass <code>target_version</code> in the vim plugin (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1319"">#1319</a>)</li>
<li>Install build tools in docker file and use multi-stage build to keep the image size
down (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2582"">#2582</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.9b0&new-version=21.11b1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1516/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1057996111,I_kwDOBm6k_c4_D71P,1517,Let `register_routes()` over-ride default routes within Datasette,9599,simonw,closed,0,,,3268330,Datasette 1.0,2,2021-11-19T00:22:15Z,2021-11-19T03:20:00Z,2021-11-19T03:07:27Z,OWNER,,"See https://github.com/simonw/datasette/issues/878#issuecomment-973554024_ - right now `register_routes()` can't replace default Datasette routes.

It would be neat if plugins could do this - especially if there was a neat documented way for them to then re-dispatch to the original route code after making some kind of modification.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1517/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1058072543,I_kwDOBm6k_c4_EOff,1518,Complete refactor of TableView and table.html template,9599,simonw,open,0,,,3268330,Datasette 1.0,45,2021-11-19T02:55:16Z,2022-03-15T18:35:49Z,,OWNER,,"Split from #878. The current `TableView` class is by far the most complex part of Datasette, and the most difficult to work on: https://github.com/simonw/datasette/blob/0.59.2/datasette/views/table.py

In #878 I started exploring a new pattern for building views. In doing so it became clear that `TableView` is the first beast that I need to slay - if I can refactor that into something neat the pattern for building other views will emerge as a natural consequence.

I've been trying to build this as a `register_routes()` plugin, as originally suggested in #870 - though unfortunately it looks like those plugins can't replace existing Datasette default views at the moment, see #1517. [UPDATE: I was wrong about this, plugins can over-ride default views just fine]

I also know that I want to have a fully documented template context for `table.html` as a major step on the way to Datasette 1.0, see #1510.

All of this adds up to the `TableView` factor being a major project that will unblock a whole flurry of other things - so I'm going to work on that in this separate issue.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1518/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1058790545,I_kwDOBm6k_c4_G9yR,1519,base_url is omitted in JSON and CSV views,157158,phubbard,closed,0,,,,,22,2021-11-19T18:10:45Z,2021-12-01T17:50:09Z,2021-11-20T19:11:21Z,NONE,,"I have a datasette deployment, using Apache2 to reverse proxy:

       ProxyPass /ged http://thor.phfactor.net:8001
       ProxyPreserveHost On

In settings.json I have
```json
{
    ""base_url"": ""/ged/"",
    ""trace_debug"": 1,
    ""template_debug"": 1
}
```
and datasette works correctly. However, if you view a query and then click on the 'This data as json, CSV' both links omit the base_url prefix and are therefore 404.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1519/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
276765070,MDU6SXNzdWUyNzY3NjUwNzA=,152,Incorrect display of rows page for tables with a primary key,9599,simonw,closed,0,,,2949431,Custom templates edition,0,2017-11-25T17:29:54Z,2017-12-07T05:23:20Z,2017-12-07T05:23:19Z,OWNER,,"This is a regression. Here's the old version:

<img width=""420"" alt=""parlgov__party_family"" src=""https://user-images.githubusercontent.com/9599/33232903-031ef1a0-d1c3-11e7-94b4-10f9f0076b4f.png"">

And here's the new, broken one:

https://parlgov-xtxlddmtiz.now.sh/parlgov-25f9855/party_family/1

<img width=""420"" alt=""parlgov__party_family"" src=""https://user-images.githubusercontent.com/9599/33232908-12f885be-d1c3-11e7-97f1-cb420f2a5de4.png"">

The JSON output is the same for both - it's only the HTML representation that exhibits the bug.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/152/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1058803238,I_kwDOBm6k_c4_HA4m,1520,Pattern for avoiding accidental URL over-rides,9599,simonw,open,0,,,,,1,2021-11-19T18:28:05Z,2021-11-19T18:29:26Z,,OWNER,,"Following #1517 I'm experimenting with a plugin that does this:
```python
@hookimpl
def register_routes():
    return [
        (r""/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?)$"", Table().view),
    ]
```
This is supposed to replace the default table page with new code... but there's a problem: `/-/versions` on that instance now returns 404 `Database '-' does not exist`!

Need to figure out a pattern to avoid that happening. Plugins get to add their routes before Datasette's default routes, which is why this is happening here.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1520/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1058815557,I_kwDOBm6k_c4_HD5F,1521,Docker configuration for exercising Datasette behind Apache mod_proxy,9599,simonw,closed,0,,,,,10,2021-11-19T18:46:18Z,2021-11-19T20:32:29Z,2021-11-19T20:32:29Z,OWNER,,"> Having a live demo running on Cloud Run that proxies through Apache and uses `base_url` would be incredibly useful for replicating and debugging this kind of thing. I wonder how hard it is to run Apache and `mod_proxy` in the same Docker container as Datasette?

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1519#issuecomment-974310208_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1521/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1058896236,I_kwDOBm6k_c4_HXls,1522,Deploy a live instance of demos/apache-proxy,9599,simonw,closed,0,,,,,34,2021-11-19T20:32:55Z,2021-11-23T03:00:34Z,2021-11-20T18:51:56Z,OWNER,,"> I'll get this working on my laptop first, but then I want to get it up and running on Cloud Run - maybe with a GitHub Actions  workflow in this repo that re-deploys it on manual execution.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1521#issuecomment-974322178_

I started by following https://ahmet.im/blog/cloud-run-multiple-processes-easy-way/ - see example in https://github.com/ahmetb/multi-process-container-lazy-solution",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1522/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1059209412,I_kwDOBm6k_c4_IkDE,1523,Come up with a more elegant solution for base_url than ds.urls.path(),9599,simonw,open,0,,,,,0,2021-11-20T19:05:22Z,2021-11-20T19:05:22Z,,OWNER,,"While fixing #1519 I added a lot of ugly code that looks like this: https://github.com/simonw/datasette/blob/08947fa76433d18988aa1ee1d929bd8320c75fe2/datasette/facets.py#L228-L230

See these two commits in particular: fe687fd0207c4c56c4778d3e92e3505fc4b18172  and 08947fa76433d18988aa1ee1d929bd8320c75fe2

It would be great to come up with a less verbose and error-prone way of handling this problem.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1523/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1059219106,I_kwDOBm6k_c4_Imai,1524,"Improve Apache proxy documentation, link to demo",9599,simonw,closed,0,,,,,4,2021-11-20T20:03:14Z,2021-11-20T23:34:03Z,2021-11-20T23:34:03Z,OWNER,,"> The latest demo is now live at https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures/sortable?_facet=pk2

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1519#issuecomment-974697824_

I'm going to put out 0.59.3 bugfix release with this, but I'd like to first improve the documentation on https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration to highlight the new demo.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1524/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1059509927,I_kwDOBm6k_c4_Jtan,1525,"""Links from other tables"" broken for columns starting with underscore",9599,simonw,closed,0,,,,,3,2021-11-21T22:55:08Z,2021-11-30T06:39:01Z,2021-11-30T06:34:35Z,OWNER,,"Same bug as #1506, this time it's this link or the row page:

<img width=""341"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/142782167-fe9b6ffa-cd81-4a30-9f3b-401118c498e9.png"">
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1525/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1059549523,I_kwDOBm6k_c4_J3FT,1526,"Add to vercel.json, rather than overwriting it.",192568,mroswell,closed,0,,,,,2,2021-11-22T00:47:12Z,2021-11-22T04:49:45Z,2021-11-22T04:13:47Z,CONTRIBUTOR,,"I'd like to be able to add to vercel.json. But Datasette overwrites whatever I put in that file. I originally reported this here:
https://github.com/simonw/datasette-publish-vercel/issues/51

In that case, I wanted to do a rewrite... and now I need to do 301 redirects (because we had to rename our site).

Can this be addressed?
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1526/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1059555791,I_kwDOBm6k_c4_J4nP,1527,Columns starting with an underscore behave poorly in filters,9599,simonw,closed,0,,,7571612,Datasette 0.60,7,2021-11-22T01:01:36Z,2022-01-14T00:57:08Z,2022-01-14T00:57:08Z,OWNER,,"Similar bug to #1525 (and #1506 before it). Start on https://latest.datasette.io/fixtures/facetable?_facet=_neighborhood - then select a neighborhood - then try to remove that filter using the little ""x"" and submitting the form again.

![filter-bug](https://user-images.githubusercontent.com/9599/142786754-31d265a2-944d-4ea2-af6f-305d445a2ccb.gif)
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1527/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1060631257,I_kwDOBm6k_c4_N_LZ,1528,"Add new `""sql_file""` key to Canned Queries in metadata?",15178711,asg017,open,0,,,,,3,2021-11-22T21:58:01Z,2022-06-10T03:23:08Z,,CONTRIBUTOR,,"Currently for canned queries, you have to inline SQL in your `metadata.yaml` like so:

```yaml
databases:
  fixtures:
    queries:
      neighborhood_search:
        sql: |-
          select neighborhood, facet_cities.name, state
          from facetable
            join facet_cities on facetable.city_id = facet_cities.id
          where neighborhood like '%' || :text || '%'
          order by neighborhood
        title: Search neighborhoods
```

This works fine, but for a few reasons, I usually have my canned queries already written in separate `.sql` files. I'd like to instead re-use those instead of re-writing it. 

So, I'd like to see a new `""sql_file""` key that works like so:

`metadata.yaml`:

```yaml
databases:
  fixtures:
    queries:
      neighborhood_search:
        sql_file: neighborhood_search.sql
        title: Search neighborhoods
```
`neighborhood_search.sql`:
```sql
select neighborhood, facet_cities.name, state
from facetable
join facet_cities on facetable.city_id = facet_cities.id
where neighborhood like '%' || :text || '%'
order by neighborhood
```

Both of these would work in the exact same way, where Datasette would instead open + include `neighborhood_search.sql` on startup. 


A few reasons why I'd like to keep my canned queries SQL separate from metadata.yaml:

- Keeping SQL in standalone SQL files means syntax highlighting and other text editor integrations in my code
- Multiline strings in yaml, while functional, are a tad cumbersome and are hard to edit
- Works well with other tools (can pipe `.sql` files into the `sqlite3` CLI, or use with other SQLite clients easier)
- Typically my canned queries are quite long compared to everything else in my metadata.yaml, so I'd love to separate it where possible

Let me know if this is a feature you'd like to see, I can try to send up a PR if this sounds right!",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1528/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1062414013,PR_kwDOBm6k_c4u9wQq,1529,"Update janus requirement from <0.7,>=0.6.2 to >=0.6.2,<0.8",49699333,dependabot[bot],closed,0,,,,,1,2021-11-24T13:12:42Z,2021-11-30T02:37:13Z,2021-11-30T02:37:13Z,CONTRIBUTOR,simonw/datasette/pulls/1529,"Updates the requirements on [janus](https://github.com/aio-libs/janus) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/aio-libs/janus/blob/master/CHANGES.rst"">janus's changelog</a>.</em></p>
<blockquote>
<h2>0.7.0 (2021-11-24)</h2>
<ul>
<li>Add SyncQueue and AsyncQueue Protocols to provide type hints for sync and async queues <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/374"">#374</a></li>
</ul>
<h2>0.6.2 (2021-10-24)</h2>
<ul>
<li>Fix Python 3.10 compatibility <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/358"">#358</a></li>
</ul>
<h2>0.6.1 (2020-10-26)</h2>
<ul>
<li>
<p>Raise RuntimeError on queue.join() after queue closing. <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/295"">#295</a></p>
</li>
<li>
<p>Replace <code>timeout</code> type from <code>Optional[int]</code> to <code>Optional[float]</code> <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/267"">#267</a></p>
</li>
</ul>
<h2>0.6.0 (2020-10-10)</h2>
<ul>
<li>
<p>Drop Python 3.5, the minimal supported version is Python 3.6</p>
</li>
<li>
<p>Support Python 3.9</p>
</li>
<li>
<p>Refomat with <code>black</code></p>
</li>
</ul>
<h2>0.5.0 (2020-04-23)</h2>
<ul>
<li>Remove explicit loop arguments and forbid creating queues outside event loops <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/246"">#246</a></li>
</ul>
<h2>0.4.0 (2018-07-28)</h2>
<ul>
<li>
<p>Add <code>py.typed</code> macro <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/89"">#89</a></p>
</li>
<li>
<p>Drop python 3.4 support and fix minimal version python3.5.3 <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/88"">#88</a></p>
</li>
<li>
<p>Add property with that indicates if queue is closed <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/86"">#86</a></p>
</li>
</ul>
<h2>0.3.2 (2018-07-06)</h2>
<ul>
<li>Fixed python 3.7 support <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/97"">#97</a></li>
</ul>
<h2>0.3.1 (2018-01-30)</h2>
<ul>
<li>Fixed bug with join() in case tasks are added by sync_q.put() <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/75"">#75</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/aio-libs/janus/commit/7b72d8577517422dd0719da1c6a0fb33d8a10e23""><code>7b72d85</code></a> Bump to 0.7</li>
<li><a href=""https://github.com/aio-libs/janus/commit/5389dec41aedb1d85243f146d655d123ed348ef0""><code>5389dec</code></a> add SyncQueue and AsyncQueue Protocols (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/374"">#374</a>)</li>
<li><a href=""https://github.com/aio-libs/janus/commit/d54ae4230fe1fc78657262771ca8b96172dc91c4""><code>d54ae42</code></a> Bump docutils from 0.18 to 0.18.1 (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/381"">#381</a>)</li>
<li><a href=""https://github.com/aio-libs/janus/commit/76ccc92890e4fa28966a1b3fe873fc360a2a70e9""><code>76ccc92</code></a> Bump black from 21.11b0 to 21.11b1 (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/380"">#380</a>)</li>
<li><a href=""https://github.com/aio-libs/janus/commit/967c28426d509671e8f18327848046f7413805f6""><code>967c284</code></a> Bump black from 21.10b0 to 21.11b0 (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/379"">#379</a>)</li>
<li><a href=""https://github.com/aio-libs/janus/commit/4d625a2138ef85ca964b6a776472d19a92334433""><code>4d625a2</code></a> Bump bandit from 1.7.0 to 1.7.1 (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/378"">#378</a>)</li>
<li><a href=""https://github.com/aio-libs/janus/commit/e21eec2c04acfba189f7177d2981e24fc3674fd1""><code>e21eec2</code></a> Bump coverage from 6.1.1 to 6.1.2 (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/377"">#377</a>)</li>
<li><a href=""https://github.com/aio-libs/janus/commit/7ff20afb10149c2d020c388e81a22ed715149914""><code>7ff20af</code></a> Bump isort from 5.10.0 to 5.10.1 (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/376"">#376</a>)</li>
<li><a href=""https://github.com/aio-libs/janus/commit/67e3782dc540ee13858027c3521691e779fcdc16""><code>67e3782</code></a> Switch from setup.py to declarative setup.cfg (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/375"">#375</a>)</li>
<li><a href=""https://github.com/aio-libs/janus/commit/8a0c4983da0b412644d1e84e6e1e2da6c671221e""><code>8a0c498</code></a> Bump isort from 5.9.3 to 5.10.0 (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/373"">#373</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/aio-libs/janus/compare/v0.6.2...v0.7.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1529/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
276842536,MDU6SXNzdWUyNzY4NDI1MzY=,153,Ability to customize presentation of specific columns in HTML view,20264,ftrain,closed,0,,,2949431,Custom templates edition,14,2017-11-26T17:46:11Z,2017-12-10T02:08:45Z,2017-12-07T06:17:33Z,NONE,,"This ties into https://github.com/simonw/datasette/issues/3 in some ways. It would be great to have some adaptability in the HTML views and to specific some columns as displaying in certain ways.

- [x] 1. **Auto-parsing URIs into in-browser links.** Why? Lots of public data around cultural commons stuff links to a specific URL. This would be a great utility to turn on at the command line, just parse everything for URLs. Maybe they need to be underlined or represented in a different way than internal URLs.
- [x] 2. **Ability to identify a column as plain/preformatted text.** Why? Was trying to import the Enron emails, the body collapses. Hard to read. These fields also tend to screw up the ability to scan a table view. If you knew it was text the system could set an `overflow` property on the relevant CSS, so you could still scan.
- [x] 3. **Ability to identify a column as HTML.** Why? I want to spider some stuff and drop sections into SQLite, and just keep them as HTML.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/153/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1065429936,I_kwDOBm6k_c4_gSuw,1532,Use datasette-table Web Component to guide the design of the JSON API for 1.0,9599,simonw,open,0,,,3268330,Datasette 1.0,4,2021-11-28T20:37:18Z,2022-03-16T20:13:34Z,,OWNER,,"I realized that one of the reasons I'm having trouble committing to nailing down the JSON API for 1.0 is that I don't use it much myself - I use the `?_shape=array` one quite often, but I don't have any projects that are using the default, more fully-featured API.

As an experiment I built a Web Component for embedding Datasette tables on pages - https://github.com/simonw/datasette-table - and I think it's actually going to be a really useful tool for helping me dog food the v1.0 API design.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1532/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1065431383,I_kwDOBm6k_c4_gTFX,1533,"Add `Link: rel=""alternate""` header pointing to JSON for a table/query",9599,simonw,closed,0,,,3268330,Datasette 1.0,4,2021-11-28T20:43:25Z,2022-02-02T07:56:51Z,2022-02-02T07:49:33Z,OWNER,,"Originally explored in https://github.com/simonw/datasette-notebook/issues/2#issuecomment-980789406 - I wanted an efficient way to scan a list of URLs and figure out which if any of those corresponded to Datasette tables, canned queries or SQL output that could be represented as a table on a page.

It looks like a neat way to do that is with ` Link:` header like this:

`Link: http://127.0.0.1:8058/fixtures/compound_three_primary_keys.json; rel=""alternate""; type=""application/datasette+json""`

I can put a `<link href=...` in the page header too.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1533/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1065432388,I_kwDOBm6k_c4_gTVE,1534,Maybe return JSON from HTML pages if `Accept: application/json` is sent,9599,simonw,closed,0,,,,,4,2021-11-28T20:48:09Z,2022-04-27T21:59:34Z,2022-02-02T23:39:33Z,OWNER,,"Relates to #1533 - and to the work I've been doing on the https://github.com/simonw/datasette-table Web Component.

It would be useful to support users pasting in a URL to a Datasette table or query without first having to add the `.json` extension themselves - since then other systems could hit that URL with `Accept: application/json` to get back the JSON representation without first needing to read the `Link: ` header from #1533 to figure out what the URL to that JSON is.

(There is weird logic deep in Datasette that says that you add `.json` to the path UNLESS the table name itself ends with `.json`, in which case you add `?_format=json` - this is super-confusing).

[Update: I removed that confusing feature here: [https://simonwillison.net/2022/Mar/19/weeknotes/](https://simonwillison.net/2022/Mar/19/weeknotes/)]",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1534/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1066023866,PR_kwDOBm6k_c4vIJqi,1537,"Update aiofiles requirement from <0.8,>=0.4 to >=0.4,<0.9",49699333,dependabot[bot],closed,0,,,,,1,2021-11-29T13:13:52Z,2021-11-30T02:29:55Z,2021-11-30T02:29:54Z,CONTRIBUTOR,simonw/datasette/pulls/1537,"Updates the requirements on [aiofiles](https://github.com/Tinche/aiofiles) to permit the latest version.
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/Tinche/aiofiles/commit/d010ff4d789598213334a32ec3d3f55caaab766c""><code>d010ff4</code></a> 0.8.0</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/3e3830daad8f41b1d3548a59c9913e4daf7be327""><code>3e3830d</code></a> Tweak README</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/713984eb46a62f8d504f31ee9b1305a55581675d""><code>713984e</code></a> Update README</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/95709758e635e9155ee5d96c973676cd0815221a""><code>9570975</code></a> Use os.sep</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/2b2500d555769460bc1f226f078d86eeaae06daf""><code>2b2500d</code></a> Add prefix, suffix, dir args to TemporaryDirectory</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/8895eb48b6004c8677da5841e2504a73eb8fad5e""><code>8895eb4</code></a> Fix ospath.exists test</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/83083555512320cf5c4cb10dd4dbda57c4cfbea1""><code>8308355</code></a> Fix typo: aiofiles.ospath.exists</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/b8cf8dd19369c2529181b3a4b8f1c485a2c05dbd""><code>b8cf8dd</code></a> Update Readme with os.makedirs and os.removedirs</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/aea353ebdd2771f2e4ae934077eceb12daaa3481""><code>aea353e</code></a> Merge branch 'patch-2'</li>
<li><a href=""https://github.com/Tinche/aiofiles/commit/761f06ede5925d0cdd22926b0d83564e9c1e1ab5""><code>761f06e</code></a> Fix README</li>
<li>Additional commits viewable in <a href=""https://github.com/Tinche/aiofiles/compare/v0.4.0...v0.8.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1537/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1066288689,I_kwDOBm6k_c4_jkYx,1538,Research pattern for re-registering existing Click tools with register_commands,9599,simonw,closed,0,,,,,3,2021-11-29T17:09:47Z,2021-11-29T17:32:44Z,2021-11-29T17:27:16Z,OWNER,,"Building a Datasette plugin that imports an existing Click CLI tool and re-registers it is proving hard - Click doesn't really want you to do that. I tried this:
```python
from datasette import hookimpl
from git_history.cli import file as git_history_file


@hookimpl
def register_commands(cli):
    cli.command(name=""git-history"")(git_history_file.callback)
```
But when I run this:
```
 % datasette git-history --help     
Usage: datasette git-history [OPTIONS]

  Analyze the history of a specific file and write it to SQLite

Options:
  --help  Show this message and exit.
```
The options are all missing - which means that the command doesn't actually work. Will need to research this pattern separately.

_Originally posted by @simonw in https://github.com/simonw/git-history/issues/21#issuecomment-981835305_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1538/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1067775061,I_kwDOBm6k_c4_pPRV,1539,Research PRAGMA query_only,9599,simonw,open,0,,,,,0,2021-11-30T23:30:24Z,2021-11-30T23:30:24Z,,OWNER,,"https://www.sqlite.org/pragma.html#pragma_query_only

> The query_only pragma prevents data changes on database files when enabled. When this pragma is enabled, any attempt to CREATE, DELETE, DROP, INSERT, or UPDATE will result in an [SQLITE_READONLY](https://www.sqlite.org/rescode.html#readonly) error. However, the database is not truly read-only. You can still run a [checkpoint](https://www.sqlite.org/wal.html#ckpt) or a [COMMIT](https://www.sqlite.org/lang_transaction.html) and the return value of the [sqlite3_db_readonly()](https://www.sqlite.org/c3ref/db_readonly.html) routine is not affected.

Would it be worth adding this as an extra protection against accidental writes to a DB file over a read-only connection?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1539/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
276873891,MDU6SXNzdWUyNzY4NzM4OTE=,154,Datasette CSS should include content hash in the URL,9599,simonw,closed,0,,,2949431,Custom templates edition,3,2017-11-27T00:57:36Z,2017-12-09T03:10:23Z,2017-12-09T03:10:22Z,OWNER,,"When I deployed the latest version of datasette to https://fivethirtyeight.datasettes.com/ I noticed I was getting served stale CSS since it had been cached. Including the sha of he contents in its URL should fix that.

I can calculate this on server start.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/154/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1068791148,I_kwDOBm6k_c4_tHVs,1540,Idea: hover to reveal details of linked row,9599,simonw,open,0,,,,,6,2021-12-01T19:28:07Z,2021-12-09T23:38:39Z,,OWNER,,"<img width=""698"" alt=""fara__item_version__7_rows_where_where__item___5236"" src=""https://user-images.githubusercontent.com/9599/144300364-ee621811-8f5a-4c7e-9950-199119092cf3.png"">

Hovering over that could work a little bit like GitHub issue links:

![hover](https://user-images.githubusercontent.com/9599/144300537-9cd9e9af-ac16-42db-842f-37661bc94063.gif)
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1540/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1069881276,I_kwDOBm6k_c4_xRe8,1541,Different default layout for row page,9599,simonw,open,0,,,,,1,2021-12-02T18:56:36Z,2021-12-02T18:56:54Z,,OWNER,,"The row page displays as a table even though it only has one table row.

maybe default to the same display as the narrow page version, even for wide pages?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1541/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1072106103,I_kwDOBm6k_c4_5wp3,1542,feature request: order and dependency of plugins (that use js),33631,fs111,open,0,,,,,1,2021-12-06T12:40:45Z,2021-12-15T17:47:08Z,,NONE,,"I have been playing with datasette for the last couple of weeks and it is great! I am a big fan of `datasette-cluster-map` and wanted to enhance it a bit with a what I would call a sub-plugin. I basically want to add more controls to the map that cluster map provides. I have been looking into its code and how the plugin management works, but it seems what I am trying to do is not doable without hacks in js.

Basically what would like to have is a way to say load my plugin after the plugins I depend on have been loaded and rendered. There seems to be no prior art where plugins have these dependencies on the js level so I was wondering if that could be added or if it exists how to do it.

Basically what I want to do is:

my-awesome-plugin has a dependency on datastte-cluster-map. Whenever datasette cluster map has finished rendering on page load, call my plugin, but no earlier. To make that work datasette probably needs some total order in which way plugins are loaded intialized.

Since I am new to datastte, I may be missing something obvious, so please let me know if the above makes no sense.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1542/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1072135269,PR_kwDOBm6k_c4vb__Y,1543,Bump black from 21.11b1 to 21.12b0,49699333,dependabot[bot],closed,0,,,,,1,2021-12-06T13:11:16Z,2021-12-13T23:22:29Z,2021-12-13T23:22:29Z,CONTRIBUTOR,simonw/datasette/pulls/1543,"Bumps [black](https://github.com/psf/black) from 21.11b1 to 21.12b0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>21.12b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Fix determination of f-string expression spans (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2654"">#2654</a>)</li>
<li>Fix bad formatting of error messages about EOF in multi-line statements (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2343"">#2343</a>)</li>
<li>Functions and classes in blocks now have more consistent surrounding spacing (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2472"">#2472</a>)</li>
</ul>
<h4>Jupyter Notebook support</h4>
<ul>
<li>Cell magics are now only processed if they are known Python cell magics. Earlier, all cell magics were tokenized, leading to possible indentation errors e.g. with <code>%%writefile</code>. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2630"">#2630</a>)</li>
<li>Fix assignment to environment variables in Jupyter Notebooks (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2642"">#2642</a>)</li>
</ul>
<h4>Python 3.10 support</h4>
<ul>
<li>Point users to using <code>--target-version py310</code> if we detect 3.10-only syntax (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2668"">#2668</a>)</li>
<li>Fix <code>match</code> statements with open sequence subjects, like <code>match a, b:</code> or <code>match a, *b:</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2639"">#2639</a>) (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2659"">#2659</a>)</li>
<li>Fix <code>match</code>/<code>case</code> statements that contain <code>match</code>/<code>case</code> soft keywords multiple times, like <code>match re.match()</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2661"">#2661</a>)</li>
<li>Fix <code>case</code> statements with an inline body (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2665"">#2665</a>)</li>
<li>Fix styling of starred expressions inside <code>match</code> subject (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2667"">#2667</a>)</li>
<li>Fix parser error location on invalid syntax in a <code>match</code> statement (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2649"">#2649</a>)</li>
<li>Fix Python 3.10 support on platforms without ProcessPoolExecutor (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2631"">#2631</a>)</li>
<li>Improve parsing performance on code that uses <code>match</code> under <code>--target-version py310</code> up to ~50% (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2670"">#2670</a>)</li>
</ul>
<h3>Packaging</h3>
<ul>
<li>Remove dependency on <code>regex</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2644"">#2644</a>) (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2663"">#2663</a>)</li>
</ul>
<hr />
<h4>Thank you!</h4>
<ul>
<li><a href=""https://github.com/isidentical""><code>@​isidentical</code></a> for the polishing up 3.10 syntax support (which they contributed in the first place!)</li>
<li><a href=""https://github.com/MarcoGorelli""><code>@​MarcoGorelli</code></a> for their ever-continuing work on Black's jupyter support</li>
<li><a href=""https://github.com/jalaziz""><code>@​jalaziz</code></a> for cleaning up our Pyinstaller CD workflow</li>
<li><a href=""https://github.com/hauntsaninja""><code>@​hauntsaninja</code></a> for helping us drop the <code>regex</code> dependency</li>
</ul>
<p>And also congrats to first contributors!</p>
<ul>
<li><a href=""https://github.com/MatthewScholefield""><code>@​MatthewScholefield</code></a> made their first contribution in <a href=""https://github-redirect.dependabot.com/psf/black/pull/2631"">psf/black#2631</a></li>
<li><a href=""https://github.com/AshIsbitt""><code>@​AshIsbitt</code></a> made their first contribution in <a href=""https://github-redirect.dependabot.com/psf/black/pull/2632"">psf/black#2632</a></li>
<li><a href=""https://github.com/kalbasit""><code>@​kalbasit</code></a> made their first contribution in <a href=""https://github-redirect.dependabot.com/psf/black/pull/2638"">psf/black#2638</a></li>
<li><a href=""https://github.com/danieleades""><code>@​danieleades</code></a> made their first contribution in <a href=""https://github-redirect.dependabot.com/psf/black/pull/2653"">psf/black#2653</a></li>
<li><a href=""https://github.com/danielsparing""><code>@​danielsparing</code></a> made their first contribution in <a href=""https://github-redirect.dependabot.com/psf/black/pull/2630"">psf/black#2630</a></li>
<li><a href=""https://github.com/tanvimoharir""><code>@​tanvimoharir</code></a> made their first contribution in <a href=""https://github-redirect.dependabot.com/psf/black/pull/2343"">psf/black#2343</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>21.12b0</h2>
<h3><em>Black</em></h3>
<ul>
<li>Fix determination of f-string expression spans (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2654"">#2654</a>)</li>
<li>Fix bad formatting of error messages about EOF in multi-line statements (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2343"">#2343</a>)</li>
<li>Functions and classes in blocks now have more consistent surrounding spacing (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2472"">#2472</a>)</li>
</ul>
<h4>Jupyter Notebook support</h4>
<ul>
<li>Cell magics are now only processed if they are known Python cell magics. Earlier, all
cell magics were tokenized, leading to possible indentation errors e.g. with
<code>%%writefile</code>. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2630"">#2630</a>)</li>
<li>Fix assignment to environment variables in Jupyter Notebooks (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2642"">#2642</a>)</li>
</ul>
<h4>Python 3.10 support</h4>
<ul>
<li>Point users to using <code>--target-version py310</code> if we detect 3.10-only syntax (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2668"">#2668</a>)</li>
<li>Fix <code>match</code> statements with open sequence subjects, like <code>match a, b:</code> or
<code>match a, *b:</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2639"">#2639</a>) (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2659"">#2659</a>)</li>
<li>Fix <code>match</code>/<code>case</code> statements that contain <code>match</code>/<code>case</code> soft keywords multiple
times, like <code>match re.match()</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2661"">#2661</a>)</li>
<li>Fix <code>case</code> statements with an inline body (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2665"">#2665</a>)</li>
<li>Fix styling of starred expressions inside <code>match</code> subject (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2667"">#2667</a>)</li>
<li>Fix parser error location on invalid syntax in a <code>match</code> statement (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2649"">#2649</a>)</li>
<li>Fix Python 3.10 support on platforms without ProcessPoolExecutor (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2631"">#2631</a>)</li>
<li>Improve parsing performance on code that uses <code>match</code> under <code>--target-version py310</code>
up to ~50% (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2670"">#2670</a>)</li>
</ul>
<h3>Packaging</h3>
<ul>
<li>Remove dependency on <code>regex</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2644"">#2644</a>) (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2663"">#2663</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.11b1&new-version=21.12b0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1543/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1073712378,I_kwDOBm6k_c4__4z6,1544,Code that detects the label column for a table is case-sensitive,9599,simonw,closed,0,,,,,2,2021-12-07T20:01:25Z,2021-12-07T20:03:43Z,2021-12-07T20:03:43Z,OWNER,,I just noticed that a column called `Name` is not being picked up as the label column for a table.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1544/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1075893249,I_kwDOBm6k_c5AINQB,1545,Custom pages don't work on windows,559711,ryascott,closed,0,,,,,3,2021-12-09T18:53:05Z,2022-02-03T02:08:31Z,2022-02-03T01:58:35Z,NONE,,"It seems that custom pages don't work when put in templates/pages

To reproduce on datasette version 0.59.4  using PowerShell on WIndows 10 with Python 3.10.0

    mkdir -p templates/pages

    echo ""hello world"" >> templates/pages/about.html

Start datasette
   
    datasette --template-dir templates/

Navigate to [http://127.0.0.1:8001/about](url) and receive:
    
   Error 404:
   Database not found: about





",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1545/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1076057610,I_kwDOBm6k_c5AI1YK,1546,validating the sql,50336793,jadsongmatos,closed,0,,,,,1,2021-12-09T21:35:57Z,2021-12-18T02:05:17Z,2021-12-18T02:05:16Z,NONE,,Could someone tell me that part of the code is responsible for validating the sql that guarantees that only a table can be read,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1546/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1076388044,I_kwDOBm6k_c5AKGDM,1547,Writable canned queries fail to load custom templates,127565,wragge,closed,0,,,7571612,Datasette 0.60,6,2021-12-10T03:31:48Z,2022-01-13T22:27:59Z,2021-12-19T21:12:00Z,CONTRIBUTOR,,"I've created a canned query with `""write"": true` set. I've also created a custom template for it, but the template doesn't seem to be found. If I look in the HTML I see (`stock_exchange` is the db name):

`<!-- Templates considered: query-stock_exchange.html, *query.html -->`

My non-writeable canned queries pick up custom templates as expected, and if I look at their HTML I see the canned query name added to the templates considered (the canned query here is `date_search`):

`<!-- Templates considered: query-stock_exchange-date_search.html, query-stock_exchange.html, *query.html -->`

So it seems like the writeable canned query is behaving differently for some reason. Is it an authentication thing? I'm using the built in `--root` authentication.

Thanks!
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1547/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1076834768,PR_kwDOBm6k_c4vrZxV,1548,"Update pytest-xdist requirement from <2.5,>=2.2.1 to >=2.2.1,<2.6",49699333,dependabot[bot],closed,0,,,,,1,2021-12-10T13:12:06Z,2021-12-13T23:22:22Z,2021-12-13T23:22:21Z,CONTRIBUTOR,simonw/datasette/pulls/1548,"Updates the requirements on [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst"">pytest-xdist's changelog</a>.</em></p>
<blockquote>
<h1>pytest-xdist 2.5.0 (2021-12-10)</h1>
<h2>Features</h2>
<ul>
<li>
<p><code>[#722](https://github.com/pytest-dev/pytest-xdist/issues/722) &lt;https://github.com/pytest-dev/pytest-xdist/issues/722&gt;</code>_: Full compatibility with pytest 7 - no deprecation warnings or use of legacy features.</p>
</li>
<li>
<p><code>[#733](https://github.com/pytest-dev/pytest-xdist/issues/733) &lt;https://github.com/pytest-dev/pytest-xdist/issues/733&gt;</code>_: New <code>--dist=loadgroup</code> option, which ensures all tests marked with <code>@pytest.mark.xdist_group</code> run in the same session/worker. Other tests run distributed as in <code>--dist=load</code>.</p>
</li>
</ul>
<h2>Trivial Changes</h2>
<ul>
<li>
<p><code>[#708](https://github.com/pytest-dev/pytest-xdist/issues/708) &lt;https://github.com/pytest-dev/pytest-xdist/issues/708&gt;</code>_: Use <code>@pytest.hookspec</code> decorator to declare hook options in <code>newhooks.py</code> to avoid warnings in <code>pytest 7.0</code>.</p>
</li>
<li>
<p><code>[#719](https://github.com/pytest-dev/pytest-xdist/issues/719) &lt;https://github.com/pytest-dev/pytest-xdist/issues/719&gt;</code>_: Use up-to-date <code>setup.cfg</code>/<code>pyproject.toml</code> packaging setup.</p>
</li>
<li>
<p><code>[#720](https://github.com/pytest-dev/pytest-xdist/issues/720) &lt;https://github.com/pytest-dev/pytest-xdist/issues/720&gt;</code>_: Require pytest&gt;=6.2.0.</p>
</li>
<li>
<p><code>[#721](https://github.com/pytest-dev/pytest-xdist/issues/721) &lt;https://github.com/pytest-dev/pytest-xdist/issues/721&gt;</code>_: Started using type annotations and mypy checking internally. The types are incomplete and not published.</p>
</li>
</ul>
<h1>pytest-xdist 2.4.0 (2021-09-20)</h1>
<h2>Features</h2>
<ul>
<li>
<p><code>[#696](https://github.com/pytest-dev/pytest-xdist/issues/696) &lt;https://github.com/pytest-dev/pytest-xdist/issues/696&gt;</code>_: On Linux, the process title now changes to indicate the current worker state (running/idle).</p>
<p>Depends on the <code>setproctitle &lt;https://pypi.org/project/setproctitle/&gt;</code>__ package, which can be installed with <code>pip install pytest-xdist[setproctitle]</code>.</p>
</li>
<li>
<p><code>[#704](https://github.com/pytest-dev/pytest-xdist/issues/704) &lt;https://github.com/pytest-dev/pytest-xdist/issues/704&gt;</code>_: Add support for Python 3.10.</p>
</li>
</ul>
<h1>pytest-xdist 2.3.0 (2021-06-16)</h1>
<h2>Deprecations and Removals</h2>
<ul>
<li><code>[#654](https://github.com/pytest-dev/pytest-xdist/issues/654) &lt;https://github.com/pytest-dev/pytest-xdist/issues/654&gt;</code>_: Python 3.5 is no longer supported.</li>
</ul>
<h2>Features</h2>
<ul>
<li><code>[#646](https://github.com/pytest-dev/pytest-xdist/issues/646) &lt;https://github.com/pytest-dev/pytest-xdist/issues/646&gt;</code>_: Add <code>--numprocesses=logical</code> flag, which automatically uses the number of logical CPUs available, instead of physical CPUs with <code>auto</code>.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/13f39349c6950a881c1fe4fcd5984af2e8b7c220""><code>13f3934</code></a> Remove unnecessary skip from test_logfinish_hook as we require pytest&gt;=6.2</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/c76d5622f17135e892965d742377870eb9b07933""><code>c76d562</code></a> Skip test_warning_captured_deprecated_in_pytest_6 in pytest&gt;=7.1</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/5f78c7155e66ab73bdc7631c4ac6bfe684b82500""><code>5f78c71</code></a> Fix CHANGELOG header</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/c8bbc03e49d5a53b5da808c7328e8f3ad6ed2d7e""><code>c8bbc03</code></a> Release 2.5.0</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/8dbf3677dc7cc26ada33cf8a27d7ac51a9be467b""><code>8dbf367</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/738"">#738</a> from pytest-dev/pre-commit-ci-update-config</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/a25c14bef59ad728e39cabc64f71190aaad73b0a""><code>a25c14b</code></a> [pre-commit.ci] pre-commit autoupdate</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/110c114025202d11570737be823de158d1bb8d99""><code>110c114</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/734"">#734</a> from nicoddemus/revamp-readme</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/83bdbf4b95c914a889d1faa8fba8d506bcc2f8c7""><code>83bdbf4</code></a> Revamp README</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/630c1eb6f2c31dcb4c38c75bb62f868237cdde94""><code>630c1eb</code></a> Merge pull request <a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/733"">#733</a> from baekdohyeop/feature-loadgroup</li>
<li><a href=""https://github.com/pytest-dev/pytest-xdist/commit/62e50d00977b41e175b5f119381f9db760459ddc""><code>62e50d0</code></a> Address review</li>
<li>Additional commits viewable in <a href=""https://github.com/pytest-dev/pytest-xdist/compare/v2.2.1...v2.5.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1548/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1077620955,I_kwDOBm6k_c5AOzDb,1549,Redesign CSV export to improve usability,536941,fgregg,open,0,,,3268330,Datasette 1.0,5,2021-12-11T19:02:12Z,2022-04-04T11:17:13Z,,CONTRIBUTOR,,"*Original title: Set content type for CSV so that browsers will attempt to download instead opening in the browser*

Right now, if the user clicks on the CSV related to a <s>table or a</s> query, the response header for the content type is 

""content-type: text/plain; charset=utf-8""

Most browsers will try to open a file with this content-type in the browser. 

This is not what most people want to do, and lots of folks don't know that if they want to download the CSV and open it in the a spreadsheet program they next need to save the page through their browser.

It would be great if the response header could be something like 

```
'Content-type: text/csv');
'Content-disposition: attachment;filename=MyVerySpecial.csv');
```

which would lead browsers to open a download dialog.


",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1549/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
277589569,MDU6SXNzdWUyNzc1ODk1Njk=,155,A primary key column that has foreign key restriction associated won't rendering label column,388154,wsxiaoys,closed,0,,,2949431,Custom templates edition,4,2017-11-29T00:40:02Z,2017-12-07T05:39:53Z,2017-12-07T05:39:53Z,NONE,,,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/155/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1077628073,I_kwDOBm6k_c5AO0yp,1550,Research option for returning all rows from arbitrary query,9599,simonw,open,0,,,,,2,2021-12-11T19:31:11Z,2021-12-11T23:43:24Z,,OWNER,,"Inspired by thinking about #1549 - returning ALL rows from an arbitrary query is a lot easier if you just run that query and keep iterating over the cursor.

I've avoided doing that in the past because it could tie up a connection for a long time - but in private instances this wouldn't be such a problem.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1550/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1077893013,I_kwDOBm6k_c5AP1eV,1551,`keep_blank_values=True` when parsing `request.args`,9599,simonw,closed,0,,,7571612,Datasette 0.60,3,2021-12-12T19:53:07Z,2022-01-13T22:26:04Z,2021-12-12T20:02:01Z,OWNER,,"This code in `TableView` wouldn't be necessary: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/views/table.py#L396-L399

If that happened here instead: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/utils/asgi.py#L98-L100

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1518#issuecomment-991827468_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1551/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1078702875,I_kwDOBm6k_c5AS7Mb,1552,Allow to set `facets_array` in metadata (like current `facets`),3556,davidbgk,closed,0,,,7571612,Datasette 0.60,9,2021-12-13T16:00:44Z,2022-01-13T22:26:15Z,2021-12-16T18:47:48Z,CONTRIBUTOR,,"For now, you can set a `facets` value (array) in your metadata file but I couldn't find a way to set a `facets_array` in order to provide default facets for arrays (like tags). My use-case is to access to [that kind of view](https://latest.datasette.io/fixtures/facetable?_facet_array=tags) by default without URL's parameters as with other default facets.

_I'm new to datasette, and I'm willing to help with a PR if that is not already implemented and I missed it!_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1552/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1079111498,I_kwDOBm6k_c5AUe9K,1553,if csv export is truncated in non streaming mode set informative response header,536941,fgregg,open,0,,,,,3,2021-12-13T22:50:44Z,2021-12-16T19:17:28Z,,CONTRIBUTOR,,"streaming mode is currently not enabled for custom queries, so the queries will be truncated to max row limit.

it would be great if a response is truncated that an header signalling that was set in the header.

i need to write some pagination code for getting full results back for a custom query and it would make the code much better if i could reliably known when there is nothing more to limit/offset ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1553/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1079129258,PR_kwDOBm6k_c4vynly,1554,TableView refactor,9599,simonw,closed,0,,,,,3,2021-12-13T23:16:04Z,2021-12-20T23:52:11Z,2021-12-20T23:52:04Z,OWNER,simonw/datasette/pulls/1554,"I'm starting a PR with almost nothing in it so I can use the GitHub code commenting feature to add a bunch of comments to the code I intend to refactor.

Related issues:

- #617 
- #715 
- #870
- #1518",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1554/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1079149656,I_kwDOBm6k_c5AUoRY,1555,Optimize all those calls to index_list and foreign_key_list,9599,simonw,closed,0,,,7571612,Datasette 0.60,27,2021-12-13T23:50:56Z,2022-01-13T22:27:32Z,2021-12-19T20:55:59Z,OWNER,,"On the first hit to a restarted index I'm seeing this in the SQL traces: https://latest-with-plugins.datasette.io/github/commits?_trace=1

<img width=""826"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/145907011-e4abc93c-8dbf-468d-afef-13d5e7463f92.png"">

I imagine this could be sped up a lot using tricks like this one from the SQLite documentation: https://sqlite.org/pragma.html#pragfunc

```sql
SELECT DISTINCT m.name || '.' || ii.name AS 'indexed-columns'
  FROM sqlite_schema AS m,
       pragma_index_list(m.name) AS il,
       pragma_index_info(il.name) AS ii
 WHERE m.type='table'
 ORDER BY 1;
```
https://latest-with-plugins.datasette.io/fixtures?sql=SELECT+DISTINCT+m.name+%7C%7C+%27.%27+%7C%7C+ii.name+AS+%27indexed-columns%27%0D%0A++FROM+sqlite_schema+AS+m%2C%0D%0A+++++++pragma_index_list%28m.name%29+AS+il%2C%0D%0A+++++++pragma_index_info%28il.name%29+AS+ii%0D%0A+WHERE+m.type%3D%27table%27%0D%0A+ORDER+BY+1%3B",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1555/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1081318247,I_kwDOBm6k_c5Ac5tn,1556,"Show count of facet values always, not just for `?_facet_size=max`",9599,simonw,closed,0,,,7571612,Datasette 0.60,1,2021-12-15T17:49:01Z,2022-01-13T22:26:07Z,2021-12-15T17:58:06Z,OWNER,,"> You've caused me to rethink this feature - I no longer think there's value in only showing these numbers if `?_facet_size=max` as opposed to all of the time.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1423#issuecomment-995023410_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1556/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,completed
1082564912,I_kwDOBm6k_c5AhqEw,1557,`?_nosuggest=1` parameter for disabling facet suggestions on table view,9599,simonw,closed,0,,,7571612,Datasette 0.60,1,2021-12-16T19:21:42Z,2022-01-13T22:26:48Z,2021-12-16T19:24:59Z,OWNER,,"Found I wanted this while I was debugging #625 just to clean up the debug traces, but it makes sense as a partner to `?_nofacet=1` and `?_nocount=1` from #1350 and #1353.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1557/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1082584499,I_kwDOBm6k_c5Ahu2z,1558,Redesign `facet_results` JSON structure prior to Datasette 1.0,9599,simonw,open,0,,,3268330,Datasette 1.0,3,2021-12-16T19:45:10Z,2023-01-09T15:31:17Z,,OWNER,,"> Decision: as an initial fix I'm going to de-duplicate those keys by using `tags__array` etc - with a `_2` on the end if that key is already used.
>
> I'll open a separate issue to redesign this better for Datasette 1.0.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/625#issuecomment-996130862_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1558/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1082743068,PR_kwDOBm6k_c4v-izc,1559,"filters_from_request plugin hook, now used in TableView",9599,simonw,closed,0,,,,,6,2021-12-16T23:59:33Z,2021-12-17T23:09:41Z,2021-12-17T19:02:15Z,OWNER,simonw/datasette/pulls/1559,"New plugin hook, refs #473

Used it to extract the logic from TableView that handles _search and
_through and _where - refs #1518",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1559/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
278189708,MDU6SXNzdWUyNzgxODk3MDg=,156,Document CSS hooks and custom templates,9599,simonw,closed,0,,,2949431,Custom templates edition,1,2017-11-30T16:43:15Z,2017-11-30T17:11:34Z,2017-11-30T17:10:58Z,OWNER,,Documentation currently lives in commit messages on https://github.com/simonw/datasette/commit/8ab3a169d42d096f2c7979c6d3d7746618d30f0b and 3cd06729f457d690603b6060dc552b535517ab09,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/156/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1082746149,I_kwDOBm6k_c5AiWUl,1560,"Table page title has ""where where"" in it",9599,simonw,closed,0,,,7571612,Datasette 0.60,0,2021-12-17T00:05:48Z,2022-01-13T22:28:35Z,2022-01-13T22:20:15Z,OWNER,,"Just noticed this while working on #1518.

```
% curl -s 'https://latest.datasette.io/fixtures/facetable?_sort=pk&on_earth__exact=1' | grep -C 1 '<title>'
<head>
    <title>fixtures: facetable: 14 rows
    where where on_earth = 1 sorted by pk</title>
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1560/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1082765654,I_kwDOBm6k_c5AibFW,1561,"add hash id to ""_memory"" url if hashed url mode is turned on and crossdb is also turned on",536941,fgregg,closed,0,,,,,3,2021-12-17T00:45:12Z,2022-03-19T04:45:40Z,2022-03-19T04:45:40Z,CONTRIBUTOR,,"If hashed_url mode is turned on and crossdb is also turned on, then queries to _memory should have a hash_id. 

One way that it could work is to have the _memory hash be a hash of all the individual databases.

Otherwise, crossdb queries can get quit out of data if using aggressive caching.

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1561/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1083246400,PR_kwDOBm6k_c4wAMK8,1562,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1",49699333,dependabot[bot],closed,0,,,,,4,2021-12-17T13:11:10Z,2021-12-17T23:08:29Z,2021-12-17T23:08:28Z,CONTRIBUTOR,simonw/datasette/pulls/1562,"Updates the requirements on [janus](https://github.com/aio-libs/janus) to permit the latest version.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/aio-libs/janus/releases"">janus's releases</a>.</em></p>
<blockquote>
<h2>janus 1.0.0 release</h2>
<ul>
<li>Dropped Python 3.6 support</li>
<li>Janus is marked as stable, no API changes was made for years</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/aio-libs/janus/blob/master/CHANGES.rst"">janus's changelog</a>.</em></p>
<blockquote>
<h2>1.0.0 (2021-12-17)</h2>
<ul>
<li>Drop Python 3.6 support</li>
</ul>
<h2>0.7.0 (2021-11-24)</h2>
<ul>
<li>Add SyncQueue and AsyncQueue Protocols to provide type hints for sync and async queues <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/374"">#374</a></li>
</ul>
<h2>0.6.2 (2021-10-24)</h2>
<ul>
<li>Fix Python 3.10 compatibility <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/358"">#358</a></li>
</ul>
<h2>0.6.1 (2020-10-26)</h2>
<ul>
<li>
<p>Raise RuntimeError on queue.join() after queue closing. <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/295"">#295</a></p>
</li>
<li>
<p>Replace <code>timeout</code> type from <code>Optional[int]</code> to <code>Optional[float]</code> <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/267"">#267</a></p>
</li>
</ul>
<h2>0.6.0 (2020-10-10)</h2>
<ul>
<li>
<p>Drop Python 3.5, the minimal supported version is Python 3.6</p>
</li>
<li>
<p>Support Python 3.9</p>
</li>
<li>
<p>Refomat with <code>black</code></p>
</li>
</ul>
<h2>0.5.0 (2020-04-23)</h2>
<ul>
<li>Remove explicit loop arguments and forbid creating queues outside event loops <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/246"">#246</a></li>
</ul>
<h2>0.4.0 (2018-07-28)</h2>
<ul>
<li>
<p>Add <code>py.typed</code> macro <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/89"">#89</a></p>
</li>
<li>
<p>Drop python 3.4 support and fix minimal version python3.5.3 <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/88"">#88</a></p>
</li>
<li>
<p>Add property with that indicates if queue is closed <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/86"">#86</a></p>
</li>
</ul>
<h2>0.3.2 (2018-07-06)</h2>
<ul>
<li>Fixed python 3.7 support <a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/97"">#97</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/aio-libs/janus/commit/0783f9b7a9bb7e1c095e93ebb4aad4f1e219f512""><code>0783f9b</code></a> Fix coverage upload</li>
<li><a href=""https://github.com/aio-libs/janus/commit/41c49bafb1b192d2ee25b7394cead2386e452dc2""><code>41c49ba</code></a> Make deployment only if checks are green</li>
<li><a href=""https://github.com/aio-libs/janus/commit/ec94b35b2ae095dcb97827f1369c0cd31b7e8e5e""><code>ec94b35</code></a> Fix CI again</li>
<li><a href=""https://github.com/aio-libs/janus/commit/2303208c2f972e38445e7ecec54fda0f3203f566""><code>2303208</code></a> Fix CI</li>
<li><a href=""https://github.com/aio-libs/janus/commit/dff507895bf8d77efea2c4cc1d8b04a8a2986a0b""><code>dff5078</code></a> Bump to 1.0.0</li>
<li><a href=""https://github.com/aio-libs/janus/commit/3421545f3954b7ef6d90e02b7653a7ab685f3e78""><code>3421545</code></a> Bump mypy from 0.910 to 0.920 (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/384"">#384</a>)</li>
<li><a href=""https://github.com/aio-libs/janus/commit/56b2d1d8dbd10cce28302a4e1c4224ce219c6246""><code>56b2d1d</code></a> Bump black from 21.11b1 to 21.12b0 (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/383"">#383</a>)</li>
<li><a href=""https://github.com/aio-libs/janus/commit/883e82bea0af1d12a68e92148b75b3344b31227a""><code>883e82b</code></a> Update README.rst</li>
<li><a href=""https://github.com/aio-libs/janus/commit/2e30d8a0f3c77c383a39da9b5c233a5c93a049fb""><code>2e30d8a</code></a> Bump coverage from 6.1.2 to 6.2 (<a href=""https://github-redirect.dependabot.com/aio-libs/janus/issues/382"">#382</a>)</li>
<li><a href=""https://github.com/aio-libs/janus/commit/7b72d8577517422dd0719da1c6a0fb33d8a10e23""><code>7b72d85</code></a> Bump to 0.7</li>
<li>Additional commits viewable in <a href=""https://github.com/aio-libs/janus/compare/v0.6.2...v1.0.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1562/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1083573206,I_kwDOBm6k_c5AlgPW,1563,Datasette(... files=) should not be a required argument,9599,simonw,closed,0,,,7571612,Datasette 0.60,2,2021-12-17T19:54:18Z,2022-01-13T22:27:18Z,2021-12-18T02:19:40Z,OWNER,,"```pycon
>>> ds = Datasette(memory=True)
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
TypeError: __init__() missing 1 required positional argument: 'files'
>>> ds = Datasette(memory=True, files=[])
```
I wanted to create an in-memory Datasette for running some tests, no point in forcing me to pass `files=[]` to do that.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1563/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1083581011,I_kwDOBm6k_c5AliJT,1564,_prepare_connection not called on write connections,9599,simonw,closed,0,,,7571612,Datasette 0.60,1,2021-12-17T20:06:47Z,2022-01-20T21:29:43Z,2021-12-18T01:58:44Z,OWNER,,"I was trying to initalize SpatiaLite in a write connection:
```pycon
>>> from datasette.app import Datasette
>>> ds = Datasette(memory=True, files=[], sqlite_extensions=[""spatialite""])
>>> db = ds.add_memory_database('geo')
>>> await db.execute_write(""select InitSpatialMetadata(1)"")
UUID('3f143baa-4e3d-5842-a36f-4fa2f683b72f')
no such function: InitSpatialMetadata
```
It looks like the code that loads additional modules only works on read-only connections, not on write connections:

https://github.com/simonw/datasette/blob/92a5280d2e75c39424a75ad6226fc74400ae984f/datasette/database.py#L146-L153

Compared to:

https://github.com/simonw/datasette/blob/92a5280d2e75c39424a75ad6226fc74400ae984f/datasette/database.py#L124-L132",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1564/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1083657868,I_kwDOBm6k_c5Al06M,1565,Documented JavaScript variables on different templates made available for plugins,9599,simonw,open,0,,,,,8,2021-12-17T22:30:51Z,2021-12-19T22:37:29Z,,OWNER,,"While working on https://github.com/simonw/datasette-leaflet-freedraw/issues/10 I found myself writing this atrocity to figure out the SQL query used for a specific table page:

```javascript
let innerSql = Array.from(document.getElementsByTagName(""span"")).filter(
    el => el.innerText == ""View and edit SQL""
)[0].parentElement.getAttribute(""title"")
```
This is obviously bad - it's very brittle, and will break if I ever change the text on that link (like localizing it for example).

Instead, I think pages like that one should have a block of script at the bottom something like this:
```javascript
window.datasette = window.datasette || {};
datasette.view_name = 'table';
datasette.table_sql = 'select * from ...';
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1565/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1083669410,I_kwDOBm6k_c5Al3ui,1566,Release Datasette 0.60,9599,simonw,closed,0,,,7571612,Datasette 0.60,6,2021-12-17T22:58:12Z,2022-01-14T01:59:55Z,2022-01-14T01:59:55Z,OWNER,,Using this as a tracking issue. I'm hoping to get the bulk of the JSON redesign work from the refactor in #1554 in for this release.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1566/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1083718998,I_kwDOBm6k_c5AmD1W,1567,Remove undocumented sqlite_functions mechanism,9599,simonw,closed,0,,,7571612,Datasette 0.60,0,2021-12-18T01:51:10Z,2022-01-13T22:27:04Z,2021-12-18T01:54:46Z,OWNER,,"I added this in 0b8c1b0a6da9cb8ac0d28cc90dd783de87554036 but it's never been documented and the same thing can now be achieved using the `prepare_connection` plugin hook.

https://github.com/simonw/datasette/blob/0c91e59d2bbfc08884cfcf5d1b902a2f4968b7ff/datasette/app.py#L262

https://github.com/simonw/datasette/blob/0c91e59d2bbfc08884cfcf5d1b902a2f4968b7ff/datasette/app.py#L551-L552

It's used here in the tests:

https://github.com/simonw/datasette/blob/69244a617b1118dcbd04a8f102173f04680cf08c/tests/fixtures.py#L156",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1567/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1083726550,I_kwDOBm6k_c5AmFrW,1568,Trace should show queries on the write connection too,9599,simonw,closed,0,,,7571612,Datasette 0.60,2,2021-12-18T02:34:12Z,2022-01-13T22:27:23Z,2021-12-18T02:42:34Z,OWNER,,"> Here's why - `trace` only applies to read, not write SQL operations: https://github.com/simonw/datasette/blob/7c8f8aa209e4ba7bf83976f8495d67c28fbfca24/datasette/database.py#L209-L211

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1555#issuecomment-997128508_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1568/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1083895395,I_kwDOBm6k_c5Amu5j,1569,"db.execute_write(..., executescript=True) parameter",9599,simonw,closed,0,,,7571612,Datasette 0.60,2,2021-12-18T18:20:47Z,2022-01-13T22:27:27Z,2021-12-18T18:34:18Z,OWNER,,"> Idea: teach `execute_write` to accept an optional `executescript=True` parameter, like this:
```diff
diff --git a/datasette/database.py b/datasette/database.py
index 468e936..1a424f5 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -94,10 +94,14 @@ class Database:
             f""file:{self.path}{qs}"", uri=True, check_same_thread=False
         )
 
-    async def execute_write(self, sql, params=None, block=False):
+    async def execute_write(self, sql, params=None, executescript=False, block=False):
+        assert not executescript and params, ""Cannot use params with executescript=True""
         def _inner(conn):
             with conn:
-                return conn.execute(sql, params or [])
+                if executescript:
+                    return conn.executescript(sql)
+                else:
+                    return conn.execute(sql, params or [])
 
         with trace(""sql"", database=self.name, sql=sql.strip(), params=params):
             results = await self.execute_write_fn(_inner, block=block)
```

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1555#issuecomment-997248364_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1569/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
278190321,MDU6SXNzdWUyNzgxOTAzMjE=,157,"Teach ""datasette publish"" about custom template directories",9599,simonw,closed,0,,,2949431,Custom templates edition,1,2017-11-30T16:44:57Z,2020-01-15T16:05:13Z,2017-12-09T18:28:54Z,OWNER,,"The following command should copy the custom templates into the deployment and ensure `datasette serve` correctly serves them:

    datasette publish now mydb.db --template-dir=custom-templates/",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/157/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1083921371,I_kwDOBm6k_c5Am1Pb,1570,Separate db.execute_write() into three methods,9599,simonw,closed,0,,,7571612,Datasette 0.60,2,2021-12-18T18:45:54Z,2022-01-13T22:27:38Z,2021-12-18T18:57:25Z,OWNER,,"> Rather than adding a `executemany=True` parameter, I'm now thinking a better design might be to have three methods:
>
> - `db.execute_write(sql, params=None, block=False)`
> - `db.execute_write_script(sql, block=False)`
> - `db.execute_write_many(sql, params_seq, block=False)`

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1555#issuecomment-997267416_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1570/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1083927147,I_kwDOBm6k_c5Am2pr,1571,Track number of executions for execute_write_many() in traces,9599,simonw,closed,0,,,7571612,Datasette 0.60,0,2021-12-18T19:16:17Z,2022-01-13T22:27:49Z,2021-12-19T20:30:40Z,OWNER,,"Spotted while working on #1555

<img width=""1171"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/146653166-ce5358bc-b822-4961-a2dd-ed3efaa8dd13.png"">

There's no indication there of how many times `execute_write_many()` executed the SQL.

Solving this is a tiny bit tricky because `params_seq` is an iterator that we don't want to exhaust before passing it to `conn.executemany()` - so we need to instead wrap it in something that counts how many times it was called.

But then we need a way to attach that to the trace here: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/database.py#L115-L122

So probably need to redesign the `trace()` decorator to allow extra pairs to be attached to it within the `with` statement.

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1571/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1084007781,I_kwDOBm6k_c5AnKVl,1572,"""Query took"" should be ""Queries took""",9599,simonw,closed,0,,,7571612,Datasette 0.60,0,2021-12-19T04:03:00Z,2022-01-13T22:27:43Z,2021-12-19T04:03:24Z,OWNER,,"This is misleading, since usually there have been more than one query executed:

![CleanShot 2021-12-18 at 20 02 35@2x](https://user-images.githubusercontent.com/9599/146663457-9c4c2900-5cc0-4650-a565-bb1ff0b8a725.png)

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1572/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1084185188,I_kwDOBm6k_c5An1pk,1573,Make trace() a documented internal API,9599,simonw,open,0,,,,,1,2021-12-19T20:32:56Z,2021-12-19T21:13:13Z,,OWNER,,"This should be documented so plugin authors can use it to add their own custom traces: https://github.com/simonw/datasette/blob/8f311d6c1d9f73f4ec643009767749c17b5ca5dd/datasette/tracer.py#L28-L52

Including the new `kwargs` pattern I added in #1571: https://github.com/simonw/datasette/blob/f65817000fdf87ce8a0c23edc40784ebe33b5842/datasette/database.py#L128-L132",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1573/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1084193403,PR_kwDOBm6k_c4wDKmb,1574,introduce new option for datasette package to use a slim base image,33631,fs111,closed,0,,,,,6,2021-12-19T21:18:19Z,2022-08-15T08:49:31Z,2022-08-15T08:49:31Z,NONE,simonw/datasette/pulls/1574,"The official python images on docker hub come with a slim variant that is significantly smaller than the default. The diff does not change the default, but allows to switch to the `slim` variant with commandline switch (`--slim-base-image`)

Size comparison:

```
$ datasette package some.db -t fat --install ""datasette-basemap datasette-cluster-map""

$ datasette  package some.db -t slim --slim-base-image --install ""datasette-basemap datasette-cluster-map""

$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
fat          latest    807b393ace0d   9 seconds ago    978MB
slim         latest    31bc5e63505c   8 minutes ago    191MB

```",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1574/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1084257842,I_kwDOBm6k_c5AoHYy,1575,__call__() got an unexpected keyword argument 'specname',9599,simonw,closed,0,,,,,1,2021-12-20T01:24:04Z,2021-12-20T01:48:03Z,2021-12-20T01:47:57Z,OWNER,,"> I've installed the alpha version but get an error when starting up Datasette:

```
Traceback (most recent call last):
  File ""/Users/tim/.pyenv/versions/stock-exchange/bin/datasette"", line 5, in <module>
    from datasette.cli import cli
  File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/cli.py"", line 15, in <module>
    from .app import Datasette, DEFAULT_SETTINGS, SETTINGS, SQLITE_LIMIT_ATTACHED, pm
  File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/app.py"", line 31, in <module>
    from .views.database import DatabaseDownload, DatabaseView
  File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/views/database.py"", line 25, in <module>
    from datasette.plugins import pm
  File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/plugins.py"", line 29, in <module>
    mod = importlib.import_module(plugin)
  File ""/Users/tim/.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py"", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/filters.py"", line 9, in <module>
    @hookimpl(specname=""filters_from_request"")
TypeError: __call__() got an unexpected keyword argument 'specname'
```

_Originally posted by @wragge in https://github.com/simonw/datasette/issues/1547#issuecomment-997511968_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1575/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1087181951,I_kwDOBm6k_c5AzRR_,1576,Traces should include SQL executed by subtasks created with `asyncio.gather`,9599,simonw,closed,0,,,3268330,Datasette 1.0,12,2021-12-22T20:52:02Z,2022-02-05T05:21:35Z,2022-02-05T05:19:53Z,OWNER,,"I tried running some parallel SQL queries using `asyncio.gather()` but the SQL that was executed didn't show up in the trace rendered by https://datasette.io/plugins/datasette-pretty-traces

I realized that was because traces are keyed against the current task ID, which changes when a sub-task is run using `asyncio.gather` or similar.

The faceting and suggest faceting queries are missing from this trace:

![image](https://user-images.githubusercontent.com/9599/147153855-2d611f07-922a-4d18-9e6e-4be89e010dc4.png)

> The reason they aren't showing up in the traces is that traces are stored just for the currently executing `asyncio` task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25
>
> This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively ""child tasks"" of the current request, and hence should be tracked the same.
>
> https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using `asyncio.gather()` to work around this kind of problem.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1518#issuecomment-999870993_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1576/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1087913724,I_kwDOBm6k_c5A2D78,1577,Drop support for Python 3.6,9599,simonw,closed,0,,,3268330,Datasette 1.0,6,2021-12-23T18:17:03Z,2022-01-25T23:30:03Z,2022-01-20T04:31:41Z,OWNER,,"*Original title: Decide when to drop support for Python 3.6*

> `context_vars` can solve this but they were introduced in Python 3.7: https://www.python.org/dev/peps/pep-0567/
>
> Python 3.6 support ends in a few days time, and it looks like Glitch has updated to 3.7 now - so maybe I can get away with Datasette needing 3.7 these days?
>
> Tweeted about that here: https://twitter.com/simonw/status/1473761478155010048

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1576#issuecomment-999878907_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1577/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1087919372,I_kwDOBm6k_c5A2FUM,1578,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key,9599,simonw,open,0,,,,,4,2021-12-23T18:27:59Z,2021-12-24T21:33:19Z,,OWNER,,"Found this while working on https://github.com/simonw/datasette-tiddlywiki

<img width=""1254"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/147279097-e02f80f3-cc88-4bdd-a26f-03f924c13b5e.png"">

Then clicking on `/tiddlywiki/tiddlers/%24%3A%2FDefaultTiddlers` returns a 404.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1578/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1087931918,I_kwDOBm6k_c5A2IYO,1579,`.execute_write(... block=True)` should be the default behaviour,9599,simonw,closed,0,,,7571612,Datasette 0.60,7,2021-12-23T18:54:28Z,2022-01-13T22:28:08Z,2021-12-23T19:18:26Z,OWNER,,"Every single piece of code I've written against the write APIs has used the `block=True` option to wait for the result.

Without that, it instead fires the write into the queue but then continues even before it has finished executing.

`block=True` should clearly be the default behaviour here!",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1579/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
278190981,MDU6SXNzdWUyNzgxOTA5ODE=,158,Ensure default templates are designed to be extended,9599,simonw,closed,0,,,2949431,Custom templates edition,1,2017-11-30T16:46:41Z,2017-12-07T05:41:09Z,2017-12-07T05:41:08Z,OWNER,,"Since custom templates can do `{% extends ""default:table.html"" %}` the default templates should include sensible named `{% block %}` components designed to support common extension patterns. Since we already support `{{ super() }}` we may not have much if anything to add here.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/158/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1089529555,I_kwDOBm6k_c5A8ObT,1581,"when hashed urls are turned on, the _memory db has improperly long-lived cache expiry",536941,fgregg,closed,0,,,,,1,2021-12-28T00:05:48Z,2022-03-24T04:08:18Z,2022-03-24T04:08:18Z,CONTRIBUTOR,,"if hashed_urls are on, then a -000 suffix is added to the `_memory` database, and the cache settings are set just as if it was a normal hashed database.

in particular, this header is set:

`cache-control: max-age=31536000`

this is not appropriate because the `_memory-000` database isn't really hashed based on the contents of the databases (see #1561).

Either the cache-control header should be changed, or the _memory db should have a hash suffix that does depend on the contents of the databases.

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1581/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1090055810,PR_kwDOBm6k_c4wWDxH,1582,don't set far expiry if hash is '000',536941,fgregg,closed,0,,,,,1,2021-12-28T18:16:13Z,2022-03-24T04:07:58Z,2022-03-24T04:07:58Z,CONTRIBUTOR,simonw/datasette/pulls/1582,"This will close #1581.

I couldn't find any unit tests related to the testing hashed urls, and I know that you want to break that code out of the core application (#1561), so I'm not quite sure what you would like me to for testing.",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1582/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1090810196,I_kwDOBm6k_c5BBHFU,1583,consider adding deletion step of cloudbuild artifacts to gcloud publish,536941,fgregg,open,0,,,,,1,2021-12-30T00:33:23Z,2021-12-30T00:34:16Z,,CONTRIBUTOR,,"right now, as part of the the publish process images and other artifacts are stored to gcloud's cloud storage before being deployed to cloudrun.

after successfully deploying, it would be nice if the the script deleted these artifacts. otherwise, if you have regularly scheduled build process, you can end up paying to store lots of out of date artifacts.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1583/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1091257796,I_kwDOBm6k_c5BC0XE,1584,give error with recursive sql,58088336,tunguyenatwork,open,0,,,,,0,2021-12-30T18:53:16Z,2021-12-30T18:53:16Z,,NONE,,"I got an error ""near ""WITH"": syntax error"" after I upgraded to version 0.59 from 0.52.4.    This error is related to recursive sql.  It works great on the previous version but it failed after upgraded.  Below is an example of sql:

WITH RECURSIVE manager_of(position, super_position) AS (SELECT position, case ifnull(INDIRECT_SUPER_POSITION,'') when '' then super_position else INDIRECT_SUPER_POSITION end as SUPER_POSITION FROM position where super_position<>'SGV000000001' and super_position!=''  and position <> super_position),chain_manager_of_position(position, level) AS (SELECT super_position, 1 as level FROM manager_of WHERE super_position!='' and (position=:pos or position in (Select position from employee where employee=:ein)) UNION ALL SELECT super_position, level+1 as level FROM manager_of JOIN chain_manager_of_position USING(position)) SELECT * FROM chain_manager_of_position left join employee using(position)  where  employee is not NULL order by level limit 1",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1584/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1091838742,I_kwDOBm6k_c5BFCMW,1585,Fire base caching for `publish cloudrun`,9599,simonw,open,0,,,,,1,2022-01-01T15:38:15Z,2022-01-01T15:40:38Z,,OWNER,,"https://gist.github.com/steren/03d3e58c58c9a53fd49bb78f58541872 has a recipe for this, via https://twitter.com/steren/status/1477038411114446848

Could this enable easier vanity URLs of the format `https://$project_id.web.app/`? How about CDN caching?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1585/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1096536240,I_kwDOBm6k_c5BW9Cw,1586,run analyze on all databases as part of start up or publishing,536941,fgregg,open,0,,,,,1,2022-01-07T17:52:34Z,2022-02-02T07:13:37Z,,CONTRIBUTOR,,"Running `analyze;` lets sqlite's query planner make *much* better use of any indices.

It might be nice if the analyze was run as part of the start up of ""serve"" or ""publish"".",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1586/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1097040427,I_kwDOBm6k_c5BY4Ir,1587,Add `sqlite_stat1`(-4) tables to hidden table list,9599,simonw,closed,0,,,,,2,2022-01-08T21:28:20Z,2022-01-20T04:12:59Z,2022-01-20T04:12:59Z,OWNER,,"> Running `ANALYZE` creates a new visible table called `sqlite_stat1`: https://www.sqlite.org/fileformat.html#the_sqlite_stat1_table
>
> This should be added to the default list of hidden tables in Datasette.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1587/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1097101917,I_kwDOBm6k_c5BZHJd,1588,`explain query plan select` is too strict about whitespace,9599,simonw,closed,0,,,7571612,Datasette 0.60,3,2022-01-09T04:22:42Z,2022-01-13T22:28:19Z,2022-01-13T20:35:05Z,OWNER,,"`explain query plan select * from facetable` is allowed: https://latest.datasette.io/fixtures?sql=explain+query+plan+select+*+from+facetable

But... `explain query plan  select * from facetable` (with two spaces before the `select`) returns a ""Statement must be a SELECT"" error: https://latest.datasette.io/fixtures?sql=explain+query+plan++select+*+from+facetable",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1588/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1098275181,PR_kwDOBm6k_c4wwNCl,1589,Typo in docs about default redirect status code,3556,davidbgk,closed,0,,,,,1,2022-01-10T19:14:36Z,2022-03-06T02:27:49Z,2022-03-06T01:58:32Z,CONTRIBUTOR,simonw/datasette/pulls/1589,,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1589/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
278191223,MDU6SXNzdWUyNzgxOTEyMjM=,159,Come up with an elegant mechanism for per-row template customization,9599,simonw,closed,0,,,2949431,Custom templates edition,0,2017-11-30T16:47:26Z,2017-12-07T06:12:27Z,2017-12-07T06:12:26Z,OWNER,,It would be nice if customizing the display of an individual row in a custom table template was as simple as possible - refs #153 ,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/159/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1099723916,I_kwDOBm6k_c5BjHSM,1590,Table+query JSON and CSV links broken when using `base_url` setting,1001306,eelkevdbos,closed,0,,,7571612,Datasette 0.60,11,2022-01-11T23:46:39Z,2022-01-14T01:16:34Z,2022-01-14T01:16:08Z,NONE,,"Datasette appends the prefix found in the `base_url` setting twice if a `base_url` is set.

In the follow asgi example, I'm hosting a custom Datasette instance:

```python
# asgi.py
import pathlib
from asgi_cors import asgi_cors
from channels.routing import URLRouter
from django.urls import re_path
from datasette.app import Datasette

datasette_ = Datasette(
    files=[],
    settings={
        ""base_url"": ""/datasettes/"",
        ""plugins"": {}
    },
    config_dir=pathlib.Path('.'),
)
application = URLRouter([
  re_path(r""^datasettes/.*"", asgi_cors(datasette_.app(), allow_all=True)),
])
```

Running it with:
```shell
$ daphne -p 8002 asgi:application
```

Using a simple query on the `_memory` table: 
```sql
select sqlite_version()
```

http://localhost:8002/datasettes/_memory?sql=select+sqlite_version%28%29

It renders the following upon inspection:
![image](https://user-images.githubusercontent.com/1001306/149038851-aa842950-126a-467c-9a86-fae13bce6221.png)

I am using datasette version `0.59.4`",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1590/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1100015398,I_kwDOBm6k_c5BkOcm,1591,Maybe let plugins define custom serve options?,9599,simonw,open,0,,,,,7,2022-01-12T08:18:47Z,2022-01-15T11:56:59Z,,OWNER,,"https://twitter.com/psychemedia/status/1481171650934714370

> can extensions be passed their own cli args? eg `--ext-tiddlywiki-dbname tiddlywiki2.sqlite` ?

I've thought something like this might be useful for other plugins in the past, too.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1591/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1100499619,I_kwDOBm6k_c5BmEqj,1592,Row pages should show links to foreign keys,9599,simonw,open,0,,,,,1,2022-01-12T15:50:20Z,2022-01-12T15:52:17Z,,OWNER,,Refs #1518 refactor.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1592/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1101705012,PR_kwDOBm6k_c4w7eqc,1593,"Update pytest-asyncio requirement from <0.17,>=0.10 to >=0.10,<0.18",49699333,dependabot[bot],closed,0,,,,,2,2022-01-13T13:11:50Z,2022-02-07T13:13:24Z,2022-02-07T13:13:23Z,CONTRIBUTOR,simonw/datasette/pulls/1593,"Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/pytest-dev/pytest-asyncio/releases"">pytest-asyncio's releases</a>.</em></p>
<blockquote>
<h2>pytest-asyncio 0.17.0</h2>
<hr />
<h2>title: 'pytest-asyncio: pytest support for asyncio'</h2>
<p><a href=""https://pypi.python.org/pypi/pytest-asyncio""><img src=""https://img.shields.io/pypi/v/pytest-asyncio.svg"" alt=""image"" /></a></p>
<p><a href=""https://github.com/pytest-dev/pytest-asyncio/actions?workflow=CI""><img src=""https://github.com/pytest-dev/pytest-asyncio/workflows/CI/badge.svg"" alt=""image"" /></a></p>
<p><a href=""https://codecov.io/gh/pytest-dev/pytest-asyncio""><img src=""https://codecov.io/gh/pytest-dev/pytest-asyncio/branch/master/graph/badge.svg"" alt=""image"" /></a></p>
<p><a href=""https://github.com/pytest-dev/pytest-asyncio""><img src=""https://img.shields.io/pypi/pyversions/pytest-asyncio.svg"" alt=""Supported Python versions"" /></a></p>
<p><a href=""https://github.com/ambv/black""><img src=""https://img.shields.io/badge/code%20style-black-000000.svg"" alt=""image"" /></a></p>
<p>pytest-asyncio is an Apache2 licensed library, written in Python, for
testing asyncio code with pytest.</p>
<p>asyncio code is usually written in the form of coroutines, which makes
it slightly more difficult to test using normal testing tools.
pytest-asyncio provides useful fixtures and markers to make testing
easier.</p>
<pre lang=""{.sourceCode"" data-meta="".python}""><code>@pytest.mark.asyncio
async def test_some_asyncio_code():
    res = await library.do_something()
    assert b&quot;expected result&quot; == res
</code></pre>
<p>pytest-asyncio has been strongly influenced by
<a href=""https://github.com/eugeniy/pytest-tornado"">pytest-tornado</a>.</p>
<h1>Features</h1>
<ul>
<li>fixtures for creating and injecting versions of the asyncio event
loop</li>
<li>fixtures for injecting unused tcp/udp ports</li>
<li>pytest markers for treating tests as asyncio coroutines</li>
<li>easy testing with non-default event loops</li>
<li>support for [async def]{.title-ref} fixtures and async generator
fixtures</li>
<li>support <em>auto</em> mode to handle all async fixtures and tests
automatically by asyncio; provide <em>strict</em> mode if a test suite
should work with different async frameworks simultaneously, e.g.
<code>asyncio</code> and <code>trio</code>.</li>
</ul>
<h1>Installation</h1>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/2e2d5d202ecfea6fd403d92a9dd6ab6734aa0f85""><code>2e2d5d2</code></a> Bump to 0.17 release</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/90436c98dd27e670f0c721b48bb28001ccdcbbda""><code>90436c9</code></a> Fix pandoc installation procedure</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/d291c666870d2d903fc99543dccd77dca8496d5b""><code>d291c66</code></a> Convert README.rst to Markdown for making githun release</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/141937b89aa6ced9856ed3f997818ef8e3fbea57""><code>141937b</code></a> Fix release artifacts</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/696cf7d5e0a458825c27ae8c7d621fb538c70827""><code>696cf7d</code></a> Fix trove classifier for asyncio</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/8ccb30650068f58d6f3e7314d2c1f2d59cdba1e9""><code>8ccb306</code></a> Build on tag</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/cd8498709d5cf763c7a73cac9b43a0bd2a2d4fb7""><code>cd84987</code></a> Release process automation (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/252"">#252</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/2eb12a7b0591dfb8578303235d87bb25ddeedf77""><code>2eb12a7</code></a> Setup GitHub Workflows linter and yaml-reformatter (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/253"">#253</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/d28b826b8acb329401cceed286ef3b42dc82df05""><code>d28b826</code></a> Bump codecov/codecov-action from 1 to 2.1.0 (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/251"">#251</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/2f523bad98ee1256c26ffe94c78ebcd8a1d03688""><code>2f523ba</code></a> Configure dependabot version updater (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/250"">#250</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/pytest-dev/pytest-asyncio/compare/v0.10.0...v0.17.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1593/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1102359726,I_kwDOBm6k_c5BtKyu,1594,"Add a CLI reference page to the docs, inspired by sqlite-utils",9599,simonw,closed,0,,,7571612,Datasette 0.60,3,2022-01-13T20:55:08Z,2022-01-13T22:28:22Z,2022-01-13T21:38:48Z,OWNER,,"Thought of this while posting this comment: https://github.com/simonw/datasette/issues/1591#issuecomment-1012506595

I added https://sqlite-utils.datasette.io/en/stable/cli-reference.html to `sqlite-utils` in https://github.com/simonw/sqlite-utils/issues/383 and I _really_ like it - it's a page showing the `--help` output of every CLI command for that tool.

It's maintained using `cog`. One of the benefits is that I get a free commit history of changes to `--help` at https://github.com/simonw/sqlite-utils/commits/main/docs/cli-reference.rst",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1594/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1102484126,I_kwDOBm6k_c5BtpKe,1595,Release notes for 0.60,9599,simonw,closed,0,,,7571612,Datasette 0.60,4,2022-01-13T22:23:14Z,2022-01-14T01:37:39Z,2022-01-14T01:37:39Z,OWNER,,,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1595/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1102568047,I_kwDOBm6k_c5Bt9pv,1596,Documentation page warning of changes coming in 1.0,9599,simonw,open,0,,,,,0,2022-01-13T23:26:04Z,2022-01-13T23:26:04Z,,OWNER,,I should start this relatively soon.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1596/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1102612922,I_kwDOBm6k_c5BuIm6,1597,"""datasette inspect"" has no help summary",9599,simonw,closed,0,,,,,1,2022-01-14T00:02:16Z,2022-01-14T00:07:36Z,2022-01-14T00:07:36Z,OWNER,,"Made obvious by the new CLI reference page added in #1594. https://docs.datasette.io/en/latest/cli-reference.html#datasette-inspect-help
```
Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect
  install    Install Python packages - e.g.
```
```
Usage: datasette inspect [OPTIONS] [FILES]...

Options:
  --inspect-file TEXT
  --load-extension TEXT  Path to a SQLite extension to load
  --help                 Show this message and exit.
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1597/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1102637351,I_kwDOBm6k_c5BuOkn,1598,Replace update-docs-help.py script with cog,9599,simonw,closed,0,,,,,1,2022-01-14T00:33:27Z,2022-01-14T00:47:57Z,2022-01-14T00:47:57Z,OWNER,,"I introduced `cog` in #1594 - I can use this to replace the older `update-docs-help.py` mechanism:

https://github.com/simonw/datasette/blob/76d66d5b2bf10249c0beaac0999b93ac8d757f48/tests/test_docs.py#L36-L53",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1598/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1102966378,I_kwDOBm6k_c5Bve5q,1599,Add architecture documentation,9599,simonw,open,0,,,,,0,2022-01-14T04:55:38Z,2022-01-14T04:56:03Z,,OWNER,,"Inspired by https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html

Good example: https://github.com/rust-analyzer/rust-analyzer/blob/d7c99931d05e3723d878bea5dc26766791fa4e69/docs/dev/architecture.md",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1599/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
267726219,MDU6SXNzdWUyNjc3MjYyMTk=,16,Default HTML/CSS needs to look reasonable and be responsive,9599,simonw,closed,0,,,2857392,Ship first public release,6,2017-10-23T16:05:22Z,2017-11-11T20:19:07Z,2017-11-11T20:19:07Z,OWNER,,"Version one should have the following characteristics:

- Looks OK
- Works great on mobile
- Loads extremely fast
- No JavaScript! At least not in v1.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/16/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
278208011,MDU6SXNzdWUyNzgyMDgwMTE=,160,Ability to bundle and serve additional static files,9599,simonw,closed,0,,,2949431,Custom templates edition,8,2017-11-30T17:37:51Z,2019-02-02T00:58:20Z,2017-12-09T18:29:11Z,OWNER,,"Since we now have custom templates, we should support including custom static files with them as well. Maybe something like this:

    datasette mydb.db --template-dir=templates/ --static-dir=static/

This should also be supported by datasette publish - see also #157 ",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/160/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1104691662,I_kwDOBm6k_c5B2EHO,1600,plugins --all example should use cog,9599,simonw,closed,0,,,,,1,2022-01-15T11:47:49Z,2022-01-20T05:06:21Z,2022-01-20T05:04:16Z,OWNER,,The example output for `datasette plugins --all`on this page has got out of date: https://docs.datasette.io/en/stable/plugins.html#seeing-what-plugins-are-installed,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1600/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1105916061,I_kwDOBm6k_c5B6vCd,1601,Add KNN and data_licenses to hidden tables list,25778,eyeseast,closed,0,,,,,5,2022-01-17T14:19:57Z,2022-01-20T21:29:44Z,2022-01-20T04:38:54Z,CONTRIBUTOR,,"They're generated by Spatialite and not very interesting in most cases.

<img width=""626"" alt=""Screen Shot 2022-01-17 at 9 17 31 AM"" src=""https://user-images.githubusercontent.com/25778/149786443-7832363c-2a2c-4533-9edc-048cd00e4a08.png"">

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1601/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1108084641,PR_kwDOBm6k_c4xQ0uZ,1602,"Update pytest-timeout requirement from <2.1,>=1.4.2 to >=1.4.2,<2.2",49699333,dependabot[bot],closed,0,,,,,1,2022-01-19T13:11:50Z,2022-03-06T01:41:50Z,2022-03-06T01:41:49Z,CONTRIBUTOR,simonw/datasette/pulls/1602,"Updates the requirements on [pytest-timeout](https://github.com/pytest-dev/pytest-timeout) to permit the latest version.
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pytest-dev/pytest-timeout/commit/8e4800ecbcf149790bb8035ec37dc30f3d48a127""><code>8e4800e</code></a> Fixup readme</li>
<li><a href=""https://github.com/pytest-dev/pytest-timeout/commit/dc1efca167ef1d466ab805595528e6c64996cdeb""><code>dc1efca</code></a> 2.1.0 release</li>
<li><a href=""https://github.com/pytest-dev/pytest-timeout/commit/dd9d6080763c559de1ff5b15deb9c27559dd3431""><code>dd9d608</code></a> Add custom hooks specifications for overriding setup_timeout and teardown_tim...</li>
<li><a href=""https://github.com/pytest-dev/pytest-timeout/commit/ed8ecd6cbb8b8acccd57f6782dab94dcde0f83ea""><code>ed8ecd6</code></a> module names, they're difficult</li>
<li><a href=""https://github.com/pytest-dev/pytest-timeout/commit/3ab4319c5170bc9b9d8052019103c93c2ed74c86""><code>3ab4319</code></a> Add changelog</li>
<li><a href=""https://github.com/pytest-dev/pytest-timeout/commit/4f7ebae5789902b3097dbe618b109b8cafa3fd64""><code>4f7ebae</code></a> Replace deprecated py.io.get_terminal_width() with shutil.get_terminal_size()...</li>
<li><a href=""https://github.com/pytest-dev/pytest-timeout/commit/b8a2fa6e56330ca6510c50b59ba8ee75ec962822""><code>b8a2fa6</code></a> Prep release</li>
<li><a href=""https://github.com/pytest-dev/pytest-timeout/commit/951972da1c11ea5ca8ac76691f7c78e5deb1faaf""><code>951972d</code></a> Update changelog</li>
<li><a href=""https://github.com/pytest-dev/pytest-timeout/commit/748a9c37466b7381626a5f4152ec5dc9d2cc3907""><code>748a9c3</code></a> Making detection of whether a debugger is currently attached more flexible. (...</li>
<li><a href=""https://github.com/pytest-dev/pytest-timeout/commit/f8a46a194c89f1459fd97d5a8fa618e8326a2bfa""><code>f8a46a1</code></a> Github removed the git protocol (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-timeout/issues/112"">#112</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/pytest-dev/pytest-timeout/compare/1.4.2...2.1.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1602/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1108235694,I_kwDOBm6k_c5CDlWu,1603,A proper favicon,9599,simonw,closed,0,,,3268330,Datasette 1.0,19,2022-01-19T15:24:55Z,2022-03-19T04:04:49Z,2022-01-20T06:07:31Z,OWNER,,"Tips here: https://adamj.eu/tech/2022/01/18/how-to-add-a-favicon-to-your-django-site/ - I think a PNG served at `/favicon.ico` is the best option, since safari doesn't support SVG yet.

Relevant code: https://github.com/simonw/datasette/blob/cb29119db9115b1f40de2fb45263ed77e3bfbb3e/datasette/app.py#L182-L183

I can reuse the icon for https://datasette.io/desktop",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1603/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1108300685,I_kwDOBm6k_c5CD1ON,1604,Option to assign a domain/subdomain using `datasette publish cloudrun`,9599,simonw,open,0,,,,,1,2022-01-19T16:21:17Z,2022-01-19T16:23:54Z,,OWNER,,Looks like this API should be able to do that: https://twitter.com/steren/status/1483835859191304192 - https://cloud.google.com/run/docs/reference/rest/v1/namespaces.domainmappings/create,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1604/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1108671952,I_kwDOBm6k_c5CFP3Q,1605,Scripted exports,25778,eyeseast,open,0,,,,,10,2022-01-19T23:45:55Z,2022-11-30T15:06:38Z,,CONTRIBUTOR,,"Posting this while I'm thinking about it: I mentioned at the end of [this thread](https://twitter.com/eyeseast/status/1483893011658551299) that I'm usually doing `datasette --get` to export canned queries.

I used to use a tool called [datafreeze](https://github.com/pudo/datafreeze) to do scripted exports, but that project looks dead now. The ergonomics of it are pretty nice, though, and the `Freezefile.yml` structure is actually not too far from Datasette's canned queries.

This is related to the idea for `datasette query` (#1356) but I think it's a distinct feature. It's most likely a plugin, but I want to raise it here because it's probably something other people have thought about.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1605/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1108846067,I_kwDOBm6k_c5CF6Xz,1606,Tests failing against Python 3.6,9599,simonw,closed,0,,,,,3,2022-01-20T04:22:44Z,2022-01-20T04:36:42Z,2022-01-20T04:36:42Z,OWNER,,"https://github.com/simonw/datasette/runs/4877484366

```
E           File ""/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/uvicorn/server.py"", line 67, in run
E             return asyncio.run(self.serve(sockets=sockets))
E         AttributeError: module 'asyncio' has no attribute 'run'
```
I think this may mean `uvicorn` has dropped support for Python 3.6.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1606/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1109783030,I_kwDOBm6k_c5CJfH2,1607,More detailed information about installed SpatiaLite version,9599,simonw,closed,0,,,3268330,Datasette 1.0,2,2022-01-20T21:28:03Z,2022-02-09T06:42:02Z,2022-02-09T06:32:28Z,OWNER,,"https://www.gaia-gis.it/gaia-sins/spatialite-sql-5.0.0.html#version has a whole bunch of interesting functions for things like `freexl_version()` and `geos_version()` and `HasMathSQL()` and suchlike.

These could be shown on the `/-/versions` page.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1607/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1109808154,I_kwDOBm6k_c5CJlQa,1608,Documentation should clarify /stable/ vs /latest/,9599,simonw,closed,0,,,,,15,2022-01-20T22:02:59Z,2023-03-26T23:41:12Z,2022-01-20T22:53:17Z,OWNER,,"It's not currently clear what the difference between https://docs.datasette.io/en/latest/ and https://docs.datasette.io/en/stable/ is - I should fix that.

On Twitter: https://twitter.com/simonw/status/1484285006243528705",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1608/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1109884720,I_kwDOBm6k_c5CJ38w,1609,"Ensure ""pip install datasette"" still works with Python 3.6",9599,simonw,closed,0,,,,,12,2022-01-21T00:08:10Z,2022-01-24T19:20:09Z,2022-01-21T02:24:13Z,OWNER,,"## Original title: Can I keep ""pip install datasette"" working on Python 3.6?

I dropped support for 3.6 in:
- #1577

I'm getting reports that `pip3 install datasette` throws an error on that Python, even though I haven't made that new release yet - presumably due to lack of pinning of Uvicorn: https://twitter.com/ldodds/status/1484289475195080706

Is it possible to get `pip` on that version of Python to install the highest possible version of the packages that are still known to support Python 3.6?

If so, how?",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1609/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
278814220,MDU6SXNzdWUyNzg4MTQyMjA=,161,Support WITH query ,388154,wsxiaoys,closed,0,,,,,4,2017-12-03T20:00:40Z,2017-12-08T06:18:12Z,2017-12-04T04:52:41Z,NONE,,"Currently datasettle failed with error message: Statement must begin with SELECT

Example query
```sql
WITH RECURSIVE
  cnt(x) AS (
     SELECT 1
     UNION ALL
     SELECT x+1 FROM cnt
      LIMIT 1000000
  )
SELECT x FROM cnt;
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/161/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1112633417,PR_kwDOBm6k_c4xfryi,1610,"Update asgiref requirement from <3.5.0,>=3.2.10 to >=3.2.10,<3.6.0",49699333,dependabot[bot],closed,0,,,,,0,2022-01-24T13:14:18Z,2022-03-06T01:30:27Z,2022-03-06T01:30:27Z,CONTRIBUTOR,simonw/datasette/pulls/1610,"Updates the requirements on [asgiref](https://github.com/django/asgiref) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/django/asgiref/blob/main/CHANGELOG.txt"">asgiref's changelog</a>.</em></p>
<blockquote>
<h2>3.5.0 (2022-01-22)</h2>
<ul>
<li>
<p>Python 3.6 is no longer supported, and asyncio calls have been changed to
use only the modern versions of the APIs as a result</p>
</li>
<li>
<p>Several causes of RuntimeErrors in cases where an event loop was assigned
to a thread but not running</p>
</li>
<li>
<p>Speed improvements in the Local class</p>
</li>
</ul>
<h2>3.4.1 (2021-07-01)</h2>
<ul>
<li>Fixed an issue with the deadlock detection where it had false positives
during exception handling.</li>
</ul>
<h2>3.4.0 (2021-06-27)</h2>
<ul>
<li>
<p>Calling sync_to_async directly from inside itself (which causes a deadlock
when in the default, thread-sensitive mode) now has deadlock detection.</p>
</li>
<li>
<p>asyncio usage has been updated to use the new versions of get_event_loop,
ensure_future, wait and gather, avoiding deprecation warnings in Python 3.10.
Python 3.6 installs continue to use the old versions; this is only for 3.7+</p>
</li>
<li>
<p>sync_to_async and async_to_sync now have improved type hints that pass
through the underlying function type correctly.</p>
</li>
<li>
<p>All Websocket* types are now spelled WebSocket, to match our specs and the
official spelling. The old names will work until release 3.5.0, but will
raise deprecation warnings.</p>
</li>
<li>
<p>The typing for WebSocketScope and HTTPScope's <code>extensions</code> key has been
fixed.</p>
</li>
</ul>
<h2>3.3.4 (2021-04-06)</h2>
<ul>
<li>The async_to_sync type error is now a warning due the high false negative
rate when trying to detect coroutine-returning callables in Python.</li>
</ul>
<h2>3.3.3 (2021-04-06)</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/django/asgiref/commit/8b615134bd481c4262356ad3194fbeff9240d0d3""><code>8b61513</code></a> Releasing 3.5.0</li>
<li><a href=""https://github.com/django/asgiref/commit/b2e1c9dfdef212e69fc00adc8c83ffbc6fbac73c""><code>b2e1c9d</code></a> Fixed pytest_asyncio deprecation warning.</li>
<li><a href=""https://github.com/django/asgiref/commit/2eda55127a82618f83bb54788fecd239babb86ce""><code>2eda551</code></a> Added testing for Python 3.10.</li>
<li><a href=""https://github.com/django/asgiref/commit/02fecb6046bb5ec0dbbad00ffcd2043e893fcea5""><code>02fecb6</code></a> Drop Python 3.6 (<a href=""https://github-redirect.dependabot.com/django/asgiref/issues/307"">#307</a>)</li>
<li><a href=""https://github.com/django/asgiref/commit/6689c0a1e820a5c32532f069e945da8230d808ad""><code>6689c0a</code></a> Added stacklevel to warning in AsyncToSync.</li>
<li><a href=""https://github.com/django/asgiref/commit/4364f9b4bb052d9ec54b806867fd859b07c39186""><code>4364f9b</code></a> Changed how StatelessServer handles event loops</li>
<li><a href=""https://github.com/django/asgiref/commit/7bc055c0883db13ca8622e11ab116d0bd180acaf""><code>7bc055c</code></a> Update implementations.rst (<a href=""https://github-redirect.dependabot.com/django/asgiref/issues/295"">#295</a>)</li>
<li><a href=""https://github.com/django/asgiref/commit/c758984f4ab4864ac5787ef90d03b3589b2bcb90""><code>c758984</code></a> Move current_task import choice to module definition time</li>
<li><a href=""https://github.com/django/asgiref/commit/dfe87b2633d111f4f67c2cdb29a8c1ebe2eb8dae""><code>dfe87b2</code></a> Fixed <a href=""https://github-redirect.dependabot.com/django/asgiref/issues/292"">#292</a>: Use get_event_loop in class-level code</li>
<li><a href=""https://github.com/django/asgiref/commit/b3a65e34e9f445a1ccc234282deedafe4b8b5de2""><code>b3a65e3</code></a> Removed class variable which has been unused since a0bbe90</li>
<li>Additional commits viewable in <a href=""https://github.com/django/asgiref/compare/3.2.10...3.5.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1610/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1113384383,I_kwDOBm6k_c5CXOW_,1611,Avoid ever running count(*) against SpatiaLite KNN table,9599,simonw,open,0,,,,,1,2022-01-25T03:32:54Z,2022-02-02T06:45:47Z,,OWNER,,"Got this in a trace:

<img width=""941"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150906011-5f09dc6d-4def-433a-8546-e4fd94e0edf0.png"">

Looks like running `count(*)` against KNN took 83s! It ignored the time limit. And still only returned a count of 0.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1611/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1114147905,I_kwDOBm6k_c5CaIxB,1612,Move canned queries closer to the SQL input area,639012,jsfenfen,closed,0,,,3268330,Datasette 1.0,5,2022-01-25T17:06:39Z,2022-03-19T04:04:49Z,2022-01-25T18:34:21Z,CONTRIBUTOR,,"*Original title:  Consider placing example queries above the sql input?*

Hi! Have been enjoying deploying ad hoc datasettes for collaborators to pick over! 

I keep finding myself manually ""fixing"" the database.html template so that the ""example queries"" (canned queries) appear directly *over* the sql box? So they are sorta more a suggestion for collaborators who aren't inclined to write their own queries? 

My sense is any time I go to the trouble of writing canned queries my users should see 'em? 

(( I have also considered a client-side reactive-ish option where selecting a query just places the raw SQL in the box and doesn't execute it, but this seems to end up being an inconvenience, rather than a teaching tool. )) 

",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1612/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1114628238,I_kwDOBm6k_c5Cb-CO,1613,Improvements to help make Datasette a better tool for learning SQL,9599,simonw,open,0,,,,,5,2022-01-26T04:56:07Z,2022-01-26T16:41:46Z,,OWNER,,Tracking issue for the general goal of making Datasette a better tool for learning SQL.,107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1613/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1115435536,I_kwDOBm6k_c5CfDIQ,1614,Try again with SQLite codemirror support,9599,simonw,open,0,,,,,1,2022-01-26T20:05:20Z,2022-12-23T21:27:10Z,,OWNER,,"I tried and failed to implement autocomplete a while ago. Relevant code:

https://github.com/codemirror/legacy-modes/blob/8f36abca5f55024258cd23d9cfb0203d8d244f0d/mode/sql.js#L335

Sounds like upgrading to CodeMirror 6 ASAP would be worthwhile since it has better accessibility and touch screen support: https://codemirror.net/6/",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1614/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1117132741,I_kwDOBm6k_c5ClhfF,1615,Potential simplified publishing mechanism,369053,aidansteele,closed,0,,,,,2,2022-01-28T08:34:50Z,2022-02-02T07:34:21Z,2022-02-02T07:34:17Z,NONE,,"Hi,

Forewarning: this idea is one I've only been thinking about for a while and it's not fully fleshed-out yet.

I love Datasette and what it stands for. I was thinking about how we could make it accessible to more people, especially those without access to credit cards required for a lot of hosting options. Or they might not feel comfortable signing up for said services.

So I was thinking I might create a service that hosts Datasette instances for folks. I'd probably stick it on AWS Lambda and limit requests to something like n/month to avoid bankrupting myself. If I did build such a hypothetical service, I was thinking I would rely on GitHub Actions to do the heavy lifting. 

E.g. user `johndoe` creates a repo `my-animals` with a couple of files: `dogs.csv`, `cats.csv` and the following GitHub Actions workflow:

```yaml
# .github/workflows/push.yml
on:
  push

# this allows the publish action to use OIDC to authenticate johndoe/my-animals
permissions:
  id-token: write
  contents: read

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v2

      - run: pip install sqlite-utils

      - uses: actions/checkout@v2

      - run: |
          set -eux
          
          sqlite-utils create-database animals.db          
          sqlite-utils insert animals.db dogs dogs.csv --csv          
          sqlite-utils insert animals.db cats cats.csv --csv

      - uses: datasette-hub/publish@v1
        with:
          db: animals.db
          metadata: meta.yml

        # this step is helpful for debugging why the
        # generated sqlite db was rejected
      - uses: actions/upload-artifact@v2
        if: failure()
        with:
          path: animals.db
          retention-days: 1
```

This would then cause a Datasette instance to be available at `https://johndoe-my-animals.datasette-hub.test/`. It feels like this could significantly reduce the friction to someone being able to go from data set to Datasette.

What do you think? Does this address a real need? Or am I perhaps misunderstanding the main friction points? As a bonus: it feels like this would pair well with [git scraping](https://simonwillison.net/2020/Oct/9/git-scraping/).",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1615/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1119413338,PR_kwDOBm6k_c4x1kCu,1616,Bump black from 21.12b0 to 22.1.0,49699333,dependabot[bot],closed,0,,,,,2,2022-01-31T13:13:46Z,2022-02-02T22:23:52Z,2022-02-02T22:23:51Z,CONTRIBUTOR,simonw/datasette/pulls/1616,"Bumps [black](https://github.com/psf/black) from 21.12b0 to 22.1.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>22.1.0</h2>
<p>At long last, Black is no longer a beta product! This is the first non-beta release and the first release covered by our new stability policy.</p>
<h3>Highlights</h3>
<ul>
<li>Remove Python 2 support (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2740"">#2740</a>)</li>
<li>Introduce the <code>--preview</code> flag (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2752"">#2752</a>)</li>
</ul>
<h3>Style</h3>
<ul>
<li>Deprecate <code>--experimental-string-processing</code> and move the functionality under <code>--preview</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2789"">#2789</a>)</li>
<li>For stubs, one blank line between class attributes and methods is now kept if there's at least one pre-existing blank line (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2736"">#2736</a>)</li>
<li>Black now normalizes string prefix order (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2297"">#2297</a>)</li>
<li>Remove spaces around power operators if both operands are simple (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2726"">#2726</a>)</li>
<li>Work around bug that causes unstable formatting in some cases in the presence of the magic trailing comma (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2807"">#2807</a>)</li>
<li>Use parentheses for attribute access on decimal float and int literals (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2799"">#2799</a>)</li>
<li>Don't add whitespace for attribute access on hexadecimal, binary, octal, and complex literals (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2799"">#2799</a>)</li>
<li>Treat blank lines in stubs the same inside top-level if statements (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2820"">#2820</a>)</li>
<li>Fix unstable formatting with semicolons and arithmetic expressions (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2817"">#2817</a>)</li>
<li>Fix unstable formatting around magic trailing comma (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2572"">#2572</a>)</li>
</ul>
<h3>Parser</h3>
<ul>
<li>Fix mapping cases that contain as-expressions, like <code>case {&quot;key&quot;: 1 | 2 as password}</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2686"">#2686</a>)</li>
<li>Fix cases that contain multiple top-level as-expressions, like <code>case 1 as a, 2 as b</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2716"">#2716</a>)</li>
<li>Fix call patterns that contain as-expressions with keyword arguments, like <code>case Foo(bar=baz as quux)</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2749"">#2749</a>)</li>
<li>Tuple unpacking on <code>return</code> and <code>yield</code> constructs now implies 3.8+ (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2700"">#2700</a>)</li>
<li>Unparenthesized tuples on annotated assignments (e.g <code>values: Tuple[int, ...] = 1, 2, 3</code>) now implies 3.8+ (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2708"">#2708</a>)</li>
<li>Fix handling of standalone <code>match()</code> or <code>case()</code> when there is a trailing newline or a comment inside of the parentheses. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2760"">#2760</a>)</li>
<li><code>from __future__ import annotations</code> statement now implies Python 3.7+ (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2690"">#2690</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Speed-up the new backtracking parser about 4X in general (enabled when <code>--target-version</code> is set to 3.10 and higher). (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2728"">#2728</a>)</li>
<li>Black is now compiled with mypyc for an overall 2x speed-up. 64-bit Windows, MacOS, and Linux (not including musl) are supported. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1009"">#1009</a>, <a href=""https://github-redirect.dependabot.com/psf/black/issues/2431"">#2431</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>Do not accept bare carriage return line endings in pyproject.toml (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2408"">#2408</a>)</li>
<li>Add configuration option (<code>python-cell-magics</code>) to format cells with custom magics in Jupyter Notebooks (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2744"">#2744</a>)</li>
<li>Allow setting custom cache directory on all platforms with environment variable <code>BLACK_CACHE_DIR</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2739"">#2739</a>).</li>
<li>Enable Python 3.10+ by default, without any extra need to specify -<code>-target-version=py310</code>. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2758"">#2758</a>)</li>
<li>Make passing <code>SRC</code> or <code>--code</code> mandatory and mutually exclusive (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2804"">#2804</a>)</li>
</ul>
<h3>Output</h3>
<ul>
<li>Improve error message for invalid regular expression (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2678"">#2678</a>)</li>
<li>Improve error message when parsing fails during AST safety check by embedding the underlying SyntaxError (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2693"">#2693</a>)</li>
<li>No longer color diff headers white as it's unreadable in light themed terminals (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2691"">#2691</a>)</li>
<li>Text coloring added in the final statistics (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2712"">#2712</a>)</li>
<li>Verbose mode also now describes how a project root was discovered and which paths will be formatted. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2526"">#2526</a>)</li>
</ul>
<h3>Packaging</h3>
<ul>
<li>All upper version bounds on dependencies have been removed (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2718"">#2718</a>)</li>
<li><code>typing-extensions</code> is no longer a required dependency in Python 3.10+ (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2772"">#2772</a>)</li>
<li>Set <code>click</code> lower bound to <code>8.0.0</code> as <em>Black</em> crashes on <code>7.1.2</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2791"">#2791</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>22.1.0</h2>
<p>At long last, <em>Black</em> is no longer a beta product! This is the first non-beta release
and the first release covered by our new stability policy.</p>
<h3>Highlights</h3>
<ul>
<li><strong>Remove Python 2 support</strong> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2740"">#2740</a>)</li>
<li>Introduce the <code>--preview</code> flag (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2752"">#2752</a>)</li>
</ul>
<h3>Style</h3>
<ul>
<li>Deprecate <code>--experimental-string-processing</code> and move the functionality under
<code>--preview</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2789"">#2789</a>)</li>
<li>For stubs, one blank line between class attributes and methods is now kept if there's
at least one pre-existing blank line (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2736"">#2736</a>)</li>
<li>Black now normalizes string prefix order (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2297"">#2297</a>)</li>
<li>Remove spaces around power operators if both operands are simple (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2726"">#2726</a>)</li>
<li>Work around bug that causes unstable formatting in some cases in the presence of the
magic trailing comma (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2807"">#2807</a>)</li>
<li>Use parentheses for attribute access on decimal float and int literals (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2799"">#2799</a>)</li>
<li>Don't add whitespace for attribute access on hexadecimal, binary, octal, and complex
literals (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2799"">#2799</a>)</li>
<li>Treat blank lines in stubs the same inside top-level <code>if</code> statements (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2820"">#2820</a>)</li>
<li>Fix unstable formatting with semicolons and arithmetic expressions (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2817"">#2817</a>)</li>
<li>Fix unstable formatting around magic trailing comma (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2572"">#2572</a>)</li>
</ul>
<h3>Parser</h3>
<ul>
<li>Fix mapping cases that contain as-expressions, like <code>case {&quot;key&quot;: 1 | 2 as password}</code>
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/2686"">#2686</a>)</li>
<li>Fix cases that contain multiple top-level as-expressions, like <code>case 1 as a, 2 as b</code>
(<a href=""https://github-redirect.dependabot.com/psf/black/issues/2716"">#2716</a>)</li>
<li>Fix call patterns that contain as-expressions with keyword arguments, like
<code>case Foo(bar=baz as quux)</code> (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2749"">#2749</a>)</li>
<li>Tuple unpacking on <code>return</code> and <code>yield</code> constructs now implies 3.8+ (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2700"">#2700</a>)</li>
<li>Unparenthesized tuples on annotated assignments (e.g
<code>values: Tuple[int, ...] = 1, 2, 3</code>) now implies 3.8+ (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2708"">#2708</a>)</li>
<li>Fix handling of standalone <code>match()</code> or <code>case()</code> when there is a trailing newline or a
comment inside of the parentheses. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2760"">#2760</a>)</li>
<li><code>from __future__ import annotations</code> statement now implies Python 3.7+ (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2690"">#2690</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Speed-up the new backtracking parser about 4X in general (enabled when
<code>--target-version</code> is set to 3.10 and higher). (<a href=""https://github-redirect.dependabot.com/psf/black/issues/2728"">#2728</a>)</li>
<li><em>Black</em> is now compiled with <a href=""https://github.com/mypyc/mypyc"">mypyc</a> for an overall 2x
speed-up. 64-bit Windows, MacOS, and Linux (not including musl) are supported. (<a href=""https://github-redirect.dependabot.com/psf/black/issues/1009"">#1009</a>,
<a href=""https://github-redirect.dependabot.com/psf/black/issues/2431"">#2431</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href=""https://github.com/psf/black/commits/22.1.0"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.12b0&new-version=22.1.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1616/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1120990806,PR_kwDOBm6k_c4x6zZ5,1617,"Ensure template_path always uses ""/"" to match jinja",3526913,cb160,closed,0,,,,,3,2022-02-01T17:20:30Z,2022-02-03T01:58:35Z,2022-02-03T01:58:35Z,CONTRIBUTOR,simonw/datasette/pulls/1617,"This PR shoudl fix #1545 

The existing code substituted / for \, assuming this was the right behaviour for windows. But on Windows, Jinja still uses / for the template list - See https://github.com/pallets/jinja/blob/896a62135bcc151f2997e028c5125bec2cb2431f/src/jinja2/loaders.py#L225",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1617/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1121121305,I_kwDOBm6k_c5C0vQZ,1618,"Reconsider policy on blocking queries containing the string ""pragma""",770231,strada,open,0,,,,,6,2022-02-01T19:39:46Z,2022-02-02T19:42:03Z,,NONE,,"First of all, thanks for creating this cool project, and also supporting publishing to various hosting services out of the box.

While testing out, I noticed legitimate queries such as 
```
select * from books where title like 'Pragmatic%'
```
or
```
select * from books where title = 'The Pragmatic Programmer'
```
are blocked, due to the regular expression check here:
https://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L185

Example as seen from a Datasette instance:
https://fivethirtyeight.datasettes.com/polls?sql=select+*+from+books+where+title+like+%27Pragmatic%25%27%0D%0A

I'd propose a regular expression like 
```
re.compile(f""pragma_(?!({'|'.join(allowed_pragmas)}))""),
```
instead of
```
re.compile(f""pragma(?!_({'|'.join(allowed_pragmas)}))""),
```

I can create a pull request with this change, unless the maintainers think it would allow unwanted queries to be executed.
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1618/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1121583414,I_kwDOBm6k_c5C2gE2,1619,JSON link on row page is 404 if base_url setting is used,9599,simonw,open,0,,,,,5,2022-02-02T07:09:53Z,2023-03-24T15:38:04Z,,OWNER,,"On my local environment:

    datasette fixtures.db -p 3344 --setting base_url /foo/bar/

Then hit http://127.0.0.1:3344/foo/bar/fixtures/table%2Fwith%2Fslashes.csv/3

<img width=""434"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/152109515-64e2a0f8-989c-4e90-9755-65c60da677ad.png"">

But... that `json` link goes here, which is a 404:

http://127.0.0.1:3344/foo/bar/foo/bar/fixtures/table%2Fwith%2Fslashes.csv/3?_format=json",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1619/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
279199916,MDU6SXNzdWUyNzkxOTk5MTY=,162,Link should not show up in the column selection dropdowns,9599,simonw,closed,0,,,2949431,Custom templates edition,0,2017-12-05T00:19:04Z,2017-12-07T05:05:58Z,2017-12-07T05:05:58Z,OWNER,,"e.g. on https://san-francisco.datasettes.com/food-trucks-921342f/Applicant

<img width=""729"" alt=""food-trucks__applicant__131_rows"" src=""https://user-images.githubusercontent.com/9599/33583381-ece18de6-d90e-11e7-8287-d5e4813dcb78.png"">


",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/162/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1121618041,I_kwDOBm6k_c5C2oh5,1620,"Link: rel=""alternate"" to JSON for queries too",9599,simonw,closed,0,,,3268330,Datasette 1.0,3,2022-02-02T08:02:42Z,2022-02-02T21:53:02Z,2022-02-02T21:33:00Z,OWNER,,"Following:
- #1533

I implemented it for tables and rows but I should have done queries as well.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1620/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1122413719,I_kwDOBm6k_c5C5qyX,1621,Test against Python 3.11 dev version,9599,simonw,closed,0,,,3268330,Datasette 1.0,0,2022-02-02T21:38:57Z,2022-03-19T04:04:49Z,2022-02-02T21:58:54Z,OWNER,,"To avoid another surprise like we got with 3.10: https://simonwillison.net/2021/Oct/9/finding-and-reporting-a-bug/

From a quick GitHub code search it looks like `3.11-dev` should work: https://cs.github.com/urllib3/urllib3/blob/7bec77e81aa0a194c98381053225813f5347c9d2/.github/workflows/ci.yml#L60",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1621/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1122414274,PR_kwDOBm6k_c4x_evE,1622,Test against Python 3.11-dev,9599,simonw,closed,0,,,,,1,2022-02-02T21:39:38Z,2022-02-02T21:58:53Z,2022-02-02T21:58:53Z,OWNER,simonw/datasette/pulls/1622,Refs #1621,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1622/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1122416919,I_kwDOBm6k_c5C5rkX,1623,/-/patterns returns link: alternate JSON header to 404,9599,simonw,closed,0,,,3268330,Datasette 1.0,2,2022-02-02T21:42:49Z,2022-03-19T04:04:49Z,2022-02-02T21:48:56Z,OWNER,,"Bug from:
- #1620

```
% curl -s -I 'https://latest.datasette.io/-/patterns' | grep link
link: https://latest.datasette.io/-/patterns.json; rel=""alternate""; type=""application/json+datasette""
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1623/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1122427321,I_kwDOBm6k_c5C5uG5,1624,Index page `/` has no CORS headers,9599,simonw,open,0,,,,,2,2022-02-02T21:56:10Z,2022-09-28T16:54:22Z,,OWNER,,"Compare the following:
```
% curl -I 'https://latest.datasette.io/fixtures'
HTTP/1.1 200 OK
link: https://latest.datasette.io/fixtures.json; rel=""alternate""; type=""application/json+datasette""
cache-control: max-age=5
referrer-policy: no-referrer
access-control-allow-origin: *
access-control-allow-headers: Authorization
access-control-expose-headers: Link
content-type: text/html; charset=utf-8
x-databases: _memory, _internal, fixtures, extra_database
Date: Wed, 02 Feb 2022 21:55:49 GMT
Server: Google Frontend
Transfer-Encoding: chunked

% curl -I 'https://latest.datasette.io/'        
HTTP/1.1 200 OK
link: https://latest.datasette.io/.json; rel=""alternate""; type=""application/json+datasette""
content-type: text/html; charset=utf-8
x-databases: _memory, _internal, fixtures, extra_database
Date: Wed, 02 Feb 2022 21:55:52 GMT
Server: Google Frontend
Transfer-Encoding: chunked
```",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1624/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1122450452,I_kwDOBm6k_c5C5zwU,1625,Try running tests against macOS and Windows in addition to Ubuntu,9599,simonw,open,0,,,,,0,2022-02-02T22:25:57Z,2022-02-02T22:25:57Z,,OWNER,,"I already do this for `sqlite-utils`: https://github.com/simonw/sqlite-utils/blob/3.22.1/.github/workflows/test.yml

Related:
- #1617
- #1545",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1625/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1122451096,PR_kwDOBm6k_c4x_mXy,1626,Try test suite against macOS and Windows,9599,simonw,open,0,,,,,3,2022-02-02T22:26:51Z,2022-02-03T01:22:44Z,,OWNER,simonw/datasette/pulls/1626,Refs #1625,107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1626/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1122557010,I_kwDOBm6k_c5C6NxS,1627,Get the tests passing against Windows,9599,simonw,open,0,,,,,0,2022-02-03T01:23:06Z,2022-02-03T01:23:32Z,,OWNER,,"> OK, the tests do NOT pass against Windows! https://github.com/simonw/datasette/runs/5044105941
>
> <img width=""767"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/152265501-5b5af51b-0fae-4aa7-b86d-7b61aba24b3c.png"">

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1626#issuecomment-1028515161_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1627/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1124191982,PR_kwDOBm6k_c4yFTCp,1629,"Update pytest requirement from <6.3.0,>=5.2.2 to >=5.2.2,<7.1.0",49699333,dependabot[bot],closed,0,,,,,1,2022-02-04T13:14:10Z,2022-03-06T01:30:06Z,2022-03-06T01:30:06Z,CONTRIBUTOR,simonw/datasette/pulls/1629,"Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/pytest-dev/pytest/releases"">pytest's releases</a>.</em></p>
<blockquote>
<h2>7.0.0</h2>
<h1>pytest 7.0.0 (2022-02-03)</h1>
<p>(<strong>Please see the full set of changes for this release also in the 7.0.0rc1 notes below</strong>)</p>
<h2>Deprecations</h2>
<ul>
<li>
<p><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9488"">#9488</a>: If custom subclasses of nodes like <code>pytest.Item</code>{.interpreted-text role=&quot;class&quot;} override the
<code>__init__</code> method, they should take <code>**kwargs</code>. See
<code>uncooperative-constructors-deprecated</code>{.interpreted-text role=&quot;ref&quot;} for details.</p>
<p>Note that a deprection warning is only emitted when there is a conflict in the
arguments pytest expected to pass. This deprecation was already part of pytest
7.0.0rc1 but wasn't documented.</p>
</li>
</ul>
<h2>Bug Fixes</h2>
<ul>
<li><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9355"">#9355</a>: Fixed error message prints function decorators when using assert in Python 3.8 and above.</li>
<li><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9396"">#9396</a>: Ensure <code>pytest.Config.inifile</code>{.interpreted-text role=&quot;attr&quot;} is available during the <code>pytest_cmdline_main &lt;_pytest.hookspec.pytest_cmdline_main&gt;</code>{.interpreted-text role=&quot;func&quot;} hook (regression during <code>7.0.0rc1</code>).</li>
</ul>
<h2>Improved Documentation</h2>
<ul>
<li><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9404"">#9404</a>: Added extra documentation on alternatives to common misuses of [pytest.warns(None)]{.title-ref} ahead of its deprecation.</li>
<li><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9505"">#9505</a>: Clarify where the configuration files are located. To avoid confusions documentation mentions
that configuration file is located in the root of the repository.</li>
</ul>
<h2>Trivial/Internal Changes</h2>
<ul>
<li><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9521"">#9521</a>: Add test coverage to assertion rewrite path.</li>
</ul>
<h1>pytest 7.0.0rc1 (2021-12-06)</h1>
<h2>Breaking Changes</h2>
<ul>
<li>
<p><a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7259"">#7259</a>: The <code>Node.reportinfo() &lt;non-python tests&gt;</code>{.interpreted-text role=&quot;ref&quot;} function first return value type has been expanded from [py.path.local | str]{.title-ref} to [os.PathLike[str] | str]{.title-ref}.</p>
<p>Most plugins which refer to [reportinfo()]{.title-ref} only define it as part of a custom <code>pytest.Item</code>{.interpreted-text role=&quot;class&quot;} implementation.
Since [py.path.local]{.title-ref} is a [os.PathLike[str]]{.title-ref}, these plugins are unaffacted.</p>
<p>Plugins and users which call [reportinfo()]{.title-ref}, use the first return value and interact with it as a [py.path.local]{.title-ref}, would need to adjust by calling [py.path.local(fspath)]{.title-ref}.
Although preferably, avoid the legacy [py.path.local]{.title-ref} and use [pathlib.Path]{.title-ref}, or use [item.location]{.title-ref} or [item.path]{.title-ref}, instead.</p>
<p>Note: pytest was not able to provide a deprecation period for this change.</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pytest-dev/pytest/commit/3554b833c24f541af5c7272e47cfa52b89d8138d""><code>3554b83</code></a> Add note to changelog</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/6ea7f990a040cacfbc671a8c29f2d76533d9d523""><code>6ea7f99</code></a> Prepare release version 7.0.0</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/737b220516870b3f90e87109c2c6657856dca506""><code>737b220</code></a> [7.0.x] releasing: Add template for major releases (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9597"">#9597</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/7fa397296339773ee7a94df8684573fe7bdf7072""><code>7fa3972</code></a> [7.0.x] releasing: Always set doc_version (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9590"">#9590</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/b304499925c4ce39a6a45dcc3394d5d0f0aa6b94""><code>b304499</code></a> [7.0.x] Make 'warnings' and 'deselected' in assert_outcomes optional (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9566"">#9566</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/f17525df26e65b8de304dad919f66aaeb61f1f5c""><code>f17525d</code></a> [7.0.x] doc: Add ellipsis to warning usecase list (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9562"">#9562</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/0a7be971d2b37907099e870095e49a31dc25d194""><code>0a7be97</code></a> ci: Bump up timeout (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9565"">#9565</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/c17908cdb3566508d8ac896552ac7cec092be24d""><code>c17908c</code></a> [7.0.x] doc: Recategorize 7.0.0 changelog items (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9564"">#9564</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/ab549bba8a9ee7f55f3f9a2dc884f47cca34f6c4""><code>ab549bb</code></a> [7.0.x] Add missing cooperative constructor changelog (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9563"">#9563</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest/commit/4b1707ff7041256381b95b5647ee6cc8336b54d2""><code>4b1707f</code></a> [7.0.x] Autouse linearization graph (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest/issues/9557"">#9557</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/pytest-dev/pytest/compare/5.2.2...7.0.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1629/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
279547886,MDU6SXNzdWUyNzk1NDc4ODY=,163,Document the querystring argument for setting a different time limit,9599,simonw,closed,0,,,,,2,2017-12-05T22:05:08Z,2021-03-23T02:44:33Z,2017-12-06T15:06:57Z,OWNER,,"http://datasette.readthedocs.io/en/latest/sql_queries.html#query-limits

Need to explain why this is useful too.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/163/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1125576543,I_kwDOBm6k_c5DFu9f,1630,Review datasette.utils and decide which functions should be documented for 1.0,9599,simonw,open,0,,,3268330,Datasette 1.0,0,2022-02-07T06:39:52Z,2022-02-07T06:39:52Z,,OWNER,,"Follows:
- #1176",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1630/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1125973221,PR_kwDOBm6k_c4yK44E,1631,"Update pytest-asyncio requirement from <0.17,>=0.10 to >=0.10,<0.19",49699333,dependabot[bot],closed,0,,,,,1,2022-02-07T13:13:19Z,2022-03-06T01:29:54Z,2022-03-06T01:29:53Z,CONTRIBUTOR,simonw/datasette/pulls/1631,"Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/pytest-dev/pytest-asyncio/releases"">pytest-asyncio's releases</a>.</em></p>
<blockquote>
<h2>pytest-asyncio 0.18.0</h2>
<hr />
<h2>title: 'pytest-asyncio: pytest support for asyncio'</h2>
<p><a href=""https://pypi.python.org/pypi/pytest-asyncio""><img src=""https://img.shields.io/pypi/v/pytest-asyncio.svg"" alt=""image"" /></a></p>
<p><a href=""https://github.com/pytest-dev/pytest-asyncio/actions?workflow=CI""><img src=""https://github.com/pytest-dev/pytest-asyncio/workflows/CI/badge.svg"" alt=""image"" /></a></p>
<p><a href=""https://codecov.io/gh/pytest-dev/pytest-asyncio""><img src=""https://codecov.io/gh/pytest-dev/pytest-asyncio/branch/master/graph/badge.svg"" alt=""image"" /></a></p>
<p><a href=""https://github.com/pytest-dev/pytest-asyncio""><img src=""https://img.shields.io/pypi/pyversions/pytest-asyncio.svg"" alt=""Supported Python versions"" /></a></p>
<p><a href=""https://github.com/ambv/black""><img src=""https://img.shields.io/badge/code%20style-black-000000.svg"" alt=""image"" /></a></p>
<p>pytest-asyncio is an Apache2 licensed library, written in Python, for
testing asyncio code with pytest.</p>
<p>asyncio code is usually written in the form of coroutines, which makes
it slightly more difficult to test using normal testing tools.
pytest-asyncio provides useful fixtures and markers to make testing
easier.</p>
<pre lang=""{.sourceCode"" data-meta="".python}""><code>@pytest.mark.asyncio
async def test_some_asyncio_code():
    res = await library.do_something()
    assert b&quot;expected result&quot; == res
</code></pre>
<p>pytest-asyncio has been strongly influenced by
<a href=""https://github.com/eugeniy/pytest-tornado"">pytest-tornado</a>.</p>
<h1>Features</h1>
<ul>
<li>fixtures for creating and injecting versions of the asyncio event
loop</li>
<li>fixtures for injecting unused tcp/udp ports</li>
<li>pytest markers for treating tests as asyncio coroutines</li>
<li>easy testing with non-default event loops</li>
<li>support for [async def]{.title-ref} fixtures and async generator
fixtures</li>
<li>support <em>auto</em> mode to handle all async fixtures and tests
automatically by asyncio; provide <em>strict</em> mode if a test suite
should work with different async frameworks simultaneously, e.g.
<code>asyncio</code> and <code>trio</code>.</li>
</ul>
<h1>Installation</h1>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/07e9922f1dc9cb84f0948e6c84ad9921c3662969""><code>07e9922</code></a> Prepare release of v0.18.0 (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/282"">#282</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/a92d8947ab57c7d88082dff9dcded686622ed47b""><code>a92d894</code></a> Relax dependency on typing-extensions (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/284"">#284</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/e78b59d5d8f8fed7d3aaf39c184a7d4a337676de""><code>e78b59d</code></a> Fix setuptools_scm configuration (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/279"">#279</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/d8efa640f0aa1ba8856b908ba486150588018209""><code>d8efa64</code></a> Support parametrized event_loop fixture (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/278"">#278</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/dab3b5184c0bdd0bd93dddeb6968de7f565fb8db""><code>dab3b51</code></a> Raise a warning if <a href=""https://github.com/pytest""><code>@​pytest</code></a>.mark.asyncio is applied to non-async function (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/275"">#275</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/048a6edc56e7d2c4a5f1283b5d2dd93a1270bfce""><code>048a6ed</code></a> Code cleanup: drop 'request' argument from async fixtures (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/270"">#270</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/435332735305417413c04ecf29056243137d1261""><code>4353327</code></a> Update changelog</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/c919fd7be35b7f10ba47c953a336baa73cfa8653""><code>c919fd7</code></a> Remove unconditional test item attribute access (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/268"">#268</a>)</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/39c6d0ce88868d8398606ea10541befa7d7241cd""><code>39c6d0c</code></a> Update README</li>
<li><a href=""https://github.com/pytest-dev/pytest-asyncio/commit/547933dabaa56b3d9c244c8302252f3b638463d2""><code>547933d</code></a> Require typing-extensions on py&lt;3.8 only (<a href=""https://github-redirect.dependabot.com/pytest-dev/pytest-asyncio/issues/269"">#269</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/pytest-dev/pytest-asyncio/compare/v0.10.0...v0.18.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>",107914493,datasette,pull,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1631/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,
1126604194,I_kwDOBm6k_c5DJp2i,1632,"datasette one.db one.db opens database twice, as one and one_2",9599,simonw,closed,0,,,3268330,Datasette 1.0,6,2022-02-07T23:14:47Z,2022-03-19T04:04:49Z,2022-02-07T23:50:01Z,OWNER,,"> ```
> % mkdir /tmp/data
> % cp ~/Dropbox/Development/datasette/fixtures.db /tmp/data 
> % datasette /tmp/data/*.db /tmp/data/created.db --create -p 8852
> ...
> INFO:     Uvicorn running on http://127.0.0.1:8852 (Press CTRL+C to quit)
> ^CINFO:     Shutting down
> % datasette /tmp/data/*.db /tmp/data/created.db --create -p 8852
> ...
> INFO:     127.0.0.1:49533 - ""GET / HTTP/1.1"" 200 OK
> ```
> The first time I ran Datasette I got two databases - `fixtures` and `created`
> 
> BUT... when I ran Datasette the second time it looked like this:
> 
> <img width=""697"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/152887735-5e2b4f21-001d-4e7a-a23a-16a32fdcf554.png"">
> 
> This is the same result you get if you run:
> 
>     datasette /tmp/data/fixtures.db /tmp/data/created.db /tmp/data/created.db
> 
> This is caused by this Datasette issue:
> - https://github.com/simonw/datasette/issues/509
> 
> So... either I teach Datasette to de-duplicate multiple identical file paths passed to the command, or I can't use `/data/*.db` in the `Dockerfile` here and I need to go back to other solutions for the challenge described in this comment: https://github.com/simonw/datasette-publish-fly/pull/12#issuecomment-1031971831

_Originally posted by @simonw in https://github.com/simonw/datasette-publish-fly/pull/12#issuecomment-1032029874_",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1632/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1129052172,I_kwDOBm6k_c5DS_gM,1633,base_url or prefix does not work with _exact match,6613091,henrikek,open,0,,,,,2,2022-02-09T21:45:07Z,2022-04-28T09:12:56Z,,NONE,,"When i hit ""Apply"" button to search with ""_exact"" for a column syntax the URL prefix is removed from the url.

![image](https://user-images.githubusercontent.com/6613091/153293758-0b757d55-5757-4987-992e-9426e69a7956.png)

And the result is:
![image](https://user-images.githubusercontent.com/6613091/153294672-87be7809-bb7b-455d-bf1a-41e90bbfa4ae.png)

If I add the marked row to url_builder.py it seams to work:
![image](https://user-images.githubusercontent.com/6613091/153295231-bdd52e37-efcf-4b21-9d37-69f182a922f4.png)
",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1633/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1131295060,I_kwDOBm6k_c5DbjFU,1634,Update Dockerfile generated by `datasette publish`,9599,simonw,open,0,,,3268330,Datasette 1.0,4,2022-02-11T00:07:26Z,2022-03-11T17:38:08Z,,OWNER,,"The generated `Dockerfile` currently looks something like this:
```Dockerfile
FROM python:3.8
COPY . /app
WORKDIR /app

ENV DATASETTE_SECRET 'edab49cbc5d5f6f33238f54852037e3fee710821960b73edd2ce743454182ae2'
RUN pip install -U datasette datasette-auth-passwords datasette-tiddlywiki datasette-graphql
RUN datasette inspect fixtures.db other.db --inspect-file inspect-data.json
ENV PORT 8080
EXPOSE 8080
CMD datasette serve --host 0.0.0.0 -i fixtures.db -i other.db --cors --inspect-file inspect-data.json --metadata metadata.json --create --port $PORT /data/*.db
```
This is still on Python 3.8, and it generates a pretty large image compared to the `Dockerfile` used for https://hub.docker.com/datasetteproject/datasette - https://github.com/simonw/datasette/blob/0.60.2/Dockerfile

Here's the code that generates it: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/utils/__init__.py#L389-L400",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1634/reactions"", ""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 2, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1138008042,I_kwDOBm6k_c5D1J_q,1636,"""permissions"" propery in metadata for configuring arbitrary permissions",9599,simonw,closed,0,,,8711695, Datasette 1.0a2,14,2022-02-15T00:25:59Z,2022-12-13T02:40:50Z,2022-12-13T02:40:50Z,OWNER,,"The `""allow""` block mechanism can already be used to configure various default permissions. When adding permissions to `datasette-tiddlywiki` I realized it would be good to be able to configure arbitrary permissions such as `edit-tiddlywiki` there too.",107914493,datasette,issue,,,"{""url"": ""https://api.github.com/repos/simonw/datasette/issues/1636/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed