Allow block
@@ -55,4 +55,82 @@ p.message-warning {
{% if result == ""False"" %}
Result: deny
{% endif %}
+
Test permission check
+
+
This tool lets you simulate an actor and a permission check for that actor.
+
+
+
+
+
+
{% endblock %}
diff --git a/datasette/views/special.py b/datasette/views/special.py
index 9922a621..d46fc280 100644
--- a/datasette/views/special.py
+++ b/datasette/views/special.py
@@ -1,6 +1,8 @@
import json
+from datasette.permissions import PERMISSIONS
from datasette.utils.asgi import Response, Forbidden
from datasette.utils import actor_matches_allow, add_cors_headers
+from datasette.permissions import PERMISSIONS
from .base import BaseView
import secrets
import time
@@ -138,9 +140,34 @@ class AllowDebugView(BaseView):
""error"": ""\n\n"".join(errors) if errors else """",
""actor_input"": actor_input,
""allow_input"": allow_input,
+ ""permissions"": PERMISSIONS,
},
)
+ async def post(self, request):
+ vars = await request.post_vars()
+ actor = json.loads(vars[""actor""])
+ permission = vars[""permission""]
+ resource_1 = vars[""resource_1""]
+ resource_2 = vars[""resource_2""]
+ resource = []
+ if resource_1:
+ resource.append(resource_1)
+ if resource_2:
+ resource.append(resource_2)
+ resource = tuple(resource)
+ result = await self.ds.permission_allowed(
+ actor, permission, resource, default=""USE_DEFAULT""
+ )
+ return Response.json(
+ {
+ ""actor"": actor,
+ ""permission"": permission,
+ ""resource"": resource,
+ ""result"": result,
+ }
+ )
+
class MessagesDebugView(BaseView):
name = ""messages_debug""
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors,
https://github.com/simonw/datasette/issues/1855#issuecomment-1301594495,https://api.github.com/repos/simonw/datasette/issues/1855,1301594495,IC_kwDOBm6k_c5NlMF_,9599,simonw,2022-11-03T03:11:17Z,2022-11-03T03:11:17Z,OWNER,"Maybe the way to do this is through a new standard mechanism on the actor: a set of additional restrictions, e.g.:
```
{
""id"": ""root"",
""_r"": {
""a"": [""ir"", ""ur"", ""dr""],
""d"": {
""fixtures"": [""ir"", ""ur"", ""dr""]
},
""t"": {
""fixtures"": {
""searchable"": [""ir""]
}
}
}
```
`""a""` is ""all permissions"" - these apply to everything.
`""d""` permissions only apply to the specified database
`""t""` permissions only apply to the specified table
The way this works is there's a default [permission_allowed(datasette, actor, action, resource)](https://docs.datasette.io/en/stable/plugin_hooks.html#id25) hook which only consults these, and crucially just says NO if those rules do not match.
In this way it would apply as an extra layer of permission rules over the defaults (which for this `root` instance would all return yes).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions,
https://github.com/simonw/datasette/issues/1880#issuecomment-1301043042,https://api.github.com/repos/simonw/datasette/issues/1880,1301043042,IC_kwDOBm6k_c5NjFdi,525934,amitkoth,2022-11-02T18:20:14Z,2022-11-02T18:20:14Z,NONE,"Follow on question - is all memory use @simonw - for both datasette and SQLlite confined to the ""query time"" itself i.e. the memory use is relevant only to a particular transaction or query - and then subsequently released?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1433576351,Datasette with many and large databases > Memory use,
https://github.com/simonw/datasette/issues/1871#issuecomment-1299607082,https://api.github.com/repos/simonw/datasette/issues/1871,1299607082,IC_kwDOBm6k_c5Ndm4q,9599,simonw,2022-11-02T05:45:31Z,2022-11-02T05:45:31Z,OWNER,"I'm going to add a link to the Datasette API docs for the current running version of Datasette, e.g. to https://docs.datasette.io/en/0.63/json_api.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool,
https://github.com/simonw/datasette/issues/1871#issuecomment-1299600257,https://api.github.com/repos/simonw/datasette/issues/1871,1299600257,IC_kwDOBm6k_c5NdlOB,9599,simonw,2022-11-02T05:36:40Z,2022-11-02T05:36:40Z,OWNER,"The API Explorer should definitely link to the `/-/create-token` page for users who have permission though.
And it should probably go in the Datasette application menu?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool,
https://github.com/simonw/datasette/issues/1871#issuecomment-1299599461,https://api.github.com/repos/simonw/datasette/issues/1871,1299599461,IC_kwDOBm6k_c5NdlBl,9599,simonw,2022-11-02T05:35:36Z,2022-11-02T05:36:15Z,OWNER,"Here's a slightly wild idea: what if there was a button on `/-/api` that you could click to turn on ""API explorer mode"" for the rest of the Datasette interface - which sets a cookie, and that cookie means you then see ""API explorer"" links in all sorts of other relevant places in the Datasette UI (maybe tucked away in cog menus).
Only reason I don't want to show these to everyone is that I don't think this is a very user-friendly feature: if you don't know what an API is I don't want to expose you to it unnecessarily.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool,
https://github.com/simonw/datasette/issues/1871#issuecomment-1299598570,https://api.github.com/repos/simonw/datasette/issues/1871,1299598570,IC_kwDOBm6k_c5Ndkzq,9599,simonw,2022-11-02T05:34:28Z,2022-11-02T05:34:28Z,OWNER,"This is pretty useful now. Two features I still want to add:
- The ability to link to the API explorer such that the form is pre-filled with material from the URL. Need to guard against clickjacking first though, so no-one can link to it in an invisible iframe and trick the user into hitting POST.
- Some kind of list of endpoints so people can click links to start using the API explorer. A list of every table the user can write to with each of their `/db/table/-/insert` endpoints for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool,
https://github.com/simonw/datasette/issues/1871#issuecomment-1299597066,https://api.github.com/repos/simonw/datasette/issues/1871,1299597066,IC_kwDOBm6k_c5NdkcK,9599,simonw,2022-11-02T05:32:22Z,2022-11-02T05:32:22Z,OWNER,"Demo of the latest API explorer:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool,
https://github.com/simonw/datasette/issues/1871#issuecomment-1299388341,https://api.github.com/repos/simonw/datasette/issues/1871,1299388341,IC_kwDOBm6k_c5Ncxe1,9599,simonw,2022-11-02T00:24:28Z,2022-11-02T00:25:00Z,OWNER,"I want JSON syntax highlighting.
https://github.com/luyilin/json-format-highlight is an MIT licensed tiny highlighter that looks decent for this.
https://unpkg.com/json-format-highlight@1.0.1/dist/json-format-highlight.js","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool,
https://github.com/simonw/datasette/issues/1871#issuecomment-1299349741,https://api.github.com/repos/simonw/datasette/issues/1871,1299349741,IC_kwDOBm6k_c5NcoDt,9599,simonw,2022-11-01T23:22:55Z,2022-11-01T23:22:55Z,OWNER,"It's weird that the API explorer only lets you explore POST APIs. It should probably also let you explore GET APIs, or be renamed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool,
https://github.com/simonw/datasette/issues/1879#issuecomment-1299098458,https://api.github.com/repos/simonw/datasette/issues/1879,1299098458,IC_kwDOBm6k_c5Nbqta,9599,simonw,2022-11-01T20:27:40Z,2022-11-01T20:33:52Z,OWNER,"https://github.com/simonw/datasette-x-forwarded-host/blob/main/datasette_x_forwarded_host/__init__.py could happen in core controlled by:
`--setting trust_forwarded_host 1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems,
https://github.com/simonw/datasette/issues/1879#issuecomment-1299102108,https://api.github.com/repos/simonw/datasette/issues/1879,1299102108,IC_kwDOBm6k_c5Nbrmc,9599,simonw,2022-11-01T20:30:54Z,2022-11-01T20:33:06Z,OWNER,One idea: add a `/-/debug` page (or `/-/tips` or `/-/checks`) which shows the incoming requests headers and could even detect if there's an `x-forwarded-host` header that isn't being repeated and show a tip on how to fix that.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems,
https://github.com/simonw/datasette/issues/1879#issuecomment-1299102755,https://api.github.com/repos/simonw/datasette/issues/1879,1299102755,IC_kwDOBm6k_c5Nbrwj,9599,simonw,2022-11-01T20:31:37Z,2022-11-01T20:31:37Z,OWNER,And some JavaScript that can spot if Datasette thinks it is being served over HTTP when it's actually being served over HTTPS.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems,
https://github.com/simonw/datasette/issues/1879#issuecomment-1299096850,https://api.github.com/repos/simonw/datasette/issues/1879,1299096850,IC_kwDOBm6k_c5NbqUS,9599,simonw,2022-11-01T20:26:12Z,2022-11-01T20:26:12Z,OWNER,"The other relevant plugin here is https://datasette.io/plugins/datasette-x-forwarded-host
Maybe that should be rolled into core too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems,
https://github.com/simonw/datasette/issues/1879#issuecomment-1299090678,https://api.github.com/repos/simonw/datasette/issues/1879,1299090678,IC_kwDOBm6k_c5Nboz2,9599,simonw,2022-11-01T20:20:28Z,2022-11-01T20:20:28Z,OWNER,My first step in debugging these is to install https://datasette.io/plugins/datasette-debug-asgi - but now I'm thinking maybe something like that should be part of core.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems,
https://github.com/simonw/datasette/issues/1862#issuecomment-1299073433,https://api.github.com/repos/simonw/datasette/issues/1862,1299073433,IC_kwDOBm6k_c5NbkmZ,9599,simonw,2022-11-01T20:04:31Z,2022-11-01T20:04:31Z,OWNER,"It really feels like this should be accompanied by a `/db/-/create` API for creating tables. I had to add that to `sqlite-utils` eventually (initially it only supported creating by passing in an example document):
https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425011030,"Create a new table from one or more records, `sqlite-utils` style",
https://github.com/simonw/datasette/issues/1878#issuecomment-1299071456,https://api.github.com/repos/simonw/datasette/issues/1878,1299071456,IC_kwDOBm6k_c5NbkHg,9599,simonw,2022-11-01T20:02:43Z,2022-11-01T20:02:43Z,OWNER,"Note that ""update"" is partially covered by the `replace` option to `/-/insert`, added here:
- https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API,
https://github.com/simonw/datasette/issues/1873#issuecomment-1298919552,https://api.github.com/repos/simonw/datasette/issues/1873,1298919552,IC_kwDOBm6k_c5Na_CA,9599,simonw,2022-11-01T18:11:27Z,2022-11-01T18:11:27Z,OWNER,"I forgot to document `ignore` and `replace`. Also I need to add tests that cover:
- Forgetting to include a primary key on a non-autoincrement table
- Compound primary keys
- Rowid only tables with and without rowid specified
I think my validation logic here will get caught out by the fact that `rowid` does not show up as a valid column name: https://github.com/simonw/datasette/blob/9bec7c38eb93cde5afb16df9bdd96aea2a5b0459/datasette/views/table.py#L1151-L1160
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables,
https://github.com/simonw/datasette/issues/1873#issuecomment-1298905135,https://api.github.com/repos/simonw/datasette/issues/1873,1298905135,IC_kwDOBm6k_c5Na7gv,9599,simonw,2022-11-01T17:59:59Z,2022-11-01T17:59:59Z,OWNER,"It's a bit surprising that you can send `""ignore"": true, ""return_rows"": true` and the returned `""inserted""` key will list rows that were NOT inserted (since they were ignored).
Three options:
1. Ignore that and document it
2. Fix it so `""inserted""` only returns rows that were actually inserted (bit tricky)
3. Change the name of `""inserted""` to something else
I'm picking 3 - I'm going to change it to be called `""rows""` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables,
https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451,https://api.github.com/repos/simonw/datasette/issues/1873,1298885451,IC_kwDOBm6k_c5Na2tL,9599,simonw,2022-11-01T17:42:20Z,2022-11-01T17:42:20Z,OWNER,"Design decision:
```json
{
""rows"": [{""id"": 1, ""title"": ""The title""}],
""ignore"": true
}
```
Or `""replace"": true`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables,
https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298879701,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1298879701,IC_kwDOCGYnMM5Na1TV,9599,simonw,2022-11-01T17:37:13Z,2022-11-01T17:37:13Z,OWNER,"The question I was originally trying to answer here was this: how many rows were actually inserted by that call to `.insert_all()`?
I don't know that `.rowcount` would ever be useful here, since the ""correct"" answer depends on other factors - had I determined to ignore or replace records with a primary key that matches an existing record for example?
So I think if people need `rowcount` they can get it by using a `cursor` directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix),
https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298877872,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1298877872,IC_kwDOCGYnMM5Na02w,9599,simonw,2022-11-01T17:35:30Z,2022-11-01T17:35:30Z,OWNER,"This may not make sense.
First, `.last_rowid` is a property on table - but that doesn't make sense for `rowcount` since it should clearly be a property on the database itself (you can run a query directly using `db.execute()` without going through a `Table` object).
So I tried this prototype:
```diff
diff --git a/docs/python-api.rst b/docs/python-api.rst
index 206e5e6..78d3a8d 100644
--- a/docs/python-api.rst
+++ b/docs/python-api.rst
@@ -186,6 +186,15 @@ The ``db.query(sql)`` function executes a SQL query and returns an iterator over
# {'name': 'Cleo'}
# {'name': 'Pancakes'}
+After executing a query the ``db.rowcount`` property on that database instance will reflect the number of rows affected by any insert, update or delete operations performed by that query:
+
+.. code-block:: python
+
+ db = Database(memory=True)
+ db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}])
+ print(db.rowcount)
+ # Outputs: 2
+
.. _python_api_execute:
db.execute(sql, params)
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index a06f4b7..c19c2dd 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -294,6 +294,8 @@ class Database:
_counts_table_name = ""_counts""
use_counts_table = False
+ # Number of rows inserted, updated or deleted
+ rowcount: Optional[int] = None
def __init__(
self,
@@ -480,9 +482,11 @@ class Database:
if self._tracer:
self._tracer(sql, parameters)
if parameters is not None:
- return self.conn.execute(sql, parameters)
+ cursor = self.conn.execute(sql, parameters)
else:
- return self.conn.execute(sql)
+ cursor = self.conn.execute(sql)
+ self.rowcount = cursor.rowcount
+ return cursor
def executescript(self, sql: str) -> sqlite3.Cursor:
""""""
```
But this happens:
```pycon
>>> from sqlite_utils import Database
>>> db = Database(memory=True)
>>> db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}])
>>> db.rowcount
-1
```
Turning on query tracing demonstrates why:
```pycon
>>> db = Database(memory=True, tracer=print)
PRAGMA recursive_triggers=on; None
>>> db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}])
select name from sqlite_master where type = 'view' None
select name from sqlite_master where type = 'table' None
select name from sqlite_master where type = 'view' None
CREATE TABLE [dogs] (
[name] TEXT
);
None
select name from sqlite_master where type = 'view' None
INSERT INTO [dogs] ([name]) VALUES (?), (?); ['Cleo', 'Pancakes']
select name from sqlite_master where type = 'table' None
select name from sqlite_master where type = 'table' None
PRAGMA table_info([dogs]) None
>>>
```
The `.insert_all()` function does a bunch of other queries too, so `.rowcount` is quickly over-ridden by the same result from extra queries that it executed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix),
https://github.com/simonw/datasette/issues/1876#issuecomment-1298856054,https://api.github.com/repos/simonw/datasette/issues/1876,1298856054,IC_kwDOBm6k_c5Navh2,9599,simonw,2022-11-01T17:16:01Z,2022-11-01T17:16:01Z,OWNER,`ta.style.height = ta.scrollHeight + 'px'` is an easy way to do that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1431786951,SQL query should wrap on SQL interrupted screen,
https://github.com/simonw/datasette/issues/1876#issuecomment-1298854321,https://api.github.com/repos/simonw/datasette/issues/1876,1298854321,IC_kwDOBm6k_c5NavGx,9599,simonw,2022-11-01T17:14:33Z,2022-11-01T17:14:33Z,OWNER,"I could use a `textarea` here (would need to figure out a neat pattern to expand it to fit the query):
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #508 +/- ##
==========================================
- Coverage 96.25% 96.09% -0.16%
==========================================
Files 4 4
Lines 2401 2407 +6
==========================================
+ Hits 2311 2313 +2
- Misses 90 94 +4
```
| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/508/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `96.79% <63.63%> (-0.30%)` | :arrow_down: |
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #1870 +/- ##
=======================================
Coverage 92.55% 92.55%
=======================================
Files 35 35
Lines 4432 4432
=======================================
Hits 4102 4102
Misses 330 330
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1870/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.30% <ø> (ø)` | |
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
Or:
datasette create-token --expire-after 10m","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423369494,datasette create-token CLI command,
https://github.com/simonw/datasette/issues/1858#issuecomment-1291435464,https://api.github.com/repos/simonw/datasette/issues/1858,1291435464,IC_kwDOBm6k_c5M-b3I,9599,simonw,2022-10-26T03:07:16Z,2022-10-26T03:07:16Z,OWNER,"This setting will disable the ""Token never expires"" option:
`
I think because this is text inside a `:ref:` block, not regular text.
Consider the following:
The \--convert and the :ref:`insert \--convert ` and
It's rendered like this:
>>> conn = _
>>> for f in psutil.Process().open_files(): print(f)
...
popenfile(path='/private/tmp/blah.db', fd=3)
>>> conn.close()
>>> for f in psutil.Process().open_files(): print(f)
...
>>>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11,
https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291111357,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291111357,IC_kwDOCGYnMM5M9Mu9,9599,simonw,2022-10-25T20:36:06Z,2022-10-25T20:36:06Z,OWNER,... or maybe Windows doesn't like attempts to remove a file that the process has opened?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11,
https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291103021,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291103021,IC_kwDOCGYnMM5M9Kst,9599,simonw,2022-10-25T20:32:01Z,2022-10-25T20:32:01Z,OWNER,"This test reliably fails on Windows with Python 3.11.
I'm going to skip the test for the moment to get back to green CI... but I'll leave this issue open. This is definitely concerning, I just don't have the right local environment to solve this at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11,
https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291093581,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291093581,IC_kwDOCGYnMM5M9IZN,9599,simonw,2022-10-25T20:23:00Z,2022-10-25T20:23:00Z,OWNER,"I'm not hugely happy with my fix there:
https://github.com/simonw/sqlite-utils/blob/c5d7ec1dd71fa1dce829bc8bb82b639018befd63/sqlite_utils/db.py#L321-L328
The problem here was that in the case where the `os.remove()` failed the `self.conn` property was NOT being set to a valid connection - which caused `__repr__` to fail later on.
So now I catch the `os.remove()` error, set `self.conn` to a memory connection, then raise the error again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11,
https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291088108,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291088108,IC_kwDOCGYnMM5M9HDs,9599,simonw,2022-10-25T20:17:36Z,2022-10-25T20:17:36Z,OWNER,"Now `mypy` is failing:
```
sqlite_utils/db.py:474: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute""
sqlite_utils/db.py:476: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute""
sqlite_utils/db.py:486: error: Item ""None"" of ""Optional[Any]"" has no attribute ""executescript""
sqlite_utils/db.py:603: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__enter__""
sqlite_utils/db.py:603: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__exit__""
sqlite_utils/db.py:604: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute""
sqlite_utils/db.py:607: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute""
sqlite_utils/db.py:1082: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__enter__""
sqlite_utils/db.py:1082: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__exit__""
sqlite_utils/db.py:1083: error: Item ""None"" of ""Optional[Any]"" has no attribute ""cursor""
sqlite_utils/db.py:1155: error: Item ""None"" of ""Optional[Any]"" has no attribute ""enable_load_extension""
sqlite_utils/db.py:1156: error: Item ""None"" of ""Optional[Any]"" has no attribute ""load_extension""
Found 12 errors in 1 file (checked 51 source files)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11,
https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291083188,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291083188,IC_kwDOCGYnMM5M9F20,9599,simonw,2022-10-25T20:12:52Z,2022-10-25T20:12:52Z,OWNER,"Failed again, but just noticed this: https://github.com/simonw/sqlite-utils/actions/runs/3323932266/jobs/5494890223
```
> Database(filepath, recreate=True)[""t2""].insert({""foo"": ""bar""})
tests\test_recreate.py:31:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <[AttributeError(""'Database' object has no attribute 'conn'"") raised in repr()] Database object at 0x29fc125aa90>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11,
https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291076031,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291076031,IC_kwDOCGYnMM5M9EG_,9599,simonw,2022-10-25T20:06:28Z,2022-10-25T20:06:28Z,OWNER,"This is the failing test: https://github.com/simonw/sqlite-utils/blob/7b2d1c0ffd0b874e280292b926f328a61cb31e2c/tests/test_recreate.py#L21-L32
I'm going to try a different way of creating the temporary file: https://docs.pytest.org/en/7.1.x/how-to/tmp_path.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11,
https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291071627,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291071627,IC_kwDOCGYnMM5M9DCL,9599,simonw,2022-10-25T20:02:18Z,2022-10-25T20:02:18Z,OWNER,Passes on Windows with other Python versions for some reason.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11,
https://github.com/simonw/datasette/issues/1854#issuecomment-1291047214,https://api.github.com/repos/simonw/datasette/issues/1854,1291047214,IC_kwDOBm6k_c5M89Eu,9599,simonw,2022-10-25T19:39:36Z,2022-10-25T19:39:48Z,OWNER,"This pattern should work (for the http server at least):
```python
# Loop until port 8041 serves traffic
while True:
try:
httpx.get(""http://localhost:8041/"")
break
except httpx.ConnectError:
time.sleep(0.1)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422973111,Flaky test: test_serve_localhost_http,
https://github.com/simonw/datasette/issues/1854#issuecomment-1291046958,https://api.github.com/repos/simonw/datasette/issues/1854,1291046958,IC_kwDOBm6k_c5M89Au,9599,simonw,2022-10-25T19:39:22Z,2022-10-25T19:39:22Z,OWNER,"Here's the code that starts those various servers: https://github.com/simonw/datasette/blob/613ad05c095f92653221db267ef53d54d00cdfbb/tests/conftest.py#L104-L177
I don't like those `time.sleep(1.5)` lines much - I'm going to try polling for readiness instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422973111,Flaky test: test_serve_localhost_http,
https://github.com/simonw/datasette/issues/1854#issuecomment-1291045997,https://api.github.com/repos/simonw/datasette/issues/1854,1291045997,IC_kwDOBm6k_c5M88xt,9599,simonw,2022-10-25T19:38:28Z,2022-10-25T19:38:28Z,OWNER,"Also:
```
@pytest.mark.serial
@pytest.mark.skipif(
not hasattr(socket, ""AF_UNIX""), reason=""Requires socket.AF_UNIX support""
)
def test_serve_unix_domain_socket(ds_unix_domain_socket_server):
_, uds = ds_unix_domain_socket_server
transport = httpx.HTTPTransport(uds=uds)
client = httpx.Client(transport=transport)
> response = client.get(""http://localhost/_memory.json"")
/home/runner/work/datasette/datasette/tests/test_cli_serve_server.py:35:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:1039: in get
return self.request(
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:815: in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:902: in send
response = self._send_handling_auth(
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:930: in _send_handling_auth
response = self._send_handling_redirects(
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:967: in _send_handling_redirects
response = self._send_single_request(request)
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:1003: in _send_single_request
response = transport.handle_request(request)
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_transports/default.py:217: in handle_request
with map_httpcore_exceptions():
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/contextlib.py:153: in __exit__
self.gen.throw(typ, value, traceback)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
@contextlib.contextmanager
def map_httpcore_exceptions() -> typing.Iterator[None]:
try:
yield
except Exception as exc: # noqa: PIE-786
mapped_exc = None
for from_exc, to_exc in HTTPCORE_EXC_MAP.items():
if not isinstance(exc, from_exc):
continue
# We want to map to the most specific exception we can find.
# Eg if `exc` is an `httpcore.ReadTimeout`, we want to map to
# `httpx.ReadTimeout`, not just `httpx.TimeoutException`.
if mapped_exc is None or issubclass(to_exc, mapped_exc):
mapped_exc = to_exc
if mapped_exc is None: # pragma: nocover
raise
message = str(exc)
> raise mapped_exc(message) from exc
E httpx.ConnectError: [Errno 2] No such file or directory
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422973111,Flaky test: test_serve_localhost_http,
https://github.com/simonw/datasette/issues/1853#issuecomment-1291036623,https://api.github.com/repos/simonw/datasette/issues/1853,1291036623,IC_kwDOBm6k_c5M86fP,9599,simonw,2022-10-25T19:28:56Z,2022-10-25T19:28:56Z,OWNER,"Opened an issue here:
- https://github.com/coleifer/pysqlite3/issues/43","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11,
https://github.com/simonw/datasette/issues/1853#issuecomment-1291032289,https://api.github.com/repos/simonw/datasette/issues/1853,1291032289,IC_kwDOBm6k_c5M85bh,9599,simonw,2022-10-25T19:24:27Z,2022-10-25T19:24:27Z,OWNER,https://latest.datasette.io/-/versions now shows 3.11.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11,
https://github.com/simonw/sqlite-utils/issues/502#issuecomment-1291029761,https://api.github.com/repos/simonw/sqlite-utils/issues/502,1291029761,IC_kwDOCGYnMM5M840B,9599,simonw,2022-10-25T19:21:44Z,2022-10-25T19:21:44Z,OWNER,"Replicated locally using a fresh virtual environment with Python 3.11 and:
pytest -k test_query_invalid_function","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422954582,Fix tests for Python 3.11,
https://github.com/simonw/datasette/issues/1853#issuecomment-1291023926,https://api.github.com/repos/simonw/datasette/issues/1853,1291023926,IC_kwDOBm6k_c5M83Y2,9599,simonw,2022-10-25T19:15:49Z,2022-10-25T19:15:49Z,OWNER,"This broke the deploy of `https://latest.datasette.io/` - because it tries to install `pysqlite3-binary` which doesn't have a 3.11 release yet:
https://github.com/simonw/datasette/blob/2e9751672d4fe329b3c359d5b7b1992283185820/.github/workflows/deploy-latest.yml#L77
I started using that for the `latest.datasette.io` demo in https://github.com/simonw/datasette/commit/a970276b9999687b96c5e11ea1c817d814f5d267 because I wanted a version of SQLite that supported generated columns.
Those were added in [SQLite 3.31.0](https://www.sqlite.org/changes.html#version_3_31_0) - and the SQLite version in the new base image is 3.34.1 - so I don't actually need `pysqlite3-binary` any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11,
https://github.com/simonw/datasette/issues/1853#issuecomment-1291012637,https://api.github.com/repos/simonw/datasette/issues/1853,1291012637,IC_kwDOBm6k_c5M80od,9599,simonw,2022-10-25T19:04:03Z,2022-10-25T19:04:09Z,OWNER,"And tested `datasette package` like this:
```
datasette package fixtures.db -t datasette-package-python-upgrade-3-11
```
Then:
```
docker run -p 8081:8001 datasette-package-python-upgrade-3-11
```
And tested it like this:
```
curl http://localhost:8081/-/versions.json | jq
```
Output:
```
{
""python"": {
""version"": ""3.11.0"",
""full"": ""3.11.0 (main, Oct 25 2022, 05:00:36) [GCC 10.2.1 20210110]""
},
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11,
https://github.com/simonw/datasette/issues/1853#issuecomment-1291009987,https://api.github.com/repos/simonw/datasette/issues/1853,1291009987,IC_kwDOBm6k_c5M8z_D,9599,simonw,2022-10-25T19:01:23Z,2022-10-25T19:01:23Z,OWNER,"Also tested by running this locally:
datasette publish cloudrun fixtures.db --service issue-1853
https://issue-1853-j7hipcg4aq-uc.a.run.app/-/versions now shows Python 3.11.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11,
https://github.com/simonw/datasette/issues/1853#issuecomment-1291006149,https://api.github.com/repos/simonw/datasette/issues/1853,1291006149,IC_kwDOBm6k_c5M8zDF,9599,simonw,2022-10-25T18:57:33Z,2022-10-25T18:57:33Z,OWNER,"Ran the upgrade on the Datasette Cloud image first, works fine there.
https://simon.datasette.cloud/-/versions shows me:
```
{
""python"": {
""version"": ""3.11.0"",
""full"": ""3.11.0 (main, Oct 25 2022, 05:00:36) [GCC 10.2.1 20210110]""
},
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11,
https://github.com/simonw/datasette/issues/1853#issuecomment-1290995178,https://api.github.com/repos/simonw/datasette/issues/1853,1290995178,IC_kwDOBm6k_c5M8wXq,9599,simonw,2022-10-25T18:46:33Z,2022-10-25T18:46:33Z,OWNER,"I ran a very crude benchmark on my laptop using Locust (against the official macOS packages from www.python.org for Python 3.10 and Python 3.11) and saw a substantial speed increase:
533.89 requests/second on 3.11
413.56 requests/second on 3.10
That was from running Locust against this `locustfile.py`:
```python
from locust import HttpUser, task
class CounterOne(HttpUser):
@task
def hello(self):
self.client.get(""/-/static/app.css"")
```
Using:
locust --headless --users 4 --spawn-rate 4 -H http://127.0.0.1:8001","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11,
https://github.com/simonw/datasette/issues/1851#issuecomment-1290615599,https://api.github.com/repos/simonw/datasette/issues/1851,1290615599,IC_kwDOBm6k_c5M7Tsv,25778,eyeseast,2022-10-25T14:05:12Z,2022-10-25T14:05:12Z,CONTRIBUTOR,"This could use a new plugin hook, too. I don't want to complicate your life too much, but for things like GIS, I'd want a way to turn regular JSON into SpatiaLite geometries or combine X/Y coordinates into point geometries and such. Happy to help however I can.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table,
https://github.com/simonw/datasette/issues/1851#issuecomment-1289865317,https://api.github.com/repos/simonw/datasette/issues/1851,1289865317,IC_kwDOBm6k_c5M4chl,9599,simonw,2022-10-25T01:42:47Z,2022-10-25T01:42:47Z,OWNER,"This is going to tie into Datasette's existing permissions mechanism, so plugins will be able to define their own custom mechanisms for tokens to be attached to a specific identity: https://docs.datasette.io/en/stable/authentication.html
There's only one plugin for API tokens at the moment, which is this one: https://datasette.io/plugins/datasette-auth-tokens
I'm actually planning on adding another, default token mechanism to Datasette itself as part of this work:
- #1852
It may well be that `datasette-sandstorm-support` needs to add something custom here too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table,
https://github.com/simonw/datasette/issues/1852#issuecomment-1289776707,https://api.github.com/repos/simonw/datasette/issues/1852,1289776707,IC_kwDOBm6k_c5M4G5D,9599,simonw,2022-10-24T23:29:03Z,2022-10-24T23:29:03Z,OWNER,"I'm going to implement the first version of this token mechanism using permissions that exist already. Right now that's:
https://docs.datasette.io/en/0.62/authentication.html#built-in-permissions
Here are the shortcuts I'll use for them:
- `view-instance` - `vi`
- `view-database` - `vd`
- `view-database-download` - `vdd`
- `view-table` - `vt`
- `view-query` - `vq`
- `execute-sql` - `es`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism,
https://github.com/simonw/datasette/issues/1852#issuecomment-1289775162,https://api.github.com/repos/simonw/datasette/issues/1852,1289775162,IC_kwDOBm6k_c5M4Gg6,9599,simonw,2022-10-24T23:27:00Z,2022-10-24T23:27:00Z,OWNER,"Might be neat for API tokens to be signed with an additional secret than can be rotated independently of `DATASETTE_SECRET` itself, in order to invalidate all tokens without needing to invalidate logged in users too.
But again, I don't want to implement something like that until I see an actual need for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism,
https://github.com/simonw/datasette/issues/1852#issuecomment-1289774183,https://api.github.com/repos/simonw/datasette/issues/1852,1289774183,IC_kwDOBm6k_c5M4GRn,9599,simonw,2022-10-24T23:25:52Z,2022-10-24T23:25:52Z,OWNER,"... also, maybe there should be a UI (perhaps on that page) for resetting the Datasette secret? Useful for emergency invalidation of all tokens.
No, I'm not going to build that unless someone asks for it. Restarting the server with a fresh secret should be easy enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism,
https://github.com/simonw/datasette/issues/1852#issuecomment-1289773634,https://api.github.com/repos/simonw/datasette/issues/1852,1289773634,IC_kwDOBm6k_c5M4GJC,9599,simonw,2022-10-24T23:25:06Z,2022-10-24T23:25:06Z,OWNER,"If you start Datasette without providing a `DATASETTE_SECRET` environment variable of `--secret` option it creates a random signing secret that only lasts for the lifetime of the server.
This means any signed API tokens you create will stop working if the server restarts.
I think the `/-/create-token` UI should know when this happens and show a warning message about it, to avoid confusion.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism,
https://github.com/simonw/datasette/issues/1852#issuecomment-1289766513,https://api.github.com/repos/simonw/datasette/issues/1852,1289766513,IC_kwDOBm6k_c5M4EZx,9599,simonw,2022-10-24T23:16:00Z,2022-10-24T23:16:00Z,OWNER,"Here's what that example looks like signed:
```python
from datasette.app import Datasette
ds = Datasette()
ds.sign('{""t"":{""a"":[""ir"",""ur"",""dr""],""d"":{""fixtures"":[""ir"",""ur"",""dr""]},""t"":{""fixtures"":{""searchable"":[""ir""]}}}}')
```
```
.eJxTqo5RKolRsgJSiUAqOkYpsyhGSSdGqRRCpQCpWBANUZOWWVFSWpRajFNprQ7cPCS1QF5xamJRckZiUk4qQm9sLRAoAQCC8yph.O0Gaej6-VOLbbtPq7xU6T77jEO0
```
That's 129 characters.
Note that Datasette doesn't have its own mechanism for signing things for a specific duration yet: https://docs.datasette.io/en/stable/internals.html#sign-value-namespace-default
So I'll need to add a `""e"": 1666739744` field with the UTC timestamp at which the token should expire.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism,
https://github.com/simonw/datasette/issues/1852#issuecomment-1289733483,https://api.github.com/repos/simonw/datasette/issues/1852,1289733483,IC_kwDOBm6k_c5M38Vr,9599,simonw,2022-10-24T22:54:37Z,2022-10-24T23:12:10Z,OWNER,"Token design concept:
```json
{
""t"": {
""a"": [""ir"", ""ur"", ""dr""],
""d"": {
""fixtures"": [""ir"", ""ur"", ""dr""]
},
""t"": {
""fixtures"": {
""searchable"": [""ir""]
}
}
}
}
```
That JSON would be minified and signed.
Minified version of the above looks like this (101 characters):
`{""t"":{""a"":[""ir"",""ur"",""dr""],""d"":{""fixtures"":[""ir"",""ur"",""dr""]},""t"":{""fixtures"":{""searchable"":[""ir""]}}}}`
The `""t""` key shows this is a token that as a default API key.
`""a""` means ""all"" - these are permissions that have been granted on all tables and databases.
`""d""` means ""databases"" - this is a way to set permissions for all tables in a specific database.
`""t""` means ""tables"" - this lets you set permissions at a finely grained table level.
Then the permissions themselves are two character codes which are shortened versions - so:
- `ir` = `insert-row`
- `ur` = `update-row`
- `dr` = `delete-row`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism,
https://github.com/simonw/datasette/issues/1851#issuecomment-1289752130,https://api.github.com/repos/simonw/datasette/issues/1851,1289752130,IC_kwDOBm6k_c5M4A5C,4399499,ocdtrekkie,2022-10-24T23:07:30Z,2022-10-24T23:07:30Z,NONE,"How are you tying the bearer token to identity? I'm excited to see this feature, and since Sandstorm controls API access using the same header, it also will transparently support the API documentation here, but we strip the bearer before the request reaches the app (replacing it with our existing auth headers, of course).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table,
https://github.com/simonw/datasette/issues/1852#issuecomment-1289718660,https://api.github.com/repos/simonw/datasette/issues/1852,1289718660,IC_kwDOBm6k_c5M34uE,9599,simonw,2022-10-24T22:35:01Z,2022-10-24T22:35:01Z,OWNER,"Maybe these tokens can be restricted to specific databases and tables when they are first created?
Since they're signed tokens, I could bundle a bunch of extra stuff in them - this token is allowed to do these permissions against these tables/rows for example.
General wisdom seems to be that 8KB is a sensible maximum length for this kind of token, which is easily long enough to fit in a bunch of database / table / permissions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism,
https://github.com/simonw/datasette/issues/1851#issuecomment-1289713513,https://api.github.com/repos/simonw/datasette/issues/1851,1289713513,IC_kwDOBm6k_c5M33dp,9599,simonw,2022-10-24T22:29:58Z,2022-10-24T22:30:15Z,OWNER,"Interesting open question: how should validation errors (if any) be returned?
The two forms of validation I can think of at first are:
- Missing keys which are marked as `not null` in the schema
- Keys that do not match to existing columns (if you didn't pass `""alter"": true`, an option I am considering)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table,
https://github.com/simonw/datasette/issues/1850#issuecomment-1289707357,https://api.github.com/repos/simonw/datasette/issues/1850,1289707357,IC_kwDOBm6k_c5M319d,9599,simonw,2022-10-24T22:23:12Z,2022-10-24T22:23:12Z,OWNER,Here's the implementation of `datasette-auth-tokens`: https://github.com/simonw/datasette-auth-tokens/blob/main/datasette_auth_tokens/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core,
https://github.com/simonw/datasette/issues/1850#issuecomment-1289706439,https://api.github.com/repos/simonw/datasette/issues/1850,1289706439,IC_kwDOBm6k_c5M31vH,9599,simonw,2022-10-24T22:22:17Z,2022-10-24T22:22:17Z,OWNER,"API authentication will be via `Authorization: Bearer XXX` request headers.
I'm inclined to add a default token mechanism to Datasette based on tokens that are signed with the `DATASETTE_SECRET`. Maybe the root user can access `/-/create-token` which provides a UI for generating a time-limited signed token? Could also have a `datasette create-token` command for creating such tokens at the command-line.
Plugins can then define alternative ways of creating tokens, such as the existing https://datasette.io/plugins/datasette-auth-tokens plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core,
https://github.com/simonw/datasette/issues/1850#issuecomment-1289703432,https://api.github.com/repos/simonw/datasette/issues/1850,1289703432,IC_kwDOBm6k_c5M31AI,9599,simonw,2022-10-24T22:19:48Z,2022-10-24T22:19:48Z,OWNER,It may turn out that it makes sense to also add a UI for these actions as part of this project. That's still to be determined.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core,
https://github.com/simonw/datasette/issues/1850#issuecomment-1289702146,https://api.github.com/repos/simonw/datasette/issues/1850,1289702146,IC_kwDOBm6k_c5M30sC,9599,simonw,2022-10-24T22:19:04Z,2022-10-24T22:19:04Z,OWNER,"This is going to need a whole bunch of new permissions.
To review: the existing set of permissions are listed here: https://docs.datasette.io/en/0.62/authentication.html#built-in-permissions
- `view-instance`
- `view-database`
- `view-database-download`
- `view-table`
- `view-query`
- `execute-sql`
- `permissions-debug`
- `debug-menu`
I'm going to reuse database terminology for the new permissions. So first draft of those is:
- `insert-row`
- `update-row`
- `delete-row`
- `create-table`
- `drop-table`
- `alter-table`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core,
https://github.com/simonw/datasette/issues/1850#issuecomment-1289696171,https://api.github.com/repos/simonw/datasette/issues/1850,1289696171,IC_kwDOBm6k_c5M3zOr,9599,simonw,2022-10-24T22:15:57Z,2022-10-24T22:15:57Z,OWNER,"I'm going to treat this as a bit of a research spike, at least until I like the direction it is going enough to commit to it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core,
https://github.com/simonw/datasette/issues/1849#issuecomment-1288384907,https://api.github.com/repos/simonw/datasette/issues/1849,1288384907,IC_kwDOBm6k_c5MyzGL,9599,simonw,2022-10-24T04:04:02Z,2022-10-24T04:04:02Z,OWNER,"Refs:
- #1831 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor',
https://github.com/simonw/datasette/issues/1849#issuecomment-1288384098,https://api.github.com/repos/simonw/datasette/issues/1849,1288384098,IC_kwDOBm6k_c5Myy5i,9599,simonw,2022-10-24T04:03:09Z,2022-10-24T04:03:09Z,OWNER,"Looks like the new breadcrumbs code can't handle the case where `request` is `None`.
Need a test that demonstrates this too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor',
https://github.com/simonw/datasette/issues/1848#issuecomment-1288340476,https://api.github.com/repos/simonw/datasette/issues/1848,1288340476,IC_kwDOBm6k_c5MyoP8,9599,simonw,2022-10-24T02:50:29Z,2022-10-24T02:50:29Z,OWNER,"https://latest.datasette.io/_internal now looks like this:
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #1842 +/- ##
==========================================
+ Coverage 92.52% 92.54% +0.02%
==========================================
Files 35 35
Lines 4415 4428 +13
==========================================
+ Hits 4085 4098 +13
Misses 330 330
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1842?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.28% <100.00%> (+0.05%)` | :arrow_up: |
| [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.29% <100.00%> (+0.06%)` | :arrow_up: |
| [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <100.00%> (ø)` | |
| [datasette/views/row.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3Jvdy5weQ==) | `88.70% <100.00%> (+0.37%)` | :arrow_up: |
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.20% <100.00%> (+0.01%)` | :arrow_up: |
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
{% for item in items %}
{{ item.label }}
{% if not loop.last %}
/
{% endif %}
{% endfor %}
{% endif %}
{%- endmacro %}
```
This means custom template authors can use their own `_crumbs.html` template to do something different with the breadcrumbs.
In the actual templates I display breadcrumbs like this:
```html+jinja
{% block crumbs %}
{{ crumbs.nav(request=request, database=database) }}
{% endblock %}
```
Pass `database=` to get `home / database_name` - pass `table=` as well to get `home / database_name / table_name` - if you just send `request=` you just get `home`.
I've also made the default base template show the `home` breadcrumbs - other pages such as `table.html` and `row.html` can then over-ride `{% block crumbs %}` to get different breadcrumbs.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display,
https://github.com/simonw/datasette/issues/1831#issuecomment-1278175798,https://api.github.com/repos/simonw/datasette/issues/1831,1278175798,IC_kwDOBm6k_c5ML2o2,9599,simonw,2022-10-13T21:02:52Z,2022-10-13T21:02:52Z,OWNER,This patch to `default_permissions.py` made debugging easier: https://gist.github.com/simonw/daddf022e75a98ea6246ac1e12dc8759,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display,
https://github.com/simonw/datasette/issues/1831#issuecomment-1276882359,https://api.github.com/repos/simonw/datasette/issues/1831,1276882359,IC_kwDOBm6k_c5MG623,9599,simonw,2022-10-13T00:36:09Z,2022-10-13T00:36:19Z,OWNER,"It's important that, however this works, it supports custom templates changing how the breadcrumbs are displayed. Probably needs a `_crumbs.html` template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display,
https://github.com/simonw/datasette/issues/1831#issuecomment-1276856836,https://api.github.com/repos/simonw/datasette/issues/1831,1276856836,IC_kwDOBm6k_c5MG0oE,9599,simonw,2022-10-12T23:57:28Z,2022-10-12T23:57:28Z,OWNER,"As part of this I think I want `request` to always be available in the template context, which will remove the need for https://datasette.io/plugins/datasette-template-request","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display,
https://github.com/simonw/datasette/issues/1831#issuecomment-1276706181,https://api.github.com/repos/simonw/datasette/issues/1831,1276706181,IC_kwDOBm6k_c5MGP2F,9599,simonw,2022-10-12T20:34:00Z,2022-10-12T23:43:39Z,OWNER,"Maybe new template functions: `table_crumbs(table, database)` and `database_crumbs(database)` and `instance_crumbs()` - which know how to both check the permissions and display the `` full of links.
Or just `crumbs(table=table, database=database)` where both of those are optional. Maybe accept `actor=` or `request=` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display,
https://github.com/simonw/datasette/issues/1831#issuecomment-1276848084,https://api.github.com/repos/simonw/datasette/issues/1831,1276848084,IC_kwDOBm6k_c5MGyfU,9599,simonw,2022-10-12T23:39:02Z,2022-10-12T23:39:02Z,OWNER,"Made some notes on the Jinja documentation while thinking about the best way to do this:
- https://github.com/simonw/public-notes/issues/4","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display,
https://github.com/simonw/datasette/issues/1831#issuecomment-1276696138,https://api.github.com/repos/simonw/datasette/issues/1831,1276696138,IC_kwDOBm6k_c5MGNZK,9599,simonw,2022-10-12T20:25:39Z,2022-10-12T20:25:39Z,OWNER,"Related issue:
- #1829
I'd like a code design for this that helps solve both these issues at once.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display,
https://github.com/simonw/datasette/issues/1831#issuecomment-1276694152,https://api.github.com/repos/simonw/datasette/issues/1831,1276694152,IC_kwDOBm6k_c5MGM6I,9599,simonw,2022-10-12T20:23:44Z,2022-10-12T20:23:44Z,OWNER,The fact that this code is inconsistent already (`urls.instance()` v.s. `base_url` for example) suggests that properly refactoring this is well overdue already.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display,
https://github.com/simonw/datasette/issues/1831#issuecomment-1276693475,https://api.github.com/repos/simonw/datasette/issues/1831,1276693475,IC_kwDOBm6k_c5MGMvj,9599,simonw,2022-10-12T20:23:01Z,2022-10-12T20:23:01Z,OWNER,"Here are all the places that display that navigation bar at the moment:
https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/row.html#L18-L25
https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/table.html#L19-L25
https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/query.html#L21-L27
https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/error.html#L5-L10
https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/database.html#L12-L17
https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/show_json.html#L7-L12
https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/permissions_debug.html#L25-L30
https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/permissions_debug.html#L25-L30t
Interesting to note that failing to include that `super()` call means the nav menu itself will not be rendered:
https://github.com/simonw/datasette/blob/b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9/datasette/templates/base.html#L20-L49","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display,
https://github.com/simonw/sqlite-utils/pull/499#issuecomment-1275248761,https://api.github.com/repos/simonw/sqlite-utils/issues/499,1275248761,IC_kwDOCGYnMM5MAsB5,22429695,codecov[bot],2022-10-11T20:38:13Z,2022-10-11T20:38:13Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/499?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Base: **96.37**% // Head: **96.25**% // Decreases project coverage by **`-0.11%`** :warning:
> Coverage data is based on head [(`a25e838`)](https://codecov.io/gh/simonw/sqlite-utils/pull/499?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`d792dad`)](https://codecov.io/gh/simonw/sqlite-utils/commit/d792dad1cf5f16525da81b1e162fb71d469995f3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> Patch coverage: 40.00% of modified lines in pull request are covered.
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #499 +/- ##
==========================================
- Coverage 96.37% 96.25% -0.12%
==========================================
Files 4 4
Lines 2401 2406 +5
==========================================
+ Hits 2314 2316 +2
- Misses 87 90 +3
```
| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/499?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.09% <40.00%> (-0.23%)` | :arrow_down: |
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #498 +/- ##
=======================================
Coverage 96.37% 96.37%
=======================================
Files 4 4
Lines 2401 2402 +1
=======================================
+ Hits 2314 2315 +1
Misses 87 87
```
| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/498?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/498/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.32% <100.00%> (+<0.01%)` | :arrow_up: |
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #1820 +/- ##
==========================================
+ Coverage 92.50% 92.51% +0.01%
==========================================
Files 35 35
Lines 4400 4406 +6
==========================================
+ Hits 4070 4076 +6
Misses 330 330
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.11% <ø> (ø)` | |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `94.80% <100.00%> (+0.05%)` | :arrow_up: |
| [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.29% <100.00%> (+0.06%)` | :arrow_up: |
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
` elements that link to off-domain pages and adds `target=""_blank""` to them via the DOM.
That way people who like `target=""_blank""` can have it!
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400494162,Open Datasette link in new tab,
https://github.com/simonw/datasette/issues/1836#issuecomment-1271008997,https://api.github.com/repos/simonw/datasette/issues/1836,1271008997,IC_kwDOBm6k_c5Lwg7l,536941,fgregg,2022-10-07T02:00:37Z,2022-10-07T02:00:49Z,CONTRIBUTOR,"yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in `immutable` mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!
running a test of that now. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1271006020,https://api.github.com/repos/simonw/datasette/issues/1836,1271006020,IC_kwDOBm6k_c5LwgNE,9599,simonw,2022-10-07T01:54:07Z,2022-10-07T01:54:07Z,OWNER,Just overlapped with your comment here: https://github.com/simonw/datasette/issues/1836#issuecomment-1271003212 - which notes that opening with `?immutable=1` DOES seem to cause the file to be duplicated!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1271004167,https://api.github.com/repos/simonw/datasette/issues/1836,1271004167,IC_kwDOBm6k_c5LwfwH,9599,simonw,2022-10-07T01:53:05Z,2022-10-07T01:53:05Z,OWNER,"Oh this is interesting!
Is your hunch here that running this line is causing the file to be stored as a second layer?
https://github.com/simonw/datasette/blob/5aa359b86907d11b3ee601510775a85a90224da8/datasette/utils/__init__.py#L399
I guess it's possible that running a non-read-only query against the database causes one or two bytes to be changed (maybe a transaction ID or similar?)
Modifying the `inspect` command to use `?mode=ro` seems sensible to me. Except.... it should already be opening those files in immutable mode according to this line: https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/cli.py#L172-L173
Here's what opening as a `immutables` does:
https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/app.py#L258-L260
https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/database.py#L98
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1271003212,https://api.github.com/repos/simonw/datasette/issues/1836,1271003212,IC_kwDOBm6k_c5LwfhM,536941,fgregg,2022-10-07T01:52:04Z,2022-10-07T01:52:04Z,CONTRIBUTOR,"and if we try immutable mode, which is how things are opened by `datasette inspect` we duplicate the files!!!
```python
# test_sql_immutable.py
import sqlite3
import sys
db_name = sys.argv[1]
conn = sqlite3.connect(f'file:/app/{db_name}?immutable=1', uri=True)
cur = conn.cursor()
cur.execute('select count(*) from filing')
print(cur.fetchone())
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1270992795,https://api.github.com/repos/simonw/datasette/issues/1836,1270992795,IC_kwDOBm6k_c5Lwc-b,536941,fgregg,2022-10-07T01:29:15Z,2022-10-07T01:50:14Z,CONTRIBUTOR,"fascinatingly, telling python to open sqlite in read only mode makes this layer have a size of 0
```python
# test_sql_ro.py
import sqlite3
import sys
db_name = sys.argv[1]
conn = sqlite3.connect(f'file:/app/{db_name}?mode=ro', uri=True)
cur = conn.cursor()
cur.execute('select count(*) from filing')
print(cur.fetchone())
```
that's quite weird because setting the file permissions to read only didn't do anything. (on reflection, that chmod isn't doing anything because the dockerfile commands are run as root)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1270988081,https://api.github.com/repos/simonw/datasette/issues/1836,1270988081,IC_kwDOBm6k_c5Lwb0x,536941,fgregg,2022-10-07T01:19:01Z,2022-10-07T01:27:35Z,CONTRIBUTOR,"okay, some progress!! running some sql against a database file causes that file to get duplicated even if it doesn't apparently change the file.
make a little test script like this:
```python
# test_sql.py
import sqlite3
import sys
db_name = sys.argv[1]
conn = sqlite3.connect(f'file:/app/{db_name}', uri=True)
cur = conn.cursor()
cur.execute('select count(*) from filing')
print(cur.fetchone())
```
then
```docker
RUN python test_sql.py nlrb.db
```
produced a layer that's the same size as `nlrb.db`!!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1270936982,https://api.github.com/repos/simonw/datasette/issues/1836,1270936982,IC_kwDOBm6k_c5LwPWW,536941,fgregg,2022-10-07T00:52:41Z,2022-10-07T00:52:41Z,CONTRIBUTOR,"it's not that the inspect command is somehow changing the db files. if i set them to only read-only, the ""inspect"" layer still has the same very large size.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1270923537,https://api.github.com/repos/simonw/datasette/issues/1836,1270923537,IC_kwDOBm6k_c5LwMER,536941,fgregg,2022-10-07T00:46:08Z,2022-10-07T00:46:08Z,CONTRIBUTOR,"i thought it was maybe to do with reading through all the files, but that does not seem to be the case
if i make a little test file like:
```python
# test_read.py
import hashlib
import sys
import pathlib
HASH_BLOCK_SIZE = 1024 * 1024
def inspect_hash(path):
""""""Calculate the hash of a database, efficiently.""""""
m = hashlib.sha256()
with path.open(""rb"") as fp:
while True:
data = fp.read(HASH_BLOCK_SIZE)
if not data:
break
m.update(data)
return m.hexdigest()
inspect_hash(pathlib.Path(sys.argv[1]))
```
then a line in the Dockerfile like
```docker
RUN python test_read.py nlrb.db && echo ""[]"" > /etc/inspect.json
```
just produes a layer of `3B`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/pull/1837#issuecomment-1270855853,https://api.github.com/repos/simonw/datasette/issues/1837,1270855853,IC_kwDOBm6k_c5Lv7it,22429695,codecov[bot],2022-10-07T00:01:20Z,2022-10-07T00:01:20Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1837?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Base: **92.50**% // Head: **92.50**% // No change to project coverage :thumbsup:
> Coverage data is based on head [(`c12447e`)](https://codecov.io/gh/simonw/datasette/pull/1837?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`eff1124`)](https://codecov.io/gh/simonw/datasette/commit/eff112498ecc499323c26612d707908831446d25?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> Patch has no changes to coverable lines.
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #1837 +/- ##
=======================================
Coverage 92.50% 92.50%
=======================================
Files 35 35
Lines 4400 4400
=======================================
Hits 4070 4070
Misses 330 330
```
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #1835 +/- ##
==========================================
+ Coverage 91.71% 92.50% +0.78%
==========================================
Files 38 35 -3
Lines 4754 4400 -354
==========================================
- Hits 4360 4070 -290
+ Misses 394 330 -64
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1835?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1835/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | | |
| [datasette/utils/shutil\_backport.py](https://codecov.io/gh/simonw/datasette/pull/1835/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3NodXRpbF9iYWNrcG9ydC5weQ==) | | |
| [datasette/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1835/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL19faW5pdF9fLnB5) | | |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1835/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `94.75% <0.00%> (+0.01%)` | :arrow_up: |
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #1823 +/- ##
==========================================
+ Coverage 91.58% 92.50% +0.91%
==========================================
Files 36 35 -1
Lines 4444 4400 -44
==========================================
Hits 4070 4070
+ Misses 374 330 -44
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1823?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1823/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `91.06% <88.23%> (ø)` | |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1823/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.11% <100.00%> (ø)` | |
| [datasette/utils/shutil\_backport.py](https://codecov.io/gh/simonw/datasette/pull/1823/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3NodXRpbF9iYWNrcG9ydC5weQ==) | | |
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
)
E + where = .rows
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/xk/github/xk/lb/xklb/check.py(11)test_transaction()
9 with db1.conn:
10 db1[""t""].insert({""foo"": 1})
---> 11 assert list(db2[""t""].rows) == []
12 assert list(db2[""t""].rows) == [{""foo"": 1}]
```
It fails because it is already inserted.
btw if you put these two lines in you pyproject.toml you can get `ipdb` in pytest
```
[tool.pytest.ini_options]
addopts = ""--pdbcls=IPython.terminal.debugger:TerminalPdb --ignore=tests/data --capture=tee-sys --log-cli-level=ERROR""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149661489,`with db:` for transactions,
https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1264219650,https://api.github.com/repos/simonw/sqlite-utils/issues/493,1264219650,IC_kwDOCGYnMM5LWnYC,7908073,chapmanjacobd,2022-10-01T03:22:50Z,2022-10-01T03:23:58Z,CONTRIBUTOR,"this is likely what you are looking for: https://stackoverflow.com/a/51076749/697964
but yeah I would say just disable smart quotes","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386562662,Tiny typographical error in install/uninstall docs,
https://github.com/simonw/datasette/pull/1827#issuecomment-1263570186,https://api.github.com/repos/simonw/datasette/issues/1827,1263570186,IC_kwDOBm6k_c5LUI0K,22429695,codecov[bot],2022-09-30T13:22:15Z,2022-09-30T13:22:15Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1827?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Base: **92.50**% // Head: **92.50**% // No change to project coverage :thumbsup:
> Coverage data is based on head [(`1f0c557`)](https://codecov.io/gh/simonw/datasette/pull/1827?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`34defdc`)](https://codecov.io/gh/simonw/datasette/commit/34defdc10aa293294ca01cfab70780755447e1d7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> Patch has no changes to coverable lines.
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #1827 +/- ##
=======================================
Coverage 92.50% 92.50%
=======================================
Files 35 35
Lines 4400 4400
=======================================
Hits 4070 4070
Misses 330 330
```
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
,
> or unsubscribe
>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1388631785,render_cell documentation example doesn't match the method signature,
https://github.com/simonw/datasette/pull/1825#issuecomment-1260368537,https://api.github.com/repos/simonw/datasette/issues/1825,1260368537,IC_kwDOBm6k_c5LH7KZ,9599,simonw,2022-09-28T04:21:18Z,2022-09-28T04:21:18Z,OWNER,"This is great, thank you very much!
https://datasette--1825.org.readthedocs.build/en/1825/deploying.html#running-datasette-using-openrc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1388227245,Add documentation for serving via OpenRC,
https://github.com/simonw/datasette/pull/1825#issuecomment-1260368122,https://api.github.com/repos/simonw/datasette/issues/1825,1260368122,IC_kwDOBm6k_c5LH7D6,22429695,codecov[bot],2022-09-28T04:20:28Z,2022-09-28T04:20:28Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1825?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Base: **91.58**% // Head: **91.58**% // No change to project coverage :thumbsup:
> Coverage data is based on head [(`b16eb2f`)](https://codecov.io/gh/simonw/datasette/pull/1825?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`5f9f567`)](https://codecov.io/gh/simonw/datasette/commit/5f9f567acbc58c9fcd88af440e68034510fb5d2b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> Patch has no changes to coverable lines.
> :exclamation: Current head b16eb2f differs from pull request most recent head e7e96dc. Consider uploading reports for the commit e7e96dc to get more accurate results
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #1825 +/- ##
=======================================
Coverage 91.58% 91.58%
=======================================
Files 36 36
Lines 4444 4444
=======================================
Hits 4070 4070
Misses 374 374
```
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
>>> (await ds.client.get(""/.json"")).json()
{'_memory': {'name': '_memory', 'hash': None, 'color': 'a6c7b9', 'path': '/_memory', 'tables_and_views_truncated': [], 'tab
les_and_views_more': False, 'tables_count': 0, 'table_rows_sum': 0, 'show_table_row_counts': False, 'hidden_table_rows_sum'
: 0, 'hidden_tables_count': 0, 'views_count': 0, 'private': False}}
>>>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide,
https://github.com/simonw/datasette/issues/1733#issuecomment-1115318417,https://api.github.com/repos/simonw/datasette/issues/1733,1115318417,IC_kwDOBm6k_c5CemiR,9599,simonw,2022-05-02T20:13:43Z,2022-05-02T20:13:43Z,OWNER,This is good enough to push an alpha.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide,
https://github.com/simonw/datasette/issues/1733#issuecomment-1115318303,https://api.github.com/repos/simonw/datasette/issues/1733,1115318303,IC_kwDOBm6k_c5Cemgf,9599,simonw,2022-05-02T20:13:36Z,2022-05-02T20:13:36Z,OWNER,"I got a build from the `pyodide` branch to work!
```
Welcome to the Pyodide terminal emulator 🐍
Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM
Type ""help"", ""copyright"", ""credits"" or ""license"" for more information.
>>> import micropip
>>> await micropip.install(""https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl"")
Traceback (most recent call last):
File """", line 1, in
File ""/lib/python3.10/asyncio/futures.py"", line 284, in __await__
yield self # This tells Task to wait for completion.
File ""/lib/python3.10/asyncio/tasks.py"", line 304, in __wakeup
future.result()
File ""/lib/python3.10/asyncio/futures.py"", line 201, in result
raise self._exception
File ""/lib/python3.10/asyncio/tasks.py"", line 234, in __step
result = coro.throw(exc)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 183, in install
transaction = await self.gather_requirements(requirements, ctx, keep_going)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 173, in gather_requirements
await gather(*requirement_promises)
File ""/lib/python3.10/asyncio/futures.py"", line 284, in __await__
yield self # This tells Task to wait for completion.
File ""/lib/python3.10/asyncio/tasks.py"", line 304, in __wakeup
future.result()
File ""/lib/python3.10/asyncio/futures.py"", line 201, in result
raise self._exception
File ""/lib/python3.10/asyncio/tasks.py"", line 232, in __step
result = coro.send(None)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 245, in add_requirement
await self.add_wheel(name, wheel, version, (), ctx, transaction)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 316, in add_wheel
await self.add_requirement(recurs_req, ctx, transaction)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 291, in add_requirement
await self.add_wheel(
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 316, in add_wheel
await self.add_requirement(recurs_req, ctx, transaction)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 291, in add_requirement
await self.add_wheel(
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 316, in add_wheel
await self.add_requirement(recurs_req, ctx, transaction)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 276, in add_requirement
raise ValueError(
ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed
>>> await micropip.install(""https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl"")
Traceback (most recent call last):
File """", line 1, in
File ""/lib/python3.10/asyncio/futures.py"", line 284, in __await__
yield self # This tells Task to wait for completion.
File ""/lib/python3.10/asyncio/tasks.py"", line 304, in __wakeup
future.result()
File ""/lib/python3.10/asyncio/futures.py"", line 201, in result
raise self._exception
File ""/lib/python3.10/asyncio/tasks.py"", line 234, in __step
result = coro.throw(exc)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 183, in install
transaction = await self.gather_requirements(requirements, ctx, keep_going)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 173, in gather_requirements
await gather(*requirement_promises)
File ""/lib/python3.10/asyncio/futures.py"", line 284, in __await__
yield self # This tells Task to wait for completion.
File ""/lib/python3.10/asyncio/tasks.py"", line 304, in __wakeup
future.result()
File ""/lib/python3.10/asyncio/futures.py"", line 201, in result
raise self._exception
File ""/lib/python3.10/asyncio/tasks.py"", line 232, in __step
result = coro.send(None)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 245, in add_requirement
await self.add_wheel(name, wheel, version, (), ctx, transaction)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 316, in add_wheel
await self.add_requirement(recurs_req, ctx, transaction)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 291, in add_requirement
await self.add_wheel(
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 316, in add_wheel
await self.add_requirement(recurs_req, ctx, transaction)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 291, in add_requirement
await self.add_wheel(
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 316, in add_wheel
await self.add_requirement(recurs_req, ctx, transaction)
File ""/lib/python3.10/site-packages/micropip/_micropip.py"", line 276, in add_requirement
raise ValueError(
ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed
>>> await micropip.install(""h11==0.12"")
>>> await micropip.install(""https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl"")
>>> import datasette
>>> from datasette.app import Datasette
Traceback (most recent call last):
File """", line 1, in
File ""/lib/python3.10/site-packages/datasette/app.py"", line 9, in
import httpx
File ""/lib/python3.10/site-packages/httpx/__init__.py"", line 2, in
from ._api import delete, get, head, options, patch, post, put, request, stream
File ""/lib/python3.10/site-packages/httpx/_api.py"", line 4, in
from ._client import Client
File ""/lib/python3.10/site-packages/httpx/_client.py"", line 9, in
from ._auth import Auth, BasicAuth, FunctionAuth
File ""/lib/python3.10/site-packages/httpx/_auth.py"", line 10, in
from ._models import Request, Response
File ""/lib/python3.10/site-packages/httpx/_models.py"", line 16, in
from ._content import ByteStream, UnattachedStream, encode_request, encode_response
File ""/lib/python3.10/site-packages/httpx/_content.py"", line 17, in
from ._multipart import MultipartStream
File ""/lib/python3.10/site-packages/httpx/_multipart.py"", line 7, in
from ._types import (
File ""/lib/python3.10/site-packages/httpx/_types.py"", line 5, in
import ssl
File ""/lib/python3.10/ssl.py"", line 98, in
import _ssl # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'
>>> import ssl
>>> from datasette.app import Datasette
Traceback (most recent call last):
File """", line 1, in
File ""/lib/python3.10/site-packages/datasette/app.py"", line 14, in
import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
>>> import setuptools
>>> from datasette.app import Datasette
>>> ds = Datasette(memory=True)
>>> ds
>>> await ds.client.get(""/"")
Traceback (most recent call last):
File ""/lib/python3.10/site-packages/datasette/app.py"", line 1268, in route_path
response = await view(request, send)
File ""/lib/python3.10/site-packages/datasette/views/base.py"", line 134, in view
return await self.dispatch_request(request)
File ""/lib/python3.10/site-packages/datasette/views/base.py"", line 89, in dispatch_request
await self.ds.refresh_schemas()
File ""/lib/python3.10/site-packages/datasette/app.py"", line 353, in refresh_schemas
await self._refresh_schemas()
File ""/lib/python3.10/site-packages/datasette/app.py"", line 358, in _refresh_schemas
await init_internal_db(internal_db)
File ""/lib/python3.10/site-packages/datasette/utils/internal_db.py"", line 65, in init_internal_db
await db.execute_write_script(create_tables_sql)
File ""/lib/python3.10/site-packages/datasette/database.py"", line 116, in execute_write_script
results = await self.execute_write_fn(_inner, block=block)
File ""/lib/python3.10/site-packages/datasette/database.py"", line 155, in execute_write_fn
self._write_thread.start()
File ""/lib/python3.10/threading.py"", line 928, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
>>> ds = Datasette(memory=True, settings={""num_sql_threads"": 0})
>>> await ds.client.get(""/"")
>>> (await ds.client.get(""/"")).text
'\n\n\n Datasette: _memory \n
\n
\n\n\n\n \n\n\n\n
\n\nDatasette \n\n\n\n\n\n
r detailsClickedWithin = null;\n while (target && target.tagName != \'DETAILS\') {\n target = target.parentNode;\
n }\n if (target && target.tagName == \'DETAILS\') {\n detailsClickedWithin = target;\n }\n Array.from(d
ocument.getElementsByTagName(\'details\')).filter(\n (details) => details.open && details != detailsClickedWithin\n
).forEach(details => details.open = false);\n});\n\n\n\n\n\n\n
'
>>>
```
That `ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed` error is annoying. I assume it's a `uvicorn` dependency clash of some sort, because I wasn't getting that when I removed `uvicorn` as a dependency.
I can avoid it by running this first though:
await micropip.install(""h11==0.12"")","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide,
https://github.com/simonw/datasette/issues/1735#issuecomment-1115301733,https://api.github.com/repos/simonw/datasette/issues/1735,1115301733,IC_kwDOBm6k_c5Ceidl,9599,simonw,2022-05-02T19:57:19Z,2022-05-02T19:59:03Z,OWNER,"This code breaks if that setting is 0:
https://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/app.py#L291-L293
It's used here:
https://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/database.py#L188-L190","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223263540,Datasette setting to disable threading (for Pyodide),
https://github.com/simonw/datasette/issues/1733#issuecomment-1115288284,https://api.github.com/repos/simonw/datasette/issues/1733,1115288284,IC_kwDOBm6k_c5CefLc,9599,simonw,2022-05-02T19:40:33Z,2022-05-02T19:40:33Z,OWNER,"I'll release this as a `0.62a0` as soon as it's ready, so I can start testing it out in Pyodide for real.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide,
https://github.com/simonw/datasette/issues/1734#issuecomment-1115283922,https://api.github.com/repos/simonw/datasette/issues/1734,1115283922,IC_kwDOBm6k_c5CeeHS,9599,simonw,2022-05-02T19:35:32Z,2022-05-02T19:35:32Z,OWNER,I'll use my original from 2009: https://www.djangosnippets.org/snippets/1431/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223241647,Remove python-baseconv dependency,
https://github.com/simonw/datasette/issues/1734#issuecomment-1115282773,https://api.github.com/repos/simonw/datasette/issues/1734,1115282773,IC_kwDOBm6k_c5Ced1V,9599,simonw,2022-05-02T19:34:15Z,2022-05-02T19:34:15Z,OWNER,I'm going to vendor it and update the documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223241647,Remove python-baseconv dependency,
https://github.com/simonw/datasette/issues/1733#issuecomment-1115278325,https://api.github.com/repos/simonw/datasette/issues/1733,1115278325,IC_kwDOBm6k_c5Cecv1,9599,simonw,2022-05-02T19:29:05Z,2022-05-02T19:29:05Z,OWNER,"I'm going to add a Datasette setting to disable threading entirely, designed for usage in this particular case.
I thought about adding a new setting, then I noticed this:
datasette mydatabase.db --setting num_sql_threads 10
I'm going to let users set that to `0` to disable threaded execution of SQL queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide,
https://github.com/simonw/datasette/issues/1733#issuecomment-1115268245,https://api.github.com/repos/simonw/datasette/issues/1733,1115268245,IC_kwDOBm6k_c5CeaSV,9599,simonw,2022-05-02T19:18:11Z,2022-05-02T19:18:11Z,OWNER,"Maybe I can leave `uvicorn` as a dependency? Installing it works OK, it only generates errors when you try to import it:
```pycon
Welcome to the Pyodide terminal emulator 🐍
Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM
Type ""help"", ""copyright"", ""credits"" or ""license"" for more information.
>>> import micropip
>>> await micropip.install(""uvicorn"")
>>> import uvicorn
Traceback (most recent call last):
File """", line 1, in
File ""/lib/python3.10/site-packages/uvicorn/__init__.py"", line 1, in
from uvicorn.config import Config
File ""/lib/python3.10/site-packages/uvicorn/config.py"", line 8, in
import ssl
File ""/lib/python3.10/ssl.py"", line 98, in
import _ssl # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'
>>> import ssl
>>> import uvicorn
Traceback (most recent call last):
File """", line 1, in
File ""/lib/python3.10/site-packages/uvicorn/__init__.py"", line 2, in
from uvicorn.main import Server, main, run
File ""/lib/python3.10/site-packages/uvicorn/main.py"", line 24, in
from uvicorn.supervisors import ChangeReload, Multiprocess
File ""/lib/python3.10/site-packages/uvicorn/supervisors/__init__.py"", line 3, in
from uvicorn.supervisors.basereload import BaseReload
File ""/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py"", line 12, in
from uvicorn.subprocess import get_subprocess
File ""/lib/python3.10/site-packages/uvicorn/subprocess.py"", line 14, in
multiprocessing.allow_connection_pickling()
File ""/lib/python3.10/multiprocessing/context.py"", line 170, in allow_connection_pickling
from . import connection
File ""/lib/python3.10/multiprocessing/connection.py"", line 21, in
import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'
>>> import multiprocessing
>>> import uvicorn
Traceback (most recent call last):
File """", line 1, in
File ""/lib/python3.10/site-packages/uvicorn/__init__.py"", line 2, in
from uvicorn.main import Server, main, run
File ""/lib/python3.10/site-packages/uvicorn/main.py"", line 24, in
from uvicorn.supervisors import ChangeReload, Multiprocess
File ""/lib/python3.10/site-packages/uvicorn/supervisors/__init__.py"", line 3, in
from uvicorn.supervisors.basereload import BaseReload
File ""/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py"", line 12, in
from uvicorn.subprocess import get_subprocess
File ""/lib/python3.10/site-packages/uvicorn/subprocess.py"", line 14, in
multiprocessing.allow_connection_pickling()
File ""/lib/python3.10/multiprocessing/context.py"", line 170, in allow_connection_pickling
from . import connection
File ""/lib/python3.10/multiprocessing/connection.py"", line 21, in
import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'
>>>
```
Since the `import ssl` trick fixed the `_ssl` error I was hopeful that `import multiprocessing` could fix the `_multiprocessing` one, but sadly it did not.
But it looks like i can address this issue just by making `import uvicorn` in `app.py` an optional import.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide,
https://github.com/simonw/datasette/issues/1733#issuecomment-1115262218,https://api.github.com/repos/simonw/datasette/issues/1733,1115262218,IC_kwDOBm6k_c5CeY0K,9599,simonw,2022-05-02T19:11:51Z,2022-05-02T19:14:01Z,OWNER,"Here's the full diff I applied to Datasette to get it fully working in Pyodide:
https://github.com/simonw/datasette/compare/94a3171b01fde5c52697aeeff052e3ad4bab5391...8af32bc5b03c30b1f7a4a8cc4bd80eb7e2ee7b81
And as a visible diff:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index d269372..6c0c5fc 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -15,7 +15,6 @@ import pkg_resources
import re
import secrets
import sys
-import threading
import traceback
import urllib.parse
from concurrent import futures
@@ -26,7 +25,6 @@ from itsdangerous import URLSafeSerializer
from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
from jinja2.environment import Template
from jinja2.exceptions import TemplateNotFound
-import uvicorn
from .views.base import DatasetteError, ureg
from .views.database import DatabaseDownload, DatabaseView
@@ -813,7 +811,6 @@ class Datasette:
},
""datasette"": datasette_version,
""asgi"": ""3.0"",
- ""uvicorn"": uvicorn.__version__,
""sqlite"": {
""version"": sqlite_version,
""fts_versions"": fts_versions,
@@ -854,23 +851,7 @@ class Datasette:
]
def _threads(self):
- threads = list(threading.enumerate())
- d = {
- ""num_threads"": len(threads),
- ""threads"": [
- {""name"": t.name, ""ident"": t.ident, ""daemon"": t.daemon} for t in threads
- ],
- }
- # Only available in Python 3.7+
- if hasattr(asyncio, ""all_tasks""):
- tasks = asyncio.all_tasks()
- d.update(
- {
- ""num_tasks"": len(tasks),
- ""tasks"": [_cleaner_task_str(t) for t in tasks],
- }
- )
- return d
+ return {""num_threads"": 0, ""threads"": []}
def _actor(self, request):
return {""actor"": request.actor}
diff --git a/datasette/database.py b/datasette/database.py
index ba594a8..b50142d 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -4,7 +4,6 @@ from pathlib import Path
import janus
import queue
import sys
-import threading
import uuid
from .tracer import trace
@@ -21,8 +20,6 @@ from .utils import (
)
from .inspect import inspect_hash
-connections = threading.local()
-
AttachedDatabase = namedtuple(""AttachedDatabase"", (""seq"", ""name"", ""file""))
@@ -43,12 +40,12 @@ class Database:
self.hash = None
self.cached_size = None
self._cached_table_counts = None
- self._write_thread = None
- self._write_queue = None
if not self.is_mutable and not self.is_memory:
p = Path(path)
self.hash = inspect_hash(p)
self.cached_size = p.stat().st_size
+ self._read_connection = None
+ self._write_connection = None
@property
def cached_table_counts(self):
@@ -134,60 +131,17 @@ class Database:
return results
async def execute_write_fn(self, fn, block=True):
- task_id = uuid.uuid5(uuid.NAMESPACE_DNS, ""datasette.io"")
- if self._write_queue is None:
- self._write_queue = queue.Queue()
- if self._write_thread is None:
- self._write_thread = threading.Thread(
- target=self._execute_writes, daemon=True
- )
- self._write_thread.start()
- reply_queue = janus.Queue()
- self._write_queue.put(WriteTask(fn, task_id, reply_queue))
- if block:
- result = await reply_queue.async_q.get()
- if isinstance(result, Exception):
- raise result
- else:
- return result
- else:
- return task_id
-
- def _execute_writes(self):
- # Infinite looping thread that protects the single write connection
- # to this database
- conn_exception = None
- conn = None
- try:
- conn = self.connect(write=True)
- self.ds._prepare_connection(conn, self.name)
- except Exception as e:
- conn_exception = e
- while True:
- task = self._write_queue.get()
- if conn_exception is not None:
- result = conn_exception
- else:
- try:
- result = task.fn(conn)
- except Exception as e:
- sys.stderr.write(""{}\n"".format(e))
- sys.stderr.flush()
- result = e
- task.reply_queue.sync_q.put(result)
+ # We always treat it as if block=True now
+ if self._write_connection is None:
+ self._write_connection = self.connect(write=True)
+ self.ds._prepare_connection(self._write_connection, self.name)
+ return fn(self._write_connection)
async def execute_fn(self, fn):
- def in_thread():
- conn = getattr(connections, self.name, None)
- if not conn:
- conn = self.connect()
- self.ds._prepare_connection(conn, self.name)
- setattr(connections, self.name, conn)
- return fn(conn)
-
- return await asyncio.get_event_loop().run_in_executor(
- self.ds.executor, in_thread
- )
+ if self._read_connection is None:
+ self._read_connection = self.connect()
+ self.ds._prepare_connection(self._read_connection, self.name)
+ return fn(self._read_connection)
async def execute(
self,
diff --git a/setup.py b/setup.py
index 7f0562f..c41669c 100644
--- a/setup.py
+++ b/setup.py
@@ -44,20 +44,20 @@ setup(
install_requires=[
""asgiref>=3.2.10,<3.6.0"",
""click>=7.1.1,<8.2.0"",
- ""click-default-group~=1.2.2"",
+ # ""click-default-group~=1.2.2"",
""Jinja2>=2.10.3,<3.1.0"",
""hupper~=1.9"",
""httpx>=0.20"",
""pint~=0.9"",
""pluggy>=1.0,<1.1"",
- ""uvicorn~=0.11"",
+ # ""uvicorn~=0.11"",
""aiofiles>=0.4,<0.9"",
""janus>=0.6.2,<1.1"",
""asgi-csrf>=0.9"",
""PyYAML>=5.3,<7.0"",
""mergedeep>=1.1.1,<1.4.0"",
""itsdangerous>=1.1,<3.0"",
- ""python-baseconv==1.2.2"",
+ # ""python-baseconv==1.2.2"",
],
entry_points=""""""
[console_scripts]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide,
https://github.com/simonw/datasette/issues/1734#issuecomment-1115260999,https://api.github.com/repos/simonw/datasette/issues/1734,1115260999,IC_kwDOBm6k_c5CeYhH,9599,simonw,2022-05-02T19:10:34Z,2022-05-02T19:10:34Z,OWNER,"This is actually mostly a documentation thing: here: https://docs.datasette.io/en/0.61.1/authentication.html#including-an-expiry-time
In the code it's only used in these two places:
https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/actor_auth_cookie.py#L16-L20
https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_auth.py#L56-L60","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223241647,Remove python-baseconv dependency,
https://github.com/simonw/datasette/issues/1733#issuecomment-1115258737,https://api.github.com/repos/simonw/datasette/issues/1733,1115258737,IC_kwDOBm6k_c5CeX9x,9599,simonw,2022-05-02T19:08:17Z,2022-05-02T19:08:17Z,OWNER,"I was going to vendor `baseconv.py`, but then I reconsidered - what if there are plugins out there that expect `import baseconv` to work because they have dependend on Datasette?
I used https://cs.github.com/ and as far as I can tell there aren't any!
So I'm going to remove that dependency and work out a smarter way to do this - probably by providing a utility function within Datasette itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide,
https://github.com/simonw/datasette/issues/1733#issuecomment-1115256318,https://api.github.com/repos/simonw/datasette/issues/1733,1115256318,IC_kwDOBm6k_c5CeXX-,9599,simonw,2022-05-02T19:05:55Z,2022-05-02T19:05:55Z,OWNER,"I released a `click-default-group-wheel` package to solve that dependency issue. I've already upgraded `sqlite-utils` to that, so now you can use that in Pyodide:
- https://github.com/simonw/sqlite-utils/pull/429
`python-baseconv` is only used for actor cookie expiration times:
https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/actor_auth_cookie.py#L16-L20
Datasette never actually sets that cookie itself - it instead encourages plugins to set it in the authentication documentation here: https://docs.datasette.io/en/0.61.1/authentication.html#including-an-expiry-time","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223234932,Get Datasette compatible with Pyodide,
https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115196863,https://api.github.com/repos/simonw/sqlite-utils/issues/429,1115196863,IC_kwDOCGYnMM5CeI2_,9599,simonw,2022-05-02T18:03:47Z,2022-05-02T18:52:42Z,OWNER,"I made a build of this branch and tested it like this: https://pyodide.org/en/stable/console.html
```pycon
>>> import micropip
>>> await micropip.install(""https://s3.amazonaws.com/simonwillison-cors-allowed-public/sqlite_utils-3.26-py3-none-any.whl"")
>>> import sqlite_utils
>>> db = sqlite_utils.Database(memory=True)
>>> list(db.query(""select 32443 + 55""))
[{'32443 + 55': 32498}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223177069,Depend on click-default-group-wheel,
https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115197644,https://api.github.com/repos/simonw/sqlite-utils/issues/429,1115197644,IC_kwDOCGYnMM5CeJDM,9599,simonw,2022-05-02T18:04:28Z,2022-05-02T18:04:28Z,OWNER,I'm going to ship this straight away as `3.26.1`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223177069,Depend on click-default-group-wheel,
https://github.com/simonw/datasette/issues/1479#issuecomment-1114601882,https://api.github.com/repos/simonw/datasette/issues/1479,1114601882,IC_kwDOBm6k_c5Cb3ma,32839123,Rik-de-Kort,2022-05-02T08:10:27Z,2022-05-02T11:54:49Z,NONE,"Also ran into this issue today using `datasette package`. The stack trace takes up my whole PowerShell history, though (recursionerror), but it also concerns the temporary directory.
Our development machines have a very zealous scanner that appears to insert itself between every call to the filesystem. I suspected that was causing some racing, but this turned out not to be the case: inserting `time.sleep(3)` on line 451 of `datasette/datasette/utils/__init__.py` does not make the problem go away. Commenting out the `tmp.cleanup()` line does.
The next error I get is docker-specific, so that probably does resolve the Datasette error here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish",
https://github.com/simonw/datasette/issues/1727#issuecomment-1114058210,https://api.github.com/repos/simonw/datasette/issues/1727,1114058210,IC_kwDOBm6k_c5CZy3i,9599,simonw,2022-04-30T21:39:34Z,2022-04-30T21:39:34Z,OWNER,"Something to consider if I look into subprocesses for parallel query execution:
https://sqlite.org/howtocorrupt.html#_carrying_an_open_database_connection_across_a_fork_
> Do not open an SQLite database connection, then fork(), then try to use that database connection in the child process. All kinds of locking problems will result and you can easily end up with a corrupt database. SQLite is not designed to support that kind of behavior. Any database connection that is used in a child process must be opened in the child process, not inherited from the parent. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1729#issuecomment-1114038259,https://api.github.com/repos/simonw/datasette/issues/1729,1114038259,IC_kwDOBm6k_c5CZt_z,9599,simonw,2022-04-30T19:06:03Z,2022-04-30T19:06:03Z,OWNER,"> but actually the facet results would be better if they were a list rather than a dictionary
I think `facet_results` in the JSON should match this (used by the HTML) instead:
https://github.com/simonw/datasette/blob/942411ef946e9a34a2094944d3423cddad27efd3/datasette/views/table.py#L737-L741
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView,
https://github.com/simonw/datasette/issues/1729#issuecomment-1114036946,https://api.github.com/repos/simonw/datasette/issues/1729,1114036946,IC_kwDOBm6k_c5CZtrS,9599,simonw,2022-04-30T18:56:25Z,2022-04-30T19:04:03Z,OWNER,"Related:
- #1558
Which talks about how there was confusion in this example: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1&_size=0
Which I fixed in #625 by introducing `tags` and `tags_2` keys, but actually the facet results would be better if they were a list rather than a dictionary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView,
https://github.com/simonw/datasette/issues/1729#issuecomment-1114037521,https://api.github.com/repos/simonw/datasette/issues/1729,1114037521,IC_kwDOBm6k_c5CZt0R,9599,simonw,2022-04-30T19:01:07Z,2022-04-30T19:01:07Z,OWNER,"I had to look up what `hideable` means - it means that you can't hide the current facet because it was defined in metadata, not as a `?_facet=` parameter:
https://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/datasette/facets.py#L228
That's a bit of a weird thing to expose in the API. Maybe change that to `source` so it can be `metadata` or `request`? That's very slightly less coupled to how the UI works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView,
https://github.com/simonw/datasette/issues/1729#issuecomment-1114013757,https://api.github.com/repos/simonw/datasette/issues/1729,1114013757,IC_kwDOBm6k_c5CZoA9,9599,simonw,2022-04-30T16:15:51Z,2022-04-30T18:54:39Z,OWNER,"Deployed a preview of this here: https://latest-1-0-alpha.datasette.io/
Examples:
- https://latest-1-0-alpha.datasette.io/fixtures/facetable.json
- https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count
Second example produces:
```json
{
""rows"": [],
""next"": null,
""next_url"": null,
""count"": 15,
""facet_results"": {
""state"": {
""name"": ""state"",
""type"": ""column"",
""hideable"": true,
""toggle_url"": ""/fixtures/facetable.json?_size=0&_extra=facet_results&_extra=count"",
""results"": [
{
""value"": ""CA"",
""label"": ""CA"",
""count"": 10,
""toggle_url"": ""https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=CA"",
""selected"": false
},
{
""value"": ""MI"",
""label"": ""MI"",
""count"": 4,
""toggle_url"": ""https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=MI"",
""selected"": false
},
{
""value"": ""MC"",
""label"": ""MC"",
""count"": 1,
""toggle_url"": ""https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=MC"",
""selected"": false
}
],
""truncated"": false
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView,
https://github.com/simonw/datasette/issues/1727#issuecomment-1112889800,https://api.github.com/repos/simonw/datasette/issues/1727,1112889800,IC_kwDOBm6k_c5CVVnI,9599,simonw,2022-04-29T05:29:38Z,2022-04-29T05:29:38Z,OWNER,"OK, I just got the most incredible result with that!
I started up a container running `bash` like this, from my `datasette` checkout. I'm mapping port 8005 on my laptop to port 8001 inside the container because laptop port 8001 was already doing something else:
```
docker run -it --rm --name my-running-script -p 8005:8001 -v ""$PWD"":/usr/src/myapp \
-w /usr/src/myapp nogil/python bash
```
Then in `bash` I ran the following commands to install Datasette and its dependencies:
```
pip install -e '.[test]'
pip install datasette-pretty-traces # For debug tracing
```
Then I started Datasette against my `github.db` database (from github-to-sqlite.dogsheep.net/github.db) like this:
```
datasette github.db -h 0.0.0.0 --setting trace_debug 1
```
I hit the following two URLs to compare the parallel v.s. not parallel implementations:
- `http://127.0.0.1:8005/github/issues?_facet=milestone&_facet=repo&_trace=1&_size=10`
- `http://127.0.0.1:8005/github/issues?_facet=milestone&_facet=repo&_trace=1&_size=10&_noparallel=1`
And... the parallel one beat the non-parallel one decisively, on multiple page refreshes!
Not parallel: 77ms
Parallel: 47ms
wrote:\n>I have a solution for FTS already, but I'm interested in apsw as a\n>mechanism for allowing custom virtual tables to be written in Python\n>(pysqlite only lets you write custom functions)\n>\n>Not having PyPI support is pretty tough though. I'm planning a\n>plugin/extension system which would be ideal for things like an\n>optional apsw mode, but that's a lot harder if apsw isn't in PyPI.\n>\n>-- \n>You are receiving this because you authored the thread.\n>Reply to this email directly or view it on GitHub:\n>https://github.com/simonw/datasette/issues/144#issuecomment-346405660\n""
},
{
""id"": 348252037,
""body"": ""WOW!\n\n--\nPaul Ford // (646) 369-7128 // @ftrain\n\nOn Thu, Nov 30, 2017 at 11:47 AM, Simon Willison \nwrote:\n\n> Remaining work on this now lives in a milestone:\n> https://github.com/simonw/datasette/milestone/6\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> ,\n> or mute the thread\n> \n> .\n>\n""
},
{
""id"": 391141391,
""body"": ""I'm going to clean this up for consistency tomorrow morning so hold off\nmerging until then please\n\nOn Tue, May 22, 2018 at 6:34 PM, Simon Willison \nwrote:\n\n> Yeah let's try this without pysqlite3 and see if we still get the correct\n> version.\n>\n> —\n> You are receiving this because you authored the thread.\n> Reply to this email directly, view it on GitHub\n> , or mute\n> the thread\n> \n> .\n>\n""
},
{
""id"": 391355030,
""body"": ""No objections;\r\nIt's good to go @simonw\r\n\r\nOn Wed, 23 May 2018, 14:51 Simon Willison, wrote:\r\n\r\n> @r4vi any objections to me merging this?\r\n>\r\n> —\r\n> You are receiving this because you were mentioned.\r\n> Reply to this email directly, view it on GitHub\r\n> , or mute\r\n> the thread\r\n> \r\n> .\r\n>\r\n""
}
],
""next"": ""391355030,391355030"",
""next_url"": ""http://127.0.0.1:8001/github_memory/issue_comments.json?_search=simon&_size=5&_extra=query_ms&_extra=count&_col=body&_next=391355030%2C391355030&_sort=id"",
""count"": 57,
""query_ms"": 21.780223003588617
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView,
https://github.com/simonw/datasette/issues/1729#issuecomment-1112730416,https://api.github.com/repos/simonw/datasette/issues/1729,1112730416,IC_kwDOBm6k_c5CUusw,9599,simonw,2022-04-28T23:01:21Z,2022-04-28T23:01:21Z,OWNER,"I'm not sure what to do about the `""truncated"": true/false` key.
It's not really relevant to table results, since they are paginated whether or not you ask for them to be.
It plays a role in query results, where you might run `select * from table` and get back 1000 results because Datasette truncates at that point rather than returning everything.
Adding it to every table result and always setting it to `""truncated"": false` feels confusing.
I think I'm going to keep it exclusively in the default representation for the `/db?sql=...` query endpoint, and not return it at all for tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView,
https://github.com/simonw/datasette/issues/1729#issuecomment-1112721321,https://api.github.com/repos/simonw/datasette/issues/1729,1112721321,IC_kwDOBm6k_c5CUsep,9599,simonw,2022-04-28T22:44:05Z,2022-04-28T22:44:14Z,OWNER,I may be able to implement this mostly in the `json_renderer()` function: https://github.com/simonw/datasette/blob/94a3171b01fde5c52697aeeff052e3ad4bab5391/datasette/renderer.py#L29-L34,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView,
https://github.com/simonw/datasette/issues/1729#issuecomment-1112717745,https://api.github.com/repos/simonw/datasette/issues/1729,1112717745,IC_kwDOBm6k_c5CUrmx,9599,simonw,2022-04-28T22:38:39Z,2022-04-28T22:39:05Z,OWNER,"(I remain keen on the idea of shipping a plugin that restores the old default API shape to people who have written pre-Datasette-1.0 code against it, but I'll tackle that much later. I really like how jQuery has a culture of doing this.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView,
https://github.com/simonw/datasette/issues/1729#issuecomment-1112717210,https://api.github.com/repos/simonw/datasette/issues/1729,1112717210,IC_kwDOBm6k_c5CUrea,9599,simonw,2022-04-28T22:37:37Z,2022-04-28T22:37:37Z,OWNER,"This means `filtered_table_rows_count` is going to become `count`. I had originally picked that terrible name to avoid confusion between the count of all rows in the table and the count of rows that were filtered.
I'll add `?_extra=table_count` for getting back the full table count instead. I think `count` is clear enough!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView,
https://github.com/simonw/datasette/issues/1729#issuecomment-1112716611,https://api.github.com/repos/simonw/datasette/issues/1729,1112716611,IC_kwDOBm6k_c5CUrVD,9599,simonw,2022-04-28T22:36:24Z,2022-04-28T22:36:24Z,OWNER,"Then I'm going to implement the following `?_extra=` options:
- `?_extra=facet_results` - to see facet results
- `?_extra=suggested_facets` - for suggested facets
- `?_extra=count` - for the count of total rows
- `?_extra=columns` - for a list of column names
- `?_extra=primary_keys` - for a list of primary keys
- `?_extra=query` - a `{""sql"" ""select ..."", ""params"": {}}` object
I thought about having `?_extra=facet_results` returned automatically if the user specifies at least one `?_facet` - but that doesn't work for default facets configured in `metadata.json` - how can the user opt out of those being returned? So I'm going to say you don't see facets at all if you don't include `?_extra=facet_results`.
I'm tempted to add `?_extra=_all` to return everything, but I can decide if that's a good idea later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView,
https://github.com/simonw/datasette/issues/1729#issuecomment-1112713581,https://api.github.com/repos/simonw/datasette/issues/1729,1112713581,IC_kwDOBm6k_c5CUqlt,9599,simonw,2022-04-28T22:31:11Z,2022-04-28T22:31:11Z,OWNER,"I'm going to change the default API response to look like this:
```json
{
""rows"": [
{
""pk"": 1,
""created"": ""2019-01-14 08:00:00"",
""planet_int"": 1,
""on_earth"": 1,
""state"": ""CA"",
""_city_id"": 1,
""_neighborhood"": ""Mission"",
""tags"": ""[\""tag1\"", \""tag2\""]"",
""complex_array"": ""[{\""foo\"": \""bar\""}]"",
""distinct_some_null"": ""one"",
""n"": ""n1""
},
{
""pk"": 2,
""created"": ""2019-01-14 08:00:00"",
""planet_int"": 1,
""on_earth"": 1,
""state"": ""CA"",
""_city_id"": 1,
""_neighborhood"": ""Dogpatch"",
""tags"": ""[\""tag1\"", \""tag3\""]"",
""complex_array"": ""[]"",
""distinct_some_null"": ""two"",
""n"": ""n2""
}
],
""next"": null,
""next_url"": null
}
```
Basically https://latest.datasette.io/fixtures/facetable.json?_shape=objects but with just the `rows`, `next` and `next_url` fields returned by default.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1219385669,Implement ?_extra and new API design for TableView,
https://github.com/simonw/datasette/issues/1715#issuecomment-1112711115,https://api.github.com/repos/simonw/datasette/issues/1715,1112711115,IC_kwDOBm6k_c5CUp_L,9599,simonw,2022-04-28T22:26:56Z,2022-04-28T22:26:56Z,OWNER,"I'm not going to use `asyncinject` in this refactor - at least not until I really need it. My research in these issues has put me off the idea ( in favour of `asyncio.gather()` or even not trying for parallel execution at all):
- #1727","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject,
https://github.com/simonw/datasette/issues/1727#issuecomment-1112668411,https://api.github.com/repos/simonw/datasette/issues/1727,1112668411,IC_kwDOBm6k_c5CUfj7,9599,simonw,2022-04-28T21:25:34Z,2022-04-28T21:25:44Z,OWNER,"The two most promising theories at the moment, from here and Twitter and the SQLite forum, are:
- SQLite is I/O bound - it generally only goes as fast as it can load data from disk. Multiple connections all competing for the same file on disk are going to end up blocked at the file system layer. But maybe this means in-memory databases will perform better?
- It's the GIL. The sqlite3 C code may release the GIL, but the bits that do things like assembling `Row` objects to return still happen in Python, and that Python can only run on a single core.
A couple of ways to research the in-memory theory:
- Use a RAM disk on macOS (or Linux). https://stackoverflow.com/a/2033417/6083 has instructions - short version:
hdiutil attach -nomount ram://$((2 * 1024 * 100))
diskutil eraseVolume HFS+ RAMDisk name-returned-by-previous-command (was `/dev/disk2` when I tried it)
cd /Volumes/RAMDisk
cp ~/fixtures.db .
- Copy Datasette databases into an in-memory database on startup. I built a new plugin to do that here: https://github.com/simonw/datasette-copy-to-memory
I need to do some more, better benchmarks using these different approaches.
https://twitter.com/laurencerowe/status/1519780174560169987 also suggests:
> Maybe try:
> 1. Copy the sqlite file to /dev/shm and rerun (all in ram.)
> 2. Create a CTE which calculates Fibonacci or similar so you can test something completely cpu bound (only return max value or something to avoid crossing between sqlite/Python.)
I like that second idea a lot - I could use the mandelbrot example from https://www.sqlite.org/lang_with.html#outlandish_recursive_query_examples","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1633#issuecomment-1111955628,https://api.github.com/repos/simonw/datasette/issues/1633,1111955628,IC_kwDOBm6k_c5CRxis,6613091,henrikek,2022-04-28T09:12:56Z,2022-04-28T09:12:56Z,NONE,I have verified that the problem with base_url still exists in the latest version 0.61.1. I would need some guidance if my code change suggestion is correct or if base_url should be included in some other code?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1129052172,base_url or prefix does not work with _exact match,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111752676,https://api.github.com/repos/simonw/datasette/issues/1728,1111752676,IC_kwDOBm6k_c5CQ__k,127565,wragge,2022-04-28T05:11:54Z,2022-04-28T05:11:54Z,CONTRIBUTOR,"And in terms of the bug, yep I agree that option 2 would be the most useful and least frustrating.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111751734,https://api.github.com/repos/simonw/datasette/issues/1728,1111751734,IC_kwDOBm6k_c5CQ_w2,127565,wragge,2022-04-28T05:09:59Z,2022-04-28T05:09:59Z,CONTRIBUTOR,"Thanks, I'll give it a try!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111726586,https://api.github.com/repos/simonw/datasette/issues/1727,1111726586,IC_kwDOBm6k_c5CQ5n6,9599,simonw,2022-04-28T04:17:16Z,2022-04-28T04:19:31Z,OWNER,"I could experiment with the `await asyncio.run_in_executor(processpool_executor, fn)` mechanism described in https://stackoverflow.com/a/29147750
Code examples: https://cs.github.com/?scopeName=All+repos&scope=&q=run_in_executor+ProcessPoolExecutor","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111725638,https://api.github.com/repos/simonw/datasette/issues/1727,1111725638,IC_kwDOBm6k_c5CQ5ZG,9599,simonw,2022-04-28T04:15:15Z,2022-04-28T04:15:15Z,OWNER,"Useful theory from Keith Medcalf https://sqlite.org/forum/forumpost/e363c69d3441172e
> This is true, but the concurrency is limited to the execution which occurs with the GIL released (that is, in the native C sqlite3 library itself). Each row (for example) can be retrieved in parallel but ""constructing the python return objects for each row"" will be serialized (by the GIL).
>
> That is to say that if your have two python threads each with their own connection, and each one is performing a select that returns 1,000,000 rows (lets say that is 25% of the candidates for each select) then the difference in execution time between executing two python threads in parallel vs a single serial thead will not be much different (if even detectable at all). In fact it is possible that the multiple-threaded version takes longer to run both queries to completion because of the increased contention over a shared resource (the GIL).
So maybe this is a GIL thing.
I should test with some expensive SQL queries (maybe big aggregations against large tables) and see if I can spot an improvement there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111714665,https://api.github.com/repos/simonw/datasette/issues/1728,1111714665,IC_kwDOBm6k_c5CQ2tp,9599,simonw,2022-04-28T03:52:47Z,2022-04-28T03:52:58Z,OWNER,"Nice custom template/theme!
Yeah, for that I'd recommend hosting elsewhere - on a regular VPS (I use `systemd` like this: https://docs.datasette.io/en/stable/deploying.html#running-datasette-using-systemd ) or using Fly if you want to tub containers without managing a full server.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111712953,https://api.github.com/repos/simonw/datasette/issues/1728,1111712953,IC_kwDOBm6k_c5CQ2S5,127565,wragge,2022-04-28T03:48:36Z,2022-04-28T03:48:36Z,CONTRIBUTOR,"I don't think that'd work for this project. The db is very big, and my aim was to have an environment where researchers could be making use of the data, but be easily able to add corrections to the HTR/OCR extracted data when they came across problems. It's in its immutable (!) form here: https://sydney-stock-exchange-xqtkxtd5za-ts.a.run.app/stock_exchange/stocks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111708206,https://api.github.com/repos/simonw/datasette/issues/1728,1111708206,IC_kwDOBm6k_c5CQ1Iu,9599,simonw,2022-04-28T03:38:56Z,2022-04-28T03:38:56Z,OWNER,"In terms of this bug, there are a few potential fixes:
1. Detect the write to a immutable database and show the user a proper, meaningful error message in the red error box at the top of the page
2. Don't allow the user to even submit the form - show a message saying that this canned query is unavailable because the database cannot be written to
3. Don't even allow Datasette to start running at all - if there's a canned query configured in `metadata.yml` and the database it refers to is in `-i` immutable mode throw an error on startup
I'm not keen on that last one because it would be frustrating if you couldn't launch Datasette just because you had an old canned query lying around in your metadata file.
So I'm leaning towards option 2.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111707384,https://api.github.com/repos/simonw/datasette/issues/1728,1111707384,IC_kwDOBm6k_c5CQ074,9599,simonw,2022-04-28T03:36:46Z,2022-04-28T03:36:56Z,OWNER,"A more realistic solution (which I've been using on several of my own projects) is to keep the data itself in GitHub and encourage users to edit it there - using the GitHub web interface to edit YAML files or similar.
Needs your users to be comfortable hand-editing YAML though! You can at least guard against critical errors by having CI run tests against their YAML before deploying.
I have a dream of building a more friendly web forms interface which edits the YAML back on GitHub for the user, but that's just a concept at the moment.
Even more fun would be if a user-friendly form could submit PRs for review without the user having to know what a PR is!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111706519,https://api.github.com/repos/simonw/datasette/issues/1728,1111706519,IC_kwDOBm6k_c5CQ0uX,9599,simonw,2022-04-28T03:34:49Z,2022-04-28T03:34:49Z,OWNER,"I've wanted to do stuff like that on Cloud Run too. So far I've assumed that it's not feasible, but recently I've been wondering how hard it would be to have a small (like less than 100KB or so) Datasette instance which persists data to a backing GitHub repository such that when it starts up it can pull the latest copy and any time someone edits it can push their changes.
I'm still not sure it would work well on Cloud Run due to the uncertainty at what would happen if Cloud Run decided to boot up a second instance - but it's still an interesting thought exercise.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111705323,https://api.github.com/repos/simonw/datasette/issues/1728,1111705323,IC_kwDOBm6k_c5CQ0br,127565,wragge,2022-04-28T03:32:06Z,2022-04-28T03:32:06Z,CONTRIBUTOR,"Ah, that would be it! I have a core set of data which doesn't change to which I want authorised users to be able to submit corrections. I was going to deal with the persistence issue by just grabbing the user corrections at regular intervals and saving to GitHub. I might need to rethink. Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111705069,https://api.github.com/repos/simonw/datasette/issues/1728,1111705069,IC_kwDOBm6k_c5CQ0Xt,9599,simonw,2022-04-28T03:31:33Z,2022-04-28T03:31:33Z,OWNER,"Confirmed - this is a bug where immutable databases fail to show a useful error if you write to them with a canned query.
Steps to reproduce:
```
echo '
databases:
writable:
queries:
add_name:
sql: insert into names(name) values (:name)
write: true
' > write-metadata.yml
echo '{""name"": ""Simon""}' | sqlite-utils insert writable.db names -
datasette writable.db -m write-metadata.yml
```
Then visit http://127.0.0.1:8001/writable/add_name - adding names works.
Now do this instead:
```
datasette -i writable.db -m write-metadata.yml
```
And I'm getting a broken error:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111699175,https://api.github.com/repos/simonw/datasette/issues/1727,1111699175,IC_kwDOBm6k_c5CQy7n,9599,simonw,2022-04-28T03:19:48Z,2022-04-28T03:20:08Z,OWNER,"I ran `py-spy` and then hammered refresh a bunch of times on the `http://127.0.0.1:8856/github/commits?_facet=repo&_facet=committer&_trace=1&_noparallel=` page - it generated this SVG profile for me.
The area on the right is the threads running the DB queries:
Interactive version here: https://static.simonwillison.net/static/2022/datasette-parallel-profile.svg","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111698307,https://api.github.com/repos/simonw/datasette/issues/1728,1111698307,IC_kwDOBm6k_c5CQyuD,9599,simonw,2022-04-28T03:18:02Z,2022-04-28T03:18:02Z,OWNER,If the behaviour you are seeing is because the database is running in immutable mode then that's a bug - you should get a useful error message instead!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111697985,https://api.github.com/repos/simonw/datasette/issues/1728,1111697985,IC_kwDOBm6k_c5CQypB,9599,simonw,2022-04-28T03:17:20Z,2022-04-28T03:17:20Z,OWNER,"How did you deploy to Cloud Run?
`datasette publish cloudrun` defaults to running databases there in `-i` immutable mode, because if you managed to change a file on disk on Cloud Run those changes would be lost the next time your container restarted there.
That's why I upgraded `datasette-publish-fly` to provide a way of working with their volumes support - they're the best option I know of right now for running Datasette in a container with a persistent volume that can accept writes: https://simonwillison.net/2022/Feb/15/fly-volumes/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111683539,https://api.github.com/repos/simonw/datasette/issues/1727,1111683539,IC_kwDOBm6k_c5CQvHT,9599,simonw,2022-04-28T02:47:57Z,2022-04-28T02:47:57Z,OWNER,"Maybe this is the Python GIL after all?
I've been hoping that the GIL won't be an issue because the `sqlite3` module releases the GIL for the duration of the execution of a SQL query - see https://github.com/python/cpython/blob/f348154c8f8a9c254503306c59d6779d4d09b3a9/Modules/_sqlite/cursor.c#L749-L759
So I've been hoping this means that SQLite code itself can run concurrently on multiple cores even when Python threads cannot.
But maybe I'm misunderstanding how that works?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111681513,https://api.github.com/repos/simonw/datasette/issues/1727,1111681513,IC_kwDOBm6k_c5CQunp,9599,simonw,2022-04-28T02:44:26Z,2022-04-28T02:44:26Z,OWNER,"I could try `py-spy top`, which I previously used here:
- https://github.com/simonw/datasette/issues/1673","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111661331,https://api.github.com/repos/simonw/datasette/issues/1727,1111661331,IC_kwDOBm6k_c5CQpsT,9599,simonw,2022-04-28T02:07:31Z,2022-04-28T02:07:31Z,OWNER,Asked on the SQLite forum about this here: https://sqlite.org/forum/forumpost/ffbfa9f38e,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111602802,https://api.github.com/repos/simonw/datasette/issues/1727,1111602802,IC_kwDOBm6k_c5CQbZy,9599,simonw,2022-04-28T00:21:35Z,2022-04-28T00:21:35Z,OWNER,"Tried this but I'm getting back an empty JSON array of traces at the bottom of the page most of the time (intermittently it works correctly):
```diff
diff --git a/datasette/database.py b/datasette/database.py
index ba594a8..d7f9172 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -7,7 +7,7 @@ import sys
import threading
import uuid
-from .tracer import trace
+from .tracer import trace, trace_child_tasks
from .utils import (
detect_fts,
detect_primary_keys,
@@ -207,30 +207,31 @@ class Database:
time_limit_ms = custom_time_limit
with sqlite_timelimit(conn, time_limit_ms):
- try:
- cursor = conn.cursor()
- cursor.execute(sql, params if params is not None else {})
- max_returned_rows = self.ds.max_returned_rows
- if max_returned_rows == page_size:
- max_returned_rows += 1
- if max_returned_rows and truncate:
- rows = cursor.fetchmany(max_returned_rows + 1)
- truncated = len(rows) > max_returned_rows
- rows = rows[:max_returned_rows]
- else:
- rows = cursor.fetchall()
- truncated = False
- except (sqlite3.OperationalError, sqlite3.DatabaseError) as e:
- if e.args == (""interrupted"",):
- raise QueryInterrupted(e, sql, params)
- if log_sql_errors:
- sys.stderr.write(
- ""ERROR: conn={}, sql = {}, params = {}: {}\n"".format(
- conn, repr(sql), params, e
+ with trace(""sql"", database=self.name, sql=sql.strip(), params=params):
+ try:
+ cursor = conn.cursor()
+ cursor.execute(sql, params if params is not None else {})
+ max_returned_rows = self.ds.max_returned_rows
+ if max_returned_rows == page_size:
+ max_returned_rows += 1
+ if max_returned_rows and truncate:
+ rows = cursor.fetchmany(max_returned_rows + 1)
+ truncated = len(rows) > max_returned_rows
+ rows = rows[:max_returned_rows]
+ else:
+ rows = cursor.fetchall()
+ truncated = False
+ except (sqlite3.OperationalError, sqlite3.DatabaseError) as e:
+ if e.args == (""interrupted"",):
+ raise QueryInterrupted(e, sql, params)
+ if log_sql_errors:
+ sys.stderr.write(
+ ""ERROR: conn={}, sql = {}, params = {}: {}\n"".format(
+ conn, repr(sql), params, e
+ )
)
- )
- sys.stderr.flush()
- raise
+ sys.stderr.flush()
+ raise
if truncate:
return Results(rows, truncated, cursor.description)
@@ -238,9 +239,8 @@ class Database:
else:
return Results(rows, False, cursor.description)
- with trace(""sql"", database=self.name, sql=sql.strip(), params=params):
- results = await self.execute_fn(sql_operation_in_thread)
- return results
+ with trace_child_tasks():
+ return await self.execute_fn(sql_operation_in_thread)
@property
def size(self):
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111597176,https://api.github.com/repos/simonw/datasette/issues/1727,1111597176,IC_kwDOBm6k_c5CQaB4,9599,simonw,2022-04-28T00:11:44Z,2022-04-28T00:11:44Z,OWNER,"Though it would be interesting to also have the trace reveal how much time is spent in the functions that wrap that core SQL - the stuff that is being measured at the moment.
I have a hunch that this could help solve the over-arching performance mystery.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111595319,https://api.github.com/repos/simonw/datasette/issues/1727,1111595319,IC_kwDOBm6k_c5CQZk3,9599,simonw,2022-04-28T00:09:45Z,2022-04-28T00:11:01Z,OWNER,"Here's where read queries are instrumented: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L241-L242
So the instrumentation is actually capturing quite a bit of Python activity before it gets to SQLite:
https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L179-L190
And then:
https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L204-L233
Ideally I'd like that `trace()` block to wrap just the `cursor.execute()` and `cursor.fetchmany(...)` or `cursor.fetchall()` calls.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111558204,https://api.github.com/repos/simonw/datasette/issues/1727,1111558204,IC_kwDOBm6k_c5CQQg8,9599,simonw,2022-04-27T22:58:39Z,2022-04-27T22:58:39Z,OWNER,"I should check my timing mechanism. Am I capturing the time taken just in SQLite or does it include time spent in Python crossing between async and threaded world and waiting for a thread pool worker to become available?
That could explain the longer query times.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111553029,https://api.github.com/repos/simonw/datasette/issues/1727,1111553029,IC_kwDOBm6k_c5CQPQF,9599,simonw,2022-04-27T22:48:21Z,2022-04-27T22:48:21Z,OWNER,I wonder if it would be worth exploring multiprocessing here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111551076,https://api.github.com/repos/simonw/datasette/issues/1727,1111551076,IC_kwDOBm6k_c5CQOxk,9599,simonw,2022-04-27T22:44:51Z,2022-04-27T22:45:04Z,OWNER,Really wild idea: what if I created three copies of the SQLite database file - as three separate file names - and then balanced the parallel queries across all these? Any chance that could avoid any mysterious locking issues?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111535818,https://api.github.com/repos/simonw/datasette/issues/1727,1111535818,IC_kwDOBm6k_c5CQLDK,9599,simonw,2022-04-27T22:18:45Z,2022-04-27T22:18:45Z,OWNER,"Another avenue: https://twitter.com/weargoggles/status/1519426289920270337
> SQLite has its own mutexes to provide thread safety, which as another poster noted are out of play in multi process setups. Perhaps downgrading from the “serializable” to “multi-threaded” safety would be okay for Datasette? https://sqlite.org/c3ref/c_config_covering_index_scan.html#sqliteconfigmultithread
Doesn't look like there's an obvious way to access that from Python via the `sqlite3` module though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-1111506339,https://api.github.com/repos/simonw/sqlite-utils/issues/159,1111506339,IC_kwDOCGYnMM5CQD2j,154364,dracos,2022-04-27T21:35:13Z,2022-04-27T21:35:13Z,NONE,"Just stumbled across this, wondering why none of my deletes were working.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/datasette/issues/1727#issuecomment-1111485722,https://api.github.com/repos/simonw/datasette/issues/1727,1111485722,IC_kwDOBm6k_c5CP-0a,9599,simonw,2022-04-27T21:08:20Z,2022-04-27T21:08:20Z,OWNER,"Tried that and it didn't seem to make a difference either.
I really need a much deeper view of what's going on here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111462442,https://api.github.com/repos/simonw/datasette/issues/1727,1111462442,IC_kwDOBm6k_c5CP5Iq,9599,simonw,2022-04-27T20:40:59Z,2022-04-27T20:42:49Z,OWNER,"This looks VERY relevant: [SQLite Shared-Cache Mode](https://www.sqlite.org/sharedcache.html):
> SQLite includes a special ""shared-cache"" mode (disabled by default) intended for use in embedded servers. If shared-cache mode is enabled and a thread establishes multiple connections to the same database, the connections share a single data and schema cache. This can significantly reduce the quantity of memory and IO required by the system.
Enabled as part of the URI filename:
ATTACH 'file:aux.db?cache=shared' AS aux;
Turns out I'm already using this for in-memory databases that have `.memory_name` set, but not (yet) for regular file-backed databases:
https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L73-L75
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111460068,https://api.github.com/repos/simonw/datasette/issues/1727,1111460068,IC_kwDOBm6k_c5CP4jk,9599,simonw,2022-04-27T20:38:32Z,2022-04-27T20:38:32Z,OWNER,WAL mode didn't seem to make a difference. I thought there was a chance it might help multiple read connections operate at the same time but it looks like it really does only matter for when writes are going on.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1727#issuecomment-1111456500,https://api.github.com/repos/simonw/datasette/issues/1727,1111456500,IC_kwDOBm6k_c5CP3r0,9599,simonw,2022-04-27T20:36:01Z,2022-04-27T20:36:01Z,OWNER,"Yeah all of this is pretty much assuming read-only connections. Datasette has a separate mechanism for ensuring that writes are executed one at a time against a dedicated connection from an in-memory queue:
- https://github.com/simonw/datasette/issues/682","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,