.status_code
tests/test_graphql.py:142: AssertionError
----------------------------- Captured stderr call -----------------------------
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
Traceback (most recent call last):
File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py"", line 1171, in route_path
response = await view(request, send)
File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py"", line 151, in view
request, **request.scope[""url_route""][""kwargs""]
File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py"", line 123, in dispatch_request
await self.ds.refresh_schemas()
File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py"", line 338, in refresh_schemas
await init_internal_db(internal_db)
File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/utils/internal_db.py"", line 16, in init_internal_db
block=True,
File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 102, in execute_write
return await self.execute_write_fn(_inner, block=block)
File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 118, in execute_write_fn
raise result
File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 139, in _execute_writes
result = task.fn(conn)
File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 100, in _inner
return conn.execute(sql, params or [])
sqlite3.OperationalError: table databases already exists
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,
https://github.com/simonw/datasette/issues/1231#issuecomment-881204343,https://api.github.com/repos/simonw/datasette/issues/1231,881204343,IC_kwDOBm6k_c40hhx3,9599,2021-07-16T06:13:11Z,2021-07-16T06:13:11Z,OWNER,This just broke the `datasette-graphql` test suite: https://github.com/simonw/datasette-graphql/issues/77 - I need to figure out a solution here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,
https://github.com/simonw/datasette/issues/1231#issuecomment-881663968,https://api.github.com/repos/simonw/datasette/issues/1231,881663968,IC_kwDOBm6k_c40jR_g,9599,2021-07-16T19:18:42Z,2021-07-16T19:18:42Z,OWNER,The race condition happens inside this method - initially with the call to `await init_internal_db()`: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/app.py#L334-L359,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,
https://github.com/simonw/datasette/issues/1231#issuecomment-881664408,https://api.github.com/repos/simonw/datasette/issues/1231,881664408,IC_kwDOBm6k_c40jSGY,9599,2021-07-16T19:19:35Z,2021-07-16T19:19:35Z,OWNER,"The only place that calls `refresh_schemas()` is here: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/views/base.py#L120-L124
Ideally only one call to `refresh_schemas()` would be running at any one time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,
https://github.com/simonw/datasette/issues/1231#issuecomment-881665383,https://api.github.com/repos/simonw/datasette/issues/1231,881665383,IC_kwDOBm6k_c40jSVn,9599,2021-07-16T19:21:35Z,2021-07-16T19:21:35Z,OWNER,"https://stackoverflow.com/a/25799871/6083 has a good example of using `asyncio.Lock()`:
```python
stuff_lock = asyncio.Lock()
async def get_stuff(url):
async with stuff_lock:
if url in cache:
return cache[url]
stuff = await aiohttp.request('GET', url)
cache[url] = stuff
return stuff
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,
https://github.com/simonw/datasette/issues/1231#issuecomment-881671706,https://api.github.com/repos/simonw/datasette/issues/1231,881671706,IC_kwDOBm6k_c40jT4a,9599,2021-07-16T19:32:05Z,2021-07-16T19:32:05Z,OWNER,The test suite passes with that change.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,
https://github.com/simonw/datasette/issues/1231#issuecomment-881668759,https://api.github.com/repos/simonw/datasette/issues/1231,881668759,IC_kwDOBm6k_c40jTKX,9599,2021-07-16T19:27:46Z,2021-07-16T19:27:46Z,OWNER,"Second attempt at this:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 5976d8b..5f348cb 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -224,6 +224,7 @@ class Datasette:
self.inspect_data = inspect_data
self.immutables = set(immutables or [])
self.databases = collections.OrderedDict()
+ self._refresh_schemas_lock = asyncio.Lock()
self.crossdb = crossdb
if memory or crossdb or not self.files:
self.add_database(Database(self, is_memory=True), name=""_memory"")
@@ -332,6 +333,12 @@ class Datasette:
self.client = DatasetteClient(self)
async def refresh_schemas(self):
+ if self._refresh_schemas_lock.locked():
+ return
+ async with self._refresh_schemas_lock:
+ await self._refresh_schemas()
+
+ async def _refresh_schemas(self):
internal_db = self.databases[""_internal""]
if not self.internal_db_created:
await init_internal_db(internal_db)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,
https://github.com/simonw/datasette/issues/1231#issuecomment-881674857,https://api.github.com/repos/simonw/datasette/issues/1231,881674857,IC_kwDOBm6k_c40jUpp,9599,2021-07-16T19:38:39Z,2021-07-16T19:38:39Z,OWNER,I can't replicate the race condition locally with or without this patch. I'm going to push the commit and then test the CI run from `datasette-graphql` that was failing against it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,
https://github.com/simonw/datasette/issues/1231#issuecomment-881677620,https://api.github.com/repos/simonw/datasette/issues/1231,881677620,IC_kwDOBm6k_c40jVU0,9599,2021-07-16T19:44:12Z,2021-07-16T19:44:12Z,OWNER,"That fixed the race condition in the `datasette-graphql` tests, which is the only place that I've been able to successfully replicate this. I'm going to land this change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,
https://github.com/simonw/datasette/issues/1396#issuecomment-881686662,https://api.github.com/repos/simonw/datasette/issues/1396,881686662,IC_kwDOBm6k_c40jXiG,9599,2021-07-16T20:02:44Z,2021-07-16T20:02:44Z,OWNER,Confirmed fixed: 0.58.1 was successfully published to Docker Hub in https://github.com/simonw/datasette/runs/3089447346 and the `latest` tag on https://hub.docker.com/r/datasetteproject/datasette/tags was updated.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,
https://github.com/simonw/datasette/issues/1199#issuecomment-881932880,https://api.github.com/repos/simonw/datasette/issues/1199,881932880,IC_kwDOBm6k_c40kTpQ,9599,2021-07-17T17:39:17Z,2021-07-17T17:39:17Z,OWNER,"I asked about optimizing performance on the SQLite forum and this came up as a suggestion: https://sqlite.org/forum/forumpost/9a6b9ae8e2048c8b?t=c
I can start by trying this:
PRAGMA mmap_size=268435456;","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792652391,
https://github.com/simonw/datasette/issues/123#issuecomment-882138084,https://api.github.com/repos/simonw/datasette/issues/123,882138084,IC_kwDOBm6k_c40lFvk,9599,2021-07-19T00:04:31Z,2021-07-19T00:04:31Z,OWNER,"I've been thinking more about this one today too. An extension of this (touched on in #417, Datasette Library) would be to support pointing Datasette at a directory and having it automatically load any CSV files it finds anywhere in that folder or its descendants - either loading them fully, or providing a UI that allows users to select a file to open it in Datasette.
For larger files I think the right thing to do is import them into an on-disk SQLite database, which is limited only by available disk space. For smaller files loading them into an in-memory database should work fine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,
https://github.com/simonw/datasette/issues/1445#issuecomment-904024939,https://api.github.com/repos/simonw/datasette/issues/1445,904024939,IC_kwDOBm6k_c414lNr,9599,2021-08-23T18:52:35Z,2021-08-23T18:52:35Z,OWNER,"The downside of the current implementation of this trick is that it only works for exact LIKE partial matches in a specific table - if you search for `dog cat` and `dog` appears in `title` but `cat` appears in `description` you won't get back that result.
I think that's fine though. If you want more advanced search there are other mechanisms you can use. This is meant to be a very quick and dirty starting point for exploring a brand new table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,
https://github.com/simonw/datasette/issues/1445#issuecomment-904026253,https://api.github.com/repos/simonw/datasette/issues/1445,904026253,IC_kwDOBm6k_c414liN,9599,2021-08-23T18:54:49Z,2021-08-23T18:54:49Z,OWNER,"The bigger problem here is UI design. This feels like a pretty niche requirement to me, so adding a prominent search box to the table page (which already has the filters interface, plus the full-text search box for tables that have FTS configured) feels untidy.
I could tuck it away in the table cog menu, but that's a weird place for something like this to live.
Maybe add it as a new type of filter? Filters apply to specific columns though, so this would be the first filter that applied to _all_ columns - which doesn't really fit the existing filter interface very well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,
https://github.com/simonw/datasette/issues/1445#issuecomment-904027166,https://api.github.com/repos/simonw/datasette/issues/1445,904027166,IC_kwDOBm6k_c414lwe,9599,2021-08-23T18:56:20Z,2021-08-23T18:56:20Z,OWNER,A related but potentially even more useful ability would be running a search across every column of every table in a whole database. For anything less than a few 100MB this could be incredibly useful.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,
https://github.com/simonw/datasette/issues/1445#issuecomment-904036200,https://api.github.com/repos/simonw/datasette/issues/1445,904036200,IC_kwDOBm6k_c414n9o,9599,2021-08-23T19:08:54Z,2021-08-23T19:08:54Z,OWNER,"Figured out a query for searching across every column in every table! https://til.simonwillison.net/datasette/search-all-columns-trick#user-content-same-trick-for-the-entire-database
```sql
with tables as (
select
name as table_name
from
sqlite_master
where
type = 'table'
),
queries as (
select
'select ''' || tables.table_name || ''' as _table, rowid from ""' || tables.table_name || '"" where ' || group_concat(
'""' || name || '"" like ''%'' || :search || ''%''',
' or '
) as query
from
pragma_table_info(tables.table_name),
tables
group by
tables.table_name
)
select
group_concat(query, ' union all ')
from
queries
```
The SQL query this generates for larger databases is _extremely_ long - but it does seem to work for smaller databases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,
https://github.com/simonw/datasette/issues/1445#issuecomment-904037087,https://api.github.com/repos/simonw/datasette/issues/1445,904037087,IC_kwDOBm6k_c414oLf,9599,2021-08-23T19:10:17Z,2021-08-23T19:10:17Z,OWNER,"Rather than trying to run that monstrosity in a single `union all` query, a better approach may be to use `fetch()` requests as seen in https://datasette.io/plugins/datasette-search-all","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,
https://github.com/simonw/datasette/issues/1446#issuecomment-904866495,https://api.github.com/repos/simonw/datasette/issues/1446,904866495,IC_kwDOBm6k_c417yq_,9599,2021-08-24T18:13:49Z,2021-08-24T18:13:49Z,OWNER,"OK, now the following optional CSS gives us a sticky footer:
```css
html, body {
height: 100%;
}
body {
display: flex;
flex-direction: column;
}
.not-footer {
flex: 1 0 auto;
}
footer {
flex-shrink: 0;
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978357984,
https://github.com/simonw/datasette/issues/1446#issuecomment-904954530,https://api.github.com/repos/simonw/datasette/issues/1446,904954530,IC_kwDOBm6k_c418IKi,9599,2021-08-24T20:32:47Z,2021-08-24T20:32:47Z,OWNER,"Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978357984,
https://github.com/simonw/datasette/pull/1447#issuecomment-905097468,https://api.github.com/repos/simonw/datasette/issues/1447,905097468,IC_kwDOBm6k_c418rD8,9599,2021-08-25T01:28:53Z,2021-08-25T01:28:53Z,OWNER,"Good catch, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978614898,
https://github.com/simonw/datasette/issues/1405#issuecomment-889525741,https://api.github.com/repos/simonw/datasette/issues/1405,889525741,IC_kwDOBm6k_c41BRXt,9599,2021-07-29T23:33:30Z,2021-07-29T23:33:30Z,OWNER,New documentation section for `datasette.utils` is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",955316250,
https://github.com/simonw/datasette/issues/1241#issuecomment-889539227,https://api.github.com/repos/simonw/datasette/issues/1241,889539227,IC_kwDOBm6k_c41BUqb,9599,2021-07-30T00:15:26Z,2021-07-30T00:15:26Z,OWNER,"One possible treatment:
```html
{% if query.sql and allow_execute_sql %}
View and edit SQL
{% endif %}
Copy and share link
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,
https://github.com/simonw/datasette/issues/1406#issuecomment-889548536,https://api.github.com/repos/simonw/datasette/issues/1406,889548536,IC_kwDOBm6k_c41BW74,9599,2021-07-30T00:43:47Z,2021-07-30T00:43:47Z,OWNER,"Still couldn't replicate on my laptop. On a hunch, I'm going to add `@pytest.mark.serial` to every test that uses `runner.isolated_filesystem()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,
https://github.com/simonw/datasette/issues/1406#issuecomment-889547142,https://api.github.com/repos/simonw/datasette/issues/1406,889547142,IC_kwDOBm6k_c41BWmG,9599,2021-07-30T00:39:49Z,2021-07-30T00:39:49Z,OWNER,It happens in CI but not on my laptop. I think I need to run the tests on my laptop like this: https://github.com/simonw/datasette/blob/121e10c29c5b412fddf0326939f1fe46c3ad9d4a/.github/workflows/test.yml#L27-L30,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,
https://github.com/simonw/datasette/issues/1406#issuecomment-889550391,https://api.github.com/repos/simonw/datasette/issues/1406,889550391,IC_kwDOBm6k_c41BXY3,9599,2021-07-30T00:49:31Z,2021-07-30T00:49:31Z,OWNER,That fixed it. My hunch is that Click's `runner.isolated_filesystem()` mechanism doesn't play well with `pytest-xdist`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,
https://github.com/simonw/datasette/issues/1406#issuecomment-889553052,https://api.github.com/repos/simonw/datasette/issues/1406,889553052,IC_kwDOBm6k_c41BYCc,9599,2021-07-30T00:58:43Z,2021-07-30T00:58:43Z,OWNER,Tests are still failing in the job that calculates coverage.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,
https://github.com/simonw/datasette/issues/1406#issuecomment-889555977,https://api.github.com/repos/simonw/datasette/issues/1406,889555977,IC_kwDOBm6k_c41BYwJ,9599,2021-07-30T01:06:57Z,2021-07-30T01:06:57Z,OWNER,"Looking at the source code in Click for `isolated_filesystem()`: https://github.com/pallets/click/blob/9da166957f5848b641231d485467f6140bca2bc0/src/click/testing.py#L450-L468
```python
@contextlib.contextmanager
def isolated_filesystem(
self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None
) -> t.Iterator[str]:
""""""A context manager that creates a temporary directory and
changes the current working directory to it. This isolates tests
that affect the contents of the CWD to prevent them from
interfering with each other.
:param temp_dir: Create the temporary directory under this
directory. If given, the created directory is not removed
when exiting.
.. versionchanged:: 8.0
Added the ``temp_dir`` parameter.
""""""
cwd = os.getcwd()
t = tempfile.mkdtemp(dir=temp_dir)
os.chdir(t)
```
How about if I pass in that optional `temp_dir` as a temp directory created using the `pytest-xdist` aware pytest mechanisms: https://docs.pytest.org/en/6.2.x/tmpdir.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,
https://github.com/simonw/datasette/issues/1406#issuecomment-890259755,https://api.github.com/repos/simonw/datasette/issues/1406,890259755,IC_kwDOBm6k_c41EEkr,9599,2021-07-31T00:04:54Z,2021-07-31T00:04:54Z,OWNER,"STILL failing. I'm going to try removing all instances of `isolated_filesystem()` in favour of a different pattern using pytest temporary files, then see if I can get that to work without the serial hack. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,
https://github.com/simonw/datasette/issues/1407#issuecomment-890388200,https://api.github.com/repos/simonw/datasette/issues/1407,890388200,IC_kwDOBm6k_c41Ej7o,9599,2021-07-31T18:38:41Z,2021-07-31T18:38:41Z,OWNER,"The `path` variable there looked like this:
`/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-696/popen-gw0/uds0/datasette.sock`
I think what's happening here is that `pytest-xdist` causes `tmp_path_factory.mktemp(""uds"")` to create significantly longer paths, which in this case is breaking some limit.
So for this code to work with `pytest-xdist` I need to make sure the random path to `datasette.sock` is shorter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957298475,
https://github.com/simonw/datasette/issues/1407#issuecomment-890388656,https://api.github.com/repos/simonw/datasette/issues/1407,890388656,IC_kwDOBm6k_c41EkCw,9599,2021-07-31T18:42:41Z,2021-07-31T18:42:41Z,OWNER,I'll try `tempfile.gettempdir()` - on macOS it returns something like `'/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T'` which is still long but hopefully not too long.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957298475,
https://github.com/simonw/datasette/issues/1406#issuecomment-890390198,https://api.github.com/repos/simonw/datasette/issues/1406,890390198,IC_kwDOBm6k_c41Eka2,9599,2021-07-31T18:55:33Z,2021-07-31T18:55:33Z,OWNER,"To clarify: the core problem here is that an error is thrown any time you call `os.getcwd()` but the directory you are currently in has been deleted.
`runner.isolated_filesystem()` assumes that the current directory in has not been deleted. But the various temporary directory utilities in `pytest` work by creating directories and then deleting them.
Maybe there's a larger problem here that I play a bit fast and loose with `os.chdir()` in both the test suite and in various lines of code in Datasette itself (in particular in the publish commands)?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,
https://github.com/simonw/datasette/issues/1406#issuecomment-890390342,https://api.github.com/repos/simonw/datasette/issues/1406,890390342,IC_kwDOBm6k_c41EkdG,9599,2021-07-31T18:56:35Z,2021-07-31T18:56:35Z,OWNER,"But... I've lost enough time to this already, and removing `runner.isolated_filesystem()` has the tests passing again. So I'm not going to work on this any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,
https://github.com/simonw/datasette/issues/1408#issuecomment-890390495,https://api.github.com/repos/simonw/datasette/issues/1408,890390495,IC_kwDOBm6k_c41Ekff,9599,2021-07-31T18:57:39Z,2021-07-31T18:57:39Z,OWNER,Opening this issue as an optional follow-up to the work I did in #1406.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957302085,
https://github.com/simonw/datasette/issues/1408#issuecomment-890390845,https://api.github.com/repos/simonw/datasette/issues/1408,890390845,IC_kwDOBm6k_c41Ekk9,9599,2021-07-31T19:00:32Z,2021-07-31T19:00:32Z,OWNER,"When I revisit this I can also look at dropping the `@pytest.mark.serial` hack, and maybe the `restore_working_directory()` fixture hack too:
https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/pytest.ini#L9-L10
https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/tests/conftest.py#L62-L75","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957302085,
https://github.com/simonw/datasette/issues/1409#issuecomment-890400059,https://api.github.com/repos/simonw/datasette/issues/1409,890400059,IC_kwDOBm6k_c41Em07,9599,2021-07-31T20:21:51Z,2021-07-31T20:21:51Z,OWNER,"One of these two options:
- `--setting default_allow_sql off`
- `--setting allow_sql_default off`
Existing settings from https://docs.datasette.io/en/0.58.1/settings.html with similar names that I need to be consistent with:
- `default_page_size`
- `allow_facet`
- `default_facet_size`
- `allow_download`
- `default_cache_ttl`
- `default_cache_ttl_hashed`
- `allow_csv_stream`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,
https://github.com/simonw/datasette/issues/1409#issuecomment-890400121,https://api.github.com/repos/simonw/datasette/issues/1409,890400121,IC_kwDOBm6k_c41Em15,9599,2021-07-31T20:22:21Z,2021-07-31T20:23:34Z,OWNER,"I think `default_allow_sql` is more consistent with the current naming conventions, because both `allow` and `default` are used as prefixes at the moment but neither of them are ever used as a suffix.
Plus `default_allow_sql off` makes sense to me but `allow_default_sql off` does not - what is ""default SQL""?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,
https://github.com/simonw/datasette/issues/1409#issuecomment-890400425,https://api.github.com/repos/simonw/datasette/issues/1409,890400425,IC_kwDOBm6k_c41Em6p,9599,2021-07-31T20:25:16Z,2021-07-31T20:26:25Z,OWNER,"If I was prone to over-thinking (which I am) I'd note that `allow_facet` and `allow_download` and `allow_csv_stream` are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent.
So maybe there's a future task where I introduce those as both permissions and metadata `""allow_x""` blocks, then rename the settings themselves to be called `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`.
If I was going to do that I should get it in before Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,
https://github.com/simonw/datasette/issues/1409#issuecomment-890397124,https://api.github.com/repos/simonw/datasette/issues/1409,890397124,IC_kwDOBm6k_c41EmHE,9599,2021-07-31T19:51:10Z,2021-07-31T19:51:10Z,OWNER,"I think I may like `disable_sql` better. Some options:
- `--setting allow_sql off` (consistent with `allow_facet` and `allow_download` and `allow_csv_stream` - all which default to `on` already)
- `--setting disable_sql on`
- `--setting disable_custom_sql on`
The existence of three `allow_*` settings does make a strong argument for staying consistent with that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,
https://github.com/simonw/datasette/issues/1409#issuecomment-890397169,https://api.github.com/repos/simonw/datasette/issues/1409,890397169,IC_kwDOBm6k_c41EmHx,9599,2021-07-31T19:51:35Z,2021-07-31T19:51:35Z,OWNER,I'm going to stick with `--setting allow_sql off`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,
https://github.com/simonw/datasette/issues/1409#issuecomment-890397261,https://api.github.com/repos/simonw/datasette/issues/1409,890397261,IC_kwDOBm6k_c41EmJN,9599,2021-07-31T19:52:25Z,2021-07-31T19:52:25Z,OWNER,I think I can make this modification by teaching the default permissions code here to take the `allow_sql` setting into account: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/datasette/default_permissions.py#L38-L45,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,
https://github.com/simonw/datasette/issues/1409#issuecomment-890397652,https://api.github.com/repos/simonw/datasette/issues/1409,890397652,IC_kwDOBm6k_c41EmPU,9599,2021-07-31T19:56:48Z,2021-07-31T19:56:48Z,OWNER,"The other option would be to use the setting to pick the `default=` argument when calling `self.ds.permission_allowed( request.actor, ""execute-sql"", resource=database, default=True)`.
The problem with that is that there are actually a few different places which perform that check, so changing all of them raises the risk of missing one in the future:
https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L436-L444
https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L964-L966
https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L220-L221
https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L343-L345
https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L134-L136
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,
https://github.com/simonw/datasette/issues/1409#issuecomment-890397753,https://api.github.com/repos/simonw/datasette/issues/1409,890397753,IC_kwDOBm6k_c41EmQ5,9599,2021-07-31T19:57:56Z,2021-07-31T19:57:56Z,OWNER,"I think the correct solution is for the default permissions logic to take the `allow_sql` setting into account, and to return `False` if that setting is set to `off` AND the current actor fails the `actor_matches_allow` checks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,
https://github.com/simonw/datasette/issues/1409#issuecomment-890399806,https://api.github.com/repos/simonw/datasette/issues/1409,890399806,IC_kwDOBm6k_c41Emw-,9599,2021-07-31T20:18:46Z,2021-07-31T20:18:46Z,OWNER,"My rationale for removing it: https://github.com/simonw/datasette/issues/813#issuecomment-640916290
> Naming problem: Datasette already has a config option with this name:
>
> $ datasette serve data.db --config allow_sql:1
>
> https://datasette.readthedocs.io/en/stable/config.html#allow-sql
>
> It's confusing to have two things called `allow_sql` that do slightly different things.
>
> I could retire the `--config allow_sql:0` option entirely, since the new `metadata.json` mechanism can be used to achieve the exact same thing.
>
> I'm going to do that.
This is true. The `""allow_sql""` permissions block in `metadata.json` does indeed have a name that is easily confused with `--setting allow_sql off`.
So I definitely need to pick a different name from the setting. `--setting default_allow_sql off` is a good option here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,
https://github.com/simonw/datasette/issues/1411#issuecomment-890441844,https://api.github.com/repos/simonw/datasette/issues/1411,890441844,IC_kwDOBm6k_c41ExB0,9599,2021-08-01T03:27:30Z,2021-08-01T03:27:30Z,OWNER,Confirmed: https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1 no longer exhibits the bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957345476,
https://github.com/simonw/datasette/issues/1227#issuecomment-891352132,https://api.github.com/repos/simonw/datasette/issues/1227,891352132,IC_kwDOBm6k_c41IPRE,9599,2021-08-02T21:38:39Z,2021-08-02T21:38:39Z,OWNER,Relevant TIL: https://til.simonwillison.net/vscode/vs-code-regular-expressions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810394616,
https://github.com/simonw/datasette/issues/1417#issuecomment-891979858,https://api.github.com/repos/simonw/datasette/issues/1417,891979858,IC_kwDOBm6k_c41KohS,9599,2021-08-03T16:15:30Z,2021-08-03T16:15:30Z,OWNER,"Docs: https://pypi.org/project/codespell/
There's a `codespell --ignore-words=FILE` option for ignoring words. I don't have any that need ignoring yet but I'm going to add that file anyway, that way I can have codespell be a failing test but still provide a way to work around it if it incorrectly flags a correct word.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959278472,
https://github.com/simonw/datasette/pull/1418#issuecomment-891984359,https://api.github.com/repos/simonw/datasette/issues/1418,891984359,IC_kwDOBm6k_c41Kpnn,9599,2021-08-03T16:21:56Z,2021-08-03T16:21:56Z,OWNER,"Failed with:
```
docs/authentication.rst:63: perfom ==> perform
docs/authentication.rst:76: perfom ==> perform
docs/changelog.rst:429: repsonse ==> response
docs/changelog.rst:503: permissons ==> permissions
docs/changelog.rst:717: compatibilty ==> compatibility
docs/changelog.rst:1172: browseable ==> browsable
docs/deploying.rst:191: similiar ==> similar
docs/internals.rst:434: Respons ==> Response, respond
docs/internals.rst:440: Respons ==> Response, respond
docs/internals.rst:717: tha ==> than, that, the
docs/performance.rst:42: databse ==> database
docs/plugin_hooks.rst:667: utilites ==> utilities
docs/publish.rst:168: countainer ==> container
docs/settings.rst:352: inalid ==> invalid
docs/sql_queries.rst:406: preceeded ==> preceded, proceeded
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959284434,
https://github.com/simonw/datasette/issues/1420#issuecomment-892365639,https://api.github.com/repos/simonw/datasette/issues/1420,892365639,IC_kwDOBm6k_c41MGtH,9599,2021-08-04T05:05:07Z,2021-08-04T05:05:07Z,OWNER,https://github.com/simonw/datasette/blob/cd8b7bee8fb5c1cdce7c8dbfeb0166011abc72c6/datasette/publish/cloudrun.py#L153-L158,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,
https://github.com/simonw/datasette/issues/1420#issuecomment-892372509,https://api.github.com/repos/simonw/datasette/issues/1420,892372509,IC_kwDOBm6k_c41MIYd,9599,2021-08-04T05:22:29Z,2021-08-04T05:22:29Z,OWNER,"Testing this manually with:
datasette publish cloudrun fixtures.db --memory 8G --cpu 4 \
--service fixtures-over-provisioned-issue-1420 --install datasette-psutil
And for comparison:
datasette publish cloudrun fixtures.db --service fixtures-default-issue-1420 \
--install datasette-psutil
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,
https://github.com/simonw/datasette/issues/1420#issuecomment-892374253,https://api.github.com/repos/simonw/datasette/issues/1420,892374253,IC_kwDOBm6k_c41MIzt,9599,2021-08-04T05:27:21Z,2021-08-04T05:29:59Z,OWNER,"I'll delete these services shortly afterwards. Right now:
https://fixtures-over-provisioned-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil (deployed with `--memory 8G --cpu 4`) returns:
```
process.memory_info()
pmem(rss=60456960, vms=518930432, shared=0, text=0, lib=0, data=0, dirty=0)
...
psutil.cpu_times(True)
scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
psutil.virtual_memory()
svmem(total=2147483648, available=2092531712, percent=2.6, used=33103872, free=2092531712, active=44130304, inactive=10792960, buffers=0, cached=21848064, shared=262144, slab=0)
```
https://fixtures-default-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil returns:
```
process.memory_info()
pmem(rss=49324032, vms=140595200, shared=0, text=0, lib=0, data=0, dirty=0)
...
psutil.cpu_times(True)
scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
psutil.virtual_memory()
svmem(total=2147483648, available=2091188224, percent=2.6, used=40071168, free=2091188224, active=41586688, inactive=7983104, buffers=0, cached=16224256, shared=262144, slab=0)
```
These numbers are different enough that I assume this works as advertised. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,
https://github.com/simonw/datasette/issues/1420#issuecomment-892376353,https://api.github.com/repos/simonw/datasette/issues/1420,892376353,IC_kwDOBm6k_c41MJUh,9599,2021-08-04T05:33:12Z,2021-08-04T05:33:12Z,OWNER,"In the Cloud Run console (before I deleted these services) when I click ""Edit and deploy new revision"" I see this for the default one:
And this for the big one:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,
https://github.com/simonw/datasette/issues/1420#issuecomment-893079520,https://api.github.com/repos/simonw/datasette/issues/1420,893079520,IC_kwDOBm6k_c41O0_g,9599,2021-08-05T00:54:59Z,2021-08-05T00:54:59Z,OWNER,"Just saw this error:
`ERROR: (gcloud.run.deploy) The `--cpu` flag is not supported on the fully managed version of Cloud Run. Specify `--platform gke` or run `gcloud config set run/platform gke` to work with Cloud Run for Anthos deployed on Google Cloud.`
Which is weird because I managed to run this successfully the other day. Maybe a region difference thing?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,
https://github.com/simonw/datasette/issues/1422#issuecomment-893122356,https://api.github.com/repos/simonw/datasette/issues/1422,893122356,IC_kwDOBm6k_c41O_c0,9599,2021-08-05T02:52:31Z,2021-08-05T02:52:44Z,OWNER,"If you do this it should still be possible to view the SQL - which means we need a new parameter. I propose `?_show_sql=1` to over-ride the hidden default.
I think the configuration should use `hide_sql: true` - looking like this:
```yaml
databases:
fixtures:
queries:
neighborhood_search:
hide_sql: true
sql: |-
select neighborhood, facet_cities.name, state
from facetable
join facet_cities on facetable.city_id = facet_cities.id
where neighborhood like '%' || :text || '%'
order by neighborhood
title: Search neighborhoods
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,
https://github.com/simonw/datasette/issues/1422#issuecomment-893131703,https://api.github.com/repos/simonw/datasette/issues/1422,893131703,IC_kwDOBm6k_c41PBu3,9599,2021-08-05T03:16:46Z,2021-08-05T03:16:46Z,OWNER,"The logic for this is a little bit fiddly, due to the need to switch to using `?_show_sql=1` on the link depending on the context.
- If metadata says hide and there's no query string, hide and link to `?_show_sql=1`
- If metadata says hide but query string says `?_show_sql=1`, show and have hide link linking to URL without `?_show_sql=1`
- Otherwise, show and link to `?_hide_sql=1`
- ... or if that query string is there then hide and link to URL without `?_hide_sql=1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,
https://github.com/simonw/datasette/issues/1419#issuecomment-893133496,https://api.github.com/repos/simonw/datasette/issues/1419,893133496,IC_kwDOBm6k_c41PCK4,9599,2021-08-05T03:22:44Z,2021-08-05T03:22:44Z,OWNER,"I ran into this exact same problem today! I only just learned how to use filter on aggregates: https://til.simonwillison.net/sqlite/sqlite-aggregate-filter-clauses
A workaround I used is to add this to the deploy command:
datasette publish cloudrun ... --install=pysqlite3-binary
This will install the https://pypi.org/project/pysqlite3-binary for package which bundles a more recent SQLite version.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959710008,
https://github.com/simonw/datasette/issues/1423#issuecomment-893996604,https://api.github.com/repos/simonw/datasette/issues/1423,893996604,IC_kwDOBm6k_c41SU48,9599,2021-08-06T04:43:07Z,2021-08-06T04:43:37Z,OWNER,"Problem: on a page which doesn't have quite enough facet values to trigger the display of the ""..."" link that links to `?_facet_size=max` the user would still have to manually count the values - up to 30 by default.
So maybe the count should always be shown, perhaps as a non-bold light colored number?
I could even hide it in a non-discoverable tooltip.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,
https://github.com/simonw/datasette/issues/1423#issuecomment-894452990,https://api.github.com/repos/simonw/datasette/issues/1423,894452990,IC_kwDOBm6k_c41UET-,9599,2021-08-06T18:49:37Z,2021-08-06T18:49:37Z,OWNER,"Could display them always, like this:
That's with `23`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,
https://github.com/simonw/datasette/issues/1423#issuecomment-894453520,https://api.github.com/repos/simonw/datasette/issues/1423,894453520,IC_kwDOBm6k_c41UEcQ,9599,2021-08-06T18:50:40Z,2021-08-06T18:50:40Z,OWNER,"Point of confusion: if only 30 options are shown, but there's a `...` at the end, what would the number be? It can't be the total number of facets because we haven't counted them all - but if it's just the number of displayed facets that's like to be confusing.
So the original idea of showing the counts only if `_facet_size=max` is a good one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,
https://github.com/simonw/datasette/issues/1423#issuecomment-894454087,https://api.github.com/repos/simonw/datasette/issues/1423,894454087,IC_kwDOBm6k_c41UElH,9599,2021-08-06T18:51:42Z,2021-08-06T18:51:42Z,OWNER,"The invisible tooltip could say ""Showing 30 items, more available"" (helping save you from counting up to 20 if you know about the secret feature). The numbers could then be fully displayed on the ""..."" page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,
https://github.com/simonw/datasette/issues/1423#issuecomment-894454644,https://api.github.com/repos/simonw/datasette/issues/1423,894454644,IC_kwDOBm6k_c41UEt0,9599,2021-08-06T18:52:49Z,2021-08-06T18:52:49Z,OWNER,"This means that the counts would be unavailable to users who cannot see tooltips (e.g. mobile users) on pages that did not have any facets that broke the 30 limit and hence displayed that ""..."" link.
I think I'm OK with that, for the moment. May revisit in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,
https://github.com/simonw/datasette/issues/1422#issuecomment-894589140,https://api.github.com/repos/simonw/datasette/issues/1422,894589140,IC_kwDOBm6k_c41UljU,9599,2021-08-07T01:58:16Z,2021-08-07T01:58:24Z,OWNER,Also need to consider this hidden field - it should pass the `_hide_sql` or `_show_sql` parameters depending on the same logic: https://github.com/simonw/datasette/blob/acc22436622ff8476c30acf45ed60f54b4aaa5d9/datasette/templates/query.html#L47-L49,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,
https://github.com/simonw/datasette/issues/1421#issuecomment-894606843,https://api.github.com/repos/simonw/datasette/issues/1421,894606843,IC_kwDOBm6k_c41Up37,9599,2021-08-07T05:17:12Z,2021-08-07T05:17:12Z,OWNER,Marking this blocked because I don't have a way around the needing-a-SQLite-SQL-parser problem at the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,
https://github.com/simonw/datasette/issues/1421#issuecomment-894606796,https://api.github.com/repos/simonw/datasette/issues/1421,894606796,IC_kwDOBm6k_c41Up3M,9599,2021-08-07T05:16:39Z,2021-08-07T05:16:39Z,OWNER,"Urgh, yeah I've seen this one before. Fixing it pretty much requires writing a full SQLite SQL syntax parser in Python, which is frustratingly complicated for solving this issue!
You can work around this for a canned query by using the optional `params:` argument documented here: https://docs.datasette.io/en/stable/sql_queries.html#canned-query-parameters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,
https://github.com/simonw/datasette/issues/1422#issuecomment-894607989,https://api.github.com/repos/simonw/datasette/issues/1422,894607989,IC_kwDOBm6k_c41UqJ1,9599,2021-08-07T05:31:57Z,2021-08-07T05:31:57Z,OWNER,"Demo: https://latest.datasette.io/fixtures/neighborhood_search
Documentation: https://docs.datasette.io/en/latest/sql_queries.html#additional-canned-query-options","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,
https://github.com/simonw/datasette/issues/1421#issuecomment-894922145,https://api.github.com/repos/simonw/datasette/issues/1421,894922145,IC_kwDOBm6k_c41V22h,9599,2021-08-09T03:07:38Z,2021-08-09T03:07:38Z,OWNER,"I hoped this would work:
```sql
with foo as (
explain select * from facetable
where state = :state
and on_earth = :on_earth
and neighborhood not like '00:04'
)
select p4 from foo where opcode = 'Variable'
```
But sadly [it returns an error](https://latest.datasette.io/fixtures?sql=with+foo+as+%28%0D%0A++explain+select+*+from+facetable%0D%0A++where+state+%3D+%3Astate%0D%0A++and+on_earth+%3D+%3Aon_earth%0D%0A++and+neighborhood+not+like+%2700%3A04%27%0D%0A%29%0D%0Aselect+p4+from+foo+where+opcode+%3D+%27Variable%27&state=&on_earth=&04=):
> near ""explain"": syntax error","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,
https://github.com/simonw/datasette/issues/1421#issuecomment-894922703,https://api.github.com/repos/simonw/datasette/issues/1421,894922703,IC_kwDOBm6k_c41V2_P,9599,2021-08-09T03:09:29Z,2021-08-09T03:09:29Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/ad90a72afa21b737b162e2bbdddc301a97d575cd/datasette/views/database.py#L225-L231,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,
https://github.com/simonw/datasette/issues/1421#issuecomment-894921512,https://api.github.com/repos/simonw/datasette/issues/1421,894921512,IC_kwDOBm6k_c41V2so,9599,2021-08-09T03:05:26Z,2021-08-09T03:05:26Z,OWNER,"I may have a way to work around this, using `explain`. Consider this query:
```sql
select * from facetable
where state = :state
and on_earth = :on_earth
and neighborhood not like '00:04'
```
Datasette currently gets confused and shows three form fields: https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04=
But... if I run `explain` [against that](https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04=) I get this (truncated):
addr | opcode | p1 | p2 | p3 | p4 | p5 | comment
-- | -- | -- | -- | -- | -- | -- | --
20 | ResultRow | 6 | 10 | 0 | | 0 |
21 | Next | 0 | 3 | 0 | | 1 |
22 | Halt | 0 | 0 | 0 | | 0 |
23 | Transaction | 0 | 0 | 35 | 0 | 1 |
24 | Variable | 1 | 2 | 0 | :state | 0 |
25 | Variable | 2 | 3 | 0 | :on_earth | 0 |
26 | String8 | 0 | 4 | 0 | 00:04 | 0 |
27 | Goto | 0 | 1 | 0 | | 0 |
Could it be as simple as pulling out those `Variable` rows to figure out the names of the variables in the query?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,
https://github.com/simonw/datasette/issues/1421#issuecomment-894925437,https://api.github.com/repos/simonw/datasette/issues/1421,894925437,IC_kwDOBm6k_c41V3p9,9599,2021-08-09T03:19:00Z,2021-08-09T03:19:00Z,OWNER,"This may not work:
> `ERROR: sql = 'explain select 1 + :one + :two', params = None: You did not supply a value for binding 1.`
The `explain` queries themselves want me to pass them parameters.
I could try using the regex to pull out candidates and passing `None` for each of those, including incorrect ones like `:31`.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,
https://github.com/simonw/datasette/issues/1421#issuecomment-894925914,https://api.github.com/repos/simonw/datasette/issues/1421,894925914,IC_kwDOBm6k_c41V3xa,9599,2021-08-09T03:20:42Z,2021-08-09T03:20:42Z,OWNER,"I think this works!
```python
_re_named_parameter = re.compile("":([a-zA-Z0-9_]+)"")
async def derive_named_parameters(db, sql):
explain = 'explain {}'.format(sql.strip().rstrip("";""))
possible_params = _re_named_parameter.findall(sql)
try:
results = await db.execute(explain, {p: None for p in possible_params})
return [row[""p4""].lstrip("":"") for row in results if row[""opcode""] == ""Variable""]
except sqlite3.DatabaseError:
return []
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,
https://github.com/simonw/datasette/issues/1421#issuecomment-894927185,https://api.github.com/repos/simonw/datasette/issues/1421,894927185,IC_kwDOBm6k_c41V4FR,9599,2021-08-09T03:25:01Z,2021-08-09T03:25:01Z,OWNER,"One catch with this approach: if the SQL query is invalid, the parameters will not be extracted and shown as form fields.
Maybe that's completely fine? Why display a form if it's going to break when the user actually runs the query?
But it does bother me. I worry that someone who is still iterating on and editing their query before actually starting to use it might find the behaviour confusing.
So maybe if the query raises an exception it could fall back on the regular expression results?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,
https://github.com/simonw/datasette/issues/1421#issuecomment-894929080,https://api.github.com/repos/simonw/datasette/issues/1421,894929080,IC_kwDOBm6k_c41V4i4,9599,2021-08-09T03:33:02Z,2021-08-09T03:33:02Z,OWNER,"Fixed! Fantastic, this one has been bothering me for *years*.
https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,