html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/pull/2118#issuecomment-1652681136,https://api.github.com/repos/simonw/datasette/issues/2118,1652681136,IC_kwDOBm6k_c5igemw,9599,simonw,2023-07-26T23:30:44Z,2023-07-26T23:30:44Z,OWNER,"The `_shape=` stuff should use `json_renderer` instead - that's how the table view did it:
https://github.com/simonw/datasette/commit/d97e82df3c8a3f2e97038d7080167be9bb74a68d#diff-5c9ef29c33ed0fde413565b23fa258d60fc3a2bb205b016db9e915c9bd5ecfb3
https://github.com/simonw/datasette/blob/c3e3ecf590ca5fa61b00aba4c78599e33d370b60/datasette/views/table.py#L822-L850
Instead of:
https://github.com/simonw/datasette/blob/c3e3ecf590ca5fa61b00aba4c78599e33d370b60/datasette/views/database.py#L239-L288","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views,
https://github.com/simonw/sqlite-utils/issues/581#issuecomment-1652496702,https://api.github.com/repos/simonw/sqlite-utils/issues/581,1652496702,IC_kwDOCGYnMM5ifxk-,9599,simonw,2023-07-26T21:07:45Z,2023-07-26T21:07:45Z,OWNER,Docs: https://sqlite-utils.datasette.io/en/latest/cli.html#using-the-debugger,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823160748,`sqlite-utils convert --pdb` option,
https://github.com/simonw/datasette/issues/2111#issuecomment-1652407208,https://api.github.com/repos/simonw/datasette/issues/2111,1652407208,IC_kwDOBm6k_c5ifbuo,9599,simonw,2023-07-26T19:54:20Z,2023-07-26T19:54:20Z,OWNER,"I implemented `/content` and `/content.json` but I left `/content?sql=...` as this:
https://github.com/simonw/datasette/blob/2e40a506ad45b44fd7642474f630a31ef18b5911/datasette/views/database.py#L220-L221","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822937426,Implement new /content.json?sql=...,
https://github.com/simonw/datasette/issues/2111#issuecomment-1652354606,https://api.github.com/repos/simonw/datasette/issues/2111,1652354606,IC_kwDOBm6k_c5ifO4u,9599,simonw,2023-07-26T19:16:35Z,2023-07-26T19:17:45Z,OWNER,"I just noticed that this URL: https://latest.datasette.io/fixtures.csv
Returns a 500 error right now!
![]()
It's fine with a `?sql=` query: https://latest.datasette.io/fixtures.csv?sql=select+*+from+facetable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822937426,Implement new /content.json?sql=...,
https://github.com/simonw/datasette/issues/2110#issuecomment-1652330111,https://api.github.com/repos/simonw/datasette/issues/2110,1652330111,IC_kwDOBm6k_c5ifI5_,9599,simonw,2023-07-26T18:55:31Z,2023-07-26T18:55:31Z,OWNER,Changed my mind on this - I'm going to have the `query_view` mapped to `/db` but have the first code on there notice if `?sql=` is missing and return a `database_view()` function instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822936521,Merge database index page and query view,
https://github.com/simonw/datasette/issues/2109#issuecomment-1652325193,https://api.github.com/repos/simonw/datasette/issues/2109,1652325193,IC_kwDOBm6k_c5ifHtJ,9599,simonw,2023-07-26T18:51:15Z,2023-07-26T18:51:15Z,OWNER,"Here's the code I'm going to be entirely replacing:
https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/database.py#L213-L530
Plus this weird class in `views/table.py`:
https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/table.py#L701-L749","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working,
https://github.com/simonw/datasette/issues/2114#issuecomment-1652321419,https://api.github.com/repos/simonw/datasette/issues/2114,1652321419,IC_kwDOBm6k_c5ifGyL,9599,simonw,2023-07-26T18:48:03Z,2023-07-26T18:48:03Z,OWNER,"This is also where I'll bring back writable canned queries:
https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/database.py#L281-L334","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822940263,Implement canned queries against new query JSON work,
https://github.com/simonw/datasette/issues/2111#issuecomment-1652318269,https://api.github.com/repos/simonw/datasette/issues/2111,1652318269,IC_kwDOBm6k_c5ifGA9,9599,simonw,2023-07-26T18:45:23Z,2023-07-26T18:45:23Z,OWNER,"To avoid confusion I'll start by having `/content` return a HTML ""TODO: implement this"" message.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822937426,Implement new /content.json?sql=...,
https://github.com/simonw/datasette/issues/2116#issuecomment-1652315926,https://api.github.com/repos/simonw/datasette/issues/2116,1652315926,IC_kwDOBm6k_c5ifFcW,9599,simonw,2023-07-26T18:43:17Z,2023-07-26T18:43:17Z,OWNER,"Tests pass, and manually tested like this too:
```bash
datasette -i pelicans.db
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822949756,Turn DatabaseDownload into an async view function,
https://github.com/simonw/datasette/issues/2116#issuecomment-1652304289,https://api.github.com/repos/simonw/datasette/issues/2116,1652304289,IC_kwDOBm6k_c5ifCmh,9599,simonw,2023-07-26T18:33:07Z,2023-07-26T18:33:07Z,OWNER,"This runs three tests:
```bash
pytest -k test_database_download
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822949756,Turn DatabaseDownload into an async view function,
https://github.com/simonw/datasette/issues/2116#issuecomment-1652303471,https://api.github.com/repos/simonw/datasette/issues/2116,1652303471,IC_kwDOBm6k_c5ifCZv,9599,simonw,2023-07-26T18:32:24Z,2023-07-26T18:32:24Z,OWNER,https://github.com/simonw/datasette/blob/dc5171eb1b1d9f1d55e367f8a4d93edb55a43351/datasette/views/database.py#L172-L212,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822949756,Turn DatabaseDownload into an async view function,
https://github.com/simonw/datasette/issues/2106#issuecomment-1652298879,https://api.github.com/repos/simonw/datasette/issues/2106,1652298879,IC_kwDOBm6k_c5ifBR_,9599,simonw,2023-07-26T18:28:33Z,2023-07-26T18:28:33Z,OWNER,"Applied the same fix as here:
- https://github.com/simonw/llm/issues/136","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816857442,`datasette install -e` option,
https://github.com/simonw/datasette/pull/2053#issuecomment-1652296467,https://api.github.com/repos/simonw/datasette/issues/2053,1652296467,IC_kwDOBm6k_c5ifAsT,9599,simonw,2023-07-26T18:26:44Z,2023-07-26T18:26:44Z,OWNER,"I'm abandoning this branch in favour of a fresh attempt, described here:
- https://github.com/simonw/datasette/issues/2109
I'll copy bits and pieces of this branch across as-needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/issues/2109#issuecomment-1652295866,https://api.github.com/repos/simonw/datasette/issues/2109,1652295866,IC_kwDOBm6k_c5ifAi6,9599,simonw,2023-07-26T18:26:18Z,2023-07-26T18:26:18Z,OWNER,"I'm going to do this work in a fresh branch, copying bits over from here as needed:
- https://github.com/simonw/datasette/pull/2053","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working,
https://github.com/simonw/datasette/issues/2109#issuecomment-1652294920,https://api.github.com/repos/simonw/datasette/issues/2109,1652294920,IC_kwDOBm6k_c5ifAUI,9599,simonw,2023-07-26T18:25:34Z,2023-07-26T18:25:34Z,OWNER,"OK, these issues will do for the plan.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working,
https://github.com/simonw/datasette/pull/2053#issuecomment-1651904060,https://api.github.com/repos/simonw/datasette/issues/2053,1651904060,IC_kwDOBm6k_c5idg48,9599,simonw,2023-07-26T14:19:00Z,2023-07-26T15:25:15Z,OWNER,"Worth noting that the `register_output_renderer()` is actually pretty easy to extend, because it returns a dictionary which could have more keys (like the required set of extras) added to it:
```python
@hookimpl
def register_output_renderer(datasette):
return {
""extension"": ""test"",
""render"": render_demo,
""can_render"": can_render_demo, # Optional
}
```
https://docs.datasette.io/en/0.64.3/plugin_hooks.html#register-output-renderer-datasette
One slight hiccup with that plugin hook is this:
> rows - list of `sqlite3.Row` objects
I could turn that into a Datasette defined object that behaves like a [sqlite3.Row](https://docs.python.org/3/library/sqlite3.html#sqlite3.Row) though, which would give me extra flexibility in the future.
A bit tricky though since it's implemented in C for performance: https://github.com/python/cpython/blob/b0202a4e5d6b629ba5acbc703e950f08ebaf07df/Modules/_sqlite/row.c
Pasted that into Claude for the following explanation:
> - pysqlite_Row is the structure defining the Row object. It contains the tuple of data (self->data) and description of columns (self->description).
> - pysqlite_row_new() is the constructor which creates a new Row object given a cursor and tuple of data.
> - pysqlite_row_dealloc() frees the memory when Row object is deleted.
> - pysqlite_row_keys() returns the column names of the row.
> - pysqlite_row_length() and pysqlite_row_subscript() implement sequence like behavior to access row elements by index.
> - pysqlite_row_subscript() also allows accessing by column name by doing a lookup in description.
> - pysqlite_row_hash() and pysqlite_row_richcompare() implement equality checks and hash function.
I could use protocols in Python to make my own `DatasetteRow` which can be used interchangeably with `sqlite3.Row` - https://docs.python.org/3/library/typing.html#typing.Protocol
Turned this into a TIL: https://til.simonwillison.net/python/protocols","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/pull/2053#issuecomment-1651894668,https://api.github.com/repos/simonw/datasette/issues/2053,1651894668,IC_kwDOBm6k_c5idemM,9599,simonw,2023-07-26T14:14:34Z,2023-07-26T14:14:34Z,OWNER,"Another point of confusion is how `/content` sometimes serves the database index page (with a list of tables) and sometimes solves the results of a query.
I could resolve this by turning the information on the index page into extras, which can optionally be requested any time a query is run but default to being shown if there is no query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/pull/2053#issuecomment-1651883505,https://api.github.com/repos/simonw/datasette/issues/2053,1651883505,IC_kwDOBm6k_c5idb3x,9599,simonw,2023-07-26T14:08:20Z,2023-07-26T14:08:20Z,OWNER,"I think the hardest part of getting this working is dealing with the different formats.
Idea: refactor `.html` as a format (since it's by far the most complex) and tweak the plugin hook a bit as part of that, then use what I learn from that to get the other formats working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/pull/2053#issuecomment-1651874649,https://api.github.com/repos/simonw/datasette/issues/2053,1651874649,IC_kwDOBm6k_c5idZtZ,9599,simonw,2023-07-26T14:03:37Z,2023-07-26T14:03:37Z,OWNER,"Big chunk of commented-out code I just removed:
```python
import pdb
pdb.set_trace()
if isinstance(output, dict) and output.get(""ok"") is False:
# TODO: Other error codes?
response.status_code = 400
if datasette.cors:
add_cors_headers(response.headers)
return response
# registry = Registry(
# extra_count,
# extra_facet_results,
# extra_facets_timed_out,
# extra_suggested_facets,
# facet_instances,
# extra_human_description_en,
# extra_next_url,
# extra_columns,
# extra_primary_keys,
# run_display_columns_and_rows,
# extra_display_columns,
# extra_display_rows,
# extra_debug,
# extra_request,
# extra_query,
# extra_metadata,
# extra_extras,
# extra_database,
# extra_table,
# extra_database_color,
# extra_table_actions,
# extra_filters,
# extra_renderers,
# extra_custom_table_templates,
# extra_sorted_facet_results,
# extra_table_definition,
# extra_view_definition,
# extra_is_view,
# extra_private,
# extra_expandable_columns,
# extra_form_hidden_args,
# )
results = await registry.resolve_multi(
[""extra_{}"".format(extra) for extra in extras]
)
data = {
""ok"": True,
""next"": next_value and str(next_value) or None,
}
data.update(
{
key.replace(""extra_"", """"): value
for key, value in results.items()
if key.startswith(""extra_"") and key.replace(""extra_"", """") in extras
}
)
raw_sqlite_rows = rows[:page_size]
data[""rows""] = [dict(r) for r in raw_sqlite_rows]
private = False
if canned_query:
# Respect canned query permissions
visible, private = await datasette.check_visibility(
request.actor,
permissions=[
(""view-query"", (database, canned_query)),
(""view-database"", database),
""view-instance"",
],
)
if not visible:
raise Forbidden(""You do not have permission to view this query"")
else:
await datasette.ensure_permissions(request.actor, [(""execute-sql"", database)])
# If there's no sql, show the database index page
if not sql:
return await database_index_view(request, datasette, db)
validate_sql_select(sql)
# Extract any :named parameters
named_parameters = named_parameters or await derive_named_parameters(db, sql)
named_parameter_values = {
named_parameter: params.get(named_parameter) or """"
for named_parameter in named_parameters
if not named_parameter.startswith(""_"")
}
# Set to blank string if missing from params
for named_parameter in named_parameters:
if named_parameter not in params and not named_parameter.startswith(""_""):
params[named_parameter] = """"
extra_args = {}
if params.get(""_timelimit""):
extra_args[""custom_time_limit""] = int(params[""_timelimit""])
if _size:
extra_args[""page_size""] = _size
templates = [f""query-{to_css_class(database)}.html"", ""query.html""]
if canned_query:
templates.insert(
0,
f""query-{to_css_class(database)}-{to_css_class(canned_query)}.html"",
)
query_error = None
# Execute query - as write or as read
if write:
raise NotImplementedError(""Write queries not yet implemented"")
# if request.method == ""POST"":
# # If database is immutable, return an error
# if not db.is_mutable:
# raise Forbidden(""Database is immutable"")
# body = await request.post_body()
# body = body.decode(""utf-8"").strip()
# if body.startswith(""{"") and body.endswith(""}""):
# params = json.loads(body)
# # But we want key=value strings
# for key, value in params.items():
# params[key] = str(value)
# else:
# params = dict(parse_qsl(body, keep_blank_values=True))
# # Should we return JSON?
# should_return_json = (
# request.headers.get(""accept"") == ""application/json""
# or request.args.get(""_json"")
# or params.get(""_json"")
# )
# if canned_query:
# params_for_query = MagicParameters(params, request, self.ds)
# else:
# params_for_query = params
# ok = None
# try:
# cursor = await self.ds.databases[database].execute_write(
# sql, params_for_query
# )
# message = metadata.get(
# ""on_success_message""
# ) or ""Query executed, {} row{} affected"".format(
# cursor.rowcount, """" if cursor.rowcount == 1 else ""s""
# )
# message_type = self.ds.INFO
# redirect_url = metadata.get(""on_success_redirect"")
# ok = True
# except Exception as e:
# message = metadata.get(""on_error_message"") or str(e)
# message_type = self.ds.ERROR
# redirect_url = metadata.get(""on_error_redirect"")
# ok = False
# if should_return_json:
# return Response.json(
# {
# ""ok"": ok,
# ""message"": message,
# ""redirect"": redirect_url,
# }
# )
# else:
# self.ds.add_message(request, message, message_type)
# return self.redirect(request, redirect_url or request.path)
# else:
# async def extra_template():
# return {
# ""request"": request,
# ""db_is_immutable"": not db.is_mutable,
# ""path_with_added_args"": path_with_added_args,
# ""path_with_removed_args"": path_with_removed_args,
# ""named_parameter_values"": named_parameter_values,
# ""canned_query"": canned_query,
# ""success_message"": request.args.get(""_success"") or """",
# ""canned_write"": True,
# }
# return (
# {
# ""database"": database,
# ""rows"": [],
# ""truncated"": False,
# ""columns"": [],
# ""query"": {""sql"": sql, ""params"": params},
# ""private"": private,
# },
# extra_template,
# templates,
# )
# Not a write
rows = []
if canned_query:
params_for_query = MagicParameters(params, request, datasette)
else:
params_for_query = params
try:
results = await datasette.execute(
database, sql, params_for_query, truncate=True, **extra_args
)
columns = [r[0] for r in results.description]
rows = list(results.rows)
except sqlite3.DatabaseError as e:
query_error = e
results = None
columns = []
allow_execute_sql = await datasette.permission_allowed(
request.actor, ""execute-sql"", database
)
format_ = request.url_vars.get(""format"") or ""html""
if format_ == ""csv"":
raise NotImplementedError(""CSV format not yet implemented"")
elif format_ in datasette.renderers.keys():
# Dispatch request to the correct output format renderer
# (CSV is not handled here due to streaming)
result = call_with_supported_arguments(
datasette.renderers[format_][0],
datasette=datasette,
columns=columns,
rows=rows,
sql=sql,
query_name=None,
database=db.name,
table=None,
request=request,
view_name=""table"", # TODO: should this be ""query""?
# These will be deprecated in Datasette 1.0:
args=request.args,
data={
""rows"": rows,
}, # TODO what should this be?
)
result = await await_me_maybe(result)
if result is None:
raise NotFound(""No data"")
if isinstance(result, dict):
r = Response(
body=result.get(""body""),
status=result.get(""status_code"") or 200,
content_type=result.get(""content_type"", ""text/plain""),
headers=result.get(""headers""),
)
elif isinstance(result, Response):
r = result
# if status_code is not None:
# # Over-ride the status code
# r.status = status_code
else:
assert False, f""{result} should be dict or Response""
elif format_ == ""html"":
headers = {}
templates = [f""query-{to_css_class(database)}.html"", ""query.html""]
template = datasette.jinja_env.select_template(templates)
alternate_url_json = datasette.absolute_url(
request,
datasette.urls.path(path_with_format(request=request, format=""json"")),
)
headers.update(
{
""Link"": '{}; rel=""alternate""; type=""application/json+datasette""'.format(
alternate_url_json
)
}
)
r = Response.html(
await datasette.render_template(
template,
dict(
data,
append_querystring=append_querystring,
path_with_replaced_args=path_with_replaced_args,
fix_path=datasette.urls.path,
settings=datasette.settings_dict(),
# TODO: review up all of these hacks:
alternate_url_json=alternate_url_json,
datasette_allow_facet=(
""true"" if datasette.setting(""allow_facet"") else ""false""
),
is_sortable=any(c[""sortable""] for c in data[""display_columns""]),
allow_execute_sql=await datasette.permission_allowed(
request.actor, ""execute-sql"", resolved.db.name
),
query_ms=1.2,
select_templates=[
f""{'*' if template_name == template.name else ''}{template_name}""
for template_name in templates
],
),
request=request,
view_name=""table"",
),
headers=headers,
)
else:
assert False, ""Invalid format: {}"".format(format_)
# if next_url:
# r.headers[""link""] = f'<{next_url}>; rel=""next""'
return r
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,