issue_comments
8,358 rows where author_association = "OWNER" sorted by updated_at descending
This data as json, CSV (advanced)
issue >30
- Show column metadata plus links for foreign keys on arbitrary query results 50
- Updated Dockerfile with SpatiaLite version 5.0 45
- Complete refactor of TableView and table.html template 45
- Redesign default .json format 43
- Port Datasette to ASGI 38
- Authentication (and permissions) as a core concept 38
- JavaScript plugin hooks mechanism similar to pluggy 38
- await datasette.client.get(path) mechanism for executing internal requests 33
- Maintain an in-memory SQLite table of connected databases and their tables 31
- Deploy a live instance of demos/apache-proxy 31
- Ability to sort (and paginate) by column 29
- Export to CSV 27
- Optimize all those calls to index_list and foreign_key_list 27
- Ability for a canned query to write to the database 26
- table.transform() method for advanced alter table 26
- Proof of concept for Datasette on AWS Lambda with EFS 25
- Support cross-database joins 24
- Redesign register_output_renderer callback 24
- New pattern for views that return either JSON or HTML, available for plugins 23
- "datasette insert" command and plugin hook 23
- Datasette Plugins 21
- table.extract(...) method and "sqlite-utils extract" command 21
- ?sort=colname~numeric to sort by by column cast to real 21
- Idea: import CSV to memory, run SQL, export in a single command 21
- base_url is omitted in JSON and CSV views 21
- "flash messages" mechanism 20
- Move CI to GitHub Issues 20
- load_template hook doesn't work for include/extends 20
- Better way of representing binary data in .csv output 19
- Introspect if table is FTS4 or FTS5 19
- …
user 1
- simonw 5,552
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
999874886 | https://github.com/simonw/datasette/issues/1576#issuecomment-999874886 | https://api.github.com/repos/simonw/datasette/issues/1576 | IC_kwDOBm6k_c47mOFG | simonw 9599 | 2021-12-22T20:55:42Z | 2021-12-22T20:57:28Z | OWNER | One way to solve this would be to introduce a It would be really nice if I could solve this using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Traces should include SQL executed by subtasks created with `asyncio.gather` 1087181951 | |
999874484 | https://github.com/simonw/datasette/issues/1576#issuecomment-999874484 | https://api.github.com/repos/simonw/datasette/issues/1576 | IC_kwDOBm6k_c47mN-0 | simonw 9599 | 2021-12-22T20:54:52Z | 2021-12-22T20:54:52Z | OWNER | Here's the full current relevant code from |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Traces should include SQL executed by subtasks created with `asyncio.gather` 1087181951 | |
999870993 | https://github.com/simonw/datasette/issues/1518#issuecomment-999870993 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c47mNIR | simonw 9599 | 2021-12-22T20:47:18Z | 2021-12-22T20:50:24Z | OWNER | The reason they aren't showing up in the traces is that traces are stored just for the currently executing This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively "child tasks" of the current request, and hence should be tracked the same. https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
999870282 | https://github.com/simonw/datasette/issues/1518#issuecomment-999870282 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c47mM9K | simonw 9599 | 2021-12-22T20:45:56Z | 2021-12-22T20:46:08Z | OWNER |
I wrote code to execute those in parallel using ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 9808fd2..ec9db64 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import urllib import itertools import json @@ -615,44 +616,37 @@ class TableView(RowTableShared): if request.args.get("_timelimit"): extra_args["custom_time_limit"] = int(request.args.get("_timelimit"))
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
999863269 | https://github.com/simonw/datasette/issues/1518#issuecomment-999863269 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c47mLPl | simonw 9599 | 2021-12-22T20:35:41Z | 2021-12-22T20:37:13Z | OWNER | It looks like the count has to be executed before facets can be, because the facet_class constructor needs that total count figure: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L660-L671 It's used in facet suggestion logic here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/facets.py#L172-L178 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
999850191 | https://github.com/simonw/datasette/issues/1518#issuecomment-999850191 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c47mIDP | simonw 9599 | 2021-12-22T20:29:38Z | 2021-12-22T20:29:38Z | OWNER | New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
999837569 | https://github.com/simonw/datasette/issues/1518#issuecomment-999837569 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c47mE-B | simonw 9599 | 2021-12-22T20:15:45Z | 2021-12-22T20:15:45Z | OWNER | Also the whole |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
999837220 | https://github.com/simonw/datasette/issues/1518#issuecomment-999837220 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c47mE4k | simonw 9599 | 2021-12-22T20:15:04Z | 2021-12-22T20:15:04Z | OWNER | I think I can move this much higher up in the method, it's a bit confusing having it half way through: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L414-L436 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
999831967 | https://github.com/simonw/datasette/issues/1518#issuecomment-999831967 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c47mDmf | simonw 9599 | 2021-12-22T20:04:47Z | 2021-12-22T20:10:11Z | OWNER | I think I might be able to clean up a lot of the stuff in here using the The catch with that hook - https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette - is that it gets called for every single cell. I don't want the overhead of looking up the foreign key relationships etc once for every value in a specific column. But maybe I could extend the hook to include a shared cache that gets used for all of the cells in a specific table? Something like this:
It's a bit of a gross hack though, and would it ever be useful for plugins outside of the default plugin in Datasette which does the foreign key stuff? If I can think of one other potential application for this No, this optimization doesn't make sense: the most complex cell enrichment logic is the stuff that does a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
998354538 | https://github.com/simonw/datasette/pull/1554#issuecomment-998354538 | https://api.github.com/repos/simonw/datasette/issues/1554 | IC_kwDOBm6k_c47ga5q | simonw 9599 | 2021-12-20T23:52:04Z | 2021-12-20T23:52:04Z | OWNER | Abandoning this since it didn't work how I wanted. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
TableView refactor 1079129258 | |
997514220 | https://github.com/simonw/datasette/issues/1547#issuecomment-997514220 | https://api.github.com/repos/simonw/datasette/issues/1547 | IC_kwDOBm6k_c47dNvs | simonw 9599 | 2021-12-20T01:26:25Z | 2021-12-20T01:26:25Z | OWNER | OK, this should hopefully fix that for you:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Writable canned queries fail to load custom templates 1076388044 | |
997513369 | https://github.com/simonw/datasette/issues/1547#issuecomment-997513369 | https://api.github.com/repos/simonw/datasette/issues/1547 | IC_kwDOBm6k_c47dNiZ | simonw 9599 | 2021-12-20T01:24:43Z | 2021-12-20T01:24:43Z | OWNER | @wragge thanks, that's a bug! Working on that in #1575. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Writable canned queries fail to load custom templates 1076388044 | |
997513177 | https://github.com/simonw/datasette/issues/1575#issuecomment-997513177 | https://api.github.com/repos/simonw/datasette/issues/1575 | IC_kwDOBm6k_c47dNfZ | simonw 9599 | 2021-12-20T01:24:25Z | 2021-12-20T01:24:25Z | OWNER | Looks like |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
__call__() got an unexpected keyword argument 'specname' 1084257842 | |
997507074 | https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997507074 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | IC_kwDOCGYnMM47dMAC | simonw 9599 | 2021-12-20T01:10:06Z | 2021-12-20T01:16:11Z | OWNER | Work-in-progress improved help: ``` Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE Insert records from FILE into a table, creating the table if it does not already exist. By default the input is expected to be a JSON array of objects. Or:
You can also use --convert to pass a fragment of Python code that will be used to convert each input. Your Python code will be passed a "row" variable representing the imported row, and can return a modified row. If you are using --lines your code will be passed a "line" variable, and for --all an "all" variable. Options: --pk TEXT Columns to use as the primary key, e.g. id --flatten Flatten nested JSON objects, so {"a": {"b": 1}} becomes {"a_b": 1} --nl Expect newline-delimited JSON -c, --csv Expect CSV input --tsv Expect TSV input --lines Treat each line as a single value called 'line' --all Treat input as a single value called 'all' --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default <TEXT TEXT>... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`sqlite-utils insert --convert` option 1077431957 | |
997508728 | https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997508728 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | IC_kwDOCGYnMM47dMZ4 | simonw 9599 | 2021-12-20T01:14:43Z | 2021-12-20T01:14:43Z | OWNER | (This makes me want |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`sqlite-utils insert --convert` option 1077431957 | |
997502242 | https://github.com/simonw/sqlite-utils/issues/163#issuecomment-997502242 | https://api.github.com/repos/simonw/sqlite-utils/issues/163 | IC_kwDOCGYnMM47dK0i | simonw 9599 | 2021-12-20T00:56:45Z | 2021-12-20T00:56:52Z | OWNER |
I implemented that in: - #251 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: conversions= could take Python functions 706001517 | |
997497262 | https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997497262 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | IC_kwDOCGYnMM47dJmu | simonw 9599 | 2021-12-20T00:40:15Z | 2021-12-20T00:40:15Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`sqlite-utils insert --convert` option 1077431957 | |
997496931 | https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496931 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | IC_kwDOCGYnMM47dJhj | simonw 9599 | 2021-12-20T00:39:14Z | 2021-12-20T00:39:52Z | OWNER | ``` % sqlite-utils insert --help Usage: sqlite-utils insert [OPTIONS] PATH TABLE JSON_FILE Insert records from JSON file into a table, creating the table if it does not already exist. Input should be a JSON array of objects, unless --nl or --csv is used. Options: --pk TEXT Columns to use as the primary key, e.g. id --nl Expect newline-delimited JSON --flatten Flatten nested JSON objects -c, --csv Expect CSV --tsv Expect TSV --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default <TEXT TEXT>... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ``` I can add a bunch of extra help at the top there to explain all of this stuff. That "Input should be a JSON array of objects" bit could be expanded to several paragraphs. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`sqlite-utils insert --convert` option 1077431957 | |
997492872 | https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997492872 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | IC_kwDOCGYnMM47dIiI | simonw 9599 | 2021-12-20T00:23:31Z | 2021-12-20T00:23:31Z | OWNER | I think this should work on JSON, or CSV, or individual lines, or the entire content at once. So I'll require What would |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`sqlite-utils insert --convert` option 1077431957 | |
997486156 | https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997486156 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | IC_kwDOCGYnMM47dG5M | simonw 9599 | 2021-12-19T23:51:02Z | 2021-12-19T23:51:02Z | OWNER | This is going to need a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`sqlite-utils insert --convert` option 1077431957 | |
997485361 | https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997485361 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | IC_kwDOCGYnMM47dGsx | simonw 9599 | 2021-12-19T23:45:30Z | 2021-12-19T23:45:30Z | OWNER | Really interesting example input for this: https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/iOS13.txt - see https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/ |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`sqlite-utils insert --convert` option 1077431957 | |
997474022 | https://github.com/simonw/datasette/issues/1565#issuecomment-997474022 | https://api.github.com/repos/simonw/datasette/issues/1565 | IC_kwDOBm6k_c47dD7m | simonw 9599 | 2021-12-19T22:36:49Z | 2021-12-19T22:37:29Z | OWNER | No way with a tagged template literal to pass an extra database name argument, so instead I need a method that returns a callable that can be used for the tagged template literal for a specific database - or the default database. This could work (bit weird looking though):
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Documented JavaScript variables on different templates made available for plugins 1083657868 | |
997473856 | https://github.com/simonw/datasette/issues/1565#issuecomment-997473856 | https://api.github.com/repos/simonw/datasette/issues/1565 | IC_kwDOBm6k_c47dD5A | simonw 9599 | 2021-12-19T22:35:20Z | 2021-12-19T22:35:20Z | OWNER | Quick prototype of that tagged template
var id = 4;
console.log(query |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Documented JavaScript variables on different templates made available for plugins 1083657868 | |
997472639 | https://github.com/simonw/datasette/issues/1565#issuecomment-997472639 | https://api.github.com/repos/simonw/datasette/issues/1565 | IC_kwDOBm6k_c47dDl_ | simonw 9599 | 2021-12-19T22:25:50Z | 2021-12-19T22:25:50Z | OWNER | Or...
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Documented JavaScript variables on different templates made available for plugins 1083657868 | |
997472509 | https://github.com/simonw/datasette/issues/1565#issuecomment-997472509 | https://api.github.com/repos/simonw/datasette/issues/1565 | IC_kwDOBm6k_c47dDj9 | simonw 9599 | 2021-12-19T22:24:50Z | 2021-12-19T22:24:50Z | OWNER | ... huh, it could even expose a JavaScript function that can be called to execute a SQL query.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Documented JavaScript variables on different templates made available for plugins 1083657868 | |
997472370 | https://github.com/simonw/datasette/issues/1565#issuecomment-997472370 | https://api.github.com/repos/simonw/datasette/issues/1565 | IC_kwDOBm6k_c47dDhy | simonw 9599 | 2021-12-19T22:23:36Z | 2021-12-19T22:23:36Z | OWNER | This should also expose the JSON API endpoints used to execute SQL against this database. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Documented JavaScript variables on different templates made available for plugins 1083657868 | |
997472214 | https://github.com/simonw/datasette/issues/1518#issuecomment-997472214 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c47dDfW | simonw 9599 | 2021-12-19T22:22:08Z | 2021-12-19T22:22:08Z | OWNER | I sketched out a chained SQL builder pattern that might be useful for further tidying up this code - though with the new plugin hook I'm less excited about it than I was: ```python class TableQuery: def init(self, table, columns, pks, is_view=False, prev=None): self.table = table self.columns = columns self.pks = pks self.is_view = is_view self.prev = prev
'select count(*) from facetable where foo = bar and baz = 1'``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
997471672 | https://github.com/simonw/datasette/issues/1547#issuecomment-997471672 | https://api.github.com/repos/simonw/datasette/issues/1547 | IC_kwDOBm6k_c47dDW4 | simonw 9599 | 2021-12-19T22:18:26Z | 2021-12-19T22:18:26Z | OWNER | I released this in an alpha, so you can try out this fix using:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Writable canned queries fail to load custom templates 1076388044 | |
997470633 | https://github.com/simonw/datasette/issues/1566#issuecomment-997470633 | https://api.github.com/repos/simonw/datasette/issues/1566 | IC_kwDOBm6k_c47dDGp | simonw 9599 | 2021-12-19T22:12:00Z | 2021-12-19T22:12:00Z | OWNER | Released another alpha, 0.60a1: https://github.com/simonw/datasette/releases/tag/0.60a1 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Release Datasette 0.60 1083669410 | |
997462604 | https://github.com/simonw/datasette/issues/1545#issuecomment-997462604 | https://api.github.com/repos/simonw/datasette/issues/1545 | IC_kwDOBm6k_c47dBJM | simonw 9599 | 2021-12-19T21:17:08Z | 2021-12-19T21:17:08Z | OWNER | Here's the relevant code: https://github.com/simonw/datasette/blob/4094741c2881c2ada3f3f878b532fdaec7914953/datasette/app.py#L1204-L1219 It's using I need to get myself a proper Windows development environment setup to investigate this one. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Custom pages don't work on windows 1075893249 | |
997462117 | https://github.com/simonw/datasette/issues/1573#issuecomment-997462117 | https://api.github.com/repos/simonw/datasette/issues/1573 | IC_kwDOBm6k_c47dBBl | simonw 9599 | 2021-12-19T21:13:13Z | 2021-12-19T21:13:13Z | OWNER | This might also be the impetus I need to bring the https://datasette.io/plugins/datasette-pretty-traces plugin into Datasette core itself. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Make trace() a documented internal API 1084185188 | |
997460731 | https://github.com/simonw/datasette/issues/1547#issuecomment-997460731 | https://api.github.com/repos/simonw/datasette/issues/1547 | IC_kwDOBm6k_c47dAr7 | simonw 9599 | 2021-12-19T21:02:15Z | 2021-12-19T21:02:15Z | OWNER | Yes, this is a bug. It looks like the problem is with the Is missing this bit of code: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Writable canned queries fail to load custom templates 1076388044 | |
997460061 | https://github.com/simonw/datasette/issues/1570#issuecomment-997460061 | https://api.github.com/repos/simonw/datasette/issues/1570 | IC_kwDOBm6k_c47dAhd | simonw 9599 | 2021-12-19T20:56:54Z | 2021-12-19T20:56:54Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Separate db.execute_write() into three methods 1083921371 | ||
997459958 | https://github.com/simonw/datasette/issues/1555#issuecomment-997459958 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47dAf2 | simonw 9599 | 2021-12-19T20:55:59Z | 2021-12-19T20:55:59Z | OWNER | Closing this issue because I've optimized this a whole bunch, and it's definitely good enough for the moment. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997325189 | https://github.com/simonw/datasette/issues/1555#issuecomment-997325189 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cfmF | simonw 9599 | 2021-12-19T03:55:01Z | 2021-12-19T20:54:51Z | OWNER | It's a bit annoying that the queries no longer show up in the trace at all now, thanks to running in 5fac26aa221a111d7633f2dd92014641f7c0ade9 has the same problem. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997459637 | https://github.com/simonw/datasette/issues/1555#issuecomment-997459637 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47dAa1 | simonw 9599 | 2021-12-19T20:53:46Z | 2021-12-19T20:53:46Z | OWNER | Using #1571 showed me that the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997457790 | https://github.com/simonw/datasette/issues/1566#issuecomment-997457790 | https://api.github.com/repos/simonw/datasette/issues/1566 | IC_kwDOBm6k_c47c_9- | simonw 9599 | 2021-12-19T20:40:50Z | 2021-12-19T20:40:57Z | OWNER | Also release new version of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Release Datasette 0.60 1083669410 | |
997342494 | https://github.com/simonw/datasette/issues/1555#issuecomment-997342494 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cj0e | simonw 9599 | 2021-12-19T07:22:04Z | 2021-12-19T07:22:04Z | OWNER | Another option would be to provide an abstraction that makes it easier to run a group of SQL queries in the same thread at the same time, and have them traced correctly. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997324666 | https://github.com/simonw/datasette/issues/1555#issuecomment-997324666 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cfd6 | simonw 9599 | 2021-12-19T03:47:51Z | 2021-12-19T03:48:09Z | OWNER | Here's a hacked together prototype of running all of that stuff inside a single function passed to ```diff diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index 95055d8..58f9982 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -1,4 +1,5 @@ import textwrap +from datasette.utils import table_column_details async def init_internal_db(db): @@ -70,49 +71,70 @@ async def populate_schema_tables(internal_db, db): "DELETE FROM tables WHERE database_name = ?", [database_name], block=True ) tables = (await db.execute("select * from sqlite_master WHERE type = 'table'")).rows - tables_to_insert = [] - columns_to_delete = [] - columns_to_insert = [] - foreign_keys_to_delete = [] - foreign_keys_to_insert = [] - indexes_to_delete = [] - indexes_to_insert = []
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997324156 | https://github.com/simonw/datasette/issues/1555#issuecomment-997324156 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cfV8 | simonw 9599 | 2021-12-19T03:40:05Z | 2021-12-19T03:40:05Z | OWNER | Using the prototype of this: - https://github.com/simonw/datasette-pretty-traces/issues/5 I'm seeing about 180ms spent running all of these queries on startup! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997321767 | https://github.com/simonw/datasette/issues/1555#issuecomment-997321767 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cewn | simonw 9599 | 2021-12-19T03:10:58Z | 2021-12-19T03:10:58Z | OWNER | I wonder how much overhead there is switching between the Would there be a performance boost if I gathered all of the column/index information in a single function run on the thread using Would be great to understand how much of an impact that would have. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997321653 | https://github.com/simonw/datasette/issues/1555#issuecomment-997321653 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47ceu1 | simonw 9599 | 2021-12-19T03:09:43Z | 2021-12-19T03:09:43Z | OWNER | On that same documentation page I just spotted this:
This makes me nervous to rely on pragma function optimizations in Datasette itself. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997321477 | https://github.com/simonw/datasette/issues/1555#issuecomment-997321477 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cesF | simonw 9599 | 2021-12-19T03:07:33Z | 2021-12-19T03:07:33Z | OWNER | If I want to continue supporting SQLite prior to 3.16.0 (2017-01-02) I'll need this optimization to only kick in with versions that support table-valued PRAGMA functions, while keeping the old That's feasible, but it's a bit more work - and I need to make sure I have robust testing in place for SQLite 3.15.0. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997321327 | https://github.com/simonw/datasette/issues/1555#issuecomment-997321327 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cepv | simonw 9599 | 2021-12-19T03:05:39Z | 2021-12-19T03:05:44Z | OWNER | This caught me out once before in: - https://github.com/simonw/datasette/issues/1276 Turns out Glitch was running SQLite 3.11.0 from 2016-02-15. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997321217 | https://github.com/simonw/datasette/issues/1555#issuecomment-997321217 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47ceoB | simonw 9599 | 2021-12-19T03:04:16Z | 2021-12-19T03:04:16Z | OWNER | One thing to watch out for though, from https://sqlite.org/pragma.html#pragfunc
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997321115 | https://github.com/simonw/datasette/issues/1555#issuecomment-997321115 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cemb | simonw 9599 | 2021-12-19T03:03:12Z | 2021-12-19T03:03:12Z | OWNER | Table columns is a bit harder, because So if that function is available: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_xinfo.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_xinfo%28sqlite_master.name%29+AS+table_xinfo%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997320824 | https://github.com/simonw/datasette/issues/1555#issuecomment-997320824 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47ceh4 | simonw 9599 | 2021-12-19T02:59:57Z | 2021-12-19T03:00:44Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997272328 | https://github.com/simonw/datasette/issues/1566#issuecomment-997272328 | https://api.github.com/repos/simonw/datasette/issues/1566 | IC_kwDOBm6k_c47cSsI | simonw 9599 | 2021-12-18T19:18:01Z | 2021-12-18T19:18:01Z | OWNER | Added some useful new documented internal methods in: - #1570 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Release Datasette 0.60 1083669410 | |
997272223 | https://github.com/simonw/datasette/issues/1555#issuecomment-997272223 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cSqf | simonw 9599 | 2021-12-18T19:17:13Z | 2021-12-18T19:17:13Z | OWNER | That's a good optimization. Still need to deal with the huge flurry of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997267583 | https://github.com/simonw/datasette/issues/1570#issuecomment-997267583 | https://api.github.com/repos/simonw/datasette/issues/1570 | IC_kwDOBm6k_c47cRh_ | simonw 9599 | 2021-12-18T18:46:05Z | 2021-12-18T18:46:12Z | OWNER | This will replace the work done in #1569. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Separate db.execute_write() into three methods 1083921371 | |
997267416 | https://github.com/simonw/datasette/issues/1555#issuecomment-997267416 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cRfY | simonw 9599 | 2021-12-18T18:44:53Z | 2021-12-18T18:45:28Z | OWNER | Rather than adding a
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997266687 | https://github.com/simonw/datasette/issues/1569#issuecomment-997266687 | https://api.github.com/repos/simonw/datasette/issues/1569 | IC_kwDOBm6k_c47cRT_ | simonw 9599 | 2021-12-18T18:41:40Z | 2021-12-18T18:41:40Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
db.execute_write(..., executescript=True) parameter 1083895395 | ||
997266100 | https://github.com/simonw/datasette/issues/1555#issuecomment-997266100 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cRK0 | simonw 9599 | 2021-12-18T18:40:02Z | 2021-12-18T18:40:02Z | OWNER | The implementation of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997262475 | https://github.com/simonw/datasette/issues/1555#issuecomment-997262475 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cQSL | simonw 9599 | 2021-12-18T18:34:18Z | 2021-12-18T18:34:18Z | OWNER | Using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997249563 | https://github.com/simonw/datasette/issues/1569#issuecomment-997249563 | https://api.github.com/repos/simonw/datasette/issues/1569 | IC_kwDOBm6k_c47cNIb | simonw 9599 | 2021-12-18T18:21:23Z | 2021-12-18T18:21:23Z | OWNER | Goal here is to gain the ability to use https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.executescript |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
db.execute_write(..., executescript=True) parameter 1083895395 | |
997248364 | https://github.com/simonw/datasette/issues/1555#issuecomment-997248364 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cM1s | simonw 9599 | 2021-12-18T18:20:10Z | 2021-12-18T18:20:10Z | OWNER | Idea: teach
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997245301 | https://github.com/simonw/datasette/issues/1555#issuecomment-997245301 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cMF1 | simonw 9599 | 2021-12-18T18:17:04Z | 2021-12-18T18:17:04Z | OWNER | One downside of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997241969 | https://github.com/simonw/datasette/issues/1555#issuecomment-997241969 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cLRx | simonw 9599 | 2021-12-18T18:13:04Z | 2021-12-18T18:13:04Z | OWNER | Also: running all of those |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997241645 | https://github.com/simonw/datasette/issues/1555#issuecomment-997241645 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cLMt | simonw 9599 | 2021-12-18T18:12:26Z | 2021-12-18T18:12:26Z | OWNER | A simpler optimization would be just to turn all of those column and index reads into a single efficient UNION query against each database, then figure out the most efficient pattern to send them all as writes in one go as opposed to calling |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997235388 | https://github.com/simonw/datasette/issues/1566#issuecomment-997235388 | https://api.github.com/repos/simonw/datasette/issues/1566 | IC_kwDOBm6k_c47cJq8 | simonw 9599 | 2021-12-18T17:32:07Z | 2021-12-18T17:32:07Z | OWNER | I can release a new version of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Release Datasette 0.60 1083669410 | |
997235086 | https://github.com/simonw/datasette/issues/1555#issuecomment-997235086 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cJmO | simonw 9599 | 2021-12-18T17:30:13Z | 2021-12-18T17:30:13Z | OWNER | Now that trace sees write queries (#1568) it's clear that there is a whole lot more DB activity then I had realized: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997234858 | https://github.com/simonw/datasette/issues/1555#issuecomment-997234858 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47cJiq | simonw 9599 | 2021-12-18T17:28:44Z | 2021-12-18T17:28:44Z | OWNER | Maybe it would be worth exploring attaching each DB in turn to the _internal connection in order to perform these queries faster. I'm a bit worried about leaks though: the internal database isn't meant to be visible, even temporarily attaching another DB to it could cause SQL queries against that DB to be able to access the internal data. So maybe instead the _internal connection gets to connect to the other DBs? There's a maximum of ten there I think, which is good for most but not all cases. But the cases with the most connected databases will see the worst performance! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997153253 | https://github.com/simonw/datasette/issues/1568#issuecomment-997153253 | https://api.github.com/repos/simonw/datasette/issues/1568 | IC_kwDOBm6k_c47b1nl | simonw 9599 | 2021-12-18T06:20:23Z | 2021-12-18T06:20:23Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Trace should show queries on the write connection too 1083726550 | ||
997128950 | https://github.com/simonw/datasette/issues/1568#issuecomment-997128950 | https://api.github.com/repos/simonw/datasette/issues/1568 | IC_kwDOBm6k_c47bvr2 | simonw 9599 | 2021-12-18T02:38:01Z | 2021-12-18T02:38:01Z | OWNER | Prototype: ```diff diff --git a/datasette/database.py b/datasette/database.py index 0a0c104..468e936 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -99,7 +99,9 @@ class Database: with conn: return conn.execute(sql, params or [])
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Trace should show queries on the write connection too 1083726550 | |
997128508 | https://github.com/simonw/datasette/issues/1555#issuecomment-997128508 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47bvk8 | simonw 9599 | 2021-12-18T02:33:57Z | 2021-12-18T02:33:57Z | OWNER | Here's why - |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997128368 | https://github.com/simonw/datasette/issues/1555#issuecomment-997128368 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47bviw | simonw 9599 | 2021-12-18T02:32:43Z | 2021-12-18T02:32:43Z | OWNER | I wonder why the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997128251 | https://github.com/simonw/datasette/issues/1555#issuecomment-997128251 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47bvg7 | simonw 9599 | 2021-12-18T02:31:51Z | 2021-12-18T02:31:51Z | OWNER | I was thinking it might even be possible to convert this into a But the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997128080 | https://github.com/simonw/datasette/issues/1555#issuecomment-997128080 | https://api.github.com/repos/simonw/datasette/issues/1555 | IC_kwDOBm6k_c47bveQ | simonw 9599 | 2021-12-18T02:30:19Z | 2021-12-18T02:30:19Z | OWNER | I think all of these queries happen in one place - in the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optimize all those calls to index_list and foreign_key_list 1079149656 | |
997127784 | https://github.com/simonw/datasette/issues/1561#issuecomment-997127784 | https://api.github.com/repos/simonw/datasette/issues/1561 | IC_kwDOBm6k_c47bvZo | simonw 9599 | 2021-12-18T02:27:56Z | 2021-12-18T02:27:56Z | OWNER | Oh that's an interesting solution, combining the hashes of all of the individual databases. I'm actually not a big fan of
I've since not found myself wanting to use it at all for any of my projects - which makes me nervous, because it means there's a pretty complex feature that I'm not using at all, so it's only really protected by the existing unit tests for it. What I'd really like to do is figure out how to have hashed URL mode work entirely as a plugin - then I could extract it from Datasette core entirely (which would simplify a bunch of stuff) but people who find the optimization useful would be able to access it. I'm not sure that the existing plugin hooks are robust enough to do that yet though. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
add hash id to "_memory" url if hashed url mode is turned on and crossdb is also turned on 1082765654 | |
997127084 | https://github.com/simonw/datasette/issues/1563#issuecomment-997127084 | https://api.github.com/repos/simonw/datasette/issues/1563 | IC_kwDOBm6k_c47bvOs | simonw 9599 | 2021-12-18T02:22:30Z | 2021-12-18T02:22:30Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Datasette(... files=) should not be a required argument 1083573206 | ||
997125191 | https://github.com/simonw/datasette/issues/1563#issuecomment-997125191 | https://api.github.com/repos/simonw/datasette/issues/1563 | IC_kwDOBm6k_c47buxH | simonw 9599 | 2021-12-18T02:10:20Z | 2021-12-18T02:10:20Z | OWNER | I should document the usage of this constructor in https://docs.datasette.io/en/stable/internals.html#datasette-class |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Datasette(... files=) should not be a required argument 1083573206 | |
997124280 | https://github.com/simonw/datasette/issues/1546#issuecomment-997124280 | https://api.github.com/repos/simonw/datasette/issues/1546 | IC_kwDOBm6k_c47bui4 | simonw 9599 | 2021-12-18T02:05:16Z | 2021-12-18T02:05:16Z | OWNER | Sure - there are actually several levels to this. The code that creates connections to the database is this: https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/database.py#L72-L95 For files on disk, it does this: ```python For read-only connectionsconn = sqlite3.connect( "file:my.db?mode=ro", uri=True, check_same_thread=False) For connections that should be treated as immutable:conn = sqlite3.connect( "file:my.db?immutable=1", uri=True, check_same_thread=False)
So this function runs against any incoming SQL to verify that it looks like a https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/utils/init.py#L195-L204 You can see the tests for that here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/tests/test_utils.py#L136-L170 |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0 } |
validating the sql 1076057610 | |
997122938 | https://github.com/simonw/datasette/issues/1564#issuecomment-997122938 | https://api.github.com/repos/simonw/datasette/issues/1564 | IC_kwDOBm6k_c47buN6 | simonw 9599 | 2021-12-18T01:55:25Z | 2021-12-18T01:55:46Z | OWNER | Made this change while working on this issue: - #1567 I'm going to write a test for this that uses that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_prepare_connection not called on write connections 1083581011 | |
997121215 | https://github.com/simonw/datasette/issues/1565#issuecomment-997121215 | https://api.github.com/repos/simonw/datasette/issues/1565 | IC_kwDOBm6k_c47bty_ | simonw 9599 | 2021-12-18T01:45:44Z | 2021-12-18T01:45:44Z | OWNER | I want to get this into Datasette 0.60 - #1566 - it's a small change that can unlock a lot of potential. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Documented JavaScript variables on different templates made available for plugins 1083657868 | |
997120723 | https://github.com/simonw/datasette/issues/621#issuecomment-997120723 | https://api.github.com/repos/simonw/datasette/issues/621 | IC_kwDOBm6k_c47btrT | simonw 9599 | 2021-12-18T01:42:33Z | 2021-12-18T01:42:33Z | OWNER | I refactored this code out into the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Syntax for ?_through= that works as a form field 520681725 | |
552253893 | https://github.com/simonw/datasette/issues/617#issuecomment-552253893 | https://api.github.com/repos/simonw/datasette/issues/617 | MDEyOklzc3VlQ29tbWVudDU1MjI1Mzg5Mw== | simonw 9599 | 2019-11-11T00:46:42Z | 2021-12-18T01:41:47Z | OWNER | As noted in https://github.com/simonw/datasette/issues/621#issuecomment-552253208 a common pattern in this method is blocks of code that append new items to the Code that fits this pattern:
The keyset pagination code modifies [UPDATE December 2021 - this comment became the basis for a new |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor TableView.data() method 519613116 | |
981153060 | https://github.com/simonw/datasette/issues/1518#issuecomment-981153060 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c46ezUk | simonw 9599 | 2021-11-28T21:13:09Z | 2021-12-17T23:37:08Z | OWNER | Two new requirements inspired by work on the
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
997082845 | https://github.com/simonw/datasette/issues/1518#issuecomment-997082845 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c47bkbd | simonw 9599 | 2021-12-17T23:10:09Z | 2021-12-17T23:10:17Z | OWNER | These changes so far are now in the 0.60a0 alpha: https://github.com/simonw/datasette/releases/tag/0.60a0 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
997082676 | https://github.com/simonw/datasette/pull/1559#issuecomment-997082676 | https://api.github.com/repos/simonw/datasette/issues/1559 | IC_kwDOBm6k_c47bkY0 | simonw 9599 | 2021-12-17T23:09:41Z | 2021-12-17T23:09:41Z | OWNER | This is now available to try out in Datasette 0.60a0: https://github.com/simonw/datasette/releases/tag/0.60a0 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
filters_from_request plugin hook, now used in TableView 1082743068 | |
997082189 | https://github.com/simonw/datasette/pull/1562#issuecomment-997082189 | https://api.github.com/repos/simonw/datasette/issues/1562 | IC_kwDOBm6k_c47bkRN | simonw 9599 | 2021-12-17T23:08:14Z | 2021-12-17T23:08:14Z | OWNER | Oh that makes sense: In Python 3.6 this happens:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1 1083246400 | |
997081673 | https://github.com/simonw/datasette/pull/1562#issuecomment-997081673 | https://api.github.com/repos/simonw/datasette/issues/1562 | IC_kwDOBm6k_c47bkJJ | simonw 9599 | 2021-12-17T23:06:38Z | 2021-12-17T23:06:38Z | OWNER | From this diff between It looks like the only change relevant to compatibility is |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1 1083246400 | |
997080352 | https://github.com/simonw/datasette/pull/1562#issuecomment-997080352 | https://api.github.com/repos/simonw/datasette/issues/1562 | IC_kwDOBm6k_c47bj0g | simonw 9599 | 2021-12-17T23:03:08Z | 2021-12-17T23:03:08Z | OWNER | They say they've dropped 3.6 support, but Datasette's tests against 3.6 are still passing. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1 1083246400 | |
997078812 | https://github.com/simonw/datasette/issues/1566#issuecomment-997078812 | https://api.github.com/repos/simonw/datasette/issues/1566 | IC_kwDOBm6k_c47bjcc | simonw 9599 | 2021-12-17T22:58:55Z | 2021-12-17T22:58:55Z | OWNER | The release notes for the 0.60a0 alpha will be useful here: https://github.com/simonw/datasette/releases/tag/0.60a0 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Release Datasette 0.60 1083669410 | |
997077410 | https://github.com/simonw/datasette/issues/1565#issuecomment-997077410 | https://api.github.com/repos/simonw/datasette/issues/1565 | IC_kwDOBm6k_c47bjGi | simonw 9599 | 2021-12-17T22:54:45Z | 2021-12-17T22:54:45Z | OWNER | The table page should expose the query both with and without the
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Documented JavaScript variables on different templates made available for plugins 1083657868 | |
997069128 | https://github.com/simonw/datasette/issues/1565#issuecomment-997069128 | https://api.github.com/repos/simonw/datasette/issues/1565 | IC_kwDOBm6k_c47bhFI | simonw 9599 | 2021-12-17T22:31:18Z | 2021-12-17T22:31:18Z | OWNER | This should aim to be as consistent as possible with the various arguments to hooks on https://docs.datasette.io/en/stable/plugin_hooks.html |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Documented JavaScript variables on different templates made available for plugins 1083657868 | |
996961196 | https://github.com/simonw/datasette/pull/1559#issuecomment-996961196 | https://api.github.com/repos/simonw/datasette/issues/1559 | IC_kwDOBm6k_c47bGus | simonw 9599 | 2021-12-17T19:00:53Z | 2021-12-17T19:00:53Z | OWNER | I'm going to merge this to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
filters_from_request plugin hook, now used in TableView 1082743068 | |
996959325 | https://github.com/simonw/datasette/pull/1559#issuecomment-996959325 | https://api.github.com/repos/simonw/datasette/issues/1559 | IC_kwDOBm6k_c47bGRd | simonw 9599 | 2021-12-17T18:59:54Z | 2021-12-17T18:59:54Z | OWNER | I've convinced myself that this plugin hook design is good through this |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
filters_from_request plugin hook, now used in TableView 1082743068 | |
996958442 | https://github.com/simonw/datasette/issues/473#issuecomment-996958442 | https://api.github.com/repos/simonw/datasette/issues/473 | IC_kwDOBm6k_c47bGDq | simonw 9599 | 2021-12-17T18:59:27Z | 2021-12-17T18:59:27Z | OWNER | I'm happy with how the prototype that used this plugin in |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook: filters_from_request 445850934 | |
996345233 | https://github.com/simonw/datasette/issues/473#issuecomment-996345233 | https://api.github.com/repos/simonw/datasette/issues/473 | IC_kwDOBm6k_c47YwWR | simonw 9599 | 2021-12-17T01:20:31Z | 2021-12-17T18:13:01Z | OWNER | I could use this hook to add table filtering on a map to the existing |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook: filters_from_request 445850934 | |
996895423 | https://github.com/simonw/datasette/pull/1559#issuecomment-996895423 | https://api.github.com/repos/simonw/datasette/issues/1559 | IC_kwDOBm6k_c47a2q_ | simonw 9599 | 2021-12-17T17:28:44Z | 2021-12-17T17:28:44Z | OWNER | Before I land this I'm going to build one prototype plugin against it to confirm that the new hook is useful in its current shape. I'll add support for filtering a table by drawing on a map to https://datasette.io/plugins/datasette-leaflet-freedraw |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
filters_from_request plugin hook, now used in TableView 1082743068 | |
996488925 | https://github.com/simonw/datasette/pull/1204#issuecomment-996488925 | https://api.github.com/repos/simonw/datasette/issues/1204 | IC_kwDOBm6k_c47ZTbd | simonw 9599 | 2021-12-17T07:10:48Z | 2021-12-17T07:10:48Z | OWNER | I think this is missing the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Plugin includes 793002853 | |
996484551 | https://github.com/simonw/datasette/issues/473#issuecomment-996484551 | https://api.github.com/repos/simonw/datasette/issues/473 | IC_kwDOBm6k_c47ZSXH | simonw 9599 | 2021-12-17T07:02:21Z | 2021-12-17T07:04:23Z | OWNER | The one slightly weird thing about this hook is how it adds Maybe I need the proposed mechanism from - #1191 Which has an in-progress PR: - #1204 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook: filters_from_request 445850934 | |
761104933 | https://github.com/simonw/datasette/issues/1191#issuecomment-761104933 | https://api.github.com/repos/simonw/datasette/issues/1191 | MDEyOklzc3VlQ29tbWVudDc2MTEwNDkzMw== | simonw 9599 | 2021-01-15T18:21:26Z | 2021-12-17T07:03:02Z | OWNER | Also related: #857 (comprehensive documentation of variables available to templates) - since then the plugin hook could be fed the full template context and use that to do its thing. Or maybe the plugin hooks gets to return the name of a template that should be |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability for plugins to collaborate when adding extra HTML to blocks in default templates 787098345 | |
996286808 | https://github.com/simonw/datasette/pull/1559#issuecomment-996286808 | https://api.github.com/repos/simonw/datasette/issues/1559 | IC_kwDOBm6k_c47YiFY | simonw 9599 | 2021-12-17T00:01:43Z | 2021-12-17T00:01:43Z | OWNER | This already has tests and documentation, and I've used it to refactor out the logic for Do I like this enough to land it on Also, I think I can still use it to refactor out the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
filters_from_request plugin hook, now used in TableView 1082743068 | |
996286199 | https://github.com/simonw/datasette/issues/473#issuecomment-996286199 | https://api.github.com/repos/simonw/datasette/issues/473 | IC_kwDOBm6k_c47Yh73 | simonw 9599 | 2021-12-17T00:00:22Z | 2021-12-17T00:00:22Z | OWNER | Documentation for that hook in the PR branch: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook: filters_from_request 445850934 | |
996286104 | https://github.com/simonw/datasette/issues/1518#issuecomment-996286104 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c47Yh6Y | simonw 9599 | 2021-12-17T00:00:07Z | 2021-12-17T00:00:07Z | OWNER | Documentation of the new hook in the PR: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
996275108 | https://github.com/simonw/datasette/issues/473#issuecomment-996275108 | https://api.github.com/repos/simonw/datasette/issues/473 | IC_kwDOBm6k_c47YfOk | simonw 9599 | 2021-12-16T23:32:22Z | 2021-12-16T23:32:30Z | OWNER | This filter design can only influence the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook: filters_from_request 445850934 | |
996272906 | https://github.com/simonw/datasette/issues/1518#issuecomment-996272906 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c47YesK | simonw 9599 | 2021-12-16T23:27:42Z | 2021-12-16T23:27:42Z | OWNER | Got a TIL out of this: https://til.simonwillison.net/pluggy/multiple-hooks-same-file |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 | |
996267817 | https://github.com/simonw/datasette/issues/473#issuecomment-996267817 | https://api.github.com/repos/simonw/datasette/issues/473 | IC_kwDOBm6k_c47Ydcp | simonw 9599 | 2021-12-16T23:17:52Z | 2021-12-16T23:19:00Z | OWNER | I revisited this idea in #1518 and came up with a slightly different name and design for the hook:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook: filters_from_request 445850934 | |
996264617 | https://github.com/simonw/datasette/issues/1518#issuecomment-996264617 | https://api.github.com/repos/simonw/datasette/issues/1518 | IC_kwDOBm6k_c47Ycqp | simonw 9599 | 2021-12-16T23:11:12Z | 2021-12-16T23:11:12Z | OWNER | I managed to extract both |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Complete refactor of TableView and table.html template 1058072543 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
updated_at (date) >30 ✖