github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997508728 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | 997508728 | IC_kwDOCGYnMM47dMZ4 | 9599 | 2021-12-20T01:14:43Z | 2021-12-20T01:14:43Z | OWNER | (This makes me want `--extract` from #352 even more.) | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077431957 | |
https://github.com/simonw/sqlite-utils/issues/163#issuecomment-997502242 | https://api.github.com/repos/simonw/sqlite-utils/issues/163 | 997502242 | IC_kwDOCGYnMM47dK0i | 9599 | 2021-12-20T00:56:45Z | 2021-12-20T00:56:52Z | OWNER | > Maybe `sqlite-utils` should absorb all of the functionality from `sqlite-transform` - having two separate tools doesn't necessarily make sense. I implemented that in: - #251 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
706001517 | |
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997497262 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | 997497262 | IC_kwDOCGYnMM47dJmu | 9599 | 2021-12-20T00:40:15Z | 2021-12-20T00:40:15Z | OWNER | `--flatten` could do with a better description too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077431957 | |
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496931 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | 997496931 | IC_kwDOCGYnMM47dJhj | 9599 | 2021-12-20T00:39:14Z | 2021-12-20T00:39:52Z | OWNER | ``` % sqlite-utils insert --help Usage: sqlite-utils insert [OPTIONS] PATH TABLE JSON_FILE Insert records from JSON file into a table, creating the table if it does not already exist. Input should be a JSON array of objects, unless --nl or --csv is used. Options: --pk TEXT Columns to use as the primary key, e.g. id --nl Expect newline-delimited JSON --flatten Flatten nested JSON objects -c, --csv Expect CSV --tsv Expect TSV --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default <TEXT TEXT>... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ``` I can add a bunch of extra help at the top there to explain all of this stuff. That "Input should be a JSON array of objects" bit could be expanded to several paragraphs. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077431957 | |
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997492872 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | 997492872 | IC_kwDOCGYnMM47dIiI | 9599 | 2021-12-20T00:23:31Z | 2021-12-20T00:23:31Z | OWNER | I think this should work on JSON, or CSV, or individual lines, or the entire content at once. So I'll require `--lines --convert ...` to import individual lines, or `--all --convert` to run the conversion against the entire input at once. What would `--lines` or `--all` do without `--convert`? Maybe insert records as `{"line": "line of text"}` or `{"all": "whole input}`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077431957 | |
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997486156 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | 997486156 | IC_kwDOCGYnMM47dG5M | 9599 | 2021-12-19T23:51:02Z | 2021-12-19T23:51:02Z | OWNER | This is going to need a `--import` multi option too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077431957 | |
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997485361 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | 997485361 | IC_kwDOCGYnMM47dGsx | 9599 | 2021-12-19T23:45:30Z | 2021-12-19T23:45:30Z | OWNER | Really interesting example input for this: https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/iOS13.txt - see https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/ | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077431957 | |
https://github.com/simonw/datasette/issues/1565#issuecomment-997474022 | https://api.github.com/repos/simonw/datasette/issues/1565 | 997474022 | IC_kwDOBm6k_c47dD7m | 9599 | 2021-12-19T22:36:49Z | 2021-12-19T22:37:29Z | OWNER | No way with a tagged template literal to pass an extra database name argument, so instead I need a method that returns a callable that can be used for the tagged template literal for a specific database - or the default database. This could work (bit weird looking though): ```javascript var rows = await datasette.query("fixtures")`select * from foo`; ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083657868 | |
https://github.com/simonw/datasette/issues/1565#issuecomment-997473856 | https://api.github.com/repos/simonw/datasette/issues/1565 | 997473856 | IC_kwDOBm6k_c47dD5A | 9599 | 2021-12-19T22:35:20Z | 2021-12-19T22:35:20Z | OWNER | Quick prototype of that tagged template `query` function: ```javascript function query(pieces, ...parameters) { var qs = new URLSearchParams(); var sql = pieces[0]; parameters.forEach((param, i) => { sql += `:p${i}${pieces[i + 1]}`; qs.append(`p${i}`, param); }); qs.append("sql", sql); return qs.toString(); } var id = 4; console.log(query`select * from ids where id > ${id}`); ``` Outputs: ``` p0=4&sql=select+*+from+ids+where+id+%3E+%3Ap0 ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083657868 | |
https://github.com/simonw/datasette/issues/1565#issuecomment-997472639 | https://api.github.com/repos/simonw/datasette/issues/1565 | 997472639 | IC_kwDOBm6k_c47dDl_ | 9599 | 2021-12-19T22:25:50Z | 2021-12-19T22:25:50Z | OWNER | Or... ```javascript rows = await datasette.query`select * from searchable where id > ${id}`; ``` And it knows how to turn that into a parameterized call using tagged template literals. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083657868 | |
https://github.com/simonw/datasette/issues/1565#issuecomment-997472509 | https://api.github.com/repos/simonw/datasette/issues/1565 | 997472509 | IC_kwDOBm6k_c47dDj9 | 9599 | 2021-12-19T22:24:50Z | 2021-12-19T22:24:50Z | OWNER | ... huh, it could even expose a JavaScript function that can be called to execute a SQL query. ```javascript datasette.query("select * from blah").then(...) ``` Maybe it takes an optional second argument that specifies the database - defaulting to the one for the current page. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083657868 | |
https://github.com/simonw/datasette/issues/1565#issuecomment-997472370 | https://api.github.com/repos/simonw/datasette/issues/1565 | 997472370 | IC_kwDOBm6k_c47dDhy | 9599 | 2021-12-19T22:23:36Z | 2021-12-19T22:23:36Z | OWNER | This should also expose the JSON API endpoints used to execute SQL against this database. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083657868 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-997472214 | https://api.github.com/repos/simonw/datasette/issues/1518 | 997472214 | IC_kwDOBm6k_c47dDfW | 9599 | 2021-12-19T22:22:08Z | 2021-12-19T22:22:08Z | OWNER | I sketched out a chained SQL builder pattern that might be useful for further tidying up this code - though with the new plugin hook I'm less excited about it than I was: ```python class TableQuery: def __init__(self, table, columns, pks, is_view=False, prev=None): self.table = table self.columns = columns self.pks = pks self.is_view = is_view self.prev = prev # These can be changed for different instances in the chain: self._where_clauses = None self._order_by = None self._page_size = None self._offset = None self._select_columns = None self.select_all_columns = '*' self.select_specified_columns = '*' @property def where_clauses(self): wheres = [] current = self while current: if current._where_clauses is not None: wheres.extend(current._where_clauses) current = current.prev return list(reversed(wheres)) def where(self, where): new_cls = TableQuery(self.table, self.columns, self.pks, self.is_view, self) new_cls._where_clauses = [where] return new_cls @classmethod async def introspect(cls, db, table): return cls( table, columns = await db.table_columns(table), pks = await db.primary_keys(table), is_view = bool(await db.get_view_definition(table)) ) @property def sql_from(self): return f"from {self.table}{self.sql_where}" @property def sql_where(self): if not self.where_clauses: return "" else: return f" where {' and '.join(self.where_clauses)}" @property def sql_no_order_no_limit(self): return f"select {self.select_all_columns} from {self.table}{self.sql_where}" @property def sql(self): return f"select {self.select_specified_columns} from {se… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/datasette/issues/1547#issuecomment-997471672 | https://api.github.com/repos/simonw/datasette/issues/1547 | 997471672 | IC_kwDOBm6k_c47dDW4 | 9599 | 2021-12-19T22:18:26Z | 2021-12-19T22:18:26Z | OWNER | I released this [in an alpha](https://github.com/simonw/datasette/releases/tag/0.60a1), so you can try out this fix using: pip install datasette==0.60a1 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1076388044 | |
https://github.com/simonw/datasette/issues/1566#issuecomment-997470633 | https://api.github.com/repos/simonw/datasette/issues/1566 | 997470633 | IC_kwDOBm6k_c47dDGp | 9599 | 2021-12-19T22:12:00Z | 2021-12-19T22:12:00Z | OWNER | Released another alpha, 0.60a1: https://github.com/simonw/datasette/releases/tag/0.60a1 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083669410 | |
https://github.com/simonw/datasette/issues/1545#issuecomment-997462604 | https://api.github.com/repos/simonw/datasette/issues/1545 | 997462604 | IC_kwDOBm6k_c47dBJM | 9599 | 2021-12-19T21:17:08Z | 2021-12-19T21:17:08Z | OWNER | Here's the relevant code: https://github.com/simonw/datasette/blob/4094741c2881c2ada3f3f878b532fdaec7914953/datasette/app.py#L1204-L1219 It's using `route_path.split("/")` which should be OK because that's the incoming `request.path` path - which I would expect to use `/` even on Windows. Then it uses `os.path.join` which should do the right thing. I need to get myself a proper Windows development environment setup to investigate this one. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1075893249 | |
https://github.com/simonw/datasette/issues/1573#issuecomment-997462117 | https://api.github.com/repos/simonw/datasette/issues/1573 | 997462117 | IC_kwDOBm6k_c47dBBl | 9599 | 2021-12-19T21:13:13Z | 2021-12-19T21:13:13Z | OWNER | This might also be the impetus I need to bring the https://datasette.io/plugins/datasette-pretty-traces plugin into Datasette core itself. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1084185188 | |
https://github.com/simonw/datasette/issues/1547#issuecomment-997460731 | https://api.github.com/repos/simonw/datasette/issues/1547 | 997460731 | IC_kwDOBm6k_c47dAr7 | 9599 | 2021-12-19T21:02:15Z | 2021-12-19T21:02:15Z | OWNER | Yes, this is a bug. It looks like the problem is with the `if write:` branch in this code here: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L252-L327 Is missing this bit of code: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L343-L347 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1076388044 | |
https://github.com/simonw/datasette/issues/1570#issuecomment-997460061 | https://api.github.com/repos/simonw/datasette/issues/1570 | 997460061 | IC_kwDOBm6k_c47dAhd | 9599 | 2021-12-19T20:56:54Z | 2021-12-19T20:56:54Z | OWNER | Documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083921371 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997459958 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997459958 | IC_kwDOBm6k_c47dAf2 | 9599 | 2021-12-19T20:55:59Z | 2021-12-19T20:55:59Z | OWNER | Closing this issue because I've optimized this a whole bunch, and it's definitely good enough for the moment. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997325189 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997325189 | IC_kwDOBm6k_c47cfmF | 9599 | 2021-12-19T03:55:01Z | 2021-12-19T20:54:51Z | OWNER | It's a bit annoying that the queries no longer show up in the trace at all now, thanks to running in `.execute_fn()`. I wonder if there's something smart I can do about that - maybe have `trace()` record that function with a traceback even though it doesn't have the executed SQL string? 5fac26aa221a111d7633f2dd92014641f7c0ade9 has the same problem. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997459637 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997459637 | IC_kwDOBm6k_c47dAa1 | 9599 | 2021-12-19T20:53:46Z | 2021-12-19T20:53:46Z | OWNER | Using #1571 showed me that the `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ? and table_name = ?` queries were running way more times than I expected. I came up with a new optimization that just does `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ?` instead. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1566#issuecomment-997457790 | https://api.github.com/repos/simonw/datasette/issues/1566 | 997457790 | IC_kwDOBm6k_c47c_9- | 9599 | 2021-12-19T20:40:50Z | 2021-12-19T20:40:57Z | OWNER | Also release new version of `datasette-pretty-traces` with this feature: - https://github.com/simonw/datasette-pretty-traces/issues/7 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083669410 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997342494 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997342494 | IC_kwDOBm6k_c47cj0e | 9599 | 2021-12-19T07:22:04Z | 2021-12-19T07:22:04Z | OWNER | Another option would be to provide an abstraction that makes it easier to run a group of SQL queries in the same thread at the same time, and have them traced correctly. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997324666 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997324666 | IC_kwDOBm6k_c47cfd6 | 9599 | 2021-12-19T03:47:51Z | 2021-12-19T03:48:09Z | OWNER | Here's a hacked together prototype of running all of that stuff inside a single function passed to `.execute_fn()`: ```diff diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index 95055d8..58f9982 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -1,4 +1,5 @@ import textwrap +from datasette.utils import table_column_details async def init_internal_db(db): @@ -70,49 +71,70 @@ async def populate_schema_tables(internal_db, db): "DELETE FROM tables WHERE database_name = ?", [database_name], block=True ) tables = (await db.execute("select * from sqlite_master WHERE type = 'table'")).rows - tables_to_insert = [] - columns_to_delete = [] - columns_to_insert = [] - foreign_keys_to_delete = [] - foreign_keys_to_insert = [] - indexes_to_delete = [] - indexes_to_insert = [] - for table in tables: - table_name = table["name"] - tables_to_insert.append( - (database_name, table_name, table["rootpage"], table["sql"]) - ) - columns_to_delete.append((database_name, table_name)) - columns = await db.table_column_details(table_name) - columns_to_insert.extend( - { - **{"database_name": database_name, "table_name": table_name}, - **column._asdict(), - } - for column in columns - ) - foreign_keys_to_delete.append((database_name, table_name)) - foreign_keys = ( - await db.execute(f"PRAGMA foreign_key_list([{table_name}])") - ).rows - foreign_keys_to_insert.extend( - { - **{"database_name": database_name, "table_name": table_name}, - **dict(foreign_key), - } - for foreign_key in foreign_keys - ) - indexes_to_delete.append((database_name, table_name)) - indexes = (await db.execute(f"PRAGMA index_list([{table_name}])")).rows - … | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997324156 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997324156 | IC_kwDOBm6k_c47cfV8 | 9599 | 2021-12-19T03:40:05Z | 2021-12-19T03:40:05Z | OWNER | Using the prototype of this: - https://github.com/simonw/datasette-pretty-traces/issues/5 I'm seeing about 180ms spent running all of these queries on startup! ![CleanShot 2021-12-18 at 19 38 37@2x](https://user-images.githubusercontent.com/9599/146663045-46bda669-90de-474f-8870-345182725dc1.png) | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997321767 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997321767 | IC_kwDOBm6k_c47cewn | 9599 | 2021-12-19T03:10:58Z | 2021-12-19T03:10:58Z | OWNER | I wonder how much overhead there is switching between the `async` event loop main code and the thread that runs the SQL queries. Would there be a performance boost if I gathered all of the column/index information in a single function run on the thread using `db.execute_fn()` I wonder? It would eliminate a bunch of switching between threads. Would be great to understand how much of an impact that would have. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997321653 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997321653 | IC_kwDOBm6k_c47ceu1 | 9599 | 2021-12-19T03:09:43Z | 2021-12-19T03:09:43Z | OWNER | On that same documentation page I just spotted this: > This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported. This makes me nervous to rely on pragma function optimizations in Datasette itself. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997321477 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997321477 | IC_kwDOBm6k_c47cesF | 9599 | 2021-12-19T03:07:33Z | 2021-12-19T03:07:33Z | OWNER | If I want to continue supporting SQLite prior to 3.16.0 (2017-01-02) I'll need this optimization to only kick in with versions that support table-valued PRAGMA functions, while keeping the old `PRAGMA foreign_key_list(table)` stuff working for those older versions. That's feasible, but it's a bit more work - and I need to make sure I have robust testing in place for SQLite 3.15.0. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997321327 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997321327 | IC_kwDOBm6k_c47cepv | 9599 | 2021-12-19T03:05:39Z | 2021-12-19T03:05:44Z | OWNER | This caught me out once before in: - https://github.com/simonw/datasette/issues/1276 Turns out Glitch was running SQLite 3.11.0 from 2016-02-15. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997321217 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997321217 | IC_kwDOBm6k_c47ceoB | 9599 | 2021-12-19T03:04:16Z | 2021-12-19T03:04:16Z | OWNER | One thing to watch out for though, from https://sqlite.org/pragma.html#pragfunc > The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997321115 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997321115 | IC_kwDOBm6k_c47cemb | 9599 | 2021-12-19T03:03:12Z | 2021-12-19T03:03:12Z | OWNER | Table columns is a bit harder, because `table_xinfo` is only in SQLite 3.26.0 or higher: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/utils/__init__.py#L565-L581 So if that function is available: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_xinfo.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_xinfo%28sqlite_master.name%29+AS+table_xinfo%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, table_xinfo.* FROM sqlite_master, pragma_table_xinfo(sqlite_master.name) AS table_xinfo WHERE sqlite_master.type = 'table' ``` And otherwise, using `table_info`: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_info.*%2C%0D%0A++0+as+hidden%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_info%28sqlite_master.name%29+AS+table_info%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, table_info.*, 0 as hidden FROM sqlite_master, pragma_table_info(sqlite_master.name) AS table_info WHERE sqlite_master.type = 'table' ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997320824 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997320824 | IC_kwDOBm6k_c47ceh4 | 9599 | 2021-12-19T02:59:57Z | 2021-12-19T03:00:44Z | OWNER | To list all indexes: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++index_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_index_list%28sqlite_master.name%29+AS+index_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, index_list.* FROM sqlite_master, pragma_index_list(sqlite_master.name) AS index_list WHERE sqlite_master.type = 'table' ``` Foreign keys: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++foreign_key_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_foreign_key_list%28sqlite_master.name%29+AS+foreign_key_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, foreign_key_list.* FROM sqlite_master, pragma_foreign_key_list(sqlite_master.name) AS foreign_key_list WHERE sqlite_master.type = 'table' ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1566#issuecomment-997272328 | https://api.github.com/repos/simonw/datasette/issues/1566 | 997272328 | IC_kwDOBm6k_c47cSsI | 9599 | 2021-12-18T19:18:01Z | 2021-12-18T19:18:01Z | OWNER | Added some useful new documented internal methods in: - #1570 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083669410 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997272223 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997272223 | IC_kwDOBm6k_c47cSqf | 9599 | 2021-12-18T19:17:13Z | 2021-12-18T19:17:13Z | OWNER | That's a good optimization. Still need to deal with the huge flurry of `PRAGMA` queries though before I can consider this done. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1570#issuecomment-997267583 | https://api.github.com/repos/simonw/datasette/issues/1570 | 997267583 | IC_kwDOBm6k_c47cRh_ | 9599 | 2021-12-18T18:46:05Z | 2021-12-18T18:46:12Z | OWNER | This will replace the work done in #1569. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083921371 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997267416 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997267416 | IC_kwDOBm6k_c47cRfY | 9599 | 2021-12-18T18:44:53Z | 2021-12-18T18:45:28Z | OWNER | Rather than adding a `executemany=True` parameter, I'm now thinking a better design might be to have three methods: - `db.execute_write(sql, params=None, block=False)` - `db.execute_writescript(sql, block=False)` - `db.execute_writemany(sql, params_seq, block=False)` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1569#issuecomment-997266687 | https://api.github.com/repos/simonw/datasette/issues/1569 | 997266687 | IC_kwDOBm6k_c47cRT_ | 9599 | 2021-12-18T18:41:40Z | 2021-12-18T18:41:40Z | OWNER | Updated documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-executescript-false-block-false | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083895395 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997266100 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997266100 | IC_kwDOBm6k_c47cRK0 | 9599 | 2021-12-18T18:40:02Z | 2021-12-18T18:40:02Z | OWNER | The implementation of `cursor.executemany()` looks very efficient - it turns into a call to this C function with `multiple` set to `1`: https://github.com/python/cpython/blob/e002bbc6cce637171fb2b1391ffeca8643a13843/Modules/_sqlite/cursor.c#L468-L469 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997262475 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997262475 | IC_kwDOBm6k_c47cQSL | 9599 | 2021-12-18T18:34:18Z | 2021-12-18T18:34:18Z | OWNER | <img width="1055" alt="image" src="https://user-images.githubusercontent.com/9599/146652142-1c0bc34e-4a18-407d-bd59-28d565b631a6.png"> Using `executescript=True` that call now takes 1.89ms to create all of those tables. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1569#issuecomment-997249563 | https://api.github.com/repos/simonw/datasette/issues/1569 | 997249563 | IC_kwDOBm6k_c47cNIb | 9599 | 2021-12-18T18:21:23Z | 2021-12-18T18:21:23Z | OWNER | Goal here is to gain the ability to use `conn.executescript()` and still have it show up in the tracer. https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.executescript | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083895395 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997248364 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997248364 | IC_kwDOBm6k_c47cM1s | 9599 | 2021-12-18T18:20:10Z | 2021-12-18T18:20:10Z | OWNER | Idea: teach `execute_write` to accept an optional `executescript=True` parameter, like this: ```diff diff --git a/datasette/database.py b/datasette/database.py index 468e936..1a424f5 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -94,10 +94,14 @@ class Database: f"file:{self.path}{qs}", uri=True, check_same_thread=False ) - async def execute_write(self, sql, params=None, block=False): + async def execute_write(self, sql, params=None, executescript=False, block=False): + assert not executescript and params, "Cannot use params with executescript=True" def _inner(conn): with conn: - return conn.execute(sql, params or []) + if executescript: + return conn.executescript(sql) + else: + return conn.execute(sql, params or []) with trace("sql", database=self.name, sql=sql.strip(), params=params): results = await self.execute_write_fn(_inner, block=block) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997245301 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997245301 | IC_kwDOBm6k_c47cMF1 | 9599 | 2021-12-18T18:17:04Z | 2021-12-18T18:17:04Z | OWNER | One downside of `conn.executescript()` is that it won't be picked up by the tracing mechanism - in fact nothing that uses `await db.execute_write_fn(fn, block=True)` or `await db.execute_fn(fn, block=True)` gets picked up by tracing. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997241969 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997241969 | IC_kwDOBm6k_c47cLRx | 9599 | 2021-12-18T18:13:04Z | 2021-12-18T18:13:04Z | OWNER | Also: running all of those `CREATE TABLE IF NOT EXISTS` in a single call to `conn.executescript()` rather than as separate queries may speed things up too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997241645 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997241645 | IC_kwDOBm6k_c47cLMt | 9599 | 2021-12-18T18:12:26Z | 2021-12-18T18:12:26Z | OWNER | A simpler optimization would be just to turn all of those column and index reads into a single efficient UNION query against each database, then figure out the most efficient pattern to send them all as writes in one go as opposed to calling `.execute_write()` in a loop. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1566#issuecomment-997235388 | https://api.github.com/repos/simonw/datasette/issues/1566 | 997235388 | IC_kwDOBm6k_c47cJq8 | 9599 | 2021-12-18T17:32:07Z | 2021-12-18T17:32:07Z | OWNER | I can release a new version of `datasette-leaflet-freedraw` as soon as this is out. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083669410 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997235086 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997235086 | IC_kwDOBm6k_c47cJmO | 9599 | 2021-12-18T17:30:13Z | 2021-12-18T17:30:13Z | OWNER | Now that trace sees write queries (#1568) it's clear that there is a whole lot more DB activity then I had realized: <img width="1292" alt="image" src="https://user-images.githubusercontent.com/9599/146626249-cc8609f6-590c-49e3-abab-0cbc132916d1.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997234858 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997234858 | IC_kwDOBm6k_c47cJiq | 9599 | 2021-12-18T17:28:44Z | 2021-12-18T17:28:44Z | OWNER | Maybe it would be worth exploring attaching each DB in turn to the _internal connection in order to perform these queries faster. I'm a bit worried about leaks though: the internal database isn't meant to be visible, even temporarily attaching another DB to it could cause SQL queries against that DB to be able to access the internal data. So maybe instead the _internal connection gets to connect to the other DBs? There's a maximum of ten there I think, which is good for most but not all cases. But the cases with the most connected databases will see the worst performance! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1568#issuecomment-997153253 | https://api.github.com/repos/simonw/datasette/issues/1568 | 997153253 | IC_kwDOBm6k_c47b1nl | 9599 | 2021-12-18T06:20:23Z | 2021-12-18T06:20:23Z | OWNER | Now running at https://latest-with-plugins.datasette.io/github/commits?_trace=1 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083726550 | |
https://github.com/simonw/datasette/issues/1568#issuecomment-997128950 | https://api.github.com/repos/simonw/datasette/issues/1568 | 997128950 | IC_kwDOBm6k_c47bvr2 | 9599 | 2021-12-18T02:38:01Z | 2021-12-18T02:38:01Z | OWNER | Prototype: ```diff diff --git a/datasette/database.py b/datasette/database.py index 0a0c104..468e936 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -99,7 +99,9 @@ class Database: with conn: return conn.execute(sql, params or []) - return await self.execute_write_fn(_inner, block=block) + with trace("sql", database=self.name, sql=sql.strip(), params=params): + results = await self.execute_write_fn(_inner, block=block) + return results async def execute_write_fn(self, fn, block=False): task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io") ``` <img width="1292" alt="image" src="https://user-images.githubusercontent.com/9599/146626249-cc8609f6-590c-49e3-abab-0cbc132916d1.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083726550 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997128508 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997128508 | IC_kwDOBm6k_c47bvk8 | 9599 | 2021-12-18T02:33:57Z | 2021-12-18T02:33:57Z | OWNER | Here's why - `trace` only applies to read, not write SQL operations: https://github.com/simonw/datasette/blob/7c8f8aa209e4ba7bf83976f8495d67c28fbfca24/datasette/database.py#L209-L211 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997128368 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997128368 | IC_kwDOBm6k_c47bviw | 9599 | 2021-12-18T02:32:43Z | 2021-12-18T02:32:43Z | OWNER | I wonder why the `INSERT INTO` queries don't show up in that `?trace=1` view? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997128251 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997128251 | IC_kwDOBm6k_c47bvg7 | 9599 | 2021-12-18T02:31:51Z | 2021-12-18T02:31:51Z | OWNER | I was thinking it might even be possible to convert this into a `insert into tables select from ...` query: https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L102-L112 But the `SELECT` runs against a separate database from the `INSERT INTO`, so I would have to setup a cross-database connection for this which feels a little too complicated. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1555#issuecomment-997128080 | https://api.github.com/repos/simonw/datasette/issues/1555 | 997128080 | IC_kwDOBm6k_c47bveQ | 9599 | 2021-12-18T02:30:19Z | 2021-12-18T02:30:19Z | OWNER | I think all of these queries happen in one place - in the `populate_schema_tables()` function - so optimizing them might be localized to just that area of the code, which would be nice: https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L97-L183 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079149656 | |
https://github.com/simonw/datasette/issues/1561#issuecomment-997127784 | https://api.github.com/repos/simonw/datasette/issues/1561 | 997127784 | IC_kwDOBm6k_c47bvZo | 9599 | 2021-12-18T02:27:56Z | 2021-12-18T02:27:56Z | OWNER | Oh that's an interesting solution, combining the hashes of all of the individual databases. I'm actually not a big fan of `hashed_url` mode - I implemented it right at the start of the project because it felt like a clever hack, and then ended up making it not-the-default a few years ago: - #418 - #419 - #421 I've since not found myself wanting to use it at all for any of my projects - which makes me nervous, because it means there's a pretty complex feature that I'm not using at all, so it's only really protected by the existing unit tests for it. What I'd really like to do is figure out how to have hashed URL mode work entirely as a plugin - then I could extract it from Datasette core entirely (which would simplify a bunch of stuff) but people who find the optimization useful would be able to access it. I'm not sure that the existing plugin hooks are robust enough to do that yet though. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1082765654 | |
https://github.com/simonw/datasette/issues/1563#issuecomment-997127084 | https://api.github.com/repos/simonw/datasette/issues/1563 | 997127084 | IC_kwDOBm6k_c47bvOs | 9599 | 2021-12-18T02:22:30Z | 2021-12-18T02:22:30Z | OWNER | Docs here: https://docs.datasette.io/en/latest/internals.html#datasette-class | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083573206 | |
https://github.com/simonw/datasette/issues/1563#issuecomment-997125191 | https://api.github.com/repos/simonw/datasette/issues/1563 | 997125191 | IC_kwDOBm6k_c47buxH | 9599 | 2021-12-18T02:10:20Z | 2021-12-18T02:10:20Z | OWNER | I should document the usage of this constructor in https://docs.datasette.io/en/stable/internals.html#datasette-class | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083573206 | |
https://github.com/simonw/datasette/issues/1546#issuecomment-997124280 | https://api.github.com/repos/simonw/datasette/issues/1546 | 997124280 | IC_kwDOBm6k_c47bui4 | 9599 | 2021-12-18T02:05:16Z | 2021-12-18T02:05:16Z | OWNER | Sure - there are actually several levels to this. The code that creates connections to the database is this: https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/database.py#L72-L95 For files on disk, it does this: ```python # For read-only connections conn = sqlite3.connect( "file:my.db?mode=ro", uri=True, check_same_thread=False) # For connections that should be treated as immutable: conn = sqlite3.connect( "file:my.db?immutable=1", uri=True, check_same_thread=False) ``` For in-memory databases it runs this after the connection has been created: ```python conn.execute("PRAGMA query_only=1") ``` SQLite `PRAGMA` queries are treated as dangerous: someone could run `PRAGMA query_only=0` to turn that previous option off for example. So this function runs against any incoming SQL to verify that it looks like a `SELECT ...` and doesn't have anything like that in it. https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/utils/__init__.py#L195-L204 You can see the tests for that here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/tests/test_utils.py#L136-L170 | { "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0 } |
1076057610 | |
https://github.com/simonw/datasette/issues/1564#issuecomment-997122938 | https://api.github.com/repos/simonw/datasette/issues/1564 | 997122938 | IC_kwDOBm6k_c47buN6 | 9599 | 2021-12-18T01:55:25Z | 2021-12-18T01:55:46Z | OWNER | Made this change while working on this issue: - #1567 I'm going to write a test for this that uses that `sleep()` SQL function from c35b84a2aabe2f14aeacf6cda4110ae1e94d6059. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083581011 | |
https://github.com/simonw/datasette/issues/1565#issuecomment-997121215 | https://api.github.com/repos/simonw/datasette/issues/1565 | 997121215 | IC_kwDOBm6k_c47bty_ | 9599 | 2021-12-18T01:45:44Z | 2021-12-18T01:45:44Z | OWNER | I want to get this into Datasette 0.60 - #1566 - it's a small change that can unlock a lot of potential. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083657868 | |
https://github.com/simonw/datasette/issues/621#issuecomment-997120723 | https://api.github.com/repos/simonw/datasette/issues/621 | 997120723 | IC_kwDOBm6k_c47btrT | 9599 | 2021-12-18T01:42:33Z | 2021-12-18T01:42:33Z | OWNER | I refactored this code out into the `filters.py` module in aa7f0037a46eb76ae6fe9bf2a1f616c58738ecdf | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
520681725 | |
https://github.com/simonw/datasette/issues/617#issuecomment-552253893 | https://api.github.com/repos/simonw/datasette/issues/617 | 552253893 | MDEyOklzc3VlQ29tbWVudDU1MjI1Mzg5Mw== | 9599 | 2019-11-11T00:46:42Z | 2021-12-18T01:41:47Z | OWNER | As noted in https://github.com/simonw/datasette/issues/621#issuecomment-552253208 a common pattern in this method is blocks of code that append new items to the `where_clauses`, `params` and `extra_human_descriptions` arrays. This is a useful refactoring opportunity. Code that fits this pattern: * The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)` * Code that handles `?_where=`: `where_clauses.extend(request.args["_where"])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses * The `_through=` code, see #621 for details * The code that deals with `?_search=` FTS The keyset pagination code modifies `where_clauses` and `params` too, but I don't think it's quite going to work with the same abstraction that would cover the above examples. [UPDATE December 2021 - this comment became the basis for a new `filters_from_request` plugin hook, see also #473] | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
519613116 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-981153060 | https://api.github.com/repos/simonw/datasette/issues/1518 | 981153060 | IC_kwDOBm6k_c46ezUk | 9599 | 2021-11-28T21:13:09Z | 2021-12-17T23:37:08Z | OWNER | Two new requirements inspired by work on the `datasette-table` (and `datasette-notebook`) projects: - #1533 - #1534 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-997082845 | https://api.github.com/repos/simonw/datasette/issues/1518 | 997082845 | IC_kwDOBm6k_c47bkbd | 9599 | 2021-12-17T23:10:09Z | 2021-12-17T23:10:17Z | OWNER | These changes so far are now in the 0.60a0 alpha: https://github.com/simonw/datasette/releases/tag/0.60a0 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/datasette/pull/1559#issuecomment-997082676 | https://api.github.com/repos/simonw/datasette/issues/1559 | 997082676 | IC_kwDOBm6k_c47bkY0 | 9599 | 2021-12-17T23:09:41Z | 2021-12-17T23:09:41Z | OWNER | This is now available to try out in Datasette 0.60a0: https://github.com/simonw/datasette/releases/tag/0.60a0 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1082743068 | |
https://github.com/simonw/datasette/pull/1562#issuecomment-997082189 | https://api.github.com/repos/simonw/datasette/issues/1562 | 997082189 | IC_kwDOBm6k_c47bkRN | 9599 | 2021-12-17T23:08:14Z | 2021-12-17T23:08:14Z | OWNER | Oh that makes sense: In Python 3.6 this happens: ``` Collecting janus<1.1,>=0.6.2 Using cached janus-0.7.0-py3-none-any.whl (6.9 kB) ``` While in Python 3.7 or higher this happens: ``` Collecting janus<1.1,>=0.6.2 Downloading janus-1.0.0-py3-none-any.whl (6.9 kB) ``` So this is safe to apply because `pip` is smart enough to pick the version of Janus that works for that Python version. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083246400 | |
https://github.com/simonw/datasette/pull/1562#issuecomment-997081673 | https://api.github.com/repos/simonw/datasette/issues/1562 | 997081673 | IC_kwDOBm6k_c47bkJJ | 9599 | 2021-12-17T23:06:38Z | 2021-12-17T23:06:38Z | OWNER | From this diff between `0.7.0` and `1.0`: https://github.com/aio-libs/janus/compare/v0.7.0...v1.0.0 It looks like the only change relevant to compatibility is `loop = asyncio.get_running_loop()` directly instead of falling back to `asyncio.get_event_loop()` if `get_running_loop` isn't available. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083246400 | |
https://github.com/simonw/datasette/pull/1562#issuecomment-997080352 | https://api.github.com/repos/simonw/datasette/issues/1562 | 997080352 | IC_kwDOBm6k_c47bj0g | 9599 | 2021-12-17T23:03:08Z | 2021-12-17T23:03:08Z | OWNER | They say they've dropped 3.6 support, but Datasette's tests against 3.6 are still passing. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083246400 | |
https://github.com/simonw/datasette/issues/1566#issuecomment-997078812 | https://api.github.com/repos/simonw/datasette/issues/1566 | 997078812 | IC_kwDOBm6k_c47bjcc | 9599 | 2021-12-17T22:58:55Z | 2021-12-17T22:58:55Z | OWNER | The release notes for the 0.60a0 alpha will be useful here: https://github.com/simonw/datasette/releases/tag/0.60a0 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083669410 | |
https://github.com/simonw/datasette/issues/1565#issuecomment-997077410 | https://api.github.com/repos/simonw/datasette/issues/1565 | 997077410 | IC_kwDOBm6k_c47bjGi | 9599 | 2021-12-17T22:54:45Z | 2021-12-17T22:54:45Z | OWNER | The table page should expose the query both with and without the `limit` clause. The above gave me back: ```sql select id, ACCESS_TYP, UNIT_ID, UNIT_NAME, SUID_NMA, AGNCY_ID, AGNCY_NAME, AGNCY_LEV, AGNCY_TYP, AGNCY_WEB, LAYER, MNG_AG_ID, MNG_AGENCY, MNG_AG_LEV, MNG_AG_TYP, PARK_URL, COUNTY, ACRES, LABEL_NAME, YR_EST, DES_TP, GAP_STS, geometry from CPAD_2020a_Units where "AGNCY_LEV" = :p0 order by id limit 101 ``` But I actually wanted to run a `fetch()` against a version of that without the `order by id limit 101` bit (I wanted to figure out the `Extent()` of the `geometry` column) - so I need something like `datasette.table_sql_no_order_no_limit`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083657868 | |
https://github.com/simonw/datasette/issues/1565#issuecomment-997069128 | https://api.github.com/repos/simonw/datasette/issues/1565 | 997069128 | IC_kwDOBm6k_c47bhFI | 9599 | 2021-12-17T22:31:18Z | 2021-12-17T22:31:18Z | OWNER | This should aim to be as consistent as possible with the various arguments to hooks on https://docs.datasette.io/en/stable/plugin_hooks.html | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1083657868 | |
https://github.com/simonw/datasette/pull/1559#issuecomment-996961196 | https://api.github.com/repos/simonw/datasette/issues/1559 | 996961196 | IC_kwDOBm6k_c47bGus | 9599 | 2021-12-17T19:00:53Z | 2021-12-17T19:00:53Z | OWNER | I'm going to merge this to `main` now. I can continue the refactoring there, but having it in `main` means I can put out an alpha release with the new hook which will unblock me from running tests against it in this repo: https://github.com/simonw/datasette-leaflet-freedraw/pull/8 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1082743068 | |
https://github.com/simonw/datasette/pull/1559#issuecomment-996959325 | https://api.github.com/repos/simonw/datasette/issues/1559 | 996959325 | IC_kwDOBm6k_c47bGRd | 9599 | 2021-12-17T18:59:54Z | 2021-12-17T18:59:54Z | OWNER | I've convinced myself that this plugin hook design is good through this `datasette-leaflet-freedraw` prototype: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1082743068 | |
https://github.com/simonw/datasette/issues/473#issuecomment-996958442 | https://api.github.com/repos/simonw/datasette/issues/473 | 996958442 | IC_kwDOBm6k_c47bGDq | 9599 | 2021-12-17T18:59:27Z | 2021-12-17T18:59:27Z | OWNER | I'm happy with how the prototype that used this plugin in `datasette-leaflet-freedraw` turned out: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
445850934 | |
https://github.com/simonw/datasette/issues/473#issuecomment-996345233 | https://api.github.com/repos/simonw/datasette/issues/473 | 996345233 | IC_kwDOBm6k_c47YwWR | 9599 | 2021-12-17T01:20:31Z | 2021-12-17T18:13:01Z | OWNER | I could use this hook to add table filtering on a map to the existing `datasette-leaflet-freedraw` plugin. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
445850934 | |
https://github.com/simonw/datasette/pull/1559#issuecomment-996895423 | https://api.github.com/repos/simonw/datasette/issues/1559 | 996895423 | IC_kwDOBm6k_c47a2q_ | 9599 | 2021-12-17T17:28:44Z | 2021-12-17T17:28:44Z | OWNER | Before I land this I'm going to build one prototype plugin against it to confirm that the new hook is useful in its current shape. I'll add support for filtering a table by drawing on a map to https://datasette.io/plugins/datasette-leaflet-freedraw | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1082743068 | |
https://github.com/simonw/datasette/pull/1204#issuecomment-996488925 | https://api.github.com/repos/simonw/datasette/issues/1204 | 996488925 | IC_kwDOBm6k_c47ZTbd | 9599 | 2021-12-17T07:10:48Z | 2021-12-17T07:10:48Z | OWNER | I think this is missing the `_macro.html` template file but I have that in my Dropbox. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
793002853 | |
https://github.com/simonw/datasette/issues/473#issuecomment-996484551 | https://api.github.com/repos/simonw/datasette/issues/473 | 996484551 | IC_kwDOBm6k_c47ZSXH | 9599 | 2021-12-17T07:02:21Z | 2021-12-17T07:04:23Z | OWNER | The one slightly weird thing about this hook is how it adds `extra_context` without an obvious way for plugins to add extra HTML to the templates based on that context. Maybe I need the proposed mechanism from - #1191 Which has an in-progress PR: - #1204 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
445850934 | |
https://github.com/simonw/datasette/issues/1191#issuecomment-761104933 | https://api.github.com/repos/simonw/datasette/issues/1191 | 761104933 | MDEyOklzc3VlQ29tbWVudDc2MTEwNDkzMw== | 9599 | 2021-01-15T18:21:26Z | 2021-12-17T07:03:02Z | OWNER | Also related: #857 (comprehensive documentation of variables available to templates) - since then the plugin hook could be fed the full template context and use that to do its thing. Or maybe the plugin hooks gets to return the name of a template that should be `{% include %}` into the page at that point? But the plugin may want to add extra context that is available to that template include. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
787098345 | |
https://github.com/simonw/datasette/pull/1559#issuecomment-996286808 | https://api.github.com/repos/simonw/datasette/issues/1559 | 996286808 | IC_kwDOBm6k_c47YiFY | 9599 | 2021-12-17T00:01:43Z | 2021-12-17T00:01:43Z | OWNER | This already has tests and documentation, and I've used it to refactor out the logic for `?_where=` and `?_search=` and `?_through=`. Do I like this enough to land it on `main`? Also, I think I can still use it to refactor out the `Filters` code that implements `?col=x` and `?col__lt=5` and suchlike. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1082743068 | |
https://github.com/simonw/datasette/issues/473#issuecomment-996286199 | https://api.github.com/repos/simonw/datasette/issues/473 | 996286199 | IC_kwDOBm6k_c47Yh73 | 9599 | 2021-12-17T00:00:22Z | 2021-12-17T00:00:22Z | OWNER | Documentation for that hook in the PR branch: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
445850934 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-996286104 | https://api.github.com/repos/simonw/datasette/issues/1518 | 996286104 | IC_kwDOBm6k_c47Yh6Y | 9599 | 2021-12-17T00:00:07Z | 2021-12-17T00:00:07Z | OWNER | Documentation of the new hook in the PR: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/datasette/issues/473#issuecomment-996275108 | https://api.github.com/repos/simonw/datasette/issues/473 | 996275108 | IC_kwDOBm6k_c47YfOk | 9599 | 2021-12-16T23:32:22Z | 2021-12-16T23:32:30Z | OWNER | This filter design can only influence the `where` component of the SQL clause - it's not able to modify the `SELECT` columns or adjust the `ORDER BY` or `OFFSET LIMIT` parts. I think that's OK. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
445850934 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-996272906 | https://api.github.com/repos/simonw/datasette/issues/1518 | 996272906 | IC_kwDOBm6k_c47YesK | 9599 | 2021-12-16T23:27:42Z | 2021-12-16T23:27:42Z | OWNER | Got a TIL out of this: https://til.simonwillison.net/pluggy/multiple-hooks-same-file | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/datasette/issues/473#issuecomment-996267817 | https://api.github.com/repos/simonw/datasette/issues/473 | 996267817 | IC_kwDOBm6k_c47Ydcp | 9599 | 2021-12-16T23:17:52Z | 2021-12-16T23:19:00Z | OWNER | I revisited this idea in #1518 and came up with a slightly different name and design for the hook: ```python @hookspec def filters_from_request(request, database, table, datasette): """ Return FilterArguments( where_clauses=[str, str, str], params={}, human_descriptions=[str, str, str], extra_context={} ) based on the request""" ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
445850934 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-996264617 | https://api.github.com/repos/simonw/datasette/issues/1518 | 996264617 | IC_kwDOBm6k_c47Ycqp | 9599 | 2021-12-16T23:11:12Z | 2021-12-16T23:11:12Z | OWNER | I managed to extract both `_search=` and `_where=` out using a prototype of that hook. I wonder if it could extract the complex code for `?_next` too? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-996250585 | https://api.github.com/repos/simonw/datasette/issues/1518 | 996250585 | IC_kwDOBm6k_c47YZPZ | 9599 | 2021-12-16T22:43:37Z | 2021-12-16T22:45:07Z | OWNER | Ran into a problem prototyping that hook up for handling `?_where=` - that feature also adds a little bit of extra template context in order to show the interface for removing wheres - the `extra_wheres_for_ui` variable: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L457-L463 Maybe change to this? ```python class FilterArguments(NamedTuple): where_clauses: List[str] params: Dict[str, Union[str, int, float]] human_descriptions: List[str] extra_context: Dict[str, Any] ``` That might be necessary for `_search` too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-996248713 | https://api.github.com/repos/simonw/datasette/issues/1518 | 996248713 | IC_kwDOBm6k_c47YYyJ | 9599 | 2021-12-16T22:39:47Z | 2021-12-16T22:39:47Z | OWNER | The hook could return a named tuple like this one: ```python from typing import NamedTuple, List, Optional, Union, Dict class FilterArguments(NamedTuple): where_clauses: List[str] params: Dict[str, Union[str, int, float]] human_descriptions: List[str] ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-996240802 | https://api.github.com/repos/simonw/datasette/issues/1518 | 996240802 | IC_kwDOBm6k_c47YW2i | 9599 | 2021-12-16T22:25:00Z | 2021-12-16T22:36:04Z | OWNER | I think that plugin hook would get given the `request` object (and `datasette` and the name of the database and table) and returns a list of SQL fragments, a dictionary of lookup arguments and a list of human-description fragments - or an awaitable. `filters_from_request(request, database, table, datasette)` perhaps? (Similar in name to `actor_from_request`). ```python @hookspec def filters_from_request(request, database, table, datasette): """Return (where_clauses, params_dict, human_descriptions) based on the request""" ``` Turns out that's pretty much exactly what I implemented in 5116c4ec8aed5091e1f75415424b80f613518dc6 for #473: ```python @hookspec def table_filter(): "Custom filtering of the current table based on the request" ``` ```python TableFilter = namedtuple("TableFilter", ( "human_description_extras", "where_clauses", "params") ) ``` ```python # filter_arguments plugin hook support for awaitable_fn in pm.hook.table_filter(): extras = await awaitable_fn( view=self, name=name, table=table, request=request ) human_description_extras.extend(extras.human_description_extras) where_clauses.extend(extras.where_clauses) params.update(extras.params) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/sqlite-utils/issues/358#issuecomment-996232461 | https://api.github.com/repos/simonw/sqlite-utils/issues/358 | 996232461 | IC_kwDOCGYnMM47YU0N | 9599 | 2021-12-16T22:10:39Z | 2021-12-16T22:10:39Z | OWNER | This goes beyond the `transform()` method - the curious methods that create new SQL tables could benefit from the ability to add `CHECK` constraints too. I haven't used these myself, do you have any `CREATE TABLE` examples that use them that you can share? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1082651698 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-996227713 | https://api.github.com/repos/simonw/datasette/issues/1518 | 996227713 | IC_kwDOBm6k_c47YTqB | 9599 | 2021-12-16T22:02:35Z | 2021-12-16T22:03:55Z | OWNER | Is there an opportunity to refactor things using a new plugin hook here? Maybe the `register_filters` hook from #473, where the hook becomes responsible for building where clauses (and human descriptions of them) based on the incoming query string. That version dealt with `Filter` classes, but those might be a bit too low-level for this. `?_spatial_within=GEOJSON` was an interesting idea attached to that issue. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-996225889 | https://api.github.com/repos/simonw/datasette/issues/1518 | 996225889 | IC_kwDOBm6k_c47YTNh | 9599 | 2021-12-16T21:59:32Z | 2021-12-16T22:00:42Z | OWNER | I added a ton of comments to the `data()` method which really helps get a better feel for how this all works: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L322 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-996225235 | https://api.github.com/repos/simonw/datasette/issues/1518 | 996225235 | IC_kwDOBm6k_c47YTDT | 9599 | 2021-12-16T21:58:24Z | 2021-12-16T21:58:41Z | OWNER | A fundamental operation of this view is to construct the SQL query and accompanying human description based on the incoming query string parameters. The human description is the bit at the top of https://latest.datasette.io/fixtures/searchable?_search=dog&_sort=pk&_facet=text2&text2=sara+weasel that says: > 1 row where search matches "dog" and text2 = "sara weasel" sorted by pk (Also used in the page `<title>`). The code actually gathers three things: - Fragments of the `where` clause, for example ` "text2" = :p0` - Parameters, e.g. `{"p0": "sara weasel"}` - Human description components, e.g. `text2 = "sara weasel"` Some operations such as `?_where=` don't currently provide an extra human description component. `_where=` also doesn't populate a parameter, but maybe it could? Would be neat if in the future `?_where=foo+=+:bar` worked and added a `bar` input field to the screen, as seen with custom queries. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/datasette/issues/1518#issuecomment-996219117 | https://api.github.com/repos/simonw/datasette/issues/1518 | 996219117 | IC_kwDOBm6k_c47YRjt | 9599 | 2021-12-16T21:47:51Z | 2021-12-16T21:49:24Z | OWNER | Should facets really not be displayed on pages past page one (where `?_next=` is set)? That made sense to me at the time, but I'm now having second thoughts about it. I guess it's a useful performance tweak for when crawlers keep hitting the `?_next=` link. Actually it looks like facets DO display on subsequent pages, e.g. on https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 - but facet suggestions do not, thanks to this code: https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/views/table.py#L777-L785 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1058072543 | |
https://github.com/simonw/datasette/issues/1558#issuecomment-996204369 | https://api.github.com/repos/simonw/datasette/issues/1558 | 996204369 | IC_kwDOBm6k_c47YN9R | 9599 | 2021-12-16T21:23:25Z | 2021-12-16T21:23:25Z | OWNER | Related: Following the fix for #625 I noticed that `facets_timed_out` gives you just the column name, but doesn't let you know which particular type of facet (`date` or `array` for example) suffered the timeout: https://github.com/simonw/datasette/blob/0d4145d0f4d8b2a7edc1ba4aac1be56cd536a10a/datasette/facets.py#L269-L270 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1082584499 | |
https://github.com/simonw/sqlite-utils/issues/357#issuecomment-996179930 | https://api.github.com/repos/simonw/sqlite-utils/issues/357 | 996179930 | IC_kwDOCGYnMM47YH_a | 9599 | 2021-12-16T20:43:19Z | 2021-12-16T20:43:19Z | OWNER | Thanks! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1079422215 | |
https://github.com/simonw/datasette/issues/625#issuecomment-996170510 | https://api.github.com/repos/simonw/datasette/issues/625 | 996170510 | IC_kwDOBm6k_c47YFsO | 9599 | 2021-12-16T20:27:41Z | 2021-12-16T20:27:41Z | OWNER | And here's the new JSON: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1 ``` { "database": "fixtures", "table": "facetable", "is_view": false, "human_description_en": "", ... "facet_results": { "created": { "name": "created", "type": "column", ... }, "tags": { "name": "tags", "type": "column", ... }, "created_2": { "name": "created", "type": "date", ... }, "tags_2": { "name": "tags", "type": "array", ... } } } ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
520740741 | |
https://github.com/simonw/datasette/issues/625#issuecomment-996165659 | https://api.github.com/repos/simonw/datasette/issues/625 | 996165659 | IC_kwDOBm6k_c47YEgb | 9599 | 2021-12-16T20:19:53Z | 2021-12-16T20:19:53Z | OWNER | Demo of the fix: https://latest.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags#facet-tags | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
520740741 | |
https://github.com/simonw/datasette/issues/625#issuecomment-996161380 | https://api.github.com/repos/simonw/datasette/issues/625 | 996161380 | IC_kwDOBm6k_c47YDdk | 9599 | 2021-12-16T20:13:05Z | 2021-12-16T20:13:05Z | OWNER | I updated the example code in the facet plugin hook documentation: https://github.com/simonw/datasette/blob/95d0dd7a1cf6be6b7da41e1404184217eb93f64a/docs/plugin_hooks.rst#register_facet_classes | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
520740741 | |
https://github.com/simonw/datasette/issues/625#issuecomment-996152213 | https://api.github.com/repos/simonw/datasette/issues/625 | 996152213 | IC_kwDOBm6k_c47YBOV | 9599 | 2021-12-16T19:59:46Z | 2021-12-16T20:00:05Z | OWNER | Since no-one is using that plugin hook I'm going to alter its contract slightly. I'll still keep the existing JSON format working though (until 1.0), since it's much more likely that people are using that JSON somewhere. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
520740741 | |
https://github.com/simonw/datasette/issues/830#issuecomment-996151246 | https://api.github.com/repos/simonw/datasette/issues/830 | 996151246 | IC_kwDOBm6k_c47YA_O | 9599 | 2021-12-16T19:58:22Z | 2021-12-16T19:58:22Z | OWNER | As of today, 16 December 2021, I'm still not seeing any evidence that anyone is using this hook (yet) according to GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
636511683 |