html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/pull/2052#issuecomment-1530817667,https://api.github.com/repos/simonw/datasette/issues/2052,1530817667,IC_kwDOBm6k_c5bPmyD,193185,2023-05-02T03:24:53Z,2023-05-02T03:24:53Z,CONTRIBUTOR,"Thanks for putting this together! I've been slammed with work/personal stuff so haven't been able to actually prototype anything with this. :(
tl;dr: I think this would be useful immediately as is. It might also be nice if the plugins could return `Promise`s.
The long version: I read the design notes and example plugin. I think I'd be able to use this in [datasette-ui-extras](https://github.com/cldellow/datasette-ui-extras) for my lazy-facets feature.
The lazy-facets feature tries to provide a snappier user experience. It does this by altering how suggested facets work.
First, at page render time:
(A) it lies to Datasette and claims that no columns support facets, this avoids the lengthy delays/timeouts that can happen if the dataset is large.
(B) there's a python plugin that implements the [extra_body_script](https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette) hook, to write out the list of column names for future use by JavaScript
Second, at page load time: there is some JavaScript that:
(C) makes AJAX requests to suggest facets for each column - it makes 1 request per column, using the data from (B)
(D) wires up the column menus to add Facet-by-this options for each facet
With the currently proposed plugin scheme, I think (D) could be moved into the plugin. I'd do the ajax requests, then register the plugin.
If the plugin scheme also supported promises, I think (B) and (C) could also be moved into the plugin.
Does that make sense? Sorry for the wall of text!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421081939,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421081939,IC_kwDOCGYnMM5Us_1T,193185,2023-02-07T16:42:25Z,2023-02-07T16:43:42Z,NONE,"Ha, yes, I might end up making something very niche. That's OK.
I'm building a UI for [Datasette](https://datasette.io/) that lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed. And so on. I had hoped that I could just lean on sqlite-utils, but I think it's a little too surprising.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421033725,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421033725,IC_kwDOCGYnMM5Us0D9,193185,2023-02-07T16:12:13Z,2023-02-07T16:12:13Z,NONE,"I think the bigger issue is that `sqlite-utils` mixes mechanism (it implements the [12-step way to alter SQLite tables](https://www.sqlite.org/lang_altertable.html#otheralter)) and policy (it has an opinionated stance on what column types should be used).
That might be a design choice to make it accessible to users by providing a reasonable set of defaults, but it doesn't quite fit my use case.
It might make sense to extract a separate library that provides just the mechanisms, and then `sqlite-utils` would sit on top of that library with its opinionated set of policies.
That would be a very big change, though.
I might take a stab at extracting the library, but just for the table schema migration piece, not all the other features that `sqlite-utils` supports. I wouldn't expect `sqlite-utils` to depend on it.
Part of my motivation is that I want to provide some other abilities, too, like support for CHECK constraints. I see that the issue in this repo (https://github.com/simonw/sqlite-utils/issues/358) proposes a bunch of short-hand constraints, which I wouldn't want to accidentally expose to people -- I want a layer that is a 1:1 mapping to SQLite.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420992261,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420992261,IC_kwDOCGYnMM5Usp8F,193185,2023-02-07T15:45:58Z,2023-02-07T15:45:58Z,NONE,"I'd support that, but I'm not the author of this library.
One challenge is that would be a breaking change. Do you see a way to enable it without affecting existing users or bumping the major version number?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420809773,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420809773,IC_kwDOCGYnMM5Ur9Yt,193185,2023-02-07T13:53:01Z,2023-02-07T13:53:01Z,NONE,"Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178
I suspect you could overwrite the datetime entry to achieve what you want","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419740776,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1419740776,IC_kwDOCGYnMM5Un4Zo,193185,2023-02-06T20:59:01Z,2023-02-06T20:59:01Z,NONE,"That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,
https://github.com/simonw/datasette/issues/1696#issuecomment-1407767434,https://api.github.com/repos/simonw/datasette/issues/1696,1407767434,IC_kwDOBm6k_c5T6NOK,193185,2023-01-29T20:56:20Z,2023-01-29T20:56:20Z,CONTRIBUTOR,"I did some horrible things in https://github.com/cldellow/datasette-ui-extras/issues/2 to enable this in my plugin -- example here: https://dux-demo.fly.dev/cooking/posts?_facet=owner_user_id&owner_user_id=67
The implementation relies on two things:
- a `filters_from_request` hook that adds a good human description (unfortunately, without the benefit of the CSS styling you mention)
- doing something evil to hijack the `exact` and `not` operators in the `Filters` class. We can't leave them as is, or we'll get 2 human descriptions -- the built-in Datasette one and the one from my plugin. We can't remove them, or the filters UI will stop supporting the `=` and `!=` operators
This got me thinking: it'd be neat if the list of operators that the filters UI supported wasn't a closed set.
A motivating example: adding a geospatial `NEAR` operator. Ideally it'd take two arguments - a target point and a radius, so you could express a filter like `find me all rows whose lat/lng are within 10km of 43.4516° N, 80.4925° W`. (Optionally, the UI could be enhanced if the geonames database was loaded and queried, so a user could say `find me all rows whose lat/lng are within 10km of Kitchener, ON`, and the city gets translated to a lat/lng for them)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1186696202,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407716963,https://api.github.com/repos/simonw/datasette/issues/2008,1407716963,IC_kwDOBm6k_c5T6A5j,193185,2023-01-29T17:04:03Z,2023-01-29T17:04:03Z,CONTRIBUTOR,"Performance tests - I think most places don't have them as a formal gate enforced by CI. TypeScript and scalac seem to have tests that run to capture timings. The timings are included by a bot as a comment or build check, and also stored in a database so you can graph changes over time to spot regressions. Probably overkill for Datasette!
Window functions - oh, good point. Looks like Ubuntu shipped JSON1 support as far back as sqlite 3.11. I'll let this PR linger until there's a way to run against different SQLite versions. For now, I'm shipping this with `datasette-ui-extras`, since I think it's OK for a plugin to enforce a higher minimum requirement.
Tests - there actually did end up being test changes to capture the undercount bug of the current implementation, so the current implementation would fail against the new tests.
Perhaps a non-window function version could be written that uses `random()` instead of `row_number() over ()` in order to get a unique key. It's technically not unique, but in practice, I imagine it'll work well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407561308,https://api.github.com/repos/simonw/datasette/issues/2008,1407561308,IC_kwDOBm6k_c5T5a5c,193185,2023-01-29T04:50:50Z,2023-01-29T04:50:50Z,CONTRIBUTOR,I pushed a revised version which ends up being faster -- the example which currently takes 4 seconds now runs in 500ms.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407558284,https://api.github.com/repos/simonw/datasette/issues/2008,1407558284,IC_kwDOBm6k_c5T5aKM,193185,2023-01-29T04:23:58Z,2023-01-29T04:24:27Z,CONTRIBUTOR,"Ack, this PR is broken. I see now that the `inner.*` is necessary for ensuring the correct count in the face of rows having duplicate values in views.
That fixes the overcounting, but I think can undercount when the rows have the same data, eg a view like:
```sql
SELECT '[""bar""]' tags UNION ALL SELECT '[""bar""]'
```
will produce a count of `{""bar"": 1 }`, when it should be `{""bar"": 2}`. In fact, this could apply in tables without primary keys, too.
If `inner` came from a base table that had a primary key or a rowid, we could use those column(s) to solve that case.
I guess a general solution would be to compute a window function so we have a distinct ID for each row. Will fiddle to see if I can get that working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,
https://github.com/simonw/datasette/issues/1973#issuecomment-1407523547,https://api.github.com/repos/simonw/datasette/issues/1973,1407523547,IC_kwDOBm6k_c5T5Rrb,193185,2023-01-29T00:40:31Z,2023-01-29T00:40:31Z,CONTRIBUTOR,"A +1 for switching to `CustomRow`: I think you currently only get a `CustomRow` if the result set had a column that was an fkey ([this code](https://github.com/simonw/datasette/blob/3c352b7132ef09b829abb69a0da0ad00be5edef9/datasette/views/table.py#L667-L682))
Otherwise you get vanilla `sqlite3.Row`s, which will fail if you try to access `.columns` or lookup the cell by name, which surprised me recently","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515815014,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407470429,https://api.github.com/repos/simonw/datasette/issues/2008,1407470429,IC_kwDOBm6k_c5T5Etd,193185,2023-01-28T19:34:29Z,2023-01-28T19:34:29Z,CONTRIBUTOR,"I don't know how/if you do automated tests for performance, so I haven't changed any of the tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,
https://github.com/simonw/datasette/issues/2001#issuecomment-1403078134,https://api.github.com/repos/simonw/datasette/issues/2001,1403078134,IC_kwDOBm6k_c5ToUX2,193185,2023-01-25T04:20:43Z,2023-01-25T04:22:28Z,CONTRIBUTOR,"I'm on Ubuntu, unfortunately. :( Would it still be relevant?
I think I've narrowed things down a bit more.
Even `sqlite3_free(sqlite3_malloc(128))` segfaults -- this suggests to me that it's something about the sqlite3 library that was loaded, vs, say, getting the wrong db handle when I go spelunking in the Connection object.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1553615704,
https://github.com/simonw/datasette/issues/2001#issuecomment-1403053144,https://api.github.com/repos/simonw/datasette/issues/2001,1403053144,IC_kwDOBm6k_c5ToORY,193185,2023-01-25T03:34:53Z,2023-01-25T03:34:53Z,CONTRIBUTOR,"Your comment introduced me to this issue in sqlite and to the `ctypes` module - thanks!
> I also hope that the datasette developers will enable this mode in a test environment [...]
> perhaps we could figure out how to invoke it using `ctypes`
I'm not a Datasette developer, but I _am_ curious to learn more about getting unholy access to the sqlite C APIs inside of Datasette. (Such access could also help #1293, and if done without grovelling inside of pysqlite's Connection object for the db handle, could even be relatively safe.)
I experimented a bit. I came up with https://gist.github.com/cldellow/85bba507c314b127f85563869cd94820
If you run `python3 enable-strict-quoting-sqlite3.py`, it seems to set those flags correctly -- `SELECT ""foo""` fails where it would normally succeed.
But if you put it in a `plugins/` dir and run `datasette --plugins-dir plugins/`, it segfaults when it tries to call `sqlite3_db_config` on the connections created by Datasette.
I am... confused. I'm _pretty_ sure I'm using the same python and the same libsqlite3 in both scenarios, so I would expect it to work.
@gwk do you know anything that might help me debug the segfault? I gather that my approach of going grovelling inside of a `PyObject` is particularly dangerous, but I was thinking (a) it's necessary in order to test Datasette's use of the sqlite3 library and (b) even if it's not portable, it'd be good enough for running the tests on a single machine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1553615704,
https://github.com/simonw/datasette/issues/2000#issuecomment-1399847946,https://api.github.com/repos/simonw/datasette/issues/2000,1399847946,IC_kwDOBm6k_c5Tb_wK,193185,2023-01-23T06:08:00Z,2023-01-23T06:08:00Z,CONTRIBUTOR,"Actually, I discovered [your post](https://til.simonwillison.net/datasette/register-new-plugin-hooks) showing how a plugin can add a Datasette hook. That's wild! I've released `datasette-rewrite-sql` that adds this ability, albeit via monkey patching.
I had hoped to be able to expose `request` to the hook (or, even better `actor`) when the SQL was being run as a result of a user's HTTP request.
But some spelunking in the code makes me suspect that would actually require co-operation from Datasette itself. I'd be happy to be wrong and pointed in the right direction, though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1552368054,
https://github.com/simonw/datasette/issues/1973#issuecomment-1369044959,https://api.github.com/repos/simonw/datasette/issues/1973,1369044959,IC_kwDOBm6k_c5Rmfff,193185,2023-01-02T15:41:40Z,2023-01-02T15:41:40Z,CONTRIBUTOR,"Thanks for the response!
Yes, it does seem like a pretty nice developer experience--both the automagical labelling of fkeys, and the ability to index the row by column name in addition to column index.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515815014,
https://github.com/simonw/datasette/issues/236#issuecomment-612216820,https://api.github.com/repos/simonw/datasette/issues/236,612216820,MDEyOklzc3VlQ29tbWVudDYxMjIxNjgyMA==,193185,2020-04-10T21:03:38Z,2020-04-10T21:03:38Z,CONTRIBUTOR,"I made a repo at https://github.com/code402/datasette-lambda to demonstrate the idea, and scratch my personal itch for this.
The demo relies on some central authority having already published a public, reusable Lambda layer with Datasette & its dependencies. I think that differs from the other publish plugins which seem to mainly publish Dockerfiles that the host will interpret to install deps from a requirements.txt file.
I chose that approach because `uvloop` appears to be a dependency with native code that needs to be compiled for the target runtime environment. In this case, that's Amazon Linux 2. I'm not 100% clear on whether that's still required, because:
- maybe `uvloop` is only needed for `uvicorn`, which the demo doesn't actually use since HTTP routing is handled by API Gateway
- it seems like `uvloop` may be an optional, drop-in optimization for `asyncio` in any case (but I may be misreading this; I'm very much a Python noob)
If it's the case that `uvloop` is truly optional, then I think the publish plugin could do the packaging on the user's machine, regardless of what flavour of operating system they're on. That'd be a bit slower for the user, but would provide the most long-term flexibility in terms of supporting plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,
https://github.com/simonw/datasette/issues/236#issuecomment-608716819,https://api.github.com/repos/simonw/datasette/issues/236,608716819,MDEyOklzc3VlQ29tbWVudDYwODcxNjgxOQ==,193185,2020-04-03T22:19:00Z,2020-04-03T22:19:00Z,CONTRIBUTOR,"Hi Simon,
I'm thinking of attempting this. Can you clarify some questions I have?
1) I assume the goal is to have a CORS-friendly HTTPS endpoint that hosts the datasette service + user's db.
2) If that's the goal, I think Lambda alone is insufficient. Lambda provides the compute fabric, but not the HTTP routing. You'd also need to add Application Load Balancer or API Gateway to provide an HTTP endpoint that routes to the lambda function.
Do you have a preference between ALB or API GW? ALB has better economics at scale, but has a minimum monthly cost. API GW has worse per-request economics, but scales to zero when no requests are happening.
3) Does Datasette have any native components, or is it all pure python? If it has native bits, they'll likely need to be recompiled to work on Amazon Linux 2.
4) There are a few disparate services that need to be wired together to expose a Python service securely to the web. If I was doing this outside of the datasette publish system, I'd use an AWS CloudFormation template. Even within datasette, I think it still makes sense to use a CloudFormation template and just have the publish plugin invoke it (via the standard `aws` cli) with user-specified parameters. Does that sound reasonable to you?
Thanks for your help!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,