{"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1548617257", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1548617257, "node_id": "IC_kwDOBm6k_c5cTgYp", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-05-15T21:32:20Z", "updated_at": "2023-05-15T21:32:20Z", "author_association": "CONTRIBUTOR", "body": "> Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like makeColumnActions or makeAboveTablePanelConfigs supported returning a promise of arrays instead only returning plain arrays?\r\n\r\nThe latter - that you could return a promise of arrays, so it parallels the [\"await me maybe\" pattern in Datasette](https://simonwillison.net/2020/Sep/2/await-me-maybe/), where you can return either a value, a callable or an awaitable.\r\n\r\n> I have a hunch that what you're describing might be achievable without adding Promises to the API with something\r\n\r\nOops, I did a poor job explaining. Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.\r\n\r\nThis isn't that big of a deal - it'd be a nice ergonomic improvement, but nowhere near as a big of an improvement as having an officially sanctioned way to add stuff to the column menus in the first place.\r\n\r\nThis could also be layered on in a future commit without breaking v1 users, too, so it's not at all urgent.\r\n\r\n> especially if those lines are encapsulated by a function we provide (maybe something that's available on the window provided by Datasette as an inline script tag\r\n\r\nAh, this is maybe the the key point. Since it's all hosted inside Datasette, Datasette can provide some arbitrary sugar to make it easier to work with.\r\n\r\nMy experience with async scripts in JS is that people sometimes don't understand the race conditions inherent to them. If they copy/paste from a tutorial, it does just work. But then they'll delete half the code, and by chance it still works on their machine/Datasette templates, and now someone's headed for an annoying debugging session -- maybe them, maybe someone else who tries to re-use their plugin.\r\n\r\nAgain, a fairly minor thing, though.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1530822437", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1530822437, "node_id": "IC_kwDOBm6k_c5bPn8l", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-05-02T03:35:30Z", "updated_at": "2023-05-02T16:02:38Z", "author_association": "CONTRIBUTOR", "body": "Also, just checking - is this how I'd write bulletproof plugin registration code that is robust against the order in which the script tags load (eg if both my code and the Datasette code are loaded via a `<script async src='...'/>` tag)?\r\n\r\n```js\r\nif (window.__DATASETTE__)\r\n  go(window.__DATASETTE__);\r\nelse\r\n  document.addEventListener(\"datasette_init\", (evt) => go(evt.detail));\r\n\r\nfunction go(manager) {\r\n  manager.registerPlugin(...)\r\n}\r\n```\r\n\r\nI don't know if it'd make sense, but you could also consider the asynchronous queuing pattern that Google Analytics uses (see [this Stack Overflow post](https://stackoverflow.com/questions/6963779/whats-the-name-of-google-analytics-async-design-pattern-and-where-is-it-used) for more details):\r\n\r\n```js\r\n__DATASETTE__ = __DATASETTE__ || [];\r\n__DATASETTE__.push(go);\r\n\r\nfunction go(manager) {\r\n  manager.registerPlugin(...);\r\n}\r\n```", "reactions": "{\"total_count\": 2, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/2052#issuecomment-1530817667", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2052", "id": 1530817667, "node_id": "IC_kwDOBm6k_c5bPmyD", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-05-02T03:24:53Z", "updated_at": "2023-05-02T03:24:53Z", "author_association": "CONTRIBUTOR", "body": "Thanks for putting this together! I've been slammed with work/personal stuff so haven't been able to actually prototype anything with this. :(\r\n\r\ntl;dr: I think this would be useful immediately as is. It might also be nice if the plugins could return `Promise`s.\r\n\r\nThe long version: I read the design notes and example plugin. I think I'd be able to use this in [datasette-ui-extras](https://github.com/cldellow/datasette-ui-extras) for my lazy-facets feature.\r\n\r\nThe lazy-facets feature tries to provide a snappier user experience. It does this by altering how suggested facets work.\r\n\r\nFirst, at page render time:\r\n(A) it lies to Datasette and claims that no columns support facets, this avoids the lengthy delays/timeouts that can happen if the dataset is large.\r\n(B) there's a python plugin that implements the [extra_body_script](https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette) hook, to write out the list of column names for future use by JavaScript\r\n\r\nSecond, at page load time: there is some JavaScript that:\r\n(C) makes AJAX requests to suggest facets for each column - it makes 1 request per column, using the data from (B)\r\n(D) wires up the column menus to add Facet-by-this options for each facet\r\n\r\nWith the currently proposed plugin scheme, I think (D) could be moved into the plugin. I'd do the ajax requests, then register the plugin.\r\n\r\nIf the plugin scheme also supported promises, I think (B) and (C) could also be moved into the plugin.\r\n\r\nDoes that make sense? Sorry for the wall of text!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1651082214, "label": "feat: Javascript Plugin API (Custom panels, column menu items with JS actions)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2023#issuecomment-1425974877", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2023", "id": 1425974877, "node_id": "IC_kwDOBm6k_c5U_qZd", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-02-10T15:32:41Z", "updated_at": "2023-02-10T15:32:41Z", "author_association": "CONTRIBUTOR", "body": "I think this feature was removed in Datasette 0.61 and moved to a plugin. People who want hashed URLs can use the [datasette-hashed-urls](https://docs.datasette.io/en/stable/performance.html#performance-hashed-urls) plugin to achieve the same affect.\r\n\r\nIt looks like you're trying to disable hashed urls, so I think you can just remove that config setting and things will work.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1579695809, "label": "Error: Invalid setting 'hash_urls' in settings.json in 0.64.1"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421081939", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/524", "id": 1421081939, "node_id": "IC_kwDOCGYnMM5Us_1T", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-02-07T16:42:25Z", "updated_at": "2023-02-07T16:43:42Z", "author_association": "NONE", "body": "Ha, yes, I might end up making something very niche. That's OK.\r\n\r\nI'm building a UI for [Datasette](https://datasette.io/) that lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed. And so on. I had hoped that I could just lean on sqlite-utils, but I think it's a little too surprising.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1572766460, "label": "Transformation type `--type DATETIME`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421033725", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/524", "id": 1421033725, "node_id": "IC_kwDOCGYnMM5Us0D9", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-02-07T16:12:13Z", "updated_at": "2023-02-07T16:12:13Z", "author_association": "NONE", "body": "I think the bigger issue is that `sqlite-utils` mixes mechanism (it implements the [12-step way to alter SQLite tables](https://www.sqlite.org/lang_altertable.html#otheralter)) and policy (it has an opinionated stance on what column types should be used).\r\n\r\nThat might be a design choice to make it accessible to users by providing a reasonable set of defaults, but it doesn't quite fit my use case.\r\n\r\nIt might make sense to extract a separate library that provides just the mechanisms, and then `sqlite-utils` would sit on top of that library with its opinionated set of policies.\r\n\r\nThat would be a very big change, though.\r\n\r\nI might take a stab at extracting the library, but just for the table schema migration piece, not all the other features that `sqlite-utils` supports. I wouldn't expect `sqlite-utils` to depend on it.\r\n\r\nPart of my motivation is that I want to provide some other abilities, too, like support for CHECK constraints. I see that the issue in this repo (https://github.com/simonw/sqlite-utils/issues/358) proposes a bunch of short-hand constraints, which I wouldn't want to accidentally expose to people -- I want a layer that is a 1:1 mapping to SQLite.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1572766460, "label": "Transformation type `--type DATETIME`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420992261", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/524", "id": 1420992261, "node_id": "IC_kwDOCGYnMM5Usp8F", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-02-07T15:45:58Z", "updated_at": "2023-02-07T15:45:58Z", "author_association": "NONE", "body": "I'd support that, but I'm not the author of this library.\r\n\r\nOne challenge is that would be a breaking change. Do you see a way to enable it without affecting existing users or bumping the major version number?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1572766460, "label": "Transformation type `--type DATETIME`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420809773", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/524", "id": 1420809773, "node_id": "IC_kwDOCGYnMM5Ur9Yt", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-02-07T13:53:01Z", "updated_at": "2023-02-07T13:53:01Z", "author_association": "NONE", "body": "Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178\r\n\r\nI suspect you could overwrite the datetime entry to achieve what you want", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1572766460, "label": "Transformation type `--type DATETIME`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419734229", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/524", "id": 1419734229, "node_id": "IC_kwDOCGYnMM5Un2zV", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-02-06T20:53:28Z", "updated_at": "2023-02-06T21:16:29Z", "author_association": "NONE", "body": "I think it's not currently possible: sqlite-utils requires that it be one of `integer`, `text`, `float`, `blob` ([see code](https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/cli.py#L2266))\r\n\r\nIMO, this is a bit of friction and it would be nice if it was more permissive. SQLite permits developers to use any data type when creating a table. For example, this is a perfectly cromulent sqlite session that creates a table with columns of type `baz` and `bar`:\r\n\r\n```\r\nsqlite> create table foo(column1 baz, column2 bar);\r\nsqlite> .schema foo\r\nCREATE TABLE foo(column1 baz, column2 bar);\r\nsqlite> select * from pragma_table_info('foo');\r\ncid         name        type        notnull     dflt_value  pk        \r\n----------  ----------  ----------  ----------  ----------  ----------\r\n0           column1     baz         0                       0         \r\n1           column2     bar         0                       0   \r\n```\r\n\r\nThe idea is that the application developer will know what meaning to ascribe to those types. For example, I'm working on a plugin to Datasette. Dates are tricky to handle. If you have some existing rows, you can look at the values in them to know how a user is serializing the dates -- as an ISO 8601 string? An RFC 3339 string? With millisecond precision? With timezone offset? But if you don't yet have any rows, you have to guess. If the column is of type `TEXT`, you don't even know that it's meant to hold a date! In this case, my plugin will look to see if the column is of type `DATE` or `DATETIME`, and assume a certain representation when writing.\r\n\r\nPerhaps there is an argument that sqlite-utils is trying to conform to SQLite's strict mode, and that is why it limits the choices. In strict mode, SQLite requires that the data type be one of `INT`, `INTEGER`, `REAL`, `TEXT`, `BLOB`, `ANY`. But that can't be the case -- sqlite-utils supports `FLOAT`, which is not one of the valid types in strict mode, and it rejects `INT`, `REAL` and `ANY`, which _are_ valid.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1572766460, "label": "Transformation type `--type DATETIME`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419740776", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/524", "id": 1419740776, "node_id": "IC_kwDOCGYnMM5Un4Zo", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-02-06T20:59:01Z", "updated_at": "2023-02-06T20:59:01Z", "author_association": "NONE", "body": "That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1572766460, "label": "Transformation type `--type DATETIME`"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1696#issuecomment-1407767434", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1696", "id": 1407767434, "node_id": "IC_kwDOBm6k_c5T6NOK", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-01-29T20:56:20Z", "updated_at": "2023-01-29T20:56:20Z", "author_association": "CONTRIBUTOR", "body": "I did some horrible things in https://github.com/cldellow/datasette-ui-extras/issues/2 to enable this in my plugin -- example here: https://dux-demo.fly.dev/cooking/posts?_facet=owner_user_id&owner_user_id=67\r\n\r\nThe implementation relies on two things:\r\n\r\n- a `filters_from_request` hook that adds a good human description (unfortunately, without the benefit of the CSS styling you mention)\r\n- doing something evil to hijack the `exact` and `not` operators in the `Filters` class. We can't leave them as is, or we'll get 2 human descriptions -- the built-in Datasette one and the one from my plugin. We can't remove them, or the filters UI will stop supporting the `=` and `!=` operators\r\n\r\nThis got me thinking: it'd be neat if the list of operators that the filters UI supported wasn't a closed set.\r\n\r\nA motivating example: adding a geospatial `NEAR` operator. Ideally it'd take two arguments - a target point and a radius, so you could express a filter like `find me all rows whose lat/lng are within 10km of 43.4516\u00b0 N, 80.4925\u00b0 W`. (Optionally, the UI could be enhanced if the geonames database was loaded and queried, so a user could say `find me all rows whose lat/lng are within 10km of Kitchener, ON`, and the city gets translated to a lat/lng for them)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1186696202, "label": "Show foreign key label when filtering"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/2008#issuecomment-1407716963", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2008", "id": 1407716963, "node_id": "IC_kwDOBm6k_c5T6A5j", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-01-29T17:04:03Z", "updated_at": "2023-01-29T17:04:03Z", "author_association": "CONTRIBUTOR", "body": "Performance tests - I think most places don't have them as a formal gate enforced by CI. TypeScript and scalac seem to have tests that run to capture timings. The timings are included by a bot as a comment or build check, and also stored in a database so you can graph changes over time to spot regressions. Probably overkill for Datasette!\r\n\r\nWindow functions - oh, good point. Looks like Ubuntu shipped JSON1 support as far back as sqlite 3.11. I'll let this PR linger until there's a way to run against different SQLite versions. For now, I'm shipping this with `datasette-ui-extras`, since I think it's OK for a plugin to enforce a higher minimum requirement.\r\n\r\nTests - there actually did end up being test changes to capture the undercount bug of the current implementation, so the current implementation would fail against the new tests.\r\n\r\nPerhaps a non-window function version could be written that uses `random()` instead of `row_number() over ()` in order to get a unique key. It's technically not unique, but in practice, I imagine it'll work well.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1560982210, "label": "array facet: don't materialize unnecessary columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/2008#issuecomment-1407561308", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2008", "id": 1407561308, "node_id": "IC_kwDOBm6k_c5T5a5c", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-01-29T04:50:50Z", "updated_at": "2023-01-29T04:50:50Z", "author_association": "CONTRIBUTOR", "body": "I pushed a revised version which ends up being faster -- the example which currently takes 4 seconds now runs in 500ms.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1560982210, "label": "array facet: don't materialize unnecessary columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/2008#issuecomment-1407558284", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2008", "id": 1407558284, "node_id": "IC_kwDOBm6k_c5T5aKM", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-01-29T04:23:58Z", "updated_at": "2023-01-29T04:24:27Z", "author_association": "CONTRIBUTOR", "body": "Ack, this PR is broken. I see now that the `inner.*` is necessary for ensuring the correct count in the face of rows having duplicate values in views.\r\n\r\nThat fixes the overcounting, but I think can undercount when the rows have the same data, eg a view like:\r\n\r\n```sql\r\nSELECT '[\"bar\"]' tags UNION ALL SELECT '[\"bar\"]'\r\n```\r\n\r\nwill produce a count of `{\"bar\": 1 }`, when it should be `{\"bar\": 2}`. In fact, this could apply in tables without primary keys, too.\r\n\r\nIf `inner` came from a base table that had a primary key or a rowid, we could use those column(s) to solve that case.\r\n\r\nI guess a general solution would be to compute a window function so we have a distinct ID for each row. Will fiddle to see if I can get that working.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1560982210, "label": "array facet: don't materialize unnecessary columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1973#issuecomment-1407523547", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1973", "id": 1407523547, "node_id": "IC_kwDOBm6k_c5T5Rrb", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-01-29T00:40:31Z", "updated_at": "2023-01-29T00:40:31Z", "author_association": "CONTRIBUTOR", "body": "A +1 for switching to `CustomRow`: I think you currently only get a `CustomRow` if the result set had a column that was an fkey ([this code](https://github.com/simonw/datasette/blob/3c352b7132ef09b829abb69a0da0ad00be5edef9/datasette/views/table.py#L667-L682))\r\n\r\nOtherwise you get vanilla `sqlite3.Row`s, which will fail if you try to access `.columns` or lookup the cell by name, which surprised me recently", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1515815014, "label": "render_cell plugin hook's row object is not a sqlite.Row"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/2008#issuecomment-1407470429", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2008", "id": 1407470429, "node_id": "IC_kwDOBm6k_c5T5Etd", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-01-28T19:34:29Z", "updated_at": "2023-01-28T19:34:29Z", "author_association": "CONTRIBUTOR", "body": "I don't know how/if you do automated tests for performance, so I haven't changed any of the tests.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1560982210, "label": "array facet: don't materialize unnecessary columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2001#issuecomment-1403084856", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2001", "id": 1403084856, "node_id": "IC_kwDOBm6k_c5ToWA4", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-01-25T04:31:02Z", "updated_at": "2023-01-25T04:31:02Z", "author_association": "CONTRIBUTOR", "body": "Aha, it's user error on my part.\r\n\r\nAdding\r\n\r\n```\r\nsqlite3_db_config.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_int, ctypes.c_int]\r\n```\r\n\r\nmakes it work reliably both on the CLI and from datasette, and now I can reproduce the errors you mentioned in the issue description.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1553615704, "label": "Datasette is not compatible with SQLite's strict quoting compilation option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2001#issuecomment-1403078134", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2001", "id": 1403078134, "node_id": "IC_kwDOBm6k_c5ToUX2", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-01-25T04:20:43Z", "updated_at": "2023-01-25T04:22:28Z", "author_association": "CONTRIBUTOR", "body": "I'm on Ubuntu, unfortunately. :( Would it still be relevant?\r\n\r\nI think I've narrowed things down a bit more.\r\n\r\nEven `sqlite3_free(sqlite3_malloc(128))` segfaults -- this suggests to me that it's something about the sqlite3 library that was loaded, vs, say, getting the wrong db handle when I go spelunking in the Connection object.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1553615704, "label": "Datasette is not compatible with SQLite's strict quoting compilation option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2001#issuecomment-1403053144", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2001", "id": 1403053144, "node_id": "IC_kwDOBm6k_c5ToORY", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-01-25T03:34:53Z", "updated_at": "2023-01-25T03:34:53Z", "author_association": "CONTRIBUTOR", "body": "Your comment introduced me to this issue in sqlite and to the `ctypes` module - thanks!\r\n\r\n> I also hope that the datasette developers will enable this mode in a test environment [...]\r\n> perhaps we could figure out how to invoke it using `ctypes`\r\n\r\nI'm not a Datasette developer, but I _am_ curious to learn more about getting unholy access to the sqlite C APIs inside of Datasette. (Such access could also help #1293, and if done without grovelling inside of pysqlite's Connection object for the db handle, could even be relatively safe.)\r\n\r\nI experimented a bit. I came up with https://gist.github.com/cldellow/85bba507c314b127f85563869cd94820\r\n\r\nIf you run `python3 enable-strict-quoting-sqlite3.py`, it seems to set those flags correctly -- `SELECT \"foo\"` fails where it would normally succeed.\r\n\r\nBut if you put it in a `plugins/` dir and run `datasette --plugins-dir plugins/`, it segfaults when it tries to call `sqlite3_db_config` on the connections created by Datasette.\r\n\r\nI am... confused. I'm _pretty_ sure I'm using the same python and the same libsqlite3 in both scenarios, so I would expect it to work.\r\n\r\n@gwk do you know anything that might help me debug the segfault? I gather that my approach of going grovelling inside of a `PyObject` is particularly dangerous, but I was thinking (a) it's necessary in order to test Datasette's use of the sqlite3 library and (b) even if it's not portable, it'd be good enough for running the tests on a single machine.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1553615704, "label": "Datasette is not compatible with SQLite's strict quoting compilation option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2000#issuecomment-1399847946", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2000", "id": 1399847946, "node_id": "IC_kwDOBm6k_c5Tb_wK", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-01-23T06:08:00Z", "updated_at": "2023-01-23T06:08:00Z", "author_association": "CONTRIBUTOR", "body": "Actually, I discovered [your post](https://til.simonwillison.net/datasette/register-new-plugin-hooks) showing how a plugin can add a Datasette hook. That's wild! I've released `datasette-rewrite-sql` that adds this ability, albeit via monkey patching.\r\n\r\nI had hoped to be able to expose `request` to the hook (or, even better `actor`) when the SQL was being run as a result of a user's HTTP request.\r\n\r\nBut some spelunking in the code makes me suspect that would actually require co-operation from Datasette itself. I'd be happy to be wrong and pointed in the right direction, though!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1552368054, "label": "rewrite_sql hook"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1159#issuecomment-1399589414", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1159", "id": 1399589414, "node_id": "IC_kwDOBm6k_c5TbAom", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-01-22T19:48:41Z", "updated_at": "2023-01-22T19:48:41Z", "author_association": "CONTRIBUTOR", "body": "Hey @lovasoa, I hope you don't mind - I pulled this PR into [datasette-ui-extras](https://github.com/cldellow/datasette-ui-extras), a plugin I'm making that collects UI tweaks to Datasette.\r\n\r\nYou can apply it to your own Datasette instance by running `datasette install datasette-ui-extras`", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 774332247, "label": "Improve the display of facets information"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1973#issuecomment-1369044959", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1973", "id": 1369044959, "node_id": "IC_kwDOBm6k_c5Rmfff", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2023-01-02T15:41:40Z", "updated_at": "2023-01-02T15:41:40Z", "author_association": "CONTRIBUTOR", "body": "Thanks for the response!\r\n\r\nYes, it does seem like a pretty nice developer experience--both the automagical labelling of fkeys, and the ability to index the row by column name in addition to column index.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1515815014, "label": "render_cell plugin hook's row object is not a sqlite.Row"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/236#issuecomment-612216820", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/236", "id": 612216820, "node_id": "MDEyOklzc3VlQ29tbWVudDYxMjIxNjgyMA==", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2020-04-10T21:03:38Z", "updated_at": "2020-04-10T21:03:38Z", "author_association": "CONTRIBUTOR", "body": "I made a repo at https://github.com/code402/datasette-lambda to demonstrate the idea, and scratch my personal itch for this.\r\n\r\nThe demo relies on some central authority having already published a public, reusable Lambda layer with Datasette & its dependencies. I think that differs from the other publish plugins which seem to mainly publish Dockerfiles that the host will interpret to install deps from a requirements.txt file.\r\n\r\nI chose that approach because `uvloop` appears to be a dependency with native code that needs to be compiled for the target runtime environment. In this case, that's Amazon Linux 2. I'm not 100% clear on whether that's still required, because:\r\n\r\n- maybe `uvloop` is only needed for `uvicorn`, which the demo doesn't actually use since HTTP routing is handled by API Gateway\r\n- it seems like `uvloop` may be an optional, drop-in optimization for `asyncio` in any case (but I may be misreading this; I'm very much a Python noob)\r\n\r\nIf it's the case that `uvloop` is truly optional, then I think the publish plugin could do the packaging on the user's machine, regardless of what flavour of operating system they're on. That'd be a bit slower for the user, but would provide the most long-term flexibility in terms of supporting plugins.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 317001500, "label": "datasette publish lambda plugin"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/236#issuecomment-608716819", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/236", "id": 608716819, "node_id": "MDEyOklzc3VlQ29tbWVudDYwODcxNjgxOQ==", "user": {"value": 193185, "label": "cldellow"}, "created_at": "2020-04-03T22:19:00Z", "updated_at": "2020-04-03T22:19:00Z", "author_association": "CONTRIBUTOR", "body": "Hi Simon,\r\n\r\nI'm thinking of attempting this. Can you clarify some questions I have?\r\n\r\n1) I assume the goal is to have a CORS-friendly HTTPS endpoint that hosts the datasette service + user's db.\r\n\r\n2) If that's the goal, I think Lambda alone is insufficient. Lambda provides the compute fabric, but not the HTTP routing. You'd also need to add Application Load Balancer or API Gateway to provide an HTTP endpoint that routes to the lambda function.\r\n\r\nDo you have a preference between ALB or API GW? ALB has better economics at scale, but has a minimum monthly cost. API GW has worse per-request economics, but scales to zero when no requests are happening.\r\n\r\n3) Does Datasette have any native components, or is it all pure python? If it has native bits, they'll likely need to be recompiled to work on Amazon Linux 2.\r\n\r\n4) There are a few disparate services that need to be wired together to expose a Python service securely to the web. If I was doing this outside of the datasette publish system, I'd use an AWS CloudFormation template. Even within datasette, I think it still makes sense to use a CloudFormation template and just have the publish plugin invoke it (via the standard `aws` cli) with user-specified parameters. Does that sound reasonable to you?\r\n\r\nThanks for your help!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 317001500, "label": "datasette publish lambda plugin"}, "performed_via_github_app": null}