issue_comments
10,495 rows sorted by node_id
This data as json, CSV (advanced)
issue >30
- Redesign default .json format 53
- ?_extra= support (draft) 49
- Upgrade to CodeMirror 6, add SQL autocomplete 48
- JavaScript plugin hooks mechanism similar to pluggy 47
- Updated Dockerfile with SpatiaLite version 5.0 45
- Port Datasette to ASGI 42
- Authentication (and permissions) as a core concept 40
- invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 36
- await datasette.client.get(path) mechanism for executing internal requests 33
- Maintain an in-memory SQLite table of connected databases and their tables 32
- Ability to sort (and paginate) by column 31
- Server hang on parallel execution of queries to named in-memory databases 31
- Default API token authentication mechanism 30
- Port as many tests as possible to async def tests against ds_client 29
- link_or_copy_directory() error - Invalid cross-device link 28
- Export to CSV 27
- base_url configuration setting 27
- Documentation with recommendations on running Datasette in production without using Docker 27
- Support cross-database joins 26
- Ability for a canned query to write to the database 26
- table.transform() method for advanced alter table 26
- Add ?_extra= mechanism for requesting extra properties in JSON 25
- Proof of concept for Datasette on AWS Lambda with EFS 25
- WIP: Add Gmail takeout mbox import 25
- Make it easier to insert geometries, with documentation and maybe code 25
- DeprecationWarning: pkg_resources is deprecated as an API 25
- Redesign register_output_renderer callback 24
- API explorer tool 24
- De-tangling Metadata before Datasette 1.0 24
- "datasette insert" command and plugin hook 23
- …
id | html_url | issue_url | node_id ▼ | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
1249355888 | https://github.com/simonw/datasette/pull/1812#issuecomment-1249355888 | https://api.github.com/repos/simonw/datasette/issues/1812 | IC_kwDOBm6k_c5Kd6hw | codecov[bot] 22429695 | 2022-09-16T13:18:37Z | 2022-09-16T13:18:37Z | NONE | Codecov ReportBase: 91.70% // Head: 91.70% // No change to project coverage :thumbsup:
Additional details and impacted files```diff @@ Coverage Diff @@ ## main #1812 +/- ## ======================================= Coverage 91.70% 91.70% ======================================= Files 38 38 Lines 4735 4735 ======================================= Hits 4342 4342 Misses 393 393 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison):umbrella: View full report at Codecov. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Bump furo from 2022.6.21 to 2022.9.15 1375930971 | |
1249746777 | https://github.com/simonw/datasette/pull/1812#issuecomment-1249746777 | https://api.github.com/repos/simonw/datasette/issues/1812 | IC_kwDOBm6k_c5KfZ9Z | simonw 9599 | 2022-09-16T19:50:45Z | 2022-09-16T19:50:45Z | OWNER | Main difference I can see: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Bump furo from 2022.6.21 to 2022.9.15 1375930971 | |
1249745637 | https://github.com/simonw/datasette/pull/1812#issuecomment-1249745637 | https://api.github.com/repos/simonw/datasette/issues/1812 | IC_kwDOBm6k_c5KfZrl | simonw 9599 | 2022-09-16T19:49:12Z | 2022-09-16T19:49:12Z | OWNER | Preview looks good. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Bump furo from 2022.6.21 to 2022.9.15 1375930971 | |
1249985741 | https://github.com/simonw/datasette/issues/1809#issuecomment-1249985741 | https://api.github.com/repos/simonw/datasette/issues/1809 | IC_kwDOBm6k_c5KgUTN | simonw 9599 | 2022-09-17T03:04:51Z | 2022-09-17T03:04:51Z | OWNER | I'm going to throw an error in |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`prepare_jinja2_environment()` hook should take `datasette` argument 1373595927 | |
1249985971 | https://github.com/simonw/datasette/issues/1809#issuecomment-1249985971 | https://api.github.com/repos/simonw/datasette/issues/1809 | IC_kwDOBm6k_c5KgUWz | simonw 9599 | 2022-09-17T03:06:32Z | 2022-09-17T03:06:32Z | OWNER | This is likely going to cause some tests in plugins to break, but I'm OK with that - I'll fix them as I find them once this release is out. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`prepare_jinja2_environment()` hook should take `datasette` argument 1373595927 | |
1249986079 | https://github.com/simonw/datasette/issues/1809#issuecomment-1249986079 | https://api.github.com/repos/simonw/datasette/issues/1809 | IC_kwDOBm6k_c5KgUYf | simonw 9599 | 2022-09-17T03:07:24Z | 2022-09-17T03:07:24Z | OWNER | Datasette's own tests started to break because calls to the
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`prepare_jinja2_environment()` hook should take `datasette` argument 1373595927 | |
1249987643 | https://github.com/simonw/datasette/issues/1809#issuecomment-1249987643 | https://api.github.com/repos/simonw/datasette/issues/1809 | IC_kwDOBm6k_c5KgUw7 | simonw 9599 | 2022-09-17T03:19:24Z | 2022-09-17T03:19:24Z | OWNER | In looking at the documentation on writing tests, there are a lot of examples like this:
Since it's safe to call that function multiple times, I'm going to have |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`prepare_jinja2_environment()` hook should take `datasette` argument 1373595927 | |
1249990033 | https://github.com/simonw/datasette/issues/1809#issuecomment-1249990033 | https://api.github.com/repos/simonw/datasette/issues/1809 | IC_kwDOBm6k_c5KgVWR | simonw 9599 | 2022-09-17T03:39:05Z | 2022-09-17T03:39:05Z | OWNER | New docs section on the need to call |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`prepare_jinja2_environment()` hook should take `datasette` argument 1373595927 | |
1250901367 | https://github.com/simonw/datasette/issues/1813#issuecomment-1250901367 | https://api.github.com/repos/simonw/datasette/issues/1813 | IC_kwDOBm6k_c5Kjz13 | adipasquale 883348 | 2022-09-19T11:34:45Z | 2022-09-19T11:34:45Z | CONTRIBUTOR | oh and by writing this I just realized the difference: the URL on fly.io is with a custom SQL command whereas the local one is without. It seems that there is no pagination when using custom SQL commands which makes sense Sorry for this useless issue, maybe this can be useful for someone else / me in the future. Thanks again for this wonderful project ! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
missing next and next_url in JSON responses from an instance deployed on Fly 1377811868 | |
1251724180 | https://github.com/simonw/datasette/issues/1816#issuecomment-1251724180 | https://api.github.com/repos/simonw/datasette/issues/1816 | IC_kwDOBm6k_c5Km8uU | simonw 9599 | 2022-09-20T01:13:05Z | 2022-09-20T01:13:05Z | OWNER | Oops, that has a bug:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Validate settings.json on startup in configuration directory mode 1378640768 | |
1251677220 | https://github.com/simonw/datasette/issues/1814#issuecomment-1251677220 | https://api.github.com/repos/simonw/datasette/issues/1814 | IC_kwDOBm6k_c5KmxQk | simonw 9599 | 2022-09-19T23:34:30Z | 2022-09-19T23:34:30Z | OWNER | The The
|
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0 } |
Static files not served 1378495690 | |
1251677554 | https://github.com/simonw/datasette/issues/1814#issuecomment-1251677554 | https://api.github.com/repos/simonw/datasette/issues/1814 | IC_kwDOBm6k_c5KmxVy | simonw 9599 | 2022-09-19T23:35:06Z | 2022-09-19T23:35:06Z | OWNER | It might have been useful for Datasette to show an error when started against a |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Static files not served 1378495690 | |
1251682970 | https://github.com/simonw/datasette/issues/1816#issuecomment-1251682970 | https://api.github.com/repos/simonw/datasette/issues/1816 | IC_kwDOBm6k_c5Kmyqa | simonw 9599 | 2022-09-19T23:44:54Z | 2022-09-19T23:44:54Z | OWNER | I was going to add type validation too, but that's actually a bit tricky because the logic for that currently lives in Click option parsing here: https://github.com/simonw/datasette/blob/ddc999ad1296e8c69cffede3e367dda059b8adad/datasette/cli.py#L71-L88 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Validate settings.json on startup in configuration directory mode 1378640768 | |
1254064260 | https://github.com/simonw/datasette/issues/526#issuecomment-1254064260 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5Kv4CE | fgregg 536941 | 2022-09-21T18:17:04Z | 2022-09-21T18:18:01Z | CONTRIBUTOR | hi @simonw, this is becoming more of a bother for my labor data warehouse. Is there any research or a spike i could do that would help you investigate this issue? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1272149176 | https://github.com/simonw/datasette/issues/1646#issuecomment-1272149176 | https://api.github.com/repos/simonw/datasette/issues/1646 | IC_kwDOBm6k_c5L03S4 | simonw 9599 | 2022-10-07T23:06:17Z | 2022-10-07T23:06:17Z | OWNER | Updated documentation: https://docs.datasette.io/en/latest/settings.html#configuration-directory-mode |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Configuration directory mode does not pick up other file extensions than .db 1157182254 | |
1272228740 | https://github.com/simonw/datasette/issues/1362#issuecomment-1272228740 | https://api.github.com/repos/simonw/datasette/issues/1362 | IC_kwDOBm6k_c5L1KuE | simonw 9599 | 2022-10-08T05:03:56Z | 2022-10-08T05:03:56Z | OWNER | Useful example: how Play framework does this https://www.playframework.com/documentation/2.8.1/CspFilter |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Consider using CSP to protect against future XSS 912864936 | |
1272344884 | https://github.com/simonw/datasette/issues/1836#issuecomment-1272344884 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5L1nE0 | simonw 9599 | 2022-10-08T15:41:28Z | 2022-10-08T15:41:28Z | OWNER | Lets switch to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1272357976 | https://github.com/simonw/datasette/issues/1836#issuecomment-1272357976 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5L1qRY | fgregg 536941 | 2022-10-08T16:56:51Z | 2022-10-08T16:56:51Z | CONTRIBUTOR | when you are running from docker, you always will want to run as |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1272369443 | https://github.com/simonw/datasette/issues/1362#issuecomment-1272369443 | https://api.github.com/repos/simonw/datasette/issues/1362 | IC_kwDOBm6k_c5L1tEj | simonw 9599 | 2022-10-08T18:03:03Z | 2022-10-08T18:03:03Z | OWNER | Asked for tips on Twitter: https://twitter.com/simonw/status/1578561096520114176 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Consider using CSP to protect against future XSS 912864936 | |
1272369603 | https://github.com/simonw/datasette/issues/1362#issuecomment-1272369603 | https://api.github.com/repos/simonw/datasette/issues/1362 | IC_kwDOBm6k_c5L1tHD | simonw 9599 | 2022-10-08T18:03:56Z | 2022-10-08T18:03:56Z | OWNER | This document is useful: https://csp.withgoogle.com/docs/strict-csp.html |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Consider using CSP to protect against future XSS 912864936 | |
1272369712 | https://github.com/simonw/datasette/issues/1362#issuecomment-1272369712 | https://api.github.com/repos/simonw/datasette/issues/1362 | IC_kwDOBm6k_c5L1tIw | simonw 9599 | 2022-10-08T18:04:31Z | 2022-10-08T18:05:05Z | OWNER | Also this series: https://scotthelme.co.uk/tag/csp/ - via https://twitter.com/adamchainz/status/1578762884481368065 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Consider using CSP to protect against future XSS 912864936 | |
1272376377 | https://github.com/simonw/datasette/issues/1362#issuecomment-1272376377 | https://api.github.com/repos/simonw/datasette/issues/1362 | IC_kwDOBm6k_c5L1uw5 | simonw 9599 | 2022-10-08T18:42:09Z | 2022-10-08T18:42:09Z | OWNER | And a useful cheat sheet https://scotthelme.co.uk/csp-cheat-sheet/ |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Consider using CSP to protect against future XSS 912864936 | |
1258337011 | https://github.com/simonw/datasette/issues/526#issuecomment-1258337011 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LALLz | fgregg 536941 | 2022-09-26T16:49:48Z | 2022-09-26T16:49:48Z | CONTRIBUTOR | i think the smallest change that gets close to what i want is to change the behavior so that there are some infelicities for that approach, but i'll make a PR to make it easier to discuss. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1258803261 | https://github.com/simonw/datasette/pull/1820#issuecomment-1258803261 | https://api.github.com/repos/simonw/datasette/issues/1820 | IC_kwDOBm6k_c5LB9A9 | fgregg 536941 | 2022-09-27T00:03:09Z | 2022-09-27T00:03:09Z | CONTRIBUTOR | the pattern in this PR i think it would be better to have each of these different query renderers have more direct control for how many rows to fetch, instead of relying on the internals of the generally, users will not want to paginate through tens of thousands of results, but often will want to download a full query as json or as csv. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
[SPIKE] Don't truncate query CSVs 1386456717 | |
1258601033 | https://github.com/simonw/datasette/pull/1820#issuecomment-1258601033 | https://api.github.com/repos/simonw/datasette/issues/1820 | IC_kwDOBm6k_c5LBLpJ | codecov[bot] 22429695 | 2022-09-26T20:32:47Z | 2022-10-07T03:58:13Z | NONE | Codecov ReportBase: 92.50% // Head: 92.51% // Increases project coverage by
Additional details and impacted files```diff @@ Coverage Diff @@ ## main #1820 +/- ## ========================================== + Coverage 92.50% 92.51% +0.01% ========================================== Files 35 35 Lines 4400 4406 +6 ========================================== + Hits 4070 4076 +6 Misses 330 330 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.11% <ø> (ø)` | | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `94.80% <100.00%> (+0.05%)` | :arrow_up: | | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.29% <100.00%> (+0.06%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison):umbrella: View full report at Codecov. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
[SPIKE] Don't truncate query CSVs 1386456717 | |
1258692555 | https://github.com/simonw/datasette/issues/1821#issuecomment-1258692555 | https://api.github.com/repos/simonw/datasette/issues/1821 | IC_kwDOBm6k_c5LBh_L | simonw 9599 | 2022-09-26T22:06:39Z | 2022-09-26T22:06:39Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Release Datasette 0.63a0 1386734383 | ||
1258735283 | https://github.com/simonw/datasette/issues/1818#issuecomment-1258735283 | https://api.github.com/repos/simonw/datasette/issues/1818 | IC_kwDOBm6k_c5LBsaz | simonw 9599 | 2022-09-26T22:47:19Z | 2022-09-26T22:47:19Z | OWNER | That's a really interesting idea: for a lot of databases (those made out of straight imports from CSV) |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Setting to turn off table row counts entirely 1384549993 | |
1258735747 | https://github.com/simonw/datasette/issues/1818#issuecomment-1258735747 | https://api.github.com/repos/simonw/datasette/issues/1818 | IC_kwDOBm6k_c5LBsiD | simonw 9599 | 2022-09-26T22:47:59Z | 2022-09-26T22:47:59Z | OWNER | Another option here is to tie into a feature I built in |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Setting to turn off table row counts entirely 1384549993 | |
1258738435 | https://github.com/simonw/datasette/issues/1819#issuecomment-1258738435 | https://api.github.com/repos/simonw/datasette/issues/1819 | IC_kwDOBm6k_c5LBtMD | simonw 9599 | 2022-09-26T22:52:19Z | 2022-09-26T22:52:19Z | OWNER | This is a good idea. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Preserve query on timeout 1385026210 | |
1258738740 | https://github.com/simonw/datasette/issues/1818#issuecomment-1258738740 | https://api.github.com/repos/simonw/datasette/issues/1818 | IC_kwDOBm6k_c5LBtQ0 | nelsonjchen 5363 | 2022-09-26T22:52:45Z | 2022-09-26T22:55:57Z | NONE | thoughts on order of precedence to use:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Setting to turn off table row counts entirely 1384549993 | |
1258746600 | https://github.com/simonw/datasette/issues/1819#issuecomment-1258746600 | https://api.github.com/repos/simonw/datasette/issues/1819 | IC_kwDOBm6k_c5LBvLo | simonw 9599 | 2022-09-26T23:05:40Z | 2022-09-26T23:05:40Z | OWNER | Implementing it like this, so at least you can copy and paste the SQL query back out again: I'm not doing a full textarea because this error can be raised in multiple places, including on the table page itself. It's not just an error associated with the manual query page. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Preserve query on timeout 1385026210 | |
1258757544 | https://github.com/simonw/datasette/issues/1822#issuecomment-1258757544 | https://api.github.com/repos/simonw/datasette/issues/1822 | IC_kwDOBm6k_c5LBx2o | simonw 9599 | 2022-09-26T23:21:23Z | 2022-09-26T23:21:23Z | OWNER | Everything on https://docs.datasette.io/en/stable/internals.html that uses keyword arguments should do this I think. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Switch to keyword-only arguments for a bunch of internal methods 1386854246 | |
1258754105 | https://github.com/simonw/datasette/issues/1819#issuecomment-1258754105 | https://api.github.com/repos/simonw/datasette/issues/1819 | IC_kwDOBm6k_c5LBxA5 | simonw 9599 | 2022-09-26T23:16:15Z | 2022-09-26T23:16:15Z | OWNER | Demo: https://latest.datasette.io/_memory?sql=with+recursive+counter(x)+as+(%0D%0A++select+0%0D%0A++++union%0D%0A++select+x+%2B+1+from+counter%0D%0A)%2C%0D%0Ablah+as+(select++from+counter+limit+5000000)%0D%0Aselect+count()+from+blah |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Preserve query on timeout 1385026210 | |
1258756231 | https://github.com/simonw/datasette/issues/1817#issuecomment-1258756231 | https://api.github.com/repos/simonw/datasette/issues/1817 | IC_kwDOBm6k_c5LBxiH | simonw 9599 | 2022-09-26T23:19:34Z | 2022-09-26T23:19:34Z | OWNER | This is a good idea - it's something I should do before Datasette 1.0. I was a tiny bit worried about compatibility (Datasette is 3.7+) but it looks like they have been in Python since 3.0! |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Expose `sql` and `params` arguments to various plugin hooks 1384273985 | |
1258760299 | https://github.com/simonw/datasette/issues/1822#issuecomment-1258760299 | https://api.github.com/repos/simonw/datasette/issues/1822 | IC_kwDOBm6k_c5LByhr | simonw 9599 | 2022-09-26T23:25:12Z | 2022-09-26T23:25:55Z | OWNER | A start: ```diff diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py index 8a2fa060..41ade961 100644 --- a/datasette/utils/asgi.py +++ b/datasette/utils/asgi.py @@ -118,7 +118,7 @@ class Request: return dict(parse_qsl(body.decode("utf-8"), keep_blank_values=True))
-async def asgi_send_json(send, info, status=200, headers=None): +async def asgi_send_json(send, info, *, status=200, headers=None): headers = headers or {} await asgi_send( send, @@ -215,7 +215,7 @@ async def asgi_send_json(send, info, status=200, headers=None): ) -async def asgi_send_html(send, html, status=200, headers=None): +async def asgi_send_html(send, html, *, status=200, headers=None): headers = headers or {} await asgi_send( send, @@ -226,7 +226,7 @@ async def asgi_send_html(send, html, status=200, headers=None): ) -async def asgi_send_redirect(send, location, status=302): +async def asgi_send_redirect(send, location, *, status=302): await asgi_send( send, "", @@ -236,12 +236,12 @@ async def asgi_send_redirect(send, location, status=302): ) -async def asgi_send(send, content, status, headers=None, content_type="text/plain"): +async def asgi_send(send, content, status, *, headers=None, content_type="text/plain"): await asgi_start(send, status, headers, content_type) await send({"type": "http.response.body", "body": content.encode("utf-8")}) -async def asgi_start(send, status, headers=None, content_type="text/plain"): +async def asgi_start(send, status, *, headers=None, content_type="text/plain"): headers = headers or {} # Remove any existing content-type header headers = {k: v for k, v in headers.items() if k.lower() != "content-type"} @@ -259,7 +259,7 @@ async def asgi_start(send, status, headers=None, content_type="text/plain"): async def asgi_send_file( - send, filepath, filename=None, content_type=None, chunk_size=4096, headers=None + send, filepath, filename=None, *, content_type=None, chunk_size=4096, headers=None ): headers = headers or {} if filename: @@ -284,7 +284,7 @@ async def asgi_send_file( ) -def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None): +def asgi_static(root_path, *, chunk_size=4096, headers=None, content_type=None): root_path = Path(root_path)
@@ -313,7 +313,7 @@ def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None): class Response: - def init(self, body=None, status=200, headers=None, content_type="text/plain"): + def init(self, body=None, , status=200, headers=None, content_type="text/plain"): self.body = body self.status = status self.headers = headers or {} @@ -346,6 +346,7 @@ class Response: self, key, value="", + , max_age=None, expires=None, path="/", @@ -374,7 +375,7 @@ class Response: self._set_cookie_headers.append(cookie.output(header="").strip())
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Switch to keyword-only arguments for a bunch of internal methods 1386854246 | |
1258818028 | https://github.com/simonw/datasette/issues/1817#issuecomment-1258818028 | https://api.github.com/repos/simonw/datasette/issues/1817 | IC_kwDOBm6k_c5LCAns | simonw 9599 | 2022-09-27T00:27:53Z | 2022-09-27T00:27:53Z | OWNER | Made a start on this: ```diff diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py index 34e19664..fe0971e5 100644 --- a/datasette/hookspecs.py +++ b/datasette/hookspecs.py @@ -31,25 +31,29 @@ def prepare_jinja2_environment(env, datasette): @hookspec -def extra_css_urls(template, database, table, columns, view_name, request, datasette): +def extra_css_urls( + template, database, table, columns, sql, params, view_name, request, datasette +): """Extra CSS URLs added by this plugin""" @hookspec -def extra_js_urls(template, database, table, columns, view_name, request, datasette): +def extra_js_urls( + template, database, table, columns, sql, params, view_name, request, datasette +): """Extra JavaScript URLs added by this plugin""" @hookspec def extra_body_script( - template, database, table, columns, view_name, request, datasette + template, database, table, columns, sql, params, view_name, request, datasette ): """Extra JavaScript code to be included in <script> at bottom of body""" @hookspec
def extra_template_vars(
- template, database, table, columns, view_name, request, datasette
+ template, database, table, columns, sql, params, view_name, request, datasette
):
"""Extra template variables to be made available to the template - can return dict or callable or awaitable"""
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Expose `sql` and `params` arguments to various plugin hooks 1384273985 | |
1258827688 | https://github.com/simonw/datasette/issues/1822#issuecomment-1258827688 | https://api.github.com/repos/simonw/datasette/issues/1822 | IC_kwDOBm6k_c5LCC-o | simonw 9599 | 2022-09-27T00:44:04Z | 2022-09-27T00:44:04Z | OWNER | I'll do this in a PR. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Switch to keyword-only arguments for a bunch of internal methods 1386854246 | |
1258828509 | https://github.com/simonw/datasette/pull/1823#issuecomment-1258828509 | https://api.github.com/repos/simonw/datasette/issues/1823 | IC_kwDOBm6k_c5LCDLd | simonw 9599 | 2022-09-27T00:45:26Z | 2022-09-27T00:45:26Z | OWNER | I should update the documentation to reflect this change. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Keyword-only arguments for a bunch of internal methods 1386917344 | |
1258828705 | https://github.com/simonw/datasette/pull/1823#issuecomment-1258828705 | https://api.github.com/repos/simonw/datasette/issues/1823 | IC_kwDOBm6k_c5LCDOh | simonw 9599 | 2022-09-27T00:45:46Z | 2022-09-27T00:45:46Z | OWNER | Also need to do a bit more of an audit to see if there is anywhere else that this style should be applied. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Keyword-only arguments for a bunch of internal methods 1386917344 | |
1258833358 | https://github.com/simonw/datasette/pull/1823#issuecomment-1258833358 | https://api.github.com/repos/simonw/datasette/issues/1823 | IC_kwDOBm6k_c5LCEXO | codecov[bot] 22429695 | 2022-09-27T00:54:15Z | 2022-10-05T04:37:54Z | NONE | Codecov ReportBase: 91.58% // Head: 92.50% // Increases project coverage by
Additional details and impacted files```diff @@ Coverage Diff @@ ## main #1823 +/- ## ========================================== + Coverage 91.58% 92.50% +0.91% ========================================== Files 36 35 -1 Lines 4444 4400 -44 ========================================== Hits 4070 4070 + Misses 374 330 -44 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1823?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1823/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `91.06% <88.23%> (ø)` | | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1823/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.11% <100.00%> (ø)` | | | [datasette/utils/shutil\_backport.py](https://codecov.io/gh/simonw/datasette/pull/1823/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3NodXRpbF9iYWNrcG9ydC5weQ==) | | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison):umbrella: View full report at Codecov. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Keyword-only arguments for a bunch of internal methods 1386917344 | |
1258846992 | https://github.com/simonw/datasette/issues/526#issuecomment-1258846992 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LCHsQ | simonw 9599 | 2022-09-27T01:21:41Z | 2022-09-27T01:21:41Z | OWNER | My main concern here is that public Datasette instances could easily have all of their available database connections consumed by long-running queries - either accidentally or deliberately. I do totally understand the need for this feature though. I think it can absolutely make sense provided it's protected by authentication and permissions. Maybe even limit the number of concurrent downloads at once such that there's always at least one database connection free for other requests. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1258849766 | https://github.com/simonw/datasette/issues/526#issuecomment-1258849766 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LCIXm | fgregg 536941 | 2022-09-27T01:27:03Z | 2022-09-27T01:27:03Z | CONTRIBUTOR | i agree with that concern! but if i'm understanding the code correctly, |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1258864140 | https://github.com/simonw/datasette/issues/526#issuecomment-1258864140 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LCL4M | simonw 9599 | 2022-09-27T01:55:32Z | 2022-09-27T01:55:32Z | OWNER | That recursive query is a great example of the kind of thing having a maximum row limit protects against. Imagine if Datasette CSVs did allow unlimited retrievals. Someone could hit the CSV endpoint for that recursive query and tie up Datasette's SQL connection effectively forever. Even if this feature becomes a permission-guarded thing we still need to take that case into account. At the very least it would be good if the query could be cancelled if the client disconnects - so if someone accidentally starts an infinite query they can cancel the request and free up the server resources. It might be a good idea to implement a page that shows "currently running" queries and allows users with the right permission to terminate them from that page. Another option: a "limit of last resource" - either a very high row limit (10,000,000 perhaps) or even a time limit, saying that all queries will be cancelled if they take longer than thirty minutes or similar. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1258860845 | https://github.com/simonw/datasette/issues/526#issuecomment-1258860845 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LCLEt | simonw 9599 | 2022-09-27T01:48:31Z | 2022-09-27T01:50:01Z | OWNER | The protection is supposed to be from this line:
SQLite and the ```pycon
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1258871525 | https://github.com/simonw/datasette/issues/526#issuecomment-1258871525 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LCNrl | fgregg 536941 | 2022-09-27T02:09:32Z | 2022-09-27T02:14:53Z | CONTRIBUTOR | thanks @simonw, i learned something i didn't know about sqlite's execution model!
why wouldn't the on my local version which has the code to turn off truncations for query csv, |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1258878311 | https://github.com/simonw/datasette/issues/526#issuecomment-1258878311 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LCPVn | fgregg 536941 | 2022-09-27T02:19:48Z | 2022-09-27T02:19:48Z | CONTRIBUTOR | this sql query doesn't trip up
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1258905781 | https://github.com/simonw/datasette/issues/526#issuecomment-1258905781 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LCWC1 | simonw 9599 | 2022-09-27T03:03:35Z | 2022-09-27T03:03:47Z | OWNER | Yes good point, the time limit does already protect against that. I've been contemplating a permissioned-users-only relaxation of that time limit too, and I got that idea mixed up with this one in my head. On that basis maybe this feature would be safe after all? Would need to do some testing, but it may be that the existing time limit provides enough protection here already. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1258906440 | https://github.com/simonw/datasette/issues/526#issuecomment-1258906440 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LCWNI | simonw 9599 | 2022-09-27T03:04:37Z | 2022-09-27T03:04:37Z | OWNER | It would be really neat if we could explore this idea in a plugin, but I don't think Datasette has plugin hooks in the right place for that at the moment. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1258910228 | https://github.com/simonw/datasette/issues/526#issuecomment-1258910228 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LCXIU | fgregg 536941 | 2022-09-27T03:11:07Z | 2022-09-27T03:11:07Z | CONTRIBUTOR | i think this feature would be safe, as its really only the time limit that can, and imo, should protect against long running queries, as it is pretty easy to make very expensive queries that don't return many rows. moving away from
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1259693536 | https://github.com/simonw/datasette/issues/526#issuecomment-1259693536 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LFWXg | simonw 9599 | 2022-09-27T15:42:55Z | 2022-09-27T15:42:55Z | OWNER | It's interesting to note WHY the time limit works against this so well. The time limit as-implemented looks like this: https://github.com/simonw/datasette/blob/5f9f567acbc58c9fcd88af440e68034510fb5d2b/datasette/utils/init.py#L181-L201 The key here is The handler function then checks to see if too much time has transpired and conditionally cancels the query. This also doubles up as a "maximum number of operations" guard, which is what's happening when you attempt to fetch an infinite number of rows from an infinite table. That limit code could even be extended to say "exit the query after either 5s or 50,000,000 operations". I don't think that's necessary though. To be honest I'm having trouble with the idea of dropping |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1259718517 | https://github.com/simonw/datasette/issues/526#issuecomment-1259718517 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LFcd1 | fgregg 536941 | 2022-09-27T16:02:51Z | 2022-09-27T16:04:46Z | CONTRIBUTOR | i think that if you are potentially yielding out hundreds of thousands or even millions of rows, you need to be quite careful about data flow to not run out of memory on the server, or on the client. you have a lot of places in your code that are protective of that right now, but so, given that, it makes sense to have removing that could dramatically lower the surface area for a memory-bomb attack. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1260355224 | https://github.com/simonw/datasette/issues/526#issuecomment-1260355224 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5LH36Y | simonw 9599 | 2022-09-28T04:01:25Z | 2022-09-28T04:01:25Z | OWNER | The ultimate protection against those memory bombs is to support more streaming output formats. Related issues:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1260357583 | https://github.com/simonw/datasette/issues/1826#issuecomment-1260357583 | https://api.github.com/repos/simonw/datasette/issues/1826 | IC_kwDOBm6k_c5LH4fP | simonw 9599 | 2022-09-28T04:05:16Z | 2022-09-28T04:05:16Z | OWNER | This is deliberate. The Datasette plugin system allows you to specify only a subset of the parameters for a hook - in this example, only the There's a note about this at the very top of that documentation page: https://docs.datasette.io/en/stable/plugin_hooks.html#plugin-hooks
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
render_cell documentation example doesn't match the method signature 1388631785 | |
1260357878 | https://github.com/simonw/datasette/issues/1826#issuecomment-1260357878 | https://api.github.com/repos/simonw/datasette/issues/1826 | IC_kwDOBm6k_c5LH4j2 | simonw 9599 | 2022-09-28T04:05:45Z | 2022-09-28T04:05:45Z | OWNER | Though now I notice that the copy right there needs to be updated to reflect the new |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
render_cell documentation example doesn't match the method signature 1388631785 | |
1260368122 | https://github.com/simonw/datasette/pull/1825#issuecomment-1260368122 | https://api.github.com/repos/simonw/datasette/issues/1825 | IC_kwDOBm6k_c5LH7D6 | codecov[bot] 22429695 | 2022-09-28T04:20:28Z | 2022-09-28T04:20:28Z | NONE | Codecov ReportBase: 91.58% // Head: 91.58% // No change to project coverage :thumbsup:
Additional details and impacted files```diff @@ Coverage Diff @@ ## main #1825 +/- ## ======================================= Coverage 91.58% 91.58% ======================================= Files 36 36 Lines 4444 4444 ======================================= Hits 4070 4070 Misses 374 374 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison):umbrella: View full report at Codecov. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add documentation for serving via OpenRC 1388227245 | |
1260368537 | https://github.com/simonw/datasette/pull/1825#issuecomment-1260368537 | https://api.github.com/repos/simonw/datasette/issues/1825 | IC_kwDOBm6k_c5LH7KZ | simonw 9599 | 2022-09-28T04:21:18Z | 2022-09-28T04:21:18Z | OWNER | This is great, thank you very much! https://datasette--1825.org.readthedocs.build/en/1825/deploying.html#running-datasette-using-openrc |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add documentation for serving via OpenRC 1388227245 | |
1260373403 | https://github.com/simonw/datasette/issues/1826#issuecomment-1260373403 | https://api.github.com/repos/simonw/datasette/issues/1826 | IC_kwDOBm6k_c5LH8Wb | pjamargh 66709385 | 2022-09-28T04:30:27Z | 2022-09-28T04:30:27Z | NONE | I'm glad the bug report served some purpose. Frankly I just needed the method signature, that is why the documentation you mention wasn't read. On Tue, Sep 27, 2022, 9:05 PM Simon Willison @.***> wrote:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
render_cell documentation example doesn't match the method signature 1388631785 | |
1260909128 | https://github.com/simonw/datasette/issues/1062#issuecomment-1260909128 | https://api.github.com/repos/simonw/datasette/issues/1062 | IC_kwDOBm6k_c5LJ_JI | fgregg 536941 | 2022-09-28T13:22:53Z | 2022-09-28T14:09:54Z | CONTRIBUTOR | if you went this route:
then i wonder if this was why you were thinking this feature would need a dedicated connection? reading more, there's no real limit i can find on the number of active cursors (or more precisely active prepared statements objects, because sqlite doesn't really have cursors). maybe something like this would be okay?
this seems quite weird that there's not more of limit of the number of active prepared statements, but i haven't been able to find one. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows 732674148 | |
1260829829 | https://github.com/simonw/datasette/issues/1062#issuecomment-1260829829 | https://api.github.com/repos/simonw/datasette/issues/1062 | IC_kwDOBm6k_c5LJryF | fgregg 536941 | 2022-09-28T12:27:19Z | 2022-09-28T12:27:19Z | CONTRIBUTOR | for teaching
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows 732674148 | |
1261194164 | https://github.com/simonw/datasette/issues/1624#issuecomment-1261194164 | https://api.github.com/repos/simonw/datasette/issues/1624 | IC_kwDOBm6k_c5LLEu0 | palfrey 38532 | 2022-09-28T16:54:22Z | 2022-09-28T16:54:22Z | NONE | https://github.com/simonw/datasette-cors seems to workaround this |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Index page `/` has no CORS headers 1122427321 | |
1261930179 | https://github.com/simonw/datasette/issues/370#issuecomment-1261930179 | https://api.github.com/repos/simonw/datasette/issues/370 | IC_kwDOBm6k_c5LN4bD | MichaelTiemannOSC 72577720 | 2022-09-29T08:17:46Z | 2022-09-29T08:17:46Z | CONTRIBUTOR | Just watched this video which demonstrates the integration of any webapp into JupyterLab: https://youtu.be/FH1dKKmvFtc Maybe this is the answer? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Integration with JupyterLab 377155320 | |
1263570186 | https://github.com/simonw/datasette/pull/1827#issuecomment-1263570186 | https://api.github.com/repos/simonw/datasette/issues/1827 | IC_kwDOBm6k_c5LUI0K | codecov[bot] 22429695 | 2022-09-30T13:22:15Z | 2022-09-30T13:22:15Z | NONE | Codecov ReportBase: 92.50% // Head: 92.50% // No change to project coverage :thumbsup:
Additional details and impacted files```diff @@ Coverage Diff @@ ## main #1827 +/- ## ======================================= Coverage 92.50% 92.50% ======================================= Files 35 35 Lines 4400 4400 ======================================= Hits 4070 4070 Misses 330 330 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison):umbrella: View full report at Codecov. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Bump furo from 2022.9.15 to 2022.9.29 1392426838 | |
1264738081 | https://github.com/simonw/datasette/issues/1828#issuecomment-1264738081 | https://api.github.com/repos/simonw/datasette/issues/1828 | IC_kwDOBm6k_c5LYl8h | simonw 9599 | 2022-10-02T21:34:37Z | 2022-10-02T21:34:37Z | OWNER | I'm running a build of that demo instance here (takes ~30m) https://github.com/dogsheep/github-to-sqlite/actions/runs/3170164705 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
word-wrap: anywhere resulting in weird display 1393903845 | |
1264736537 | https://github.com/simonw/datasette/issues/1805#issuecomment-1264736537 | https://api.github.com/repos/simonw/datasette/issues/1805 | IC_kwDOBm6k_c5LYlkZ | simonw 9599 | 2022-10-02T21:25:37Z | 2022-10-02T21:25:37Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
truncate_cells_html does not work for links? 1363552780 | |
1264737290 | https://github.com/simonw/datasette/issues/485#issuecomment-1264737290 | https://api.github.com/repos/simonw/datasette/issues/485 | IC_kwDOBm6k_c5LYlwK | simonw 9599 | 2022-10-02T21:29:59Z | 2022-10-02T21:29:59Z | OWNER | To clarify: the feature this issue is talking about relates to the way Datasette automatically displays foreign key relationships, for example on this page: https://github-to-sqlite.dogsheep.net/github/commits Each of those columns is a foreign key to another table. The link text that is displayed there comes from the "label column" that has either been configured or automatically detected for that other table. I wonder if this could be handled with a tiny machine learning model that's trained to help pick the best label column? Inputs to that model could include:
Output would be the most likely label column, or some indicator that no likely candidates had been found. My hunch is that this would be better solved using a few extra heuristics rather than by training a model, but it does feel like an interesting opportunity to experiment with a tiny ML model. Asked for tips about this on Twitter: https://twitter.com/simonw/status/1576680930680262658 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Improvements to table label detection 447469253 | |
1264753439 | https://github.com/simonw/datasette/issues/1828#issuecomment-1264753439 | https://api.github.com/repos/simonw/datasette/issues/1828 | IC_kwDOBm6k_c5LYpsf | simonw 9599 | 2022-10-02T23:01:17Z | 2022-10-02T23:01:17Z | OWNER | That change deployed and https://github-to-sqlite.dogsheep.net/github/commits now looks like this: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
word-wrap: anywhere resulting in weird display 1393903845 | |
1264753725 | https://github.com/simonw/datasette/issues/1805#issuecomment-1264753725 | https://api.github.com/repos/simonw/datasette/issues/1805 | IC_kwDOBm6k_c5LYpw9 | simonw 9599 | 2022-10-02T23:02:17Z | 2022-10-02T23:02:17Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
truncate_cells_html does not work for links? 1363552780 | ||
1264753894 | https://github.com/simonw/datasette/issues/1805#issuecomment-1264753894 | https://api.github.com/repos/simonw/datasette/issues/1805 | IC_kwDOBm6k_c5LYpzm | simonw 9599 | 2022-10-02T23:02:54Z | 2022-10-02T23:02:54Z | OWNER | I'm tempted to add |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
truncate_cells_html does not work for links? 1363552780 | |
1264769569 | https://github.com/simonw/datasette/issues/485#issuecomment-1264769569 | https://api.github.com/repos/simonw/datasette/issues/485 | IC_kwDOBm6k_c5LYtoh | simonw 9599 | 2022-10-03T00:04:42Z | 2022-10-03T00:04:42Z | OWNER | I love these tips - tools that can compile a simple machine learning model to a SQL query! Would be pretty cool if I could bundle a model in Datasette itself as a big in-memory SQLite SQL query: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Improvements to table label detection 447469253 | |
1265161668 | https://github.com/simonw/datasette/issues/1805#issuecomment-1265161668 | https://api.github.com/repos/simonw/datasette/issues/1805 | IC_kwDOBm6k_c5LaNXE | CharlesNepote 562352 | 2022-10-03T09:18:05Z | 2022-10-03T09:18:05Z | NONE |
Make sense IMHO. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
truncate_cells_html does not work for links? 1363552780 | |
1267708232 | https://github.com/simonw/datasette/issues/1829#issuecomment-1267708232 | https://api.github.com/repos/simonw/datasette/issues/1829 | IC_kwDOBm6k_c5Lj7FI | simonw 9599 | 2022-10-04T23:17:36Z | 2022-10-04T23:17:36Z | OWNER | Here's the relevant code from the table page: Note how Here's the same code for the database page: And for canned query pages: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Table/database that is private due to inherited permissions does not show padlock 1396948693 | |
1267709546 | https://github.com/simonw/datasette/issues/1829#issuecomment-1267709546 | https://api.github.com/repos/simonw/datasette/issues/1829 | IC_kwDOBm6k_c5Lj7Zq | simonw 9599 | 2022-10-04T23:19:24Z | 2022-10-04T23:21:07Z | OWNER | There's also a And is defined here: https://github.com/simonw/datasette/blob/4218c9cd742b79b1e3cb80878e42b7e39d16ded2/datasette/app.py#L694-L710 It's actually documented as a public method here: https://docs.datasette.io/en/stable/internals.html#await-check-visibility-actor-action-resource-none
Note that this documented method cannot actually do the right thing - because it's not being given the multiple permissions that need to be checked in order to completely answer the question. So I probably need to redesign that method a bit. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Table/database that is private due to inherited permissions does not show padlock 1396948693 | |
1267918117 | https://github.com/simonw/datasette/issues/1832#issuecomment-1267918117 | https://api.github.com/repos/simonw/datasette/issues/1832 | IC_kwDOBm6k_c5LkuUl | simonw 9599 | 2022-10-05T04:19:52Z | 2022-10-05T04:19:52Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
__bool__ method on Results 1397193691 | ||
1267925830 | https://github.com/simonw/datasette/issues/1832#issuecomment-1267925830 | https://api.github.com/repos/simonw/datasette/issues/1832 | IC_kwDOBm6k_c5LkwNG | simonw 9599 | 2022-10-05T04:31:57Z | 2022-10-05T04:31:57Z | OWNER | Turns out this already works -
I'll add a test to demonstrate this. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
__bool__ method on Results 1397193691 | |
1268398461 | https://github.com/simonw/datasette/issues/1824#issuecomment-1268398461 | https://api.github.com/repos/simonw/datasette/issues/1824 | IC_kwDOBm6k_c5Lmjl9 | CharlesNepote 562352 | 2022-10-05T12:55:05Z | 2022-10-05T12:55:05Z | NONE | Here is some working javascript code. There might be better solution, I'm not a JS expert. ```javascript var show_hide = document.querySelector(".show-hide-sql > a");
``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Convert &_hide_sql=1 to #_hide_sql 1387712501 | |
1268613335 | https://github.com/simonw/datasette/issues/1480#issuecomment-1268613335 | https://api.github.com/repos/simonw/datasette/issues/1480 | IC_kwDOBm6k_c5LnYDX | fgregg 536941 | 2022-10-05T15:45:49Z | 2022-10-05T15:45:49Z | CONTRIBUTOR | running into this as i continue to grow my labor data warehouse. Here a CloudRun PM says the container size should not count against memory: https://stackoverflow.com/a/56570717 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Exceeding Cloud Run memory limits when deploying a 4.8G database 1015646369 | |
1268629159 | https://github.com/simonw/datasette/issues/1480#issuecomment-1268629159 | https://api.github.com/repos/simonw/datasette/issues/1480 | IC_kwDOBm6k_c5Lnb6n | fgregg 536941 | 2022-10-05T16:00:55Z | 2022-10-05T16:00:55Z | CONTRIBUTOR | as a next step, i'll fetch the docker image from the google registry, and see what memory and disk usage looks like when i run it locally. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Exceeding Cloud Run memory limits when deploying a 4.8G database 1015646369 | |
1269275153 | https://github.com/simonw/datasette/issues/1480#issuecomment-1269275153 | https://api.github.com/repos/simonw/datasette/issues/1480 | IC_kwDOBm6k_c5Lp5oR | simonw 9599 | 2022-10-06T03:54:33Z | 2022-10-06T03:54:33Z | OWNER | I've been having success using Fly recently for a project which I thought would be too large for Cloud Run. I wrote about that here: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Exceeding Cloud Run memory limits when deploying a 4.8G database 1015646369 | |
1269847461 | https://github.com/simonw/datasette/issues/1480#issuecomment-1269847461 | https://api.github.com/repos/simonw/datasette/issues/1480 | IC_kwDOBm6k_c5LsFWl | fgregg 536941 | 2022-10-06T11:21:49Z | 2022-10-06T11:21:49Z | CONTRIBUTOR | thanks @simonw, i'll spend a little more time trying to figure out why this isn't working on cloudrun, and then will flip over to fly if i can't. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Exceeding Cloud Run memory limits when deploying a 4.8G database 1015646369 | |
1270586897 | https://github.com/simonw/datasette/pull/1835#issuecomment-1270586897 | https://api.github.com/repos/simonw/datasette/issues/1835 | IC_kwDOBm6k_c5Lu54R | simonw 9599 | 2022-10-06T19:34:00Z | 2022-10-06T19:34:00Z | OWNER | Wow, great catch! The whole point of inspect data was to avoid this kind of expensive operation on startup so this makes total sense - I had no idea Datasette was still trying to hash a giant file every time the server started. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
use inspect data for hash and file size 1400121355 | |
1270595328 | https://github.com/simonw/datasette/pull/1835#issuecomment-1270595328 | https://api.github.com/repos/simonw/datasette/issues/1835 | IC_kwDOBm6k_c5Lu78A | codecov[bot] 22429695 | 2022-10-06T19:42:25Z | 2022-10-06T19:42:25Z | NONE | Codecov ReportBase: 91.71% // Head: 92.50% // Increases project coverage by
Additional details and impacted files```diff @@ Coverage Diff @@ ## main #1835 +/- ## ========================================== + Coverage 91.71% 92.50% +0.78% ========================================== Files 38 35 -3 Lines 4754 4400 -354 ========================================== - Hits 4360 4070 -290 + Misses 394 330 -64 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1835?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1835/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | | | | [datasette/utils/shutil\_backport.py](https://codecov.io/gh/simonw/datasette/pull/1835/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3NodXRpbF9iYWNrcG9ydC5weQ==) | | | | [datasette/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1835/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL19faW5pdF9fLnB5) | | | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1835/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `94.75% <0.00%> (+0.01%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison):umbrella: View full report at Codecov. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
use inspect data for hash and file size 1400121355 | |
1270855853 | https://github.com/simonw/datasette/pull/1837#issuecomment-1270855853 | https://api.github.com/repos/simonw/datasette/issues/1837 | IC_kwDOBm6k_c5Lv7it | codecov[bot] 22429695 | 2022-10-07T00:01:20Z | 2022-10-07T00:01:20Z | NONE | Codecov ReportBase: 92.50% // Head: 92.50% // No change to project coverage :thumbsup:
Additional details and impacted files```diff @@ Coverage Diff @@ ## main #1837 +/- ## ======================================= Coverage 92.50% 92.50% ======================================= Files 35 35 Lines 4400 4400 ======================================= Hits 4070 4070 Misses 330 330 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison):umbrella: View full report at Codecov. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Make hash and size a lazy property 1400431789 | |
1271103097 | https://github.com/simonw/datasette/issues/1836#issuecomment-1271103097 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5Lw355 | fgregg 536941 | 2022-10-07T04:43:41Z | 2022-10-07T04:43:41Z | CONTRIBUTOR | @simonw, should i open up a new issue for investigating the differences between "immutable=1" and "mode=ro" and possibly switching to "mode=ro". Or would you like to keep that conversation in this issue? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1271100651 | https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5Lw3Tr | fgregg 536941 | 2022-10-07T04:38:14Z | 2022-10-07T04:38:14Z | CONTRIBUTOR |
this completely addressed #1480 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1271101072 | https://github.com/simonw/datasette/issues/1480#issuecomment-1271101072 | https://api.github.com/repos/simonw/datasette/issues/1480 | IC_kwDOBm6k_c5Lw3aQ | fgregg 536941 | 2022-10-07T04:39:10Z | 2022-10-07T04:39:10Z | CONTRIBUTOR | switching from |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Exceeding Cloud Run memory limits when deploying a 4.8G database 1015646369 | |
1270923537 | https://github.com/simonw/datasette/issues/1836#issuecomment-1270923537 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5LwMER | fgregg 536941 | 2022-10-07T00:46:08Z | 2022-10-07T00:46:08Z | CONTRIBUTOR | i thought it was maybe to do with reading through all the files, but that does not seem to be the case if i make a little test file like: ```python test_read.pyimport hashlib import sys import pathlib HASH_BLOCK_SIZE = 1024 * 1024 def inspect_hash(path): """Calculate the hash of a database, efficiently.""" m = hashlib.sha256() with path.open("rb") as fp: while True: data = fp.read(HASH_BLOCK_SIZE) if not data: break m.update(data)
inspect_hash(pathlib.Path(sys.argv[1])) ``` then a line in the Dockerfile like
just produes a layer of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1270936982 | https://github.com/simonw/datasette/issues/1836#issuecomment-1270936982 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5LwPWW | fgregg 536941 | 2022-10-07T00:52:41Z | 2022-10-07T00:52:41Z | CONTRIBUTOR | it's not that the inspect command is somehow changing the db files. if i set them to only read-only, the "inspect" layer still has the same very large size. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1270988081 | https://github.com/simonw/datasette/issues/1836#issuecomment-1270988081 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5Lwb0x | fgregg 536941 | 2022-10-07T01:19:01Z | 2022-10-07T01:27:35Z | CONTRIBUTOR | okay, some progress!! running some sql against a database file causes that file to get duplicated even if it doesn't apparently change the file. make a little test script like this: ```python test_sql.pyimport sqlite3 import sys db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ``` then
produced a layer that's the same size as |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1270992795 | https://github.com/simonw/datasette/issues/1836#issuecomment-1270992795 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5Lwc-b | fgregg 536941 | 2022-10-07T01:29:15Z | 2022-10-07T01:50:14Z | CONTRIBUTOR | fascinatingly, telling python to open sqlite in read only mode makes this layer have a size of 0 ```python test_sql_ro.pyimport sqlite3 import sys db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}?mode=ro', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ``` that's quite weird because setting the file permissions to read only didn't do anything. (on reflection, that chmod isn't doing anything because the dockerfile commands are run as root) |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1271003212 | https://github.com/simonw/datasette/issues/1836#issuecomment-1271003212 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5LwfhM | fgregg 536941 | 2022-10-07T01:52:04Z | 2022-10-07T01:52:04Z | CONTRIBUTOR | and if we try immutable mode, which is how things are opened by ```python test_sql_immutable.pyimport sqlite3 import sys db_name = sys.argv[1] conn = sqlite3.connect(f'file:/app/{db_name}?immutable=1', uri=True) cur = conn.cursor() cur.execute('select count(*) from filing') print(cur.fetchone()) ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1271004167 | https://github.com/simonw/datasette/issues/1836#issuecomment-1271004167 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5LwfwH | simonw 9599 | 2022-10-07T01:53:05Z | 2022-10-07T01:53:05Z | OWNER | Oh this is interesting! Is your hunch here that running this line is causing the file to be stored as a second layer? https://github.com/simonw/datasette/blob/5aa359b86907d11b3ee601510775a85a90224da8/datasette/utils/init.py#L399 I guess it's possible that running a non-read-only query against the database causes one or two bytes to be changed (maybe a transaction ID or similar?) Modifying the Here's what opening as a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1271009214 | https://github.com/simonw/datasette/pull/1838#issuecomment-1271009214 | https://api.github.com/repos/simonw/datasette/issues/1838 | IC_kwDOBm6k_c5Lwg-- | simonw 9599 | 2022-10-07T02:01:07Z | 2022-10-07T02:01:07Z | OWNER | The argument that has always convinced me NOT to use If you click a link with `target="_blank" on it you get a new browser window... with a disabled back button. You have to then know to close that browser window in order to return to the previous page - as opposed to hitting the "back" button like usual. You'll note that Datasette doesn't use So I'm very firmly in the anti-target-blank camp! This is the kind of change which I'd suggest implementing as a plugin. That way people who like |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Open Datasette link in new tab 1400494162 | |
1271008997 | https://github.com/simonw/datasette/issues/1836#issuecomment-1271008997 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5Lwg7l | fgregg 536941 | 2022-10-07T02:00:37Z | 2022-10-07T02:00:49Z | CONTRIBUTOR | yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in running a test of that now. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1271006020 | https://github.com/simonw/datasette/issues/1836#issuecomment-1271006020 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5LwgNE | simonw 9599 | 2022-10-07T01:54:07Z | 2022-10-07T01:54:07Z | OWNER | Just overlapped with your comment here: https://github.com/simonw/datasette/issues/1836#issuecomment-1271003212 - which notes that opening with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1271020193 | https://github.com/simonw/datasette/issues/1836#issuecomment-1271020193 | https://api.github.com/repos/simonw/datasette/issues/1836 | IC_kwDOBm6k_c5Lwjqh | fgregg 536941 | 2022-10-07T02:15:05Z | 2022-10-07T02:21:08Z | CONTRIBUTOR | when i hack the connect method to open non mutable files with "mode=ro" and not "immutable=1" https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/database.py#L79 then:
the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
docker image is duplicating db files somehow 1400374908 | |
1271024708 | https://github.com/simonw/datasette/pull/1838#issuecomment-1271024708 | https://api.github.com/repos/simonw/datasette/issues/1838 | IC_kwDOBm6k_c5LwkxE | ocdtrekkie 4399499 | 2022-10-07T02:19:49Z | 2022-10-07T02:19:49Z | NONE | Ooh, I didn't even think about links in tables! You're definitely right on the approach to this. It might also be a really good "stupidly simple" plugin for me to try to build myself, which could be fun. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Open Datasette link in new tab 1400494162 | |
1271035998 | https://github.com/simonw/datasette/issues/1301#issuecomment-1271035998 | https://api.github.com/repos/simonw/datasette/issues/1301 | IC_kwDOBm6k_c5Lwnhe | fgregg 536941 | 2022-10-07T02:38:04Z | 2022-10-07T02:38:04Z | CONTRIBUTOR | the only mode that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publishing to cloudrun with immutable mode? 860722711 | |
1271803298 | https://github.com/simonw/datasette/pull/1838#issuecomment-1271803298 | https://api.github.com/repos/simonw/datasette/issues/1838 | IC_kwDOBm6k_c5Lzi2i | simonw 9599 | 2022-10-07T16:28:41Z | 2022-10-07T16:28:41Z | OWNER | ... and here's @ocdtrekkie's plugin! https://github.com/ocdtrekkie/datasette-external-links-new-tabs |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Open Datasette link in new tab 1400494162 | |
1291392887 | https://github.com/simonw/datasette/issues/1852#issuecomment-1291392887 | https://api.github.com/repos/simonw/datasette/issues/1852 | IC_kwDOBm6k_c5M-Rd3 | simonw 9599 | 2022-10-26T02:04:48Z | 2022-10-26T02:04:48Z | OWNER | Implemented that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Default API token authentication mechanism 1421552095 | |
1291397623 | https://github.com/simonw/datasette/issues/1852#issuecomment-1291397623 | https://api.github.com/repos/simonw/datasette/issues/1852 | IC_kwDOBm6k_c5M-Sn3 | simonw 9599 | 2022-10-26T02:11:40Z | 2022-10-26T02:11:40Z | OWNER | Built a prototype of the
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Default API token authentication mechanism 1421552095 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
user >30