github
| html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
|---|---|---|---|---|---|---|---|---|---|---|---|
| https://github.com/simonw/datasette/issues/1356#issuecomment-1017016553 | https://api.github.com/repos/simonw/datasette/issues/1356 | 1017016553 | IC_kwDOBm6k_c48nnDp | 9599 | 2022-01-20T01:06:37Z | 2022-01-20T01:06:37Z | OWNER | > A problem with this is that if you're using `--query` you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is `.csv` and plugin formats can't handle full streams, see #1062 and #1177. I figured out a neat pattern for streaming JSON arrays in this TIL: https://til.simonwillison.net/python/output-json-array-streaming | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
910092577 | |
| https://github.com/simonw/datasette/issues/1605#issuecomment-1016994329 | https://api.github.com/repos/simonw/datasette/issues/1605 | 1016994329 | IC_kwDOBm6k_c48nhoZ | 25778 | 2022-01-20T00:27:17Z | 2022-01-20T00:27:17Z | CONTRIBUTOR | Right now, I usually have a line in a Makefile like this: ```make combined.geojson: project.db pipenv run datasette project.db --get /project/combined.geojson \ --load-extension spatialite \ --setting sql_time_limit_ms 5000 \ --setting max_returned_rows 20000 \ -m metadata.yml > $@ ``` That all assumes I've loaded whatever I need into `project.db` and created a canned query called `combined` (and then uses `datasette-geojson` for geojson output). It works, but as you can see, it's a lot to manage, a lot of boilerplate, and it wasn't obvious how to get there. If there's an error in the canned query, I get an HTML error page, so that's hard to debug. And it's only one query, so each output needs a line like this. Make isn't ideal, either, for that reason. The thing I really liked with `datafreeze` was doing templated filenames. I have a project now where I need to export a bunch of litttle geojson files, based on queries, and it would be awesome to be able to do something like this: ```yml databases: project: queries: boundaries: sql: "SELECT * FROM boundaries" filename: "boundaries/{id}.geojson" mode: "item" format: geojson ``` And then do: ```sh datasette freeze -m metadata.yml project.db ``` For HTML export, maybe there's a `template` argument, or `format: template` or something. And that gets you a static site generator, kinda for free. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1108671952 | |
| https://github.com/simonw/datasette/issues/1605#issuecomment-1016977725 | https://api.github.com/repos/simonw/datasette/issues/1605 | 1016977725 | IC_kwDOBm6k_c48ndk9 | 9599 | 2022-01-19T23:55:08Z | 2022-01-19T23:55:08Z | OWNER | Oh that's interesting. I was thinking about this from a slightly different angle recently - pondering what a static site generator built on top of Datasette might look like. Just a sketch at the moment, but I was imagining a YAML configuration file with a SQL query that returns a list of paths - then a tool that runs that query and uses the equivalent of `datasette --get` to create a static copy of each of those paths. I think these two ideas can probably be merged. I'd love to know more about how you are solving this right now! | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1108671952 | |
| https://github.com/simonw/datasette/issues/1601#issuecomment-1016651485 | https://api.github.com/repos/simonw/datasette/issues/1601 | 1016651485 | IC_kwDOBm6k_c48mN7d | 25778 | 2022-01-19T16:39:03Z | 2022-01-19T16:39:03Z | CONTRIBUTOR | I think both of these are Spatialite specific. They get generated when you first initialize the extension. KNN is actually deprecated in favor of [KNN2](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN2), as I understand it. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1105916061 | |
| https://github.com/simonw/datasette/issues/1601#issuecomment-1016639294 | https://api.github.com/repos/simonw/datasette/issues/1601 | 1016639294 | IC_kwDOBm6k_c48mK8- | 9599 | 2022-01-19T16:26:44Z | 2022-01-19T16:26:56Z | OWNER | I need to add `sqlite_stat1` to the hidden tables too, see: - https://github.com/simonw/sqlite-utils/issues/366 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1105916061 | |
| https://github.com/simonw/datasette/issues/1601#issuecomment-1016637722 | https://api.github.com/repos/simonw/datasette/issues/1601 | 1016637722 | IC_kwDOBm6k_c48mKka | 9599 | 2022-01-19T16:25:07Z | 2022-01-19T16:25:07Z | OWNER | Good idea - though I'm nervous about accidentally hiding a `data_licenses` table outside of the SpatiaLite case. I'll only hide that one if SpatiaLite is loaded. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1105916061 | |
| https://github.com/simonw/datasette/issues/1604#issuecomment-1016636561 | https://api.github.com/repos/simonw/datasette/issues/1604 | 1016636561 | IC_kwDOBm6k_c48mKSR | 9599 | 2022-01-19T16:23:54Z | 2022-01-19T16:23:54Z | OWNER | Potential design: datasette publish cloudrun data.db \ --service my-service \ --domain demo.datasette.io I think I'm OK with calling this `--domain` even when it is being used with a subdomain. This will require `datasette.io` to already have been verified with Google. Not sure how best to handle the DNS part - maybe print out instructions for the necessary CNAME? | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1108300685 | |
| https://github.com/simonw/datasette/issues/1603#issuecomment-1016589519 | https://api.github.com/repos/simonw/datasette/issues/1603 | 1016589519 | IC_kwDOBm6k_c48l-zP | 9599 | 2022-01-19T15:36:38Z | 2022-01-19T15:36:38Z | OWNER | Also people can use a custom base template and link to a custom favicon if they want to. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1108235694 | |
| https://github.com/simonw/datasette/issues/1603#issuecomment-1016589140 | https://api.github.com/repos/simonw/datasette/issues/1603 | 1016589140 | IC_kwDOBm6k_c48l-tU | 9599 | 2022-01-19T15:36:16Z | 2022-01-19T15:36:16Z | OWNER | For the moment then I will hard-code a new favicon and leave it to ASGI plugins if people want to define their own. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1108235694 | |
| https://github.com/simonw/datasette/issues/1603#issuecomment-1016588326 | https://api.github.com/repos/simonw/datasette/issues/1603 | 1016588326 | IC_kwDOBm6k_c48l-gm | 9599 | 2022-01-19T15:35:33Z | 2022-01-19T15:35:33Z | OWNER | I don't think abusing the template loader mechanism for this will work: Jinja provides an API for loading text templates, but I don't think it can be sensibly abused to open binary image files instead. Loaded code is here: https://github.com/pallets/jinja/blob/main/src/jinja2/loaders.py | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1108235694 | |
| https://github.com/simonw/datasette/issues/1603#issuecomment-1016579661 | https://api.github.com/repos/simonw/datasette/issues/1603 | 1016579661 | IC_kwDOBm6k_c48l8ZN | 9599 | 2022-01-19T15:27:05Z | 2022-01-19T15:27:05Z | OWNER | I'd like this to be customizable. I'm going to load it from the template loading system such that a custom favicon for a specific instance can be dropped in using a file in `templates/`. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1108235694 | |
| https://github.com/simonw/datasette/pull/1602#issuecomment-1016456784 | https://api.github.com/repos/simonw/datasette/issues/1602 | 1016456784 | IC_kwDOBm6k_c48leZQ | 22429695 | 2022-01-19T13:17:24Z | 2022-01-19T13:17:24Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1602](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (29f5772) into [main](https://codecov.io/gh/simonw/datasette/commit/cb29119db9115b1f40de2fb45263ed77e3bfbb3e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (cb29119) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1602 +/- ## ======================================= Coverage 92.09% 92.09% ======================================= Files 34 34 Lines 4516 4516 ======================================= Hits 4159 4159 Misses 357 357 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [cb29119...29f5772](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1108084641 | |
| https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1013698557 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | 1013698557 | IC_kwDOCGYnMM48a8_9 | 25778 | 2022-01-15T15:15:22Z | 2022-01-15T15:15:22Z | NONE | @simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
557842245 | |
| https://github.com/simonw/datasette/issues/1591#issuecomment-1013669543 | https://api.github.com/repos/simonw/datasette/issues/1591 | 1013669543 | IC_kwDOBm6k_c48a16n | 9599 | 2022-01-15T11:56:59Z | 2022-01-15T11:56:59Z | OWNER | There's actually already a way to move regular Datasette `--setting` options to a `settings.json` file thanks to configuration directory mode: https://docs.datasette.io/en/stable/settings.html Maybe extending that mechanism to handle plugins would be a neat path forward here. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1100015398 | |
| https://github.com/simonw/datasette/issues/1591#issuecomment-1013668967 | https://api.github.com/repos/simonw/datasette/issues/1591 | 1013668967 | IC_kwDOBm6k_c48a1xn | 9599 | 2022-01-15T11:53:21Z | 2022-01-15T11:53:21Z | OWNER | The `datasette publish --plugin-secret name setting value` option already implements something a bit like this. https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values It's a bit of a messy hack to compensate for metadata being visible. Maybe I could replace that mechanism with the proposed plugin configuration rethink from this issue. I still like the debug benefits of making plugin settings public - perhaps add a rule that if a plugin setting has a `secret:` prefix it gets redacted on a new `/-/plugin-settings` page. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1100015398 | |
| https://github.com/simonw/datasette/issues/1566#issuecomment-1012680228 | https://api.github.com/repos/simonw/datasette/issues/1566 | 1012680228 | IC_kwDOBm6k_c48XEYk | 9599 | 2022-01-14T01:59:54Z | 2022-01-14T01:59:54Z | OWNER | This is now shipped! | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1083669410 | |
| https://github.com/simonw/datasette/issues/1595#issuecomment-1012664607 | https://api.github.com/repos/simonw/datasette/issues/1595 | 1012664607 | IC_kwDOBm6k_c48XAkf | 9599 | 2022-01-14T01:22:58Z | 2022-01-14T01:22:58Z | OWNER | - Upgraded Pluggy dependency to 1.0. #1575 - Now using [Plausible](https://plausible.io/) analytics for the Datasette documentation. - The `db.execute_write()` internals method now defaults to blocking until the write operation has completed. Previously it defaulted to queuing the write and then continuing to run code while the write was in the queue. #1579 - `explain query plan` is now allowed with varying amounts of white space in the query. #1588 - New CLI reference page showing the output of `--help` for each of the `datasette` sub-commands. This lead to several small improvements to the help copy. #1594 - Fixed bug where columns with a underscore prefix could result in unnecessary hidden form fields. #1527 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1102484126 | |
| https://github.com/simonw/datasette/issues/1595#issuecomment-1012626410 | https://api.github.com/repos/simonw/datasette/issues/1595 | 1012626410 | IC_kwDOBm6k_c48W3Pq | 9599 | 2022-01-14T00:00:56Z | 2022-01-14T01:17:47Z | OWNER | Commits since 0.60a1: https://github.com/simonw/datasette/compare/0.60a1...3664ddd40 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1102484126 | |
| https://github.com/simonw/datasette/issues/1590#issuecomment-1012661522 | https://api.github.com/repos/simonw/datasette/issues/1590 | 1012661522 | IC_kwDOBm6k_c48W_0S | 9599 | 2022-01-14T01:16:08Z | 2022-01-14T01:16:34Z | OWNER | OK, I'm going to recommend a workaround for this instead. Here's `asgi.py` updated to strip the prefix before passing the request on to Datasette: ```python import pathlib from asgi_cors import asgi_cors from channels.routing import URLRouter from django.urls import re_path from datasette.app import Datasette def rewrite_path(app, prefix_to_strip): async def rewrite_path_app(scope, receive, send): if ( scope["type"] == "http" and "path" in scope and scope["path"].startswith(prefix_to_strip) ): scope["path"] = scope["path"][len(prefix_to_strip) :] if "raw_path" in scope: scope["raw_path"] = scope["raw_path"][len(prefix_to_strip) :] await app(scope, receive, send) return rewrite_path_app datasette_ = Datasette( files=["fixtures.db"], settings={"base_url": "/datasettes/", "plugins": {}}, ) application = URLRouter( [ re_path( r"^datasettes/.*", asgi_cors(rewrite_path(datasette_.app(), "/datasettes"), allow_all=True), ), ] ) ``` This works on my laptop - please re-open the ticket if it doesn't work for you! | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099723916 | |
| https://github.com/simonw/datasette/issues/1590#issuecomment-1012656790 | https://api.github.com/repos/simonw/datasette/issues/1590 | 1012656790 | IC_kwDOBm6k_c48W-qW | 9599 | 2022-01-14T01:05:34Z | 2022-01-14T01:05:34Z | OWNER | I think this prefixed string mechanism is supposed to prevent the `base_url` prefix from being applied twice: https://github.com/simonw/datasette/blob/3664ddd400062123e99500d28b160c7944408c1a/datasette/url_builder.py#L9-L16 But with a bit of extra logging all of the inputs to that are NOT prefixed strings: ``` Urls.path called with: /datasettes/fixtures/compound_three_primary_keys?_sort=content (PrefixedUrlString = False) returning /datasettes/datasettes/fixtures/compound_three_primary_keys?_sort=content ``` So it looks like `urls.path(...)` is indeed the code responsible for doubling up that `/datasettes/` prefix. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099723916 | |
| https://github.com/simonw/datasette/issues/1590#issuecomment-1012653966 | https://api.github.com/repos/simonw/datasette/issues/1590 | 1012653966 | IC_kwDOBm6k_c48W9-O | 9599 | 2022-01-14T00:59:07Z | 2022-01-14T00:59:07Z | OWNER | Since this is a special case bug for when using Datasette as a library I wonder if a good fix here would be to support something like this: ```python application = URLRouter([ re_path(r"^datasettes/.*", asgi_cors(datasette_.app(remove_path_prefix="datasettes/"), allow_all=True)), ]) ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099723916 | |
| https://github.com/simonw/datasette/issues/1527#issuecomment-1012653109 | https://api.github.com/repos/simonw/datasette/issues/1527 | 1012653109 | IC_kwDOBm6k_c48W9w1 | 9599 | 2022-01-14T00:57:08Z | 2022-01-14T00:57:08Z | OWNER | Bug is fixed on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1059555791 | |
| https://github.com/simonw/datasette/issues/1598#issuecomment-1012643882 | https://api.github.com/repos/simonw/datasette/issues/1598 | 1012643882 | IC_kwDOBm6k_c48W7gq | 9599 | 2022-01-14T00:34:49Z | 2022-01-14T00:34:49Z | OWNER | There are four places in the documentation that use `.. literalinclude::` at the moment which I can replace - I can actually just link directly to the new https://docs.datasette.io/en/latest/cli-reference.html page instead of embedding the help directly. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1102637351 | |
| https://github.com/simonw/datasette/issues/1527#issuecomment-1012635696 | https://api.github.com/repos/simonw/datasette/issues/1527 | 1012635696 | IC_kwDOBm6k_c48W5gw | 9599 | 2022-01-14T00:19:10Z | 2022-01-14T00:20:36Z | OWNER | Oh! This is because `_city_id` has a leading underscore (for testing purposes). I think I need to filter out any keys that contain `__` in that case. What happens to columns that contain a `__`? They shouldn't be reflected in the hidden arguments either - this code is really only supposed to catch things like `_where` and `_m2m_through=` and `_col` - basically most of the list on https://docs.datasette.io/en/stable/json_api.html#special-table-arguments | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1059555791 | |
| https://github.com/simonw/datasette/issues/1527#issuecomment-1012634659 | https://api.github.com/repos/simonw/datasette/issues/1527 | 1012634659 | IC_kwDOBm6k_c48W5Qj | 9599 | 2022-01-14T00:17:00Z | 2022-01-14T00:18:11Z | OWNER | That's because that page has this unnecessary hidden form field: ```html <input type="hidden" name="_city_id__gt" value="1"> ``` That field is added by this bit in the template: https://github.com/simonw/datasette/blob/515f8d38ebae203efc15ca79a8b42848276b35e5/datasette/templates/table.html#L119-L122 Which is populated here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/views/table.py#L813-L821 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1059555791 | |
| https://github.com/simonw/datasette/issues/1597#issuecomment-1012629825 | https://api.github.com/repos/simonw/datasette/issues/1597 | 1012629825 | IC_kwDOBm6k_c48W4FB | 9599 | 2022-01-14T00:07:36Z | 2022-01-14T00:07:36Z | OWNER | Fixed: ``` % datasette --help Usage: datasette [OPTIONS] COMMAND [ARGS]... Datasette is an open source multi-tool for exploring and publishing data About Datasette: https://datasette.io/ Full documentation: https://docs.datasette.io/ Options: --version Show the version and exit. --help Show this message and exit. Commands: serve* Serve up specified SQLite database files with a web UI inspect Generate JSON summary of provided database files ... % datasette inspect --help Usage: datasette inspect [OPTIONS] [FILES]... Generate JSON summary of provided database files This can then be passed to "datasette --inspect-file" to speed up count operations against immutable database files. Options: --inspect-file TEXT --load-extension TEXT Path to a SQLite extension to load --help Show this message and exit. ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1102612922 | |
| https://github.com/simonw/datasette/issues/1595#issuecomment-1012626243 | https://api.github.com/repos/simonw/datasette/issues/1595 | 1012626243 | IC_kwDOBm6k_c48W3ND | 9599 | 2022-01-14T00:00:33Z | 2022-01-14T00:00:33Z | OWNER | Copying those in here: - New plugin hook: [filters_from_request(request, database, table, datasette)](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-filters-from-request), which runs on the table page and can be used to support new custom query string parameters that modify the SQL query. ([#473](https://github.com/simonw/datasette/issues/473)) - The number of unique values in a facet is now always displayed. Previously it was only displayed if the user specified `?_facet_size=max`. ([#1556](https://github.com/simonw/datasette/issues/1556)) - Fixed bug where `?_facet_array=tags&_facet=tags` would only display one of the two selected facets. ([#625](https://github.com/simonw/datasette/issues/625)) - Facets of type `date` or `array` can now be configured in `metadata.json`, see [Facets in metadata.json](https://docs.datasette.io/en/latest/facets.html#facets-metadata). Thanks, David Larlet. ([#1552](https://github.com/simonw/datasette/issues/1552)) - New `?_nosuggest=1` parameter for table views, which disables facet suggestion. ([#1557](https://github.com/simonw/datasette/issues/1557)) - Label columns detected for foreign keys are now case-insensitive, so `Name` or `TITLE` will be detected in the same way as `name` or `title`. ([#1544](https://github.com/simonw/datasette/issues/1544)) - The query string variables exposed by `request.args` will now include blank strings for arguments such as `foo` in `?foo=&bar=1` rather than ignoring those parameters entirely. ([#1551](https://github.com/simonw/datasette/issues/1551)) - Database write connections now execute the [prepare_connection(conn, database, datasette)](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-prepare-connection) plugin hook. ([#1564](https://github.com/simonw/datasette/issues/1564)) - The `Datasette()` constructor no longer requires the `files=` argument, and is now documented at [Datasette class](https://docs.datasette.io/en/latest/internals.html#internals-datasette). ([#1563](https://github.com/simonw/datasette/issues/1563)) - The tracing feature now traces write queries, not just read queries. ([#1568](https://github.com/simonw/datasette/issues/1568)) - Added two methods for writing to the database: [await db.execute_write_script(sql, block=False)](https://docs.datasette.io/en/latest/internals.html#database-execute-write-script) and [await db.execute_write_many(sql, params_seq, block=False)](https://docs.datasette.io/en/latest/internals.html#database-execute-write-many). ([#1570](https://github.com/simonw/datasette/issues/1570)) - Made several performance improvements to the database schema introspection code that runs when Datasette first starts up. ([#1555](https://github.com/simonw/datasette/issues/1555)) - Fixed bug where writable canned queries could not be used with custom templates. ([#1547](https://github.com/simonw/datasette/issues/1547)) | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1102484126 | |
| https://github.com/simonw/datasette/issues/1590#issuecomment-1012583091 | https://api.github.com/repos/simonw/datasette/issues/1590 | 1012583091 | IC_kwDOBm6k_c48Wsqz | 9599 | 2022-01-13T22:41:15Z | 2022-01-13T22:41:15Z | OWNER | Seeing as this area of the code has produced so many bugs in the past, I think part of the fix may be to write comprehensive documentation about how routing works for the internals documentation. Doing so might help me figure this bug out! | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099723916 | |
| https://github.com/simonw/datasette/issues/1595#issuecomment-1012575013 | https://api.github.com/repos/simonw/datasette/issues/1595 | 1012575013 | IC_kwDOBm6k_c48Wqsl | 9599 | 2022-01-13T22:29:22Z | 2022-01-13T22:29:22Z | OWNER | Most of these are already written for these two alpha releases: - https://github.com/simonw/datasette/releases/tag/0.60a0 - https://github.com/simonw/datasette/releases/tag/0.60a1 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1102484126 | |
| https://github.com/simonw/datasette/issues/1588#issuecomment-1012554208 | https://api.github.com/repos/simonw/datasette/issues/1588 | 1012554208 | IC_kwDOBm6k_c48Wlng | 9599 | 2022-01-13T22:07:15Z | 2022-01-13T22:07:15Z | OWNER | This works now: https://latest.datasette.io/fixtures?sql=explain+query+plan++select+*+from+facetable | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097101917 | |
| https://github.com/simonw/datasette/issues/650#issuecomment-1012552760 | https://api.github.com/repos/simonw/datasette/issues/650 | 1012552760 | IC_kwDOBm6k_c48WlQ4 | 9599 | 2022-01-13T22:04:56Z | 2022-01-13T22:04:56Z | OWNER | Challenge: explain the difference between view as in SQL view, and view as in the code that serves `TableView` / `DatabaseView` etc. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
534629631 | |
| https://github.com/simonw/datasette/issues/1336#issuecomment-1012546924 | https://api.github.com/repos/simonw/datasette/issues/1336 | 1012546924 | IC_kwDOBm6k_c48Wj1s | 9599 | 2022-01-13T21:55:58Z | 2022-01-13T21:55:58Z | OWNER | See also: - #1412 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
895686039 | |
| https://github.com/simonw/datasette/issues/1466#issuecomment-1012546223 | https://api.github.com/repos/simonw/datasette/issues/1466 | 1012546223 | IC_kwDOBm6k_c48Wjqv | 9599 | 2022-01-13T21:54:51Z | 2022-01-13T21:54:51Z | OWNER | Going with this for the copy: > [Datasette Desktop](https://datasette.io/desktop) is a packaged Mac application which bundles Datasette together with Python and allows you to install and run Datasette directly on your laptop. This is the best option for local installation if you are not comfortable using the command line. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
991467558 | |
| https://github.com/simonw/datasette/issues/1594#issuecomment-1012536257 | https://api.github.com/repos/simonw/datasette/issues/1594 | 1012536257 | IC_kwDOBm6k_c48WhPB | 9599 | 2022-01-13T21:38:48Z | 2022-01-13T21:38:48Z | OWNER | https://docs.datasette.io/en/latest/cli-reference.html | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1102359726 | |
| https://github.com/simonw/datasette/issues/1594#issuecomment-1012535024 | https://api.github.com/repos/simonw/datasette/issues/1594 | 1012535024 | IC_kwDOBm6k_c48Wg7w | 9599 | 2022-01-13T21:36:53Z | 2022-01-13T21:36:53Z | OWNER | I went with a simpler pattern that `sqlite-utils` because Datasette has a lot less commands: https://github.com/simonw/datasette/blob/4b23f01f3e668c8f2a2f1a294be49f49b4073969/docs/cli-reference.rst#L9-L35 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1102359726 | |
| https://github.com/simonw/datasette/issues/1594#issuecomment-1012508787 | https://api.github.com/repos/simonw/datasette/issues/1594 | 1012508787 | IC_kwDOBm6k_c48Wahz | 9599 | 2022-01-13T20:56:14Z | 2022-01-13T20:56:34Z | OWNER | The implementation can be _almost_ exactly the same as this: https://github.com/simonw/sqlite-utils/blame/74586d3cb26fa3cc3412721985ecdc1864c2a31d/docs/cli-reference.rst#L11-L76 I need to do something extra to ensure the output of `datasette --help-settings` is shown too. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1102359726 | |
| https://github.com/simonw/datasette/issues/1591#issuecomment-1012506595 | https://api.github.com/repos/simonw/datasette/issues/1591 | 1012506595 | IC_kwDOBm6k_c48WZ_j | 9599 | 2022-01-13T20:52:56Z | 2022-01-13T20:52:56Z | OWNER | You can already run `datasette --help-settings` to see detailed help on available settings. Maybe `datasette --help-plugin-settings` could do the same thing for plugin settings? Or I could even have available plugin settings show up as a list at the bottom of the `datasette --help-settings` output - which currently looks like this: ``` % datasette --help-settings Settings: default_page_size Default page size for the table view (default=100) max_returned_rows Maximum rows that can be returned from a table or custom query (default=1000) num_sql_threads Number of threads in the thread pool for executing SQLite queries (default=3) sql_time_limit_ms Time limit for a SQL query in milliseconds (default=1000) default_facet_size Number of values to return for requested facets (default=30) facet_time_limit_ms Time limit for calculating a requested facet (default=200) facet_suggest_time_limit_ms Time limit for calculating a suggested facet (default=50) hash_urls Include DB file contents hash in URLs, for far- future caching (default=False) allow_facet Allow users to specify columns to facet using ?_facet= parameter (default=True) allow_download Allow users to download the original SQLite database files (default=True) suggest_facets Calculate and display suggested facets (default=True) default_cache_ttl Default HTTP cache TTL (used in Cache-Control: max-age= header) (default=5) default_cache_ttl_hashed Default HTTP cache TTL for hashed URL pages (default=31536000) cache_size_kb SQLite cache size in KB (0 == use SQLite default) (default=0) allow_csv_stream Allow .csv?_stream=1 to download all rows (ignoring max_returned_rows) (default=True) max_csv_mb Maximum size allowed for CSV export in MB - set 0 to disable this limit (default=100) truncate_cells_html Truncate cells longer than this in HTML table view - set 0 to disable (default=2048) force_https_urls Force URLs in API output to always use https:// protocol (default=False) template_debug Allow display of template debug information with ?_context=1 (default=False) trace_debug Allow display of SQL trace debug information with ?_trace=1 (default=False) base_url Datasette URLs should use this base path (default=/) ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1100015398 | |
| https://github.com/simonw/datasette/issues/1591#issuecomment-1012505706 | https://api.github.com/repos/simonw/datasette/issues/1591 | 1012505706 | IC_kwDOBm6k_c48WZxq | 9599 | 2022-01-13T20:51:30Z | 2022-01-13T20:51:30Z | OWNER | Another option: if I make plugin settings a higher level concept in Datasette than they are at the moment, I could allow them to be set either using `--options` OR using the existing `metadata.yml/json` mechanism. https://docs.datasette.io/en/stable/plugins.html#plugin-configuration I want to make changes to that anyway, because I'm increasingly uncomfortable with plugin settings ending up in the "metadata" mechanism. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1100015398 | |
| https://github.com/simonw/datasette/issues/1591#issuecomment-1012504251 | https://api.github.com/repos/simonw/datasette/issues/1591 | 1012504251 | IC_kwDOBm6k_c48WZa7 | 9599 | 2022-01-13T20:49:19Z | 2022-01-13T20:49:59Z | OWNER | I try to stick pretty closely to what [Click](https://click.palletsprojects.com/en/8.0.x/) supports, and Click likes you to define options explicitly so that it can display them in the output of `--help`. But... that makes me think that actually showing these options in `--help` is likely a better idea. My `--plugin-setting` concept would have help that looks something like this: ``` % datasette serve --help ... --plugin-setting <TEXT TEXT TEXT>... Setting for a specified plugin. ``` That's not great help! The alternative would be to allow plugins to register their extra options with the command - which would mean the help output could look like this instead: ``` % datasette serve --help ... --tiddlywiki-db <TEXT> Name of database to use for datasette-tiddlywiki ``` This feels like a good argument to me for plugins to explicitly register their settings. I'm not sure if I should enforce the `name-of-plugin-` prefix, or if I should allow plugins to define any setting they like. I'm still nervous about plugins over-riding existing or future default options to that command. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1100015398 | |
| https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012413729 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | 1012413729 | IC_kwDOCGYnMM48WDUh | 25778 | 2022-01-13T18:50:00Z | 2022-01-13T18:50:00Z | NONE | One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
557842245 | |
| https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012253198 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | 1012253198 | IC_kwDOCGYnMM48VcIO | 25778 | 2022-01-13T15:39:14Z | 2022-01-13T15:39:14Z | NONE | Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (`gis.py` or something) just so it's easier to find later. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
557842245 | |
| https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012230212 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | 1012230212 | IC_kwDOCGYnMM48VWhE | 25778 | 2022-01-13T15:15:13Z | 2022-01-13T15:15:13Z | NONE | Some proposals I'd add to sqlite-utils: Some version of this, from [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite/blob/main/geojson_to_sqlite/utils.py#L124-L130): ```python def init_spatialite(db, lib): db.conn.enable_load_extension(True) db.conn.load_extension(lib) # Initialize SpatiaLite if not yet initialized if "spatial_ref_sys" in db.table_names(): return db.conn.execute("select InitSpatialMetadata(1)") ``` Also a function for creating a spatial index: ```python db.conn.execute("select CreateSpatialIndex(?, ?)", [table, "geometry"]) ``` I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like: ```sh sqlite-utils spatial-index spatial.db table-name column-name ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
557842245 | |
| https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012158895 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | 1012158895 | IC_kwDOCGYnMM48VFGv | 25778 | 2022-01-13T13:55:59Z | 2022-01-13T13:55:59Z | NONE | Came here to add this. I might pick it up. Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
557842245 | |
| https://github.com/simonw/datasette/pull/1593#issuecomment-1012128696 | https://api.github.com/repos/simonw/datasette/issues/1593 | 1012128696 | IC_kwDOBm6k_c48U9u4 | 22429695 | 2022-01-13T13:18:35Z | 2022-01-13T13:18:35Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1593](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (df73ebb) into [main](https://codecov.io/gh/simonw/datasette/commit/8c401ee0f054de2f568c3a8302c9223555146407?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8c401ee) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1593 +/- ## ======================================= Coverage 92.09% 92.09% ======================================= Files 34 34 Lines 4516 4516 ======================================= Hits 4159 4159 Misses 357 357 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [8c401ee...df73ebb](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1101705012 | |
| https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1011855133 | https://api.github.com/repos/simonw/sqlite-utils/issues/348 | 1011855133 | IC_kwDOCGYnMM48T68d | 9599 | 2022-01-13T07:06:59Z | 2022-01-13T07:06:59Z | OWNER | Wrote a lot more about this feature here: https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/ | {
"total_count": 1,
"+1": 1,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1067771698 | |
| https://github.com/simonw/datasette/issues/1592#issuecomment-1011185061 | https://api.github.com/repos/simonw/datasette/issues/1592 | 1011185061 | IC_kwDOBm6k_c48RXWl | 9599 | 2022-01-12T15:50:41Z | 2022-01-12T15:50:41Z | OWNER | Twitter: https://twitter.com/dracos/status/1481290103519592459 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1100499619 | |
| https://github.com/simonw/datasette/issues/1591#issuecomment-1010947634 | https://api.github.com/repos/simonw/datasette/issues/1591 | 1010947634 | IC_kwDOBm6k_c48QdYy | 82988 | 2022-01-12T11:32:17Z | 2022-01-12T11:32:17Z | CONTRIBUTOR | Is it possible to parse things like `--ext-{plugin}-{arg} VALUE` ? | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1100015398 | |
| https://github.com/simonw/datasette/issues/1591#issuecomment-1010764036 | https://api.github.com/repos/simonw/datasette/issues/1591 | 1010764036 | IC_kwDOBm6k_c48PwkE | 9599 | 2022-01-12T08:22:16Z | 2022-01-12T08:22:32Z | OWNER | The challenge here is avoiding clashes. What if a plugin adds an option that I later want to use for a new Datasette core feature? Or what if two plugins define the same option? Maybe the solution is to make them use namespaces defined by the plugin name. How about this: datasette t.db --plugin-setting datasette-tiddlywiki db t.db It's a bit verbose having an option that itself then takes THREE strings: plugin name, setting name, setting value - but it would work. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1100015398 | |
| https://github.com/simonw/datasette/issues/1590#issuecomment-1010559681 | https://api.github.com/repos/simonw/datasette/issues/1590 | 1010559681 | IC_kwDOBm6k_c48O-rB | 1001306 | 2022-01-12T02:10:20Z | 2022-01-12T02:10:20Z | NONE | In my example, path matching happens at the application layer (being the Django channels URLRouter). That might be a somewhat exotic solution that would normally be solved by a proxy like Apache or Nginx. However, in my specific use case, this is a "feature" enabling me to do simple management of databases and metadata from within a Django admin app instance mapped in that same router. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099723916 | |
| https://github.com/simonw/datasette/issues/1590#issuecomment-1010556333 | https://api.github.com/repos/simonw/datasette/issues/1590 | 1010556333 | IC_kwDOBm6k_c48O92t | 1001306 | 2022-01-12T02:03:59Z | 2022-01-12T02:03:59Z | NONE | Thank you for the quick reply! Just a quick observation, I am running this locally without a proxy, whereas your fly example seems to be running behind an apache proxy (if the name is accurate). Can it be that the apache proxy strips the prefix before it passes on the request to the daphne backend? | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099723916 | |
| https://github.com/simonw/datasette/issues/1590#issuecomment-1010540923 | https://api.github.com/repos/simonw/datasette/issues/1590 | 1010540923 | IC_kwDOBm6k_c48O6F7 | 9599 | 2022-01-12T01:33:49Z | 2022-01-12T01:33:49Z | OWNER | Looking closer at the code quoted above, it doesn't modify `path` or `raw_path` at all - ALL it does is add the `route_path` to the scope. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099723916 | |
| https://github.com/simonw/datasette/issues/1590#issuecomment-1010538188 | https://api.github.com/repos/simonw/datasette/issues/1590 | 1010538188 | IC_kwDOBm6k_c48O5bM | 9599 | 2022-01-12T01:28:41Z | 2022-01-12T01:30:43Z | OWNER | Oh wait! It looks like `route_path` is something I invented there. Yup, I added it in https://github.com/simonw/datasette/commit/a63412152518581c6a3d4e142b937e27dabdbfdb - commit message says: > - new `route_path` key in `request.scope` storing the path that was used for routing with the `base_url` prefix stripped So actually part of the mystery here is: why does the Fly hosted one NOT have that key? | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099723916 | |
| https://github.com/simonw/datasette/issues/1590#issuecomment-1010538016 | https://api.github.com/repos/simonw/datasette/issues/1590 | 1010538016 | IC_kwDOBm6k_c48O5Yg | 9599 | 2022-01-12T01:28:19Z | 2022-01-12T01:28:19Z | OWNER | The Daphne one has this key: `'route_path': '/-/asgi-scope',` Maybe Datasette's routing code needs to look out for that, if it's available, and use it to reconstruct the requested path? The code in question is here: https://github.com/simonw/datasette/blob/8c401ee0f054de2f568c3a8302c9223555146407/datasette/app.py#L1143-L1149 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099723916 | |
| https://github.com/simonw/datasette/issues/1590#issuecomment-1010537058 | https://api.github.com/repos/simonw/datasette/issues/1590 | 1010537058 | IC_kwDOBm6k_c48O5Ji | 9599 | 2022-01-12T01:26:34Z | 2022-01-12T01:26:34Z | OWNER | I'm using the https://datasette.io/plugins/datasette-debug-asgi plugin to investigate. On my laptop using Daphne I get this: http://127.0.0.1:8032/datasettes/-/asgi-scope ``` {'actor': None, 'asgi': {'version': '3.0'}, 'client': ['127.0.0.1', 53767], 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x1122aeef0>, 'headers': [(b'host', b'127.0.0.1:8032'), (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko' b'/20100101 Firefox/95.0'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/' b'avif,image/webp,*/*;q=0.8'), (b'accept-language', b'en-US,en;q=0.5'), (b'accept-encoding', b'gzip, deflate'), (b'dnt', b'1'), (b'connection', b'keep-alive'), (b'cookie', b'_ga=GA1.1.742283954.1628542653'), (b'upgrade-insecure-requests', b'1'), (b'sec-fetch-dest', b'document'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-site', b'none'), (b'sec-fetch-user', b'?1')], 'http_version': '1.1', 'method': 'GET', 'path': '/datasettes/-/asgi-scope', 'path_remaining': '', 'query_string': b'', 'raw_path': b'/datasettes/-/asgi-scope', 'root_path': '', 'route_path': '/-/asgi-scope', 'scheme': 'http', 'server': ['127.0.0.1', 8032], 'type': 'http', 'url_route': {'kwargs': {}}} ``` On the demo running on Fly (which I just redeployed with that plugin) I get this: https://datasette-apache-proxy-demo.fly.dev/prefix/-/asgi-scope ``` {'actor': None, 'asgi': {'spec_version': '2.1', 'version': '3.0'}, 'client': ('86.109.12.167', 0), 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x7f4c0413bca0>, 'headers': [(b'host', b'datasette-apache-proxy-demo.fly.dev'), (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko' b'/20100101 Firefox/95.0'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/' b'avif,image/webp,*/*;q=0.8'), (b'accept-language', b'en-US,en;q=0.5'), (b'accept-encoding', b'gzip, deflate, br'), (b'dnt', b'1'), (b'x-request-start', b't=1641950740651658'), (b'sec-fetch-dest', b'document'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-site', b'none'), (b'sec-fetch-user', b'?1'), (b'fly-client-ip', b'24.5.172.176'), (b'x-forwarded-for', b'24.5.172.176, 213.188.193.173, 86.109.12.167'), (b'fly-forwarded-proto', b'https'), (b'x-forwarded-proto', b'https'), (b'fly-forwarded-ssl', b'on'), (b'x-forwarded-ssl', b'on'), (b'fly-forwarded-port', b'443'), (b'x-forwarded-port', b'443'), (b'fly-region', b'sjc'), (b'fly-request-id', b'01FS5Y805BX43HM94T8XW610KG'), (b'via', b'2 fly.io'), (b'fly-dispatch-start', b't=1641950740683198;instance=87f188a2'), (b'x-forwarded-host', b'datasette-apache-proxy-demo.fly.dev'), (b'x-forwarded-server', b'localhost'), (b'connection', b'Keep-Alive')], 'http_version': '1.1', 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'', 'raw_path': b'/-/asgi-scope', 'root_path': '', 'scheme': 'https', 'server': ('127.0.0.1', 8001), 'type': 'http', 'url_route': {'kwargs': {}}} ``` The version that works as ` 'raw_path': b'/-/asgi-scope'` - the version that fails has `'raw_path': b'/datasettes/-/asgi-scope'`. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099723916 | |
| https://github.com/simonw/datasette/issues/1590#issuecomment-1010533133 | https://api.github.com/repos/simonw/datasette/issues/1590 | 1010533133 | IC_kwDOBm6k_c48O4MN | 9599 | 2022-01-12T01:19:19Z | 2022-01-12T01:19:19Z | OWNER | Thanks for the steps to reproduce - I have your bug running on my laptop now. I've been mostly testing this stuff using the hosted copy of Datasette here, which doesn't exhibit the bug: https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures?sql=select+sqlite_version%28%29 Something interesting definitely going on here! | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099723916 | |
| https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010462035 | https://api.github.com/repos/simonw/sqlite-utils/issues/381 | 1010462035 | IC_kwDOCGYnMM48Om1T | 9599 | 2022-01-11T23:33:37Z | 2022-01-11T23:33:37Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099584685 | |
| https://github.com/simonw/sqlite-utils/issues/382#issuecomment-1010461844 | https://api.github.com/repos/simonw/sqlite-utils/issues/382 | 1010461844 | IC_kwDOCGYnMM48OmyU | 9599 | 2022-01-11T23:33:14Z | 2022-01-11T23:33:14Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099585611 | |
| https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010441118 | https://api.github.com/repos/simonw/sqlite-utils/issues/381 | 1010441118 | IC_kwDOCGYnMM48Ohue | 9599 | 2022-01-11T22:56:53Z | 2022-01-11T22:57:09Z | OWNER | `sqlite-utils search` has `--limit` already: https://sqlite-utils.datasette.io/en/latest/cli-reference.html#search ``` --limit INTEGER Number of rows to return - defaults to everything ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099584685 | |
| https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010440166 | https://api.github.com/repos/simonw/sqlite-utils/issues/383 | 1010440166 | IC_kwDOCGYnMM48Ohfm | 9599 | 2022-01-11T22:55:05Z | 2022-01-11T22:55:05Z | OWNER | Twitter thread about this: https://twitter.com/simonw/status/1481020195074293761 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099586786 | |
| https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010387223 | https://api.github.com/repos/simonw/sqlite-utils/issues/383 | 1010387223 | IC_kwDOCGYnMM48OUkX | 9599 | 2022-01-11T21:45:32Z | 2022-01-11T21:45:32Z | OWNER | The new page of documentation: https://sqlite-utils.datasette.io/en/latest/cli-reference.html | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099586786 | |
| https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010386802 | https://api.github.com/repos/simonw/sqlite-utils/issues/383 | 1010386802 | IC_kwDOCGYnMM48OUdy | 9599 | 2022-01-11T21:44:53Z | 2022-01-11T21:44:53Z | OWNER | Here's the `cog` code I used: https://github.com/simonw/sqlite-utils/blob/1d44b0cc2784c94aed1bcf350225cd86ee1aa7e5/docs/cli-reference.rst#L11-L76 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099586786 | |
| https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010333511 | https://api.github.com/repos/simonw/sqlite-utils/issues/383 | 1010333511 | IC_kwDOCGYnMM48OHdH | 9599 | 2022-01-11T20:27:08Z | 2022-01-11T20:27:08Z | OWNER | I'll call the new page "CLI reference", for consistency with the API reference page here: https://sqlite-utils.datasette.io/en/stable/reference.html | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1099586786 | |
| https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009548580 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | 1009548580 | IC_kwDOCGYnMM48LH0k | 536941 | 2022-01-11T02:43:34Z | 2022-01-11T02:43:34Z | NONE | thanks so much! always a pleasure to see how you work through these things | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1096558279 | |
| https://github.com/simonw/sqlite-utils/issues/380#issuecomment-1009544785 | https://api.github.com/repos/simonw/sqlite-utils/issues/380 | 1009544785 | IC_kwDOCGYnMM48LG5R | 9599 | 2022-01-11T02:32:56Z | 2022-01-11T02:32:56Z | OWNER | CLI and Python library improvements to help run [ANALYZE](https://www.sqlite.org/lang_analyze.html) after creating indexes or inserting rows, to gain better performance from the SQLite query planner when it runs against indexes. Three new CLI commands: `create-database`, `analyze` and `bulk`. - New `sqlite-utils create-database` command for creating new empty database files. ([#348](https://github.com/simonw/sqlite-utils/issues/348)) - New Python methods for running `ANALYZE` against a database, table or index: `db.analyze()` and `table.analyze()`, see [Optimizing index usage with ANALYZE](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-analyze). ([#366](https://github.com/simonw/sqlite-utils/issues/366)) - New [sqlite-utils analyze command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-analyze) for running `ANALYZE` using the CLI. ([#379](https://github.com/simonw/sqlite-utils/issues/379)) - The `create-index`, `insert` and `update` commands now have a new `--analyze` option for running `ANALYZE` after the command has completed. ([#379](https://github.com/simonw/sqlite-utils/issues/379)) - New [sqlite-utils bulk command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-bulk) which can import records in the same way as `sqlite-utils insert` (from JSON, CSV or TSV) and use them to bulk execute a parametrized SQL query. ([#375](https://github.com/simonw/sqlite-utils/issues/375)) - The CLI tool can now also be run using `python -m sqlite_utils`. ([#368](https://github.com/simonw/sqlite-utils/issues/368)) - Using `--fmt` now implies `--table`, so you don't need to pass both options. ([#374](https://github.com/simonw/sqlite-utils/issues/374)) - The `--convert` function applied to rows can now modify the row in place. ([#371](https://github.com/simonw/sqlite-utils/issues/371)) - The [insert-files command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-insert-files) supports two new columns: `stem` and `suffix`. ([#372](https://github.com/simonw/sqlite-utils/issues/372)) - The `--nl` import option now ignores blank lines in the input. ([#376](https://github.com/simonw/sqlite-utils/issues/376)) - Fixed bug where streaming input to the `insert` command with `--batch-size 1` would appear to only commit after several rows had been ingested, due to unnecessary input buffering. ([#364](https://github.com/simonw/sqlite-utils/issues/364)) | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1098574572 | |
| https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1009536276 | https://api.github.com/repos/simonw/sqlite-utils/issues/375 | 1009536276 | IC_kwDOCGYnMM48LE0U | 9599 | 2022-01-11T02:12:58Z | 2022-01-11T02:12:58Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#executing-sql-in-bulk | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097251014 | |
| https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009534817 | https://api.github.com/repos/simonw/sqlite-utils/issues/377 | 1009534817 | IC_kwDOCGYnMM48LEdh | 9599 | 2022-01-11T02:09:38Z | 2022-01-11T02:09:38Z | OWNER | I tested this like so: ``` % wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' % sqlite-utils create-database test.db % sqlite-utils create-table test.db power_plants url text owner text % sqlite-utils schema test.db CREATE TABLE [power_plants] ( [url] TEXT, [owner] TEXT ); % sqlite-utils bulk test.db 'insert into power_plants (url, owner) values (:url, :owner)' global_power_plant_database.csv --csv [------------------------------------] 0% [###################################-] 99% % sqlite-utils tables --counts test.db -t table count ------------ ------- power_plants 33643 ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097477582 | |
| https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009532125 | https://api.github.com/repos/simonw/sqlite-utils/issues/377 | 1009532125 | IC_kwDOCGYnMM48LDzd | 9599 | 2022-01-11T02:03:35Z | 2022-01-11T02:03:35Z | OWNER | Documentation: https://github.com/simonw/sqlite-utils/blob/f4ea0d32c0543373eefaa9b9f3911eb07549eecb/docs/cli.rst#executing-sql-in-bulk | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097477582 | |
| https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009531863 | https://api.github.com/repos/simonw/sqlite-utils/issues/377 | 1009531863 | IC_kwDOCGYnMM48LDvX | 22429695 | 2022-01-11T02:03:00Z | 2022-01-11T02:03:00Z | NONE | # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#377](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f4ea0d3) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/b6dad08a8389736b7e960cfe9bc719cfc21a98f5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b6dad08) will **decrease** coverage by `0.01%`. > The diff coverage is `96.00%`. [](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #377 +/- ## ========================================== - Coverage 96.52% 96.50% -0.02% ========================================== Files 6 6 Lines 2330 2378 +48 ========================================== + Hits 2249 2295 +46 - Misses 81 83 +2 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/377/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.49% <94.28%> (-0.06%)` | :arrow_down: | | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/377/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.68% <100.00%> (+0.03%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b6dad08...f4ea0d3](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097477582 | |
| https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009521921 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | 1009521921 | IC_kwDOCGYnMM48LBUB | 9599 | 2022-01-11T01:37:53Z | 2022-01-11T01:37:53Z | OWNER | I decided to go with making this opt-in, mainly for consistency with the other places where I added this feature - see: - #379 - #366 You can now run the following: sqlite-utils create-index mydb.db mytable mycolumn --analyze And ``ANALYZE`` will be run on the index once it has been created. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1096558279 | |
| https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865 | https://api.github.com/repos/simonw/sqlite-utils/issues/366 | 1009508865 | IC_kwDOCGYnMM48K-IB | 9599 | 2022-01-11T01:08:51Z | 2022-01-11T01:08:51Z | OWNER | The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1096563265 | |
| https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898 | https://api.github.com/repos/simonw/sqlite-utils/issues/366 | 1009288898 | IC_kwDOCGYnMM48KIbC | 9599 | 2022-01-10T19:54:04Z | 2022-01-10T19:54:04Z | OWNER | Having browsed the API reference I think the methods that would benefit from an `analyze=True` parameter are: - `db.create_index` - `table.insert_all` - `table.upsert_all` - `table.delete_where` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1096563265 | |
| https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009285627 | https://api.github.com/repos/simonw/sqlite-utils/issues/366 | 1009285627 | IC_kwDOCGYnMM48KHn7 | 9599 | 2022-01-10T19:49:19Z | 2022-01-10T19:51:25Z | OWNER | Documentation for those two new methods: https://sqlite-utils.datasette.io/en/latest/python-api.html#optimizing-index-usage-with-analyze | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1096563265 | |
| https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009286373 | https://api.github.com/repos/simonw/sqlite-utils/issues/366 | 1009286373 | IC_kwDOCGYnMM48KHzl | 9599 | 2022-01-10T19:50:22Z | 2022-01-10T19:50:22Z | OWNER | With respect to #365, I'm now thinking that having the ability to say "... and then run ANALYZE" could be useful for a bunch of Python methods. For example: ```python db["dogs"].insert_all(list_of_dogs, analyze=True) db["dogs"].create_index(["name"], analyze=True) ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1096563265 | |
| https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009273525 | https://api.github.com/repos/simonw/sqlite-utils/issues/366 | 1009273525 | IC_kwDOCGYnMM48KEq1 | 9599 | 2022-01-10T19:32:39Z | 2022-01-10T19:32:39Z | OWNER | I'm going to implement the Python library methods based on the prototype: ```diff commit 650f97a08f29a688c530e5f6c9eedc9269ed7bdc Author: Simon Willison <swillison@gmail.com> Date: Sat Jan 8 13:34:01 2022 -0800 Initial prototype of .analyze(), refs #366 diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index dfc4723..1348b4a 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -923,6 +923,13 @@ class Database: "Run a SQLite ``VACUUM`` against the database." self.execute("VACUUM;") + def analyze(self, name=None): + "Run ``ANALYZE`` against the entire database or a named table or index." + sql = "ANALYZE" + if name is not None: + sql += " [{}]".format(name) + self.execute(sql) + class Queryable: def exists(self) -> bool: @@ -2902,6 +2909,10 @@ class Table(Queryable): ) return self + def analyze(self): + "Run ANALYZE against this table" + self.db.analyze(self.name) + def analyze_column( self, column: str, common_limit: int = 10, value_truncate=None, total_rows=None ) -> "ColumnDetails": ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1096563265 | |
| https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1009272446 | https://api.github.com/repos/simonw/sqlite-utils/issues/367 | 1009272446 | IC_kwDOCGYnMM48KEZ- | 9599 | 2022-01-10T19:31:08Z | 2022-01-10T19:31:08Z | OWNER | I'm going to implement this in a separate commit from this PR. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097041471 | |
| https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008557414 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | 1008557414 | IC_kwDOCGYnMM48HV1m | 9599 | 2022-01-10T05:36:19Z | 2022-01-10T05:36:19Z | OWNER | That did the trick. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1095570074 | |
| https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008556706 | https://api.github.com/repos/simonw/sqlite-utils/issues/375 | 1008556706 | IC_kwDOCGYnMM48HVqi | 9599 | 2022-01-10T05:33:41Z | 2022-01-10T05:33:41Z | OWNER | I tested the prototype like this: sqlite-utils blah.db 'create table blah (id integer primary key, name text)' echo 'id,name 1,Cleo 2,Chicken' > blah.csv sqlite-utils bulk blah.db 'insert into blah (id, name) values (:id, :name)' blah.csv --csv | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097251014 | |
| https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008546573 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | 1008546573 | IC_kwDOCGYnMM48HTMN | 9599 | 2022-01-10T05:05:15Z | 2022-01-10T05:05:15Z | OWNER | Bit nasty but it might work: ```python def try_until(expected): tries = 0 while True: rows = list(Database(db_path)["rows"].rows) if rows == expected: return tries += 1 if tries > 10: assert False, "Expected {}, got {}".format(expected, rows) time.sleep(tries * 0.1) try_until([{"name": "Azi"}]) proc.stdin.write(b'{"name": "Suna"}\n') proc.stdin.flush() try_until([{"name": "Azi"}, {"name": "Suna"}]) ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1095570074 | |
| https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008545140 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | 1008545140 | IC_kwDOCGYnMM48HS10 | 9599 | 2022-01-10T05:01:34Z | 2022-01-10T05:01:34Z | OWNER | Urgh, tests are still failing intermittently - for example: ``` time.sleep(0.4) > assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}] E AssertionError: assert [] == [{'name': 'Azi'}] E Right contains one more item: {'name': 'Azi'} E Full diff: E - [{'name': 'Azi'}] E + [] ``` I'm going to change this code to keep on trying up to 10 seconds - that should get the tests to pass faster on most machines. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1095570074 | |
| https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008537194 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | 1008537194 | IC_kwDOCGYnMM48HQ5q | 9599 | 2022-01-10T04:29:53Z | 2022-01-10T04:31:29Z | OWNER | After a bunch of debugging with `print()` statements it's clear that the problem isn't with when things are committed or the size of the batches - it's that the data sent to standard input is all being processed in one go, not a line at a time. I think that's because it is being buffered by this: https://github.com/simonw/sqlite-utils/blob/d2a79d200f9071a86027365fa2a576865b71064f/sqlite_utils/cli.py#L759-L770 The buffering is there so that we can sniff the first few bytes to detect if it's a CSV file - added in 99ff0a288c08ec2071139c6031eb880fa9c95310 for #230. So maybe for non-CSV inputs we should disable buffering? | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1095570074 | |
| https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008526736 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | 1008526736 | IC_kwDOCGYnMM48HOWQ | 9599 | 2022-01-10T04:07:29Z | 2022-01-10T04:07:29Z | OWNER | I think this test is right: ```python def test_insert_streaming_batch_size_1(db_path): # https://github.com/simonw/sqlite-utils/issues/364 # Streaming with --batch-size 1 should commit on each record # Can't use CliRunner().invoke() here bacuse we need to # run assertions in between writing to process stdin proc = subprocess.Popen( [ sys.executable, "-m", "sqlite_utils", "insert", db_path, "rows", "-", "--nl", "--batch-size", "1", ], stdin=subprocess.PIPE, ) proc.stdin.write(b'{"name": "Azi"}') proc.stdin.flush() assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}] proc.stdin.write(b'{"name": "Suna"}') proc.stdin.flush() assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}, {"name": "Suna"}] proc.stdin.close() proc.wait() ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1095570074 | |
| https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008383293 | https://api.github.com/repos/simonw/sqlite-utils/issues/348 | 1008383293 | IC_kwDOCGYnMM48GrU9 | 9599 | 2022-01-09T20:38:17Z | 2022-01-09T20:38:17Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#creating-an-empty-database | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1067771698 | |
| https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008367607 | https://api.github.com/repos/simonw/sqlite-utils/issues/348 | 1008367607 | IC_kwDOCGYnMM48Gnf3 | 9599 | 2022-01-09T20:22:43Z | 2022-01-09T20:22:43Z | OWNER | I'm not going to implement `--page-size` unless someone specifically requests it - I don't like having features that I've never needed to use myself. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1067771698 | |
| https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008364701 | https://api.github.com/repos/simonw/sqlite-utils/issues/371 | 1008364701 | IC_kwDOCGYnMM48Gmyd | 9599 | 2022-01-09T20:04:35Z | 2022-01-09T20:04:35Z | OWNER | The previous code for highlighting errors in syntax (which was already a bit confused thanks to the added `return`, see https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684 - isn't compatible with this approach at all. I'm going to ditch it and just show a generic `Error: Could not compile code` message. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097128334 | |
| https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008354207 | https://api.github.com/repos/simonw/sqlite-utils/issues/371 | 1008354207 | IC_kwDOCGYnMM48GkOf | 9599 | 2022-01-09T18:54:54Z | 2022-01-09T18:54:54Z | OWNER | This seems to work: ```python def _compile_code(code, imports, variable="value"): locals = {} globals = {"r": recipes, "recipes": recipes} # If user defined a convert() function, return that try: exec(code, globals, locals) return locals["convert"] except (AttributeError, SyntaxError, NameError, KeyError, TypeError): pass # Try compiling their code as a function instead body_variants = [code] # If single line and no 'return', try adding the return if "\n" not in code and not code.strip().startswith("return "): body_variants.insert(0, "return {}".format(code)) for variant in body_variants: new_code = ["def fn({}):".format(variable)] for line in variant.split("\n"): new_code.append(" {}".format(line)) try: code_o = compile("\n".join(new_code), "<string>", "exec") break except SyntaxError: # Try another variant, e.g. for 'return row["column"] = 1' continue for import_ in imports: globals[import_.split(".")[0]] = __import__(import_) exec(code_o, globals, locals) return locals["fn"] ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097128334 | |
| https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008348032 | https://api.github.com/repos/simonw/sqlite-utils/issues/371 | 1008348032 | IC_kwDOCGYnMM48GiuA | 9599 | 2022-01-09T18:14:02Z | 2022-01-09T18:14:02Z | OWNER | Here's the code in question: https://github.com/simonw/sqlite-utils/blob/b8c134059e89f0fa040b84fb7d0bda25b9a52759/sqlite_utils/utils.py#L288-L299 | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097128334 | |
| https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008347768 | https://api.github.com/repos/simonw/sqlite-utils/issues/371 | 1008347768 | IC_kwDOCGYnMM48Gip4 | 9599 | 2022-01-09T18:12:30Z | 2022-01-09T18:12:30Z | OWNER | Tried this test: ```python result = CliRunner().invoke( cli.cli, [ "insert", db_path, "rows", "-", "--convert", 'row["is_chicken"] = True', ], input='{"name": "Azi"}', ) ``` And got this error: > `E + where 1 = <Result SyntaxError('invalid syntax', ('<string>', 2, 30, ' return row["is_chicken"] = True\n'))>.exit_code` The code snippet compilation isn't currently compatible with this. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097128334 | |
| https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346841 | https://api.github.com/repos/simonw/sqlite-utils/issues/374 | 1008346841 | IC_kwDOCGYnMM48GibZ | 9599 | 2022-01-09T18:06:50Z | 2022-01-09T18:06:50Z | OWNER | In addition to a unit test I manually tested all of the above, e.g. ``` % sqlite-utils indexes global-power-plants.db sqlite_master --fmt rst ======= ============ ======= ===== ====== ====== ====== ===== table index_name seqno cid name desc coll key ======= ============ ======= ===== ====== ====== ====== ===== ======= ============ ======= ===== ====== ====== ====== ===== ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097135860 | |
| https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346338 | https://api.github.com/repos/simonw/sqlite-utils/issues/374 | 1008346338 | IC_kwDOCGYnMM48GiTi | 9599 | 2022-01-09T18:03:22Z | 2022-01-09T18:03:22Z | OWNER | Commands that support `--fmt` (via the `@output_options` decorator) are: - `tables` - `views` - `query` - `memory` - `search` - `rows` - `triggers` - `indexes` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097135860 | |
| https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008345267 | https://api.github.com/repos/simonw/sqlite-utils/issues/374 | 1008345267 | IC_kwDOCGYnMM48GiCz | 9599 | 2022-01-09T17:56:37Z | 2022-01-09T17:56:37Z | OWNER | Better: ```python if fmt: table = True ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097135860 | |
| https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344980 | https://api.github.com/repos/simonw/sqlite-utils/issues/373 | 1008344980 | IC_kwDOCGYnMM48Gh-U | 9599 | 2022-01-09T17:54:53Z | 2022-01-09T17:54:53Z | OWNER | Updated TIL: https://til.simonwillison.net/python/cog-to-update-help-in-readme#user-content-cog-for-restructuredtext | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097135732 | |
| https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344525 | https://api.github.com/repos/simonw/sqlite-utils/issues/373 | 1008344525 | IC_kwDOCGYnMM48Gh3N | 9599 | 2022-01-09T17:52:22Z | 2022-01-09T17:52:22Z | OWNER | Updated docs: https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097135732 | |
| https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008341078 | https://api.github.com/repos/simonw/sqlite-utils/issues/373 | 1008341078 | IC_kwDOCGYnMM48GhBW | 9599 | 2022-01-09T17:31:12Z | 2022-01-09T17:31:12Z | OWNER | Found an example of using `cog` in a rST file here: https://github.com/nedbat/coveragepy/blob/f3238eea7e403d13a217b30579b1a1c2cbff62e3/doc/dbschema.rst#L21 ``` .. [[[cog from coverage.sqldata import SCHEMA_VERSION print(".. code::") print() print(f" SCHEMA_VERSION = {SCHEMA_VERSION}") print() .. ]]] .. code:: SCHEMA_VERSION = 7 .. [[[end]]] ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097135732 | |
| https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008338186 | https://api.github.com/repos/simonw/sqlite-utils/issues/375 | 1008338186 | IC_kwDOCGYnMM48GgUK | 9599 | 2022-01-09T17:13:33Z | 2022-01-09T17:13:54Z | OWNER | cat blah.csv | sqlite-utils bulk blah.db - \ "insert into blah (:foo, :bar)" --csv | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097251014 | |
| https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008275546 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | 1008275546 | IC_kwDOCGYnMM48GRBa | 536941 | 2022-01-09T11:01:15Z | 2022-01-09T13:37:51Z | NONE | i don’t want to be such a partisan for analyze, but the query planner deciding *not* to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice. <s>the original poster in that stack overflow doesn’t say there’s a performance regression </s> | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1096558279 | |
| https://github.com/simonw/datasette/pull/1574#issuecomment-1008279307 | https://api.github.com/repos/simonw/datasette/issues/1574 | 1008279307 | IC_kwDOBm6k_c48GR8L | 33631 | 2022-01-09T11:26:06Z | 2022-01-09T11:26:06Z | NONE | @fgregg my thinking was backwards compatibility. I don't know what people do to their builds, I just wanted a smaller image for my use case. @simonw any chance to take a look at this? If there is no interest, feel free to close the PR | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1084193403 | |
| https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008252732 | https://api.github.com/repos/simonw/sqlite-utils/issues/374 | 1008252732 | IC_kwDOCGYnMM48GLc8 | 9599 | 2022-01-09T08:25:30Z | 2022-01-09T08:25:30Z | OWNER | Need to change `if table:` to `if table or fmt:` in a few places. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097135860 | |
| https://github.com/simonw/sqlite-utils/issues/372#issuecomment-1008247370 | https://api.github.com/repos/simonw/sqlite-utils/issues/372 | 1008247370 | IC_kwDOCGYnMM48GKJK | 9599 | 2022-01-09T07:51:18Z | 2022-01-09T07:51:18Z | OWNER | Pathlib says the stem of that would be `dogs.and.cats.jpg` - best stick with that for consistency. https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.suffix It calls the last bit `suffix` - maybe I should use that instead of `ext`. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097129710 | |
| https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246366 | https://api.github.com/repos/simonw/sqlite-utils/issues/371 | 1008246366 | IC_kwDOCGYnMM48GJ5e | 9599 | 2022-01-09T07:42:14Z | 2022-01-09T07:42:14Z | OWNER | Also need to update relevant docs for that example. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097128334 | |
| https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246239 | https://api.github.com/repos/simonw/sqlite-utils/issues/371 | 1008246239 | IC_kwDOCGYnMM48GJ3f | 9599 | 2022-01-09T07:41:24Z | 2022-01-09T07:41:24Z | OWNER | Might be a case of modifying this line: https://github.com/simonw/sqlite-utils/blob/e0c476bc380744680c8b7675c24fb0e9f5ec6dcd/sqlite_utils/cli.py#L828 To: ```python docs = (fn(doc) or doc for doc in docs) ``` | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1097128334 | |
| https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008234293 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | 1008234293 | IC_kwDOCGYnMM48GG81 | 9599 | 2022-01-09T05:37:02Z | 2022-01-09T05:37:02Z | OWNER | Calling `p.stdin.close()` and then `p.wait()` terminates the subprocess. | {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
1095570074 |