github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/datasette/issues/283#issuecomment-781077127 | https://api.github.com/repos/simonw/datasette/issues/283 | 781077127 | MDEyOklzc3VlQ29tbWVudDc4MTA3NzEyNw== | 9599 | 2021-02-18T05:56:30Z | 2021-02-18T05:57:34Z | OWNER | I'm going to to try prototyping the `--crossdb` option that causes `/_memory` to connect to all databases as a starting point and see how well that works. | { "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0 } |
325958506 | |
https://github.com/simonw/datasette/issues/1226#issuecomment-781530157 | https://api.github.com/repos/simonw/datasette/issues/1226 | 781530157 | MDEyOklzc3VlQ29tbWVudDc4MTUzMDE1Nw== | 9599 | 2021-02-18T18:00:15Z | 2021-02-18T18:00:15Z | OWNER | I can use `click.IntRange(min=None, max=None)` for this. https://click.palletsprojects.com/en/7.x/options/#ranges - inclusive on both edges. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808843401 | |
https://github.com/simonw/datasette/issues/1226#issuecomment-781546512 | https://api.github.com/repos/simonw/datasette/issues/1226 | 781546512 | MDEyOklzc3VlQ29tbWVudDc4MTU0NjUxMg== | 9599 | 2021-02-18T18:26:19Z | 2021-02-18T18:26:19Z | OWNER | This broke CI: https://github.com/simonw/datasette/runs/1929355965?check_suite_focus=true | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
808843401 | |
https://github.com/simonw/datasette/issues/1231#issuecomment-781560865 | https://api.github.com/repos/simonw/datasette/issues/1231 | 781560865 | MDEyOklzc3VlQ29tbWVudDc4MTU2MDg2NQ== | 9599 | 2021-02-18T18:50:38Z | 2021-02-18T18:50:38Z | OWNER | I started trying to use locks to resolve this but I've not figured out the right way to do that yet - here's my first experiment: ```diff diff --git a/datasette/app.py b/datasette/app.py index 9e15a16..1681c9d 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -217,6 +217,7 @@ class Datasette: self.inspect_data = inspect_data self.immutables = set(immutables or []) self.databases = collections.OrderedDict() + self._refresh_schemas_lock = threading.Lock() if memory or not self.files: self.add_database(Database(self, is_memory=True), name="_memory") # memory_name is a random string so that each Datasette instance gets its own @@ -324,6 +325,13 @@ class Datasette: self.client = DatasetteClient(self) async def refresh_schemas(self): + return + if self._refresh_schemas_lock.locked(): + return + with self._refresh_schemas_lock: + await self._refresh_schemas() + + async def _refresh_schemas(self): internal_db = self.databases["_internal"] if not self.internal_db_created: await init_internal_db(internal_db) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
811367257 | |
https://github.com/simonw/datasette/issues/1231#issuecomment-781560989 | https://api.github.com/repos/simonw/datasette/issues/1231 | 781560989 | MDEyOklzc3VlQ29tbWVudDc4MTU2MDk4OQ== | 9599 | 2021-02-18T18:50:53Z | 2021-02-18T18:50:53Z | OWNER | Ideally I'd figure out a way to replicate this error in a concurrent unit test. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
811367257 | |
https://github.com/simonw/datasette/issues/283#issuecomment-781573676 | https://api.github.com/repos/simonw/datasette/issues/283 | 781573676 | MDEyOklzc3VlQ29tbWVudDc4MTU3MzY3Ng== | 9599 | 2021-02-18T19:13:30Z | 2021-02-18T19:13:30Z | OWNER | It turns out SQLite defaults to a maximum of 10 attached databases. This can be increased using a compile-time constant, but even with that it cannot be more than 62: https://stackoverflow.com/questions/9845448/attach-limit-10 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
325958506 | |
https://github.com/simonw/datasette/issues/283#issuecomment-781574786 | https://api.github.com/repos/simonw/datasette/issues/283 | 781574786 | MDEyOklzc3VlQ29tbWVudDc4MTU3NDc4Ng== | 9599 | 2021-02-18T19:15:37Z | 2021-02-18T19:15:37Z | OWNER | `select * from pragma_database_list();` is useful - shows all attached databases for the current connection. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
325958506 | |
https://github.com/simonw/datasette/issues/283#issuecomment-781591015 | https://api.github.com/repos/simonw/datasette/issues/283 | 781591015 | MDEyOklzc3VlQ29tbWVudDc4MTU5MTAxNQ== | 9599 | 2021-02-18T19:44:02Z | 2021-02-18T19:44:02Z | OWNER | For the moment I'm going to hard-code a `SQLITE_LIMIT_ATTACHED=10` constant and only attach the first 10 databases to the `_memory` connection. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
325958506 | |
https://github.com/simonw/datasette/issues/283#issuecomment-781593169 | https://api.github.com/repos/simonw/datasette/issues/283 | 781593169 | MDEyOklzc3VlQ29tbWVudDc4MTU5MzE2OQ== | 9599 | 2021-02-18T19:47:34Z | 2021-02-18T19:47:34Z | OWNER | I have a working version now, moving development to a pull request. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
325958506 | |
https://github.com/simonw/datasette/pull/1232#issuecomment-781594632 | https://api.github.com/repos/simonw/datasette/issues/1232 | 781594632 | MDEyOklzc3VlQ29tbWVudDc4MTU5NDYzMg== | 9599 | 2021-02-18T19:50:21Z | 2021-02-18T19:50:21Z | OWNER | It would be neat if the `/_memory` page showed a list of attached databases, to indicate that the `--crossdb` option is working and give people links to click to start running queries. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
811407131 | |
https://github.com/simonw/datasette/pull/1232#issuecomment-781598585 | https://api.github.com/repos/simonw/datasette/issues/1232 | 781598585 | MDEyOklzc3VlQ29tbWVudDc4MTU5ODU4NQ== | 9599 | 2021-02-18T19:57:30Z | 2021-02-18T19:57:30Z | OWNER | It would also be neat if https://latest.datasette.io/ had multiple databases attached in order to demonstrate this feature. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
811407131 | |
https://github.com/simonw/datasette/pull/1232#issuecomment-781629841 | https://api.github.com/repos/simonw/datasette/issues/1232 | 781629841 | MDEyOklzc3VlQ29tbWVudDc4MTYyOTg0MQ== | 9599 | 2021-02-18T20:57:23Z | 2021-02-18T20:57:23Z | OWNER | The new warning looks like this: <img width="514" alt="datasette_—_pipenv_shell_▸_Python_—_182×66" src="https://user-images.githubusercontent.com/9599/108420562-d41db480-71e8-11eb-87e3-1cddd65627c2.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
811407131 | |
https://github.com/simonw/datasette/pull/1232#issuecomment-781634819 | https://api.github.com/repos/simonw/datasette/issues/1232 | 781634819 | MDEyOklzc3VlQ29tbWVudDc4MTYzNDgxOQ== | 9599 | 2021-02-18T21:06:43Z | 2021-02-18T21:06:43Z | OWNER | I'll document this option on https://docs.datasette.io/en/stable/sql_queries.html | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
811407131 | |
https://github.com/simonw/datasette/issues/1233#issuecomment-781636590 | https://api.github.com/repos/simonw/datasette/issues/1233 | 781636590 | MDEyOklzc3VlQ29tbWVudDc4MTYzNjU5MA== | 9599 | 2021-02-18T21:10:08Z | 2021-02-18T21:10:08Z | OWNER | I think the bug is here: https://github.com/simonw/datasette/blob/640ac7071b73111ba4423812cd683756e0e1936b/datasette/utils/__init__.py#L349-L373 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
811458446 | |
https://github.com/simonw/datasette/pull/1232#issuecomment-781637292 | https://api.github.com/repos/simonw/datasette/issues/1232 | 781637292 | MDEyOklzc3VlQ29tbWVudDc4MTYzNzI5Mg== | 9599 | 2021-02-18T21:11:31Z | 2021-02-18T21:11:31Z | OWNER | Due to bug #1233 I'm going to publish the additional database as `extra_database.db` rather than `extra database.db` as it is used in the tests. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
811407131 | |
https://github.com/simonw/datasette/pull/1232#issuecomment-781641728 | https://api.github.com/repos/simonw/datasette/issues/1232 | 781641728 | MDEyOklzc3VlQ29tbWVudDc4MTY0MTcyOA== | 9599 | 2021-02-18T21:19:34Z | 2021-02-18T21:19:34Z | OWNER | I tested the demo deployment like this: ``` datasette publish cloudrun fixtures.db extra_database.db \ -m fixtures.json \ --plugins-dir=plugins \ --branch=crossdb \ --extra-options="--setting template_debug 1 --crossdb" \ --install=pysqlite3-binary \ --service=datasette-latest-crossdb ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
811407131 | |
https://github.com/simonw/datasette/pull/1232#issuecomment-781651283 | https://api.github.com/repos/simonw/datasette/issues/1232 | 781651283 | MDEyOklzc3VlQ29tbWVudDc4MTY1MTI4Mw== | 9599 | 2021-02-18T21:37:55Z | 2021-02-18T21:37:55Z | OWNER | UI listing the attached tables: <img width="888" alt="_memory" src="https://user-images.githubusercontent.com/9599/108424809-8015ce80-71ee-11eb-8752-03c459f89320.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
811407131 | |
https://github.com/simonw/datasette/issues/283#issuecomment-781665560 | https://api.github.com/repos/simonw/datasette/issues/283 | 781665560 | MDEyOklzc3VlQ29tbWVudDc4MTY2NTU2MA== | 9599 | 2021-02-18T22:06:14Z | 2021-02-18T22:06:14Z | OWNER | The implementation in #1232 is ready to land. It's the simplest-thing-that-could-possibly-work: you can run `datasette one.db two.db three.db --crossdb` and then use the `/_memory` page to run joins across tables from multiple databases. It only works on the first 10 databases that were passed to the command-line. This means that if you have a Datasette instance with hundreds of attached databases (see [Datasette Library](https://github.com/simonw/datasette/issues/417)) this won't be particularly useful for you. So... a better, future version of this feature would be one that lets you join across databases on command - maybe by hitting `/_memory?attach=db1&attach=db2` to get a special connection. Also worth noting: plugins that implement the [prepare_connection()](https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette) hook can attach additional databases - so if you need better, customized support for this one way to handle that would be with a custom plugin. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
325958506 | |
https://github.com/simonw/datasette/issues/283#issuecomment-781670827 | https://api.github.com/repos/simonw/datasette/issues/283 | 781670827 | MDEyOklzc3VlQ29tbWVudDc4MTY3MDgyNw== | 9599 | 2021-02-18T22:16:46Z | 2021-02-18T22:16:46Z | OWNER | Demo is now live here: https://latest.datasette.io/_memory The documentation is at https://docs.datasette.io/en/latest/sql_queries.html#cross-database-queries - it links to this example query: https://latest.datasette.io/_memory?sql=select%0D%0A++%27fixtures%27+as+database%2C+*%0D%0Afrom%0D%0A++%5Bfixtures%5D.sqlite_master%0D%0Aunion%0D%0Aselect%0D%0A++%27extra_database%27+as+database%2C+*%0D%0Afrom%0D%0A++%5Bextra_database%5D.sqlite_master | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
325958506 | |
https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-781451701 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | 781451701 | MDEyOklzc3VlQ29tbWVudDc4MTQ1MTcwMQ== | 203343 | 2021-02-18T16:06:21Z | 2021-02-18T16:06:21Z | NONE | Awesome! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
778380836 | |
https://github.com/simonw/datasette/issues/1230#issuecomment-781330466 | https://api.github.com/repos/simonw/datasette/issues/1230 | 781330466 | MDEyOklzc3VlQ29tbWVudDc4MTMzMDQ2Ng== | 7107523 | 2021-02-18T13:06:22Z | 2021-02-18T15:22:15Z | NONE | [Edit] Oh, I just saw the "Load all" button under the cluster map as well as the [setting to alter the max number or results](https://docs.datasette.io/en/stable/settings.html#max-returned-rows). So I guess this issue only is about the Vega charts. <details> Note that datasette-cluster-map also seems to be limited to 998 displayed points: ![ss-2021-02-18_140548](https://user-images.githubusercontent.com/7107523/108361225-15fb2a80-71ea-11eb-9a19-d885e8513f55.png) </details> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
811054000 | |
https://github.com/simonw/datasette/issues/283#issuecomment-780991910 | https://api.github.com/repos/simonw/datasette/issues/283 | 780991910 | MDEyOklzc3VlQ29tbWVudDc4MDk5MTkxMA== | 9308268 | 2021-02-18T02:13:56Z | 2021-02-18T02:13:56Z | NONE | I was going ask you about this issue when we talk during your office-hours schedule this Friday, but was there any support ever added for doing this cross-database joining? I have a use-case where could be pretty neat to do analysis using this tool on time-specific databases from snapshots https://ilsweb.cincinnatilibrary.org/collection-analysis/ ![image](https://user-images.githubusercontent.com/9308268/108294883-ba3a8e00-7164-11eb-9206-fcd5a8cdd883.png) and thanks again for such an amazing tool! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
325958506 | |
https://github.com/simonw/datasette/pull/1232#issuecomment-781599929 | https://api.github.com/repos/simonw/datasette/issues/1232 | 781599929 | MDEyOklzc3VlQ29tbWVudDc4MTU5OTkyOQ== | 22429695 | 2021-02-18T19:59:54Z | 2021-02-18T22:06:42Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=h1) Report > Merging [#1232](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=desc) (8876499) into [main](https://codecov.io/gh/simonw/datasette/commit/4df548e7668b5b21d64a267964951e67894f4712?el=desc) (4df548e) will **increase** coverage by `0.03%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1232/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1232 +/- ## ========================================== + Coverage 91.42% 91.46% +0.03% ========================================== Files 32 32 Lines 3955 3970 +15 ========================================== + Hits 3616 3631 +15 Misses 339 339 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1232/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.68% <100.00%> (+0.06%)` | :arrow_up: | | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1232/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `76.62% <100.00%> (+0.36%)` | :arrow_up: | | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1232/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.19% <100.00%> (+0.01%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=footer). Last update [4df5… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
811407131 |