github
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
642572841 | MDU6SXNzdWU2NDI1NzI4NDE= | 859 | Database page loads too slowly with many large tables (due to table counts) | 3243482 | open | 0 | 21 | 2020-06-21T14:23:17Z | 2021-08-25T21:59:55Z | CONTRIBUTOR | Hey, I have a database that I save in HTML from couple of web scrapers. There are around 200k+, 50+ rows in a couple of tables, with sqlite file weighing around 600MB. The app runs on a VPS with 2 core CPU, 4GB RAM and refreshing database page regularly takes more than 10 seconds. I was suspecting that counting tables was the culprit, but manually running `select count(*) from table_name` for the largest table finishes under a second. I've looked at the source code. There's a check for index page for mutable databases larger than 100MB https://github.com/simonw/datasette/blob/799c5d53570d773203527f19530cf772dc2eeb24/datasette/views/index.py#L15 but this check is not performed for database page. I've manually crippled `Database::table_counts` method ```py async def table_counts(self, limit=10): if not self.is_mutable and self.cached_table_counts is not None: return self.cached_table_counts # Try to get counts for each table, $limit timeout for each count counts = {} for table in await self.table_names(): try: # table_count = ( # await self.execute( # "select count(*) from [{}]".format(table), # custom_time_limit=limit, # ) # ).rows[0][0] counts[table] = 10 # table_count # In some cases I saw "SQL Logic Error" here in addition to # QueryInterrupted - so we catch that too: except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError): counts[table] = None if not self.is_mutable: self.cached_table_counts = counts return counts ``` now the page loads in <100ms. Is it possible to apply size check on database page too? <details> <summary> /-/versions output </summary> <pre> { "python": { "version": "3.8.0", "full": "3.8.0 (default, Oct 28 2019, 16:14:01) \n[GCC 8.3.0]" }, "datasette": { "version": "0.44" }, "asgi": "3.… | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/859/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
947640902 | MDExOlB1bGxSZXF1ZXN0NjkyNTk2MDA2 | 1400 | Bump black from 21.6b0 to 21.7b0 | 49699333 | closed | 0 | 1 | 2021-07-19T13:13:41Z | 2021-08-25T01:29:56Z | 2021-08-25T01:29:55Z | CONTRIBUTOR | simonw/datasette/pulls/1400 | Bumps [black](https://github.com/psf/black) from 21.6b0 to 21.7b0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/psf/black/releases">black's releases</a>.</em></p> <blockquote> <h2>21.7b0</h2> <h3><em>Black</em></h3> <ul> <li>Configuration files using TOML features higher than spec v0.5.0 are now supported (<a href="https://github-redirect.dependabot.com/psf/black/issues/2301">#2301</a>)</li> <li>Add primer support and test for code piped into black via STDIN (<a href="https://github-redirect.dependabot.com/psf/black/issues/2315">#2315</a>)</li> <li>Fix internal error when <code>FORCE_OPTIONAL_PARENTHESES</code> feature is enabled (<a href="https://github-redirect.dependabot.com/psf/black/issues/2332">#2332</a>)</li> <li>Accept empty stdin (<a href="https://github-redirect.dependabot.com/psf/black/issues/2346">#2346</a>)</li> <li>Provide a more useful error when parsing fails during AST safety checks (<a href="https://github-redirect.dependabot.com/psf/black/issues/2304">#2304</a>)</li> </ul> <h3>Docker</h3> <ul> <li>Add new <code>latest_release</code> tag automation to follow latest black release on docker images (<a href="https://github-redirect.dependabot.com/psf/black/issues/2374">#2374</a>)</li> </ul> <h3>Integrations</h3> <ul> <li>The vim plugin now searches upwards from the directory containing the current buffer instead of the current working directory for pyproject.toml. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1871">#1871</a>)</li> <li>The vim plugin now reads the correct string normalization option in pyproject.toml (<a href="https://github-redirect.dependabot.com/psf/black/issues/1869">#1869</a>)</li> <li>The vim plugin no longer crashes Black when there's boolean values in pyproject.toml (<a href="https://github-redirect.dependabot.com/psf/black/issues/1869">#1869</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/psf/black/blob/main/CHANGES.md">black's ch… | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/1400/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
970386262 | MDExOlB1bGxSZXF1ZXN0NzEyMzQ2MTk5 | 1433 | Update trustme requirement from <0.9,>=0.7 to >=0.7,<0.10 | 49699333 | closed | 0 | 1 | 2021-08-13T13:10:24Z | 2021-08-25T01:29:27Z | 2021-08-25T01:29:26Z | CONTRIBUTOR | simonw/datasette/pulls/1433 | Updates the requirements on [trustme](https://github.com/python-trio/trustme) to permit the latest version. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/python-trio/trustme/commit/8fc5bf9c5afaf40843287e6eccca344713eddb50"><code>8fc5bf9</code></a> Bump version to 0.9.0</li> <li><a href="https://github.com/python-trio/trustme/commit/913e21dd94fc16a8b41742cc60338dcf638ae792"><code>913e21d</code></a> Bump types-cryptography from 3.3.3 to 3.3.5 (<a href="https://github-redirect.dependabot.com/python-trio/trustme/issues/342">#342</a>)</li> <li><a href="https://github.com/python-trio/trustme/commit/c66709d48bffc0d3a70d260313b752037ea71f8d"><code>c66709d</code></a> Bump types-pyopenssl from 20.0.4 to 20.0.5 (<a href="https://github-redirect.dependabot.com/python-trio/trustme/issues/343">#343</a>)</li> <li><a href="https://github.com/python-trio/trustme/commit/5131f79e9b7d73d993f901d8e48e6d744b0606ef"><code>5131f79</code></a> Add type annotations (<a href="https://github-redirect.dependabot.com/python-trio/trustme/issues/341">#341</a>)</li> <li><a href="https://github.com/python-trio/trustme/commit/a411dad9614f3fc2e746324d9a226e4947d55835"><code>a411dad</code></a> Bump charset-normalizer from 2.0.3 to 2.0.4 (<a href="https://github-redirect.dependabot.com/python-trio/trustme/issues/340">#340</a>)</li> <li><a href="https://github.com/python-trio/trustme/commit/be5ec8a3892ea218291ac93da8b382229b07f044"><code>be5ec8a</code></a> Bump sphinx from 4.1.1 to 4.1.2</li> <li><a href="https://github.com/python-trio/trustme/commit/d3b8865576a82189e06c735c64685fbad7ad02d0"><code>d3b8865</code></a> Bump charset-normalizer from 2.0.2 to 2.0.3</li> <li><a href="https://github.com/python-trio/trustme/commit/4503bef325bf0b23f4a0289654ee4c72d0c79fd1"><code>4503bef</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/python-trio/trustme/issues/334">#334</a> from python-trio/dependabot/pip/charset-normalizer-2.0.2</li> <li><a href="https://github.com/python-trio/trustme/commit/ce8099d9e2b5… | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/1433/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
978614898 | MDExOlB1bGxSZXF1ZXN0NzE5MTc1NTkz | 1447 | Remove underscore from search mode parameter name | 127565 | closed | 0 | 1 | 2021-08-25T01:28:04Z | 2021-08-25T01:28:58Z | 2021-08-25T01:28:58Z | CONTRIBUTOR | simonw/datasette/pulls/1447 | The fulltext search documentation refers to the parameter as `searchmode` but the `metadata.json` example uses `search_mode`. The latter doesn't actually seem to work. | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/1447/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 |