The other reason I didn't implement this in the first place is that adding offset/limit to a custom query (as opposed to a view) requires modifying the existing SQL - what if that SQL already has its own offset/limit clause?

It looks like I can solve that using a nested query: sql select * from ( select * from compound_three_primary_keys limit 1000 ) limit 10 offset 100 https://latest.datasette.io/fixtures?sql=select++from+%28%0D%0A++select++from+compound_three_primary_keys+limit+1000%0D%0A%29+limit+10+offset+100

So I can wrap any user-provided SQL query in an outer offset/limit and implement pagination that way.

Views already use offset/limit pagination so actually I may be over-thinking this.

Maybe the right thing to do here is to have the feature enabled by default, since it will work for the VAST majority of queries - the only ones that might cause problems are complex queries across millions of rows. It can continue to use aggressive internal time limits so if someone DOES trigger something expensive they'll get an error.

I can allow users to disable the feature with a config setting, or increase the time limit if they need to.

Downgrading this from a medium to a small since it's much less effort to enable the existing pagination method for this type of query.

Closing this because it doesn't really fit the new model of inspect (though we should discuss in #465 how to further evolve this feature) and because as-of #272 we no longer use Sanic - though #520 will implement the equivalent of prepare_sanic against ASGI.

https://datasette.readthedocs.io/en/latest/full_text_search.html#adding-full-text-search-to-a-sqlite-table

It's mentioned here at the moment, but I'm going to expand that:

https://github.com/simonw/datasette/blob/34e292d24dc2b8376236472bec3cce1c556ddfe5/docs/full_text_search.rst#L74

I'm suspicious of the wildcard. Does it work if you do the following?

ExecStart=/home/chris/Env/datasette/bin/datasette serve -h 0.0.0.0 /home/chris/digital-library/databases/actual-database.db --cors --metadata /home/chris/digital-library/metadata.json

If that does work then it means the ExecStart line doesn't support bash wildcard expansion. You'll need to create a separate script like this:

```

!/bin/bash

/home/chris/Env/datasette/bin/datasette serve -h 0.0.0.0 /home/chris/digital-library/databases/*.db --cors --metadata /home/chris/digital-library/metadata.json ```

Then save that as /home/chris/digital-library/run-datasette.sh and try this:

ExecStart=/home/chris/digital-library/run-datasette.sh

This is currently a deliberate feature decision.

The problem is that the streaming CSV feature relies on Datasette's automated efficient pagination under the hood. When you stream a CSV you're actually causing Datasette to paginate through the full set of "pages" under the hood, streaming each page out as a new chunk of CSV rows.

This mechanism only works if the next_url has been generated for the page. Currently the next_url is available for table views (where it uses the primary key or the sort column) and for views, but it's not set for canned queries because I can't be certain they can be efficiently paginated.

Offset/limit pagination for canned queries would be a pretty nasty performance hit, because each subsequent page would require even more time for SQLite to scroll through to the specified offset.

This does seem like it's worth fixing though: pulling every row for a canned queries would definitely be useful. The problem is that the pagination trick used elsewhere isn't right for canned queries - instead I would need to keep the database cursor open until ALL rows had been fetched. Figuring out how to do that efficiently within an asyncio managed thread pool may take some thought.

Maybe this feature ends up as something which is turned off by default (due to the risk of it causing uptime problems for public sites) but that users working on their own private environments can turn on?

I just released csvs-to-sqlite 0.9.1 with this bug fix: https://github.com/simonw/csvs-to-sqlite/releases/tag/0.9.1

Closing in favour of that bug in the csvs-to-sqlite repo.

The select rank, * feature is only available with FTS5 - it won't work with FTS4. So my best guess is that csvs-to-sqlite is setting up FTS with FTS4 when you want FTS5.

... I tested on my own machine and that is indeed what's happening! And in fact it looks like it's a known bug - I should fix that! https://github.com/simonw/csvs-to-sqlite/issues/41

The Sanic stuff I import at the moment is:

• [x] from sanic.views import HTTPMethodView
• [x] from sanic.request import Request https://github.com/huge-success/sanic/blob/0.7.0/sanic/request.py#L44
• [x] from sanic.exceptions import NotFound
• [x] from sanic import response - used for response.text(""), response.redirect(path), response.HTTPResponse and response.html() https://github.com/huge-success/sanic/blob/0.7.0/sanic/response.py
• [x] from sanic.request import RequestParameters https://github.com/huge-success/sanic/blob/0.7.0/sanic/request.py#L30

Same error when I used the full path.

On Sun, Jun 23, 2019 at 18:31 Simon Willison notifications@github.com wrote:

I suggest trying a full path in ExecStart like this:

ExecStart=/home/chris/Env/datasette/bin/datasette serve -h 0.0.0.0 /home/chris/digital-library/databases/*.db --cors --metadata metadata.json

That should eliminate the chance of some kind of path confusion.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/simonw/datasette/issues/514?email_source=notifications&email_token=AB4RUOZESVC5ZI2AZFJJD2TP372UJA5CNFSM4H2VJCSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYLISIY#issuecomment-504793379, or mute the thread https://github.com/notifications/unsubscribe-auth/AB4RUO243SPJ7YSWJU7YKF3P372UJANCNFSM4H2VJCSA .

-- Chris Persaud ChrisPersaud.com

I've seen no evidence that JSON handling is even close to being a performance bottleneck, so wontfix.

Consider this when solving #465

I'm not going to do this - there are plenty of smarter ways of achieving a similar goal.

Black is now enforced by our unit tests as of #449

I'm going to call this "done" thanks to cloudrun: #400

I did a bunch of research relevant to this a while ago: https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/

Closing as couldn't replicate

https://simonwillison.net/2019/May/19/datasette-0-28/

https://datasette.readthedocs.io/en/stable/sql_queries.html?highlight=Pagination#pagination

Yes this is definitely documented now https://datasette.readthedocs.io/en/stable/performance.html

I did this a while ago https://datasette.readthedocs.io/en/stable/contributing.html

I no longer depend on Sanic so I should be able to solve this entirely within the Datasette codebase. I need to figure out how Transfer-Encoding: chunked and the Content-Length header interact.

I've landed #272 - need to manually test if it works on Windows now!

I wrote about this on my blog: https://simonwillison.net/2019/Jun/23/datasette-asgi/

272 is closed now! This hook is next on the priority list.

It's alive! Here's the first deployed version: https://a559123.datasette.io/

You can confirm it's running under ASGI by viewing https://a559123.datasette.io/-/versions and looking for the "asgi" key.

Compare to the last version of master running on Sanic here: http://aa91112.datasette.io/

It's live! https://a559123.datasette.io/

Ah, apologies – I had found and read those issues, but I was under the impression that they refered only to the filtered row count, not the unfiltered total row count.

For databases opened in immutable mode we pre-calculate the total row count for every table precisely so we can offer this kind of functionality without too much of a performance hit. The total row count is already available to the template (you can hit the .json endpoint to see it), so implementing this should be possible just by updating the template.

For mutable databases we have a mechanism for attempting the count and giving up after a specified time limit - we can use that to get "3 of many".

It looks like this is actually a dupe of #127 and #134.