I'm building this documentation-first - here's the documentation so far: https://github.com/simonw/datasette/blob/6fc8bd9c473f4a25e0a076f24c7e5a9b2f353bb8/docs/sql_queries.rst#magic-parameters

But then what kind of magic parameters might plugins want to add?

Here's a crazy idea: _scrapedcontent_url - it would look for the url column on the data being inserted, scrape the content from it and insert that. This does suggest that the magic resolving function scrapedcontent() would need to optionally be sent the full row dictionary being inserted too.

Brainstorming more potential magic parameters:

• _actor_id
• _actor_name
• _request_ip
• _request_user_agent
• _cookie_cookiename
• _signedcookie_cookiename - reading signed cookies would be cool, not sure how to specify namespace though, maybe always use the same one? Or have the namespace come last, _signedcookie_cookiename_mynamespace. Might not need special signed cookie support since actor is already usually from a signed cookie.
• _timestamp_unix (not happy with these names yet)
• _timestamp_localtime
• _timestamp_datetime
• _timestamp_utc

Another magic parameter that would be useful would be _random. Consider https://github.com/simonw/datasette-auth-tokens/issues/1 for example - I'd like to be able to provide a writable canned query which can create new authentication tokens in the database, but ideally it would automatically populate a secure random secret for each one.

Maybe _random_chars_128 to create a 128 character long random string (using os.urandom(64).hex()).

This would be the first example of a magic parameter where part of the parameter name is used to configure the resulting value. Maybe neater to separate that with a different character? Unfortunately _random_chars:128 wouldn't work because these parameters are used in a SQLite query where : has special meaning: insert into blah (secret) values (:_random_chars:128) wouldn't make sense.

Actually this is already supported by the proposed design - _random_chars_128 would become random("chars_128") so the random() function could split off the 128 itself.

Thanks for the bug report. Yes I think #838 may be the same issue. Will investigate.

I really need to get myself a Windows 10 development environment working so I can dig into this kind of bug properly. I have a gaming PC lying around that I could re-task for that.

Codecov Report

Merging #866 into master will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

master #866 +/-

======================================= Coverage 82.99% 82.99%
======================================= Files 26 26
Lines 3547 3547
======================================= Hits 2944 2944
Misses 603 603
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1a5b7d3...fb64dda. Read the comment docs.

But also want to say thanks for a great tool

+1!

This seems to be a duplicate of: https://github.com/simonw/datasette/issues/838

I tried setting cache_size_kb=0 then cache_size_kb=100000, still getting this behavior. I even changed Database::table_counts and lowered time limit to 1

py table_count = ( await self.execute( "select count(*) from [{}]".format(table), custom_time_limit=1, ) ).rows[0][0] counts[table] = table_count

I feel like 10 seconds is a magic number, like a processing timeout and datasette gives up and returns the page. Index page loads instantly, table page, query page, as well. But when I return to database page after some time, it loads in 10s.

EDIT:

It's always like 10 + 0.3s, like 10s wait and timeout then 300ms to render the page

The ideal fix here would be to rework my BaseView subclass mechanism to work with register_routes() so that those views don't have any special privileges above plugin-provided views.

Urgh, fixing this is going to be a bit of a pain.

Here's where I added that custom dispatch_request() method - it was to implement flash messaging in #790: https://github.com/simonw/datasette/blame/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/views/base.py#L85

If I want this to be made available to register_routes() views as well, I'm going to have to move the logic somewhere else. In particular I need to make sure that the request object is created once and used throughout the whole request cycle.

Currently register_routes() view functions get their own separate request object which is created here:

https://github.com/simonw/datasette/blob/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/app.py#L1057-L1068

So I'm going to have to refactor this quite a bit to get that shared request object which can be passed both to register_routes views and to my various BaseView subclasses.