issue_comments

18 rows where issue = 638212085 sorted by updated_at descending

View and edit SQL

Suggested facets: created_at (date), updated_at (date)

user

issue

  • Magic parameters for canned queries · 18

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
650684635 https://github.com/simonw/datasette/issues/842#issuecomment-650684635 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDY4NDYzNQ== simonw 9599 2020-06-28T03:30:31Z 2020-06-28T03:30:31Z OWNER

Live demo: https://latest.datasette.io/fixtures/magic_parameters

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650679100 https://github.com/simonw/datasette/issues/842#issuecomment-650679100 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDY3OTEwMA== simonw 9599 2020-06-28T03:00:44Z 2020-06-28T03:00:44Z OWNER

I'm going to add some canned queries to the metadata.json used by the live demo that illustrate this feature.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650678951 https://github.com/simonw/datasette/issues/842#issuecomment-650678951 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDY3ODk1MQ== simonw 9599 2020-06-28T02:59:52Z 2020-06-28T02:59:52Z OWNER

Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#magic-parameters

Plugin hook documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#plugin-hook-register-magic-parameters

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650648434 https://github.com/simonw/datasette/issues/842#issuecomment-650648434 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDY0ODQzNA== simonw 9599 2020-06-27T23:27:35Z 2020-06-27T23:37:38Z OWNER

I'm going to rename _request_X to _header_X as that better reflects what it now does.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650593122 https://github.com/simonw/datasette/issues/842#issuecomment-650593122 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDU5MzEyMg== simonw 9599 2020-06-27T18:03:02Z 2020-06-27T18:03:10Z OWNER

Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter _cookie_ds_actor to be resolved and returned as JSON data that the attacker can see.

This is an open security hole in https://github.com/simonw/datasette/commit/94c1315f0030fd58ce46a9294052c5c9d9d181c7 - it's useful for testing, but I need to remove it before I land that branch.

https://github.com/simonw/datasette/blob/94c1315f0030fd58ce46a9294052c5c9d9d181c7/datasette/views/database.py#L231-L237

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650458857 https://github.com/simonw/datasette/issues/842#issuecomment-650458857 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDQ1ODg1Nw== simonw 9599 2020-06-27T00:11:04Z 2020-06-27T00:11:04Z OWNER

Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter _cookie_ds_actor to be resolved and returned as JSON data that the attacker can see.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650455793 https://github.com/simonw/datasette/issues/842#issuecomment-650455793 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDQ1NTc5Mw== simonw 9599 2020-06-26T23:57:30Z 2020-06-27T00:00:16Z OWNER

Maybe I should ship a default _scope_headers_... parameter instead, which reads from a dictionary of scope["headers"] - https://asgi-scope.now.sh/ shows what those look like.

{'client': ('148.64.98.14', 0),
 'headers': [[b'host', b'asgi-scope.now.sh'],
             [b'x-forwarded-for', b'148.64.98.14'],
             [b'x-vercel-id', b'sw72x-1593215573008-024e4e603806'],
             [b'x-forwarded-host', b'asgi-scope.now.sh'],
             [b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'webp,*/*;q=0.8'],
             [b'x-real-ip', b'148.64.98.14'],
             [b'x-vercel-deployment-url', b'asgi-scope-9eyeojbek.now.sh'],
             [b'upgrade-insecure-requests', b'1'],
             [b'x-vercel-trace', b'sfo1'],
             [b'x-forwarded-proto', b'https'],
             [b'accept-language', b'en-US,en;q=0.5'],
             [b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko'
              b'/20100101 Firefox/77.0'],
             [b'x-vercel-forwarded-for', b'148.64.98.14'],
             [b'accept-encoding', b'gzip, deflate, br'],
             [b'dnt', b'1'],
             [b'te', b'trailers']],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/',
 'query_string': b'',
 'raw_path': b'/',
 'root_path': '',
 'scheme': 'https',
 'server': ('asgi-scope.now.sh', 80),
 'type': 'http'}

I'm going to have _request_X actually mean "find the first value for X in scope["headers"]" - with underscores converted to hyphens.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650455353 https://github.com/simonw/datasette/issues/842#issuecomment-650455353 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDQ1NTM1Mw== simonw 9599 2020-06-26T23:55:40Z 2020-06-26T23:55:40Z OWNER

_request_ip is actually quite hard to implement - should it take into account things like the x-forwarded-for header?

It probably should - but that means it now needs a bunch of extra configuration to tell it which of those headers can be trusted in the current environment.

As such I think I'll leave that for a plugin.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
649014757 https://github.com/simonw/datasette/issues/842#issuecomment-649014757 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0OTAxNDc1Nw== simonw 9599 2020-06-24T19:15:46Z 2020-06-24T19:31:52Z OWNER

I'm building this documentation-first - here's the documentation so far: https://github.com/simonw/datasette/blob/6fc8bd9c473f4a25e0a076f24c7e5a9b2f353bb8/docs/sql_queries.rst#magic-parameters

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
646271834 https://github.com/simonw/datasette/issues/842#issuecomment-646271834 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0NjI3MTgzNA== simonw 9599 2020-06-18T19:49:41Z 2020-06-24T18:49:22Z OWNER

But then what kind of magic parameters might plugins want to add?

Here's a crazy idea: _scrapedcontent_url - it would look for the url column on the data being inserted, scrape the content from it and insert that. This does suggest that the magic resolving function scrapedcontent() would need to optionally be sent the full row dictionary being inserted too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
646270702 https://github.com/simonw/datasette/issues/842#issuecomment-646270702 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0NjI3MDcwMg== simonw 9599 2020-06-18T19:47:19Z 2020-06-24T18:48:48Z OWNER

Brainstorming more potential magic parameters:

  • _actor_id
  • _actor_name
  • _request_ip
  • _request_user_agent
  • _cookie_cookiename
  • _signedcookie_cookiename - reading signed cookies would be cool, not sure how to specify namespace though, maybe always use the same one? Or have the namespace come last, _signedcookie_cookiename_mynamespace. Might not need special signed cookie support since actor is already usually from a signed cookie.
  • _timestamp_unix (not happy with these names yet)
  • _timestamp_localtime
  • _timestamp_datetime
  • _timestamp_utc
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
649000075 https://github.com/simonw/datasette/issues/842#issuecomment-649000075 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0OTAwMDA3NQ== simonw 9599 2020-06-24T18:46:36Z 2020-06-24T18:47:37Z OWNER

Another magic parameter that would be useful would be _random. Consider https://github.com/simonw/datasette-auth-tokens/issues/1 for example - I'd like to be able to provide a writable canned query which can create new authentication tokens in the database, but ideally it would automatically populate a secure random secret for each one.

Maybe _random_chars_128 to create a 128 character long random string (using os.urandom(64).hex()).

This would be the first example of a magic parameter where part of the parameter name is used to configure the resulting value. Maybe neater to separate that with a different character? Unfortunately _random_chars:128 wouldn't work because these parameters are used in a SQLite query where : has special meaning: insert into blah (secret) values (:_random_chars:128) wouldn't make sense.

Actually this is already supported by the proposed design - _random_chars_128 would become random("chars_128") so the random() function could split off the 128 itself.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
646272627 https://github.com/simonw/datasette/issues/842#issuecomment-646272627 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0NjI3MjYyNw== simonw 9599 2020-06-18T19:51:32Z 2020-06-18T19:51:32Z OWNER

I'd be OK with the first version of this not including a plugin hook.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
646264051 https://github.com/simonw/datasette/issues/842#issuecomment-646264051 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0NjI2NDA1MQ== simonw 9599 2020-06-18T19:32:13Z 2020-06-18T19:32:37Z OWNER

If every magic parameter has a prefix and suffix, like _request_ip and _actor_id, then plugins could register a function for a prefix. Register a function to _actor and actor("id")will be called for _actor_id.

But does it make sense for every magic parameter to be of form _a_b? I think so.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
646246062 https://github.com/simonw/datasette/issues/842#issuecomment-646246062 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0NjI0NjA2Mg== simonw 9599 2020-06-18T18:54:41Z 2020-06-18T18:54:41Z OWNER

The _actor_id param makes this a bit trickier, because we can't just say "if you see an unknown parameter called X call this function" - our magic parameter logic isn't adding single parameters, it might add a whole family of them.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
646242172 https://github.com/simonw/datasette/issues/842#issuecomment-646242172 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0NjI0MjE3Mg== simonw 9599 2020-06-18T18:46:06Z 2020-06-18T18:53:31Z OWNER

Yes that can work - and using __missing__ (new in Python 3) is nicer because then the regular dictionary gets checked first:

import sqlite3

conn = sqlite3.connect(":memory:")


class Magic(dict):
    def __missing__(self, key):
        return key.upper()


conn.execute("select :name", Magic()).fetchall()

Outputs:

[('NAME',)]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
646238702 https://github.com/simonw/datasette/issues/842#issuecomment-646238702 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0NjIzODcwMg== simonw 9599 2020-06-18T18:39:07Z 2020-06-18T18:39:07Z OWNER

It would be nice if Datasette didn't have to do any additional work to find e.g. _request_ip if that parameter turned out not to be used by the query.

Could I do this with a custom class that implements __getitem__() and then gets passed as SQLite arguments?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
643663005 https://github.com/simonw/datasette/issues/842#issuecomment-643663005 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0MzY2MzAwNQ== simonw 9599 2020-06-13T18:51:57Z 2020-06-13T18:51:57Z OWNER

Two potential designs:

  • _actor_id, _request_ip, _now_timestamp - so special reserved parameters
  • a SQL function: update blah set up = special('ip')

I fee the first would be easier to implement.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 29.892ms · About: github-to-sqlite