issue_comments

OK, for that percentage thing: the Python core implementation of URL percentage escaping deliberately ignores two of the characters we want to escape: . and -:

https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Lib/urllib/parse.py#L780-L783

python _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-~') It also defaults to skipping / (passed as a safe= parameter to various things).

I'm going to try borrowing and modifying the core of the Python implementation: https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Lib/urllib/parse.py#L795-L814 ```python class _Quoter(dict): """A mapping from bytes numbers (in range(0,256)) to strings. String values are percent-encoded byte values, unless the key < 128, and in either of the specified safe set, or the always safe set. """ # Keeps a cache internally, via missing, for efficiency (lookups # of cached keys don't call Python code at all). def init(self, safe): """safe: bytes object.""" self.safe = _ALWAYS_SAFE.union(safe)

def __repr__(self):
    return f"<Quoter {dict(self)!r}>"

def __missing__(self, b):
    # Handle a cache miss. Store quoted string in cache and return.
    res = chr(b) if b in self.safe else '%{:02X}'.format(b)
    self[b] = res
    return res

```

I want to try regular percentage encoding, except that it also encodes both the - and the . characters, AND it uses - instead of % as the encoding character.

Should check what it does with emoji too.

This comment from glyph got me thinking:

Have you considered replacing % with some other character and then using percent-encoding?

What happens if a table name includes a % character and that ends up getting mangled by a misbehaving proxy?

I should consider % in the escaping system too. And maybe go with that suggestion of using percent-encoding directly but with a different character.

Lots of great conversations about the dash encoding implementation on Twitter: https://twitter.com/simonw/status/1500228316309061633

@dracos helped me figure out a simpler regex: https://twitter.com/dracos/status/1500236433809973248

^/(?P<database>[^/]+)/(?P<table>[^\/\-\.]*|\-/|\-\.|\-\-)*(?P<format>\.\w+)?$

Blogged about this here: https://simonwillison.net/2022/Mar/5/dash-encoding/

I'm going to move this work to a PR.

Those new docs: https://github.com/simonw/datasette/blob/d1cb73180b4b5a07538380db76298618a5fc46b6/docs/internals.rst#dash-encoding

Wrote documentation: