issue_comments

sql select sqlite_master.name as table_name, table_info.* from sqlite_master join pragma_table_info(sqlite_master.name) as table_info order by sqlite_master.name, table_info.cid https://latest.datasette.io/fixtures?sql=select%0D%0A++sqlite_master.name+as+table_name%2C%0D%0A++table_info.*%0D%0Afrom%0D%0A++sqlite_master%0D%0A++join+pragma_table_info%28sqlite_master.name%29+as+table_info%0D%0Aorder+by%0D%0A++sqlite_master.name%2C%0D%0A++table_info.cid

Thank you. Now it's ok with the url

http://localhost:8001/index/summary?_search=language%3Aeng&_sort=title&_searchmode=raw

But I'm not able to manage it in the metadata file. Here is mine (note that the sort column is taken into account) Here it is:

``` { "databases": { "index": { "tables": { "summary": { "sort": "title", "searchmode": "raw" } } } } }

``` Any idea ?

OK, I just released 0.41 with that and a bunch of other stuff: https://datasette.readthedocs.io/en/latest/changelog.html#v0-41

Actually I'm going to put that release out today. I was hoping to finish #698 first but that shouldn't delay those other features any longer.

I'll definitely get that out this week!

For the moment a trick I often use is to put a URL to the most recent commit in my requirements.txt - e.g. https://github.com/simonw/datasette/archive/0784f2ef9d3ff6dd9df05f54cb51de29a6d11764.zip

This should be safe because nothing lands on Datasette master without the full unit test suite passing. But you're right, there's a bunch of stuff now that needs to go out in a release: https://github.com/simonw/datasette/compare/0.40...0784f2ef9d3ff6dd9df05f54cb51de29a6d11764

It's interesting to hear from someone who's using this feature - I'm considering moving it out into a plugin #647.

Thanks for the suggestion! I'll add this.

This was a deliberate change from #651.

The _search_colname= alternative argument for doing this still works - compare these two:

• https://latest.datasette.io/fixtures/searchable?_search=dog
• https://latest.datasette.io/fixtures/searchable?_search_tex1=dog

If you want to use advanced search syntax on those pages you can do so using the &_searchmode=raw option - I added better documentation for this the other day: https://datasette.readthedocs.io/en/latest/full_text_search.html#advanced-sqlite-search-queries

Example: https://latest.datasette.io/fixtures/searchable?_search=text1:dog&_searchmode=raw

More demos here: https://github.com/simonw/til/blob/master/sqlite/list-all-columns-in-a-database.md

Potential recipe in https://github.com/simonw/til/blob/master/sqlite/list-all-columns-in-a-database.md

Interestingly https://latest.datasette.io/fixtures?sql=select+*+from+pragma_function_list() doesn't work, when it DOES work on my laptop.

latest.datasette.io currently runs SQLite 3.27.2 while my laptop runs 3.31.1

https://www.sqlite.org/changes.html#version_3_30_0 says that as-of 3.30.0:

The PRAGMA function_list, PRAGMA module_list, and PRAGMA pragma_list commands are now enabled in all builds by default. Disable them using -DSQLITE_OMIT_INTROSPECTION_PRAGMAS.

Some demos: * https://latest.datasette.io/fixtures?sql=select++from+pragma_database_list%28%29 * https://latest.datasette.io/fixtures?sql=select++from+pragma_foreign_key_list%28%27complex_foreign_keys%27%29 * https://latest.datasette.io/fixtures?sql=select++from+pragma_function_list%28%29 * https://latest.datasette.io/fixtures?sql=select++from+pragma_index_info%28%27idx_compound_three_primary_keys_content%27%29 * https://latest.datasette.io/fixtures?sql=select++from+pragma_index_list%28%27compound_three_primary_keys%27%29 * https://latest.datasette.io/fixtures?sql=select++from+pragma_index_xinfo%28%27idx_compound_three_primary_keys_content%27%29 * https://latest.datasette.io/fixtures?sql=select++from+pragma_page_count%28%29 * https://latest.datasette.io/fixtures?sql=select++from+pragma_max_page_count%28%29 * https://latest.datasette.io/fixtures?sql=select++from+pragma_page_size%28%29 * https://latest.datasette.io/fixtures?sql=select++from+pragma_schema_version%28%29 * https://latest.datasette.io/fixtures?sql=select++from+pragma_table_info%28%27complex_foreign_keys%27%29 * https://latest.datasette.io/fixtures?sql=select++from+pragma_table_xinfo%28%27complex_foreign_keys%27%29

For the moment I'll allow-list the following: * pragma_database_list * pragma_foreign_key_list * pragma_function_list * pragma_index_info * pragma_index_list * pragma_index_xinfo * pragma_page_count * pragma_max_page_count * pragma_page_size * pragma_schema_version * pragma_table_info * pragma_table_xinfo

I could allow-list some other useful pragma_x tables too.

SQLite calls these "pragma functions" - documented here: https://www.sqlite.org/pragma.html#pragfunc

They sound safe:

Table-valued functions exist only for PRAGMAs that return results and that have no side-effects.

Maybe use a negative lookahead assertion? https://docs.python.org/3/library/re.html#index-20

(?!...)

Matches if ... doesn’t match next. This is a negative lookahead assertion. For example, Isaac (?!Asimov) will match 'Isaac ' only if it’s not followed by 'Asimov'.

The rationale for blocking pragma entirely from statements is that it can be used to change the state of the SQLite database, e.g. from https://www.sqlite.org/pragma.html :

PRAGMA schema.application_id; PRAGMA schema.application_id = integer ; That second line is unsafe. I don't think it's possible to use the pragma_table_x variants to make writes in this way.

select * from pragma_table_info(tablename); is currently disallowed for user-provided queries via a regex restriction - but could help here too.

https://github.com/simonw/datasette/blob/d349d57cdf3d577afb62bdf784af342a4d5be660/datasette/utils/init.py#L174

Here's rendering code from my hacked-together not-yet-released S3 image proxy: ```python from starlette.responses import Response from PIL import Image, ExifTags import pyheif

for ORIENTATION_TAG in ExifTags.TAGS.keys(): if ExifTags.TAGS[ORIENTATION_TAG] == "Orientation": break ... # Load it into Pillow if ext == "heic": heic = pyheif.read_heif(image_response.content) image = Image.frombytes(mode=heic.mode, size=heic.size, data=heic.data) else: image = Image.open(io.BytesIO(image_response.content))

# Does EXIF tell us to rotate it?
try:
    exif = dict(image._getexif().items())
    if exif[ORIENTATION_TAG] == 3:
        image = image.rotate(180, expand=True)
    elif exif[ORIENTATION_TAG] == 6:
        image = image.rotate(270, expand=True)
    elif exif[ORIENTATION_TAG] == 8:
        image = image.rotate(90, expand=True)
except (AttributeError, KeyError, IndexError):
    pass

# Resize based on ?w= and ?h=, if set
width, height = image.size
w = request.query_params.get("w")
h = request.query_params.get("h")
if w is not None or h is not None:
    if h is None:
        # Set h based on w
        w = int(w)
        h = int((float(height) / width) * w)
    elif w is None:
        h = int(h)
        # Set w based on h
        w = int((float(width) / height) * h)
    w = int(w)
    h = int(h)
    image.thumbnail((w, h))

# ?bw= converts to black and white
if request.query_params.get("bw"):
    image = image.convert("L")

# ?q= sets the quality - defaults to 75
quality = 75
q = request.query_params.get("q")
if q and q.isdigit() and 1 <= int(q) <= 100:
    quality = int(q)

# Output as JPEG or PNG
output_image = io.BytesIO()
image_type = "JPEG"
kwargs = {"quality": quality}
if image.format == "PNG":
    image_type = "PNG"
    kwargs = {}

image.save(output_image, image_type, **kwargs)
return Response(
    output_image.getvalue(),
    media_type="image/jpeg",
    headers={"cache-control": "s-maxage={}, public".format(365 * 24 * 60 * 60)},
)

```

The plugin can be generalized: it can be configured to know how to take the URL path, look it up in ANY table (via a custom SQL query) to get a path on disk and then serve that.

The apple_photos table has an indexed uuid column and a path column which stores the full path to that photo file on disk.

I can write a custom Datasette plugin which takes the uuid from the URL, looks up the path, then serves up a thumbnail of the jpeg or heic image file.

I'll prototype this is a one-off plugin first, then package it on PyPI for other people to install.

Most annoying part of this is the difficulty of actually showing a photo.

Maybe I need to run a local proxy that I can link to? A custom Datasette plugin perhaps?