I need to figure out what to do about extra_context_from_filters - which was previously passed straight to the HTML context.

https://github.com/simonw/datasette/blob/11f7feb7a3f7166c71389786880863d60ed3d165/datasette/views/table.py#L406-L422

Found a neat trick: ```diff diff --git a/datasette/app.py b/datasette/app.py index 186f192d..40416713 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -23,7 +23,13 @@ from pathlib import Path

from markupsafe import Markup, escape from itsdangerous import URLSafeSerializer -from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader +from jinja2 import ( + ChoiceLoader, + Environment, + FileSystemLoader, + PrefixLoader, + StrictUndefined, +) from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound

@@ -394,7 +400,10 @@ class Datasette: ] ) self.jinja_env = Environment( - loader=template_loader, autoescape=True, enable_async=True + loader=template_loader, + autoescape=True, + enable_async=True, + undefined=StrictUndefined, ) self.jinja_env.filters["escape_css_string"] = escape_css_string self.jinja_env.filters["quote_plus"] = urllib.parse.quote_plus ``` This causes Jinja to raise a hard error if there are any variables referenced in the template that are not available in the context.

It's helping me spot things that are still missing, rather than just relying on failed unit tests.

I improved the readability by removing some unnecessary table aliases: sql with recursive nested_folders(folder_id, descendant_folder_id) as ( -- base case: select all immediate children of the root folder select id, id from folders where parent is null union all -- recursive case: select all children of the previous level of nested folders select nested_folders.folder_id, folders.id from nested_folders join folders on nested_folders.descendant_folder_id = folders.parent ) -- Find notes within all descendants of folder 1 select * from notes where folder in ( select descendant_folder_id from nested_folders where folder_id = 1 );

Here's a query that returns all notes in folder 1, including notes in descendant folders: sql with recursive nested_folders(folder_id, descendant_folder_id) as ( -- base case: select all immediate children of the root folder select id, id from folders where parent is null union all -- recursive case: select all children of the previous level of nested folders select nf.folder_id, f.id from nested_folders nf join folders f on nf.descendant_folder_id = f.parent ) -- Find notes within all descendants of folder 1 select * from notes where folder in ( select descendant_folder_id from nested_folders where folder_id = 1 ); With assistance from ChatGPT. Prompts were:

``` SQLite schema:

CREATE TABLE [folders] ( [id] INTEGER PRIMARY KEY, [long_id] TEXT, [name] TEXT, [parent] INTEGER, FOREIGN KEY([parent]) REFERENCES folders );

Write a recursive CTE that returns the following:

folder_id | descendant_folder_id

With a row for every nested child of every folder - so the top level folder has lots of rows Then I tweaked it a bit, then ran this: WITH RECURSIVE nested_folders(folder_id, descendant_folder_id) AS ( -- base case: select all immediate children of the root folder SELECT id, id FROM folders WHERE parent IS NULL UNION ALL -- recursive case: select all children of the previous level of nested folders SELECT nf.folder_id, f.id FROM nested_folders nf JOIN folders f ON nf.descendant_folder_id = f.parent ) -- select all rows from the recursive CTE SELECT * from notes where folder in (select descendant_folder_id FROM nested_folders where folder_id = 1)

Convert all SQL keywords to lower case, and re-indent ```

Here's what the CTE from that looks like:

I've implemented the test (thanks for pointing me in the right direction!).

At tmcl-it/datasette:0.64.1+row-view-expand-labels I also have a variant of this patch that applies to the 0.64.x branch. Please let me know if you'd be interested in merging that as well and I'll open another PR.

Sure, let's merge that one too - it can go out in the next 0.64.x series release (maybe even a 0.65).

Good catch, thanks.

My folders table will have:

• id - rowid
• long_id - that long unique string ID
• name - the name
• parent - foreign key to id

Improved script: zsh osascript -e 'tell application "Notes" set allFolders to folders repeat with aFolder in allFolders set folderId to id of aFolder set folderName to name of aFolder set folderContainer to container of aFolder if class of folderContainer is folder then set folderContainerId to id of folderContainer else set folderContainerId to "" end if log "ID: " & folderId log "Name: " & folderName log "Container: " & folderContainerId log " " end repeat end tell ' ``` ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113 Name: Blog posts Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698 Name: JSK Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995 Name: Nested inside blog posts Container: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526 Name: New Folder Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839 Name: New Folder 1 Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2 Name: Notes Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059 Name: Quick Notes Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283 Name: UK Christmas 2022 Container: `` I filtered out things where the parent was an account and not a folder usingif class of folderContainer is folder then`.

Created through several rounds with ChatGPT (including hints like "rewrite that using setdefault()"): ```python def topological_sort(nodes): children = {} for node in nodes: parent_id = node["parent"] if parent_id is not None: children.setdefault(parent_id, []).append(node)

def traverse(node, result):
    result.append(node)
    if node["id"] in children:
        for child in children[node["id"]]:
            traverse(child, result)

sorted_data = []

for node in nodes:
    if node["parent"] is None:
        traverse(node, sorted_data)

return sorted_data

```

I used ChatGPT to write this: osascript -e 'tell application "Notes" set allFolders to folders repeat with aFolder in allFolders set folderId to id of aFolder set folderName to name of aFolder set folderContainer to container of aFolder set folderContainerName to name of folderContainer log "Folder ID: " & folderId log "Folder Name: " & folderName log "Folder Container: " & folderContainerName log " " --check for nested folders if count of folders of aFolder > 0 then set nestedFolders to folders of aFolder repeat with aNestedFolder in nestedFolders set nestedFolderId to id of aNestedFolder set nestedFolderName to name of aNestedFolder set nestedFolderContainer to container of aNestedFolder set nestedFolderContainerName to name of nestedFolderContainer log " Nested Folder ID: " & nestedFolderId log " Nested Folder Name: " & nestedFolderName log " Nested Folder Container: " & nestedFolderContainerName log " " end repeat end if end repeat end tell ' Which for my account output this: ``` Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113 Folder Name: Blog posts Folder Container: iCloud

Nested Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
Nested Folder Name: Nested inside blog posts
Nested Folder Container: Blog posts

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698 Folder Name: JSK Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995 Folder Name: Nested inside blog posts Folder Container: Blog posts

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526 Folder Name: New Folder Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839 Folder Name: New Folder 1 Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2 Folder Name: Notes Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059 Folder Name: Quick Notes Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283 Folder Name: UK Christmas 2022 Folder Container: iCloud ``` So I think the correct approach here is to run code at the start to list all of the folders (no need to do fancy recursion though, just a flat list with the parent containers is enough) and create a model of that hierarchy in SQLite.

Then when I import notes I can foreign key reference them back to their containing folder.

I'm tempted to use rowid for the foreign keys because the official IDs are pretty long.

So it looks like folders can be hierarchical?

From the Script Editor library docs:

A note has a:

• container (folder), r/o) : the folder of the note

Here's what a folder looks like:

folder n : a folder containing notes elements:

• contains folders, notes; contained by application, accounts, folders.

properties:

• name (text) : the name of the folder
• id (text, r/o) : the unique identifier of the folder
• shared (boolean, r/o) : Is the folder shared?
• container (account or folder, r/o) : the container of the folder

In terms of the UI: I'm tempted to say that the default behaviour is for it to run until it sees a note that it already knows about AND that has matching update/created dates, and then stop.

You can do a full import again ignoring that logic with apple-notes-to-sqlite notes.db --full.

It looks like the iteration order is most-recently-modified-first - I tried editing a note a bit further back in my notes app and it was the first one output by apple-notes-to-sqlite --dump.

OK, this works!

It doesn't have tests yet. I guess I'll need to mock subprocess to test this.

Before that conversion:

Monday, March 6, 2023 at 11:55:15 AM

After:

2023-03-06T11:55:15

Converting AppleScript date strings to ISO format is hard!

https://forum.latenightsw.com/t/formatting-dates/841 has a recipe I'll try:

set todayISO to (todayDate as «class isot» as string)

Not clear to me how timezones work here. I'm going to ignore them for the moment.

My prototype showed that images embedded in notes come out in the HTML export as bas64 image URLs, which is neat.

I opened the "Script Editor" app on my computer, used Window -> Library to open the Library panel, then clicked on the Notes app there. I got this:

So the notes object has these properties:

• name (text) : the name of the note (normally the first line of the body)
• id (text, r/o) : the unique identifier of the note
• container (folder, r/o) : the folder of the note
• body (text) : the HTML content of the note
• plaintext (text, r/o) : the plaintext content of the note
• creation date (date, r/o) : the creation date of the note
• modification date (date, r/o) : the modification date of the note
• password protected (boolean, r/o) : Is the note password protected?
• shared (boolean, r/o) : Is the note shared?

I'm going to ignore the concept of attachments for the moment.

I think the AppleScript I want to pass to osascript looks like this: applescript tell application "Notes" repeat with eachNote in every note set noteId to the id of eachNote set noteTitle to the name of eachNote set noteBody to the body of eachNote log "------------------------" & "\n" log noteId & "\n" log noteTitle & "\n\n" log noteBody & "\n" end repeat end tell But there are a few more properties I'd like to get - created and updated date for example.

This did the job! Next step is to turn that into a Python script.

After a few more rounds I got to this script, which outputs them to a /tmp/notes.txt file:

```zsh

!/bin/zsh

osascript -e ' set notesFile to "/tmp/notes.txt" set fileRef to open for access notesFile with write permission tell application "Notes" repeat with eachNote in every note set noteId to the id of eachNote set noteTitle to the name of eachNote set noteBody to the body of eachNote write "------------------------" & "\n" to fileRef write noteId & "\n" to fileRef write noteTitle & "\n\n" to fileRef write noteBody & "\n" to fileRef end repeat end tell close access fileRef' Then I wrote this little Python script to load them into a database:python import sqlite_utils split = b"------------------------\n" s = open("/tmp/notes.txt", "rb").read() notes = [n.decode("mac_roman") for n in s.split(split) if n]

cleaned_notes = [{ "id": n.split("\n")[0], "title": n.split("\n")[1], "body": "\n".join(n.split("\n")[2:]).strip() } for n in notes]

db = sqlite_utils.Database("/tmp/notes.db") db["notes"].insert_all(cleaned_notes) ```

Me:

Write applescript to loop through all of my Apple Notes and output their contents

ChatGPT:

applescript tell application "Notes" repeat with eachNote in every note set noteTitle to the name of eachNote set noteBody to the body of eachNote display dialog noteTitle & return & return & noteBody end repeat end tell Me:

How can I turn that into a zsh script which will output the results to my terminal

ChatGPT: ```zsh

!/usr/bin/zsh

osascript -e 'tell application "Notes" repeat with eachNote in every note set noteTitle to the name of eachNote set noteBody to the body of eachNote set output to noteTitle & "\n" & noteBody & "\n" display dialog output log output end repeat end tell' `` I had to change that first line to#!/bin/zsh` - then when I ran it it spammed my screen with dialog boxes.

Me:

I want to log the notes contents to the terminal, not display it in a dialog

ChatGPT:

```zsh

!/bin/zsh

osascript -e 'tell application "Notes" repeat with eachNote in every note set noteTitle to the name of eachNote set noteBody to the body of eachNote set output to noteTitle & "\n" & noteBody & "\n" log output end repeat end tell' ``` This really helps!

Just ran into a no such table: columns error - which I think is because my new view code sometimes bypasses calling this method (currently done in BaseView.dispatch_request()):

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/base.py#L101-L103

Or... I could temporarily build a quick additional CannedQueryView subclass that just does the necessary bits to get the existing code to work. I'm going to try that.

Solving this is proving difficult: https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L1500-L1503

The problem is that calling .data() on QueryView only works here because we expect to ourselves be inside a .data() method, with all of the existing magic that knows how to render things that are returned by that.

So I may need to substantially re-engineer how QueryView works in order to get this to work.

127.0.0.1:8001/fixtures/neighborhood_search fails because the forwarding to a canned query does not yet work.

pytest tests/test_table_html.py

Currently 44 failed, 24 passed in 7.53s

Failures here: https://gist.github.com/simonw/df0a52cd7d820b776dc3dfc50e7cb778

I managed to get HTML view working! I did it by continuing to add more things to the extras and the _html bundle until the page loaded for me:

```diff async def extra_extras(): "Available ?_extra= blocks" return { @@ -1981,6 +2053,14 @@ async def extra_extras(): "query", "display_columns", "display_rows", + "database", + "table", + "database_color", + "table_actions", + "filters", + "renderers", + "custom_table_templates", + "sorted_facet_results", ] }

@@ -2006,6 +2086,14 @@ async def extra_extras(): extra_query, extra_metadata, extra_extras, + extra_database, + extra_table, + extra_database_color, + extra_table_actions, + extra_filters, + extra_renderers, + extra_custom_table_templates, + extra_sorted_facet_results, ) `` I'll probably refactor this into something cleaner, and maybe but a bunch of them in a"html"dictionary and update the templates to use{{ html.filters }}` or similar. Will look at that once the tests are passing.