issue_comments

Table actions

9,673 rows sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

issue >1000

user 352

author_association 4

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1498167714 https://github.com/dogsheep/github-to-sqlite/issues/79#issuecomment-1498167714 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79 IC_kwDODFdgUs5ZTDmi simonw 9599 2023-04-05T21:12:55Z 2023-04-05T21:12:55Z MEMBER

It's a rate limiting problem: https://github.com/dogsheep/github-to-sqlite/actions/runs/4622674009/jobs/8175633155

File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 407, in commits utils.save_commits(db, commits, repo_full["id"]) File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 635, in save_commits for commit in commits: File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 417, in fetch_commits for commits in paginate(url, headers): File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 478, in paginate raise GitHubError.from_response(response) github_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403) Error: Process completed with exit code 1.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Deploy demo job is failing due to rate limit 1570375808  
1495780111 https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1495780111 https://api.github.com/repos/simonw/sqlite-utils/issues/235 IC_kwDOCGYnMM5ZJ8sP Thomascountz 19786848 2023-04-04T11:09:56Z 2023-04-04T11:13:40Z NONE

@wpears' workaround also worked for me, but also required me to manually set PRAGMA writable_schema.

sql PRAGMA writable_schema = 1; UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (...)' PRAGMA writable_schema = 0;

```shell $ python --version Python 3.11.2

$ sqlite3 --version 3.41.2 2023-03-22 11:56:21 0d1fc92f94cb6b76bffe3ec34d69cffde2924203304e8ffc4155597af0c191da

$ sqlite-utils --version sqlite-utils, version 3.30 ```

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified 810618495  
1493442956 https://github.com/dogsheep/apple-notes-to-sqlite/issues/6#issuecomment-1493442956 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6 IC_kwDOJHON9s5ZBCGM amlestin 14314871 2023-04-02T21:20:43Z 2023-04-02T21:25:37Z NONE

I'm experiencing something similar. My apostrophes (') turn into (’) and the output is truncated. Hoping to debug next weekend 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Character encoding problem 1617602868  
1493052396 https://github.com/simonw/sqlite-utils/issues/265#issuecomment-1493052396 https://api.github.com/repos/simonw/sqlite-utils/issues/265 IC_kwDOCGYnMM5Y_ivs dracos 154364 2023-04-01T17:27:18Z 2023-04-01T17:27:18Z NONE

enable_fts is a function in datasette, not in this repo, which doesn't do any escaping of search terms. It sounds like from https://docs.datasette.io/en/stable/full_text_search.html#advanced-sqlite-search-queries you might want to enable raw searching, as otherwise it's disabled and everything is escaped by default.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Using enable_fts before search term 907795562  
1493051222 https://github.com/simonw/sqlite-utils/issues/159#issuecomment-1493051222 https://api.github.com/repos/simonw/sqlite-utils/issues/159 IC_kwDOCGYnMM5Y_idW dracos 154364 2023-04-01T17:21:05Z 2023-04-01T17:21:05Z NONE

In a related issue, nearly a year later I just stumbled across this again, as I wondered why none of my rebuild-fts were rebuilding. It looks like: disable_fts in db.py commits; enable_fts partly commits except the last step (due to executescript committing a pending transaction); rebuild_fts won't commit unless manually done as above with e.g. a context manager.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 .delete_where() does not auto-commit (unlike .insert() or .upsert()) 702386948  
1492777509 https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1492777509 https://api.github.com/repos/simonw/sqlite-utils/issues/235 IC_kwDOCGYnMM5Y-fol hydrosquall 9020979 2023-04-01T01:31:48Z 2023-04-01T01:31:48Z NONE

My current workaround is to use this library from a python script instead of as a CLI tool.

This lets me set the foreign key constraint at table creation time, instead of trying to modify an existing table. docs

I found this stackoverflow helpful, as it explained that Sqlite doesn't support modifying existing tables directly.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified 810618495  
1492387771 https://github.com/simonw/datasette/issues/2050#issuecomment-1492387771 https://api.github.com/repos/simonw/datasette/issues/2050 IC_kwDOBm6k_c5Y9Ae7 simonw 9599 2023-03-31T17:59:48Z 2023-03-31T17:59:48Z OWNER

Some of the extras from the table view make sense here. A few custom ones make sense too - including this already existing but undocumented one:

https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/views/row.py#L86-L89

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Row page JSON should use new ?_extra= format 1649791661  
1492206593 https://github.com/simonw/datasette/issues/2035#issuecomment-1492206593 https://api.github.com/repos/simonw/datasette/issues/2035 IC_kwDOBm6k_c5Y8UQB simonw 9599 2023-03-31T16:09:08Z 2023-03-31T16:09:08Z OWNER

I could ship this as part of: - #2049 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Potential feature: special support for `?a=1&a=2` on the query page 1615692818  
1491357104 https://github.com/simonw/datasette/issues/1989#issuecomment-1491357104 https://api.github.com/repos/simonw/datasette/issues/1989 IC_kwDOBm6k_c5Y5E2w xavdid 1231935 2023-03-31T06:17:23Z 2023-03-31T06:18:05Z NONE

I'm running into a similar use case as pax above- I made a nice view that just has the data I'm interested in (which doesn't include the id, since it's not important in this case). But, by excluding id from the view, I can't do fts queries against it because the view has no id field to tie to rowid:

ERROR: conn=<sqlite3.Connection object at 0x106521210>, sql = 'select time, text, permalink, num_children from nice where id in (select rowid from items_fts where items_fts match :search) limit 101', params = {'search': 'whatever'}: no such column: id

It works fine when I include id in my view, but now my nice view is cluttered up. Would be great to hide it permanently in the config.json.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Suggestion: Hiding columns 1531991339  
1489530555 https://github.com/simonw/datasette/issues/2049#issuecomment-1489530555 https://api.github.com/repos/simonw/datasette/issues/2049 IC_kwDOBm6k_c5YyG67 simonw 9599 2023-03-30T00:51:27Z 2023-03-30T00:51:27Z OWNER

I'd really like to refactor all of the extras functions into a datasette/extras.py module. The table ones currently rely a LOT on local variables in scope though, so I would need to rewrite those such that EVERY dependency they take is passed to asyncinject explicitly.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Custom SQL queries should use new JSON ?_extra= format 1646734246  
1489530037 https://github.com/simonw/datasette/issues/2049#issuecomment-1489530037 https://api.github.com/repos/simonw/datasette/issues/2049 IC_kwDOBm6k_c5YyGy1 simonw 9599 2023-03-30T00:50:30Z 2023-03-30T00:50:30Z OWNER

Two things to consider here: _shape= and _extra=.

Most of the shapes make sense, with the exception of ?_shape=object since we don't know which column we would use as a primary key.

Looking at the (undocumented) list of extras from the table view, here are the ones I think make sense:

  • count - YES
  • facet_results - no
  • facets_timed_out - no
  • suggested_facets - no
  • human_description_en - no
  • next_url - MAYBE
  • columns - YES
  • primary_keys - no
  • display_columns - YES
  • display_rows - YES
  • debug - YES?
  • request - YES
  • query - YES
  • metadata - YES
  • extras - YES
  • database - YES
  • table - no
  • database_color - no?
  • table_actions - no
  • filters - no
  • renderers - YES
  • custom_table_templates - no
  • sorted_facet_results - no
  • table_definition - no
  • view_definition - no
  • is_view - no
  • private - YES
  • expandable_columns - no
  • form_hidden_args - no

Just the YES ones:

  • count - this is new
  • columns
  • display_columns
  • display_rows
  • debug
  • request
  • query
  • metadata
  • extras
  • database
  • renderers
  • private

The count one is interesting - I think I can provide that by optionally running select count(*) from (inner query). It's a new feature though and not one I want to expose on the HTML view since it could result in poor performance - but having it as an extra that API users can opt into may make sense.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Custom SQL queries should use new JSON ?_extra= format 1646734246  
1489526501 https://github.com/simonw/datasette/issues/2049#issuecomment-1489526501 https://api.github.com/repos/simonw/datasette/issues/2049 IC_kwDOBm6k_c5YyF7l simonw 9599 2023-03-30T00:44:05Z 2023-03-30T00:44:05Z OWNER

As part of this I should be able to figure out which bits of the new code I wrote for the table view should actually be shared with the query view. That stuff is mostly going to be from this commit: https://github.com/simonw/datasette/commit/d97e82df3c8a3f2e97038d7080167be9bb74a68d

Here's the existing QueryView class I need to replace:

https://github.com/simonw/datasette/blob/4c1e277edbd783d06840d3f9b20bf00783478ce4/datasette/views/database.py#L215-L532

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Custom SQL queries should use new JSON ?_extra= format 1646734246  
1489306198 https://github.com/simonw/datasette/pull/2034#issuecomment-1489306198 https://api.github.com/repos/simonw/datasette/issues/2034 IC_kwDOBm6k_c5YxQJW wenhoujx 4370201 2023-03-29T20:56:19Z 2023-03-29T20:56:19Z NONE

@simonw any idea why the test coverage check fails? 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 remove an unused `app` var in cli.py 1613974869  
1489110168 https://github.com/dogsheep/hacker-news-to-sqlite/pull/6#issuecomment-1489110168 https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/6 IC_kwDODtX3eM5YwgSY xavdid 1231935 2023-03-29T18:36:16Z 2023-03-29T18:36:16Z NONE

@simonw can you take a look when you have a chance?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add permalink virtual field to items table 1641117021  
1488010837 https://github.com/simonw/datasette/issues/262#issuecomment-1488010837 https://api.github.com/repos/simonw/datasette/issues/262 IC_kwDOBm6k_c5YsT5V simonw 9599 2023-03-29T06:22:21Z 2023-03-29T06:22:21Z OWNER

I need to get the arbitrary query page to return the same format. It likely won't have nearly as many extras.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add ?_extra= mechanism for requesting extra properties in JSON 323658641  
1384039611 https://github.com/simonw/datasette/pull/1992#issuecomment-1384039611 https://api.github.com/repos/simonw/datasette/issues/1992 IC_kwDOBm6k_c5SfsS7 codecov[bot] 22429695 2023-01-16T13:12:32Z 2023-03-29T06:10:23Z NONE

Codecov Report

Patch coverage has no change and project coverage change: -0.31 :warning:

Comparison is base (c025b01) 92.41% compared to head (bfaa59b) 92.11%.

:exclamation: Current head bfaa59b differs from pull request most recent head 64dd698. Consider uploading reports for the commit 64dd698 to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1992 +/- ## ========================================== - Coverage 92.41% 92.11% -0.31% ========================================== Files 38 38 Lines 5750 5555 -195 ========================================== - Hits 5314 5117 -197 - Misses 436 438 +2 ``` [see 9 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/1992/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Bump blacken-docs from 1.12.1 to 1.13.0 1534904478  
1487999503 https://github.com/simonw/datasette/pull/2014#issuecomment-1487999503 https://api.github.com/repos/simonw/datasette/issues/2014 IC_kwDOBm6k_c5YsRIP dependabot[bot] 49699333 2023-03-29T06:09:11Z 2023-03-29T06:09:11Z CONTRIBUTOR

Superseded by #2047.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Bump black from 22.12.0 to 23.1.0 1566081801  
1487998788 https://github.com/simonw/datasette/pull/2014#issuecomment-1487998788 https://api.github.com/repos/simonw/datasette/issues/2014 IC_kwDOBm6k_c5YsQ9E simonw 9599 2023-03-29T06:08:23Z 2023-03-29T06:08:23Z OWNER

@dependabot recreate

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Bump black from 22.12.0 to 23.1.0 1566081801  
1486944644 https://github.com/simonw/datasette/pull/2043#issuecomment-1486944644 https://api.github.com/repos/simonw/datasette/issues/2043 IC_kwDOBm6k_c5YoPmE dependabot[bot] 49699333 2023-03-28T13:58:20Z 2023-03-28T13:58:20Z CONTRIBUTOR

Superseded by #2046.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Bump furo from 2022.12.7 to 2023.3.23 1639446870  
1484277416 https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484277416 https://api.github.com/repos/simonw/sqlite-utils/issues/388 IC_kwDOCGYnMM5YeEao simonw 9599 2023-03-26T23:43:12Z 2023-03-26T23:43:12Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Link to stable docs from older versions 1114543475  
1484276946 https://github.com/simonw/datasette/issues/1608#issuecomment-1484276946 https://api.github.com/repos/simonw/datasette/issues/1608 IC_kwDOBm6k_c5YeETS simonw 9599 2023-03-26T23:41:11Z 2023-03-26T23:41:11Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Documentation should clarify /stable/ vs /latest/ 1109808154  
1484276302 https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484276302 https://api.github.com/repos/simonw/sqlite-utils/issues/388 IC_kwDOCGYnMM5YeEJO simonw 9599 2023-03-26T23:39:27Z 2023-03-26T23:39:27Z OWNER

Actually final Datasette fix was: ```html+jinja {% block scripts %} {{ super() }}

<script> document.addEventListener("DOMContentLoaded", function() { // Show banner linking to /stable/ if this is a /latest/ page if (!/\/latest\//.test(location.pathname)) { return; } var stableUrl = location.pathname.replace("/latest/", "/stable/"); // Check it's not a 404 fetch(stableUrl, { method: "HEAD" }).then((response) => { if (response.status === 200) { var warning = document.createElement("div"); warning.className = "admonition warning"; warning.innerHTML = `

Note

This documentation covers the development version of Datasette.

See this page for the current stable release.

`; var mainArticle = document.querySelector("article[role=main]"); mainArticle.insertBefore(warning, mainArticle.firstChild); } }); }); </script>

{% endblock %} ```

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Link to stable docs from older versions 1114543475  
1484274796 https://github.com/simonw/datasette/issues/1608#issuecomment-1484274796 https://api.github.com/repos/simonw/datasette/issues/1608 IC_kwDOBm6k_c5YeDxs simonw 9599 2023-03-26T23:38:18Z 2023-03-26T23:38:18Z OWNER

ChatGPT prompt:

`` jQuery(function ($) { // Show banner linking to /stable/ if this is a /latest/ page if (!/\/latest\//.test(location.pathname)) { return; } var stableUrl = location.pathname.replace("/latest/", "/stable/"); // Check it's not a 404 fetch(stableUrl, { method: "HEAD" }).then((response) => { if (response.status == 200) { var warning = $(

Note

This documentation covers the development version of Datasette.

See this page for the current stable release.

` ); warning.find("a").attr("href", stableUrl); $("article[role=main]").prepend(warning); } }); });

Rewrite this to use vanilla JavaScript, no jQuery ```

Got back:

javascript document.addEventListener("DOMContentLoaded", function() { // Show banner linking to /stable/ if this is a /latest/ page if (!/\/latest\//.test(location.pathname)) { return; } var stableUrl = location.pathname.replace("/latest/", "/stable/"); // Check it's not a 404 fetch(stableUrl, { method: "HEAD" }).then((response) => { if (response.status === 200) { var warning = document.createElement("div"); warning.className = "admonition warning"; warning.innerHTML = ` <p class="first admonition-title">Note</p> <p class="last"> This documentation covers the <strong>development version</strong> of Datasette. </p> <p> See <a href="${stableUrl}">this page</a> for the current stable release. </p> `; var mainArticle = document.querySelector("article[role=main]"); mainArticle.insertBefore(warning, mainArticle.firstChild); } }); });

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Documentation should clarify /stable/ vs /latest/ 1109808154  
1484272313 https://github.com/simonw/datasette/issues/1608#issuecomment-1484272313 https://api.github.com/repos/simonw/datasette/issues/1608 IC_kwDOBm6k_c5YeDK5 simonw 9599 2023-03-26T23:36:17Z 2023-03-26T23:36:17Z OWNER

Still getting this error:

Weird because jQuery is present in the devtools console.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Documentation should clarify /stable/ vs /latest/ 1109808154  
1484269326 https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484269326 https://api.github.com/repos/simonw/sqlite-utils/issues/388 IC_kwDOCGYnMM5YeCcO simonw 9599 2023-03-26T23:24:12Z 2023-03-26T23:28:14Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Link to stable docs from older versions 1114543475  
1484270067 https://github.com/simonw/datasette/issues/1608#issuecomment-1484270067 https://api.github.com/repos/simonw/datasette/issues/1608 IC_kwDOBm6k_c5YeCnz simonw 9599 2023-03-26T23:26:10Z 2023-03-26T23:27:42Z OWNER

That didn't work, because jQuery isn't yet defined that high up the template.

I should use {% block scripts %} instead: https://github.com/pradyunsg/furo/blob/05350110cf02d033ebb57147eb54e9371a63545a/src/furo/theme/furo/base.html#L89-L102

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Documentation should clarify /stable/ vs /latest/ 1109808154  
1484268146 https://github.com/simonw/datasette/issues/1608#issuecomment-1484268146 https://api.github.com/repos/simonw/datasette/issues/1608 IC_kwDOBm6k_c5YeCJy simonw 9599 2023-03-26T23:23:22Z 2023-03-26T23:23:22Z OWNER

This broke when I upgraded to the Furo theme.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Documentation should clarify /stable/ vs /latest/ 1109808154  
1483248966 https://github.com/simonw/datasette/pull/2031#issuecomment-1483248966 https://api.github.com/repos/simonw/datasette/issues/2031 IC_kwDOBm6k_c5YaJVG tmcl-it 82332573 2023-03-24T18:35:24Z 2023-03-24T18:35:24Z NONE

I've rebased my patch on the latest main. It should be ready to merge.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Expand foreign key references in row view as well 1605481359  
1457016788 https://github.com/simonw/datasette/pull/2031#issuecomment-1457016788 https://api.github.com/repos/simonw/datasette/issues/2031 IC_kwDOBm6k_c5W2E_U codecov[bot] 22429695 2023-03-06T21:18:18Z 2023-03-24T18:29:49Z NONE

Codecov Report

Patch coverage: 97.72% and project coverage change: -0.36 :warning:

Comparison is base (3feed1f) 92.46% compared to head (c8a2904) 92.11%.

:exclamation: Current head c8a2904 differs from pull request most recent head ef25867. Consider uploading reports for the commit ef25867 to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2031 +/- ## ========================================== - Coverage 92.46% 92.11% -0.36% ========================================== Files 38 38 Lines 5750 5556 -194 ========================================== - Hits 5317 5118 -199 - Misses 433 438 +5 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `92.57% <97.56%> (-3.35%)` | :arrow_down: | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.17% <100.00%> (+2.38%)` | :arrow_up: | | [datasette/views/row.py](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3Jvdy5weQ==) | `87.93% <100.00%> (+0.10%)` | :arrow_up: | ... and [8 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/2031/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Expand foreign key references in row view as well 1605481359  
1483009959 https://github.com/simonw/datasette/issues/1619#issuecomment-1483009959 https://api.github.com/repos/simonw/datasette/issues/1619 IC_kwDOBm6k_c5YZO-n henrikek 6613091 2023-03-24T15:38:04Z 2023-03-24T15:38:04Z NONE

I also have the same problem when running behind an apache proxy server with base_url. However, I have researched the problem a bit and have come to the conclusion that if you change _facet_date to _facet in the following https://github.com/simonw/datasette/blob/3feed1f66e2b746f349ee56970a62246a18bb164/datasette/facets.py#LL493C57-L493C68 , the facets work. But I'm not sure if this has other consequences?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 JSON link on row page is 404 if base_url setting is used 1121583414  
1482859217 https://github.com/simonw/datasette/pull/2043#issuecomment-1482859217 https://api.github.com/repos/simonw/datasette/issues/2043 IC_kwDOBm6k_c5YYqLR codecov[bot] 22429695 2023-03-24T14:05:54Z 2023-03-24T14:05:54Z NONE

Codecov Report

Patch coverage has no change and project coverage change: -0.06 :warning:

Comparison is base (3feed1f) 92.46% compared to head (85b9e51) 92.41%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2043 +/- ## ========================================== - Coverage 92.46% 92.41% -0.06% ========================================== Files 38 38 Lines 5750 5750 ========================================== - Hits 5317 5314 -3 - Misses 433 436 +3 ``` [see 2 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/2043/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Bump furo from 2022.12.7 to 2023.3.23 1639446870  
1480355670 https://github.com/simonw/datasette/issues/262#issuecomment-1480355670 https://api.github.com/repos/simonw/datasette/issues/262 IC_kwDOBm6k_c5YPG9W simonw 9599 2023-03-22T22:50:30Z 2023-03-22T22:50:30Z OWNER

I just landed this PR so this feature is now in main: - #1999

Still needs documentation and maybe some extra tests too.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add ?_extra= mechanism for requesting extra properties in JSON 323658641  
1480349156 https://github.com/simonw/datasette/pull/1999#issuecomment-1480349156 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YPFXk codecov[bot] 22429695 2023-03-22T22:40:58Z 2023-03-22T22:48:57Z NONE

Codecov Report

Patch coverage: 87.89% and project coverage change: -4.43 :warning:

Comparison is base (56b0758) 92.15% compared to head (921faae) 87.73%.

:exclamation: Current head 921faae differs from pull request most recent head 69a31cd. Consider uploading reports for the commit 69a31cd to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1999 +/- ## ========================================== - Coverage 92.15% 87.73% -4.43% ========================================== Files 38 38 Lines 5560 6066 +506 ========================================== + Hits 5124 5322 +198 - Misses 436 744 +308 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.61% <ø> (ø)` | | | [datasette/views/row.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3Jvdy5weQ==) | `87.82% <ø> (ø)` | | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `69.11% <86.76%> (-23.46%)` | :arrow_down: | | [datasette/renderer.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3JlbmRlcmVyLnB5) | `93.33% <90.90%> (-0.87%)` | :arrow_down: | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `92.78% <91.66%> (-2.39%)` | :arrow_down: | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.48% <100.00%> (-0.01%)` | :arrow_down: | | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `79.93% <100.00%> (ø)` | | | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | | | [datasette/publish/cloudrun.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3B1Ymxpc2gvY2xvdWRydW4ucHk=) | `97.29% <100.00%> (ø)` | | | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.59% <100.00%> (-0.27%)` | :arrow_down: | ... and [2 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/1999/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1480343044 https://github.com/simonw/datasette/pull/1999#issuecomment-1480343044 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YPD4E simonw 9599 2023-03-22T22:33:15Z 2023-03-22T22:33:15Z OWNER

This still needs documentation, but now the tests are passing I'm going to merge it into main!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1480339527 https://github.com/simonw/datasette/pull/1999#issuecomment-1480339527 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YPDBH simonw 9599 2023-03-22T22:28:54Z 2023-03-22T22:28:54Z OWNER

I hacked at the CSV stuff until it worked.

I need to clean it up though, but I can do that in this separate task: - #1101

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1480299765 https://github.com/simonw/datasette/pull/1999#issuecomment-1480299765 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YO5T1 simonw 9599 2023-03-22T21:44:03Z 2023-03-22T21:45:10Z OWNER

Oh this is a bit tricky.

I have a failing test because a plugin that uses the extra_css_urls hook can't see the columns for the page.

Turns out that bit comes from here:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/app.py#L1203-L1217

Which assumes the context has "columns" - but that's only now available if ?_extra=columns was passed.

Actually I think I can cheat here, since it's still getting the HTML context in order to render the template.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1480179217 https://github.com/simonw/datasette/pull/1999#issuecomment-1480179217 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YOb4R simonw 9599 2023-03-22T19:55:31Z 2023-03-22T21:34:02Z OWNER

I rebased from main. Now:

FAILED tests/test_csv.py::test_table_csv - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_cors_headers - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_no_header - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_blob_columns - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_download - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101 FAILED tests/test_csv.py::test_csv_trace - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...} FAILED tests/test_plugins.py::test_hook_render_cell_demo - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_plugins.py::test_hook_render_cell_async[/fixtures/simple_primary_key] - assert b'RENDER_CELL_ASYNC_RESULT' in b'<!DOCTYPE html>\n<html>\n<head>\n <title>Error 500</title>\n <link rel="stylesheet" href="/-/static/app.css?d5...ils != detailsClickedWithin\n ).forEach(details => details.open = false);\n});\n</script>\n\n\n\n\n</b...>', ...} == {'1+1': 2, 'c... 0xXXX>', ...} FAILED tests/test_permissions.py::test_permissions_checked[/fixtures/simple_primary_key-permissions3] - assert 500 in (200, 403) FAILED tests/test_table_html.py::test_table_csv_json_export_interface - assert 500 == 200 FAILED tests/test_table_html.py::test_table_metadata - assert 500 == 200 FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/simple_primary_key-expected_classes3] - assert 500 == 200 FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert 500 == 200 FAILED tests/test_plugins.py::test_hook_table_actions[simple_view] - AssertionError: assert [] == [{'href': '/'...simple_view'}] ERROR tests/test_internals_database.py::test_execute_write_fn_connection_exception - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_mtime_ns - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_mtime_ns_is_none_for_memory - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_is_mutable - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_database_memory_name - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_in_memory_databases_forbid_writes - Failed: Timeout >1.0s =============== 21 failed, 1275 passed, 2 skipped, 1 xfailed, 6 errors in 59.18s ================

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1480163485 https://github.com/simonw/datasette/pull/1999#issuecomment-1480163485 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YOYCd simonw 9599 2023-03-22T19:48:00Z 2023-03-22T19:48:00Z OWNER

Getting close now! Only 13 failures left, mostly relating to CSV. FAILED tests/test_csv.py::test_table_csv - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_cors_headers - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_no_header - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_blob_columns - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_download - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101 FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...} FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets' FAILED tests/test_csv.py::test_csv_trace - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...} FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'1+1': 2, 'c... 0xXXX>', ...} == {'1+1': 2, 'c... 0xXXX>', ...} =============== 13 failed, 1287 passed, 2 skipped, 1 xfailed in 61.57s (0:01:01) ================

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1480154453 https://github.com/simonw/datasette/pull/1999#issuecomment-1480154453 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YOV1V simonw 9599 2023-03-22T19:41:39Z 2023-03-22T19:43:04Z OWNER

To replace this code: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L122

Maybe datasette.render_template() should optionally accept a list of templates.

https://docs.datasette.io/en/stable/internals.html#await-render-template-template-context-none-request-none - turns out it does already:

If this is a list of template file names then the first one that exists will be loaded and rendered.

It doesn't have an easy way to populate that select_templates debug template variable though.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1477155602 https://github.com/simonw/datasette/issues/2041#issuecomment-1477155602 https://api.github.com/repos/simonw/datasette/issues/2041 IC_kwDOBm6k_c5YC5sS simonw 9599 2023-03-21T01:17:43Z 2023-03-21T01:17:43Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Remove obsolete table POST code 1633077183  
1477144853 https://github.com/simonw/datasette/issues/2041#issuecomment-1477144853 https://api.github.com/repos/simonw/datasette/issues/2041 IC_kwDOBm6k_c5YC3EV simonw 9599 2023-03-21T01:02:13Z 2023-03-21T01:02:13Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Remove obsolete table POST code 1633077183  
1477082852 https://github.com/simonw/datasette/pull/1999#issuecomment-1477082852 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YCn7k simonw 9599 2023-03-20T23:27:25Z 2023-03-20T23:27:25Z OWNER

Urgh getting CSV to work is going to be so hard, because the logic for that currently lives in a huge chunk of code in BaseView which depends on the old design of the data() method:

https://github.com/simonw/datasette/blob/4bb49848697e40b8b9a1557be42b8e59eac965b3/datasette/views/base.py#L177-L343

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1476860334 https://github.com/simonw/datasette/pull/1999#issuecomment-1476860334 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YBxmu simonw 9599 2023-03-20T20:06:56Z 2023-03-20T22:09:03Z OWNER

pytest -n 8 locally says:

FAILED tests/test_canned_queries.py::test_canned_query_form_csrf_hidden_field[add_name_specify_id-True] - assert '<input type="hidden" name="csrftoken" value="' in '<!DOCTYPE html>\n<html>\n<head>\n <title>Error 500</title>\n <link rel="stylesheet" href="/-/static/app.css?d59...ils != detailsClickedWithin\n ).forEach(details => details.open = false);\n});\n</... FAILED tests/test_canned_queries.py::test_canned_query_form_csrf_hidden_field[add_name-True] - assert '<input type="hidden" name="csrftoken" value="' in '<!DOCTYPE html>\n<html>\n<head>\n <title>Error 500</title>\n <link rel="stylesheet" href="/-/static/app.css?d59...ils != detailsClickedWithin\n ).forEach(details => details.open = false);\n});\n</... FAILED tests/test_canned_queries.py::test_canned_query_permissions - AssertionError: assert 200 == 500 FAILED tests/test_canned_queries.py::test_canned_query_with_named_parameter - KeyError: 'rows' FAILED tests/test_canned_queries.py::test_canned_write_custom_template - assert 500 == 200 FAILED tests/test_canned_queries.py::test_canned_write_query_disabled_for_immutable_database - assert 500 == 200 FAILED tests/test_canned_queries.py::test_custom_params - assert '<input type="text" id="qp3" name="extra" value="foo">' in '<!DOCTYPE html>\n<html>\n<head>\n <title>Error 500</title>\n <link rel="stylesheet" href="/-/static/app.css?d59...ils != detailsClickedWithin\n ).forEach(details => details.open = false);\... FAILED tests/test_canned_queries.py::test_insert - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_insert_error - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[False-True] - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[True-True] - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_vary_header - KeyError: 'vary' FAILED tests/test_csv.py::test_csv_trace - AssertionError: assert 'application/...charset=utf-8' == 'text/html; charset=utf-8' FAILED tests/test_csv.py::test_table_csv - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_blob_columns - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_download - AssertionError: assert 'application/...charset=utf-8' == 'text/csv; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_no_header - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101 FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_counts - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_facets - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_csv.py::test_table_csv_with_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_facets.py::test_array_facet_handle_duplicate_tags - KeyError: 'facet_results' FAILED tests/test_facets.py::test_conflicting_facet_names_json - AssertionError: assert {'results', 'timed_out'} == {'created', '...gs', 'tags_2'} FAILED tests/test_facets.py::test_facet_size - KeyError: 'suggested_facets' FAILED tests/test_facets.py::test_json_array_with_blanks_and_nulls - KeyError: 'suggested_facets' FAILED tests/test_facets.py::test_other_types_of_facet_in_metadata - assert '<strong>created (date)\n' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: facetable: 15 rows</title>\n <link rel="stylesheet" href="/-...name": "table", "request_path": "/fixtures/facetable", "added": 15, "columns": null};</script>\n\n\n\n</bo... FAILED tests/test_html.py::test_alternate_url_json[/fixtures/facetable-http://localhost/fixtures/facetable.json] - KeyError: 'link' FAILED tests/test_html.py::test_alternate_url_json[/fixtures/table~2Fwith~2Fslashes~2Ecsv-http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json] - KeyError: 'link' FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert '<!-- Templates considered: table-fixtures-simple_primary_key.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: simple_primary_key: 5 rows</title>\n <link rel="stylesheet" ...able", "request_path": "/fixtures/simple_primary... FAILED tests/test_html.py::test_templates_considered[/fixtures/table~2Fwith~2Fslashes~2Ecsv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] - assert '<!-- Templates considered: table-fixtures-tablewithslashescsv-fa7563.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: table/with/slashes.csv: 1 row</title>\n <link rel="styleshee...quest_path": "/fixtures/table~2Fwith~2Fs... FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...} FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...} FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets' FAILED tests/test_plugins.py::test_hook_register_magic_parameters - KeyError: 'ds_csrftoken' FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'next': None...', ...}, ...]} == {'1+1': 2, 'c... 0xXXX>', ...} FAILED tests/test_plugins.py::test_hook_register_output_renderer_no_parameters - assert b'Hello' == b'{"ok": true... "n": null}]}' FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_broken_value - assert 200 == 500 FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_response - assert {'next': None...', ...}, ...]} == {'this_is': 'json'} === 39 failed, 1259 passed, 2 skipped, 1 xfailed in 58.07s ===

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1476898261 https://github.com/simonw/datasette/pull/1999#issuecomment-1476898261 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YB63V simonw 9599 2023-03-20T20:37:52Z 2023-03-20T20:37:52Z OWNER

Manual testing spotted a bug.

/content/repos.json?owner=9599&_facet_array=topics - does not return a facet_results key.

/content/repos.json?owner=9599&_facet_array=topics&_facet=owner does.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1476854645 https://github.com/simonw/datasette/pull/1999#issuecomment-1476854645 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YBwN1 simonw 9599 2023-03-20T20:02:24Z 2023-03-20T20:02:24Z OWNER

``` def test_routes(routes, path, expected_class, expected_matches): match, view = resolve_routes(routes, path) if expected_class is None: assert match is None else:

      assert view.view_class.__name__ == expected_class

E AttributeError: 'function' object has no attribute 'view_class' ```

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1476851525 https://github.com/simonw/datasette/pull/1999#issuecomment-1476851525 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YBvdF simonw 9599 2023-03-20T19:59:51Z 2023-03-20T20:01:40Z OWNER

Three failures in test_html.py: FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert '<!-- Templates considered: table-fixtures-simple_primary_key.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: simple_primary_key: 5 rows</title>\n <link rel="stylesheet" ...able", "request_path": "/fixtures/simple_primary_key", "ad... FAILED tests/test_html.py::test_templates_considered[/fixtures/table~2Fwith~2Fslashes~2Ecsv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] - assert '<!-- Templates considered: table-fixtures-tablewithslashescsv-fa7563.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures: table/with/slashes.csv: 1 row</title>\n <link rel="styleshee...quest_path": "/fixtures/table~2Fwith~2Fslashes~2Ec... FAILED tests/test_html.py::test_alternate_url_json[/fixtures/facetable-http://localhost/fixtures/facetable.json] - KeyError: 'link' FAILED tests/test_html.py::test_alternate_url_json[/fixtures/table~2Fwith~2Fslashes~2Ecsv-http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json] - KeyError: 'link'

In CI I'm getting this: https://github.com/simonw/datasette/actions/runs/4472331872/jobs/7858360154

====== 44 failed, 1222 passed, 1 skipped, 1 xfailed in 171.24s (0:02:51) =======

Including failures in tests/test_plugins.py and tests/test_routes.py.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1476781401 https://github.com/simonw/datasette/pull/1999#issuecomment-1476781401 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5YBeVZ simonw 9599 2023-03-20T19:01:35Z 2023-03-20T19:01:35Z OWNER

Down to just these failures: FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'attrs': {'class': ['col-Link'], 'scope': 'col', 'data-column': 'Link', 'data-column-type': '', 'data-column-not-null': '0', 'data-is-pk': '0'}, 'a_href': None}, {'attrs': {'class': ['col-pk1'], 'scope': 'col', 'data-column': 'pk1', 'data-column-typ... FAILED tests/test_table_html.py::test_table_html_simple_primary_key - AssertionError: assert 'id\xa0▼' == 'id' FAILED tests/test_table_html.py::test_table_csv_json_export_interface - AssertionError: assert ['/fixtures/simple_primary_key.json?id__gt=2', '/fixtures/simple_primary_key.testall?id__gt=2', '/fixtures/simple_primary_key.testnone?id__gt=2', '/fixtures/simple_primary_key.testresponse?id__gt=2', '/fixtures/simple_primary_key.csv?id__gt=2&_size=ma... FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys - AssertionError: assert ['/fixtures/facetable.json?_labels=on', '/fixtures/facetable.testall?_labels=on', '/fixtures/facetable.testnone?_labels=on', '/fixtures/facetable.testresponse?_labels=on', '/fixtures/facetable.csv?_labels=on&_size=max', '#export'] == ['/', '', '#export'] FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key - AssertionError: assert 'rowid\xa0▼' == 'rowid' FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/infinity-expected_column_options0] - AssertionError: assert ['- column -', 'rowid', 'value'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/primary_key_multiple_columns-expected_column_options1] - AssertionError: assert ['- column -', 'id', 'content', 'content2'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/compound_primary_key-expected_column_options2] - AssertionError: assert ['- column -', 'pk1', 'pk2', 'content'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns - AssertionError: assert ['- column -'] == ['- column -', 'pk1', 'pk2', 'content', 'sortable_with_nulls', 'sortable_with_nulls_2', 'text', 'sortable'] FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id', 'name'] FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk', 'name'] === 11 failed, 57 passed in 7.59s ===

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1460759358 https://github.com/simonw/datasette/pull/1999#issuecomment-1460759358 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XEWs- simonw 9599 2023-03-08T19:48:13Z 2023-03-20T18:47:12Z OWNER

Breaking this down into smaller steps:

  • [x] Get ?_next= working
  • [x] Implement extensions - so .json is needed again for the JSON version, and anything without an extension is passed through a new code path for HTML
  • [ ] That HTML view should only access JSON data, which can be seen by using .context - this will require a lot of updates to templates (it may be necessary to still provide access to some helper functions though). This will form the basis of the ambition to fully document the template context.
  • [ ] Get a bunch of the existing table HTML and JSON tests to pass
  • [ ] Use those tests to refactor the nasty _next code, see https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469
  • [ ] Figure out how the register_output_renderer(datasette) plugin hook should work
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1475074311 https://github.com/simonw/datasette/pull/1999#issuecomment-1475074311 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5X69kH simonw 9599 2023-03-19T02:16:08Z 2023-03-19T02:17:33Z OWNER

Next big challenge: fix all the tests in test_table_html.py:

FAILED tests/test_table_html.py::test_add_filter_redirects - assert 500 == 302 FAILED tests/test_table_html.py::test_existing_filter_redirects - assert 500 == 302 FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood-expected_hidden0] - AssertionError: assert {} == {'_facet': '_neighborhood'} FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_where=1+=+1&_col=_city_id-expected_hidden1] - AssertionError: assert {} == {'_col': '_ci...ere': '1 = 1'} FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood&_neighborhood__exact=Downtown-expected_hidden2] - AssertionError: assert {} == {'_facet': '_neighborhood'} FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood&_city_id__gt=1-expected_hidden3] - AssertionError: assert {} == {'_facet': '_neighborhood'} FAILED tests/test_table_html.py::test_empty_search_parameter_gets_removed - assert 500 == 302 FAILED tests/test_table_html.py::test_searchable_view_persists_fts_table - AssertionError: assert [('_fts_table...ts_pk', 'pk')] == [] FAILED tests/test_table_html.py::test_sort_by_desc_redirects - assert 500 == 302 FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...] FAILED tests/test_table_html.py::test_facets_persist_through_filter_form - AssertionError: assert [] == [('_facet', '...ray', 'tags')] FAILED tests/test_table_html.py::test_next_does_not_persist_in_hidden_field - AssertionError: assert [] == [('_size', '1')] FAILED tests/test_table_html.py::test_table_html_simple_primary_key - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_table_csv_json_export_interface - AssertionError: assert ['/fixtures/s...x', '#export'] == ['/', '', '#export'] FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys - AssertionError: assert ['/fixtures/f...x', '#export'] == ['/', '', '#export'] FAILED tests/test_table_html.py::test_table_html_no_primary_key - AssertionError: assert ['content', 'a', 'b', 'c'] == ['a', 'b', 'c'] FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_table_html_compound_primary_key - AssertionError: assert 'Link' == 'pk1' FAILED tests/test_table_html.py::test_table_html_foreign_key_links - assert [['<td class=...">\xa0</td>']] == [['<td class=...">\xa0</td>']] FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels - assert [['<td class=...str">b</td>']] == [['<td class=...str">b</td>']] FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['<td class=...1</em></td>']] == [['<td class=...1</em></td>']] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/infinity-expected_column_options0] - AssertionError: assert ['- column -'...wid', 'value'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/primary_key_multiple_columns-expected_column_options1] - AssertionError: assert ['- column -'...', 'content2'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/compound_primary_key-expected_column_options2] - AssertionError: assert ['- column -'...2', 'content'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns - AssertionError: assert ['- column -'] == ['- column -'...nulls_2', ...] FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references - assert [['<td class=...ne</a></td>']] == [['<td class=...ne</a></td>']] FAILED tests/test_table_html.py::test_view_html - assert None is not None FAILED tests/test_table_html.py::test_extra_where_clauses - assert [('_where', "...'_city_id=1')] == [] FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10-expected_hidden0] - AssertionError: assert [] == [('_size', '10')] FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10&_ignore=1&_ignore=2-expected_hidden1] - AssertionError: assert [] == [('_size', '1...ignore', '2')] FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated[/fixtures/searchable?_sort=text2&_where=1-expected_hidden3] - AssertionError: assert [] == [('_where', '1')] FAILED tests/test_table_html.py::test_binary_data_display_in_table - assert [['<td class=...">\xa0</td>']] == [['<td class=...">\xa0</td>']] FAILED tests/test_table_html.py::test_metadata_sort - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_metadata_sort_desc - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood-2-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 2 FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood&_facet_size=50-5-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 5 FAILED tests/test_table_html.py::test_facet_total - assert 500 == 200

Deduped that's 30 tests:

FAILED tests/test_table_html.py::test_add_filter_redirects FAILED tests/test_table_html.py::test_binary_data_display_in_table FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys FAILED tests/test_table_html.py::test_empty_search_parameter_gets_removed FAILED tests/test_table_html.py::test_existing_filter_redirects FAILED tests/test_table_html.py::test_extra_where_clauses FAILED tests/test_table_html.py::test_facet_more_links FAILED tests/test_table_html.py::test_facet_total FAILED tests/test_table_html.py::test_facets_persist_through_filter_form FAILED tests/test_table_html.py::test_metadata_sort FAILED tests/test_table_html.py::test_metadata_sort_desc FAILED tests/test_table_html.py::test_next_does_not_persist_in_hidden_field FAILED tests/test_table_html.py::test_other_hidden_form_fields FAILED tests/test_table_html.py::test_reflected_hidden_form_fields FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated FAILED tests/test_table_html.py::test_searchable_view_persists_fts_table FAILED tests/test_table_html.py::test_sort_by_desc_redirects FAILED tests/test_table_html.py::test_sort_links FAILED tests/test_table_html.py::test_table_csv_json_export_interface FAILED tests/test_table_html.py::test_table_html_compound_primary_key FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels FAILED tests/test_table_html.py::test_table_html_filter_form_column_options FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column FAILED tests/test_table_html.py::test_table_html_foreign_key_links FAILED tests/test_table_html.py::test_table_html_no_primary_key FAILED tests/test_table_html.py::test_table_html_simple_primary_key FAILED tests/test_table_html.py::test_view_html

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1475074025 https://github.com/simonw/datasette/pull/1999#issuecomment-1475074025 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5X69fp simonw 9599 2023-03-19T02:14:28Z 2023-03-19T02:14:51Z OWNER

I had to replicate quite a bit of this logic from base.py:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L526-L544

I should refactor this when I move the canned / arbitrary query views away from that base class too.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1475016834 https://github.com/simonw/datasette/pull/1999#issuecomment-1475016834 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5X6viC simonw 9599 2023-03-18T22:30:31Z 2023-03-18T22:30:31Z OWNER

test_paginate_using_link_header will be tricky.

The reason the tests are failing is that json_renderer() attempts to populate the link header using data["next_url"] - but that's not present unless ?_extra=next_url has been passed:

https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/renderer.py#L101-L102

But I can only rely on next being present, not next_url.

I thought I could maybe assemble the link header using next, by turning that into a next_url link - but there's some custom logic which kicks in for pagination against SQL views (offset/limit pagination) to calculate the next_url which isn't easily replicable at the json_renderer() layer, in the _next_value_and_url() utility function:

https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/views/table.py#L2275-L2282

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1475003292 https://github.com/simonw/datasette/pull/1999#issuecomment-1475003292 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5X6sOc simonw 9599 2023-03-18T21:46:20Z 2023-03-18T21:46:20Z OWNER

Now 25 failures in test_table_api.py: FAILED tests/test_table_api.py::test_expand_labels - assert {'2': {'pk': 2, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', '_city_id': 1, '_neighborhood': 'Dogpatch', 'tags': '["tag1", "tag3"]', 'complex_array'... FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_co... FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1, 'value': inf}, {'rowid': 2, 'value': -inf}, {'rowid': 3, 'value': 1.5}] == [{'rowid': 1, 'value': None}, {'rowid': 2, 'value': None}, {'rowid': 3, 'value'... FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_compoun... FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'rowid': 1, 'data': "b'\\x15\\x1c\\x02\\xc7\\xad\\x05\\xfe'"}, {'rowid': 2, 'data': "b'\\x15\\x1c\\x03\\xc7\\xad\\x05\\xfe'"}, {'rowid': 3, 'data': None}] == [{'rowid': 1, 'data': {'... FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}\n{"rowid": 2, "data": {"$base64": true, "encoded": "FRwDx60F/g=="}}\n{"rowid": 3, "data": null}] - assert '{"ok": false, "error": "Object of type bytes is not JSON serializable", "status": 500, "title": null}' == '{"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}\n{"rowid"... FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21 FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns' ============================================================================= 25 failed, 86 passed, 1 xfailed in 7.18s =============================================================================

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1474704790 https://github.com/simonw/datasette/pull/1999#issuecomment-1474704790 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5X5jWW simonw 9599 2023-03-18T04:52:59Z 2023-03-18T04:52:59Z OWNER

Here are the next set of tests to get passing: % pytest tests/test_table_api.py FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id-expected_facet_results0] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id&state=MI-expected_facet_results1] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int-expected_facet_results2] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int&planet_int=1-expected_facet_results3] - KeyError: 'name' FAILED tests/test_table_api.py::test_suggested_facets - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_allow_facet_off - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_suggest_facets_off - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nofacet[True] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nofacet[False] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nosuggest[True] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nosuggest[False] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nocount[True-None] - KeyError: 'count' FAILED tests/test_table_api.py::test_nocount[False-15] - KeyError: 'count' FAILED tests/test_table_api.py::test_expand_labels - AssertionError: assert {'13': {'_cit...:00:00', ...}} == {'13': {'_cit...:00:00', ...}} FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'forei...l': '1', ...}} == {'1': {'forei...': '1'}, ...}} FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1,...'value': 1.5}] == [{'rowid': 1,...'value': 1.5}] FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'foreign_ke...': None, ...}] == [{'foreign_ke...': None, ...}] FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'data': "b'..., 'rowid': 3}] == [{'data': {'$..., 'rowid': 3}] FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}\n{"rowid": 2, "data": {"$base64": true, "encoded": "FRwDx60F/g=="}}\n{"rowid": 3, "data": null}] - assert '{"ok": false...title": null}' == '{"rowid": 1,..."data": null}' FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21 FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns' ============================================================================= 38 failed, 73 passed, 1 xfailed in 7.25s =============================================================================

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1468898285 https://github.com/dogsheep/apple-notes-to-sqlite/issues/8#issuecomment-1468898285 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/8 IC_kwDOJHON9s5XjZvt RhetTbull 41546558 2023-03-14T22:00:21Z 2023-03-14T22:00:21Z NONE

Well that's embarrassing. I made a fork using macnotesapp and it's actually slower. This is because the Scripting Bridge sometimes fails to return the folder and thus macnotesapp resorts to AppleScript in this situation. The repeated AppleScript calls on a large library are slower than your "slurp it all in" approach. I've got some ideas about how to improve this--will make another attempt if I can fix the issues.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Increase performance using macnotesapp 1617823309  
1465315726 https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465315726 https://api.github.com/repos/simonw/sqlite-utils/issues/531 IC_kwDOCGYnMM5XVvGO eyeseast 25778 2023-03-12T22:21:56Z 2023-03-12T22:21:56Z CONTRIBUTOR

Exactly, that's what I was running into. On my M2 MacBook, SpatiaLite ends up in what is -- for the moment -- a non-standard location, so even when I passed in the location with --load-extension, I still hit an error on create-spatial-index.

What I learned doing this originally is that SQLite needs to load the extension for each connection, even if all the SpatiaLite stuff is already in the database. So that's why init_spatialite() gets called again.

Here's the code where I hit the error: https://github.com/eyeseast/boston-parcels/blob/main/Makefile#L30 It works using this branch.

I'm not attached to this solution if you can think of something better. And I'm not sure, TBH, my test would actually catch what I'm after here.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add paths for homebrew on Apple silicon 1620164673  
1465303378 https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465303378 https://api.github.com/repos/simonw/sqlite-utils/issues/533 IC_kwDOCGYnMM5XVsFS simonw 9599 2023-03-12T21:24:04Z 2023-03-12T21:24:04Z OWNER

Upgraded to Sphinx 6 locally and got the same error:

% just docs Cogging README.md Cogging docs/changelog.rst Cogging docs/cli-reference.rst Cogging docs/cli.rst Cogging docs/contributing.rst Cogging docs/index.rst Cogging docs/installation.rst Cogging docs/python-api.rst Cogging docs/reference.rst sphinx-autobuild -a -b html "." "_build" --watch ../sqlite_utils [sphinx-autobuild] > sphinx-build -b html -a /Users/simon/Dropbox/Development/sqlite-utils/docs /Users/simon/Dropbox/Development/sqlite-utils/docs/_build Running Sphinx v6.1.3 loading pickled environment... failed failed: Can't get attribute '_stable_repr_object' on <sphinx.deprecation._ModuleWrapper object at 0x103f155d0> building [mo]: all of 0 po files writing output... building [html]: all source files updating environment: [new config] 8 added, 0 changed, 0 removed reading sources... [ 12%] changelog Exception occurred: File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/sphinx/ext/extlinks.py", line 103, in role title = caption % part ~~~~~~~~^~~~~~ TypeError: not all arguments converted during string formatting The full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1ey36c1n.log, if you want to report the issue to the developers.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ReadTheDocs error: not all arguments converted during string formatting 1620516340  
1465302936 https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465302936 https://api.github.com/repos/simonw/sqlite-utils/issues/533 IC_kwDOCGYnMM5XVr-Y simonw 9599 2023-03-12T21:22:09Z 2023-03-12T21:22:09Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ReadTheDocs error: not all arguments converted during string formatting 1620516340  
1465302343 https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465302343 https://api.github.com/repos/simonw/sqlite-utils/issues/531 IC_kwDOCGYnMM5XVr1H simonw 9599 2023-03-12T21:19:13Z 2023-03-12T21:19:13Z OWNER

Aah, I think I see why you wrote it like that.

The problem is that init_spatialite() does other stuff too:

https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L1161-L1171

So it needs to be able to load the SpatiaLite extension from the correct place, and THEN run select InitSpatialMetadata() to configure the database, if needed.

So the problem you're trying to solve here is to let people optionally pass in the path to SpatiaLite if it's not one of the ones that are searched by default.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add paths for homebrew on Apple silicon 1620164673  
1465208436 https://github.com/simonw/datasette/issues/236#issuecomment-1465208436 https://api.github.com/repos/simonw/datasette/issues/236 IC_kwDOBm6k_c5XVU50 sopel 545193 2023-03-12T14:04:15Z 2023-03-12T14:04:15Z NONE

I keep coming back to this in search for the related exploration, so I'll just link it now:

@simonw has meanwhile researched how to deploy Datasette to AWS Lambda using function URLs and Mangum via https://github.com/simonw/public-notes/issues/6 and concluded that's everything I need to know in order to build a datasette-publish-lambda plugin.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette publish lambda plugin 317001500  
1465038901 https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465038901 https://api.github.com/repos/simonw/sqlite-utils/issues/531 IC_kwDOCGYnMM5XUrg1 codecov[bot] 22429695 2023-03-11T22:29:19Z 2023-03-11T22:29:19Z NONE

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.03 :tada:

Comparison is base (fc221f9) 96.25% compared to head (0e4750d) 96.29%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #531 +/- ## ========================================== + Coverage 96.25% 96.29% +0.03% ========================================== Files 6 6 Lines 2671 2671 ========================================== + Hits 2571 2572 +1 + Misses 100 99 -1 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.33% <ø> (ø)` | | | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `95.13% <ø> (ø)` | | | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.26% <100.00%> (+0.09%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add paths for homebrew on Apple silicon 1620164673  
1464796494 https://github.com/dogsheep/healthkit-to-sqlite/issues/24#issuecomment-1464796494 https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24 IC_kwDOC8tyDs5XTwVO Mjboothaus 956433 2023-03-11T02:23:42Z 2023-03-11T02:23:42Z NONE

@simonw - maybe put in some error handling to trap for poorly formed XML (from Apple engineers) so that it suggests that there are problems with export.zip rather than odd looking Python errors :)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 DOC: xml.etree.ElementTree.ParseError due to healthkit version 12  1515883470  
1464786643 https://github.com/dogsheep/healthkit-to-sqlite/issues/24#issuecomment-1464786643 https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24 IC_kwDOC8tyDs5XTt7T Mjboothaus 956433 2023-03-11T02:01:27Z 2023-03-11T02:01:27Z NONE

Thanks for reporting this and providing a solution -- I was puzzled by this error when I revisited my walking data and experienced this issues. I haven't tried the fix yet.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 DOC: xml.etree.ElementTree.ParseError due to healthkit version 12  1515883470  
1463113856 https://github.com/simonw/datasette/pull/1999#issuecomment-1463113856 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XNViA simonw 9599 2023-03-10T02:13:15Z 2023-03-10T02:13:15Z OWNER

Idea for if this change ends up making a bunch of breaking changes to the templates (which I think it should) - I can generate a GitHub diff link between the old and new templates and include that link in the 1.0 upgrade documentation to help people who wrote custom templates see what they might need to change - with minimal effort from myself to document those changes.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1463112173 https://github.com/simonw/datasette/issues/2038#issuecomment-1463112173 https://api.github.com/repos/simonw/datasette/issues/2038 IC_kwDOBm6k_c5XNVHt simonw 9599 2023-03-10T02:11:06Z 2023-03-10T02:11:06Z OWNER

Here's an example of something that would break:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/templates/database.html#L12-L15

Because metadata is an empty dictionary sometimes, so {{ metadata.title or database }} would raise an error and need to be replaced by {{ metadata.get("title") or database }}.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Consider a `strict_templates` setting 1618249044  
1463110978 https://github.com/simonw/datasette/issues/2038#issuecomment-1463110978 https://api.github.com/repos/simonw/datasette/issues/2038 IC_kwDOBm6k_c5XNU1C simonw 9599 2023-03-10T02:09:41Z 2023-03-10T02:09:41Z OWNER

I'm torn on this. It's useful for me right now for refactoring, but I feel like it should be a permanent thing, not a setting - or it should default to on and people should turn it off, but who would ever do that?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Consider a `strict_templates` setting 1618249044  
1463024951 https://github.com/simonw/datasette/pull/1999#issuecomment-1463024951 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XM_03 simonw 9599 2023-03-10T00:17:58Z 2023-03-10T00:17:58Z OWNER

Renderers have an impact on three different pages: query results, table page and row page.

The row page feature is incomplete though:

https://congress-legislators.datasettes.com/legislators/social_media/A000055

Why is there no .csv link there?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1463023674 https://github.com/simonw/datasette/pull/1999#issuecomment-1463023674 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XM_g6 simonw 9599 2023-03-10T00:16:03Z 2023-03-10T00:16:03Z OWNER

I also need to figure out the renderers stuff, so I can link to the right URLs for CSV and JSON and other formats:

https://github.com/simonw/datasette/blob/6d07a7da1531cd749844fc6827d9a1e57009b2ea/datasette/views/base.py#L474-L518

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1463022397 https://github.com/simonw/datasette/pull/1999#issuecomment-1463022397 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XM_M9 simonw 9599 2023-03-10T00:14:21Z 2023-03-10T00:14:21Z OWNER

Ironically the thing I most need right now is comprehensive documentation of what variables are being passed to the templates!

One big challenge is that I need to untangle the template context that happens in BaseView - I'm hacking that together at the moment, but I need a real answer for how that should work in a world in which view functions aren't using that base class.

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L145

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1463021383 https://github.com/simonw/datasette/pull/1999#issuecomment-1463021383 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XM-9H simonw 9599 2023-03-10T00:12:50Z 2023-03-10T00:12:50Z OWNER

Now at 34 failed, 34 passed.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1463005744 https://github.com/simonw/datasette/pull/1999#issuecomment-1463005744 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XM7Iw simonw 9599 2023-03-09T23:52:15Z 2023-03-09T23:52:23Z OWNER

I need to figure out what to do about extra_context_from_filters - which was previously passed straight to the HTML context.

https://github.com/simonw/datasette/blob/11f7feb7a3f7166c71389786880863d60ed3d165/datasette/views/table.py#L406-L422

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1462997800 https://github.com/simonw/datasette/pull/1999#issuecomment-1462997800 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XM5Mo simonw 9599 2023-03-09T23:39:47Z 2023-03-09T23:39:47Z OWNER

Found a neat trick: ```diff diff --git a/datasette/app.py b/datasette/app.py index 186f192d..40416713 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -23,7 +23,13 @@ from pathlib import Path

from markupsafe import Markup, escape from itsdangerous import URLSafeSerializer -from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader +from jinja2 import ( + ChoiceLoader, + Environment, + FileSystemLoader, + PrefixLoader, + StrictUndefined, +) from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound

@@ -394,7 +400,10 @@ class Datasette: ] ) self.jinja_env = Environment( - loader=template_loader, autoescape=True, enable_async=True + loader=template_loader, + autoescape=True, + enable_async=True, + undefined=StrictUndefined, ) self.jinja_env.filters["escape_css_string"] = escape_css_string self.jinja_env.filters["quote_plus"] = urllib.parse.quote_plus ``` This causes Jinja to raise a hard error if there are any variables referenced in the template that are not available in the context.

It's helping me spot things that are still missing, rather than just relying on failed unit tests.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1462968053 https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462968053 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11 IC_kwDOJHON9s5XMx71 simonw 9599 2023-03-09T23:24:01Z 2023-03-09T23:24:01Z MEMBER

I improved the readability by removing some unnecessary table aliases: sql with recursive nested_folders(folder_id, descendant_folder_id) as ( -- base case: select all immediate children of the root folder select id, id from folders where parent is null union all -- recursive case: select all children of the previous level of nested folders select nested_folders.folder_id, folders.id from nested_folders join folders on nested_folders.descendant_folder_id = folders.parent ) -- Find notes within all descendants of folder 1 select * from notes where folder in ( select descendant_folder_id from nested_folders where folder_id = 1 );

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Implement a SQL view to make it easier to query files in a nested folder 1618130434  
1462962682 https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462962682 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11 IC_kwDOJHON9s5XMwn6 simonw 9599 2023-03-09T23:20:35Z 2023-03-09T23:22:41Z MEMBER

Here's a query that returns all notes in folder 1, including notes in descendant folders: sql with recursive nested_folders(folder_id, descendant_folder_id) as ( -- base case: select all immediate children of the root folder select id, id from folders where parent is null union all -- recursive case: select all children of the previous level of nested folders select nf.folder_id, f.id from nested_folders nf join folders f on nf.descendant_folder_id = f.parent ) -- Find notes within all descendants of folder 1 select * from notes where folder in ( select descendant_folder_id from nested_folders where folder_id = 1 ); With assistance from ChatGPT. Prompts were:

``` SQLite schema:

CREATE TABLE [folders] ( [id] INTEGER PRIMARY KEY, [long_id] TEXT, [name] TEXT, [parent] INTEGER, FOREIGN KEY([parent]) REFERENCES folders );

Write a recursive CTE that returns the following:

folder_id | descendant_folder_id

With a row for every nested child of every folder - so the top level folder has lots of rows Then I tweaked it a bit, then ran this: WITH RECURSIVE nested_folders(folder_id, descendant_folder_id) AS ( -- base case: select all immediate children of the root folder SELECT id, id FROM folders WHERE parent IS NULL UNION ALL -- recursive case: select all children of the previous level of nested folders SELECT nf.folder_id, f.id FROM nested_folders nf JOIN folders f ON nf.descendant_folder_id = f.parent ) -- select all rows from the recursive CTE SELECT * from notes where folder in (select descendant_folder_id FROM nested_folders where folder_id = 1)

Convert all SQL keywords to lower case, and re-indent ```

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Implement a SQL view to make it easier to query files in a nested folder 1618130434  
1462965256 https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462965256 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11 IC_kwDOJHON9s5XMxQI simonw 9599 2023-03-09T23:22:12Z 2023-03-09T23:22:12Z MEMBER

Here's what the CTE from that looks like:

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Implement a SQL view to make it easier to query files in a nested folder 1618130434  
1462921890 https://github.com/simonw/datasette/pull/2031#issuecomment-1462921890 https://api.github.com/repos/simonw/datasette/issues/2031 IC_kwDOBm6k_c5XMmqi simonw 9599 2023-03-09T22:35:30Z 2023-03-09T22:35:30Z OWNER

I've implemented the test (thanks for pointing me in the right direction!).

At tmcl-it/datasette:0.64.1+row-view-expand-labels I also have a variant of this patch that applies to the 0.64.x branch. Please let me know if you'd be interested in merging that as well and I'll open another PR.

Sure, let's merge that one too - it can go out in the next 0.64.x series release (maybe even a 0.65).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Expand foreign key references in row view as well 1605481359  
1462921010 https://github.com/simonw/datasette/pull/2034#issuecomment-1462921010 https://api.github.com/repos/simonw/datasette/issues/2034 IC_kwDOBm6k_c5XMmcy simonw 9599 2023-03-09T22:34:29Z 2023-03-09T22:34:29Z OWNER

Good catch, thanks.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 remove an unused `app` var in cli.py 1613974869  
1462693867 https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462693867 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7 IC_kwDOJHON9s5XLu_r simonw 9599 2023-03-09T20:01:39Z 2023-03-09T20:02:11Z MEMBER

My folders table will have:

  • id - rowid
  • long_id - that long unique string ID
  • name - the name
  • parent - foreign key to id
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Folder support 1617769847  
1462691466 https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462691466 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7 IC_kwDOJHON9s5XLuaK simonw 9599 2023-03-09T19:59:52Z 2023-03-09T19:59:52Z MEMBER

Improved script: zsh osascript -e 'tell application "Notes" set allFolders to folders repeat with aFolder in allFolders set folderId to id of aFolder set folderName to name of aFolder set folderContainer to container of aFolder if class of folderContainer is folder then set folderContainerId to id of folderContainer else set folderContainerId to "" end if log "ID: " & folderId log "Name: " & folderName log "Container: " & folderContainerId log " " end repeat end tell ' ``` ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113 Name: Blog posts Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698 Name: JSK Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995 Name: Nested inside blog posts Container: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526 Name: New Folder Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839 Name: New Folder 1 Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2 Name: Notes Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059 Name: Quick Notes Container:

ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283 Name: UK Christmas 2022 Container: `` I filtered out things where the parent was an account and not a folder usingif class of folderContainer is folder then`.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Folder support 1617769847  
1462682795 https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462682795 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7 IC_kwDOJHON9s5XLsSr simonw 9599 2023-03-09T19:52:20Z 2023-03-09T19:52:44Z MEMBER

Created through several rounds with ChatGPT (including hints like "rewrite that using setdefault()"): ```python def topological_sort(nodes): children = {} for node in nodes: parent_id = node["parent"] if parent_id is not None: children.setdefault(parent_id, []).append(node)

def traverse(node, result):
    result.append(node)
    if node["id"] in children:
        for child in children[node["id"]]:
            traverse(child, result)

sorted_data = []

for node in nodes:
    if node["parent"] is None:
        traverse(node, sorted_data)

return sorted_data

```

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Folder support 1617769847  
1462570187 https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462570187 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7 IC_kwDOJHON9s5XLQzL simonw 9599 2023-03-09T18:30:24Z 2023-03-09T18:30:24Z MEMBER

I used ChatGPT to write this: osascript -e 'tell application "Notes" set allFolders to folders repeat with aFolder in allFolders set folderId to id of aFolder set folderName to name of aFolder set folderContainer to container of aFolder set folderContainerName to name of folderContainer log "Folder ID: " & folderId log "Folder Name: " & folderName log "Folder Container: " & folderContainerName log " " --check for nested folders if count of folders of aFolder > 0 then set nestedFolders to folders of aFolder repeat with aNestedFolder in nestedFolders set nestedFolderId to id of aNestedFolder set nestedFolderName to name of aNestedFolder set nestedFolderContainer to container of aNestedFolder set nestedFolderContainerName to name of nestedFolderContainer log " Nested Folder ID: " & nestedFolderId log " Nested Folder Name: " & nestedFolderName log " Nested Folder Container: " & nestedFolderContainerName log " " end repeat end if end repeat end tell ' Which for my account output this: ``` Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113 Folder Name: Blog posts Folder Container: iCloud

Nested Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
Nested Folder Name: Nested inside blog posts
Nested Folder Container: Blog posts

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698 Folder Name: JSK Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995 Folder Name: Nested inside blog posts Folder Container: Blog posts

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526 Folder Name: New Folder Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839 Folder Name: New Folder 1 Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2 Folder Name: Notes Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059 Folder Name: Quick Notes Folder Container: iCloud

Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283 Folder Name: UK Christmas 2022 Folder Container: iCloud ``` So I think the correct approach here is to run code at the start to list all of the folders (no need to do fancy recursion though, just a flat list with the parent containers is enough) and create a model of that hierarchy in SQLite.

Then when I import notes I can foreign key reference them back to their containing folder.

I'm tempted to use rowid for the foreign keys because the official IDs are pretty long.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Folder support 1617769847  
1462564717 https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462564717 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7 IC_kwDOJHON9s5XLPdt simonw 9599 2023-03-09T18:25:39Z 2023-03-09T18:25:39Z MEMBER

So it looks like folders can be hierarchical?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Folder support 1617769847  
1462562735 https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462562735 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7 IC_kwDOJHON9s5XLO-v simonw 9599 2023-03-09T18:23:56Z 2023-03-09T18:25:22Z MEMBER

From the Script Editor library docs:

A note has a:

  • container (folder), r/o) : the folder of the note

Here's what a folder looks like:

folder n : a folder containing notes elements:

  • contains folders, notes; contained by application, accounts, folders.

properties:

  • name (text) : the name of the folder
  • id (text, r/o) : the unique identifier of the folder
  • shared (boolean, r/o) : Is the folder shared?
  • container (account or folder, r/o) : the container of the folder
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Folder support 1617769847  
1462556829 https://github.com/dogsheep/apple-notes-to-sqlite/issues/4#issuecomment-1462556829 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4 IC_kwDOJHON9s5XLNid simonw 9599 2023-03-09T18:20:56Z 2023-03-09T18:20:56Z MEMBER

In terms of the UI: I'm tempted to say that the default behaviour is for it to run until it sees a note that it already knows about AND that has matching update/created dates, and then stop.

You can do a full import again ignoring that logic with apple-notes-to-sqlite notes.db --full.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support incremental updates 1616429236  
1462554175 https://github.com/dogsheep/apple-notes-to-sqlite/issues/4#issuecomment-1462554175 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4 IC_kwDOJHON9s5XLM4_ simonw 9599 2023-03-09T18:19:34Z 2023-03-09T18:19:34Z MEMBER

It looks like the iteration order is most-recently-modified-first - I tried editing a note a bit further back in my notes app and it was the first one output by apple-notes-to-sqlite --dump.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support incremental updates 1616429236  
1461285545 https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461285545 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2 IC_kwDOJHON9s5XGXKp simonw 9599 2023-03-09T05:06:24Z 2023-03-09T05:06:24Z MEMBER

OK, this works!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 First working version 1616354999  
1461262577 https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461262577 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2 IC_kwDOJHON9s5XGRjx simonw 9599 2023-03-09T04:30:00Z 2023-03-09T04:30:00Z MEMBER

It doesn't have tests yet. I guess I'll need to mock subprocess to test this.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 First working version 1616354999  
1461260978 https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461260978 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2 IC_kwDOJHON9s5XGRKy simonw 9599 2023-03-09T04:27:18Z 2023-03-09T04:27:18Z MEMBER

Before that conversion:

Monday, March 6, 2023 at 11:55:15 AM

After:

2023-03-06T11:55:15

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 First working version 1616354999  
1461259490 https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461259490 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2 IC_kwDOJHON9s5XGQzi simonw 9599 2023-03-09T04:24:27Z 2023-03-09T04:24:27Z MEMBER

Converting AppleScript date strings to ISO format is hard!

https://forum.latenightsw.com/t/formatting-dates/841 has a recipe I'll try:

set todayISO to (todayDate as «class isot» as string)

Not clear to me how timezones work here. I'm going to ignore them for the moment.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 First working version 1616354999  
1461234591 https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461234591 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2 IC_kwDOJHON9s5XGKuf simonw 9599 2023-03-09T03:56:45Z 2023-03-09T03:56:45Z MEMBER

My prototype showed that images embedded in notes come out in the HTML export as bas64 image URLs, which is neat.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 First working version 1616354999  
1461234311 https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461234311 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2 IC_kwDOJHON9s5XGKqH simonw 9599 2023-03-09T03:56:24Z 2023-03-09T03:56:24Z MEMBER

I opened the "Script Editor" app on my computer, used Window -> Library to open the Library panel, then clicked on the Notes app there. I got this:

So the notes object has these properties:

  • name (text) : the name of the note (normally the first line of the body)
  • id (text, r/o) : the unique identifier of the note
  • container (folder, r/o) : the folder of the note
  • body (text) : the HTML content of the note
  • plaintext (text, r/o) : the plaintext content of the note
  • creation date (date, r/o) : the creation date of the note
  • modification date (date, r/o) : the modification date of the note
  • password protected (boolean, r/o) : Is the note password protected?
  • shared (boolean, r/o) : Is the note shared?

I'm going to ignore the concept of attachments for the moment.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 First working version 1616354999  
1461232709 https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461232709 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2 IC_kwDOJHON9s5XGKRF simonw 9599 2023-03-09T03:54:28Z 2023-03-09T03:54:28Z MEMBER

I think the AppleScript I want to pass to osascript looks like this: applescript tell application "Notes" repeat with eachNote in every note set noteId to the id of eachNote set noteTitle to the name of eachNote set noteBody to the body of eachNote log "------------------------" & "\n" log noteId & "\n" log noteTitle & "\n\n" log noteBody & "\n" end repeat end tell But there are a few more properties I'd like to get - created and updated date for example.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 First working version 1616354999  
1461230436 https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461230436 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1 IC_kwDOJHON9s5XGJtk simonw 9599 2023-03-09T03:51:52Z 2023-03-09T03:51:52Z MEMBER

This did the job! Next step is to turn that into a Python script.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Initial proof of concept with ChatGPT 1616347574  
1461230197 https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461230197 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1 IC_kwDOJHON9s5XGJp1 simonw 9599 2023-03-09T03:51:36Z 2023-03-09T03:51:36Z MEMBER

After a few more rounds I got to this script, which outputs them to a /tmp/notes.txt file:

```zsh

!/bin/zsh

osascript -e ' set notesFile to "/tmp/notes.txt" set fileRef to open for access notesFile with write permission tell application "Notes" repeat with eachNote in every note set noteId to the id of eachNote set noteTitle to the name of eachNote set noteBody to the body of eachNote write "------------------------" & "\n" to fileRef write noteId & "\n" to fileRef write noteTitle & "\n\n" to fileRef write noteBody & "\n" to fileRef end repeat end tell close access fileRef' Then I wrote this little Python script to load them into a database:python import sqlite_utils split = b"------------------------\n" s = open("/tmp/notes.txt", "rb").read() notes = [n.decode("mac_roman") for n in s.split(split) if n]

cleaned_notes = [{ "id": n.split("\n")[0], "title": n.split("\n")[1], "body": "\n".join(n.split("\n")[2:]).strip() } for n in notes]

db = sqlite_utils.Database("/tmp/notes.db") db["notes"].insert_all(cleaned_notes) ```

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Initial proof of concept with ChatGPT 1616347574  
1461226402 https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461226402 https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1 IC_kwDOJHON9s5XGIui simonw 9599 2023-03-09T03:47:11Z 2023-03-09T03:47:11Z MEMBER

Me:

Write applescript to loop through all of my Apple Notes and output their contents

ChatGPT:

applescript tell application "Notes" repeat with eachNote in every note set noteTitle to the name of eachNote set noteBody to the body of eachNote display dialog noteTitle & return & return & noteBody end repeat end tell Me:

How can I turn that into a zsh script which will output the results to my terminal

ChatGPT: ```zsh

!/usr/bin/zsh

osascript -e 'tell application "Notes" repeat with eachNote in every note set noteTitle to the name of eachNote set noteBody to the body of eachNote set output to noteTitle & "\n" & noteBody & "\n" display dialog output log output end repeat end tell' `` I had to change that first line to#!/bin/zsh` - then when I ran it it spammed my screen with dialog boxes.

Me:

I want to log the notes contents to the terminal, not display it in a dialog

ChatGPT:

```zsh

!/bin/zsh

osascript -e 'tell application "Notes" repeat with eachNote in every note set noteTitle to the name of eachNote set noteBody to the body of eachNote set output to noteTitle & "\n" & noteBody & "\n" log output end repeat end tell' ``` This really helps!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Initial proof of concept with ChatGPT 1616347574  
1461161256 https://github.com/simonw/datasette/pull/1999#issuecomment-1461161256 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XF40o simonw 9599 2023-03-09T02:10:07Z 2023-03-09T02:10:07Z OWNER

Just ran into a no such table: columns error - which I think is because my new view code sometimes bypasses calling this method (currently done in BaseView.dispatch_request()):

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/base.py#L101-L103

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461148579 https://github.com/simonw/datasette/pull/1999#issuecomment-1461148579 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XF1uj simonw 9599 2023-03-09T01:54:10Z 2023-03-09T01:55:33Z OWNER

Or... I could temporarily build a quick additional CannedQueryView subclass that just does the necessary bits to get the existing code to work. I'm going to try that.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461148254 https://github.com/simonw/datasette/pull/1999#issuecomment-1461148254 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XF1pe simonw 9599 2023-03-09T01:53:41Z 2023-03-09T01:53:41Z OWNER

Solving this is proving difficult: https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L1500-L1503

The problem is that calling .data() on QueryView only works here because we expect to ourselves be inside a .data() method, with all of the existing magic that knows how to render things that are returned by that.

So I may need to substantially re-engineer how QueryView works in order to get this to work.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461075648 https://github.com/simonw/datasette/pull/1999#issuecomment-1461075648 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFj7A simonw 9599 2023-03-09T00:24:22Z 2023-03-09T00:24:22Z OWNER

127.0.0.1:8001/fixtures/neighborhood_search fails because the forwarding to a canned query does not yet work.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461074526 https://github.com/simonw/datasette/pull/1999#issuecomment-1461074526 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFjpe simonw 9599 2023-03-09T00:23:06Z 2023-03-09T00:23:06Z OWNER 
pytest tests/test_table_html.py

Currently 44 failed, 24 passed in 7.53s

Failures here: https://gist.github.com/simonw/df0a52cd7d820b776dc3dfc50e7cb778

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  
1461070937 https://github.com/simonw/datasette/pull/1999#issuecomment-1461070937 https://api.github.com/repos/simonw/datasette/issues/1999 IC_kwDOBm6k_c5XFixZ simonw 9599 2023-03-09T00:18:52Z 2023-03-09T00:19:36Z OWNER

I managed to get HTML view working! I did it by continuing to add more things to the extras and the _html bundle until the page loaded for me:

```diff async def extra_extras(): "Available ?_extra= blocks" return { @@ -1981,6 +2053,14 @@ async def extra_extras(): "query", "display_columns", "display_rows", + "database", + "table", + "database_color", + "table_actions", + "filters", + "renderers", + "custom_table_templates", + "sorted_facet_results", ] }

@@ -2006,6 +2086,14 @@ async def extra_extras(): extra_query, extra_metadata, extra_extras, + extra_database, + extra_table, + extra_database_color, + extra_table_actions, + extra_filters, + extra_renderers, + extra_custom_table_templates, + extra_sorted_facet_results, ) `` I'll probably refactor this into something cleaner, and maybe but a bunch of them in a"html"dictionary and update the templates to use{{ html.filters }}` or similar. Will look at that once the tests are passing.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 ?_extra= support (draft) 1551694938  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);