issues

https://github.com/simonw/datasette/actions/runs/6266449018/job/17017460074

This is a bit of a mystery, I don't think I've changed anything recently that could have broken this.

Refs: - #2102

Also includes a prototype implementation of --actor option which I'm using for testing this, from: - #2153

:books: Documentation preview :books:: https://datasette--2154.org.readthedocs.build/en/2154/

I want to use isort to order imports. A few steps here:

• [x] Add a .isort.cfg file (see below)
• [x] Use isort -rc to reformat existing code
• [ ] Commit this change
• [x] Add a unit test that ensures future changes remain isort compatible

Currently the CORS headers served are:

https://github.com/simonw/datasette/blob/9584879534ff0556e04e4c420262972884cac87b/datasette/utils/init.py#L1139-L1143

Serving Access-Control-Max-Age: 600 would allow browsers to cache that for 10 minutes, avoiding additional CORS pre-flight OPTIONS requests during that time.

Equivalent to sqlite-utils upsert: https://sqlite-utils.datasette.io/en/stable/python-api.html#upserting-data

Refs #1878

Still todo: - [x] Support "return": true properly for upserts (with tests) - [x] Require both insert-row and update-row permissions - [x] Tests are going to need to cover both rowid-only and compound primary key tables, including all of the error states - [x] Documentation

:books: Documentation preview :books:: https://datasette--1931.org.readthedocs.build/en/1931/

I noticed that a table page that is private because the database or instance is private, e.g. this one:

Is not displaying the padlock icon that indicates the table is not visible to the public.

Same issue for the database page too, which in this case is private due to view-instance.

While working on: - https://github.com/simonw/datasette/issues/1657 - https://github.com/simonw/datasette/issues/1439

It became very clear that the least maintainable part of Datasette at the moment is the way routing to the database, table and row views work - in particular the subclassing mechanism with BaseView and DataView, but also the complex variety of ways in which the URL routes capture different named regular expression groups.

Tidy up from: - #1439

As demonstrated by this piece of documentation, using SpatiaLite with sqlite-utils requires a fair bit of boilerplate: https://github.com/simonw/sqlite-utils/blob/f7289174e66ae4d91d57de94bbd9d09fabf7aff4/docs/python-api.rst#L880-L909

While working on https://github.com/simonw/datasette-leaflet-freedraw/issues/10 I found myself writing this atrocity to figure out the SQL query used for a specific table page:

javascript let innerSql = Array.from(document.getElementsByTagName("span")).filter( el => el.innerText == "View and edit SQL" )[0].parentElement.getAttribute("title") This is obviously bad - it's very brittle, and will break if I ever change the text on that link (like localizing it for example).

Instead, I think pages like that one should have a block of script at the bottom something like this: javascript window.datasette = window.datasette || {}; datasette.view_name = 'table'; datasette.table_sql = 'select * from ...';

While I'm doing this I should rename this internal variable to avoid confusion in the future:

https://github.com/simonw/datasette/blob/e837095ef35ae155b4c78cc9a8b7133a48c94f03/datasette/app.py#L203

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1431#issuecomment-898072940

e.g. https://github.com/simonw/datasette/runs/3197141955

I've seen this error before, but I don't yet have a good workaround for it.

``` @contextlib.contextmanager def isolated_filesystem( self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None ) -> t.Iterator[str]: """A context manager that creates a temporary directory and changes the current working directory to it. This isolates tests that affect the contents of the CWD to prevent them from interfering with each other.

    :param temp_dir: Create the temporary directory under this
        directory. If given, the created directory is not removed
        when exiting.

    .. versionchanged:: 8.0
        Added the ``temp_dir`` parameter.
    """

  cwd = os.getcwd()

E FileNotFoundError: [Errno 2] No such file or directory

/opt/hostedtoolcache/Python/3.6.14/x64/lib/python3.6/site-packages/click/testing.py:466: FileNotFoundError =========================== short test summary info ============================ FAILED tests/test_publish_cloudrun.py::test_publish_cloudrun_apt_get_install FAILED tests/test_publish_cloudrun.py::test_publish_cloudrun_extra_options[---setting force_https_urls on] FAILED tests/test_publish_cloudrun.py::test_publish_cloudrun_extra_options[--setting base_url /foo---setting base_url /foo --setting force_https_urls on] FAILED tests/test_publish_cloudrun.py::test_publish_cloudrun_extra_options[--setting force_https_urls off---setting force_https_urls off] FAILED tests/test_publish_heroku.py::test_publish_heroku_requires_heroku - Fi... FAILED tests/test_publish_heroku.py::test_publish_heroku_installs_plugin - Fi... FAILED tests/test_publish_heroku.py::test_publish_heroku - FileNotFoundError:... FAILED tests/test_publish_heroku.py::test_publish_heroku_plugin_secrets - Fil... ================== 8 failed, 920 passed in 188.22s (0:03:08) =================== ```

Reported in the wild on https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-3d4a4b7/bib?_facet=bib_level_callnumber - the "next page" link links to https://127.0.0.1:8010/collection-analysis/current_collection-3d4a4b7/bib?_facet=bib_level_callnumber&_next=100

That installation uses "base_url": "/collection-analysis/"

Weirdly all of the other links on that page - to facet results, sort orders, row permalinks etc - work fine. It's JUST the next_url one that is broken.

Also broken in their JSON: https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-3d4a4b7/bib.json?_size=1 returns

json "suggested_facets": [], "next": "1", "next_url": "https://127.0.0.1:8010/collection-analysis/current_collection-3d4a4b7/bib.json?_size=1&_next=1", "private": false,

Refs #272. Initial implementation only does CSV data, still needs:

• [x] Implement --save
• [x] Add --dump to the documentation
• [x] Add --attach example to the documentation
• [x] Replace :memory: in documentation

Refs #283. Still needs:

• [x] Unit test for --crossdb queries
• [x] Show warning on console if it truncates at ten databases (or on web interface)
• [x] Show connected databases on the /_memory database page
• [x] Documentation
• [x] https://latest.datasette.io/ demo should demonstrate this feature

I just spotted that csv.Sniffer in the Python standard library has a .has_header(sample) method which detects if the first row appears to be a header or not, which is interesting. https://docs.python.org/3/library/csv.html#csv.Sniffer

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778812050

https://github.com/simonw/datasette-export-notebook/blob/0.1/datasette_export_notebook/init.py

python async def render_notebook(datasette, request): return Response.html( await datasette.render_template( "export_notebook.html", { "csv_stream_url": datasette.absolute_url( request, path_with_format( request=request, format="csv", extra_qs={"_stream": "on"} ), ), "json_url": datasette.absolute_url( request, path_with_format( request=request, format="json", extra_qs={"_shape": "array"} ), ), "json": json, }, ) ) This results in https://latest-with-plugins.datasette.io/github/issue_comments.Notebook showing http://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_format=json&_shape=array

Using PyInstaller, as explored in #93 and https://til.simonwillison.net/python/packaging-pyinstaller

The bigger challenge will be the code signing bit. I'll need a Apple Developer account ($99/year) and some extensive CI fiddling.

Now that we're making a LOT more permission checks (on the DB index page we do a check for every listed table for example) the /-/permissions page gets filled up pretty quickly.

Can make this more readable by grouping permission checks by request. Have most recent request at the top of the page but the permission requests within that page sorted chronologically by most recent last.

Wrap the main table in <div style="overflow-x: auto">

I want to fix the bug in #1134 and ship it as Datasette 0.52.5 - but the main branch now has a feature in it (4c25b035b2370983c8dd5e0c8762e9154e379774 added arraynotcontains, #1132).

I'm not ready for a feature release, so instead I want to release 0.52.5 with just that bug fix.

This is the first time I will have shipped a release from a branch. I need to establish that pattern and add it to the documentation in https://docs.datasette.io/en/stable/contributing.html

Currently this is mainly handled by a plugin - https://github.com/simonw/datasette-dateutil - but I realise now that this really needs to be core functionality.

See also Twitter thread: https://twitter.com/simonw/status/1318234808653213696

``` % curl -vv -XOPTIONS https://latest.datasette.io/ * Trying 216.58.195.83:443...

OPTIONS / HTTP/1.1 Host: latest.datasette.io User-Agent: curl/7.70.0 Accept: /

• Mark bundle as not supporting multiuse < HTTP/1.1 500 Internal Server Error ```

Datasette currently bundles 5.31.0 (from October 2017) - latest version is 5.57.0 (August 2020). https://codemirror.net/doc/releases.html

Got this on a search results page.

Part of #3 - in particular I want to make sure my photos are displayed with a thumbnail.

Got this error deploying large (603MB) database with Cloud Run X Deploying... Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revi sion might contain more information. X Creating Revision... Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information. . Routing traffic... ✓ Setting IAM Policy... Deployment failed ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.

request.raw_args is not documented, and I'd like to remove it entirely. Originally posted by @simonw in https://github.com/simonw/datasette/issues/706#issuecomment-634975252

I use it in a few places in other projects though, so I'll have to fix those first: https://github.com/search?q=user%3Asimonw+raw_args&type=Code

photos-to-sqlite doesn't really capture the full scope of this project anymore.

It would be useful if I could tell from looking at the table page if a column was a integer or a text (or a float I guess?). This is particularly important for knowing if it safe to sort by that column.

It would be great if pragma_table_info(tablename) was allowed to be used in queries. See also https://github.com/simonw/til/blob/master/sqlite/list-all-columns-in-a-database.md

select * from pragma_table_info(tablename); is currently disallowed for user-provided queries via a regex restriction - but could help here too.

https://github.com/simonw/datasette/blob/d349d57cdf3d577afb62bdf784af342a4d5be660/datasette/utils/init.py#L174

Originally posted by @simonw in https://github.com/simonw/datasette/issues/760#issuecomment-624729459

Faces, albums, locations, that kind of thing.

Spun off from #24 - it would be useful if CLI users could create new tables (with explicit column types, not null rules and defaults) without having to insert an example record.

• [x] Get it working
• [x] Support --pk
• [x] Support --not-null
• [x] Support --default
• [x] Support --fk colname othertable othercol
• [x] Support --replace and --ignore
• [x] Documentation

Maybe this:

$ github-to-sqlite stargazers github.db simonw/datasette

It could accept more than one repos.

Maybe have options similar to --sql in twitter-to-sqlite so you can e.g. fetch all stargazers for all of the repos you have fetched into the database already (or all of the repos belonging to owner X)

See https://github.com/simonw/museums/issues/20#issuecomment-609103663 - at some point between 286ed286b68793532c2a38436a08343b45cfbc91 and current master (e0e7a0facfc935a835cd73c720bc46661462f0b1 today) a bug was introduced where the extra_template_vars(request, view_name) plugin hook started being passed None instead of index for the view_name parameter on the site index page.

This would be called with the context dictionary before each template is rendered. It would have the opportunity to modify that context.

Traceback (most recent call last): File "/home/ubuntu/datasette-venv/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/cli.py", line 235, in commits utils.save_commits(db, commits, repo_full["id"]) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 290, in save_commits commit_to_insert["author"] = save_user(db, commit["author"]) File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 54, in save_user for key, value in user.items() AttributeError: 'NoneType' object has no attribute 'items' Got this running the commits command from cron.

Needed by https://github.com/simonw/datasette-auth-github/issues/5

We already have a navigation breadcrumbs header on some pages, I can extend that to be present on every page and make it easy to modify with custom templates.

Since we now have custom templates, we should support including custom static files with them as well. Maybe something like this:

datasette mydb.db --template-dir=templates/ --static-dir=static/

This should also be supported by datasette publish - see also #157

Heroku has Docker container support so this should not be too hard:

https://devcenter.heroku.com/articles/container-registry-and-runtime

See also #59

This should work exactly like the existing “datasette publish now....” command except it would be “datasette publish heroku...”

Absolutely everything should have a far-future expires header

Part of the URL will be the truncated sha1 hash of the database file itself, calculated at build time