issues

This code here:

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/utils/asgi.py#L127-L135

It consumes the messages, which means if you try to call it a second time you won't be able to get at the body.

This is efficient - we don't end up with a request object property with potentially megabytes of content that we never look at again - but it's inconvenient for cases like middleware or functions where we don't know if the body has been consumed yet or not.

Potential solution: set request._body the first time it is called, and return that on subsequent calls.

Potential optimization: only do this for bodies that are shorter than a certain threshold - maybe 1MB - and raise an exception if you attempt to call post_body() multiple times against one of those larger bodies.

I'm a bit nervous about that option though, since it could result in errors that don't show up in testing but do show up in production.

https://github.com/simonw/sqlite-utils-plugins should be in the official documentation.

I can use the same pattern as https://llm.datasette.io/en/stable/plugins/directory.html

https://til.simonwillison.net/readthedocs/stable-docs

This made me think. That {"$env": "ENV_VAR"} hack was introduced back here:

• https://github.com/simonw/datasette/issues/538

The problem it was solving was that metadata was visible to everyone with access to the instance at /-/metadata but plugins clearly needed a way to set secret settings.

Now that this stuff is moving to config, we have some decisions to make:

1. Add /-/config to let people see the configuration of their instance, and keep the $env trick for secret settings.
2. Say all configuration aside from metadata is secret and make $env optional or ditch it entirely.
3. Allow plugins to announce which of their configuration options are secret so we can automatically redact them from /-/config

I've found /-/metadata extraordinarily useful as a user of Datasette - it really helps me understand exactly what's going on if I run into any problems with a plugin, if I can quickly check what the settings look like.

So I'm leaning towards option 1 or 3.

Originally posted by @simonw in https://github.com/simonw/datasette/pull/2183#discussion_r1325076924

Also refs: - #2093

In: - https://github.com/simonw/datasette.io/issues/152#issuecomment-1712323347

This turned out to be the fix:

bash dogsheep-beta index dogsheep-index.db templates/dogsheep-beta.yml sqlite-utils rebuild-fts dogsheep-index.db

The return type isn't documented here: https://github.com/simonw/datasette/blob/4c3ef033110407f3b3dbce501659d523724985e0/docs/internals.rst#L327-L350

On inspecting the code I'm not 100% sure if it's possible for this. method to return None, or if it can only return True or False. Need to confirm that.

https://github.com/simonw/datasette/blob/4c3ef033110407f3b3dbce501659d523724985e0/datasette/app.py#L822C15-L853

I just had to write a one-off custom plugin to redirect niche-musems.com to www.niche-museums.com (https://github.com/simonw/museums/issues/21) - it would be great if this kind of thing could be handled by a configurable plugin.

https://github.com/simonw/museums/blob/6b1faf00c463b2228860d4d62d104b11935e01b1/plugins/redirect_www.py

Currently looks like this: https://github.com/simonw/datasette/blob/08181823990a71ffa5a1b57b37259198eaa43e06/datasette/init.py#L1-L6

Adding __all__ = ["Permission", "Forbidden"...] would let me get rid of those # noqa comments.

Maybe put it on the <tr> but probably better to go on the td.type-pk.

https://discord.com/channels/823971286308356157/823971286941302908/1120516587036889098

One thing that would be helpful would be message at launch indicating a metadata.json is getting picked up. I'm using directory mode and was editing the wrong file for awhile before I realize nothing I was doing was having any effect.

See: - #2056

ruff spots a bunch of warnings about things like unused variables - would be good to clean up as many of these as possible.

Provides URLs to the JSON version of those rows. Maybe it persists the ?_shape= option too? Not sure about that.

e.g. first column on this page - cog menu has no option to facet.

https://datasette.io/content/tools

Now that I've landed: - #1999

See also: - #262

I want to get some feedback from people on the design of the new ?_extra= feature, before freezing it into Datasette 1.0.

The big change is that the default JSON representation is now MUCH slimmer - it only gives you keys for "next" and "rows", where rows is a list of JSON objects (not a list of arrays as was previously the default) - for example https://latest.datasette.io/fixtures/sortable.json

If you want extra stuff you can ask for it with the new ?_extra= parameter - e.g. https://latest.datasette.io/fixtures/sortable.json?_extra=columns&_extra=suggested_facets

You can use ?_extra=extras to see a list of available extras: https://latest.datasette.io/fixtures/sortable.json?_extra=extras

The full body version of the notes can get HUGE, due to embedded images. It turns out for my own purposes I'm usually happy with just the plaintext version.

I'm tempted to say you don't get HTML unless you pass a --html option.

FTS would be useful.

Maybe even extract the plain text from the notes to make that index easier to create, rather than creating it against the HTML. Can use the plaintext property for that.

Spotted this on an instance index page:

https://github.com/simonw/datasette/blob/e4ebef082de90db4e1b8527abc0d582b7ae0bc9d/datasette/utils/init.py#L1052-L1056

This logic should be reflected in the documentation on https://docs.datasette.io/en/stable/authentication.html#actors

https://datasette.simonwillison.net/simonwillisonblog/blog_entry.csv?_col=id&_col=title&_col=body&_labels=on&_size=1

csv id,id,title,body 1,1,WaSP Phase II,"<p>The <a href=""http://www.webstandards.org/"">Web Standards</a> project has launched Phase II.</p>" That should not have two id columns.

I do think this needs type checking - I just tried and you really can send a string to an integer column and have it work, which feels bad.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1863#issuecomment-1331089156

Refs: - #1850

Initially I decided that the API shouldn't be able to create new tokens at all - I don't like the idea of an API token holder creating themselves additional tokens.

Then I realized that two of the API features are specifically more useful if you can generate fresh tokens via the API:

• Tokes that expire after a time limit are MUCH more useful if they can be automatically generated
• Likewise, tokens that are restricted to a subset of permissions (see #1855) make more sense to be generated like this, especially in conjunction with expiry times

Designed in: - #1720

Part of: - #262 - #1709

Idea from here: - https://github.com/simonw/sqlite-utils/issues/516

This is a strong argument for extracting the Heroku support out to a plugin - it would allow this to be fixed with a plugin release without needing to push a full release of Datasette itself.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1905#issuecomment-1320678715

For datasette-copyable I want to show breadcrumbs that take database/instance permissions into account, so I'm removing {% block nav %} entirely and replacing it with this:

html+jinja {% block crumbs %} {{ crumbs.nav(request=request, database=database, table=table) }} {% endblock %}

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1901#issuecomment-1319588163

I should document this.

Turns out this is case-sensitive, needs to be:

request.headers.get("content-type") != "application/json"

That's not great usability. It should be case insensitive.

Turns out this isn't necessary:

https://github.com/simonw/datasette/blob/5aa359b86907d11b3ee601510775a85a90224da8/datasette/utils/init.py#L849-L858

I can use this instead: https://jinja.palletsprojects.com/en/3.1.x/templates/#jinja-filters.filesizeformat

Datasette doesn't limit URL lengths but some common web proxies do - the one in front of Google Cloud Run for example limits to 8KB total for incoming request headers: https://cloud.google.com/load-balancing/docs/quotas#https-lb-header-limits

This means longer SQL queries can break!

Need an optional mechanism for submitting queries by POST instead.

I use this pattirn in a lot of plugin tests, e.g. https://github.com/simonw/datasette-edit-templates/blob/087f6a6cabc20020f2b0524f11aa3a7836320848/tests/test_edit_templates.py#L55-L58 python actor = ds.sign({"a": {"id": "root"}}, "actor") response1 = await ds.client.get( "/-/edit-templates/_footer.html", cookies={"ds_actor": actor} ) I should add this to the documentation on this page: https://docs.datasette.io/en/latest/testing_plugins.html

I haven't done this with any of my other datasette publish tools, but I do think it's a good idea. Being able to publish the entire directory - with templates and plugins and metadata - does seem very useful to me.

Originally posted by @simonw in https://github.com/simonw/datasette-publish-fly/issues/23#issuecomment-1251673489

Could offer syntactic sugar for:

sql create table foo as select * from bar

sqlite-utils create-table data.db foo --sql "select * from bar" https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table

If your phone helpfully adds curly quotes for you then phrase searches against FTS won't work: “Rebecca Sugar”

In regular (not ?_searchmode=raw search mode Datasette could clean these up for you to help avoid that mistake.

https://scotrail.datasette.io/scotrail/assemble_sentence?terms=This+train+is+formed+of%2Cbomb+which

Current title is:

scotrail: with phrases as ( select key, value from json_each('["' || replace(:terms, ',', '","') || '"]')),matches as (select phrases.key, phrases.value, ( select File from announcements where announcements.Transcription like '%' || trim(phrases.value) || '%' order by length(announcements.Transcription) limit 1 ) as Filefrom phrases),results as ( select key, announcements.Transcription, announcements.mp3 from announcements join matches on announcements.File = matches.File order by key)select 'Combined sentence:' as mp3, group_concat(Transcription, ' ') as Transcription, -1 as keyfrom results unionselect mp3, Transcription, keyfrom resultsorder by key

I think a better title would be:

scotrail: assemble_sentence, terms = This train is formed of,bomb which

https://latest.datasette.io/fixtures/paginated_view

Compare with:

The --create option currently creates databases if they are missing, but does not enable WAL mode for them.

It turns out WAL mode is useful for databases that are accepting writes!

I think a --create-wal option that both creates them AND sets WAL mode on any that are created would be a good idea.

It seems to me Janus dependency is not necessary, async with app.database_write_mutex(): out = await app.transaction(func) may be enough.

Comment here: https://lobste.rs/s/fki4tj/architecture_notes_datasette#c_a2ihon

Because I'm using it in this plugin: - https://github.com/simonw/datasette-upload-dbs/issues/1

In #1727 I realized that the SQL tracing was measuring a whole bunch of stuff outside of the SQL query itself.

I started experimenting with this fix for that but it didn't work - I got back an empty JSON array of traces for some reason:

```diff diff --git a/datasette/database.py b/datasette/database.py index ba594a8..d7f9172 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -7,7 +7,7 @@ import sys import threading import uuid

-from .tracer import trace +from .tracer import trace, trace_child_tasks from .utils import ( detect_fts, detect_primary_keys, @@ -207,30 +207,31 @@ class Database: time_limit_ms = custom_time_limit

         with sqlite_timelimit(conn, time_limit_ms):

• try:
• cursor = conn.cursor()
• cursor.execute(sql, params if params is not None else {})
• max_returned_rows = self.ds.max_returned_rows
• if max_returned_rows == page_size:
• max_returned_rows += 1
• if max_returned_rows and truncate:
• rows = cursor.fetchmany(max_returned_rows + 1)
• truncated = len(rows) > max_returned_rows
• rows = rows[:max_returned_rows]
• else:
• rows = cursor.fetchall()
• truncated = False
• except (sqlite3.OperationalError, sqlite3.DatabaseError) as e:
• if e.args == ("interrupted",):
• raise QueryInterrupted(e, sql, params)
• if log_sql_errors:
• sys.stderr.write(
• "ERROR: conn={}, sql = {}, params = {}: {}\n".format(
• conn, repr(sql), params, e
• with trace("sql", database=self.name, sql=sql.strip(), params=params):
• try:
• cursor = conn.cursor()
• cursor.execute(sql, params if params is not None else {})
• max_returned_rows = self.ds.max_returned_rows
• if max_returned_rows == page_size:
• max_returned_rows += 1
• if max_returned_rows and truncate:
• rows = cursor.fetchmany(max_returned_rows + 1)
• truncated = len(rows) > max_returned_rows
• rows = rows[:max_returned_rows]
• else:
• rows = cursor.fetchall()
• truncated = False
• except (sqlite3.OperationalError, sqlite3.DatabaseError) as e:
• if e.args == ("interrupted",):
• raise QueryInterrupted(e, sql, params)
• if log_sql_errors:
• sys.stderr.write(
• "ERROR: conn={}, sql = {}, params = {}: {}\n".format(
• conn, repr(sql), params, e
• ) )
• )
• sys.stderr.flush()
• raise
• sys.stderr.flush()

• raise

       if truncate:
           return Results(rows, truncated, cursor.description)
  

  @@ -238,9 +239,8 @@ class Database: else: return Results(rows, False, cursor.description)

• with trace("sql", database=self.name, sql=sql.strip(), params=params):

• results = await self.execute_fn(sql_operation_in_thread)
• return results
• with trace_child_tasks():

• return await self.execute_fn(sql_operation_in_thread)

  @property def size(self): ```

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1727#issuecomment-1111602802

A page talking about how to run Datasette securely, and security concerns to take into account.

Trace from https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_trace=1&_facet=committer

What's going on in that gap? Can I improve the tracing output to show some non-SQL queries to figure that out?

Noticed this while working on: - #1715

This code here isn't showing up in traces: https://github.com/simonw/datasette/blob/579f59dcec43a91dd7d404e00b87a00afd8515f2/datasette/views/table.py#L218-L220

Because those functions don't use the regular trace-instrumented db.execute() code path - they work directly against a connection instead: https://github.com/simonw/datasette/blob/579f59dcec43a91dd7d404e00b87a00afd8515f2/datasette/utils/init.py#L610-L626

As part of the preparation for the 1.0 release, test all existing known plugins against the alpha.

For any that break, submit a PR suggesting they pin to a version <1.0 - and include a link to the documentation on how to upgrade the plugin for 1.0.

This tutorial made me notice that Datasette is pretty inconsistent with how column quoting works: https://datasette.io/tutorials/learn-sql

It has examples of each of "table_name" and [table_name] and table_name, and it uses single quoted values too.

Datasette should generate SQL as consistently as possible to support learners.

That tutorial should also provide a tiny bit of extra information about what's going on here.

I used show_json.html in the new datasette-packages plugin, which means it will break if that template changes: - https://github.com/simonw/datasette-packages/issues/3

It would be useful if it (or something like it) was documented and stable for plugins to use.

Also relevant: - #878

A reasonable question: "What are those hidden tables for?"

This could be answered by adding a small piece of explanatory text to each table - based on if it's related to FTS or to SpatiaLite or configured to be hidden for some other reason.

Permission checks right now can express one of three opinions:

• False means "so not grant this permisson"
• True means "grant this permission"
• None means "I have no opinion"

But... there's also a concept of a "default" for a given permission check, which might be False or True.

I worry this is too complicated. Could this be simplified before 1.0? In particular the default concept.

See also: - #1676

I wrote datasette-redirect-forbidden today because I needed 403 errors to redirect to /-/login and it was the quickest way to solve that problem.

This should be a feature of Datasette core.

• https://github.com/simonw/datasette-redirect-forbidden/issues/2

https://docs.datasette.io/en/stable/plugin_hooks.html#filters-from-request-request-database-table-datasette doesn't mention that you can return an async function - but you can, and in fact Datasette itself uses that here: https://github.com/simonw/datasette/blob/aa7f0037a46eb76ae6fe9bf2a1f616c58738ecdf/datasette/filters.py#L43-L47

Follows: - #1176

OK, the tests do NOT pass against Windows! https://github.com/simonw/datasette/runs/5044105941

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1626#issuecomment-1028515161

I already do this for sqlite-utils: https://github.com/simonw/sqlite-utils/blob/3.22.1/.github/workflows/test.yml

Related: - #1617 - #1545

Inspired by https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html

Good example: https://github.com/rust-analyzer/rust-analyzer/blob/d7c99931d05e3723d878bea5dc26766791fa4e69/docs/dev/architecture.md

I should start this relatively soon.

Inspired by this: https://news.ycombinator.com/item?id=29729283

Because SQLite is effectively serializing all the writes for us, we have zero locking in our code. We used to have to lock when inserting new items (to get the LastInsertRowId), but the newer version of SQLite supports the RETURNING keyword, so we don't even have to lock on inserts now.

Working on https://github.com/simonw/datasette-search-all/issues/10 made me realize that sometimes it can be difficult to calculate the URL for a database, table or row within Datasette.

It would be useful to have an optional extra JSON extension (using ?_extra= from #262) that can help with this.

https://www.sqlite.org/pragma.html#pragma_query_only

The query_only pragma prevents data changes on database files when enabled. When this pragma is enabled, any attempt to CREATE, DELETE, DROP, INSERT, or UPDATE will result in an SQLITE_READONLY error. However, the database is not truly read-only. You can still run a checkpoint or a COMMIT and the return value of the sqlite3_db_readonly() routine is not affected.

Would it be worth adding this as an extra protection against accidental writes to a DB file over a read-only connection?

While fixing #1519 I added a lot of ugly code that looks like this: https://github.com/simonw/datasette/blob/08947fa76433d18988aa1ee1d929bd8320c75fe2/datasette/facets.py#L228-L230

See these two commits in particular: fe687fd0207c4c56c4778d3e92e3505fc4b18172 and 08947fa76433d18988aa1ee1d929bd8320c75fe2

It would be great to come up with a less verbose and error-prone way of handling this problem.

Spotted in https://github.com/simonw/datasette-graphql/issues/79

Demo: https://datasette-graphql-demo.datasette.io/fixtures/bad_foreign_key

The foreign key links to a 404 page.

This can have text such as "Show 1,000 rows per page", based on the max size limit setting. Would make it easier for people to see more data at once without having to know how to hack the URL, similar to the ... for facet sizes I added in #1337.

I made a plugin for this today but I think perhaps it should be a default thing instead: https://datasette.io/plugins/datasette-template-request

If a table has 200 columns it will take multiple seconds to try and suggest facets. I should either quit after the first 20 or not suggest facets at all.

This is useful for people who are deploying Datasette with any write functionality, especially if they might be using something like EFS.

I can add a section about this to the bottom of https://docs.datasette.io/en/stable/deploying.html and then link to that section from both https://docs.datasette.io/en/stable/sql_queries.html#writable-canned-queries and https://docs.datasette.io/en/stable/internals.html#await-db-execute-write-sql-params-none-block-false

Also useful: mention that just copying a SQLite database file while it is being written to may not get a consistent file, so tell people to use one of the SQLite backup mechanisms instead.

If I was prone to over-thinking (which I am) I'd note that allow_facet and allow_download and allow_csv_stream are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent.

So maybe there's a future task where I introduce those as both permissions and metadata "allow_x" blocks, then rename the settings themselves to be called default_allow_facet and default_allow_download and default_allow_csv_stream.

If I was going to do that I should get it in before Datasette 1.0.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1409#issuecomment-890400425

https://codecov.io/gh/simonw/datasette/src/main/datasette/inspect.py shows only 36% coverage for that module at the moment.

Refs #1160

As discussed in this thread: https://twitter.com/simonw/status/1405554676993433605 - one of the disadvantages of Datasette's streaming CSV feature is that it's hard to tell if you got the whole file or if the connection ended early - or if an error occurred.

Idea: offer an optional ?_end=1 parameter which, if enabled, adds a single row to the end of the CSV file that looks like this:

END,,,,,,,,,

For however many columns the CSV file usually has.

https://docs.datasette.io/en/stable/writing_plugins.html should have tips on writing secure plugins.

This tool is currently almost entirely ignorant of the concept of branches. One example: you can't retrieve commits from any branch other than the default (usually main).

Inspiration here: https://emptystat.es/

It would be cool if you could run Datasette in a long-running query mode, for use with trusted users - something like this:

datasette --unlimited my.db

This would disable the query limit, but would also enable a feature where if a query takes longer than e.g. 1s to return Datasette returns an HTML page to the browser with a progress indicator and polls the server until the query is complete.... but also provides the user with a "cancel" button. This relates to the .interrupt() research in #1270.

Refs https://github.com/simonw/datasette/issues/878

Refs #239. Still needs documentation and CLI implementation.

I want to set far future / immutable cache headers on everything served from /-/static and /-/static-plugins

This has security implications since it will be possible to see what plugins are installed by checking for known static URLs. I'm fine with that - performance is more important here.

I added a deprecation warning to this in #992.

This is a master tracking ticket for various many-2-many features.

e.g. https://latest.datasette.io/fixtures/searchable?_search_text1=barry

In #998 I made a change that affected the table page but didn't affect the query page because I incorrectly assumed they shared rendering logic.

A link to /-/beta?category=1&timestamp__date=2018-08-01&q=swedish should be to /-/beta?category=1&timestamp__date=2018-08-01&q=swedish&sort=newest

https://sqlite-utils.readthedocs.io/en/2.4.2/python-api.html#adding-columns says:

SQLite types you can specify are "TEXT", "INTEGER", "FLOAT" or "BLOB".

As discovered in #92 this isn't the right list of values. I should expand this to match https://www.sqlite.org/datatype3.html

The sqlar format supports this, it would be useful if insert-files could support this too.

https://www.sqlite.org/sqlar.html

It's mentioned in passing in the README but it deserves its own section: $ twitter-to-sqlite fetch \ "https://api.twitter.com/1.1/account/verify_credentials.json" \ | grep '"id"' | head -n 1

https://developer.github.com/v3/gists/#list-starred-gists

It would be cool if the display_sql could return a column populated with GeoJSON which would the automatically be displayed on a map in the results (or maybe default JS would look for a class="geojson" element output by the display template) - ala https://github.com/simonw/datasette-leaflet-geojson

Then I could render workout routes on a map, or Swarm checkin points.

I just wrote this code: python results = await database.execute(SEARCH_SQL, {"query": query}) return [dict(r) for r in results.rows] How about having results.dicts as a utility property that does that?

Once I've got the Datasette plugin to a state where it's worth building a demo: #3

I can use data from my public https://github-to-sqlite.dogsheep.net/ demo plus the Pocket data subset I use for the demo in https://github.com/dogsheep/pocket-to-sqlite/issues/5 - I could pull in the https://dogsheep-photos.dogsheep.net/ photos data too.

The time_added, time_updated and time_read columns currently store data like this:

September 19, 2019 - 00:30:30 UTC

Should use ISO instead, e.g. 2020-07-26T01:05:24+00:00

This is a pattern I often find myself needing. I usually implement this in GitHub Actions like this:

https://github.com/simonw/covid-19-datasette/blob/efa01c39abc832b8641fc2a92840cc3acae2fb08/.github/workflows/scheduled.yml#L52-L63

yaml - name: Set variables to decide if we should deploy id: decide_variables run: |- echo "##[set-output name=latest;]$(datasette inspect covid.db | jq '.covid.hash' -r)" echo "##[set-output name=deployed;]$(curl -s https://covid-19.datasettes.com/-/databases.json | jq '.[0].hash' -r)" - name: Set up Cloud Run if: github.event_name == 'workflow_dispatch' || steps.decide_variables.outputs.latest != steps.decide_variables.outputs.deployed uses: GoogleCloudPlatform/github-actions/setup-gcloud@master This is pretty fiddly. It might be good for datasette publish to grow a helper option that does effectively this - hashes the databases (and the metadata.json) and compares them to the deployed version.

The photos table stores data like this:

id | createdAt | source | prefix | suffix | width | height | visibility | created ▲ | user -- | -- | -- | -- | -- | -- | -- | -- | -- | -- 5e12c9708506bc000840262a | January 06, 2020 - 05:45:20 UTC | Swarm for iOS 1 | https://fastly.4sqi.net/img/general/ | /15889193_AXxGk4I1nbzUZuyYqObgbXdJNyEHiwj6AUDq0tPZWtw.jpg | 1920 | 1440 | public | 2020-01-06T05:45:20 | 15889193

The photo URL can be derived from those pieces - define a SQL view which does that (using datasette-json-html to display the pictures)

--name works inconsistently across the different publish providers - on Cloud Run you should use --service instead for example. Need to review it across all of them and either remove it or clarify what it does.

This is a frustrating workaround. I have a restore_working_directory fixture that I wrote to solve errors that look like this: ``` /Users/simon/Dropbox/Development/datasette/tests/test_publish_cloudrun.py:148:

/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:112: in enter return next(self.gen)

self = <click.testing.CliRunner object at 0x1135ad110>

@contextlib.contextmanager
def isolated_filesystem(self):
    """A context manager that creates a temporary folder and changes
    the current working directory to it for isolated filesystem tests.
    """

  cwd = os.getcwd()

E FileNotFoundError: [Errno 2] No such file or directory `` Here's an example of it in use: removing therestore_working_directory` argument from this function causes the failure. https://github.com/simonw/datasette/blob/549b1c2063db48c4622ee5c7b478a1e3cbc1ac07/tests/test_plugins.py#L689-L690

I'd like to not have to do this.

I'm not sure why this is, but the code coverage I have running in a GitHub Action doesn't take my .coveragerc file into account. It should:

https://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/.github/workflows/test-coverage.yml#L31-L35

Here's the bit that's ignored:

https://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/.coveragerc#L1-L2

As a result my coverage score is 84%, when it should be 92%: 2020-06-13T21:41:18.4404252Z ----------- coverage: platform linux, python 3.8.3-final-0 ----------- 2020-06-13T21:41:18.4404570Z Name Stmts Miss Cover 2020-06-13T21:41:18.4404971Z -------------------------------------------------------- 2020-06-13T21:41:18.4405227Z datasette/__init__.py 3 0 100% 2020-06-13T21:41:18.4405441Z datasette/__main__.py 3 3 0% 2020-06-13T21:41:18.4405668Z datasette/_version.py 279 279 0% 2020-06-13T21:41:18.4405921Z datasette/actor_auth_cookie.py 20 0 100% 2020-06-13T21:41:18.4406135Z datasette/app.py 499 27 95% 2020-06-13T21:41:18.4406343Z datasette/cli.py 162 45 72% 2020-06-13T21:41:18.4406553Z datasette/database.py 236 17 93% 2020-06-13T21:41:18.4406761Z datasette/default_permissions.py 40 0 100% 2020-06-13T21:41:18.4406975Z datasette/facets.py 210 24 89% 2020-06-13T21:41:18.4407186Z datasette/filters.py 122 7 94% 2020-06-13T21:41:18.4407394Z datasette/hookspecs.py 34 0 100% 2020-06-13T21:41:18.4407600Z datasette/inspect.py 36 23 36% 2020-06-13T21:41:18.4407807Z datasette/plugins.py 34 6 82% 2020-06-13T21:41:18.4408014Z datasette/publish/__init__.py 0 0 100% 2020-06-13T21:41:18.4408240Z datasette/publish/cloudrun.py 57 2 96% 2020-06-13T21:41:18.4408786Z datasette/publish/common.py 19 1 95% 2020-06-13T21:41:18.4409029Z datasette/publish/heroku.py 97 13 87% 2020-06-13T21:41:18.4409243Z datasette/renderer.py 63 4 94% 2020-06-13T21:41:18.4409450Z datasette/sql_functions.py 5 0 100% 2020-06-13T21:41:18.4410480Z datasette/tracer.py 87 16 82% 2020-06-13T21:41:18.4410972Z datasette/utils/__init__.py 504 31 94% 2020-06-13T21:41:18.4411755Z datasette/utils/asgi.py 264 24 91% 2020-06-13T21:41:18.4412173Z datasette/utils/shutil_backport.py 44 44 0% 2020-06-13T21:41:18.4412822Z datasette/version.py 4 0 100% 2020-06-13T21:41:18.4413562Z datasette/views/__init__.py 0 0 100% 2020-06-13T21:41:18.4414276Z datasette/views/base.py 288 19 93% 2020-06-13T21:41:18.4414579Z datasette/views/database.py 120 2 98% 2020-06-13T21:41:18.4414860Z datasette/views/index.py 57 2 96% 2020-06-13T21:41:18.4415379Z datasette/views/special.py 72 16 78% 2020-06-13T21:41:18.4418994Z datasette/views/table.py 418 18 96% 2020-06-13T21:41:18.4428811Z -------------------------------------------------------- 2020-06-13T21:41:18.4430394Z TOTAL 3777 623 84%

datasette-atom uses this:

https://github.com/simonw/datasette-atom/blob/df98a6c43a443224b6cd232f84703ec297ef046b/datasette_atom/init.py#L36-L37 python if data.get("human_description_en"): title += ": " + data["human_description_en"] It's a nice way to generate a useful title for a filtered table.

https://dogsheep-photos.dogsheep.net/public/photos_with_apple_metadata?place_city=New+Orleans&_facet=place_city&_facet_array=albums&_facet_array=persons

Here's one way to add that: sql select rowid, photo, ( select json_group_array( json_object( 'label', normalized_string, 'href', '/photos/labelled?_hide_sql=1&label=' || normalized_string ) ) from labels where labels.uuid = photos_with_apple_metadata.uuid ) as labels, date,

This was added in SQLite 2020-01-22 (3.31.0): https://www.sqlite.org/changes.html#version_3_31_0

Add the sqlite3_hard_heap_limit64() interface and the corresponding PRAGMA hard_heap_limit command.

This sounds like it could be a nice extra safety measure.

At the moment if you want to pull all your repos, issues, issues comments etc you have to do it with a sequence of separate commands.

Consider adding a everything or all command which fetches everything that the tool knows how to fetch, and is designed to be run on a cron in a way that fetches just new stuff each time.

The logic for rendering cells in table view and in database (or canned query) view is currently very similar:

https://github.com/simonw/datasette/blob/7656fd64d8b6a32ebc34d89c1b8711cc5ea240f7/datasette/views/base.py#L514-L539

Compared with:

https://github.com/simonw/datasette/blob/7656fd64d8b6a32ebc34d89c1b8711cc5ea240f7/datasette/views/table.py#L104-L195

I'll be changing this a bit in #698 but I should still try to clean this up more further in the future.

https://pypi.org/project/pytest-benchmark/ looks great for this.

Here's how to run it as a github action: https://github.com/rhysd/github-action-benchmark/blob/master/examples/pytest/README.md

https://github.com/dogsheep/twitter-to-sqlite/blob/fb2698086d766e0333a55bb73435e7283feeb438/twitter_to_sqlite/archive.py#L98-L99

If you run Datasette against a number of database files and one of them is malformed, you get this 500 error on the index page:

It would be better if Datasette still worked and listed the databases that were NOT malformed, then showed an inline error message just for the one that could not be accessed.

The Pandas ecosystem is huge, varied and full of tools that are really good at doing interesting analysis on top of tabular data.

Pandas should not be a dependency of Datasette core, but I think there is a lot of potential in having plugins which use Pandas to apply interesting analysis to data sucked out of Datasette's SQLite tables.

One example (thanks, Tony): https://github.com/ResidentMario/missingno could form the basis of a fantastic plugin for getting a high-level overview of how complete each column in a table is.

Some thought is needed here about what shape these kind of plugins might take, and what plugin hooks they would use.