issues

The -a used for the auth credentials and the shortened form of the --all flags were in conflict on the fetch command. To be consistent with other -to-sqlite libraries in the Dogsheep ecosystem, I removed the shortened form of the --all flag.

I'm not happy with how metadata.(json|yaml) keeps growing new features. Rather than having a single plugin hook for all of metadata.json I'm going to split out the feature that shows actual real metadata for tables and databases - source, license etc - into its own plugin-powered mechanism.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/357#issuecomment-647189045_

Part of #1105. Thankfully this isn't yet part of the documented public API on https://docs.datasette.io/en/stable/internals.html

I added a deprecation warning to this in #992.

I realized I need a tracking ticket for this.

I want to start splitting things like plugin configuration and default facets / sort order out of metadata.json - so I want to start calling those things configuration. But the term configuration is already used for the --config family of global settings. So I'm rebranding that type of configuration as settings to free up the name "configuration" for more run-time concerns (default sort order) and plugin configuration.

I'd like to redirect https://docs.datasette.io/en/stable/config.html to a new https://docs.datasette.io/en/stable/settings.html page too. I can use https://docs.readthedocs.io/en/stable/user-defined-redirects.html for that.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1105#issuecomment-733190827_

Like this:

https://user-images.githubusercontent.com/9599/100154418-60bf0d80-2e5a-11eb-92bf-527a14edb5c5.png">

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1106#issuecomment-733248437_

Another knock-on effect of #992.

https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/docs/config.rst#L51-L55

As part of rebranding config to settings, see also #992.

I designed the config format before I had a good feel for CLI design using Click. --config max_page_size 2000 is better than --config max_page_size:2000.

I'd like more thorough unit test coverage of the plugins mechanism - in particular for installable plugins.

I think I can do this while still having the code live in the same repo, by creating a subdirectory in tests/example_plugin with its own setup.py and then running python setup.py install as part of the test runner.

I imagine I will need to bump the version number every time I change the plugin in case someone runs the test again in the same virtual environment.

If that doesn't work I can instead ship a datasette-plugins-tests two to PyPI and add that as a tests_require dependency.

Refs #14

Lots of base_url bugs that I'd like to solve at once.

This is a master tracking ticket for various many-2-many features.

https://docs.datasette.io/en/stable/testing_plugins.html currently shows how to use HTTPX directly.

I'd like to implement this by first extending the register_output_renderer() hook to support streaming huge responses, then switching CSV to use the plugin hook in addition to TSV using it.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1096#issuecomment-732542285_

Refs #1095

I encountered a parse error if the enex file contained š or  

Run command:
evernote-to-sqlite enex evernote.db evernote.enex

Traceback (most recent call last):
...
  File "evernote_to_sqlite/cli.py", line 31, in enex
    save_note(db, note)
  File "evernote_to_sqlite/utils.py", line 35, in save_note
    content = ET.tostring(ET.fromstring(content_xml)).decode("utf-8")
  File "/usr/lib/python3.8/xml/etree/ElementTree.py", line 1320, in XML
    parser.feed(text)
xml.etree.ElementTree.ParseError: undefined entity š: line 3, column 35

Workaround:

sed -i 's/š//g' evernote.enex
sed -i 's/ //g' evernote.enex

I got the following error when executing evernote-to-sqlite enex evernote.db evernote.enex

... 
  File "evernote_to_sqlite/cli.py", line 31, in enex
    save_note(db, note)
  File "evernote_to_sqlite/utils.py", line 28, in save_note
    updated = note.find("updated").text
AttributeError: 'NoneType' object has no attribute 'text'

Seems that in some cases the updated element is not added to the note, this is a part of the problematic note:

<created>20201019T074518Z</created>
<note-attributes>
   <source>web.clip7</source>
   <source-application>webclipper.evernote</source-application>
</note-attributes>

When I perform an OPTIONS request against a database or table datasette fails with an internal error.

All these tests result in the traceback below.

curl -XOPTIONS http://127.0.0.1:8001/test-db
curl -XOPTIONS http://127.0.0.1:8001/test-db/table1
curl -XOPTIONS http://127.0.0.1:8001/test-db/table1\?_search\=test

Traceback (most recent call last):
  File "[path-to-python]/site-packages/datasette/app.py", line 1033, in route_path
    response = await view(request, send)
  File "[path-to-python]/site-packages/datasette/views/base.py", line 146, in view
    request, **request.scope["url_route"]["kwargs"]
  File "[path-to-python]/site-packages/datasette/views/base.py", line 118, in dispatch_request
    return await handler(request, *args, **kwargs)
TypeError: object Response can't be used in 'await' expression

Making the options function in the DataView class async fixed it for me.

    async def options(self, request, *args, **kwargs):
        r = Response.text("ok")
        if self.ds.cors:
            r.headers["Access-Control-Allow-Origin"] = "*"
        return r

Trying to use datasette 0.51.1 gives the following error:

ImportError: cannot import name 'EX_CANTCREAT' from 'os' (C:\Users\drkan\AppData\Local\Programs\Python\Python39\lib\os.py)

Looks like that code is only available on unix: https://docs.python.org/3/library/os.html#os.EX_CANTCREAT

Removing the line makes it work fine (EX_CANTCREAT doesn't seem to be used anywhere?)

This should almost happen as a side-effect or moving from Sanic to Uvicorn during the port to ASGI: #272

Additional steps:

• test it manually
• update documentation
• set up some form of Windows CI

I think it'd be very useful if this tool has an option like --incremental to fetch only newer tweets. This way operations could complete very fast in sequential runs. I'd want to try to implement this feature if it seems OK for this tool's purpose.

Just tested with the latest Docker image, and it works pretty much everywhere! THANK YOU!

I did notice that if I try to export json or csv, the base is not applied. Not sure if I should reopen this issue or open a new one.

To see this, go here: https://corpora.tika.apache.org/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES

Click/hover over json or CSV and you'll see that the 'datasette' base is not included.

_Originally posted by @tballison in https://github.com/simonw/datasette/issues/865#issuecomment-726385422_

Add support for compound foreign keys, as per issue #117

Not sure if this is the right approach. In particular I'm unsure about:

• the new ForeignKey class, which replaces the namedtuple in order to ensure that column and other_column are forced into tuples. The class does the job, but doesn't feel very elegant.
• I haven't rewritten guess_foreign_table to take account of multiple columns, so it just checks for the first column in the foreign key definition. This isn't ideal.
• I haven't added any ability to the CLI to add compound foreign keys, it's only in the python API at the moment.

The PR also contains a minor related change that columns and tables are always quoted in foreign key definitions.

Just installed swarm-to-sqlite-0.3.2 and tried according to the docs:

Traceback (most recent call last):
  File "/usr/local/bin/swarm-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/swarm_to_sqlite/cli.py", line 73, in cli
    save_checkin(checkin, db)
  File "/usr/local/lib/python3.9/site-packages/swarm_to_sqlite/utils.py", line 82, in save_checkin
    checkins_table.m2m("users", user, m2m_table="likes", pk="id")
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1914, in m2m
    id = other_table.insert(record, pk=pk, replace=True).last_pk
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1647, in insert
    return self.insert_all(
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1765, in insert_all
    self.insert_chunk(
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1575, in insert_chunk
    result = self.db.execute(query, params)
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 200, in execute
    return self.conn.execute(sql, parameters)
sqlite3.OperationalError: table users has no column named lastName

Reported on Twitter here: https://twitter.com/ZaneSelvans/status/1328093641395548161

Maybe I'm doing something wrong here but the automatically generated links based foreign key relationships seem to be working here for utility_id_eia, but not for plant_id_eia & generator_id which seems odd: https://pudl-datasette-xl7xwcpe2a-uc.a.run.app/pudl/generators_eia860

Right now it seems like they're trying to, but with only one of the two keys, so it gives "Error 500. You did not supply a value for binding 2." Maybe only create the links when it's a simple foreign key?

Since Datasette now requires Python 3.6, how about some f-strings?

I ran this in the datasette root checkout:

pip install flynt
flynt .
black .

Reported as a bug in #1098 because they caused 500 errors - but it would be even better if Datasette could hyperlink to related rows via compound foreign keys.

Could look something like this:

{
    "title": "Five Thirty Eight",
    "license": "CC Attribution 4.0 License",
    "license_url": "https://creativecommons.org/licenses/by/4.0/",
    "source": "fivethirtyeight/data on GitHub",
    "source_url": "https://github.com/fivethirtyeight/data",
    "databases": {
        "fivethirtyeight": {
            "tables": {
                "mueller-polls/mueller-approval-polls": {
                    "description_html": "<p>....</p>",
                    "columns": {
                        "name_of_column": "column_description goes here"
}

running evernote-to-sqlite 0.2 on windows 10. Command:

evernote-to-sqlite enex evernote.db MyNotes.enex

I get the followinng error:

File "C:\Users\marti\AppData\Roaming\Python\Python38\site-packages\evernote_to_sqlite\utils.py", line 46, in save_note
note_id = db["notes"].insert(row, hash_id="id", replace=True, alter=True).last_pk
TypeError: insert() got an unexpected keyword argument 'replace'

Removing replace=True,

Leads to below error:

note_id = db["notes"].insert(row, hash_id="id", alter=True).last_pk
File "C:\Users\marti\AppData\Roaming\Python\Python38\site-packages\sqlite_utils\db.py", line 924, in insert
return self.insert_all(
File "C:\Users\marti\AppData\Roaming\Python\Python38\site-packages\sqlite_utils\db.py", line 1046, in insert_all
result = self.db.conn.execute(sql, values)
sqlite3.IntegrityError: UNIQUE constraint failed: notes.id

This solves a weird issue when there is record with metadata key
that is only different in letter cases.

See the test for details.

The BaseView class has a method for cascading permission checks, but it's not easily accessible to plugins.

https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/views/base.py#L75-L99

This leaves plugins like datasette-graphql having to implement their own versions of this logic, which is bad: https://github.com/simonw/datasette-graphql/issues/65

First check view-database - if that says False then disallow access, if it says True then allow access. If it says None check view-instance.

This should become a supported API that plugins are encouraged to use.

This isn't particularly straight-forward - all the more reason for Datasette to implement it for you. This article is helpful: http://charlesleifer.com/blog/using-sqlite-full-text-search-with-python/

https://cloud.google.com/run/quotas lists the maximum response size as 32MB.

I spotted a bug where attempting to download a database file larger than that from a Cloud Run deployment (in this case it was https://github-to-sqlite.dogsheep.net/github.db after I accidentally increased the size of that database) returned a 500 error because of this.

Over on Apache Tika, we're using datasette to allow users to make sense of the metadata for our file regression testing corpus.

This could be user error in how I've set up the reverse proxy!

I started datasette like so:
docker run -d -p 8001:8001 -vpwd:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/corpora-metadata.db --config sql_time_limit_ms:60000 --config base_url:/datasette/

I then reverse proxied like so:

ProxyPreserveHost On
ProxyPass /datasette http://x.y.z.q:xxxx
ProxyPassReverse /datasette http://x.y.z.q:xxx

Regular sql works perfectly:
https://corpora.tika.apache.org/datasette/corpora-metadata?sql=select+mime_string%2C+count%281%29+as+cnt%0D%0Afrom+profiles+p%0D%0Ajoin+mimes+m+on+p.mime_id%3Dm.mime_id%0D%0Agroup+by+mime_string%0D%0Aorder+by+cnt+desc

However, adding criteria and clicking 'Apply'
https://corpora.tika.apache.org/datasette/corpora-metadata/tika_1_24_1_mimes?_sort=file&mime__exact=text%2Fplain

bounces back to:
https://corpora.tika.apache.org/corpora-metadata/tika_1_24_1_mimes?_sort=file&file__contains=bug&mime__exact=text%2Fplain

Inspired by https://github.com/django/django/pull/11482

This is an idea that was cut from the first version of writable canned queries:

I really want the option to use a <textarea> for a specific value.

Idea: metadata syntax like this:

json { "databases": { "my-database": { "queries": { "add_twitter_handle": { "sql": "insert into twitter_handles (username) values (:username)", "write": true, "params": { "username": { "widget": "textarea" } } } } } } }

I can ship with some default widgets and provide a plugin hook for registering extra widgets.

This opens up some really exciting possibilities for things like map widgets that let you draw polygons.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/698#issuecomment-608125928_

I got this on my (private) https://dogsheep.simonwillison.net/twitter/tweets/1188612004572880896 page:

https://user-images.githubusercontent.com/9599/98895899-a6440980-245c-11eb-98e8-fb97d10ec2fd.png">

Working on https://github.com/simonw/datasette-search-all/issues/10 made me realize that sometimes it can be difficult to calculate the URL for a database, table or row within Datasette.

It would be useful to have an optional extra JSON extension (using ?_extra= from #262) that can help with this.

Just spotted that fixtures.db uses FTS3, which is pretty much obsolete these days.

https://github.com/simonw/datasette/blob/13d1228d80c91d382a05b1a9549ed02c300ef851/tests/fixtures.py#L488-L489

Refs #1081

https://user-images.githubusercontent.com/9599/98856656-bafbaf80-2412-11eb-8070-4f3a47511973.png">

The HTML looks like this:

<td class="col-tag_id type-int"><a href="/index/core_tag/1"></a>&nbsp;<em>1</em></td>

I'd like to get new sqlite-utils (and Datasette) releases submitted to Homebrew as painlessly as possible.

A mechanism for specifying columns that should be configured for full-text search as part of the initial data import:

sqlite-utils insert mydb.db articles articles.csv --csv -f title -f body

While working on search (#192) I've spotted a few small changes I would like to make that would break backwards compatibility in minor ways, hence requiring a 3.x release.

db[table].search() - I would like this to default to sorting by rank

Also I'd like to free up the -c and -f options for other purposes from the standard output formats here:

https://github.com/simonw/sqlite-utils/blob/43eae8b193d362f2b292df73e087ed6f10838144/sqlite_utils/cli.py#L48-L58

I'd like -f to be used to indicate a full-text search column during an insert and -c to indicate a column (so you can specify which columns you want to output).

I need to improve this method to help build sqlite-utils search in #192 (PR is #195).

The challenge is deciding how it should handle sorting by relevance - especially since that is easy in FTS5 but not at all easy in FTS4.

Latest test failure:
114 -> assert [("racoons are biting trash pandas", "USA", "bar")] == table.search( 115 "bite", order="rowid" 116 ) 117 118 119 def test_optimize_fts(fresh_db): (Pdb) table.search("bite") [(2, 'racoons are biting trash pandas', 'USA', 'bar', -9.641434262948206e-07)]
The problem here is that the table.search() method now behaves differently for FTS4 v.s. FTS5 tables.

With FTS4 you get back just the table columns.

With FTS5 you also get back the rowid as the first column and the rank score as the last column.

This is weird. It also makes me question whether having .search() return a list of tuples is the right API design.

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/195#issuecomment-722542895_

A command that knows how to run a search against a FTS enabled table and return results ranked by relevance.

Both table.search(columns=) and the sqlite-utils search -c option do not work as expected - they always return both the rowid and the rank columns even if those have not been requested.

This should be fixed before the 3.0 non-alpha release.

To let you specify the exact columns you want. Based on the -c option to sqlite-utils search in #192.

We already support --csv for output, and the insert command accepts --tsv. The output format options should accept --tsv too.

According to https://www.sqlite.org/fts5.html (not sure about FTS4) results can be sorted by relevance. At the moment results are returned by default by rowid. Perhaps a flag can be added to the search method?

Refs #192. Still needs tests.

For #192 and #197 I've decided I want to be able to order by relevance in FTS4 as well as FTS5.

This means I need to port over my work on bm25() from https://github.com/simonw/sqlite-fts4 (since I don't want to add a full dependency).

I'd like a mechanism to optionally avoid registering a custom function if it has already been registered.

SQLite doesn't seem to offer a way to introspect registered custom functions so I'll need to track what has already been registered in sqlite-utils instead.

Should I register the custom rank_bm25 SQLite function for every connection, or should I register it against the connection just the first time the user attempts an FTS4 search? I think I'd rather register it only if it is needed.

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723145383_

I want .search() to work against both FTS5 and FTS4 tables - but sort by rank should only work for FTS5.

This means I need to be able to introspect and tell if a table is FTS4 or FTS5.

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/192#issuecomment-722054264_

1. Have a sqlite db stored in Dropbox
2. Previously tried the Digital Ocean build pack minimal approach (e.g. Procfile, requirements.txt, bin/post_compile)
3. bin/post_compile with wget from Dropbox
4. download of large sqlite db is successful
5. log reveals that when building Docker container, Digital Ocean runs out of memory for 5gb+ sqlite db but works fine for 2gb+ sqlite db

There's no link to download the CSV file - the table page has that as an advanced export option, but this is missing from the query page.

The insert command currently results in string types for every column - at least when used against CSV or TSV inputs.

It would be useful if you could do the following:

• automatically detects the column types based on eg the first 1000 records
• explicitly state the rule for specific columns

--detect-types could work for the former - or it could do that by default and allow opt-out using --no-detect-types

For specific columns maybe this:

sqlite-utils insert db.db images images.tsv \
  --tsv \
  -c id int \
  -c score float

I have a dataset with many columns, of which only some are likely to be of interest for searching.

It would be great for usability if the search filters in the UI could be configured to include/exclude columns.

See also: https://github.com/simonw/datasette/issues/292

e.g. https://latest.datasette.io/fixtures/searchable?_search_text1=barry

If a table has hundreds of columns the Datasette UI starts getting unwieldy.

Addressing this would be neat. One option would be to only select the first 30 columns by default and provide a UI for selecting more.

Can use /database/-/... namespace from #296

Nothing uses this plugin hook yet, so the design is not yet proven.

I'm going to build a real plugin against it and use that process to inform any design changes that may need to be made.

I'll add a warning about this to the documentation.

Like column_actions but adds a cog menu to the database page.

On this page when signed in as root: https://latest.datasette.io/fixtures/roadside_attraction_characteristics/1

If a text column actually contains numbers, being able to "sort by column, treated as numeric" would be really useful.

Probably depends on column actions enabled by #690

I'm running datasette serve --config base_url:/foo/ …, proxying to it with this Apache config:

    ProxyPass /foo/ http://localhost:8001/ 
    ProxyPassReverse /foo/ http://localhost:8001/

and then accessing it via https://example.com/foo/.

Although many of the URLs in the pages are correct (presumably because they either use absolute paths which include base_url or relative paths), the faceting and pagination links still use fully-qualified URLs pointing at http://localhost:8001.

I looked into this a little in the source code, and it seems to be an issue anywhere request.url or request.path is used, as these contain the values for the request between the frontend (Apache) and backend (Datasette) server. Those properties are primarily used via the path_with_… family of utility functions and the Datasette.absolute_url method.

Datasette views currently work by creating a set of data that should be returned as JSON, then defining an additional, optional template_data() function which is called if the view is being rendered as HTML.

This template_data() function calculates extra template context variables which are necessary for the HTML view but should not be included in the JSON.

Example of how that is used today: https://github.com/simonw/datasette/blob/2b79f2bdeb1efa86e0756e741292d625f91cb93d/datasette/views/table.py#L672-L704

With features like Facets in #255 I'm beginning to want to move more items into the template_data() - in the case of facets it's the suggested_facets array. This saves that feature from being calculated (involving several SQL queries) for the JSON case where it is unlikely to be used.

But... as an API user, I want to still optionally be able to access that information.

Solution: Add a ?_extra=suggested_facets&_extra=table_metadata argument which can be used to optionally request additional blocks to be added to the JSON API.

Then redefine as many of the current template_data() features as extra arguments instead, and teach Datasette to return certain extras by default when rendering templates.

This could allow the JSON representation to be slimmed down further (removing e.g. the table_definition and view_definition keys) while still making that information available to API users who need it.

https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++%5Bfoo%5D

Would be useful if this page showed you the invalid SQL you entered so you can edit it and try again.

In #981 I added data-column= attributes to the <th> on the table page. These should become part of Datasette's documented API so JavaScript plugin authors can use them to derive things about the tables shown on a page (`datasette-cluster-map uses them as-of https://github.com/simonw/datasette-cluster-map/issues/18).

The datasette-write plugin provides this at the moment https://github.com/simonw/datasette-write - but it feels like it should be a built-in capability, protected by a default permission.

UI concept: if you have write permission then the existing SQL editor gets an "execute write" checkbox underneath it.

JavaScript can spot if you appear to be trying to execute an UPDATE or INSERT or DELETE query and check that checkbox for you.

If you link to a query page with a non-SELECT then that query will be displayed in the box ready for you to POST submit it. The page will also then get "cannot be embedded" headers to protect against clickjacking.

The default JSON just isn't right. I find myself using ?_shape=array for almost everything I build against the API.

Can be part of #870 - refactoring existing views to use register_routes().

I'm going to put the new check_permissions() method on BaseView as well. If I want that method to be available to plugins I can do so by turning that BaseView class into a documented API that plugins are encouraged to use themselves.
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/832#issuecomment-651995453_

https://latest.datasette.io/fixtures/facetable.json doesn't currently include the CREATE TABLE statement for the page, even though it's available on the HTML version at https://latest.datasette.io/fixtures/facetable

Using datasette to solve some frustrating problems with our fulfillment provider today, I was surprised to see repeated requests for assets under /-/static and the favicon. While it won't likely be a huge performance bottleneck, I bet datasette would feel a bit zippier if you had Uvicorn serving up some caching-related headers telling the browser it was safe to cache static assets.

It would also be interesting to try out the SQL hint mode, which can autocomplete against tables and columns. This demo shows how to configure that: https://codemirror.net/mode/sql/

Some missing documentation: https://stackoverflow.com/questions/20023381/codemirror-how-add-tables-to-sql-hint
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/948#issuecomment-679355426_

https://github.com/simonw/datasette/blob/5db7ae3ce165ded57c7fb1cfbdb3258b1cf06c10/datasette/app.py#L1307-L1313

Start by combining release notes from https://github.com/simonw/datasette/releases/tag/0.51a0 and https://github.com/simonw/datasette/releases/tag/0.51a1 and https://github.com/simonw/datasette/releases/tag/0.51a2

Split from #1040, refs #1036.

This can go on this page: https://docs.datasette.io/en/stable/deploying.html

Refs #1023, #865

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1033#issuecomment-714681365_

Refs #1023

Added in 7a67bc7a569509d65b3a8661e0ad2c65f0b09166 refs #1026. Lots of tests are failing now.

Includes like this one always go to disk, without hitting the load_template plugin hook:

<footer class="ft">{% block footer %}{% include "_footer.html" %}{% endblock %}</footer>

It bothers me that these aren't visible in any public demos. Maybe latest.datasette.io should include the my_plugins.py and my_plugins2.py plugins?
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1067#issuecomment-719961701_

Refs #1023. If Datasette is running with a base_url setting and a plugin calls e.g. datasette.client.get("/-/plugins.json") what should happen?

That datasette.urls.table("db", "table") + ".json" example is bad because if the table name contains a . it should be ?_format=json instead.

Maybe .table() should have a format="json" option that knows how to do this.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1026#issuecomment-712962517_

Follow-on from #1066.

I should include an example in the documentation that shows loading templates from a database table.
_Originally posted by @simonw in https://github.com/simonw/datasette/pull/1069#issuecomment-719664530_

I couldn't get it working correctly with async (necessary for include/extend to function), and on deeper investigation it appears that I can build something equivalent to what I wanted using the existing prepare_jinja2_environment hook.

I'm going to remove the load_template plugin hook and see if it's possible to build the edit templates extension against prepare_jinja2_environment instead.
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1072#issuecomment-719833744_

In the pattern portfolio:

https://user-images.githubusercontent.com/9599/97752509-5a36b380-1ab1-11eb-9a26-700cb295fe11.png">

But they're currently showing like this:

https://user-images.githubusercontent.com/9599/97752544-69b5fc80-1ab1-11eb-80c2-c88aabe6a832.png">

This can work with the Jinja template loaders. It would unlock things like storing templates in SQLite.

Refs #1042

https://github.com/simonw/datasette/blob/18a64fbb29271ce607937110bbdb55488c43f4e0/datasette/default_menu_links.py#L4-L6

This should check a named permission so that it can be customized by permission plugins.

Now that we have support for plugins that can write I'm seeing all sorts of places where a plugin might need to add UI to the table page.

Some examples:

• datasette-configure-fts needs to add a "configure search for this table" link
• a plugin that lets you render or delete tables needs to add a link or button somewhere
• existing plugins like datasette-vega and datasette-cluster-map already do this with JavaScript

The challenge here is that multiple plugins may want to do this, so simply overriding templates and populating names blocks doesn't entirely work as templates may override each other.

For the table actions: attaching it to a cog icon next to the table name could make sense.

https://user-images.githubusercontent.com/9599/96168896-45192c80-0ed6-11eb-9d90-6172a8346468.png">

This is the column action icon at twice the size, color #666.
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/690#issuecomment-709497595_

Closes #1064, refs #690.

Needed for #690. Prototype in https://github.com/simonw/datasette/commit/0d7ac764861d84be24d661cf4104ce61ea11a82a

I just noticed this: https://latest.datasette.io/fixtures/binary_data.csv

rowid,data
1,b'\x15\x1c\x02\xc7\xad\x05\xfe'
2,b'\x15\x1c\x03\xc7\xad\x05\xfe'

There's no good way to represent binary data in a CSV file, but this seems like one of the more-bad options.

• Update .csv output to link to these things (and get that xfail test to pass)
• <del>Add a .csv?_blob_base64=1 argument that causes them to be output in base64 in the CSV</del>

Moving the CSV work to a separate ticket.
_Originally posted by @simonw in https://github.com/simonw/datasette/pull/1061#issuecomment-719042601_