issue_comments

I renamed /:memory: to /_memory in version 0.55 - https://docs.datasette.io/en/stable/changelog.html#v0-55

But... in 0.59 I stopped following HTTP redirects by default, which is why this used to work and no longer does!

So the fix is to update the Homebrew regression test to use this instead:

datasette --get '/_memory.json?sql=select+3*5'

Thanks for catching this!

https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid is fixed now.

Since that Janus PR hasn't been merged yet, one temporary option for a fix would be to entirely vendor the fixed Janus - https://github.com/aio-libs/janus/blob/9e13d3fb74e2c93d7501443b370a455d1b302b1f/janus/__init__.py - since it's only a single module.

This is blocking an upgrade for the Homebrew Datasette package: https://github.com/Homebrew/homebrew-core/pull/86932

The test there failed because it turns out there's a whole bunch of places that set the Access-Control-Allow-Origin header. I'm going to close this PR and ship a fix that refactors those places to use the same code.

This looks like a good fix to me.

This is a great idea, thanks.

@dependabot recreate

I think this is the change in httpx which is causing the bug for me: https://github.com/encode/httpx/commit/ff9813e84dab56f0f3c4ef3a159a4cce8c644a91#diff-0d0cbe9ebcd03cc8c780b0407762540a082f70cc64257f2fcd588cc30f43c15cR96

https://user-images.githubusercontent.com/9599/137224339-0aed171f-b351-4d67-8d6c-ad22f2301f18.png">

Previously it was using path from path, _, query = full_path.partition(b"?") to populate the raw_path key - but it changed to instead using request.url.raw_path which presumably implements the logic that includes the query string.

This is weird - as far as I can tell httpx has included the query string in raw_path for well over a year: https://github.com/encode/httpx/commit/8e4a8a1c73f60fe5754f95b308beaa725cb8791d#diff-c9a78eb3b5f5c4fac4e5552165fbdd5320c7e3fadf9eedabcb5461393466c090R235

(Pdb) request.scope['path']
'/_memory.json'
(Pdb) request.scope['raw_path']
b'/_memory.json?sql=select+sqlite_version()'

So raw_path now includes the query string.

I think the issue is in route_path():

> /Users/simon/Dropbox/Development/datasette/datasette/app.py(1182)route_path()
-> response = await view(request, send)
(Pdb) path
'/_memory.json?sql=select+sqlite_version()'

Upgrading to 0.20.0 gives me lots of the following errors:

'{"ok": false, "error": "Database not found: .json?_sort=relationships", "status": 404, "title": null}'

It looks like the full query string is now being treated as the name of the database.

The core problem here is treating the ?_facet= query string parameters as the point of truth for which facets are currently enabled.

Instead, I could use a data- attribute on the displayed facets.

Thanks for this!

I'll open a separate issue for removing _next= when running a search.

I think that SQLite error message difference was caused by https://github.com/python/cpython/commit/a50e28377bcf37121b55c2de70d95a5386c478f8 or related work.

I applied my PR against Janus to my local copy of Datasette like so:

pip uninstall janus
pip install https://github.com/aio-libs/janus/archive/9e13d3fb74e2c93d7501443b370a455d1b302b1f.zip

Then I ran the Datasette tests and got a much happier pass rate.

I submitted a PR to Janus with a workaround for this: https://github.com/aio-libs/janus/pull/359

There's a tiny chance this could be a bug in Python 3.10 itself - I filed an issue here: https://bugs.python.org/issue45416 - in which I said:

In Python 3.10 it is not possible to instantiate an asyncio.Condition that wraps an asyncio.Lock without raising a "loop argument must agree with lock" exception.

This code raises that exception:

asyncio.Condition(asyncio.Lock())

This worked in previous Python versions.

Note that the error only occurs if an event loop is running. Here's a simple script that replicates the problem:

import asyncio

# This runs without an exception:
print(asyncio.Condition(asyncio.Lock()))

# This does not work:
async def example():
    print(asyncio.Condition(asyncio.Lock()))

# This raises "ValueError: loop argument must agree with lock":
asyncio.run(example())

And here's the relevant Janus code: https://github.com/aio-libs/janus/blob/d7970f8b76bcac2e087067ca4575ac845e481874/janus/__init__.py#L24-L42

class Queue(Generic[T]):
    def __init__(self, maxsize: int = 0) -> None:
        self._loop = current_loop()
        self._maxsize = maxsize

        self._init(maxsize)

        self._unfinished_tasks = 0

        self._sync_mutex = threading.Lock()
        self._sync_not_empty = threading.Condition(self._sync_mutex)
        self._sync_not_full = threading.Condition(self._sync_mutex)
        self._all_tasks_done = threading.Condition(self._sync_mutex)

        self._async_mutex = asyncio.Lock()
        # "loop argument must agree with lock" exception is raised here:
        self._async_not_empty = asyncio.Condition(self._async_mutex)
        self._async_not_full = asyncio.Condition(self._async_mutex)
        self._finished = asyncio.Event()
        self._finished.set()

Here's the code that raises that error: https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/asyncio/locks.py#L219-L234

class Condition(_ContextManagerMixin, mixins._LoopBoundMixin):
    """Asynchronous equivalent to threading.Condition.
    This class implements condition variable objects. A condition variable
    allows one or more coroutines to wait until they are notified by another
    coroutine.
    A new Lock object is created and used as the underlying lock.
    """

    def __init__(self, lock=None, *, loop=mixins._marker):
        super().__init__(loop=loop)
        if lock is None:
            lock = Lock()
        elif lock._loop is not self._get_loop():
            raise ValueError("loop argument must agree with lock")

Only mention I can find of that "loop argument must agree with lock" error is here - which doesn't have any tips for a workaround yet: https://giters.com/django/channels_redis/issues/278

So maybe this is an issue with Janus? I'm using https://pypi.org/project/janus/ 0.6.1 which is the latest release, from October 2020.

Running pytest -x --pdb helped me see this error:

  File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 122, in dispatch_request
    await self.ds.refresh_schemas()
  File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 344, in refresh_schemas
    await self._refresh_schemas()
  File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 349, in _refresh_schemas
    await init_internal_db(internal_db)
  File "/Users/simon/Dropbox/Development/datasette/datasette/utils/internal_db.py", line 5, in init_internal_db
    await db.execute_write(
  File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 102, in execute_write
    return await self.execute_write_fn(_inner, block=block)
  File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 113, in execute_write_fn
    reply_queue = janus.Queue()
  File "/Users/simon/.local/share/virtualenvs/py310-Z8fTATkJ/lib/python3.10/site-packages/janus/__init__.py", line 39, in __init__
    self._async_not_empty = asyncio.Condition(self._async_mutex)
  File "/Users/simon/.pyenv/versions/3.10.0/lib/python3.10/asyncio/locks.py", line 234, in __init__
    raise ValueError("loop argument must agree with lock")
ValueError: loop argument must agree with lock

Then I created myself a temporary 3.10 environment using pipenv like so:

cd /tmp
mkdir py310
cd py310
pipenv shell --python /Users/simon/.pyenv/versions/3.10.0/bin/python

And used that with my Datasette checkout like so:

cd ~/.../datasette
pip install -e '.[test]'
pytest

The 3.10 tests failed a lot. Trying to run this locally:

/tmp % pyenv install 3.10
python-build: definition not found: 3.10

The following versions contain `3.10' in the name:
  3.10.0a6
  3.10-dev
  miniconda-3.10.1
  miniconda3-3.10.1

See all available versions with `pyenv install --list'.

If the version you need is missing, try upgrading pyenv:

  brew update && brew upgrade pyenv

So trying:

brew update && brew upgrade pyenv

Then did this:

/tmp % brew upgrade pyenv 
==> Upgrading 1 outdated package:
pyenv 1.2.24.1 -> 2.1.0

This decided to upgrade everything by downloaded everything on the internet. Aah, Homebrew.

But it looks like I have 3.10.0 available to pyenv now.

/tmp % pyenv install 3.10.0
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.0.tar.xz...
-> https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tar.xz
Installing Python-3.10.0...
...

I've had this problem too - my solution was to not use Cloud Run for databases larger than about 2GB, but the way you describe it here makes me think that maybe there is a workaround here which could get it to work.

On further investigation this isn't related to _search at all - it happens when you explicitly sort by _sort=rowid and apply a _next

• https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 works without an error (currently)
• https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid shows that error

Here's the exception:

-> params[f"p{len(params)}"] = components[0]
(Pdb) list
603     
604                     # Figure out the SQL for next-based-on-primary-key first
605                     next_by_pk_clauses = []
606                     if use_rowid:
607                         next_by_pk_clauses.append(f"rowid > :p{len(params)}")
608  ->                     params[f"p{len(params)}"] = components[0]
609                     else:
610                         # Apply the tie-breaker based on primary keys
611                         if len(components) == len(pks):
612                             param_len = len(params)
613                             next_by_pk_clauses.append(

Debugger shows that components is an empty array, so components[0] cannot be resolved:

-> params[f"p{len(params)}"] = components[0]
(Pdb) params
{'search': 'hello'}
(Pdb) components
[]

So the bug is in this code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/views/table.py#L604-L617

Actually no, from that stack trace you provided:

  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\grott\anaconda3\lib\site-packages\datasette\cli.py", line 283, in package
    call(args)
  File "c:\users\grott\anaconda3\lib\contextlib.py", line 119, in __exit__
    next(self.gen)
  File "c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py", line 451, in temporary_docker_directory
    tmp.cleanup()

It looks like the problem occurs here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L449-L452

I suspect the root cause of this may be in this code: https://github.com/simonw/datasette/blob/63886178a649586b403966a27a45881709d2b868/datasette/utils/__init__.py#L673-L677

I'm pretty sure this is a Windows issue, not a Fly issue. I imagine it affects other forms of datasette publish too.

I could even have those replacement characters be properties of the Database class, so uses can sub-class and change them.

class Database:
    left_brace_replace = "_"
    right_brace_replace = "_"
    ...

I think I like the underscore option best. I don't like the idea of injecting surprise ( ) parenthesis, and having them vanish entirely could result in things like item[price] becoming itemprice which feels confusing.

item_price_ is a little ugly but I think I can live with it.

What are my options for replacing those characters?

• [ becomes ( and ] becomes )
• [ and ] are removed entirely
• [ and ] both become _

I'm inclined to just fix them and not have an option for opting-out of fixing them, since it adds quite a bit of cruft to the overall API design for an option that maybe no-one will ever use.

If I add a new parameter for opting in and out of fixing these, what should it be called? A few options:

• fix_columns=False
• rename_bad_columns=False
• fix_column_names=False
• error_on_invalid_column_names=True

Here's the code where this happens: https://github.com/simonw/sqlite-utils/blob/54191d4dc114d7dc21e849b48a4d5ae4f9e601ca/sqlite_utils/db.py#L2943-L2948

It's called from three different methods in db.py: create_table_sql(), update() and insert_all().

So either I do the automatic replacement, or I let the user request automatic replacement, or a third option: I do automatic replacement but let the user opt to receive an error instead.

Quoting https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856 in full:

I'm not sure what to do about this one.

I can't fix it: this bug in Python's sqlite3 module means that even if I write a database out with column names that include [] I won't be able to read them back again.

So... I could do one of the following:

• Throw an error if a column name includes those characters. That's my preferred option I think.
• Automatically replace [ in column names with ( and ] with )
• Do the automatic replacement but show a user-visible warning when I do it
• Throw an error, but give the user an option to run with e.g. --fix-column-names which applies that automatic fix.

Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.

Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-directly-against-csv-or-json

If two files have the same name they will be assigned a numeric suffix:

$ sqlite-utils memory foo/data.csv bar/data.csv "select * from data_2"

Here's the relevant code: https://github.com/simonw/sqlite-utils/blob/7427a9137f60de961b6331d0922a3f03da0d1890/sqlite_utils/cli.py#L1289-L1292

I can fix this by checking to see if csv_table is already in use and adding a suffix.

Oddly I can't replicate this on macOS:

(sqlite-utils) sqlite-utils % ls foo/*.csv
foo/bug.csv
(sqlite-utils) sqlite-utils % ls bar/*.csv
bar/bug.csv
(sqlite-utils) sqlite-utils % sqlite-utils memory foo/bug.csv bar/bug.csv --schema
CREATE TABLE "bug" (
   [col1] TEXT,
   [col2] TEXT
);
CREATE VIEW t1 AS select * from [bug];
CREATE VIEW t AS select * from [bug];

The t1 and t2 aliases were meant to handle this case, but the are no good if the tool throws an error.

The bug is in this code: https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/sqlite_utils/cli.py#L2205-L2227

Good catch, thanks.

Side-note: Django 4.0 will switch from using pytz to using the standard library zoneinfo module introduced in Python 3.9, which has a backport that works as far back as 3.6: https://github.com/pganssle/zoneinfo (https://pypi.org/project/backports.zoneinfo/)

If I need to handle timezones I'll use that, but I think I can get away without it?

Django does this: https://github.com/django/django/blob/b0ed619303d2fb723330ca9efa3acf23d49f1d19/setup.cfg#L39-L43

install_requires =
    asgiref >= 3.3.2
    backports.zoneinfo; python_version<"3.9"
    sqlparse >= 0.2.2
    tzdata; sys_platform == 'win32'

The audiences I care about here are:

• Producers of this timestamp - generally that will be users who are using datasette publish to share their data
• Human consumers of this timestamp - end users who look at a Datasette site and want to know how recent the data is
• Machine consumers of this timestamp - API integrations that might want to check if a Datasette instance has been updated before downloading new data

For producers I think there are going to be two categories. The first is users who run "publish" and want the site to reflect when they did so (probably using --updated=now when they publish). The second are users who are willing to spend more time thinking about this - for example my various git scraping projects where I want to use a date derived from the git history.

For humans... I'd like to be able to calculate a relative time (3 hours ago) in addition to showing the display time, because that helps avoid confusion over timezones.

For machine consumers, it might be nice to offer the option of a calculated Unix timestamp-since-1970, since those can be easier to work with in some languages than running a full ISO date parser.

The simplest possible version of this: it's always represented as a UTC ISO datetime, like this:

"updated": "2020-10-31T12:00:00+00:00"

Later versions of Datasette could extend this to handle other timezones or support just the date (though that's a backwards incompatible change so probably better to decide on the date thing right now).

So this is a metadata key called updated which can be applied at the table, database or instance level. It is represented as a .isoformat() timestamp.

Question: should I support just the date - yyyy-mm-dd - in addition to the datetime?

I think so. I can easily imagine situations where the exact time of day that a change was made hasn't been recorded, but the overall date is known.

But in that case, should the updated key sometimes be yyyy-mm-dd and sometimes be the full isoformat datetime? Or should there be an updated_date key that's used for just the date?

I'm going to use https://github.com/dateutil/dateutil for this - it's been maintained constantly (by an evolving team of contributors) since 2003 and is a very trustworthy dependency.

Thanks for this!

Still a good idea today too! Would be great for https://cdc-vaccination-history.datasette.io/ for example.

That's so useful @sethvincent! Really interesting reading your code there, especially clever how you're using the base_url config.

I'd be very interested to see what your demo looks like without using serverless - completely agree that the less additional dependencies there are for this the better.

I'm also very interested in figuring out a way to run Datasette in Lambda but with the SQLite database on an EFS volume. Do you have a feel for how hard that would be?

Especially relevant now that 0.2.0 is out which is a much higher quality release. https://github.com/simonw/datasette-app/releases/tag/0.2.0

Here's a already open issue for this: #972

At the moment this isn't possible - though there's a workaround which is to define a SQL view for the query, at which point facets will be displayed again.

I did a lot of the work required to support this when I refactored how facets worked a while back - but to finally implement this I need to refactor the table view and the arbitrary query view to share much more logic than they do at the moment.

Here's the code at fault: https://github.com/simonw/datasette/blob/b28b6cd2fe97f7e193a235877abeec2c8eb0a821/datasette/static/table.js#L137-L146

I think this is a setting. There are two relevant settings at the moment:

    "template_debug": false,
    "trace_debug": false,

For consistence then this should be called something_debug - but do I want a single setting that exposes the _internal database and adds those debug options to the menu, or do I want those as two separate settings?

• internal_debug to enable access to that _internal database
• menu_debug for those menu options?

I don't think I'll adopt it for this project. For example, here:

     response = Response.redirect("/")
-    response.set_cookie("ds_actor", datasette.sign({
-        "a": {
-            "id": "cleopaws"
-        }
-    }, "actor"))
+    response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor"))

I chose to use the multi-line version to help emphasize the structure - the single-line replacement loses that. I think I'll continue to make my own editorial choices about how the code examples are laid out.

Here's the diff it produced from that first run:
`diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..8008023 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,7 @@ Authentication plugins can set signedds_actor`` cookies themselves like so:
.. code-block:: python

 response = Response.redirect("/")

• response.set_cookie("ds_actor", datasette.sign({
• "a": {
• "id": "cleopaws"
• }
• }, "actor"))
• response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor"))

Note that you need to pass "actor" as the namespace to :ref:datasette_sign.

@@ -412,12 +408,16 @@ To include an expiry, add a "e" key to the cookie value containing a `base62
expires_at = int(time.time()) + (24 * 60 * 60)

 response = Response.redirect("/")

• response.set_cookie("ds_actor", datasette.sign({
• "a": {
• "id": "cleopaws"
• },
• "e": baseconv.base62.encode(expires_at),
• }, "actor"))
• response.set_cookie(
• "ds_actor",
• datasette.sign(
• {
• "a": {"id": "cleopaws"},
• "e": baseconv.base62.encode(expires_at),
• },
• "actor",
• ),
• )

The resulting cookie will encode data that looks something like this:

diff --git a/docs/spatialite.rst b/docs/spatialite.rst
index d1b300b..556bad8 100644
--- a/docs/spatialite.rst
+++ b/docs/spatialite.rst
@@ -58,19 +58,22 @@ Here's a recipe for taking a table with existing latitude and longitude columns,
.. code-block:: python

 import sqlite3

• conn = sqlite3.connect('museums.db')
  +
• conn = sqlite3.connect("museums.db")
  # Lead the spatialite extension:
  conn.enable_load_extension(True)
• conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
• conn.load_extension("/usr/local/lib/mod_spatialite.dylib")
  # Initialize spatial metadata for this database:
• conn.execute('select InitSpatialMetadata(1)')
• conn.execute("select InitSpatialMetadata(1)")
  # Add a geometry column called point_geom to our museums table:
  conn.execute("SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);")
  # Now update that geometry column with the lat/lon points
• conn.execute('''
• conn.execute(
• """
  UPDATE museums SET
  point_geom = GeomFromText('POINT('||"longitude"||' '||"latitude"||')',4326);
• ''')
• """

• )
  # Now add a spatial index to that column
  conn.execute('select CreateSpatialIndex("museums", "point_geom");')
  # If you don't commit your changes will not be persisted:
  @@ -186,13 +189,14 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
  .. code-block:: python

  import sqlite3
  - conn = sqlite3.connect('places.db')
  +
  + conn = sqlite3.connect("places.db")
  # Enable SpatialLite extension
  conn.enable_load_extension(True)
  - conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
  + conn.load_extension("/usr/local/lib/mod_spatialite.dylib")
  # Create the masic countries table
  - conn.execute('select InitSpatialMetadata(1)')
  - conn.execute('create table places (id integer primary key, name text);')
  + conn.execute("select InitSpatialMetadata(1)")
  + conn.execute("create table places (id integer primary key, name text);")
  # Add a MULTIPOLYGON Geometry column
  conn.execute("SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);")
  # Add a spatial index against the new column
  @@ -201,13 +205,17 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
  from shapely.geometry.multipolygon import MultiPolygon
  from shapely.geometry import shape
  import requests
  - geojson = requests.get('https://data.whosonfirst.org/404/227/475/404227475.geojson').json()
  +
  + geojson = requests.get(
  + "https://data.whosonfirst.org/404/227/475/404227475.geojson"
  + ).json()
  # Convert to "Well Known Text" format
  - wkt = shape(geojson['geometry']).wkt
  + wkt = shape(geojson["geometry"]).wkt
  # Insert and commit the record
  - conn.execute("INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))", (
  - "Wales", wkt
  - ))
  + conn.execute(
  + "INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))",
  + ("Wales", wkt),
  + )
  conn.commit()

Querying polygons using within()
diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst
index bd60a4b..5af01f6 100644
--- a/docs/writing_plugins.rst
+++ b/docs/writing_plugins.rst
@@ -18,9 +18,10 @@ The quickest way to start writing a plugin is to create a my_plugin.py file

 from datasette import hookimpl

+
@hookimpl
def prepare_connection(conn):
- conn.create_function('hello_world', 0, lambda: 'Hello world!')
+ conn.create_function("hello_world", 0, lambda: "Hello world!")

If you save this in plugins/my_plugin.py you can then start Datasette like this::

@@ -60,22 +61,18 @@ The example consists of two files: a setup.py file that defines the plugin:

 from setuptools import setup

• VERSION = '0.1'

• VERSION = "0.1"

  setup(
  - name='datasette-plugin-demos',
  - description='Examples of plugins for Datasette',
  - author='Simon Willison',
  - url='https://github.com/simonw/datasette-plugin-demos',
  - license='Apache License, Version 2.0',
  + name="datasette-plugin-demos",
  + description="Examples of plugins for Datasette",
  + author="Simon Willison",
  + url="https://github.com/simonw/datasette-plugin-demos",
  + license="Apache License, Version 2.0",
  version=VERSION,
  - py_modules=['datasette_plugin_demos'],
  - entry_points={
  - 'datasette': [
  - 'plugin_demos = datasette_plugin_demos'
  - ]
  - },
  - install_requires=['datasette']
  + py_modules=["datasette_plugin_demos"],
  + entry_points={"datasette": ["plugin_demos = datasette_plugin_demos"]},
  + install_requires=["datasette"],
  )

And a Python module file, datasette_plugin_demos.py, that implements the plugin:
@@ -88,12 +85,12 @@ And a Python module file, datasette_plugin_demos.py, that implements the plu

 @hookimpl
 def prepare_jinja2_environment(env):

• env.filters['uppercase'] = lambda u: u.upper()

• env.filters["uppercase"] = lambda u: u.upper()

  @hookimpl
  def prepare_connection(conn):
  - conn.create_function('random_integer', 2, random.randint)
  + conn.create_function("random_integer", 2, random.randint)

Having built a plugin in this way you can turn it into an installable package using the following command::
@@ -123,11 +120,13 @@ To bundle the static assets for a plugin in the package that you publish to PyPI

.. code-block:: python

• package_data={
• 'datasette_plugin_name': [
• 'static/plugin.js',
• ],
• },
• package_data = (
• {
• "datasette_plugin_name": [
• "static/plugin.js",
• ],
• },
• )

Where datasette_plugin_name is the name of the plugin package (note that it uses underscores, not hyphens) and static/plugin.js is the path within that package to the static file.

@@ -152,11 +151,13 @@ Templates should be bundled for distribution using the same package_data mec

.. code-block:: python

• package_data={
• 'datasette_plugin_name': [
• 'templates/my_template.html',
• ],
• },
• package_data = (
• {
• "datasette_plugin_name": [
• "templates/my_template.html",
• ],
• },
• )

You can also use wildcards here such as templates/*.html. See datasette-edit-schema <https://github.com/simonw/datasette-edit-schema>__ for an example of this pattern.
```

(datasette) datasette % blacken-docs docs/*.rst
docs/authentication.rst: Rewriting...
docs/internals.rst:169: code block parse error Cannot parse: 14:0: <line number missing in source>
docs/plugin_hooks.rst:251: code block parse error Cannot parse: 6:4:     ]
docs/plugin_hooks.rst:312: code block parse error Cannot parse: 38:0: <line number missing in source>
docs/spatialite.rst: Rewriting...
docs/testing_plugins.rst:135: code block parse error Cannot parse: 5:0: <line number missing in source>
docs/writing_plugins.rst: Rewriting...

This idea makes sense to me.

There's actually an existing option that takes a path, called --get - it returns the HTML or JSON for that oath directly to the console, eg datasette my.db --get /mydb/mytable.json

So... one option would be to allow combining that with -o to open that URL in the browser:

datasette my.db -o --get /mydb

So some options here are:

• datasette my.db --open-url /mydb
• datasette my.db --open-path /mydb
• datasette my.db --open --get /mydb

I quite like that last combination option, mainly to avoid adding even more command options.

I'll add researchers too.

Great idea!

I used the sticky footer mechanism in datasette.app: https://github.com/simonw/datasette.app/issues/3

Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#register-commands-cli

I need to push this out as an alpha so I can release a demo plugin that uses it.

Writing the test for this is proving difficult, because the cli module has already been imported when I attempt to register a new plugin - so it doesn't pick up on the additional command registrations.

Trying to work around that with importlib.reload(cli).

Partial prototype of datasette-verify:

from datasette import hookimpl
import click


@hookimpl
def register_commands(cli):
    from datasette.cli import sqlite_extensions
    @cli.command()
    @click.argument("files", type=click.Path(exists=True), nargs=-1)
    @sqlite_extensions
    def verify(files, sqlite_extensions):
        "Verify that files can be opened by Datasette"
        for file in files:
            print(file)

I had to move the from datasette.cli import sqlite_extensions inside the hook function to avoid a circular import.

I'll probably have to use this mechanism for the tests then: https://til.simonwillison.net/pytest/registering-plugins-in-tests

I don't think I can get this new hook to support the handy --plugins-dir= mechanism for loading plugins from Python files as opposed to registering them with setuptools - that mechanism is itself implemented inside of code called by datasette serve so I don't have a way of taking advantage of it from outside that command.

I'll go with this:

% datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette is an open source multi-tool for exploring and publishing data

  About Datasette: https://datasette.io/
  Full documentation: https://docs.datasette.io/

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Considering this piece of code: https://github.com/simonw/datasette/blob/a1a33bb5822214be1cebd98cd858b2058d91a4aa/datasette/cli.py#L122-L142

I think the hook itself gets called with a single argument, cli, which it can then use in the standard Click way to register extra stuff.

I can't pass it datasette (like I do with register_routes()) because the Datasette object itself is instantiated by the serve command, which will not have been called.

The first example plugin I'm going to build for this will be datasette verify file.db file2.db - it will take one or more paths to SQLite files and verify if they can be opened by Datasette or not.

I'm going to call the new hook register_commands - since it will allow ambitious plugins to register more than one command if they want to. That's also pleasingly similar to register_routes.

Here's the answer to that:

~ % datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect
  install    Install Python packages - e.g.
  package    Package specified SQLite files into a new datasette Docker...
  plugins    List currently available plugins
  publish    Publish specified SQLite database files to the internet...
  uninstall  Uninstall Python packages (e.g.

Since it's adding extra things that show up in --help under the "Commands:" heading, I should call them commands.

Terminology question: is it correct to call these subcommands or should they be commands? publish_subcommand() adds subcommands of the format datasette publish X - but are we instead adding commands with this new one?

There's also the option for plugins to muck around with existing registered commands - this could get a bit untidy if multiple plugins try to do it, but being able to replace serve with a fresh implementation that adds an additional command-line option before calling back to the original might open up some interesting possibilities.

The closest plugin hook to this right now is publish_subcommand - which looks like this:

@hookimpl
def publish_subcommand(publish):
    @publish.command()
    @add_common_publish_arguments_and_options
    @click.option(
        "-k",
        "--api_key",
        help="API key for talking to my hosting provider",
    )
    def my_hosting_provider(...):

But there are also several plugin hooks with register_ prefixes, which may be a good naming convention to stick to here: register_output_renderer(datasette), register_routes(datasette), register_facet_classes(), register_magic_parameters(datasette).

I tried this ampersand fix: https://regex101.com/r/ojU2H9/1

# https://regex101.com/r/ojU2H9/1
_invalid_ampersand_re = re.compile(r'&(?![a-z0-9]+;)')

def fix_bad_xml(xml):
    # More fixes for things like '&' not as part of an entity
    return _invalid_ampersand_re.sub('&', xml)

Even with that I'm still getting total garbage in the <en-note> content - it's just HTML, not even trying to be XML.

It looks like I was using the round-trip to dump the <?xml version="1.0" encoding="UTF-8" standalone="no"?> and <!DOCTYPE prefixes.

10-20 minutes to populate _internal! How many databases and tables is that for?

I may have to rethink the _internal mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed.

I need to get my own stress testing rig setup for this.

As far as I can tell the Python sqlite3 module doesn't actually have a mechanism for de-registering a custom SQL function.

This means that if I implement a mechanism whereby each call to .convert() registers a new SQL function with a random suffix (convert_value_23424() for example) those functions will stay registered - and if .convert() is called a large number of times the number of obsolete custom function registrations will grow without bounds.

For that reason, I'm going to wontfix this issue.

https://github.com/dogsheep/evernote-to-sqlite/blob/36a466f142e5bad52719851c2fbda0c05cd35b99/evernote_to_sqlite/utils.py#L34-L42

Not sure why I was round-tripping the content_xml like that - I will try not doing that.

The debugger showed me that it broke on a string that looked like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd">
<en-note>
  <h1 title="Q3 2018 Reflection & Development">
    <span title=Q3 2018 Reflection & Development">
      Q3 2018 Reflection & Development
    </span>
  </h1>
  ...

Yeah that is not valid XML!

Good catch, thanks!

Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-data-from-files

I'm going to skip this test on windows.

https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819 says:

Currently, TextIOWrapper uses locale.getpreferredencoding(False) (hereinafter called “locale encoding”) when encoding is not specified.
...
Package authors using macOS or Linux may forget that the default encoding
is not always UTF-8.

Hah, the error here is actually:

>           assert result.exit_code == 1, result.output
E           AssertionError: 
E             
E           assert 0 == 1
E            +  where 0 = <Result okay>.exit_code

So I was expecting an error, but instead the command worked.

I suspect this is because on Windows the default character set may not be UTF-8?

Huh, tests are failing but only on Windows!

Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.

I thought about supporting those different policies (with something like --errors ignore) but I feel like that's getting a little bit too deep into the weeds. Right now if you try to import an invalid file the behaviour is the same as for the sqlite-utils insert command (I added the same detailed error message):

Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text

'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte

The input you provided uses a character encoding other than utf-8.

You can fix this by passing the --encoding= option with the encoding of the file.

If you do not know the encoding, running 'file filename.csv' may tell you.

It's often worth trying: --encoding=latin-1

If someone has data that can't be translated to valid text using a known encoding, I'm happy leaving them to have to insert it into a BLOB column instead.

I had a few doubts about the design just now. Since content_text is supported as a special argument, an alternative way of handling the above would be:

sqlite-utils insert-files /tmp/text.db files *.txt -c path -c content_text -c size

This does exactly the same thing as just using --text and not specifying any columns, because the actual implementation of --text is as follows:

https://github.com/simonw/sqlite-utils/blob/0c796cd945b146b7395ff5f553861400be503867/sqlite_utils/cli.py#L1851-L1855

But actually I think that's OK - --text is a useful shorthand that avoids you having to remember how to manually specify those columns with -c. So I'm going to leave the design as is.

Here's the error message I have working for invalid unicode:

sqlite-utils insert-files /tmp/text.db files *.txt --text
  [------------------------------------]    0%
Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text

'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte

I'm going to assume utf-8 but allow --encoding to be used to specify something different, since that option is already supported by other commands.

Will need to re-title this section of the documentation: https://sqlite-utils.datasette.io/en/3.16/cli.html#inserting-binary-data-from-files - "Inserting binary data from files" will become "Inserting data from files"

I'm OK with keeping the default as BLOB but I could add a --text option which stores the content as text instead.

If the text can't be stored as utf-8 I'll probably raise an error.

This is a good idea.

Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll.