Similar trouble with ageinfo using 0.22. Here's what my ageinfo.js file looks like:

window.YTD.ageinfo.part0 = [
  {
    "ageMeta" : { }
  }
]

Commenting out the registration for ageinfo in archive.py gets my archive to import.

I could even have those replacement characters be properties of the Database class, so uses can sub-class and change them.

class Database:
    left_brace_replace = "_"
    right_brace_replace = "_"
    ...

I think I like the underscore option best. I don't like the idea of injecting surprise ( ) parenthesis, and having them vanish entirely could result in things like item[price] becoming itemprice which feels confusing.

item_price_ is a little ugly but I think I can live with it.

What are my options for replacing those characters?

• [ becomes ( and ] becomes )
• [ and ] are removed entirely
• [ and ] both become _

I'm inclined to just fix them and not have an option for opting-out of fixing them, since it adds quite a bit of cruft to the overall API design for an option that maybe no-one will ever use.

If I add a new parameter for opting in and out of fixing these, what should it be called? A few options:

• fix_columns=False
• rename_bad_columns=False
• fix_column_names=False
• error_on_invalid_column_names=True

Here's the code where this happens: https://github.com/simonw/sqlite-utils/blob/54191d4dc114d7dc21e849b48a4d5ae4f9e601ca/sqlite_utils/db.py#L2943-L2948

It's called from three different methods in db.py: create_table_sql(), update() and insert_all().

So either I do the automatic replacement, or I let the user request automatic replacement, or a third option: I do automatic replacement but let the user opt to receive an error instead.

Quoting https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856 in full:

I'm not sure what to do about this one.

I can't fix it: this bug in Python's sqlite3 module means that even if I write a database out with column names that include [] I won't be able to read them back again.

So... I could do one of the following:

• Throw an error if a column name includes those characters. That's my preferred option I think.
• Automatically replace [ in column names with ( and ] with )
• Do the automatic replacement but show a user-visible warning when I do it
• Throw an error, but give the user an option to run with e.g. --fix-column-names which applies that automatic fix.

Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.

Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-directly-against-csv-or-json

If two files have the same name they will be assigned a numeric suffix:

$ sqlite-utils memory foo/data.csv bar/data.csv "select * from data_2"

Here's the relevant code: https://github.com/simonw/sqlite-utils/blob/7427a9137f60de961b6331d0922a3f03da0d1890/sqlite_utils/cli.py#L1289-L1292

I can fix this by checking to see if csv_table is already in use and adding a suffix.

Oddly I can't replicate this on macOS:

(sqlite-utils) sqlite-utils % ls foo/*.csv
foo/bug.csv
(sqlite-utils) sqlite-utils % ls bar/*.csv
bar/bug.csv
(sqlite-utils) sqlite-utils % sqlite-utils memory foo/bug.csv bar/bug.csv --schema
CREATE TABLE "bug" (
   [col1] TEXT,
   [col2] TEXT
);
CREATE VIEW t1 AS select * from [bug];
CREATE VIEW t AS select * from [bug];

Wow, that was fast! Thank you!

The t1 and t2 aliases were meant to handle this case, but the are no good if the tool throws an error.

The bug is in this code: https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/sqlite_utils/cli.py#L2205-L2227

Good catch, thanks.

Side-note: Django 4.0 will switch from using pytz to using the standard library zoneinfo module introduced in Python 3.9, which has a backport that works as far back as 3.6: https://github.com/pganssle/zoneinfo (https://pypi.org/project/backports.zoneinfo/)

If I need to handle timezones I'll use that, but I think I can get away without it?

Django does this: https://github.com/django/django/blob/b0ed619303d2fb723330ca9efa3acf23d49f1d19/setup.cfg#L39-L43

install_requires =
    asgiref >= 3.3.2
    backports.zoneinfo; python_version<"3.9"
    sqlparse >= 0.2.2
    tzdata; sys_platform == 'win32'

The audiences I care about here are:

• Producers of this timestamp - generally that will be users who are using datasette publish to share their data
• Human consumers of this timestamp - end users who look at a Datasette site and want to know how recent the data is
• Machine consumers of this timestamp - API integrations that might want to check if a Datasette instance has been updated before downloading new data

For producers I think there are going to be two categories. The first is users who run "publish" and want the site to reflect when they did so (probably using --updated=now when they publish). The second are users who are willing to spend more time thinking about this - for example my various git scraping projects where I want to use a date derived from the git history.

For humans... I'd like to be able to calculate a relative time (3 hours ago) in addition to showing the display time, because that helps avoid confusion over timezones.

For machine consumers, it might be nice to offer the option of a calculated Unix timestamp-since-1970, since those can be easier to work with in some languages than running a full ISO date parser.

The simplest possible version of this: it's always represented as a UTC ISO datetime, like this:

"updated": "2020-10-31T12:00:00+00:00"

Later versions of Datasette could extend this to handle other timezones or support just the date (though that's a backwards incompatible change so probably better to decide on the date thing right now).

So this is a metadata key called updated which can be applied at the table, database or instance level. It is represented as a .isoformat() timestamp.

Question: should I support just the date - yyyy-mm-dd - in addition to the datetime?

I think so. I can easily imagine situations where the exact time of day that a change was made hasn't been recorded, but the overall date is known.

But in that case, should the updated key sometimes be yyyy-mm-dd and sometimes be the full isoformat datetime? Or should there be an updated_date key that's used for just the date?

I'm going to use https://github.com/dateutil/dateutil for this - it's been maintained constantly (by an evolving team of contributors) since 2003 and is a very trustworthy dependency.

Thanks for this!

Codecov Report

Merging #1476 (2eb01ff) into main (b28b6cd) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1476   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b28b6cd...2eb01ff. Read the comment docs.

Still a good idea today too! Would be great for https://cdc-vaccination-history.datasette.io/ for example.

I replaced:

<a class="text-3xl font-bold leading-none" href="https://www.saferortoxic.org">
<img 
src="https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp" 
class="custom-logo" 
alt="Logo: Safer or Toxic?" 
width="300px">
</a>

with:

<img src="https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp" 
alt="Logo: Safer or Toxic?" 
style="width:300px; cursor:pointer;" 
onclick="window.location.href='https://saferortoxic.org';return false;">

I'd still love to know what caused this (and how to troubleshoot to figure it out), so I'll leave it open for a bit, but I do have a functional logo linking to the Hugo home page, at least locally. I'll likely push tomorrow.

(Before trying this, I tried to apply a background image to the a tag. That didn't work.)

An update, if I remove the img tag and replace it with the text, "Safer or Toxic?" it links to the right place.

Also, if I keep things exactly as they are, and it improperly, but consistently goes to the undefined page, on THAT 404 page, a click on the image properly clicks through to the www.SaferOrToxic.org page.

Weird stuff.

That's so useful @sethvincent! Really interesting reading your code there, especially clever how you're using the base_url config.

I'd be very interested to see what your demo looks like without using serverless - completely agree that the less additional dependencies there are for this the better.

I'm also very interested in figuring out a way to run Datasette in Lambda but with the SQLite database on an EFS volume. Do you have a feel for how hard that would be?

:wave: I just put together a small example using the lambda container image support: https://github.com/sethvincent/datasette-aws-lambda-example

It uses mangum and AWS's python runtime interface client to handle the lambda event stuff.

I'd be happy to help with a publish plugin for AWS lambda as I plan to use this for upcoming projects.

The example uses the serverless cli for deployment but there might be a more suitable deployment approach for the plugin. It would be cool if users didn't have to install anything additional other than the aws cli and its associated config/credentials setup.

Codecov Report

Merging #1471 (01b05ed) into main (b28b6cd) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1471   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b28b6cd...01b05ed. Read the comment docs.

Superseded by #1471.

haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?

Not certain, but I think tests in CI run on Ubuntu and don't appear to install any additional Sqlite-related dependencies, and so my guess is the version of Sqlite installed by default on Ubuntu has the SQLITE_ENABLE_FTS3_PARENTHESIS option enabled and so doesn't run into this issue.

Codecov Report

Merging #1385 (db78094) into main (ea627ba) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head db78094 differs from pull request most recent head 8d78c8c. Consider uploading reports for the commit 8d78c8c to get more accurate results

@@           Coverage Diff           @@
##             main    #1385   +/-   ##
=======================================
  Coverage   91.70%   91.70%           
=======================================
  Files          34       34           
  Lines        4364     4364           
=======================================
  Hits         4002     4002           
  Misses        362      362           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea627ba...8d78c8c. Read the comment docs.

Especially relevant now that 0.2.0 is out which is a much higher quality release. https://github.com/simonw/datasette-app/releases/tag/0.2.0

Here's a already open issue for this: #972

At the moment this isn't possible - though there's a workaround which is to define a SQL view for the query, at which point facets will be displayed again.

I did a lot of the work required to support this when I refactored how facets worked a while back - but to finally implement this I need to refactor the table view and the arbitrary query view to share much more logic than they do at the moment.

Here's the code at fault: https://github.com/simonw/datasette/blob/b28b6cd2fe97f7e193a235877abeec2c8eb0a821/datasette/static/table.js#L137-L146

haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?

dupe of #293?

Thanks, that does look similar!

Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.

Ah that makes sense, I guess that's why this isn't just always installed already. I wonder if a possible solution to this issue could be doing feature detection on whether this feature is supported by the current Sqlite version, and if not these tests could be disabled locally? But possibly there's a better way to handle this, will see what @simonw thinks

Codecov Report

Merging #326 (078a087) into main (77c240d) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #326   +/-   ##
=======================================
  Coverage   96.58%   96.58%           
=======================================
  Files           5        5           
  Lines        2223     2223           
=======================================
  Hits         2147     2147           
  Misses         76       76           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77c240d...078a087. Read the comment docs.

thanks for the response! full errors attached; excerpt:

...


    def test_searchmode(table_metadata, querystring, expected_rows):
        with make_app_client(
            metadata={"databases": {"fixtures": {"tables": {"searchable": table_metadata}}}}
        ) as client:
            response = client.get("/fixtures/searchable.json?" + querystring)
>           assert expected_rows == response.json["rows"]
E           AssertionError: assert [[1, 'barry c...sel', 'puma']] == []
E             Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']
E             Use -v to get the full diff

/Users/t/dev/datasette/tests/test_api.py:1115: AssertionError

errors.txt

A quick scan of #1223 suggests you're right. Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.

What are the full errors you're getting?

This may be the same issue as described in https://github.com/simonw/datasette/pull/1223 - essentially the test suite (and corresponding Datasette features I assume) are by default implicitly dependent on your Sqlite installation having been compiled with the SQLITE_ENABLE_FTS3_PARENTHESIS option. If this is the same issue then I think this can be fixed either by recompiling with that option or (probably more easily) by running pip install pysqlite3-binary, which will be used in preference to your system Sqlite installation and has this option enabled.

on commit d57ab156b35ec642

Codecov Report

Merging #1463 (2dd94c5) into main (d57ab15) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1463   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d57ab15...2dd94c5. Read the comment docs.

I think this is a setting. There are two relevant settings at the moment:

    "template_debug": false,
    "trace_debug": false,

For consistence then this should be called something_debug - but do I want a single setting that exposes the _internal database and adds those debug options to the menu, or do I want those as two separate settings?

• internal_debug to enable access to that _internal database
• menu_debug for those menu options?

I don't think I'll adopt it for this project. For example, here:

     response = Response.redirect("/")
-    response.set_cookie("ds_actor", datasette.sign({
-        "a": {
-            "id": "cleopaws"
-        }
-    }, "actor"))
+    response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor"))

I chose to use the multi-line version to help emphasize the structure - the single-line replacement loses that. I think I'll continue to make my own editorial choices about how the code examples are laid out.

Here's the diff it produced from that first run:
`diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..8008023 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,7 @@ Authentication plugins can set signedds_actor`` cookies themselves like so:
.. code-block:: python

 response = Response.redirect("/")

• response.set_cookie("ds_actor", datasette.sign({
• "a": {
• "id": "cleopaws"
• }
• }, "actor"))
• response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor"))

Note that you need to pass "actor" as the namespace to :ref:datasette_sign.

@@ -412,12 +408,16 @@ To include an expiry, add a "e" key to the cookie value containing a `base62
expires_at = int(time.time()) + (24 * 60 * 60)

 response = Response.redirect("/")

• response.set_cookie("ds_actor", datasette.sign({
• "a": {
• "id": "cleopaws"
• },
• "e": baseconv.base62.encode(expires_at),
• }, "actor"))
• response.set_cookie(
• "ds_actor",
• datasette.sign(
• {
• "a": {"id": "cleopaws"},
• "e": baseconv.base62.encode(expires_at),
• },
• "actor",
• ),
• )

The resulting cookie will encode data that looks something like this:

diff --git a/docs/spatialite.rst b/docs/spatialite.rst
index d1b300b..556bad8 100644
--- a/docs/spatialite.rst
+++ b/docs/spatialite.rst
@@ -58,19 +58,22 @@ Here's a recipe for taking a table with existing latitude and longitude columns,
.. code-block:: python

 import sqlite3

• conn = sqlite3.connect('museums.db')
  +
• conn = sqlite3.connect("museums.db")
  # Lead the spatialite extension:
  conn.enable_load_extension(True)
• conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
• conn.load_extension("/usr/local/lib/mod_spatialite.dylib")
  # Initialize spatial metadata for this database:
• conn.execute('select InitSpatialMetadata(1)')
• conn.execute("select InitSpatialMetadata(1)")
  # Add a geometry column called point_geom to our museums table:
  conn.execute("SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);")
  # Now update that geometry column with the lat/lon points
• conn.execute('''
• conn.execute(
• """
  UPDATE museums SET
  point_geom = GeomFromText('POINT('||"longitude"||' '||"latitude"||')',4326);
• ''')
• """

• )
  # Now add a spatial index to that column
  conn.execute('select CreateSpatialIndex("museums", "point_geom");')
  # If you don't commit your changes will not be persisted:
  @@ -186,13 +189,14 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
  .. code-block:: python

  import sqlite3
  - conn = sqlite3.connect('places.db')
  +
  + conn = sqlite3.connect("places.db")
  # Enable SpatialLite extension
  conn.enable_load_extension(True)
  - conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
  + conn.load_extension("/usr/local/lib/mod_spatialite.dylib")
  # Create the masic countries table
  - conn.execute('select InitSpatialMetadata(1)')
  - conn.execute('create table places (id integer primary key, name text);')
  + conn.execute("select InitSpatialMetadata(1)")
  + conn.execute("create table places (id integer primary key, name text);")
  # Add a MULTIPOLYGON Geometry column
  conn.execute("SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);")
  # Add a spatial index against the new column
  @@ -201,13 +205,17 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
  from shapely.geometry.multipolygon import MultiPolygon
  from shapely.geometry import shape
  import requests
  - geojson = requests.get('https://data.whosonfirst.org/404/227/475/404227475.geojson').json()
  +
  + geojson = requests.get(
  + "https://data.whosonfirst.org/404/227/475/404227475.geojson"
  + ).json()
  # Convert to "Well Known Text" format
  - wkt = shape(geojson['geometry']).wkt
  + wkt = shape(geojson["geometry"]).wkt
  # Insert and commit the record
  - conn.execute("INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))", (
  - "Wales", wkt
  - ))
  + conn.execute(
  + "INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))",
  + ("Wales", wkt),
  + )
  conn.commit()

Querying polygons using within()
diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst
index bd60a4b..5af01f6 100644
--- a/docs/writing_plugins.rst
+++ b/docs/writing_plugins.rst
@@ -18,9 +18,10 @@ The quickest way to start writing a plugin is to create a my_plugin.py file

 from datasette import hookimpl

+
@hookimpl
def prepare_connection(conn):
- conn.create_function('hello_world', 0, lambda: 'Hello world!')
+ conn.create_function("hello_world", 0, lambda: "Hello world!")

If you save this in plugins/my_plugin.py you can then start Datasette like this::

@@ -60,22 +61,18 @@ The example consists of two files: a setup.py file that defines the plugin:

 from setuptools import setup

• VERSION = '0.1'

• VERSION = "0.1"

  setup(
  - name='datasette-plugin-demos',
  - description='Examples of plugins for Datasette',
  - author='Simon Willison',
  - url='https://github.com/simonw/datasette-plugin-demos',
  - license='Apache License, Version 2.0',
  + name="datasette-plugin-demos",
  + description="Examples of plugins for Datasette",
  + author="Simon Willison",
  + url="https://github.com/simonw/datasette-plugin-demos",
  + license="Apache License, Version 2.0",
  version=VERSION,
  - py_modules=['datasette_plugin_demos'],
  - entry_points={
  - 'datasette': [
  - 'plugin_demos = datasette_plugin_demos'
  - ]
  - },
  - install_requires=['datasette']
  + py_modules=["datasette_plugin_demos"],
  + entry_points={"datasette": ["plugin_demos = datasette_plugin_demos"]},
  + install_requires=["datasette"],
  )

And a Python module file, datasette_plugin_demos.py, that implements the plugin:
@@ -88,12 +85,12 @@ And a Python module file, datasette_plugin_demos.py, that implements the plu

 @hookimpl
 def prepare_jinja2_environment(env):

• env.filters['uppercase'] = lambda u: u.upper()

• env.filters["uppercase"] = lambda u: u.upper()

  @hookimpl
  def prepare_connection(conn):
  - conn.create_function('random_integer', 2, random.randint)
  + conn.create_function("random_integer", 2, random.randint)

Having built a plugin in this way you can turn it into an installable package using the following command::
@@ -123,11 +120,13 @@ To bundle the static assets for a plugin in the package that you publish to PyPI

.. code-block:: python

• package_data={
• 'datasette_plugin_name': [
• 'static/plugin.js',
• ],
• },
• package_data = (
• {
• "datasette_plugin_name": [
• "static/plugin.js",
• ],
• },
• )

Where datasette_plugin_name is the name of the plugin package (note that it uses underscores, not hyphens) and static/plugin.js is the path within that package to the static file.

@@ -152,11 +151,13 @@ Templates should be bundled for distribution using the same package_data mec

.. code-block:: python

• package_data={
• 'datasette_plugin_name': [
• 'templates/my_template.html',
• ],
• },
• package_data = (
• {
• "datasette_plugin_name": [
• "templates/my_template.html",
• ],
• },
• )

You can also use wildcards here such as templates/*.html. See datasette-edit-schema <https://github.com/simonw/datasette-edit-schema>__ for an example of this pattern.
```

(datasette) datasette % blacken-docs docs/*.rst
docs/authentication.rst: Rewriting...
docs/internals.rst:169: code block parse error Cannot parse: 14:0: <line number missing in source>
docs/plugin_hooks.rst:251: code block parse error Cannot parse: 6:4:     ]
docs/plugin_hooks.rst:312: code block parse error Cannot parse: 38:0: <line number missing in source>
docs/spatialite.rst: Rewriting...
docs/testing_plugins.rst:135: code block parse error Cannot parse: 5:0: <line number missing in source>
docs/writing_plugins.rst: Rewriting...

This idea makes sense to me.

There's actually an existing option that takes a path, called --get - it returns the HTML or JSON for that oath directly to the console, eg datasette my.db --get /mydb/mytable.json

So... one option would be to allow combining that with -o to open that URL in the browser:

datasette my.db -o --get /mydb

So some options here are:

• datasette my.db --open-url /mydb
• datasette my.db --open-path /mydb
• datasette my.db --open --get /mydb

I quite like that last combination option, mainly to avoid adding even more command options.

I'll add researchers too.

Great idea!

I love it! maybe 'researchers' instead? Or 'scientists and researchers'?

Additionally, assuming the line numbers match up with the provided enenx file, the mentioned line plus one before and after is as follows:

<![CDATA[>]]>
</span></div>
<div style="padding: 0px; font-family: Arial, sans-serif; font-size: 12px; line-height: 16px; white-space: pre-wrap;"><br style=" padding: 0px;"/>

Found the cause, it's the other commands. PR #59 submitted.

I used the sticky footer mechanism in datasette.app: https://github.com/simonw/datasette.app/issues/3

Codecov Report

Merging #1453 (4f492a7) into main (67cbf0a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1453   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 67cbf0a...4f492a7. Read the comment docs.

Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#register-commands-cli

I need to push this out as an alpha so I can release a demo plugin that uses it.

Writing the test for this is proving difficult, because the cli module has already been imported when I attempt to register a new plugin - so it doesn't pick up on the additional command registrations.

Trying to work around that with importlib.reload(cli).

Partial prototype of datasette-verify:

from datasette import hookimpl
import click


@hookimpl
def register_commands(cli):
    from datasette.cli import sqlite_extensions
    @cli.command()
    @click.argument("files", type=click.Path(exists=True), nargs=-1)
    @sqlite_extensions
    def verify(files, sqlite_extensions):
        "Verify that files can be opened by Datasette"
        for file in files:
            print(file)

I had to move the from datasette.cli import sqlite_extensions inside the hook function to avoid a circular import.

I'll probably have to use this mechanism for the tests then: https://til.simonwillison.net/pytest/registering-plugins-in-tests

I don't think I can get this new hook to support the handy --plugins-dir= mechanism for loading plugins from Python files as opposed to registering them with setuptools - that mechanism is itself implemented inside of code called by datasette serve so I don't have a way of taking advantage of it from outside that command.

I'll go with this:

% datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette is an open source multi-tool for exploring and publishing data

  About Datasette: https://datasette.io/
  Full documentation: https://docs.datasette.io/

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Considering this piece of code: https://github.com/simonw/datasette/blob/a1a33bb5822214be1cebd98cd858b2058d91a4aa/datasette/cli.py#L122-L142

I think the hook itself gets called with a single argument, cli, which it can then use in the standard Click way to register extra stuff.

I can't pass it datasette (like I do with register_routes()) because the Datasette object itself is instantiated by the serve command, which will not have been called.

The first example plugin I'm going to build for this will be datasette verify file.db file2.db - it will take one or more paths to SQLite files and verify if they can be opened by Datasette or not.

I'm going to call the new hook register_commands - since it will allow ambitious plugins to register more than one command if they want to. That's also pleasingly similar to register_routes.

Here's the answer to that:

~ % datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect
  install    Install Python packages - e.g.
  package    Package specified SQLite files into a new datasette Docker...
  plugins    List currently available plugins
  publish    Publish specified SQLite database files to the internet...
  uninstall  Uninstall Python packages (e.g.

Since it's adding extra things that show up in --help under the "Commands:" heading, I should call them commands.

Terminology question: is it correct to call these subcommands or should they be commands? publish_subcommand() adds subcommands of the format datasette publish X - but are we instead adding commands with this new one?

There's also the option for plugins to muck around with existing registered commands - this could get a bit untidy if multiple plugins try to do it, but being able to replace serve with a fresh implementation that adds an additional command-line option before calling back to the original might open up some interesting possibilities.

The closest plugin hook to this right now is publish_subcommand - which looks like this:

@hookimpl
def publish_subcommand(publish):
    @publish.command()
    @add_common_publish_arguments_and_options
    @click.option(
        "-k",
        "--api_key",
        help="API key for talking to my hosting provider",
    )
    def my_hosting_provider(...):

But there are also several plugin hooks with register_ prefixes, which may be a good naming convention to stick to here: register_output_renderer(datasette), register_routes(datasette), register_facet_classes(), register_magic_parameters(datasette).

I tried this ampersand fix: https://regex101.com/r/ojU2H9/1

# https://regex101.com/r/ojU2H9/1
_invalid_ampersand_re = re.compile(r'&(?![a-z0-9]+;)')

def fix_bad_xml(xml):
    # More fixes for things like '&' not as part of an entity
    return _invalid_ampersand_re.sub('&', xml)

Even with that I'm still getting total garbage in the <en-note> content - it's just HTML, not even trying to be XML.

It looks like I was using the round-trip to dump the <?xml version="1.0" encoding="UTF-8" standalone="no"?> and <!DOCTYPE prefixes.

Codecov Report

Merging #1448 (a211747) into main (a1a33bb) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1448   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4418     4418           
=======================================
  Hits         4057     4057           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a1a33bb...a211747. Read the comment docs.

Perceptual hashes might be what you're after : http://phash.org

I did two tests: one with 1000 5-30mb DBs and a second with 20 multi gig DBs. For the second, I created them like so:
for i in {1..20}; do sqlite-generate db$i.db --tables ${i}00 --rows 100,2000 --columns 5,100 --pks 0 --fks 0; done

This was for deciding whether to use lots of small DBs or to group things into a smaller number of bigger DBs. The second strategy wins.

By simply persisting the _internal DB to disk, I was able to avoid most of the performance issues I was experiencing previously. (To do this, I changed the datasette/internal_db.py:init_internal_db creates to if not exists, and changed the _internal DB instantiation in datasette/app.py:Datasette.__init__ to a path with is_mutable=True.) Super rough, but the pages now load so I can continue testing ideas.

10-20 minutes to populate _internal! How many databases and tables is that for?

I may have to rethink the _internal mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed.

I need to get my own stress testing rig setup for this.

Upon first stab, there's two issues here:
- DB/table/row counts (as discussed above). This isn't too bad if the DBs are actually above the MAX limit check.
- Populating the internal DB. On first load of a giant set of DBs, it can take 10-20 mins to populate. By altering datasette and persisting the internal DB to disk, this problem is vastly improved, but I'm sure this will cause problems elsewhere.

As far as I can tell the Python sqlite3 module doesn't actually have a mechanism for de-registering a custom SQL function.

This means that if I implement a mechanism whereby each call to .convert() registers a new SQL function with a random suffix (convert_value_23424() for example) those functions will stay registered - and if .convert() is called a large number of times the number of obsolete custom function registrations will grow without bounds.

For that reason, I'm going to wontfix this issue.

https://github.com/dogsheep/evernote-to-sqlite/blob/36a466f142e5bad52719851c2fbda0c05cd35b99/evernote_to_sqlite/utils.py#L34-L42

Not sure why I was round-tripping the content_xml like that - I will try not doing that.

The debugger showed me that it broke on a string that looked like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd">
<en-note>
  <h1 title="Q3 2018 Reflection & Development">
    <span title=Q3 2018 Reflection & Development">
      Q3 2018 Reflection & Development
    </span>
  </h1>
  ...

Yeah that is not valid XML!

Good catch, thanks!

Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-data-from-files

Codecov Report

Merging #321 (db2dd6d) into main (9258f4b) will increase coverage by 0.16%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #321      +/-   ##
==========================================
+ Coverage   96.41%   96.58%   +0.16%     
==========================================
  Files           5        5              
  Lines        2206     2223      +17     
==========================================
+ Hits         2127     2147      +20     
+ Misses         79       76       -3     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9258f4b...db2dd6d. Read the comment docs.

I'm going to skip this test on windows.

https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819 says:

Currently, TextIOWrapper uses locale.getpreferredencoding(False) (hereinafter called “locale encoding”) when encoding is not specified.
...
Package authors using macOS or Linux may forget that the default encoding
is not always UTF-8.

Hah, the error here is actually:

>           assert result.exit_code == 1, result.output
E           AssertionError: 
E             
E           assert 0 == 1
E            +  where 0 = <Result okay>.exit_code

So I was expecting an error, but instead the command worked.

I suspect this is because on Windows the default character set may not be UTF-8?

Huh, tests are failing but only on Windows!

I'm happy with this functionality left the way you describe. In my case the data is homogeneous but other cases would work just by being consistent on the encoding. Thanks a lot, Simon!

Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.

I thought about supporting those different policies (with something like --errors ignore) but I feel like that's getting a little bit too deep into the weeds. Right now if you try to import an invalid file the behaviour is the same as for the sqlite-utils insert command (I added the same detailed error message):

Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text

'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte

The input you provided uses a character encoding other than utf-8.

You can fix this by passing the --encoding= option with the encoding of the file.

If you do not know the encoding, running 'file filename.csv' may tell you.

It's often worth trying: --encoding=latin-1

If someone has data that can't be translated to valid text using a known encoding, I'm happy leaving them to have to insert it into a BLOB column instead.

I had a few doubts about the design just now. Since content_text is supported as a special argument, an alternative way of handling the above would be:

sqlite-utils insert-files /tmp/text.db files *.txt -c path -c content_text -c size

This does exactly the same thing as just using --text and not specifying any columns, because the actual implementation of --text is as follows:

https://github.com/simonw/sqlite-utils/blob/0c796cd945b146b7395ff5f553861400be503867/sqlite_utils/cli.py#L1851-L1855

But actually I think that's OK - --text is a useful shorthand that avoids you having to remember how to manually specify those columns with -c. So I'm going to leave the design as is.

Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.

From https://stackoverflow.com/questions/24616678/unicodedecodeerror-in-python-when-reading-a-file-how-to-ignore-the-error-and-ju :
- 'ignore' ignores errors. Note that ignoring encoding errors can lead to data loss.
- 'replace' causes a replacement marker (such as '?') to be inserted where there is malformed data.
- 'surrogateescape' will represent any incorrect bytes as code points in the Unicode Private Use Area ranging from U+DC80 to U+DCFF. These private code points will then be turned back into the same bytes when the surrogateescape error handler is used when writing data. This is useful for processing files in an unknown encoding.
- 'xmlcharrefreplace' is only supported when writing to a file. Characters not supported by the encoding are replaced with the appropriate XML character reference &#nnn;.
- 'backslashreplace' (also only supported when writing) replaces unsupported characters with Python’s backslashed escape sequences.

Here's the error message I have working for invalid unicode:

sqlite-utils insert-files /tmp/text.db files *.txt --text
  [------------------------------------]    0%
Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text

'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte

I'm going to assume utf-8 but allow --encoding to be used to specify something different, since that option is already supported by other commands.

I was thinking that an approach could be making FILE_COLUMNS a generator (_get_file_columns(mode)) or you can just have a different set of columns (is there something else that makes sense to be changed on the text scenario?).

About UTF-8 I was referring to the encoding to use when reading files. This can be difficult to auto-detect but I believe that UTF-8 is pretty much the standard for text files.

Will need to re-title this section of the documentation: https://sqlite-utils.datasette.io/en/3.16/cli.html#inserting-binary-data-from-files - "Inserting binary data from files" will become "Inserting data from files"

I'm OK with keeping the default as BLOB but I could add a --text option which stores the content as text instead.

If the text can't be stored as utf-8 I'll probably raise an error.

This is a good idea.

I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work.

Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll.

OK, now the following optional CSS gives us a sticky footer:

html, body {
  height: 100%;
}
body {
  display: flex;
  flex-direction: column;
}
.not-footer {
  flex: 1 0 auto;
}
footer {
  flex-shrink: 0;
}

This would fix #21 and make #22 obsolete.