Posting this while I'm thinking about it: I mentioned at the end of this thread that I'm usually doing datasette --get to export canned queries.

I used to use a tool called datafreeze to do scripted exports, but that project looks dead now. The ergonomics of it are pretty nice, though, and the Freezefile.yml structure is actually not too far from Datasette's canned queries.

This is related to the idea for datasette query (#1356) but I think it's a distinct feature. It's most likely a plugin, but I want to raise it here because it's probably something other people have thought about.

Original title: Can I keep "pip install datasette" working on Python 3.6?

I dropped support for 3.6 in:
- #1577

I'm getting reports that pip3 install datasette throws an error on that Python, even though I haven't made that new release yet - presumably due to lack of pinning of Uvicorn: https://twitter.com/ldodds/status/1484289475195080706

Is it possible to get pip on that version of Python to install the highest possible version of the packages that are still known to support Python 3.6?

If so, how?

It's not currently clear what the difference between https://docs.datasette.io/en/latest/ and https://docs.datasette.io/en/stable/ is - I should fix that.

On Twitter: https://twitter.com/simonw/status/1484285006243528705

They're generated by Spatialite and not very interesting in most cases.

https://user-images.githubusercontent.com/25778/149786443-7832363c-2a2c-4533-9edc-048cd00e4a08.png">

The version bundled in Datasette's Docker image right now is 4.4.0-RC0

https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/Dockerfile#L16-L17

5 has been out for a couple of months and has a bunch of big improvements, most notable stable KNN support.

See detailed notes in https://github.com/simonw/datasette/issues/1249 - for some reason SpatiaLite 5.0 hangs the /dbname page on Linux (inside Docker containers, both with a custom compiled SpatiaLite and one installed from the Ubuntu 20.10 package repository). This doesn't happen on macOS with SpatiaLite 5 installed using Homebrew.

https://github.com/simonw/datasette/blob/5fd02890650db790b2ffdb90eb9f78f8e0639c37/docs/spatialite.rst#L96

I was trying to initalize SpatiaLite in a write connection:

>>> from datasette.app import Datasette
>>> ds = Datasette(memory=True, files=[], sqlite_extensions=["spatialite"])
>>> db = ds.add_memory_database('geo')
>>> await db.execute_write("select InitSpatialMetadata(1)")
UUID('3f143baa-4e3d-5842-a36f-4fa2f683b72f')
no such function: InitSpatialMetadata

It looks like the code that loads additional modules only works on read-only connections, not on write connections:

https://github.com/simonw/datasette/blob/92a5280d2e75c39424a75ad6226fc74400ae984f/datasette/database.py#L146-L153

Compared to:

https://github.com/simonw/datasette/blob/92a5280d2e75c39424a75ad6226fc74400ae984f/datasette/database.py#L124-L132

https://github.com/simonw/datasette/commit/6aa5886379dd9017215904fb28567b80018902f9 added the --load-extension=spatialite shortcut looking for the extension in these places:

https://github.com/simonw/datasette/blob/12877d7a48e2aa28bb5e780f929a218f7265d849/datasette/utils/__init__.py#L56-L60

However, in the datasetteproject/datasette docker image the file is at /usr/local/lib/mod_spatialite.so.

This results in the example command here failing:

% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=spatialite
Error: Could not find SpatiaLite extension

But it does work when given an explicit path:

% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=/usr/local/lib/mod_spatialite.so
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
...

Perhaps SPATIALITE_PATHS should include /usr/local/lib/mod_spatialite.so?

https://github.com/simonw/datasette/blob/242bc89fdf2e775e340d69a4e851b3a9accb31c6/datasette/cli.py#L533-L548

This could use the find_spatialite() function and, if it finds something, suggest the user use --load-extension=spatialite

https://github.com/simonw/datasette/blob/242bc89fdf2e775e340d69a4e851b3a9accb31c6/datasette/utils/__init__.py#L1015-L1019

Hi,
datasette version: "0.26.2"
extensions:
spatialite: "4.4.0-RC0"
sqlite version: "3.22.0"

I have a table of airports with latitude and longitude columns. I've added spatialite (with KNN support). After creating the db using csvs-to-sqlit, I run these commands to setup the spatialite tables:

conn.execute('SELECT InitSpatialMetadata(1)')

conn.execute("SELECT AddGeometryColumn('airports', 'point_geom', 4326, 'POINT', 2);")

conn.execute('''UPDATE airports SET point_geom = GeomFromText('POINT('||"longitude"||' '||"latitude"||')',4326);''')

conn.execute("SELECT CreateSpatialIndex('airports', 'point_geom');")

I'm attempting to create a canned query and have this in my metadata.json file:

"find_airports_nearest_to_point":{
    "sql":"SELECT a.pos AS rank, b.id, b.name, b.country, b.latitude AS latitude, b.longitude AS longitude, a.distance / 1000.0 AS dist_km FROM KNN AS a JOIN airports AS b ON (b.rowid = a.fid) WHERE f_table_name = \"airports\" AND ref_geometry = MakePoint( :Long , :Lat ) AND max_items = 10;"}

which doesn't seem to perform the templating of the name parameters correctly and I get no results.

Have also tired:

MakePoint( || :Long || , || :Lat || )

which returns this error:

near "||": syntax error

However I cannot seem to find the correct combination of named parameter syntax (:Lat) or sqlite concatenation operator to make it work. Any ideas if using named parameters inside functions is supported?

Thanks
Darren

We currently show a useful warning (from #331) when the user tries to open a spatialite database without first loading the module:

https://github.com/simonw/datasette/blob/c692cd291111050483a32bea1ee08e994a0b781b/datasette/app.py#L547-L554

This code is part of .inspect() which is going away - see #462 - so I need to find somewhere else for it to live.

$ datasette spatial.db 
Serve! files=('spatial.db',) (immutables=()) on port 8001
ERROR: conn=<sqlite3.Connection object at 0x11e388f10>, sql = 'PRAGMA table_info(SpatialIndex);', params = None: no such module: VirtualSpatialIndex
Usage: datasette serve [OPTIONS] [FILES]...

Error: It looks like you're trying to load a SpatiaLite database without first loading the SpatiaLite module.

Read more: https://datasette.readthedocs.io/en/latest/spatialite.html

This error message could sniff around in the common locations for the SpatiaLite module and output the CLI command you should use to enable it:

datasette spatial.db --load-extension=/usr/local/lib/mod_spatialite.dylib

Even better: if Datasette had a --spatialite option which automatically loads the extension from common locations, if it can find it.

I added this to sqlite-utils in https://github.com/simonw/sqlite-utils/issues/136 and I really like it: pass a special value of spatialite and Datasette should attempt to load it from known likely installation locations.

When starting datasette on a SpatialLite database without loading the SpatiaLite extension (using eg --load-extension=/usr/local/lib/mod_spatialite.dylib) an error is thrown and the server fails to start:

datasette -p 8003 adminboundaries.db 
Serve! files=('adminboundaries.db',) on port 8003
Traceback (most recent call last):
  File "/Users/ajh59/anaconda3/bin/datasette", line 11, in <module>
    sys.exit(cli())
  File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/datasette/cli.py", line 552, in serve
    ds.inspect()
  File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/datasette/app.py", line 273, in inspect
    "tables": inspect_tables(conn, self.metadata.get("databases", {}).get(name, {}))
  File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/datasette/inspect.py", line 79, in inspect_tables
    "PRAGMA table_info({});".format(escape_sqlite(table))
sqlite3.OperationalError: no such module: VirtualSpatialIndex

It would be nice to trap this and return a message saying something like:

It looks like you're trying to load a SpatiaLite database? Make sure you load in the SpatiaLite extension when starting datasette.

Read more: https://datasette.readthedocs.io/en/latest/spatialite.html

I'd like to see spatialite geometry columns rendered more sensibly - at the moment they come through as well-known-binary unless you use custom SQL, and WKB isn't of much use to anyone on the web.

In HTML: they should be shown either as simple lat/long (if it's just a point, for example), or as a sensible placeholder if they're more complex geometries.

In JSON: they should be GeoJSON geometries, (which means they can be automatically fed into a leaflet map with no further messing around).

In CSV: they should be WKT.

I briefly wondered if this should go into a plugin, but I suspect it needs hooking in at a deeper level than the plugin architecture will support any time soon.

Split off from #243. Need to figure out how to install and configure SpatiaLite on Heroku.

https://docs.datasette.io/en/0.55/spatialite.html was written before I had tools like geojson-to-sqlite and shapefile-to-sqlite.

Spotted this in the Django SpatiaLite backend: https://github.com/django/django/blob/8f6a7a0e9e7c5404af6520ae606927e32415eb00/django/contrib/gis/db/backends/spatialite/base.py#L24-L36

ctypes.util.find_library('spatialite')

https://www.gaia-gis.it/gaia-sins/spatialite-sql-5.0.0.html#version has a whole bunch of interesting functions for things like freexl_version() and geos_version() and HasMathSQL() and suchlike.

These could be shown on the /-/versions page.

Tips here: https://adamj.eu/tech/2022/01/18/how-to-add-a-favicon-to-your-django-site/ - I think a PNG served at /favicon.ico is the best option, since safari doesn't support SVG yet.

Relevant code: https://github.com/simonw/datasette/blob/cb29119db9115b1f40de2fb45263ed77e3bfbb3e/datasette/app.py#L182-L183

I can reuse the icon for https://datasette.io/desktop

The example output for datasette plugins --allon this page has got out of date: https://docs.datasette.io/en/stable/plugins.html#seeing-what-plugins-are-installed

https://github.com/simonw/datasette/runs/4877484366

E           File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/uvicorn/server.py", line 67, in run
E             return asyncio.run(self.serve(sockets=sockets))
E         AttributeError: module 'asyncio' has no attribute 'run'

I think this may mean uvicorn has dropped support for Python 3.6.

I tried running some parallel SQL queries using asyncio.gather() but the SQL that was executed didn't show up in the trace rendered by https://datasette.io/plugins/datasette-pretty-traces

I realized that was because traces are keyed against the current task ID, which changes when a sub-task is run using asyncio.gather or similar.

The faceting and suggest faceting queries are missing from this trace:

The reason they aren't showing up in the traces is that traces are stored just for the currently executing asyncio task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25

This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively "child tasks" of the current request, and hence should be tracked the same.

https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using asyncio.gather() to work around this kind of problem.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1518#issuecomment-999870993_

context_vars can solve this but they were introduced in Python 3.7: https://www.python.org/dev/peps/pep-0567/

Python 3.6 support ends in a few days time, and it looks like Glitch has updated to 3.7 now - so maybe I can get away with Datasette needing 3.7 these days?

Tweeted about that here: https://twitter.com/simonw/status/1473761478155010048

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1576#issuecomment-999878907_

Running ANALYZE creates a new visible table called sqlite_stat1: https://www.sqlite.org/fileformat.html#the_sqlite_stat1_table

This should be added to the default list of hidden tables in Datasette.

Inspired by https://github.com/simonw/sqlite-utils/issues/264 - in particular this example:

datasette covid.db --get='/covid.yaml?sql=select * from ny_times_us_counties limit 1' 
- date: '2020-01-21'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0

Having to construct that URL - including potentially URL escaping the SQL query - isn't a great developer experience.

Imagine if you could do this instead:

datasette covid.db --query "select * from ny_times_us_counties limit 1" --format yaml

Looks like this API should be able to do that: https://twitter.com/steren/status/1483835859191304192 - https://cloud.google.com/run/docs/reference/rest/v1/namespaces.domainmappings/create

Updates the requirements on pytest-timeout to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This page jumps straight into running the tests: https://sqlite-utils.datasette.io/en/latest/contributing.html

It should add a little more about expected collaboration styles - opening an issue before filing a pull request - and probably link to https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/

Refs #79

This PR adds three new functions and consolidates Spatialite-related stuff in a new sqlite_utils.gis module.

• the find_spatialite function and SPATIALITE_PATHS constant are moved to gis.py
• init_spatialite loads the Spatialite extension and initializes it
• add_geometry_column adds a geometry column
• create_spatial_index creates a spatial index

Has tests and documentation. Feedback very welcome.

As demonstrated by this piece of documentation, using SpatiaLite with sqlite-utils requires a fair bit of boilerplate:
https://github.com/simonw/sqlite-utils/blob/f7289174e66ae4d91d57de94bbd9d09fabf7aff4/docs/python-api.rst#L880-L909

https://twitter.com/psychemedia/status/1481171650934714370

can extensions be passed their own cli args? eg --ext-tiddlywiki-dbname tiddlywiki2.sqlite ?

I've thought something like this might be useful for other plugins in the past, too.

Inspired by https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html

Good example: https://github.com/rust-analyzer/rust-analyzer/blob/d7c99931d05e3723d878bea5dc26766791fa4e69/docs/dev/architecture.md

Using this as a tracking issue. I'm hoping to get the bulk of the JSON redesign work from the refactor in #1554 in for this release.

Datasette appends the prefix found in the base_url setting twice if a base_url is set.

In the follow asgi example, I'm hosting a custom Datasette instance:

# asgi.py
import pathlib
from asgi_cors import asgi_cors
from channels.routing import URLRouter
from django.urls import re_path
from datasette.app import Datasette

datasette_ = Datasette(
    files=[],
    settings={
        "base_url": "/datasettes/",
        "plugins": {}
    },
    config_dir=pathlib.Path('.'),
)
application = URLRouter([
  re_path(r"^datasettes/.*", asgi_cors(datasette_.app(), allow_all=True)),
])

Running it with:

$ daphne -p 8002 asgi:application

Using a simple query on the _memory table:

select sqlite_version()

http://localhost:8002/datasettes/_memory?sql=select+sqlite_version%28%29

It renders the following upon inspection:

I am using datasette version 0.59.4

Similar bug to #1525 (and #1506 before it). Start on https://latest.datasette.io/fixtures/facetable?_facet=_neighborhood - then select a neighborhood - then try to remove that filter using the little "x" and submitting the form again.

I introduced cog in #1594 - I can use this to replace the older update-docs-help.py mechanism:

https://github.com/simonw/datasette/blob/76d66d5b2bf10249c0beaac0999b93ac8d757f48/tests/test_docs.py#L36-L53

Made obvious by the new CLI reference page added in #1594. https://docs.datasette.io/en/latest/cli-reference.html#datasette-inspect-help

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect
  install    Install Python packages - e.g.

Usage: datasette inspect [OPTIONS] [FILES]...

Options:
  --inspect-file TEXT
  --load-extension TEXT  Path to a SQLite extension to load
  --help                 Show this message and exit.

I should start this relatively soon.

Just noticed this while working on #1518.

% curl -s 'https://latest.datasette.io/fixtures/facetable?_sort=pk&on_earth__exact=1' | grep -C 1 '<title>'
<head>
    <title>fixtures: facetable: 14 rows
    where where on_earth = 1 sorted by pk</title>

See https://datasette.io/desktop and https://simonwillison.net/2021/Sep/8/datasette-desktop/

Thought of this while posting this comment: https://github.com/simonw/datasette/issues/1591#issuecomment-1012506595

I added https://sqlite-utils.datasette.io/en/stable/cli-reference.html to sqlite-utils in https://github.com/simonw/sqlite-utils/issues/383 and I really like it - it's a page showing the --help output of every CLI command for that tool.

It's maintained using cog. One of the benefits is that I get a free commit history of changes to --help at https://github.com/simonw/sqlite-utils/commits/main/docs/cli-reference.rst

explain query plan select * from facetable is allowed: https://latest.datasette.io/fixtures?sql=explain+query+plan+select+*+from+facetable

But... explain query plan select * from facetable (with two spaces before the select) returns a "Statement must be a SELECT" error: https://latest.datasette.io/fixtures?sql=explain+query+plan++select+*+from+facetable

Every single piece of code I've written against the write APIs has used the block=True option to wait for the result.

Without that, it instead fires the write into the queue but then continues even before it has finished executing.

block=True should clearly be the default behaviour here!

I've created a canned query with "write": true set. I've also created a custom template for it, but the template doesn't seem to be found. If I look in the HTML I see (stock_exchange is the db name):

<!-- Templates considered: query-stock_exchange.html, *query.html -->

My non-writeable canned queries pick up custom templates as expected, and if I look at their HTML I see the canned query name added to the templates considered (the canned query here is date_search):

<!-- Templates considered: query-stock_exchange-date_search.html, query-stock_exchange.html, *query.html -->

So it seems like the writeable canned query is behaving differently for some reason. Is it an authentication thing? I'm using the built in --root authentication.

Thanks!

Spotted while working on #1555

https://user-images.githubusercontent.com/9599/146653166-ce5358bc-b822-4961-a2dd-ed3efaa8dd13.png">

There's no indication there of how many times execute_write_many() executed the SQL.

Solving this is a tiny bit tricky because params_seq is an iterator that we don't want to exhaust before passing it to conn.executemany() - so we need to instead wrap it in something that counts how many times it was called.

But then we need a way to attach that to the trace here: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/database.py#L115-L122

So probably need to redesign the trace() decorator to allow extra pairs to be attached to it within the with statement.

This is misleading, since usually there have been more than one query executed:

Rather than adding a executemany=True parameter, I'm now thinking a better design might be to have three methods:

• db.execute_write(sql, params=None, block=False)
• db.execute_write_script(sql, block=False)
• db.execute_write_many(sql, params_seq, block=False)

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1555#issuecomment-997267416_

On the first hit to a restarted index I'm seeing this in the SQL traces: https://latest-with-plugins.datasette.io/github/commits?_trace=1

https://user-images.githubusercontent.com/9599/145907011-e4abc93c-8dbf-468d-afef-13d5e7463f92.png">

I imagine this could be sped up a lot using tricks like this one from the SQLite documentation: https://sqlite.org/pragma.html#pragfunc

SELECT DISTINCT m.name || '.' || ii.name AS 'indexed-columns'
  FROM sqlite_schema AS m,
       pragma_index_list(m.name) AS il,
       pragma_index_info(il.name) AS ii
 WHERE m.type='table'
 ORDER BY 1;

https://latest-with-plugins.datasette.io/fixtures?sql=SELECT+DISTINCT+m.name+%7C%7C+%27.%27+%7C%7C+ii.name+AS+%27indexed-columns%27%0D%0A++FROM+sqlite_schema+AS+m%2C%0D%0A+++++++pragma_index_list%28m.name%29+AS+il%2C%0D%0A+++++++pragma_index_info%28il.name%29+AS+ii%0D%0A+WHERE+m.type%3D%27table%27%0D%0A+ORDER+BY+1%3B

Idea: teach execute_write to accept an optional executescript=True parameter, like this:

diff --git a/datasette/database.py b/datasette/database.py
index 468e936..1a424f5 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -94,10 +94,14 @@ class Database:
             f"file:{self.path}{qs}", uri=True, check_same_thread=False
         )

-    async def execute_write(self, sql, params=None, block=False):
+    async def execute_write(self, sql, params=None, executescript=False, block=False):
+        assert not executescript and params, "Cannot use params with executescript=True"
         def _inner(conn):
             with conn:
-                return conn.execute(sql, params or [])
+                if executescript:
+                    return conn.executescript(sql)
+                else:
+                    return conn.execute(sql, params or [])

         with trace("sql", database=self.name, sql=sql.strip(), params=params):
             results = await self.execute_write_fn(_inner, block=block)

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1555#issuecomment-997248364_

Here's why - trace only applies to read, not write SQL operations: https://github.com/simonw/datasette/blob/7c8f8aa209e4ba7bf83976f8495d67c28fbfca24/datasette/database.py#L209-L211

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1555#issuecomment-997128508_

>>> ds = Datasette(memory=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__() missing 1 required positional argument: 'files'
>>> ds = Datasette(memory=True, files=[])

I wanted to create an in-memory Datasette for running some tests, no point in forcing me to pass files=[] to do that.

I added this in 0b8c1b0a6da9cb8ac0d28cc90dd783de87554036 but it's never been documented and the same thing can now be achieved using the prepare_connection plugin hook.

https://github.com/simonw/datasette/blob/0c91e59d2bbfc08884cfcf5d1b902a2f4968b7ff/datasette/app.py#L262

https://github.com/simonw/datasette/blob/0c91e59d2bbfc08884cfcf5d1b902a2f4968b7ff/datasette/app.py#L551-L552

It's used here in the tests:

https://github.com/simonw/datasette/blob/69244a617b1118dcbd04a8f102173f04680cf08c/tests/fixtures.py#L156

Start here: https://v0-30-2.datasette.io/fixtures/facetable?_facet_array=tags

https://user-images.githubusercontent.com/9599/68562011-c0d44480-03fc-11ea-8354-def0ba35d365.png">

Note that tags is offered as a suggested facet. But if you click that you get this:

https://v0-30-2.datasette.io/fixtures/facetable?_facet_array=tags&_facet=tags

The _facet=tags is added to the URL and it's removed from the list of suggested tags... but the facet itself is not displayed:

https://user-images.githubusercontent.com/9599/68562039-df3a4000-03fc-11ea-9778-81f7a931d608.png">

The _facet=tags facet should look like this:

https://user-images.githubusercontent.com/9599/68562067-f8db8780-03fc-11ea-8211-26281b7df9f9.png">

Found I wanted this while I was debugging #625 just to clean up the debug traces, but it makes sense as a partner to ?_nofacet=1 and ?_nocount=1 from #1350 and #1353.

For now, you can set a facets value (array) in your metadata file but I couldn't find a way to set a facets_array in order to provide default facets for arrays (like tags). My use-case is to access to that kind of view by default without URL's parameters as with other default facets.

I'm new to datasette, and I'm willing to help with a PR if that is not already implemented and I missed it!

You've caused me to rethink this feature - I no longer think there's value in only showing these numbers if ?_facet_size=max as opposed to all of the time.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1423#issuecomment-995023410_

This code in TableView wouldn't be necessary: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/views/table.py#L396-L399

If that happened here instead: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/utils/asgi.py#L98-L100

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1518#issuecomment-991827468_

https://prettier.io/ - I'm going to go with 2 spaces.

https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++%5Bfoo%5D

https://user-images.githubusercontent.com/9599/68550604-1ab31b00-03b9-11ea-8a33-5dc339b8c5f5.jpeg" width="500">

Would be useful if this page showed you the invalid SQL you entered so you can edit it and try again.

I'm not happy with how metadata.(json|yaml) keeps growing new features. Rather than having a single plugin hook for all of metadata.json I'm going to split out the feature that shows actual real metadata for tables and databases - source, license etc - into its own plugin-powered mechanism.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/357#issuecomment-647189045_

Could look something like this:

{
    "title": "Five Thirty Eight",
    "license": "CC Attribution 4.0 License",
    "license_url": "https://creativecommons.org/licenses/by/4.0/",
    "source": "fivethirtyeight/data on GitHub",
    "source_url": "https://github.com/fivethirtyeight/data",
    "databases": {
        "fivethirtyeight": {
            "tables": {
                "mueller-polls/mueller-approval-polls": {
                    "description_html": "<p>....</p>",
                    "columns": {
                        "name_of_column": "column_description goes here"
}

The datasette-write plugin provides this at the moment https://github.com/simonw/datasette-write - but it feels like it should be a built-in capability, protected by a default permission.

UI concept: if you have write permission then the existing SQL editor gets an "execute write" checkbox underneath it.

JavaScript can spot if you appear to be trying to execute an UPDATE or INSERT or DELETE query and check that checkbox for you.

If you link to a query page with a non-SELECT then that query will be displayed in the box ready for you to POST submit it. The page will also then get "cannot be embedded" headers to protect against clickjacking.

On this page when signed in as root: https://latest.datasette.io/fixtures/roadside_attraction_characteristics/1

If I'm going to minify it I'll need to figure out a build step in Datasette itself so that I can easily work on that minified version.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/983#issuecomment-752748496_

I'm going to need to add JavaScript unit tests for this new plugin system.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/983#issuecomment-752757289_

This isn't particularly straight-forward - all the more reason for Datasette to implement it for you. This article is helpful: http://charlesleifer.com/blog/using-sqlite-full-text-search-with-python/

Can be part of #870 - refactoring existing views to use register_routes().

I'm going to put the new check_permissions() method on BaseView as well. If I want that method to be available to plugins I can do so by turning that BaseView class into a documented API that plugins are encouraged to use themselves.
_Originally posted by @simonw in https://github.com/simonw/datasette/issues/832#issuecomment-651995453_

Using datasette to solve some frustrating problems with our fulfillment provider today, I was surprised to see repeated requests for assets under /-/static and the favicon. While it won't likely be a huge performance bottleneck, I bet datasette would feel a bit zippier if you had Uvicorn serving up some caching-related headers telling the browser it was safe to cache static assets.

https://latest.datasette.io/fixtures/facetable.json doesn't currently include the CREATE TABLE statement for the page, even though it's available on the HTML version at https://latest.datasette.io/fixtures/facetable

Datasette views currently work by creating a set of data that should be returned as JSON, then defining an additional, optional template_data() function which is called if the view is being rendered as HTML.

This template_data() function calculates extra template context variables which are necessary for the HTML view but should not be included in the JSON.

Example of how that is used today: https://github.com/simonw/datasette/blob/2b79f2bdeb1efa86e0756e741292d625f91cb93d/datasette/views/table.py#L672-L704

With features like Facets in #255 I'm beginning to want to move more items into the template_data() - in the case of facets it's the suggested_facets array. This saves that feature from being calculated (involving several SQL queries) for the JSON case where it is unlikely to be used.

But... as an API user, I want to still optionally be able to access that information.

Solution: Add a ?_extra=suggested_facets&_extra=table_metadata argument which can be used to optionally request additional blocks to be added to the JSON API.

Then redefine as many of the current template_data() features as extra arguments instead, and teach Datasette to return certain extras by default when rendering templates.

This could allow the JSON representation to be slimmed down further (removing e.g. the table_definition and view_definition keys) while still making that information available to API users who need it.

Call it glossary.rst - it can use a definition list something like this:

.. _glossary:

Glossary
========

Term
  A definition of the term.

Another term
  Another definition.

Datasette docs don't talk about WAL yet, which allows you to safely serve reads from a database file while it is accepting writes.

Updates the requirements on pytest-asyncio to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I sometimes find the need to create an empty SQLite database file - for example if I want to enable WAL on it before using it with another script. I currently do that like this:

sqlite3 my.db vacuum
sqlite-utils enable-wal my.db

It would be nice if sqlite-utils had a convenience command for doing this.

Refs #1518 refactor.

Everything on https://sqlite-utils.datasette.io/en/stable/cli-reference.html would benefit from an example.

Because I often want to use it just to preview a few rows from the database. Piping through | head -n 20 works for JSON and CSV (they stream) but not for --table.

CLI equivalent of table.rows_where() - should accept parameters too. Work on this at the same time as #381.

Can be maintained using cog from #373. Similar in purpose to the API reference page, but this is for the CLI.

sqlite's query planner depends upon analyze to make good use of indices. It would be nice if analyze was run as part of the create-index command.

If data is inserted later, things can get out date, but it would still probably be a net win.

For these commits: https://github.com/simonw/sqlite-utils/compare/3.20...129141572f249ea290e2a075437e2ebaad215859

The .executemany() method is a very efficient way to execute the same SQL query against a huge list of parameters.

sqlite-utils insert supports a bunch of ways of loading a list of dictionaries - from CSV, TSV, JSON, newline JSON and more thanks to:
- #361

What if you could load a list of dictionaries and provide a SQL query with :named parameters that correspond to keys in those dictionaries instead?

This would need to be a new command - I thought about adding a --sql option to insert but that doesn't make sense as that command already requires a table name.

Refs #375

Still needs:

• Refactor @insert_upsert_options so that it doesn't duplicate @import_options
• Tests
• Documentation
• Try it against a really big file

Is there a reason I've not added --extract as an option for sqlite-utils insert next? There's a extracts= option for the various table.insert() etc methods - last line in this code block:

https://github.com/simonw/sqlite-utils/blob/213a0ff177f23a35f3b235386366ff132eb879f1/sqlite_utils/db.py#L2483-L2495

The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865_

• sqlite-utils analyze command
• sqlite-utils create-index --analyze option (see #365)
• sqlite-utils insert --analyze option
• sqlite-utils upsert --analyze option

In #378 I also added .delete_where(..., analyze=True) but there isn't currently a sqlite-utils delete-where CLI command - deletions via CLI are expected to be handled using SQL queries.

Relevant documentation: https://www.sqlite.org/lang_analyze.html

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007633376_

This would cause ANALYZE to be run against the relevant table at the end of executing the method.

Having browsed the API reference I think the methods that would benefit from an analyze=True parameter are:

• table.create_index
• table.insert_all
• table.upsert_all
• table.delete_where

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898_

Refs #366

Spotted this while manually testing #364 - there's no reason --nl should crash if you feed it an empty line in between JSON objects.

For https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files

Given a file called dogs.jpg stem would be dogs and ext would be jpg.

Need to decide what happens for dogs.and.cats.jpg.gz.

Currently you have to do this:

$ sqlite-utils insert dogs.db dogs dogs.json --convert '
row["is_good"] = 1
return row'

Would be neat if this worked too:

$ sqlite-utils insert dogs.db dogs dogs.json \
  --convert 'row["is_good"] = 1'

Not sure why I didn't implement this.

https://sqlite-utils.datasette.io/en/stable/cli.html#table-formatted-output currently cheats and tells the user to run --help - can fix this using cog.

Add this to sqlite_utils/cli.py:

python if __name__ == "__main__": cli()
Now the tool can be run using python -m sqlite_utils.cli --help

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998_

I'm trying this, but it doesn't seem to write anything to the database file until I hit CTRL+C:

heroku logs --app=simonwillisonblog --tail | grep 'measure#nginx.service' | \
  sqlite-utils insert /tmp/herokutail.db log - --import re --convert "$(cat <<EOD
    r = re.compile(r'([^\s=]+)=(?:"(.*?)"|(\S+))')
    pairs = {}
    for key, value1, value2 in r.findall(line):
        pairs[key] = value1 or value2
    return pairs
EOD
)" --lines --batch-size 1

Per our conversation on Twitter:

It would be neat if all entities (including URLs) were captured. This way you can ensure, that URLs are parsed out exactly the same way Twitter parses URLs – we all know parsing URLs with a regex ain't fun.

Right now, I believe the tool filters out all entities that are not of type media.