Is this feature covered by any more recent updates to datasette, or via any plugins that you're aware of?

I was wondering if you have any more thoughts on this? I have a tangible use case now: adding a "vector" column to a database to support semantic search using doc2vec embeddings (example; note that the vtfunc package may no longer be reliable...).

A couple+ of possible use case examples:

• someone has a collection of articles indexed with FTS; they want to publish a simple search tool over the results;
• someone has an image collection and they want to be able to search over description text to return images;
• someone has a set of locations with descriptions, and wants to run a query over places and descriptions and get results as a listing or on a map;
• someone has a set of audio or video files with titles, descriptions and/or transcripts, and wants to be able to search over them and return playable versions of returned items.

In many cases, I suspect the raw content will be in one table, but the search table will be a second (eg FTS) table. Generally, the search may be over one or more joined tables, and the results constructed from one or more tables (which may or may not be distinct from the search tables).

Wondering if this actually relates to https://github.com/simonw/sqlite-utils/issues/402 ?

I also wonder if this would be a sensible approach for eg registering pint based quantity conversions into and out of the db, perhaps storing the quantity as a serialised magnitude measurement single column string?

My hunch is that the case where you want to consider input from more than one column will actually be pretty rare - the only case I can think of where I would want to do that is for latitude/longitude columns

Other possible pairs: unconventional date/datetime and timezone pairs eg 2022-02-16::17.00, London; or more generally, numerical value and unit of measurement pairs (eg if you want to cast into and out of different measurement units using packages like pint) or currencies etc. Actually, in that case, I guess you may be presenting things that are unit typed already, and so a conversion would need to parse things into an appropriate, possibly two column value, unit format.

The CustomColumnType() approach looks good. This pushes you into the mindspace that you are defining and working with a custom column type.

When creating the table, you could then error, or at least warn, if someone wasn't setting a column on a type or a custom column type, which I guess is where mypy comes in?

Is there any progress elsewhere on the handling of compound / composite foreign keys, or is this PR still effectively open?

Is it possible to parse things like --ext-{plugin}-{arg} VALUE ?

FWIW, I had a look at watchdog for a datasette powered Jupyter notebook search tool: https://github.com/ouseful-testing/nbsearch/blob/main/nbsearch/nbwatchdog.py

Not a production thing, just an experiment trying to explore what might be possible...

Thanks; just a note that the datasette.urls.static(path) and datasette.urls.static_plugins(plugin_name, path) items both seem to be repeated and appear in the docs twice?

That workaround is probably fine. I was trying to work out whether there might be other situations where a pre-external package load might be useful but couldn't offhand bring any other examples to mind. The static plugins option also looks interesting.

I'm trying to run something behind a MyBinder proxy, but seem to have something set up incorrectly and not sure what the fix is?

I'm starting datasette with jupyter-server-proxy setup:

```

init.py

def setup_nbsearch():

return {
    "command": [
        "datasette",
        "serve",
        f"{_NBSEARCH_DB_PATH}",
        "-p",
        "{port}",
        "--config",
        "base_url:{base_url}nbsearch/"
    ],
    "absolute_url": True,
    # The following needs a the labextension installing.
    # eg in postBuild: jupyter labextension install jupyterlab-server-proxy
    "launcher_entry": {
        "enabled": True,
        "title": "nbsearch",
    },
}

```

where the base_url gets automatically populated by the server-proxy. I define the loaders as:

```

init.py

from datasette import hookimpl

@hookimpl def extra_css_urls(database, table, columns, view_name, datasette): return [ "/-/static-plugins/nbsearch/prism.css", "/-/static-plugins/nbsearch/nbsearch.css", ] ``` but these seem to also need a base_url prefix set somehow?

Currently, the generated HTML loads properly but internal links are incorrect; eg they take the form <link rel="stylesheet" href="/-/static-plugins/nbsearch/prism.css"> which resolves to eg https://notebooks.gesis.org/hub/-/static-plugins/nbsearch/prism.css rather than required URL of form https://notebooks.gesis.org/binder/jupyter/user/ouseful-testing-nbsearch-0fx1mx67/nbsearch/-/static-plugins/nbsearch/prism.css.

The main css is loaded correctly: <link rel="stylesheet" href="/binder/jupyter/user/ouseful-testing-nbsearch-0fx1mx67/nbsearch/-/static/app.css?404439">

From the docs, I note:

datasette.urls.instance() Returns the URL to the Datasette instance root page. This is usually "/"

What about the proxy case? Eg if I am using jupyter-server-proxy on a MyBinder or local Jupyter notebook server site, https://example.com:PORT/weirdpath/datasette, what does datasette.urls.instance() refer to?

• [ ] https://example.com:PORT/weirdpath/datasette
• [ ] https://example.com:PORT/weirdpath/
• [ ] https://example.com:PORT/
• [ ] https://example.com
• [ ] something else?

How does /-/static relate to current guidance docs around static regarding the --static option and metadata formulations such as "extra_js_urls": [ "/static/app.js"] (I've not managed to get this to work in a Jupyter server proxied set up; the datasette / jupyter server proxy repo may provide a useful test example, eg via MyBinder, for folk to crib from?)

Fixed by @simonw; example here: https://github.com/simonw/jupyterserverproxy-datasette-demo

So could the polling support also allow you to call sqlite_utils to update a database with csv files? (Though I'm guessing you would only want to handle changed files? Do your scrapers check and cache csv datestamps/hashes?)

So the conundrum continues.. The simple test case above now runs, but if I upsert a large number of new records (successfully) and then try to upsert a fewer number of new records to a different table, I get the same error.

If I run the same upserts again (which in the first case means there are no new records to add, because they were already added), the second upsert works correctly.

It feels as if the number of items added via an upsert >> the number of items I try to add in an upsert immediately after, I get the error.

Hmmm... just tried with installs from pip and the repo (v2.0.0 and v2.0.1) and I get the error each time (start of second run through the second loop).

Could it be sqlite3? I'm on 3.30.1.

UPDATE: just tried it on jupyter.org/try and I get the error there, too.

I think I actually had several issues in play...

The missing key was one, but I think there is also an issue as per below.

For example, in the following:

```python def init_testdb(dbname='test.db'):

if os.path.exists(dbname):
    os.remove(dbname)

conn = sqlite3.connect(dbname)
db = Database(conn)

return conn, db

conn, db = init_testdb()

c = conn.cursor() c.executescript('CREATE TABLE "test1" ("Col1" TEXT, "Col2" TEXT, PRIMARY KEY ("Col1"));') c.executescript('CREATE TABLE "test2" ("Col1" TEXT, "Col2" TEXT, PRIMARY KEY ("Col1"));')

print('Test 1...') for i in range(3): db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))

print('Test 2...') for i in range(3): db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}, {'Col1':'c','Col2':'x'}], pk=('Col1')) print('Done...')

Test 1... Test 2... IndexError: list index out of range

IndexError Traceback (most recent call last) <ipython-input-763-444132ca189f> in <module> 22 print('Test 2...') 23 for i in range(3): ---> 24 db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) 25 db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}, 26 {'Col1':'c','Col2':'x'}], pk=('Col1'))

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts) 1157 alter=alter, 1158 extracts=extracts, -> 1159 upsert=True, 1160 ) 1161

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert) 1097 # self.last_rowid will be 0 if a "INSERT OR IGNORE" happened 1098 if (hash_id or pk) and self.last_rowid: -> 1099 row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0] 1100 if hash_id: 1101 self.last_pk = row[hash_id]

IndexError: list index out of range ```

the first test works but the second fails. Is the length of the list of items being upserted leaking somewhere?

In passing, it looks like a start was made on a datasette Jupyter server extension in https://github.com/lucasdurand/jupyter-datasette although the build fails in MyBinder.

Would cookie cutter approaches also work for creating various flavours of customised templates?

I need to try to create a couple of sites for myself to get a feel for what sorts of thing are easily doable, and what cribbable cookie cutter items might be. I'm guessing https://simonwillison.net/2019/Nov/25/niche-museums/ is a good place to start from?

Chrome and Firefox both support headless screengrabs from command line, but I don't know how parameterised they can be?

Minor UI observation:

_where= renders a [remove] link whereas _facet= gets a cross to remove it.

Hmm... nope... I see an updated timestamp from ls -al on the db but no reload?

PLEASE IGNORE THE BELOW... I did a package update and rebuilt the kernel I was working in... may just have been an old version of sqlite_utils, seems to be working now. (Too many containers / too many environments!)

Has an issue been reintroduced here with FTS? eg I'm getting an error thrown by spaces in column names here:

``` /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)

def enable_fts(self, columns, fts_version="FTS5"): --> 329 "Enables FTS on the specified columns" 330 sql = """ 331 CREATE VIRTUAL TABLE "{table}_fts" USING {fts_version} ( ```

when trying an insert_all.

Also, if a col has a . in it, I seem to get:

``` /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) 327 jsonify_if_needed(record.get(key, None)) for key in all_columns 328 ) --> 329 result = self.db.conn.execute(sql, values) 330 self.db.conn.commit() 331 self.last_id = result.lastrowid

OperationalError: near ".": syntax error ```

(Can't post a worked minimal example right now; racing trying to build something against a live timing screen that will stop until next weekend in an hour or two...)

PS Hmmm I did a test and they seem to work; I must be messing up s/where else...

``` import sqlite3 from sqlite_utils import Database

dbname='testingDB_sqlite_utils.db'

!rm $dbname

conn = sqlite3.connect(dbname, timeout=10)

Setup database tables

c = conn.cursor()

setup=''' CREATE TABLE IF NOT EXISTS "test1" ( "NO" INTEGER, "NAME" TEXT );

CREATE TABLE IF NOT EXISTS "test2" ( "NO" INTEGER, TIME OF DAY TEXT );

CREATE TABLE IF NOT EXISTS "test3" ( "NO" INTEGER, AVG. SPEED (MPH) FLOAT ); '''

c.executescript(setup)

DB = Database(conn)

import pandas as pd

df1 = pd.DataFrame({'NO':[1,2],'NAME':['a','b']}) DB['test1'].insert_all(df1.to_dict(orient='records'))

df2 = pd.DataFrame({'NO':[1,2],'TIME OF DAY':['early on','late']}) DB['test2'].insert_all(df2.to_dict(orient='records'))

df3 = pd.DataFrame({'NO':[1,2],'AVG. SPEED (MPH)':['123.3','123.4']}) DB['test3'].insert_all(df3.to_dict(orient='records')) ```

all seem to work ok. I'm still getting errors in my set up though, which is not too different to the text cases?

I've just noticed that SQLite lets you IGNORE inserts that collide with a pre-existing key. This can be quite handy if you have a dataset that keeps changing in part, and you don't want to upsert and replace pre-existing PK rows but you do want to ignore collisions to existing PK rows.

Do sqlite_utils support such (cavalier!) behaviour?

Does this also relate to https://github.com/simonw/datasette/issues/283 and the ability to ATTACH DATABASE?

This would be really interesting but several possibilities in use arise, I think?

For example:

• I put a new CSV file into the import dir and a new table is created therefrom
• I put a CSV file into the import dir that replaces a previous file / table of the same name as a pre-existing table (eg files that contain monthly data in year to date). The data may also patch previous months, so a full replace / DROP on the original table may well be in order.
• I put a CSV file into the import dir that updates a table of the same name as a pre-existing table (eg files that contain last month's data)

CSV files may also have messy names compared to the table you want. Or for an update CSV, may have the form MYTABLENAME-February2019.csv etc

We also get an error if a column name contains a .

Do you have any simple working examples of how to use --static? Inspection of default served files suggests locations such as http://example.com/-/static/app.css?0e06ee.

If datasette is being proxied to http://example.com/foo/datasette, what form should arguments to --static take so that static files are correctly referenced?

Use case is here: https://github.com/psychemedia/jupyterserverproxy-datasette-demo Trying to do a really simple datasette demo in MyBinder using jupyter-server-proxy.

Another route would be something like creating a datasette IPython magic for notebooks to take a dataframe and easily render it as a datasette. You'd need to run the app in the background rather than block execution in the notebook. Related to that, or to publishing a dataframe in notebook cell for use in other cells in a non-blocking way, there may be cribs in something like https://github.com/micahscopes/nbmultitask .

In terms of integration with pandas, I was pondering two different ways datasette/csvs_to_sqlite integration may work:

• like pandasql, to provide a SQL query layer either by a direct connection to the sqlite db or via datasette API;
• as an improvement of pandas.to_sql(), which is a bit ropey (e.g. pandas.to_sql_from_csvs(), routing the dataframe to sqlite via csvs_tosqlite rather than the dodgy mapping that pandas supports).

The pandas.publish_* idea could be quite interesting though... Would it be useful/fruitful to think about publish_ as a complement to pandas.to_?

I think you need to register a domain name you own separately in order to get a non-IP address address? https://www.digitalocean.com/docs/networking/dns/

@russs Different map projections can presumably be handled on the client side using a leaflet plugin to transform the geometry (eg kartena/Proj4Leaflet) although the leaflet side would need to detect or be informed of the original projection?

Another possibility would be to provide an easy way/guidance for users to create an FK'd table containing the WGS84 projection of a non-WGS84 geometry in the original/principle table? This could then as a proxy for serving GeoJSON to the leaflet map?

To summarise that thread:

• expose full metadata.json object to the index page template, eg to allow tables to be referred to by name;
• ability to import multiple metadata.json files, eg to allow metadata files created for a specific SQLite db to be reused in a datasette referring to several database files;

It could also be useful to allow users to import a python file containing custom functions that can that be loaded into scope and made available to custom templates.