Hi

I note that you define _hash() to create a hash_id from non-id column values in a table here.

It would be useful to be able to call a complementary function to generate a corresponding _id from a subset of specified columns when adding items to another table, eg to support the creation of foreign keys.

Or is there a better pattern for doing that?

When starting datasette on a SpatialLite database without loading the SpatiaLite extension (using eg --load-extension=/usr/local/lib/mod_spatialite.dylib) an error is thrown and the server fails to start:

datasette -p 8003 adminboundaries.db Serve! files=('adminboundaries.db',) on port 8003 Traceback (most recent call last): File "/Users/ajh59/anaconda3/bin/datasette", line 11, in <module> sys.exit(cli()) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/datasette/cli.py", line 552, in serve ds.inspect() File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/datasette/app.py", line 273, in inspect "tables": inspect_tables(conn, self.metadata.get("databases", {}).get(name, {})) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/datasette/inspect.py", line 79, in inspect_tables "PRAGMA table_info({});".format(escape_sqlite(table)) sqlite3.OperationalError: no such module: VirtualSpatialIndex

It would be nice to trap this and return a message saying something like:

``` It looks like you're trying to load a SpatiaLite database? Make sure you load in the SpatiaLite extension when starting datasette.

Read more: https://datasette.readthedocs.io/en/latest/spatialite.html ```

Provide support for launching datasette on Digital Ocean.

Example: Deploy Docker containers into Digital Ocean.

Digital Ocean also has a preconfigured VM running Docker that can be launched from the command line via the Digital Ocean API: Docker One-Click Application.

Related: - Launching containers in Digital Ocean servers running docker: How To Provision and Manage Remote Docker Hosts with Docker Machine on Ubuntu 16.04 - How To Use Doctl, the Official DigitalOcean Command-Line Client

To handle packages that require Javascript state setting prior to loading a package (eg thebelab, provide a template block before the URLs are loaded.

It is possible to expose a running datasette service in a Jupyter environment such as a MyBinder environment using the jupyter-server-proxy.

For example, using this demo Binder which has the server proxy installed, we can then upload a simple test database from the notebook homepage, from a Jupyter termianl install datasette and set it running against the test db on eg port 8001 and then view it via the path proxy/8001.

Clicking links results in 404s though because the datasette links aren't relative to the current path?

If I try to add a list of dicts to an empty table using upsert_all, I get an error:

```python import sqlite3 from sqlite_utils import Database import pandas as pd

conx = sqlite3.connect(':memory') cx = conx.cursor() cx.executescript('CREATE TABLE "test" ("Col1" TEXT);')

q="SELECT * FROM test;" pd.read_sql(q, conx) #shows empty table

db = Database(conx) db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])

TypeError Traceback (most recent call last) <ipython-input-74-8c26d93d7587> in <module> 1 db = Database(conx) ----> 2 db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts) 1157 alter=alter, 1158 extracts=extracts, -> 1159 upsert=True, 1160 ) 1161

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert) 1040 sql = "INSERT OR IGNORE INTO {table} VALUES({pk_placeholders});".format( 1041 table=self.name, -> 1042 pks=", ".join(["[{}]".format(p) for p in pks]), 1043 pk_placeholders=", ".join(["?" for p in pks]), 1044 )

TypeError: 'NoneType' object is not iterable

```

A hacky workaround in use is:

python try: db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}]) except: db['test'].insert_all([{'Col1':'a'},{'Col1':'b'}])

If a SQLite table column name contains spaces, they are usually referred to in double quotes:

SELECT * FROM mytable WHERE "gappy column name"="my value";

In the JSON metadata file, this is passed by escaping the double quotes:

"queries": {"my query": "SELECT * FROM mytable WHERE \"gappy column name\"=\"my value\";"}

When specifying a custom query in metadata.json using double quotes, these are then rendered in the datasette query box using single quotes:

SELECT * FROM mytable WHERE 'gappy column name'='my value';

which does not work.

Alternatively, a valid custom query can be passed using backticks (`) to quote the column name and single (unescaped) quotes for the matched value:

"queries": {"my query": "SELECT * FROM mytable WHERE `gappy column name`='my value';"}

My understanding of the --reload option was that if the database file changed datasette would automatically reload.

I'm running on a Mac and from the datasette UI queries don't seem to be picking up data in a newly changed db (I checked the db timestamp - it certainly updated).

I was also expecting to see some sort of log statement in the datasette logging to say that it had detected a file change and restarted, but don't see anything there?

Will try to check on an Ubuntu box when I get a chance to see if this is a Mac thing.

Irrrespective of whether using column names containing a space or - character is good practice, SQLite does allow it, but sqlite-utils throws an error in the following cases:

```python from sqlite_utils import Database

dbname = 'test.db' DB = Database(sqlite3.connect(dbname))

import pandas as pd df = pd.DataFrame({'col1':range(3), 'col2':range(3)})

Convert pandas dataframe to appropriate list/dict format

DB['test1'].insert_all( df.to_dict(orient='records') )

Works fine

```

However:

python df = pd.DataFrame({'col 1':range(3), 'col2':range(3)}) DB['test1'].insert_all(df.to_dict(orient='records'))

throws:

```

OperationalError Traceback (most recent call last) <ipython-input-27-070b758f4f92> in <module>() 1 import pandas as pd 2 df = pd.DataFrame({'col 1':range(3), 'col2':range(3)}) ----> 3 DB['test1'].insert_all(df.to_dict(orient='records'))

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) 327 jsonify_if_needed(record.get(key, None)) for key in all_columns 328 ) --> 329 result = self.db.conn.execute(sql, values) 330 self.db.conn.commit() 331 self.last_id = result.lastrowid

OperationalError: near "1": syntax error ```

and:

python df = pd.DataFrame({'col-1':range(3), 'col2':range(3)}) DB['test1'].upsert_all(df.to_dict(orient='records'))

results in:

```

OperationalError Traceback (most recent call last) <ipython-input-28-654523549d20> in <module>() 1 import pandas as pd 2 df = pd.DataFrame({'col-1':range(3), 'col2':range(3)}) ----> 3 DB['test1'].insert_all(df.to_dict(orient='records'))

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) 327 jsonify_if_needed(record.get(key, None)) for key in all_columns 328 ) --> 329 result = self.db.conn.execute(sql, values) 330 self.db.conn.commit() 331 self.last_id = result.lastrowid

OperationalError: near "-": syntax error ```

Is datasette beholden to version click==6.7?

Current release is at 7.0. Can the requirement be liberalised, eg to >=6.7?

The following works fine: ``` connX = sqlite3.connect('DELME.db', timeout=10)

dfX=pd.DataFrame({'col1':range(3),'col2':range(3)}) DBX = Database(connX) DBX['test'].upsert_all(dfX.to_dict(orient='records')) ```

But if a column is named order: ``` connX = sqlite3.connect('DELME.db', timeout=10)

dfX=pd.DataFrame({'order':range(3),'col2':range(3)}) DBX = Database(connX) DBX['test'].upsert_all(dfX.to_dict(orient='records')) ```

it throws an error:

```

OperationalError Traceback (most recent call last) <ipython-input-130-7dba33cd806c> in <module> 3 dfX=pd.DataFrame({'order':range(3),'col2':range(3)}) 4 DBX = Database(connX) ----> 5 DBX['test'].upsert_all(dfX.to_dict(orient='records'))

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order) 347 foreign_keys=foreign_keys, 348 upsert=True, --> 349 column_order=column_order, 350 ) 351

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) 327 jsonify_if_needed(record.get(key, None)) for key in all_columns 328 ) --> 329 result = self.db.conn.execute(sql, values) 330 self.db.conn.commit() 331 self.last_id = result.lastrowid

OperationalError: near "order": syntax error ```

The default preview of a database shows all columns (is the row count limited?) which is fine in many cases but can take a long time to load / offer a large overhead if the table is a SpatiaLite table containing geometry columns that include large shapefiles.

Would it make sense to have a setting that can limit the amount of text displayed in any given cell in the table preview, or (less useful?) suppress (with notification) the display of overlong columns unless enabled by the user?

An issue then arises if a user does want to see all the text in a cell:

1) for a particular cell; 2) for every cell in the table; 3) for all cells in a particular column or columns

(I haven't checked but what if a column contains e.g. raw image data? Does this display as raw data? Or can this be rendered in a context aware way as an image preview? I guess a custom template would be one way to do that?)

Trying to run datasette (version 0.14) on Heroku with a metadata.json doesn't seem to be picking up the metadata.json file?

On a Mac with dodgy tar support:

▸ Couldn't detect GNU tar. Builds could fail due to decompression errors ▸ See ▸ https://devcenter.heroku.com/articles/platform-api-deploying-slugs#create-slug-archive ▸ Please install it, or specify the '--tar' option ▸ Falling back to node's built-in compressor

Could that be causing the issue?

Also, I'm not seeing custom query links anywhere obvious when I run the metadata file with a local datasette server?

The heroku build does seem to make use of any provided metadata.json file.

Add the --metadata switch to the Heroku web launch command if a metadata.json file is available.

Addresses: https://github.com/simonw/datasette/issues/177