issues

I just watched a demo video for the JupyterLab Chart Editor which wraps the plotly chart editor app in a JupyterLab panel and lets you open a plotly chart JSON file in that editor. Essentially, it pops an HTML app into a panel in JupyterLab, and I think registers the app as a file viewer for a particular file type. (I'm not completely taken by it, tbh, because it means you can do irreproducible things to the chart definition file, but that's another issue).

JupyterLab extensions can also open files from a dialogue as the iframe/html previewer shows: https://github.com/timkpaine/jupyterlab_iframe.

This made me wonder about what datasette integration with JupyterLab might do.

For example, by right-clicking on a CSV file (for which there is already a CSV table view) in the file browser, offer a View / Run as datasette file viewer option that will:

• run the CSV file through csvs-to-sqlite;
• launch the datasette server and display the datasette view in a JupyterLab panel.

(? Create a new SQLite db for each CSV file and launch each datasette view on a new port? Or have a JupyterLab (session?) SQLite db that stores all datasette viewed CSVs and runs on a single port?)

As a freebie, the datasette API would allow you to run efficient SQL queries against the file eg using using pandas.read_sql() queries in a notebook in the same space.

Related:

• JupyterLab extensions docs
• a cookiecutter for wrting JupyterLab extensions using Javascript
• a cookiecutter for writing JupyterLab extensions using Typescript
• tutorial: Let’s Make an xkcd JupyterLab Extension

When starting datasette on a SpatialLite database without loading the SpatiaLite extension (using eg --load-extension=/usr/local/lib/mod_spatialite.dylib) an error is thrown and the server fails to start:

datasette -p 8003 adminboundaries.db Serve! files=('adminboundaries.db',) on port 8003 Traceback (most recent call last): File "/Users/ajh59/anaconda3/bin/datasette", line 11, in <module> sys.exit(cli()) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/datasette/cli.py", line 552, in serve ds.inspect() File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/datasette/app.py", line 273, in inspect "tables": inspect_tables(conn, self.metadata.get("databases", {}).get(name, {})) File "/Users/ajh59/anaconda3/lib/python3.6/site-packages/datasette/inspect.py", line 79, in inspect_tables "PRAGMA table_info({});".format(escape_sqlite(table)) sqlite3.OperationalError: no such module: VirtualSpatialIndex

It would be nice to trap this and return a message saying something like:

``` It looks like you're trying to load a SpatiaLite database? Make sure you load in the SpatiaLite extension when starting datasette.

Read more: https://datasette.readthedocs.io/en/latest/spatialite.html ```

Provide support for launching datasette on Digital Ocean.

Example: Deploy Docker containers into Digital Ocean.

Digital Ocean also has a preconfigured VM running Docker that can be launched from the command line via the Digital Ocean API: Docker One-Click Application.

Related: - Launching containers in Digital Ocean servers running docker: How To Provision and Manage Remote Docker Hosts with Docker Machine on Ubuntu 16.04 - How To Use Doctl, the Official DigitalOcean Command-Line Client

To handle packages that require Javascript state setting prior to loading a package (eg thebelab, provide a template block before the URLs are loaded.

It is possible to expose a running datasette service in a Jupyter environment such as a MyBinder environment using the jupyter-server-proxy.

For example, using this demo Binder which has the server proxy installed, we can then upload a simple test database from the notebook homepage, from a Jupyter termianl install datasette and set it running against the test db on eg port 8001 and then view it via the path proxy/8001.

Clicking links results in 404s though because the datasette links aren't relative to the current path?

If you try to publish to Heroku using no set name (i.e. the default datasette name) and a project already exists under that name, you get a meaningful error report on the first line followed by Py error messages that drown it out:

Creating datasette... ! ▸ Name datasette is already taken Traceback (most recent call last): File "/usr/local/bin/datasette", line 10, in <module> sys.exit(cli()) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/Users/NNNNN/Library/Python/3.7/lib/python/site-packages/datasette/publish/heroku.py", line 124, in heroku create_output = check_output(cmd).decode("utf8") File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 411, in check_output **kwargs).stdout File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 512, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['heroku', 'apps:create', 'datasette', '--json']' returned non-zero exit status 1.

It would be neater if:

• the Py error message was caught;
• the report suggested setting a project name using -n etc.

It may also be useful to provide a command to list the current names that are being used, which I assume is available via a Heroku call?

If a SQLite table column name contains spaces, they are usually referred to in double quotes:

SELECT * FROM mytable WHERE "gappy column name"="my value";

In the JSON metadata file, this is passed by escaping the double quotes:

"queries": {"my query": "SELECT * FROM mytable WHERE \"gappy column name\"=\"my value\";"}

When specifying a custom query in metadata.json using double quotes, these are then rendered in the datasette query box using single quotes:

SELECT * FROM mytable WHERE 'gappy column name'='my value';

which does not work.

Alternatively, a valid custom query can be passed using backticks (`) to quote the column name and single (unescaped) quotes for the matched value:

"queries": {"my query": "SELECT * FROM mytable WHERE `gappy column name`='my value';"}

My understanding of the --reload option was that if the database file changed datasette would automatically reload.

I'm running on a Mac and from the datasette UI queries don't seem to be picking up data in a newly changed db (I checked the db timestamp - it certainly updated).

I was also expecting to see some sort of log statement in the datasette logging to say that it had detected a file change and restarted, but don't see anything there?

Will try to check on an Ubuntu box when I get a chance to see if this is a Mac thing.

Is datasette beholden to version click==6.7?

Current release is at 7.0. Can the requirement be liberalised, eg to >=6.7?

In terms of small pieces lightly joined, I note that there are several tools starting to appear for building generating Dockerfiles and building Docker containers from simpler components such as requirements.txt files.

If plugin/extensions builders want to include additional packages, then things like incremental builds of composable builds that add additional items into a base datasette container may be required.

Examples of Dockerfile generators / container builders:

• openshift/source-to-image (s2i)
• jupyter/repo2docker
• stencila/dockter

Discussions / threads (via Binderhub gitter) on: - why repo2docker not s2i - why dockter not repo2docker - composability in s2i

Relates to things like:

• https://github.com/simonw/datasette/pull/280

The default preview of a database shows all columns (is the row count limited?) which is fine in many cases but can take a long time to load / offer a large overhead if the table is a SpatiaLite table containing geometry columns that include large shapefiles.

Would it make sense to have a setting that can limit the amount of text displayed in any given cell in the table preview, or (less useful?) suppress (with notification) the display of overlong columns unless enabled by the user?

An issue then arises if a user does want to see all the text in a cell:

1) for a particular cell; 2) for every cell in the table; 3) for all cells in a particular column or columns

(I haven't checked but what if a column contains e.g. raw image data? Does this display as raw data? Or can this be rendered in a context aware way as an image preview? I guess a custom template would be one way to do that?)

Trying to run datasette (version 0.14) on Heroku with a metadata.json doesn't seem to be picking up the metadata.json file?

On a Mac with dodgy tar support:

▸ Couldn't detect GNU tar. Builds could fail due to decompression errors ▸ See ▸ https://devcenter.heroku.com/articles/platform-api-deploying-slugs#create-slug-archive ▸ Please install it, or specify the '--tar' option ▸ Falling back to node's built-in compressor

Could that be causing the issue?

Also, I'm not seeing custom query links anywhere obvious when I run the metadata file with a local datasette server?

The heroku build does seem to make use of any provided metadata.json file.

Add the --metadata switch to the Heroku web launch command if a metadata.json file is available.

Addresses: https://github.com/simonw/datasette/issues/177