issue_comments

I've been thinking more about this one today too. An extension of this (touched on in #417, Datasette Library) would be to support pointing Datasette at a directory and having it automatically load any CSV files it finds anywhere in that folder or its descendants - either loading them fully, or providing a UI that allows users to select a file to open it in Datasette.

For larger files I think the right thing to do is import them into an on-disk SQLite database, which is limited only by available disk space. For smaller files loading them into an in-memory database should work fine.

@obra there's a plugin for that! https://github.com/simonw/datasette-upload-csvs

How would Datasette accepting URLs work?

I want to support not just SQLite files and CSVs but other extensible formats (geojson, Atom, shapefiles etc) as well.

So datasette serve needs to be able to take filepaths or URLs to a variety of different content types.

If it's a URL, we can use the first 200 downloaded bytes to decide which type of file it is. This is likely more reliable than hoping the web server provided the correct content-type.

Also: let's have a threshold for downloading to disk. We will start downloading to a temp file (location controlled by an environment variable) if either the content length header is above that threshold OR we hit that much data cached in memory already and don't know how much more is still to come.

There needs to be a command line option for saying "grab from this URL but force treat it as CSV" - same thing for files on disk.

datasette mydb.db --type=db http://blah/blah --type=csv

If you provide less --type options thatn you did URLs then the default behavior is used for all of the subsequent URLs.

Auto detection could be tricky. Probably do this with a plugin hook.

https://github.com/h2non/filetype.py is interesting but deals with images video etc so not right for this purpose.

I think we need our own simple content sniffing code via a plugin hook.

What if two plugin type hooks can both potentially handle a sniffed file? The CLI can quit and return an error saying content is ambiguous and you need to specify a --type, picking from the following list.

I'm reopening this one as part of #417.

Further experience with Python's CSV standard library module has convinced me that pandas is not a required dependency for this. My sqlite-utils package can do most of the work here with very few dependencies.

I'm going to keep this separate in csvs-to-sqlite.