issues

It's not currently clear what the difference between https://docs.datasette.io/en/latest/ and https://docs.datasette.io/en/stable/ is - I should fix that.

On Twitter: https://twitter.com/simonw/status/1484285006243528705

https://github.com/simonw/datasette/runs/8194907739?check_suite_focus=true

I thought this might be an intermittent failure but attempts to re-run the tests have not made it pass.

End of that trace is:

``` /home/runner/work/datasette/datasette/datasette/app.py:234: in init self._refresh_schemas_lock = asyncio.Lock() /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/locks.py:161: in init self._loop = events.get_event_loop()

self = <asyncio.unix_events._UnixDefaultEventLoopPolicy object at 0x7fb1fc799fd0>

def get_event_loop(self):
    """Get the event loop for the current context.

    Returns an instance of EventLoop or raises an exception.
    """
    if (self._local._loop is None and
            not self._local._set_called and
            isinstance(threading.current_thread(), threading._MainThread)):
        self.set_event_loop(self.new_event_loop())

    if self._local._loop is None:
        raise RuntimeError('There is no current event loop in thread %r.'

                         % threading.current_thread().name)

E RuntimeError: There is no current event loop in thread 'MainThread'. ```

Refs #356

Still TODO:

• [x] Get --lines working, with tests
• [x] Get --text working, with tests
• [x] Get regular JSON import working with --convert with tests
• [x] Get --lines working with --convert with tests
• [x] Get --text working with --convert with tests
• [x] Get --csv and --tsv import working with --convert with tests
• [x] Get --nl working with --convert with tests
• [x] Documentation for all of the above

See https://github.com/simonw/sqlite-transform/issues/11 - I built a separate sqlite-transform tool a while ago that uses the word "transform" to means something entirely different from sqlite-utils transform - I'd like to resolve this by merging the two tools.

Some columns don't make a lot of sense in their default representation - binary blobs such as SpatiaLite geometries for example, or lengthy columns that really should be truncated somehow.

We may also find that there are tables where we don't want to show all of the columns - so a mechanism to select a subset of columns would be nice.

I think there are two features here:

• the ability to request a subset of columns on the table view
• the ability to override the SQL for a specific column and/or add extra columns - AsGeoJSON(Geometry) for example

Both features should be available via both querystring arguments and in metadata.json

The querystring argument for custom SQL should only work if allow_sql config is turned on.

Refs #276

In #683 I started a new section of internals documentation covering the Database class: https://datasette.readthedocs.io/en/latest/internals.html#database-class

I decided not to document .execute() and .execute_against_connection_in_thread() yet because I'm not 100% happy with their API design yet.

https://latest.datasette.io/-/versions is stuck on 0.35.

This thread here is illuminating: https://stackoverflow.com/questions/3634984/insert-if-not-exists-else-update

The term UPSERT in SQLite has a specific meaning as-of 3.24.0 (2018-06-04): https://www.sqlite.org/lang_UPSERT.html

It means "behave as an UPDATE or a no-op if the INSERT would violate a uniqueness constraint". The syntax in 3.24.0+ looks like this (confusingly it does not use the term "upsert"): sql INSERT INTO phonebook(name,phonenumber) VALUES('Alice','704-555-1212') ON CONFLICT(name) DO UPDATE SET phonenumber=excluded.phonenumber Here's the problem: the sqlite-utils .upsert() and .upsert_all() methods don't do this. They use the following SQL:

sql INSERT OR REPLACE INTO [{table}] ({columns}) VALUES {rows};

If the record already exists, it will be entirely replaced by a new record - as opposed to updating any specified fields but leaving existing fields as they are (the behaviour of "upsert" in SQLite itself).