issue_comments

Pushed that incomplete code here: https://github.com/datasette/datasette-upgrade

Demo of that logic: $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-1.yaml New datasette.yaml file will be written to datasette.yaml $ touch metadata-new-1.yaml $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-2.yaml New datasette.yaml file will be written to datasette.yaml $ touch datasette.yaml $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-2.yaml New datasette.yaml file will be written to datasette-new.yaml

Started playing with this plugin idea, now tearing myself away to work on something more important: ```python from datasette import hookimpl import click import pathlib

@hookimpl def register_commands(cli): @cli.group() def upgrade(): """ Apply configuration upgrades to an existing Datasette instance """ pass

@upgrade.command()
@click.argument(
    "metadata", type=click.Path(exists=True)
)
@click.option(
    "new_metadata", "-m", "--new-metadata", help="Path to new metadata.yaml file", type=click.Path(exists=False)
)
@click.option(
    "new_datasette", "-c", "--new-datasette", help="Path to new datasette.yaml file", type=click.Path(exists=False)
)
@click.option(
    "output_dir", "-e", "--output-dir", help="Directory to write new files to", type=click.Path(), default="."
)
def metadata_to_config(metadata, new_metadata, new_datasette, output_dir):
    """
    Upgrade an existing metadata.json/yaml file to the new metadata.yaml and
    datasette.yaml split introduced prior to Datasette 1.0.
    """
    print("Upgrading {} to new metadata.yaml format".format(metadata))
    output_dir = pathlib.Path(output_dir)
    if not new_metadata:
        # Pick a filename for the new metadata.yaml file that does not yet exist
        new_metadata = pick_filename("metadata", output_dir)
    if not new_datasette:
        new_datasette = pick_filename("datasette", output_dir)
    print("New metadata.yaml file will be written to {}".format(new_metadata))
    print("New datasette.yaml file will be written to {}".format(new_datasette))

def pick_filename(base, output_dir): options = ["{}.yaml".format(base), "{}-new.yaml".format(base)] i = 0 while True: option = options.pop(0) option_path = output_dir / option if not option_path.exists(): return option_path # If we ran out if not options: i += 1 options = ["{}-new-{}.yaml".format(base, i)] ```

I think I'm OK with not preserving comments, just because it adds a level of complexity to the tool which I don't think is worth the value it provides.

If people want to keep their comments I'm happy to leave them to copy those over by hand.

Related idea: how about a datasette-upgrade plugin which adds a datasette upgrade command that can be used to automate this process?

Maybe something like this: bash datasette install datasette-upgrade datasette upgrade metadata-to-config metadata.json This would output two new files: metadata.yaml and datasette.yaml. If files with those names existed already in the current directory they would be called metadata-new.yaml and datasette-new.yaml.

The command would tell you what it did: ``` Your metadata.json file has been rewritten as two files:

metadata-new.yaml
datasette.yaml

Start Datasette like this to try them out:

datasette -m metadata-new.yaml -c datasette.yaml

`` The command isdatasette upgrade metadata-to-configbecausemetadata-to-config` is the name of the upgrade recipe. The first version of the plugin would only have that single recipe, but we could add more recipes in the future for other upgrades.

Some options for where this could go: - Directly in the release notes? I'm not sure about that, those are getting pretty long already. I think the release notes should link to relevant upgrade guides. - On a new page? We could have a "upgrade instructions" page in the documentation. - At the bottom of the new https://docs.datasette.io/en/latest/configuration.html page

I'm leaning towards the third option at the moment.

But... we may also need to provide upgrade instructions for plugin authors. Those could live in a separate area of the documentation though, since issues affecting end-users who configure Datasette and issues affecting plugin authors are unlikely to overlap much.