{"html_url": "https://github.com/simonw/datasette/issues/2199#issuecomment-1759952247", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2199", "id": 1759952247, "node_id": "IC_kwDOBm6k_c5o5r13", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-10-12T16:23:10Z", "updated_at": "2023-10-12T16:23:10Z", "author_association": "OWNER", "body": "Some options for where this could go:\r\n- Directly in the release notes? I'm not sure about that, those are getting pretty long already. I think the release notes should link to relevant upgrade guides.\r\n- On a new page? We could have a \"upgrade instructions\" page in the documentation.\r\n- At the bottom of the new https://docs.datasette.io/en/latest/configuration.html page\r\n\r\nI'm leaning towards the third option at the moment.\r\n\r\nBut... we may also need to provide upgrade instructions for plugin authors. Those could live in a separate area of the documentation though, since issues affecting end-users who configure Datasette and issues affecting plugin authors are unlikely to overlap much.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1940346034, "label": "Detailed upgrade instructions for metadata.yaml -> datasette.yaml"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2199#issuecomment-1760396195", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2199", "id": 1760396195, "node_id": "IC_kwDOBm6k_c5o7YOj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-10-12T21:36:25Z", "updated_at": "2023-10-12T21:36:25Z", "author_association": "OWNER", "body": "Related idea: how about a `datasette-upgrade` plugin which adds a `datasette upgrade` command that can be used to automate this process?\r\n\r\nMaybe something like this:\r\n```bash\r\ndatasette install datasette-upgrade\r\ndatasette upgrade metadata-to-config metadata.json\r\n```\r\nThis would output two new files: `metadata.yaml` and `datasette.yaml`. If files with those names existed already in the current directory they would be called `metadata-new.yaml` and `datasette-new.yaml`.\r\n\r\nThe command would tell you what it did:\r\n```\r\nYour metadata.json file has been rewritten as two files:\r\n\r\n    metadata-new.yaml\r\n    datasette.yaml\r\n\r\nStart Datasette like this to try them out:\r\n\r\n    datasette -m metadata-new.yaml -c datasette.yaml\r\n```\r\nThe command is `datasette upgrade metadata-to-config` because `metadata-to-config` is the name of the upgrade recipe. The first version of the plugin would only have that single recipe, but we could add more recipes in the future for other upgrades.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1940346034, "label": "Detailed upgrade instructions for metadata.yaml -> datasette.yaml"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2199#issuecomment-1760401731", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2199", "id": 1760401731, "node_id": "IC_kwDOBm6k_c5o7ZlD", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-10-12T21:41:42Z", "updated_at": "2023-10-12T21:41:42Z", "author_association": "CONTRIBUTOR", "body": "I dig it - I was thinking an Observable notebook where you paste your `metadata.json`/`metadata.yaml` and it would generate the new metadata + datasette.yaml files, but an extensible `datasette upgrade` plugin would be nice for future plugins.\r\n\r\nOne thing to think about: If someone has comments in their original `metadata.yaml`, could we preserve them in the new files? tbh maybe not too important bc if people cared that much they could just copy + paste, and it might be too distracting \r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1940346034, "label": "Detailed upgrade instructions for metadata.yaml -> datasette.yaml"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2199#issuecomment-1760411937", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2199", "id": 1760411937, "node_id": "IC_kwDOBm6k_c5o7cEh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-10-12T21:51:16Z", "updated_at": "2023-10-12T21:51:16Z", "author_association": "OWNER", "body": "I think I'm OK with not preserving comments, just because it adds a level of complexity to the tool which I don't think is worth the value it provides.\r\n\r\nIf people want to keep their comments I'm happy to leave them to copy those over by hand.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1940346034, "label": "Detailed upgrade instructions for metadata.yaml -> datasette.yaml"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2199#issuecomment-1760412424", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2199", "id": 1760412424, "node_id": "IC_kwDOBm6k_c5o7cMI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-10-12T21:51:44Z", "updated_at": "2023-10-12T21:51:44Z", "author_association": "OWNER", "body": "Started playing with this plugin idea, now tearing myself away to work on something more important:\r\n```python\r\nfrom datasette import hookimpl\r\nimport click\r\nimport pathlib\r\n\r\n\r\n@hookimpl\r\ndef register_commands(cli):\r\n    @cli.group()\r\n    def upgrade():\r\n        \"\"\"\r\n        Apply configuration upgrades to an existing Datasette instance\r\n        \"\"\"\r\n        pass\r\n\r\n\r\n    @upgrade.command()\r\n    @click.argument(\r\n        \"metadata\", type=click.Path(exists=True)\r\n    )\r\n    @click.option(\r\n        \"new_metadata\", \"-m\", \"--new-metadata\", help=\"Path to new metadata.yaml file\", type=click.Path(exists=False)\r\n    )\r\n    @click.option(\r\n        \"new_datasette\", \"-c\", \"--new-datasette\", help=\"Path to new datasette.yaml file\", type=click.Path(exists=False)\r\n    )\r\n    @click.option(\r\n        \"output_dir\", \"-e\", \"--output-dir\", help=\"Directory to write new files to\", type=click.Path(), default=\".\"\r\n    )\r\n    def metadata_to_config(metadata, new_metadata, new_datasette, output_dir):\r\n        \"\"\"\r\n        Upgrade an existing metadata.json/yaml file to the new metadata.yaml and\r\n        datasette.yaml split introduced prior to Datasette 1.0.\r\n        \"\"\"\r\n        print(\"Upgrading {} to new metadata.yaml format\".format(metadata))\r\n        output_dir = pathlib.Path(output_dir)\r\n        if not new_metadata:\r\n            # Pick a filename for the new metadata.yaml file that does not yet exist\r\n            new_metadata = pick_filename(\"metadata\", output_dir)\r\n        if not new_datasette:\r\n            new_datasette = pick_filename(\"datasette\", output_dir)\r\n        print(\"New metadata.yaml file will be written to {}\".format(new_metadata))\r\n        print(\"New datasette.yaml file will be written to {}\".format(new_datasette))\r\n\r\n\r\ndef pick_filename(base, output_dir):\r\n    options = [\"{}.yaml\".format(base), \"{}-new.yaml\".format(base)]\r\n    i = 0\r\n    while True:\r\n        option = options.pop(0)\r\n        option_path = output_dir / option\r\n        if not option_path.exists():\r\n            return option_path\r\n        # If we ran out\r\n        if not options:\r\n            i += 1\r\n            options = [\"{}-new-{}.yaml\".format(base, i)]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1940346034, "label": "Detailed upgrade instructions for metadata.yaml -> datasette.yaml"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2199#issuecomment-1760413191", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2199", "id": 1760413191, "node_id": "IC_kwDOBm6k_c5o7cYH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-10-12T21:52:25Z", "updated_at": "2023-10-12T21:52:25Z", "author_association": "OWNER", "body": "Demo of that logic:\r\n```\r\n$ datasette upgrade metadata-to-config ../datasette/metadata.json\r\nUpgrading ../datasette/metadata.json to new metadata.yaml format\r\nNew metadata.yaml file will be written to metadata-new-1.yaml\r\nNew datasette.yaml file will be written to datasette.yaml\r\n$ touch metadata-new-1.yaml\r\n$ datasette upgrade metadata-to-config ../datasette/metadata.json\r\nUpgrading ../datasette/metadata.json to new metadata.yaml format\r\nNew metadata.yaml file will be written to metadata-new-2.yaml\r\nNew datasette.yaml file will be written to datasette.yaml\r\n$ touch datasette.yaml\r\n$ datasette upgrade metadata-to-config ../datasette/metadata.json\r\nUpgrading ../datasette/metadata.json to new metadata.yaml format\r\nNew metadata.yaml file will be written to metadata-new-2.yaml\r\nNew datasette.yaml file will be written to datasette-new.yaml\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1940346034, "label": "Detailed upgrade instructions for metadata.yaml -> datasette.yaml"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2199#issuecomment-1760441535", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2199", "id": 1760441535, "node_id": "IC_kwDOBm6k_c5o7jS_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-10-12T22:08:42Z", "updated_at": "2023-10-12T22:08:42Z", "author_association": "OWNER", "body": "Pushed that incomplete code here: https://github.com/datasette/datasette-upgrade", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1940346034, "label": "Detailed upgrade instructions for metadata.yaml -> datasette.yaml"}, "performed_via_github_app": null}