html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/pull/2052#issuecomment-1760552652,https://api.github.com/repos/simonw/datasette/issues/2052,1760552652,IC_kwDOBm6k_c5o7-bM,9599,2023-10-12T23:59:21Z,2023-10-12T23:59:21Z,OWNER,I'm landing this despite the cog failures. I'll fix them on main if I have to.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1651082214, https://github.com/simonw/datasette/pull/2052#issuecomment-1632867333,https://api.github.com/repos/simonw/datasette/issues/2052,1632867333,IC_kwDOBm6k_c5hU5QF,22429695,2023-07-12T16:38:27Z,2023-10-12T23:52:24Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report All modified lines are covered by tests :white_check_mark: > Comparison is base [(`3feed1f`)](https://app.codecov.io/gh/simonw/datasette/commit/3feed1f66e2b746f349ee56970a62246a18bb164?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.46% compared to head [(`8ae479c`)](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%. > Report is 112 commits behind head on main.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2052 +/- ## ========================================== + Coverage 92.46% 92.69% +0.22% ========================================== Files 38 40 +2 Lines 5750 6047 +297 ========================================== + Hits 5317 5605 +288 - Misses 433 442 +9 ``` [see 19 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2052/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214, https://github.com/simonw/datasette/pull/2052#issuecomment-1760545012,https://api.github.com/repos/simonw/datasette/issues/2052,1760545012,IC_kwDOBm6k_c5o78j0,9599,2023-10-12T23:48:16Z,2023-10-12T23:48:16Z,OWNER,Oh! I think I broke Cog on `main` and these tests are running against this branch rebased against main.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214, https://github.com/simonw/datasette/pull/2052#issuecomment-1760542865,https://api.github.com/repos/simonw/datasette/issues/2052,1760542865,IC_kwDOBm6k_c5o78CR,9599,2023-10-12T23:44:53Z,2023-10-12T23:45:15Z,OWNER,"Weird, the `cog` check is failing in CI. ``` Run cog --check docs/*.rst cog --check docs/*.rst shell: /usr/bin/bash -e {0} env: pythonLocation: /opt/hostedtoolcache/Python/3.9.18/x64 PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib/pkgconfig Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib Check failed Checking docs/authentication.rst Checking docs/binary_data.rst Checking docs/changelog.rst Checking docs/cli-reference.rst Checking docs/configuration.rst (changed) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214, https://github.com/simonw/datasette/issues/2199#issuecomment-1760441535,https://api.github.com/repos/simonw/datasette/issues/2199,1760441535,IC_kwDOBm6k_c5o7jS_,9599,2023-10-12T22:08:42Z,2023-10-12T22:08:42Z,OWNER,Pushed that incomplete code here: https://github.com/datasette/datasette-upgrade,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034, https://github.com/simonw/datasette/issues/2196#issuecomment-1760417555,https://api.github.com/repos/simonw/datasette/issues/2196,1760417555,IC_kwDOBm6k_c5o7dcT,9599,2023-10-12T21:54:52Z,2023-10-12T21:54:52Z,OWNER,"I can't replicate this bug. Closing, but please re-open if it's still happening. As an aside, the link I promote is https://datasette.io/discord which redirects: ``` curl -i 'https://datasette.io/discord' HTTP/2 301 location: https://discord.gg/ktd74dm5mw content-type: text/plain x-cloud-trace-context: 8dcfd08d3d1fa44f7ee78568e0f5305e;o=1 date: Thu, 12 Oct 2023 21:54:17 GMT server: Google Frontend content-length: 0 ``` ``` curl -i 'https://discord.gg/ktd74dm5mw' HTTP/2 301 date: Thu, 12 Oct 2023 21:54:28 GMT content-type: text/plain;charset=UTF-8 content-length: 0 location: https://discord.com/invite/ktd74dm5mw strict-transport-security: max-age=31536000; includeSubDomains; preload permissions-policy: interest-cohort=() x-content-type-options: nosniff x-frame-options: DENY x-robots-tag: noindex, nofollow, noarchive, nocache, noimageindex, noodp x-xss-protection: 1; mode=block report-to: {""endpoints"":[{""url"":""https:\/\/a.nel.cloudflare.com\/report\/v3?s=Dzzrf%2FgGkfFxtzSAQ46slMVDLcFjsH9fsvVkzHtgUUiZ891rXAa6LvTRpHK%2BdSMSQ54F57hS9z1mZXXklIbONZW1bfBuFjSK9J4XmjjLjsFUulMXvpjfCLkB6PI%3D""}],""group"":""cf-nel"",""max_age"":604800} nel: {""success_fraction"":0,""report_to"":""cf-nel"",""max_age"":604800} server: cloudflare cf-ray: 815294ddff282511-SJC ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1910269679, https://github.com/simonw/datasette/issues/2199#issuecomment-1760413191,https://api.github.com/repos/simonw/datasette/issues/2199,1760413191,IC_kwDOBm6k_c5o7cYH,9599,2023-10-12T21:52:25Z,2023-10-12T21:52:25Z,OWNER,"Demo of that logic: ``` $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-1.yaml New datasette.yaml file will be written to datasette.yaml $ touch metadata-new-1.yaml $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-2.yaml New datasette.yaml file will be written to datasette.yaml $ touch datasette.yaml $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-2.yaml New datasette.yaml file will be written to datasette-new.yaml ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034, https://github.com/simonw/datasette/issues/2199#issuecomment-1760412424,https://api.github.com/repos/simonw/datasette/issues/2199,1760412424,IC_kwDOBm6k_c5o7cMI,9599,2023-10-12T21:51:44Z,2023-10-12T21:51:44Z,OWNER,"Started playing with this plugin idea, now tearing myself away to work on something more important: ```python from datasette import hookimpl import click import pathlib @hookimpl def register_commands(cli): @cli.group() def upgrade(): """""" Apply configuration upgrades to an existing Datasette instance """""" pass @upgrade.command() @click.argument( ""metadata"", type=click.Path(exists=True) ) @click.option( ""new_metadata"", ""-m"", ""--new-metadata"", help=""Path to new metadata.yaml file"", type=click.Path(exists=False) ) @click.option( ""new_datasette"", ""-c"", ""--new-datasette"", help=""Path to new datasette.yaml file"", type=click.Path(exists=False) ) @click.option( ""output_dir"", ""-e"", ""--output-dir"", help=""Directory to write new files to"", type=click.Path(), default=""."" ) def metadata_to_config(metadata, new_metadata, new_datasette, output_dir): """""" Upgrade an existing metadata.json/yaml file to the new metadata.yaml and datasette.yaml split introduced prior to Datasette 1.0. """""" print(""Upgrading {} to new metadata.yaml format"".format(metadata)) output_dir = pathlib.Path(output_dir) if not new_metadata: # Pick a filename for the new metadata.yaml file that does not yet exist new_metadata = pick_filename(""metadata"", output_dir) if not new_datasette: new_datasette = pick_filename(""datasette"", output_dir) print(""New metadata.yaml file will be written to {}"".format(new_metadata)) print(""New datasette.yaml file will be written to {}"".format(new_datasette)) def pick_filename(base, output_dir): options = [""{}.yaml"".format(base), ""{}-new.yaml"".format(base)] i = 0 while True: option = options.pop(0) option_path = output_dir / option if not option_path.exists(): return option_path # If we ran out if not options: i += 1 options = [""{}-new-{}.yaml"".format(base, i)] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034, https://github.com/simonw/datasette/issues/2199#issuecomment-1760411937,https://api.github.com/repos/simonw/datasette/issues/2199,1760411937,IC_kwDOBm6k_c5o7cEh,9599,2023-10-12T21:51:16Z,2023-10-12T21:51:16Z,OWNER,"I think I'm OK with not preserving comments, just because it adds a level of complexity to the tool which I don't think is worth the value it provides. If people want to keep their comments I'm happy to leave them to copy those over by hand.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034, https://github.com/simonw/datasette/issues/2199#issuecomment-1760401731,https://api.github.com/repos/simonw/datasette/issues/2199,1760401731,IC_kwDOBm6k_c5o7ZlD,15178711,2023-10-12T21:41:42Z,2023-10-12T21:41:42Z,CONTRIBUTOR,"I dig it - I was thinking an Observable notebook where you paste your `metadata.json`/`metadata.yaml` and it would generate the new metadata + datasette.yaml files, but an extensible `datasette upgrade` plugin would be nice for future plugins. One thing to think about: If someone has comments in their original `metadata.yaml`, could we preserve them in the new files? tbh maybe not too important bc if people cared that much they could just copy + paste, and it might be too distracting ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034, https://github.com/simonw/datasette/issues/2199#issuecomment-1760396195,https://api.github.com/repos/simonw/datasette/issues/2199,1760396195,IC_kwDOBm6k_c5o7YOj,9599,2023-10-12T21:36:25Z,2023-10-12T21:36:25Z,OWNER,"Related idea: how about a `datasette-upgrade` plugin which adds a `datasette upgrade` command that can be used to automate this process? Maybe something like this: ```bash datasette install datasette-upgrade datasette upgrade metadata-to-config metadata.json ``` This would output two new files: `metadata.yaml` and `datasette.yaml`. If files with those names existed already in the current directory they would be called `metadata-new.yaml` and `datasette-new.yaml`. The command would tell you what it did: ``` Your metadata.json file has been rewritten as two files: metadata-new.yaml datasette.yaml Start Datasette like this to try them out: datasette -m metadata-new.yaml -c datasette.yaml ``` The command is `datasette upgrade metadata-to-config` because `metadata-to-config` is the name of the upgrade recipe. The first version of the plugin would only have that single recipe, but we could add more recipes in the future for other upgrades.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034, https://github.com/simonw/datasette/issues/2199#issuecomment-1759952247,https://api.github.com/repos/simonw/datasette/issues/2199,1759952247,IC_kwDOBm6k_c5o5r13,9599,2023-10-12T16:23:10Z,2023-10-12T16:23:10Z,OWNER,"Some options for where this could go: - Directly in the release notes? I'm not sure about that, those are getting pretty long already. I think the release notes should link to relevant upgrade guides. - On a new page? We could have a ""upgrade instructions"" page in the documentation. - At the bottom of the new https://docs.datasette.io/en/latest/configuration.html page I'm leaning towards the third option at the moment. But... we may also need to provide upgrade instructions for plugin authors. Those could live in a separate area of the documentation though, since issues affecting end-users who configure Datasette and issues affecting plugin authors are unlikely to overlap much.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034, https://github.com/simonw/datasette/pull/2190#issuecomment-1759948683,https://api.github.com/repos/simonw/datasette/issues/2190,1759948683,IC_kwDOBm6k_c5o5q-L,9599,2023-10-12T16:20:41Z,2023-10-12T16:20:41Z,OWNER,I'm going to land this and open a new issue for the upgrade instructions.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874, https://github.com/simonw/datasette/pull/2190#issuecomment-1759947534,https://api.github.com/repos/simonw/datasette/issues/2190,1759947534,IC_kwDOBm6k_c5o5qsO,9599,2023-10-12T16:19:59Z,2023-10-12T16:19:59Z,OWNER,It would be nice if we could catch that and turn that into a less intimidating Click exception too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874, https://github.com/simonw/datasette/pull/2190#issuecomment-1759947021,https://api.github.com/repos/simonw/datasette/issues/2190,1759947021,IC_kwDOBm6k_c5o5qkN,9599,2023-10-12T16:19:38Z,2023-10-12T16:19:38Z,OWNER," This looks good and works well. The error from this currently looks like: ``` datasette -m metadata.json -p 8844 Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette"", line 33, in sys.exit(load_entry_point('datasette', 'console_scripts', 'datasette')()) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1130, in __call__ return self.main(*args, **kwargs) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1055, in main rv = self.invoke(ctx) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 760, in invoke return __callback(*args, **kwargs) File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 98, in wrapped return fn(*args, **kwargs) File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 546, in serve metadata_data = fail_if_plugins_in_metadata(parse_metadata(metadata.read())) File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 1282, in fail_if_plugins_in_metadata raise Exception( Exception: Datasette no longer accepts plugin configuration in --metadata. Move your ""plugins"" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details. ``` With wrapping: `Exception: Datasette no longer accepts plugin configuration in --metadata. Move your ""plugins"" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.` I think we should link directly to documentation that tells people how to perform this upgrade.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874, https://github.com/simonw/datasette/pull/2191#issuecomment-1724480716,https://api.github.com/repos/simonw/datasette/issues/2191,1724480716,IC_kwDOBm6k_c5myXzM,22429695,2023-09-18T21:28:36Z,2023-10-12T16:15:40Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report All modified lines are covered by tests :white_check_mark: > Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`0135e7c`)](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%. > Report is 14 commits behind head on main. > :exclamation: Current head 0135e7c differs from pull request most recent head 18b48f8. Consider uploading reports for the commit 18b48f8 to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2191 +/- ## ========================================== - Coverage 92.69% 92.68% -0.02% ========================================== Files 40 40 Lines 6039 6042 +3 ========================================== + Hits 5598 5600 +2 - Misses 441 442 +1 ``` | [Files](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.09% <100.00%> (-0.11%)` | :arrow_down: | | [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `97.36% <100.00%> (+0.01%)` | :arrow_up: | ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2191/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901768721,