issue_comments

This is now released. pip install datasette==0.43 should get the fix.

Thanks for helping test this @aborruso!

OK, here's a new branch you can try. Install it like this:

pip install https://github.com/simonw/datasette/archive/shutil-backport.zip

If it works for you I'll merge that branch into master.

Oh no! It looks like dirs_exist_ok is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.

What version of Python are you running?

I think this is the fix! I just landed it in master.

@aborruso you can install Datasette master to test it out like this:

pip install https://github.com/simonw/datasette/archive/cee671a58f417f827d1735b1abaa40716534ea67.zip

Actually maybe the answer here is to use dirs_exist_ok=True when calling shutil.copytree.

I just hit this bug myself, or at least a variant of it!

https://github.com/simonw/museums/runs/697063068?check_suite_focus=true

2020-05-21T17:14:02.2596471Z Traceback (most recent call last): 2020-05-21T17:14:02.2599146Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py", line 605, in link_or_copy_directory 2020-05-21T17:14:02.2599861Z shutil.copytree(src, dst, copy_function=os.link) 2020-05-21T17:14:02.2600377Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 554, in copytree 2020-05-21T17:14:02.2600786Z return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks, 2020-05-21T17:14:02.2601196Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 510, in _copytree 2020-05-21T17:14:02.2601580Z raise Error(errors) 2020-05-21T17:14:02.2604565Z shutil.Error: [('/home/runner/work/museums/museums/templates/row-browse-museums.html', '/tmp/tmpq47xi96y/datasette/templates/row-browse-museums.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/row-browse-museums.html' -> '/tmp/tmpq47xi96y/datasette/templates/row-browse-museums.html'"), ('/home/runner/work/museums/museums/templates/index.html', '/tmp/tmpq47xi96y/datasette/templates/index.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/index.html' -> '/tmp/tmpq47xi96y/datasette/templates/index.html'"), ('/home/runner/work/museums/museums/templates/query-browse-search.html', '/tmp/tmpq47xi96y/datasette/templates/query-browse-search.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/query-browse-search.html' -> '/tmp/tmpq47xi96y/datasette/templates/query-browse-search.html'"), ('/home/runner/work/museums/museums/templates/_analytics.html', '/tmp/tmpq47xi96y/datasette/templates/_analytics.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/_analytics.html' -> '/tmp/tmpq47xi96y/datasette/templates/_analytics.html'"), ('/home/runner/work/museums/museums/templates/_museum_card.html', '/tmp/tmpq47xi96y/datasette/templates/_museum_card.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/_museum_card.html' -> '/tmp/tmpq47xi96y/datasette/templates/_museum_card.html'"), ('/home/runner/work/museums/museums/templates/pages/about.html', '/tmp/tmpq47xi96y/datasette/templates/pages/about.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/pages/about.html' -> '/tmp/tmpq47xi96y/datasette/templates/pages/about.html'"), ('/home/runner/work/museums/museums/templates/pages/map.html', '/tmp/tmpq47xi96y/datasette/templates/pages/map.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/pages/map.html' -> '/tmp/tmpq47xi96y/datasette/templates/pages/map.html'")] 2020-05-21T17:14:02.2605437Z 2020-05-21T17:14:02.2605797Z During handling of the above exception, another exception occurred: 2020-05-21T17:14:02.2606102Z 2020-05-21T17:14:02.2606423Z Traceback (most recent call last): 2020-05-21T17:14:02.2606817Z File "/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette", line 8, in <module> 2020-05-21T17:14:02.2607189Z sys.exit(cli()) 2020-05-21T17:14:02.2607907Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__ 2020-05-21T17:14:02.2608347Z return self.main(*args, **kwargs) 2020-05-21T17:14:02.2609024Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 782, in main 2020-05-21T17:14:02.2609451Z rv = self.invoke(ctx) 2020-05-21T17:14:02.2610116Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke 2020-05-21T17:14:02.2610550Z return _process_result(sub_ctx.command.invoke(sub_ctx)) 2020-05-21T17:14:02.2611451Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke 2020-05-21T17:14:02.2611989Z return _process_result(sub_ctx.command.invoke(sub_ctx)) 2020-05-21T17:14:02.2612682Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke 2020-05-21T17:14:02.2613117Z return ctx.invoke(self.callback, **ctx.params) 2020-05-21T17:14:02.2613801Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke 2020-05-21T17:14:02.2614235Z return callback(*args, **kwargs) 2020-05-21T17:14:02.2614946Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/cloudrun.py", line 111, in cloudrun 2020-05-21T17:14:02.2615371Z with temporary_docker_directory( 2020-05-21T17:14:02.2615769Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/contextlib.py", line 113, in __enter__ 2020-05-21T17:14:02.2616135Z return next(self.gen) 2020-05-21T17:14:02.2616957Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py", line 392, in temporary_docker_directory 2020-05-21T17:14:02.2617403Z link_or_copy_directory( 2020-05-21T17:14:02.2618298Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py", line 607, in link_or_copy_directory 2020-05-21T17:14:02.2619270Z shutil.copytree(src, dst) 2020-05-21T17:14:02.2619872Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 554, in copytree 2020-05-21T17:14:02.2622179Z return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks, 2020-05-21T17:14:02.2622609Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 455, in _copytree 2020-05-21T17:14:02.2622878Z os.makedirs(dst, exist_ok=dirs_exist_ok) 2020-05-21T17:14:02.2623157Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/os.py", line 223, in makedirs 2020-05-21T17:14:02.2623405Z mkdir(name, mode) 2020-05-21T17:14:02.2623978Z FileExistsError: [Errno 17] File exists: '/tmp/tmpq47xi96y/datasette/templates'

I have a Windows 10 gaming PC in the house: what steps can I take to get an environment in there that matches yours? I've not installed Python on Windows before.

Have a try and let me know what happens! I'd like this stuff to just work on Windows but I need to figure out how to get an environment working.

What operating system are you using? I need to figure out a way to replicate this bug.

@aborruso I think I have a branch with a fix - could you try it out?

Install the new branch like this:

pip install https://github.com/simonw/datasette/archive/issue-744.zip Then try running datasette publish heroku again.

If this fixes your bug I'll merge that pull request!

This exception handling was introduced in #141

https://docs.python.org/3/library/shutil.html#shutil.copytree says that shutil.Error should be raised here, so I think this is a bug in the link_or_copy_directory function.

I don't have an environment that can replicate this bug. @aborruso I'm going to ship a fix in a branch that you can test against.

Is it possible that your /tmp directory is on a different volume from the template folder? That could cause a problem with the symlinks.

Here's the code in question:

https://github.com/simonw/datasette/blob/89c4ddd4828623888e91a1d2cb396cba12d4e7b4/datasette/utils/init.py#L595-L609

It looks to me like we were expecting an OSError but in your environment you got a shutil.Error instead. So a fix could be to capture that error as well.