issue_comments

Wow, I'm in some way very proud!

This is now released. pip install datasette==0.43 should get the fix.

Thanks for helping test this @aborruso!

Dear @simonw thank you for your time, now IT WORKS!!!

I hope that this edit to datasette code is not for an exceptional case (my PC configuration) and that it will be useful to other users.

Thank you again!!

OK, here's a new branch you can try. Install it like this:

pip install https://github.com/simonw/datasette/archive/shutil-backport.zip

If it works for you I'll merge that branch into master.

Oh no! It looks like dirs_exist_ok is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.

Thank you very much

Oh no! It looks like dirs_exist_ok is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.

What version of Python are you running?

@simonw now I have

Traceback (most recent call last): File "/home/aborruso/.local/bin/datasette", line 8, in <module> sys.exit(cli()) File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 103, in heroku extra_metadata, File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__ return next(self.gen) File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 191, in temporary_heroku_directory os.path.join(tmp.name, "templates"), File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 605, in link_or_copy_directory shutil.copytree(src, dst, copy_function=os.link, dirs_exist_ok=True) TypeError: copytree() got an unexpected keyword argument 'dirs_exist_ok'

Do I must open a new issue?

Thank you

Thank you very much!!

I will try and I write you here

I think this is the fix! I just landed it in master.

@aborruso you can install Datasette master to test it out like this:

pip install https://github.com/simonw/datasette/archive/cee671a58f417f827d1735b1abaa40716534ea67.zip

@simonw can I test it know? What I must do to update it?

Thank you

Actually maybe the answer here is to use dirs_exist_ok=True when calling shutil.copytree.

I just hit this bug myself, or at least a variant of it!

https://github.com/simonw/museums/runs/697063068?check_suite_focus=true

2020-05-21T17:14:02.2596471Z Traceback (most recent call last): 2020-05-21T17:14:02.2599146Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py", line 605, in link_or_copy_directory 2020-05-21T17:14:02.2599861Z shutil.copytree(src, dst, copy_function=os.link) 2020-05-21T17:14:02.2600377Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 554, in copytree 2020-05-21T17:14:02.2600786Z return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks, 2020-05-21T17:14:02.2601196Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 510, in _copytree 2020-05-21T17:14:02.2601580Z raise Error(errors) 2020-05-21T17:14:02.2604565Z shutil.Error: [('/home/runner/work/museums/museums/templates/row-browse-museums.html', '/tmp/tmpq47xi96y/datasette/templates/row-browse-museums.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/row-browse-museums.html' -> '/tmp/tmpq47xi96y/datasette/templates/row-browse-museums.html'"), ('/home/runner/work/museums/museums/templates/index.html', '/tmp/tmpq47xi96y/datasette/templates/index.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/index.html' -> '/tmp/tmpq47xi96y/datasette/templates/index.html'"), ('/home/runner/work/museums/museums/templates/query-browse-search.html', '/tmp/tmpq47xi96y/datasette/templates/query-browse-search.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/query-browse-search.html' -> '/tmp/tmpq47xi96y/datasette/templates/query-browse-search.html'"), ('/home/runner/work/museums/museums/templates/_analytics.html', '/tmp/tmpq47xi96y/datasette/templates/_analytics.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/_analytics.html' -> '/tmp/tmpq47xi96y/datasette/templates/_analytics.html'"), ('/home/runner/work/museums/museums/templates/_museum_card.html', '/tmp/tmpq47xi96y/datasette/templates/_museum_card.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/_museum_card.html' -> '/tmp/tmpq47xi96y/datasette/templates/_museum_card.html'"), ('/home/runner/work/museums/museums/templates/pages/about.html', '/tmp/tmpq47xi96y/datasette/templates/pages/about.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/pages/about.html' -> '/tmp/tmpq47xi96y/datasette/templates/pages/about.html'"), ('/home/runner/work/museums/museums/templates/pages/map.html', '/tmp/tmpq47xi96y/datasette/templates/pages/map.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/pages/map.html' -> '/tmp/tmpq47xi96y/datasette/templates/pages/map.html'")] 2020-05-21T17:14:02.2605437Z 2020-05-21T17:14:02.2605797Z During handling of the above exception, another exception occurred: 2020-05-21T17:14:02.2606102Z 2020-05-21T17:14:02.2606423Z Traceback (most recent call last): 2020-05-21T17:14:02.2606817Z File "/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette", line 8, in <module> 2020-05-21T17:14:02.2607189Z sys.exit(cli()) 2020-05-21T17:14:02.2607907Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__ 2020-05-21T17:14:02.2608347Z return self.main(*args, **kwargs) 2020-05-21T17:14:02.2609024Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 782, in main 2020-05-21T17:14:02.2609451Z rv = self.invoke(ctx) 2020-05-21T17:14:02.2610116Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke 2020-05-21T17:14:02.2610550Z return _process_result(sub_ctx.command.invoke(sub_ctx)) 2020-05-21T17:14:02.2611451Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke 2020-05-21T17:14:02.2611989Z return _process_result(sub_ctx.command.invoke(sub_ctx)) 2020-05-21T17:14:02.2612682Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke 2020-05-21T17:14:02.2613117Z return ctx.invoke(self.callback, **ctx.params) 2020-05-21T17:14:02.2613801Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke 2020-05-21T17:14:02.2614235Z return callback(*args, **kwargs) 2020-05-21T17:14:02.2614946Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/cloudrun.py", line 111, in cloudrun 2020-05-21T17:14:02.2615371Z with temporary_docker_directory( 2020-05-21T17:14:02.2615769Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/contextlib.py", line 113, in __enter__ 2020-05-21T17:14:02.2616135Z return next(self.gen) 2020-05-21T17:14:02.2616957Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py", line 392, in temporary_docker_directory 2020-05-21T17:14:02.2617403Z link_or_copy_directory( 2020-05-21T17:14:02.2618298Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py", line 607, in link_or_copy_directory 2020-05-21T17:14:02.2619270Z shutil.copytree(src, dst) 2020-05-21T17:14:02.2619872Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 554, in copytree 2020-05-21T17:14:02.2622179Z return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks, 2020-05-21T17:14:02.2622609Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 455, in _copytree 2020-05-21T17:14:02.2622878Z os.makedirs(dst, exist_ok=dirs_exist_ok) 2020-05-21T17:14:02.2623157Z File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/os.py", line 223, in makedirs 2020-05-21T17:14:02.2623405Z mkdir(name, mode) 2020-05-21T17:14:02.2623978Z FileExistsError: [Errno 17] File exists: '/tmp/tmpq47xi96y/datasette/templates'

I have installed heroku plugins:install heroku-builds, but I have the same error.

Then I have removed from datasette\publish\heroku.py

python # Check for heroku-builds plugin plugins = [ line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines() ] if b"heroku-builds" not in plugins: click.echo( "Publishing to Heroku requires the heroku-builds plugin to be installed." ) click.confirm( "Install it? (this will run `heroku plugins:install heroku-builds`)", abort=True, ) call(["heroku", "plugins:install", "heroku-builds"])

And now I have

``` Traceback (most recent call last): File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 210, in temporary_heroku_directory yield File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 96, in heroku list_output = check_output(["heroku", "apps:list", "--json"]).decode( File "c:\python37\lib\subprocess.py", line 395, in check_output kwargs).stdout File "c:\python37\lib\subprocess.py", line 472, in run with Popen(*popenargs, kwargs) as process: File "c:\python37\lib\subprocess.py", line 775, in init restore_signals, start_new_session) File "c:\python37\lib\subprocess.py", line 1178, in _execute_child startupinfo) FileNotFoundError: [WinError 2] The specified file could not be found

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "c:\python37\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "c:\python37\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\aborr\AppData\Roaming\Python\Python37\Scripts\datasette.exe__main__.py", line 9, in <module> File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 829, in call return self.main(args, kwargs) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 782, in main rv = self.invoke(ctx) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 610, in invoke return callback(args, **kwargs) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 120, in heroku call(["heroku", "builds:create", "-a", app_name, "--include-vcs-ignore"]) File "c:\python37\lib\contextlib.py", line 130, in exit self.gen.throw(type, value, traceback) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 213, in temporary_heroku_directory tmp.cleanup() File "c:\python37\lib\tempfile.py", line 809, in cleanup _shutil.rmtree(self.name) File "c:\python37\lib\shutil.py", line 513, in rmtree return _rmtree_unsafe(path, onerror) File "c:\python37\lib\shutil.py", line 401, in _rmtree_unsafe onerror(os.rmdir, path, sys.exc_info()) File "c:\python37\lib\shutil.py", line 399, in _rmtree_unsafe os.rmdir(path) PermissionError: [WinError 32] Unable to access file. The file is being used by another process: 'C:\Users\aborr\AppData\Local\Temp\tmpkcxy8i_q' ```

In Windows I'm not very strong. I use debian (inside WSL).

However these are the possible steps:

• I have installed Python 3 for win (I have 3.7.3);
• I have installed heroku cli for win64 and logged in;
• I have installed datasette running python -m pip install --upgrade --user datasette.

It's a very basic Python env that I do not use. This time only to reach my goal: try to publish using custom template

I have a Windows 10 gaming PC in the house: what steps can I take to get an environment in there that matches yours? I've not installed Python on Windows before.

@simonw another error starting from Windows.

I run

datasette publish heroku -n comunepa --template-dir template commissioniComunePalermo.db

And I have

Traceback (most recent call last): File "c:\python37\lib\runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "c:\python37\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\aborr\AppData\Roaming\Python\Python37\Scripts\datasette.exe\__main__.py", line 9, in <module> File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 829, in __call__ return self.main(*args, **kwargs) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 782, in main rv = self.invoke(ctx) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 610, in invoke return callback(*args, **kwargs) File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 53, in heroku line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines() File "c:\python37\lib\subprocess.py", line 395, in check_output **kwargs).stdout File "c:\python37\lib\subprocess.py", line 472, in run with Popen(*popenargs, **kwargs) as process: File "c:\python37\lib\subprocess.py", line 775, in __init__ restore_signals, start_new_session) File "c:\python37\lib\subprocess.py", line 1178, in _execute_child startupinfo) FileNotFoundError: [WinError 2] The specified file could not be found

files.zip

Have a try and let me know what happens! I'd like this stuff to just work on Windows but I need to figure out how to get an environment working.

Hi @simonw probably I could try to do it in Python for windows. I do not like to do these things in win enviroment.

Because probably WSL Linux env (in which I do a lot of great things) is not an environment that will be tested for datasette.

In win I shouldn't have any problems. Am I right?

Hi @simonw it's debian as Windows Subsystem for Linux

PRETTY_NAME="Pengwin" NAME="Pengwin" VERSION_ID="10" VERSION="10 (buster)" ID=debian ID_LIKE=debian HOME_URL="https://github.com/whitewaterfoundry/Pengwin" SUPPORT_URL="https://github.com/whitewaterfoundry/Pengwin" BUG_REPORT_URL="https://github.com/whitewaterfoundry/Pengwin" VERSION_CODENAME=buster

What operating system are you using? I need to figure out a way to replicate this bug.

A stupid note: I have no tmpcqv_1i5d folder in in /tmp.

It seems to me that it does not create any /tmp/tmpcqv_1i5d/templates folder (or other name folder, inside /tmp)

Hi @simonw , I have installed it and I have the below errors.

Is it possible that your /tmp directory is on a different volume from the template folder? That could cause a problem with the symlinks.

No, /tmp folder is in the same volume.

Thank you

``` Traceback (most recent call last): File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/init.py", line 607, in link_or_copy_directory shutil.copytree(src, dst, copy_function=os.link) File "/usr/lib/python3.7/shutil.py", line 365, in copytree raise Error(errors) shutil.Error: [('/var/youtubeComunePalermo/processing/./template/base.html', '/tmp/tmpcqv_1i5d/templates/base.html', "[Errno 18] Invalid cross-device link: '/var/youtubeComunePalermo/processing/./template/base.html' -> '/tmp/tmpcqv_1i5d/templates/base.html'"), ('/var/youtubeComunePalermo/processing/./template/index.html', '/tmp/tmpcqv_1i5d/templates/index.html', "[Errno 18] Invalid cross-device link: '/var/youtubeComunePalermo/processing/./template/index.html' -> '/tmp/tmpcqv_1i5d/templates/index.html'")]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/aborruso/.local/bin/datasette", line 8, in <module> sys.exit(cli()) File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke return callback(args, **kwargs) File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 103, in heroku extra_metadata, File "/usr/lib/python3.7/contextlib.py", line 112, in enter return next(self.gen) File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 191, in temporary_heroku_directory os.path.join(tmp.name, "templates"), File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/init.py", line 609, in link_or_copy_directory shutil.copytree(src, dst) File "/usr/lib/python3.7/shutil.py", line 321, in copytree os.makedirs(dst) File "/usr/lib/python3.7/os.py", line 221, in makedirs mkdir(name, mode) FileExistsError: [Errno 17] File exists: '/tmp/tmpcqv_1i5d/templates' ```

@aborruso I think I have a branch with a fix - could you try it out?

Install the new branch like this:

pip install https://github.com/simonw/datasette/archive/issue-744.zip Then try running datasette publish heroku again.

If this fixes your bug I'll merge that pull request!

This exception handling was introduced in #141

https://docs.python.org/3/library/shutil.html#shutil.copytree says that shutil.Error should be raised here, so I think this is a bug in the link_or_copy_directory function.

I don't have an environment that can replicate this bug. @aborruso I'm going to ship a fix in a branch that you can test against.

Is it possible that your /tmp directory is on a different volume from the template folder? That could cause a problem with the symlinks.

Here's the code in question:

https://github.com/simonw/datasette/blob/89c4ddd4828623888e91a1d2cb396cba12d4e7b4/datasette/utils/init.py#L595-L609

It looks to me like we were expecting an OSError but in your environment you got a shutil.Error instead. So a fix could be to capture that error as well.