Trying to move docker storage path

Hello! I'm running OMV and a few docker containers on a Raspberry Pi 4 8GB. OMV is installed on a 32GB microSD card, but I also have it hooked up to 16TB of pooled drive storage. I've successfully had a few docker containers running, but now I'm running into trouble since my microSD card is filling up. I saw that in the OMV docker settings, I have the option to change the Docker Storage path. I went ahead and changed it to a directory on my pooled drives, and when I reboot my system it lists its status as installed and running. Portainer says "No portainer container found." When I try hitting the install button again, I get: Something went wrong trying to pull and start portainer ...

When I try to restart docker, I get this over and over again:

>>> *************** Error ***************
Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; echo "Restarting docker ..." && systemctl restart docker.service ': Restarting docker ...
<<< *************************************

Not sure what else I need to do to get it running on my external drives. Don't know if this is related, but the drives are encrypted when OMV starts up, so maybe it gets thrown off because it tries to start these services before it's got access to the drives? Not sure.

Any help would be greatly appreciated!

Zitat von KM0201

go_niko

Did you move your old containers to the new container path.. just changing the path in the webUI does not move the containers, just the base path. The racing part (docker starting before your drive is mounted)... is easy enough to resolve..

First, SSH your server

systemctl restart docker

That will restart docker.. if you moved (or reinstalled) all your contaierrs correctly, etc. at the new location, everything should be accessible now

Alles anzeigen

Thanks for the reply! I did use rsync to copy everything from /var/lib/docker to /<externaldrive>/docker, and then set the same path in the webUI. I then ran systemctl restart docker and got this error:

Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

Any ideas on this? The racing issue sounds like it'll be relevant too, so thanks for sharing that resource as well.

Zitat von KM0201

Hold on, let me fire up a virtual machine and test this.. that should work (or at least it always has before).

Okay cool, I appreciate that. Not sure what else could be unique about my setup that could be throwing it off.

Zitat von Soma

I did that procedure once (changing root of docker on the OMV gui) and the same happened to me (RPi4 4Gb).

Errors showing "docker not restarting" or "containers missing", etc...

What I did was, on OMV gui, uninstall docker, set the new path I wanted for the docker root and, Reboot (to clear any cache, db, whatever)

After that, again on OMV gui, made sure that the path was still there and then install docker and Portainer.

All containers magically appeared.

Alles anzeigen

Thanks for the pointers! I'm in the process of copying everything to the new directory with the commands above:

Zitat von KM0201

OK, not sure what rsync command you ran, but what I said works, although I did fail to leave docker off the end of the path. Assuming you can switch back to /var/lib/docker and everything is working normally (ie, all your containers are still there)

1. Stop docker and Delete your containers directory from the OMV webUI

2. Create a new containers directory (or just empty the original, whichever floats your boat)

Code

3.  cd /var/lib
4.  cp -R docker/ /path/to/Containers

5. When it's done, the path in the OMV webUI, should be /path/to/Containers/docker

6. Save/Restart docker

You should be able to log into Portainer now all of your data is there. Assuming that is the case, you can delete /var/lib/docker now and reclaim that space on your drive.

Alles anzeigen

After that, first I'll just save/restart, and if that doesn't work I'll uninstall docker from the webUI, set the path, and reinstall it and see what happens. Hopefully all the containers stay intact! I'll post back with updates. Thanks again for the help!

Unfortunately, I tried all of the above and still have had no luck... after copying the container files to the new directory, changing and saving the new directory in the webUI, fully uninstalling docker, rebooting my system, and reinstalling docker through the webUI, I get this error (with the last 2 lines repeated many times):

Failed to install docker.
Try fixing install...
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up docker-ce (5:20.10.6~3-0~raspbian-buster) ...
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2021-05-06 20:16:44 EDT; 22ms ago
Docs: https://docs.docker.com
Process: 4472 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
Main PID: 4472 (code=exited, status=1/FAILURE)
dpkg: error processing package docker-ce (--configure):
installed docker-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
docker-ce


>>> *************** Error ***************


<<< *************************************

Zitat von KM0201

What is the output of:

Code

docker info | grep Root

Actually it looks like it's telling you docker didn't fully install, so that probably won't tell anything?

try

Code

apt install -f

and see if it fixes your docker install

Alles anzeigen

Indeed, the first command just gave the output: errors pretty printing info

And unfortunately the second gave this output/error:

Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up docker-ce (5:20.10.6~3-0~raspbian-buster) ...
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Fri 2021-05-07 08:59:43 EDT; 26ms ago
Docs: https://docs.docker.com
Process: 24202 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
Main PID: 24202 (code=exited, status=1/FAILURE)
dpkg: error processing package docker-ce (--configure):
installed docker-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)

Sorry that this is so difficult... could it be something about my trying to install it on drives pooled by mergerfs that's causing the problem? Not sure what else could be different between the new directory and the original.

Zitat von gderf

Do not put your docker storage directory on a mergerfs pool. It introduces another possible race condition and is a poor idea anyway.

Good to know! In that case, if I want to move it somewhere with more storage space, could I just put it on one of the individual drives that's part of the pool, rather than scattered across the pool itself?

Zitat von gderf

It depends on how your drive pool is defined and what create policy is in use whether it will or will not wind up scattered across more than one disk. Mine is on a drive that isn't pooled.

Looks like that was it! I moved the docker container files to a directory directly on one of the drives rather than the pool itself, and it was easily picked up in the OMV webUI, and my containers all seem to be working. Thanks for the help!