Trying to move docker storage path

  • Hello! I'm running OMV and a few docker containers on a Raspberry Pi 4 8GB. OMV is installed on a 32GB microSD card, but I also have it hooked up to 16TB of pooled drive storage. I've successfully had a few docker containers running, but now I'm running into trouble since my microSD card is filling up. I saw that in the OMV docker settings, I have the option to change the Docker Storage path. I went ahead and changed it to a directory on my pooled drives, and when I reboot my system it lists its status as installed and running. Portainer says "No portainer container found." When I try hitting the install button again, I get: Something went wrong trying to pull and start portainer ...


    When I try to restart docker, I get this over and over again:

    Code
    >>> *************** Error ***************
    Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; echo "Restarting docker ..." && systemctl restart docker.service ': Restarting docker ...
    <<< *************************************

    Not sure what else I need to do to get it running on my external drives. Don't know if this is related, but the drives are encrypted when OMV starts up, so maybe it gets thrown off because it tries to start these services before it's got access to the drives? Not sure.


    Any help would be greatly appreciated!

  • go_niko


    Did you move your old containers to the new container path.. just changing the path in the webUI does not move the containers, just the base path. The racing part (docker starting before your drive is mounted)... is easy enough to resolve..


    First, SSH your server

    systemctl restart docker


    That will restart docker.. if you moved (or reinstalled) all your contaierrs correctly, etc. at the new location, everything should be accessible now

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • If you are running into the "racing" issue, where docker is starting before the drives are mounting... macom provided a brilliantly simple solution to this in this thread...


    Jellyfin portainer stack completely reset after 10-hour server shutdown: why?


    That will basically delay docker starting by 30sec on a reboot, which should be plenty of time for the external storage drives to be mounted, thus making everything work normal

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • Thanks for the reply! I did use rsync to copy everything from /var/lib/docker to /<externaldrive>/docker, and then set the same path in the webUI. I then ran systemctl restart docker and got this error:

    Code
    Job for docker.service failed because the control process exited with error code.
    See "systemctl status docker.service" and "journalctl -xe" for details.

    Any ideas on this? The racing issue sounds like it'll be relevant too, so thanks for sharing that resource as well.

  • Thanks for the reply! I did use rsync to copy everything from /var/lib/docker to /<externaldrive>/docker, and then set the same path in the webUI. I then ran systemctl restart docker and got this error:

    Code
    Job for docker.service failed because the control process exited with error code.
    See "systemctl status docker.service" and "journalctl -xe" for details.

    Any ideas on this? The racing issue sounds like it'll be relevant too, so thanks for sharing that resource as well.

    Hold on, let me fire up a virtual machine and test this.. that should work (or at least it always has before).

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • Hold on, let me fire up a virtual machine and test this.. that should work (or at least it always has before).

    Okay cool, I appreciate that. Not sure what else could be unique about my setup that could be throwing it off.

  • Thanks for the reply! I did use rsync to copy everything from /var/lib/docker to /<externaldrive>/docker, and then set the same path in the webUI. I then ran systemctl restart docker and got this error:

    Code
    Job for docker.service failed because the control process exited with error code.
    See "systemctl status docker.service" and "journalctl -xe" for details.

    Any ideas on this? The racing issue sounds like it'll be relevant too, so thanks for sharing that resource as well.

    OK, not sure what rsync command you ran, but what I said works, although I did fail to leave docker off the end of the path. Assuming you can switch back to /var/lib/docker and everything is working normally (ie, all your containers are still there)


    1. Stop docker and Delete your containers directory from the OMV webUI

    2. Create a new containers directory (or just empty the original, whichever floats your boat)

    Code
    3. cd /var/lib
    4. cp -R docker/ /path/to/Containers

    5. When it's done, the path in the OMV webUI, should be /path/to/Containers/docker

    6. Save/Restart docker


    You should be able to log into Portainer now all of your data is there. Assuming that is the case, you can delete /var/lib/docker now and reclaim that space on your drive.

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • before deleting the /var/lib/docker, you can also verify it is using the correct directory with


    Code
    docker info | grep Root

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • I did that procedure once (changing root of docker on the OMV gui) and the same happened to me (RPi4 4Gb).

    Errors showing "docker not restarting" or "containers missing", etc...


    What I did was, on OMV gui, uninstall docker, set the new path I wanted for the docker root and, Reboot (to clear any cache, db, whatever)


    After that, again on OMV gui, made sure that the path was still there and then install docker and Portainer.

    All containers magically appeared.

  • Thanks for the pointers! I'm in the process of copying everything to the new directory with the commands above:

    After that, first I'll just save/restart, and if that doesn't work I'll uninstall docker from the webUI, set the path, and reinstall it and see what happens. Hopefully all the containers stay intact! I'll post back with updates. Thanks again for the help!

  • The only thing I found weird. Is portainer kept giving weird "end point" notifications.. but it otherwise worked fine... To resolve that.


    Go to the command line (assuming your portainer container is named portainer)


    docker stop portainer

    docker rm portainer


    Then go back to th omv webui and reinstall portainer.


    Thar should resolve that.

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • Unfortunately, I tried all of the above and still have had no luck... after copying the container files to the new directory, changing and saving the new directory in the webUI, fully uninstalling docker, rebooting my system, and reinstalling docker through the webUI, I get this error (with the last 2 lines repeated many times):

  • What is the output of:


    Code
    docker info | grep Root


    Actually it looks like it's telling you docker didn't fully install, so that probably won't tell anything?


    try


    Code
    apt install -f

    and see if it fixes your docker install

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • Indeed, the first command just gave the output: errors pretty printing info

    And unfortunately the second gave this output/error:

    Sorry that this is so difficult... could it be something about my trying to install it on drives pooled by mergerfs that's causing the problem? Not sure what else could be different between the new directory and the original.

  • could it be something about my trying to install it on drives pooled by mergerfs that's causing the problem?

    Do not put your docker storage directory on a mergerfs pool. It introduces another possible race condition and is a poor idea anyway.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 5.x on ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.

  • Do not put your docker storage directory on a mergerfs pool. It introduces another possible race condition and is a poor idea anyway.

    Good to know! In that case, if I want to move it somewhere with more storage space, could I just put it on one of the individual drives that's part of the pool, rather than scattered across the pool itself?

  • It depends on how your drive pool is defined and what create policy is in use whether it will or will not wind up scattered across more than one disk. Mine is on a drive that isn't pooled.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 5.x on ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.

  • It depends on how your drive pool is defined and what create policy is in use whether it will or will not wind up scattered across more than one disk. Mine is on a drive that isn't pooled.

    Looks like that was it! I moved the docker container files to a directory directly on one of the drives rather than the pool itself, and it was easily picked up in the OMV webUI, and my containers all seem to be working. Thanks for the help!

  • go_niko

    Added the Label OMV 5.x
  • go_niko

    Added the Label resolved

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!