Docker Failing to Start When Using Back-up docker Folder

  • Here is the short version of what happened.


    I setup docker to use the default directory /var/lib/docker. It has been running great for months. This past week I noticed serval failures on my docker images. After looking into it, I realized my main partition is full. The main partition is btrfs and attached HDD is xfs. The first thing I did was copy the docker folder to my attached HDD. I then tried a symlink for /var/lib/docker but it didn't work (my guess is it due to the different filesystems). I then went into OMV and created a sharedfolder to the new docker path on the attached HDD. After that was done, I configured docker to use this new folder and then fired it up. It started successfully. After I confirmed it was working, I turned the service off and then copied the contents from the backed up docker folder.


    The issue I have now is that docker will not start when I use the backed up docker folder. It only works when I point it to a blank docker folder.


    I really do not want to have to re-setup all my docker images again. Any idea on what I can do to save it?


    I had Radarr, Sonarr, Lidarr, Transmission, SabNZBD, Tautulli, etc.. As a worse case, is there an easy way to import settings and status to configure them again.


    Note: I noticed the copied docker folder is much larger than the original folder.

  • IIRC, changing the docker base path automatically copies all the files and directories from the existing base path to the new base path. This can take some time. Once this operation is complete docker should restart and be running in the new base path. I do not think that anything is done with the files and directories in the old base path. You will have to delete them yourself after verifying that all your containers are working properly.


    You don't say how you copied the docker folder, but if the permissions and/or ownership are not the same on the copy as they are on the original, then that would explain it not working. Using the Docker GUI to make this change preserves the permissions and ownerships.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

    Einmal editiert, zuletzt von gderf ()

  • I couldn’t do that as there was not enough space to even start docker. I just used cp and waited forever for it to copy. I believe cp keeps the permissions though.


    I’ll look through the logs again but the systemctl status code doesn’t give me anything.

  • Did you compare the permissions and ownerships?

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • Is the drive you moved the Docker Base Path to mounted exec enabled? The default for OMV is to mount data drives noexec.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • I have the noexec flag removed from /etc/openmediavault/config.xml as I needed exec for another app.

    I tried again and received the following from systemctl status docker

    docker.service - Docker Application Container Engine
    Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/docker.service.d
    └─openmediavault-docker-gui.conf, openmediavault.conf
    Active: failed (Result: exit-code) since Thu 2019-07-18 16:48:02 PDT; 10s ago
    Docs: https://docs.docker.com
    Process: 16300 ExecStart=/usr/bin/dockerd -H unix:///var/run/docker.sock $DOCKER_OPTS $OMVDOCKER_IMAGE_PATH (code=exited, status=1/FAILURE)
    Main PID: 16300 (code=exited, status=1/FAILURE)
    CPU: 503ms


    Jul 18 16:47:59 odroidxu4 systemd[1]: docker.service: Unit entered failed state.
    Jul 18 16:47:59 odroidxu4 systemd[1]: docker.service: Failed with result 'exit-code'.
    Jul 18 16:48:02 odroidxu4 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
    Jul 18 16:48:02 odroidxu4 systemd[1]: Stopped Docker Application Container Engine.
    Jul 18 16:48:02 odroidxu4 systemd[1]: docker.service: Start request repeated too quickly.
    Jul 18 16:48:02 odroidxu4 systemd[1]: Failed to start Docker Application Container Engine.
    Jul 18 16:48:02 odroidxu4 systemd[1]: docker.service: Unit entered failed state.
    Jul 18 16:48:02 odroidxu4 systemd[1]: docker.service: Failed with result 'exit-code'.

    Looking at the log it seems to think it is btrfs instead of overlay2. Is there a setting file or something stored in this location that is causing this? I tried looking online, but they only mention how to change it when docker is running.

    level=error msg="[graphdriver] prior storage driver btrfs failed: prerequisites for driver not satisfied (wrong filesystem?)

  • So I found the setting in /etc/docker/daemon.json


    I had to add "storage-driver": "overlay2"


    It will not start (small win), but it is not pulling up my previous docker images. Going to dive into that a bit more.

    • Offizieller Beitrag

    MOving from btrfs to ext4 don't think that will work. Back long time ago I tried something similar and didn't work. Your current running instance of docker best thing you can do is export each container as docker compose yaml files, then set up a clean docker instance on ext4(overlay2) and start running compose for each file. I can't find the tool to do the export at the moment but i do recall discussing it here in the forum. This will also re-download the images but thats the less if you consider re configuring each container. in the shell pretty much you can iterate through all the running containers and export them all in one command

  • Thank you. I just found that out on a docker forum. Looks like you have to do a special save/import to bring it to another filesystem. I just figured out how to mount the remaining space on my SD card and I will be transferring it back onto my btrfs filesystem.

  • Thank you both for you help. I was able to get it back up and running by moving it back into a btrfs filesystem. I guess I need to do some research on how to properly backup my images.

    • Offizieller Beitrag

    here is another tool (not tested) written in nodejs, this can be used maybe in conjunction with composerize. Takes an inspect command and translates it into a docker run line command. From there you can export the Run command line to compose with composerize.


    https://github.com/nexdrew/rekcod


    By the way running stand-alone containers like the omv plugin does is not sustainable and not easy to maintain. I now run a separate vm exclusively for docker, to view containers I use portainer. And modifying parameters done with compose via also the portainer compse section

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!