Restore after docker system prune --all --volumes --force

  • Hi all, today I made a stupid thing. X/

    I was trying to free up some space, because my directory /var/lib/docker/overlay2/ arrived at 4.2 GB and my Raspbian installation was going out of space. So I launched command "docker system prune --all --volumes --force", which allowed to free up over 700 MB but deleted also a couple of containers which were in stopped status.

    Now I'm looking for a way to recover them, if available.

    I have the system backup, created with OMV backup feature, stored as a mirrored filesystem in another drive. Is there a simple way to recover the missing containers?

    For the future, I suppose I should schedule also a portainer backup... I was now reading about a specific backup procedure for the product which should be more efficient than rsync-ing the source directory.


    I'm using openmediavault 5.6.16-1 (Usul) installed over Raspbian GNU/Linux 10 (buster).


    Thanks in advance for your help!

    • Offizieller Beitrag

    I don't know the procedure to restore that copy.

    One way to avoid this situation is to locate the docker folder on a separate disk. Here is a procedure to do this, if you feel like changing it now.

    [How to] Prepare OMV to install docker applications (beginners)

  • Isn't it as simple as running the docker-compose file again (if that's the way you run a container)?

    docker-compose up -d

    HP t630 Thin Cliënt (AMD Embedded G-Series GX-420GI | QuadCore | 8GB)
    7.0.5-1 (Sandworm) | 64 bit | pve-kernel-6.8 | omvextrasorg 7.0

    • Offizieller Beitrag

    Isn't it as simple as running the docker-compose file again (if that's the way you run a container)?

    docker-compose up -d

    If you've really only lost the volumes, yes, it's that simple. I don't know what he lost exactly.

  • I've lost containers and volumes. Browsing on the backup repository I identified and copied back volumes and containers from mirrored /var/lib/docker, so files are back again. But I still cannot find them in Portainer GUI, so I'm still looking for a way to rescan objects from there.

    Using "docker-compose up -d /var/lib/docker/containers/xxx" will relink them again to docker?

  • I've lost containers and volumes. Browsing on the backup repository I identified and copied back volumes and containers from mirrored /var/lib/docker, so files are back again. But I still cannot find them in Portainer GUI, so I'm still looking for a way to rescan objects from there.

    Using "docker-compose up -d /var/lib/docker/containers/xxx" will relink them again to docker?

    Using docker-compose up -d should link to the location where you stored your .yml files.

    But how do you setup/run your containers , through Portainer or CLI or ...?

    HP t630 Thin Cliënt (AMD Embedded G-Series GX-420GI | QuadCore | 8GB)
    7.0.5-1 (Sandworm) | 64 bit | pve-kernel-6.8 | omvextrasorg 7.0

    Einmal editiert, zuletzt von Frepke ()

    • Offizieller Beitrag

    I used Portainer to setup and run them.

    Did you at least move your containers folder off of your OS drive, to avoid having this issue in the future? I'm assuming your data drives are considerably larger than your OS drive.

  • Then you can run them again from within Portainer.

    The problem is that I have restored folders, but containers don't appear in the list on Portainer and I cannot start them.

    Right now I'm looking for a way to update the list in Portainer to reflect the containers in the filesystem...

  • The problem is that I have restored folders, but containers don't appear in the list on Portainer and I cannot start them.

    Right now I'm looking for a way to update the list in Portainer to reflect the containers in the filesystem...

    Aren't they visible in the Stacks tab in Portainer?

    HP t630 Thin Cliënt (AMD Embedded G-Series GX-420GI | QuadCore | 8GB)
    7.0.5-1 (Sandworm) | 64 bit | pve-kernel-6.8 | omvextrasorg 7.0

  • No Stacks tab contains an empty list.

    Sorry, I can't help you with recovering a backup :(

    The easiest way for me should be to setup the Stacks in Portainer from scratch (hope you have a backup from the settings) or setup everything with .yml files (those textfiles are easy to duplicate for a backup).


    Maybe (and I hope) someone else can help you recovering you lost containers.

    HP t630 Thin Cliënt (AMD Embedded G-Series GX-420GI | QuadCore | 8GB)
    7.0.5-1 (Sandworm) | 64 bit | pve-kernel-6.8 | omvextrasorg 7.0

    • Offizieller Beitrag

    Sorry, I can't help you with recovering a backup :(

    The easiest way for me should be to setup the Stacks in Portainer from scratch (hope you have a backup from the settings) or setup everything with .yml files (those textfiles are easy to duplicate for a backup).


    Maybe (and I hope) someone else can help you recovering you lost containers.

    I'm in the same situation.

  • So I launched command "docker system prune --all --volumes --force",

    Not trying to twist the knife but from where did you read to do this?

    The only "prune" that (sometimes) need to be done is docker image prune to clear the leftovers of old images.


    Volumes are binded to make sure they're persistent.


    But, that you NOW know, ;)



    To try to solve your problem, since you have a backup:

    I identified and copied back volumes and containers from mirrored /var/lib/docker, so files are back again

    This is valid ONLY if you had your volumes under the docker root (/var/lib/docker).

    Your approach is correct: copy all from "backup/var/lib/docker" to "/var/lib/docker" and your volumes/containers will be back in place.


    Now the issue is:

    Depending on how you edited the STACK before deploying, your volumes might be in a different path.


    For e.g:

    Code
        volumes:
          - /srv/dev-disk-by-label-DATA/appdata/mosquitto/config:/mosquitto/config/

    This volume is under /srv/dev-disk-by-label-DATA/appdata/mosquitto/config and NOT under /var/lib/docker/....


    If deleted, even if I restore the docker root, ALL the configuration of the Container would be lost.


    And when you ran that command, all volumes that were binded (and stopped), were deleted. (even if on a different path)


    Prune unused Docker objects | Docker Documentation



    If you have a copy of the YML (or the STACK) that you used to launch them in the beginning, that will might help to, at least, check if the DATA is still available.


    Otherwise, you'll have to start blank.

  • I think you lost is the portaner volume which contains your stacks.

    Now it depends, what the other containers were if you can recover.


    1. First of all: go to the cli and stop the docker daemon systemctl stop docker
    2. Next save the /var/lib/docker directory to some place and remember to keep the flafs / permissions
    3. from you backup find the directory .../docker/volumes/portainer_data If you cant't find it you are out of luck if you find it continue
    4. delete the directory /var/lib/docker/volumes/portainer_data (the original one, not the backup)
    5. copy the directory back from your backup
    6. restart the docker daemon systemctl start docker
    7. check if the volume is found docker volume ls
    8. report back here and we will see, if we can continue


    I wonder why the docker volume disappeared, because think you had it running.

    If you got help in the forum and want to give something back to the project click here (omv) or here (scroll down) (plugins) and write up your solution for others.

  • Hi Soma, I have read this command in a Stackoverflow answer. It was related to the question on how to free up some space when directory /var/lib/docker/overlay2 becomes huge. Now I know that I should find another way ;)


    Thanks to all for all the suggestions. At the end, I have used config files (config.v2.json) located on the container's directory from backup as a reference to recreate containers from blank. Now have the missing containers running.


    Thanks again!

  • liv

    Hat das Label gelöst hinzugefügt.
  • Soma ist not quite correct about the deletion of volumes.


    In the example above

    Code
        volumes:
          - /srv/dev-disk-by-label-DATA/appdata/mosquitto/config:/mosquitto/config/

    hhis is not volume in the docker terminology and will not be deleted by docker volume prune.

    docker volume prune will only delete named volumes.

    If you got help in the forum and want to give something back to the project click here (omv) or here (scroll down) (plugins) and write up your solution for others.

  • will not be deleted by docker volume prune.

    docker volume prune will only delete named volumes.

    You might be right, :)

    Haven't read all docker documentation...............yet, :D



    But, honestly, running a command with arguments "--all" and "--force" is a "invitation to disaster".

    When I read it, I only remembered the "good-old-days" of friends of mine running "format c: /q" on a DOS window.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!