Question regarding Compose -> Restore

  • How I got here:

    I had around 15 containers running via "compose files", i was quite new to all of this and prefered the "omv approach" of being able to up, down, pull and stop containers via .ymal rather than creating them via gui in portainer. (at some point found out about portainer stacks but was already used to the omv way).

    Long story short, I ran into some container network issues which triggered dark memories so i thought id be best to "practice" the restoring from a back-up. Somehow ran into issues with restoring the .grubparts so after endless tries I settled with just going a fresh install and trying to only restore my docker backup, since that contained most of the configurations.


    So far mostly all good, managed to get omv back into a similar state with all the extras, shares and users from memory. Also managed to correctly map my docker backup folder to my existing backup and when entering the restore page i see all the backups of my containers via names and timestamps.


    Situation:

    When i try to restore a container backup it gives me an error message saying that the container .ymal does not exist. When i manually create the .yaml, omv doesnt error anymore but it prints out that the container data path is missing. When i create that path and try again, it seemly restores without issue, but at this point I could have also just manually restored the files as listed in the vol.list.


    Questions:

    Is this the intended functionality of restore? Meaning that if you would delete a compose file and then try to restore it, it would give you the same error?

    I understand that this is valid way to roll back configuration changes in existing landscape, but was hoping this would also restore full paths if something happens to the folder structure.

    Is there some way to run these restores via CLI and some parameter (like mkdir -p) enabling them to create missing paths?


    While its an manageable task for me to create these 15 container structures manually it would be also nice to know for the future, since i previously naively commented out the docker paths in my omv backup, thinking that my daily docker backups would be more suitable + to overall save storage.


    I will attach error details to the post. Also noticed that the vol.list 0 entry has two forward slashes (0,//docker/compose/name). Not sure why but seemly not an issue if you leave it or remove one. Might be something with the shares, but everything maps correctly so I ignored it.


    For anyone struggling at the same point, these are the manual steps I mentioned:

    1. open vol.list stored in root of container backup via txt editor
    2. create listed paths and or adjust to new paths, either manually or via "mkdir -p" in CLI
    3. move .yaml and .env file into created compose folder (i assume it's always the 0. one, not sure if .env needs to be moved but might as well)
    4. run restore container via compose -> restore
    5. go to compose->files->add->import and select your docker compose directory (folders not .yaml)
    6. *optional* clean out omv comments out of yaml files to fully restore previos state

    Thanks in advance for any feedback or response, sorry if this was posted / answered somewhere before, was not able to find anything referencing this topic.

  • KM0201

    Hat das Thema freigeschaltet.
    • Offizieller Beitrag

    Is this the intended functionality of restore? Meaning that if you would delete a compose file and then try to restore it, it would give you the same error?

    Yes. There are two type of restores that I envisioned. One would be restoring a container to a previous point. The other would be restoring from a catastrophic failure. The latter would involve a different process than you followed. Since the plugin doesn't know about files on the filesystem without database entries, the first thing you should recover is the database entries. This can easily be done with the Import feature. Then you would use the restore functionality to get your container data restored.


    Is there some way to run these restores via CLI and some parameter (like mkdir -p) enabling them to create missing paths?

    There is but if you just switch your steps around to do the import earlier, you wouldn't need to. If you only reinstalled OMV, then the original compose file structure should still be on your data disk.


    While its an manageable task for me to create these 15 container structures manually it would be also nice to know for the future, since i previously naively commented out the docker paths in my omv backup, thinking that my daily docker backups would be more suitable + to overall save storage.

    That is why the import can import all compose files in a directory. That should be one step.


    Also noticed that the vol.list 0 entry has two forward slashes (0,//docker/compose/name). Not sure why but seemly not an issue if you leave it or remove one. Might be something with the shares, but everything maps correctly so I ignored it.

    That will cause no problems in Linux. I assume you created a sharedfolder with sharerootfs and put it on your OS disk. I wouldn't do that for the exact reason you are seeing now - when you reinstall, you lose data.


    Another option you might want to look at is omv-regen. Search the forum for it. This would restore your database on a fresh install and you wouldn't have to even do the import step. Once the database is restored, the plugin will rewrite all of the compose files.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.2 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4 | scripts 7.0


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Thanks for your response! I tried a few things, here are my questions / results:

    Since the plugin doesn't know about files on the filesystem without database entries, the first thing you should recover is the database entries. This can easily be done with the Import feature.

    So your talking about the import i reference in 5. the one at compose->files->import, correct? I tried to import the structure with the vol.list files that was created with that function but it did not do anything. Just tried it again now even going down to each folder but it does not grab it.


    The way I could see a different path would be to import the manually restored folder structure prior to restoring it. But however I turn that is just the same step in a different order.

    ---

    If you only reinstalled OMV, then the original compose file structure should still be on your data disk.

    Yeah, that could have been really easy but i attempted myself at restoring the backup via .grubparts and failed like briefly mentioned, so I at that point already had formatted the drive and was back to 0. I did manage to restore the .fsa file via fsarchiver to another drive but I naively commented out my docker folder from that "omv" backup thinking the compose restore i do daily, instead of the "omv " on every 7, would be enough / better.

    ---

    That is why the import can import all compose files in a directory. That should be one step.

    going back to import topic, this only works on the simple structure created by the compose plugin itself, like so:


    Folder: containername

    • containername.yaml
    • containername.env

    correct? Or is it also supposed to work on the backup created structure which is folders 0 - X + status and vol.list?


    Another option you might want to look at is omv-regen. Search the forum for it. This would restore your database on a fresh install and you wouldn't have to even do the import step. Once the database is restored, the plugin will rewrite all of the compose files.

    Ah, I did stumble over this while deep sea diving for google results while already mid having my backup restore issues. I was following this post and it seemed like that was just the manual way to do it. Since that didn't work for me i just fresh installed. My .fsa didn't have the container structure either way so pretty sure i was doomed either way.


    Reason why I commended it out was because I've been playing around with bigger containers like plex and jellyfin which can can bring some weight (10-15GB) with there metadata structure. Since I backuped via compose daily, I thought it would be smart to keep multiple omv backups which were only 3-4GB with docker data commented out.


    At first I had root level folders for dockerCompose and dockerData which I thought would be smart to combine into docker/compose and docker/data, but i can see now that that was exactly what caused me pain in the end. OFC having the extra few, for me 19kb, for the compose files would have been no issue at all. So thinking about it now, either just comment out the heavy hitters or at least make sure you catch the compose folder. Grated I was in the believed the compose backup would be able to recover from total loss as well.


    This can easily be done with the Import feature.

    Reading your responds again really makes me thing you are referencing a different import function from the one that is within the "ADD" of the compose file section. Can you briefly elaborate on that? Like under what section i can find it and which file it would import.


    Thanks again for taking the time and effort!

    • Offizieller Beitrag

    So your talking about the import i reference in 5. the one at compose->files->import, correct? I tried to import the structure with the vol.list files that was created with that function but it did not do anything. Just tried it again now even going down to each folder but it does not grab it.

    Yes but you want to parent folder not the container folder. So, it would be /docker/compose not /docker/compose/containername/


    Just to make sure it wasn't broken, I deleted all of the compose files on my dev machine directly in the database - 64 of them. I then imported from Files - > Add -> Import, entered the path /srv/dev-disk-by-uuid-f4986fb7-838b-41ab-bc30-cfd22e29a4d1/composeFiles, and I had all 64 right back.


    Then I deleted the database again and imported from backup folder - /srv/dev-disk-by-uuid-f4986fb7-838b-41ab-bc30-cfd22e29a4d1/dockerBackup. All containers were back again. I could now restore them to get the container volumes. So, I can't figure out what you are doing to not get this to work.

    going back to import topic, this only works on the simple structure created by the compose plugin itself, like so:


    Folder: containername

    containername.yaml
    containername.env

    correct?

    Yes. The env file is optional. https://github.com/OpenMediaVa…ned/rpc/compose.inc#L1060

    Grated I was in the believed the compose backup would be able to recover from total loss as well.

    I will see if I can improve the restore but it sure seems to be working well in my opinion.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.2 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4 | scripts 7.0


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    • Offizieller Beitrag

    Another way you can import a yml (but not env file) is to use the Files -> Add from URL option. You can put the full path the yml file - /srv/dev-disk-by-uuid-f4986fb7-838b-41ab-bc30-cfd22e29a4d1/composeFiles/nginx10/nginx10.yml - and it will import it just like it would from an https://domain.com/path/to/test.yml

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.2 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4 | scripts 7.0


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • loadin

    Hat das Label gelöst hinzugefügt.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!