How to regenerate a complete OMV system.

    • Official Post

    ____________________________________________________________________________________________________

    EDIT:


    After much testing it turned out to be necessary to perform some more actions for this to work. So I developed this app that does everything automatically.


    Install: wget -O - https://raw.githubusercontent.com/xhente/omv-regen/master/omv-regen.sh | bash


    ____________________________________________________________________________________________________



    Some time ago someone known to everyone on this forum said:

    Quote

    It would be interesting to re-architect OMV plugins to be saltstack (or better yet - ansible :) ) code instead of .deb packages. Then you could run the "playbook" and it would install the packages and configure everything. That would make things easier.

    It seems that this moment has arrived, OMV6 has already been working with salt for a year. So it occurred to me that thanks to salt maybe a complete OMV system can be recreated from scratch from the database xml file. I have put my hands to work and it has worked for me. I have managed to recreate an existing system from scratch with ease.


    The procedure is simple. In short it consists of doing a fresh installation (update, install plugins, do not configure anything, just install) add existing users, replace the config.xml file, run two commands and reboot.


    When the system boots it will be in the same state as we had it, all the GUI settings we had, file system, permissions, shared folders, services, scheduled jobs, rsync, plugin settings, etc will be in their original state.


    Exceptions and points to consider:

    • It is recommended to have the system as up-to-date as possible before extracting config.xml to copy it to the new system. Otherwise, the versions of some plugins may change and some inconsistencies may appear.
    • All manual root settings outside of the GUI will be gone, as expected. If you have done this you already know what you have to do to recreate it.
    • If you have shared folders in root, they will logically disappear. It is not advised to do this in general anyway.
    • Podman based plugins (Filebrowser, Photoprism, Wetty) will be active and configured in the GUI. But the internal configurations of the container will have been lost.
    • It is necessary to create the users again, they are not in the database. They're in /etc/passwd and the (encrypted) passwords are in /etc/shadow, but you'll end up doing it by hand first.
    • The docker containers will be configured in openmediavault-compose, you only have to start them the first time. If you installed them with Portainer you will have to reinstall them.
    • Some special plugins might lose settings. For example Openmediavault-KVM will lose the configuration of the vm. It can be fixed with backup copies of the xml files. If you had the proxmox kernel you must previously install it with the plugin.I don't know all the plugins so I can't make an exhaustive list here. But in general all will work except for some exceptions.

    I think this could be used for an OMV plugin. It is only necessary to follow the working route. As I see it, a button in the GUI sets a different folder on disk to do the backup (config.xml file, user list and plugin list). Once the system has been reinstalled by the user, there is a button to regenerate the system, you only have to say in which folder the backup is. The system installs the plugins, generates the users and resets the database. It reboots and voila, new system.


    As an example, in the following post is one of the tests I have done.

    • Official Post

    REGENERATION OF AN OMV SYSTEM


    ORIGINAL SYSTEM:


    Original system general configuration


    hostname -> testomv

    Standard configuration -> Time zone, Notification, Power management, Certificates, Scheduled Tasks .....

    Network -> br0 bridge created with fixed IP

    Filesystem -> 1 disk OS and 2 disks Raid1 BTRFS

    Smart active with tests scheduled

    2 shared folder on Raid1 BTRFS -> testfolder1 and testfolder2

    2 users -> testuser1 and testuser2 -> write permissions on testfolder1 and testfolder2 respectively

    Samba -> shared folders testfolder1 and testfolder2 working

    Rsync -> scheduled task copy from testfolder1 to testfolder2

    All container volumes are on the data disks.


    Plugins configured in the original system


    Openmediavault-Filebrowser (podman) -> configured in testfolder1 with certificate and working

    Openmediavault-OMV-Extras -> apparmor patch applied, docker installed and working

    Openmediavault-Compose -> Handbrake and Jellyfin installed and working

    Openmediavault-Symlinks -> 1 symlink created

    Openmediavault-Backup -> configured backup and scheduled task

    Openmediavault-Wetty (podman) -> configured and working


    DATA NEEDED TO REGENERATE THE SYSTEM


    To regenerate this system we need to have the following:


    - The OMV database configuration file, which is in /etc/openmediavault/config.xml -> copy to our PC.

    - A list of existing users on the system (passwords are not required). If there are many you can list them like this:

    Code
    # cat /etc/passwd | grep /bin/sh
    testuser1:x:1000:100::/home/testuser1:/bin/sh
    testuser2:x:1001:100::/home/testuser2:/bin/sh

    - A list of installed plugins. We can get it like this:

    Code
    # dpkg -l | grep openmediavault
    ii  openmediavault                    6.3.5-2                       all          openmediavault - The open network attached storage solution
    ii  openmediavault-backup             6.1                           all          backup plugin for OpenMediaVault.
    ii  openmediavault-compose            6.6.1                         all          OpenMediaVault compose plugin
    ii  openmediavault-filebrowser        6.0.4-1                       all          openmediavault File Browser plugin
    ii  openmediavault-keyring            1.0                           all          GnuPG archive keys of the OpenMediaVault archive
    ii  openmediavault-omvextrasorg       6.1.1                         all          OMV-Extras.org Package Repositories for OpenMediaVault
    ii  openmediavault-sharerootfs        6.0.2-1                       all          openmediavault share root filesystem plugin
    ii  openmediavault-symlinks           6.0.4                         all          OpenMediaVault symlinks plugin
    ii  openmediavault-wetty              6.0.4-1                       all          openmediavault WeTTY (Web + TTY) plugin

    SYSTEM REGENERATION


    - Do a clean install of OMV. Update the system and connect data disks.

    - Install the plugins that the previous system had. You don't need to configure anything at all, just install, apply the apparmor patch if you need it.

    - Create the existing users on the original machine, do it in the same order as the UIDs, 1000 first, 1001 second, 1002 third,... the passwords don't need to be the same. Change the default admin password to one of your own choosing.

    - Replace the existing /etc/openmediavault/config.xml file with the one we had on the original system and run:

    omv-salt stage run prepare

    omv-salt stage run deploy

    - If you don't want to lose the connection just set the original server IP first. If you haven't just wait 2 minutes and restart.

    - Now we should be able to access the OMV GUI on our old IP. We can verify that all the previous system configurations appear.


    RESULT CHECK


    General configuration of the new system


    hostname -> testomv -> OK

    Standard configuration -> Time zone, Notification, Power management, Certificates, Scheduled Tasks ..... -> ALL OK

    Network -> br0 bridge created with fixed IP -> OK

    Filesystem -> 1 disk OS and 2 disks Raid1 BTRFS -> OK

    Smart active with tests scheduled -> OK

    2 shared folder on Raid1 BTRFS -> testfolder1 and testfolder2 -> OK

    2 users -> testuser1 and testuser2 -> write permissions on testfolder1 and testfolder2 respectively -> OK

    Samba -> shared folders testfolder1 and testfolder2 working -> OK

    Rsync -> scheduled task copy from testfolder1 to testfolder2 -> OK


    Plugins configured in the new system


    Openmediavault-Filebrowser (podman) -> configured in testfolder1 with certificate and working -> Working and configured in the GUI but the internal configuration of the container has been lost. You could try to backup the pod configuration.

    Openmediavault-OMV-Extras -> apparmor patch applied, docker installed and working -> OK

    Openmediavault-Compose -> Handbrake and Jellyfin installed and working -> OK You only have to start the containers the first time.

    Openmediavault-Symlinks -> 1 symlink created -> OK

    Openmediavault-Backup -> configured backup and scheduled task -> OK

    Openmediavault-Wetty (podman) -> configured and working -> OK



    CONCLUSION


    If the precautions of the previous post are taken into account, a broken system could easily be regenerated without the need to make complete backups of the system disk. Just update the system, extract the config.xml file and do the rebuild.



    EDIT: The system has evolved. See post 48. RE: How to regenerate a complete OMV system.

    • Official Post

    please make this post sticky

    ^^ Patience, let's see what the wise minds think...

    • Official Post

    - Install the plugins that the previous system had. You don't need to configure anything at all, just install, apply the apparmor patch if you need it.

    I think to make this work more reliable, the OMV version and the plugin version have to be the ones which have been installed when the backup of config.xml was made.

    • Official Post

    hmm... I'm not sure I understand well. You mean this?

    It is recommended to have the system as up-to-date as possible before extracting config.xml to copy it to the new system. Otherwise, the versions of some plugins may change and some inconsistencies may appear.

    Or is there something else?

    • Official Post

    The idea would be to have the system always as up-to-date as possible. It is what we all try to do in general.

    This way you can have a cron job scheduled to backup config.xml and always have it ready to rebuild.

    However, there is also the possibility to install older versions of the plugins on a fresh install, in case you know which one were installed when making the copy of config.xml.

    That is interesting. Where can they be downloaded from?

    • Official Post

    the OMV version and the plugin version have to be the ones which have been installed when the backup of config.xml was made.

    You could add a file with the output of dpkg -l | grep openmediavault to the backup folder where the files are stored. In this way, the versions corresponding to the moment of extraction of config.xml will be known at all times.


    Something similar could be done with the users, although this is less important, it is assumed that they do not change as frequently.

    • Official Post

    You could add a file with the output of dpkg -l | grep openmediavault to the backup folder where the files are stored. In this way, the versions corresponding to the moment of extraction of config.xml will be known at all times.


    Something similar could be done with the users, although this is less important, it is assumed that they do not change as frequently.

    Actually I already mentioned this here in the first post. I did not remember it. I think I made it too long even for me :)

    As I see it, a button in the GUI sets a different folder on disk to do the backup (config.xml file, user list and plugin list).

    macom Do you see any other problem in this method? It is safe? I have tested it on new installations. I'm not sure what would happen in older installations.

    • Official Post

    Do you see any other problem in this method? It is safe? I have tested it on new installations. I'm not sure what would happen in older installations.

    Me not, but would be good if votdev would comment.

    • Official Post

    I already see the problem. Only the latest version of each package is available for installation.

    Code
    # apt-cache madison openmediavault
    openmediavault |    6.3.5-2 | https://packages.openmediavault.org/public shaitan/main amd64 Packages
    # apt-cache madison openmediavault-filebrowser
    openmediavault-filebrowser |    6.0.4-1 | https://packages.openmediavault.org/public shaitan/main amd64 Packages

    I can't find a way to install a package that is not the latest version. Unless there is another way, maybe download it from somewhere and install it manually. But this would complicate things quite a bit.


    Therefore, unless someone provides another solution, this method would only work with full guarantees in those cases where the original system is up to date.


    In other cases the new system will have higher package versions, and this will lead to possible inconsistencies. Everything will depend on the differences between the packages.


    Summarizing. This method is used to rebuild an updated system. This use case can still be very useful. Cases where the system is still operational and can be upgraded. This is very common. Upgrade, extract the database, rebuild the system.


    Someone correct me if I'm wrong.

    • Official Post

    The way I see it, regeneration can be safely applied in the following situations:

    • I want to replace the system disk to SSD, or to USB, or to nvma, or ... Or simply because it does not work correctly or because it is too small.
    • I want to replace the hardware completely and the original installation is not compatible with the new hardware.
    • I want to have a simple OS backup, which also does not take up space and is safe.
    • I want to reinstall the system because it has become unstable and the installed packages are up to date.
    • Others that haven't occurred to me yet...

    These situations are very common in the forum and generate long threads. Regeneration is an effective and efficient solution. votdev I think you should speak on this. "Silence grants, the smile confirms and the look speaks" :)

  • If I was not thinking about a complete rebuild of my OMV system (to fix the old kernel version issue from in-place update omv5->omv6 on RPI4), I would be happy to give this a go.

    I have always found it difficult to believe that there is no method of saving/retrieving settings and the ONLY way to configure OMV is via the GUI.


    In this modern day world of Infrastructure as Code with Terraform and similar structured configuration tools, relying solely on a GUI is limiting.

    • Official Post

    If I was not thinking about a complete rebuild of my OMV system (to fix the old kernel version issue from in-place update omv5->omv6 on RPI4), I would be happy to give this a go.

    If the system is fully upgraded to OMV6 I think it should work. Make sure you have a backup beforehand. Luck!

    • Official Post

    Ah ok, yes, means the same.

    However, there is also the possibility to install older versions of the plugins on a fresh install, in case you know which one were installed when making the copy of config.xml.

    That's what I was wondering. I doubt most know what version of a particular plugin they are using unless they need to reference it for some reason (I know I don't), not to mention making sure all your plugins are reinstalled beforehand, etc.


    Interesting idea though that may make a simple restore possible at some point, just not sure it's there yet. I'm assuming for most the desired procedure is you make a backup, do a new install, upload the backup and it basically installs/configures all plugins per the backup (that's how some other software's handle it).

    • Official Post

    Something simple that anyone can use to back up essential files for a rebuild might be the following. You don't need to make a script for this.


    Just create a daily scheduled task in the OMV GUI and add this command to run:

    Code
    rsync -av /etc/openmediavault/config.xml /[PATH_TO_DATA]/regen_omv/ && dpkg -l | grep openmediavault > /[PATH_TO_DATA]/regen_omv/plugins_list && cat /etc/passwd | grep /bin/sh > /[PATH_TO_DATA]/regen_omv/users_list

    Replace [PATH_TO_DATA] with the path to the folder where you want to store this information.


    This task will make a copy of the database, create a list of installed plugins including their versions, and another list of system users. The information will be updated every day. It's all you need to regenerate the system.


    Warning: Store this data securely, the database contains sensitive information from your server.

    • Official Post

    Edit: There is another more elaborate and complete script later in this thread. RE: How to regenerate a complete OMV system.



    If you have more information on the OS disk that you need to keep, this script may be useful.

    Copy it to a text file and save it with the name regen_omv.sh

    Copy this file to the server's /home folder and make it executable with chmod 700 /home/regen_omv.sh

    Then schedule a daily task in the OMV GUI that runs the command /home/regen_omv.sh

    And that's it, every day the backup data will be updated.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!