Migration from 7 to 8 and issues with ZFS

    Quote from ryecoaaron

    I released a new version of omv-extras 7.0.6 that fixed the upgrade on my test system. I was able to reproduce the failure before that.

    i have run into the very same issue as the OP by running omv-release-upgrade. i too have zfs-dkms installed and building the kernel module for kernel 6.17 failed, so i could boot only the 6.11 bookworm kernel. for now i have just reverted to version 7 and did not try any fixes, mostly because i realized that i have too many kernels installed, that are just prolonging the process.


    also i am not sure if i should entirely remove dkms, since i need it (probably) for the nvidia drivers and gpu acceleration. anyhow my understanding from the this thread was that omv-extras 7.0.6 should handle the update fine even with zfs-dkms

    SuperMicro CSE-825, X11SSH-F, Xeon E3-1240v6, 64 GB ECC RAM, LSI 9211-8i HBA controller, 6x HGST 12 TB ZFS

    Powered by Proxmox VE

    • Official Post
    Quote from molnart

    i have run into the very same issue as the OP by running omv-release-upgrade. i too have zfs-dkms installed and building the kernel module for kernel 6.17 failed, so i could boot only the 6.11 bookworm kernel. for now i have just reverted to version 7 and did not try any fixes, mostly because i realized that i have too many kernels installed, that are just prolonging the process.


    also i am not sure if i should entirely remove dkms, since i need it (probably) for the nvidia drivers and gpu acceleration. anyhow my understanding from the this thread was that omv-extras 7.0.6 should handle the update fine even with zfs-dkms

    No reason to revert to 7. You just need to get your many modules compiled on a kernel. SInce you are using zfs and nvidia, you can't get rid of dkms.


    omv-extras doesn't do anything to handle the zfs-dkms compiling issue and it won't. I made changes to handle proxmox kernel upgrades better. If you aren't using the proxmox kernel, then you will still have to fix the modules manually. It would be a nightmare to try to handle every module combination and I'm not going to try.

    omv 8.0.8-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.0.2 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    i have reverted because it was 3 AM and needed a functional OMV setup and restoring a snapshot took like 10 seconds. will give it another try today and try to start earlier ;)

    SuperMicro CSE-825, X11SSH-F, Xeon E3-1240v6, 64 GB ECC RAM, LSI 9211-8i HBA controller, 6x HGST 12 TB ZFS

    Powered by Proxmox VE

    i have spent at least 6 hours trying to update to OMV 8, with no luck so far.


    one thing that caused extra annoyance was the presence of kernel headers even after removing the kernels itself via the OMV Web UI. This has caused kernel modules to be rebuilt at basically every step, with each rebuild taking 20-30 minutes for the many headers installed - (and the modules get rebuilt even when the kernels are being removed with apt remove).


    the issue with the update is, that PVE kernel 6.17 which comes with OMV 8 is incompatibile with zfs-dkms 2.3.2

    i have tried manually installing kernel and headers 6.14, also trying to remove zfs-dkms post upgrade (since its still stuck at 2.3.2 after the update) but i still could not log in into web ui and the whole zfs service was down.

    if i try to remove zfs-dkms before the upgrade, it wants to remove the zfs plugin as well. also apt autoremove is not removing the packages marked as no longer required.

    SuperMicro CSE-825, X11SSH-F, Xeon E3-1240v6, 64 GB ECC RAM, LSI 9211-8i HBA controller, 6x HGST 12 TB ZFS

    Powered by Proxmox VE

    • Official Post

    I would install the proxmox kernel and remove all other kernels and zfs-dkms before the upgrade. Then install the kernel you want after the upgrade. There is no way to make the upgrade script help on your system.

    omv 8.0.8-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.0.2 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    i am not expecting the upgrade script to handle this, i am trying to upgrade somehow with manual interventions, but so far they are failing. i have the proxmox kernel installed and i ve been running it for the past years. 6.11.11-2-pve is the only kernel i have installed. yet, i cannot remove zfs-dkms, because it wants to remove omv-zfs as well. i could try to remove it with dpkg, maybe it will keep omv-zfs intact, but i am bit reluctant to do that.


    i tried to remove the dkms module via sudo dkms remove zfs/2.3.2 then update to kernel 6.14, but somehow all the old headers i have removed before crept back it, so its again rebuilding the kernel modules for 40 minutes, for 6.8, 6.11, 6.14 and 6.17. the rebuild in this case fails because there is already a newer module in the kernel:


    Code
    Error! Module version 2.3.2-2 for zfs.ko
is not newer than what is already found in kernel 6.14.11-4-pve (2.3.4-pve1).
You may override by specifying --force.

Error! Module version 2.3.2-2 for spl.ko
is not newer than what is already found in kernel 6.14.11-4-pve (2.3.4-pve1).
You may override by specifying --force.


    Anyhow, tried to update, then reboot, then remove the zfs-dkms package, then reboot again. i had to manually import the zfs pools and unlock them, but i still cannot get into the OMV webui and getting that 502 error.


    there is also a remaining dkms error for nvidia drivers and kernel 6.17 but i don't think thats blocking OMV since i am booted into kernel 6.14

    SuperMicro CSE-825, X11SSH-F, Xeon E3-1240v6, 64 GB ECC RAM, LSI 9211-8i HBA controller, 6x HGST 12 TB ZFS

    Powered by Proxmox VE

    • Official Post
    Quote from molnart

    i am not expecting the upgrade script to handle this, i am trying to upgrade somehow with manual interventions, but so far they are failing. i have the proxmox kernel installed and i ve been running it for the past years. 6.11.11-2-pve is the only kernel i have installed. yet, i cannot remove zfs-dkms, because it wants to remove omv-zfs as well. i could try to remove it with dpkg, maybe it will keep omv-zfs intact, but i am bit reluctant to do that.

    The plugin isn't preventing zfs-dkms from being removed IF you have pve-headers or proxmox-headers installed - https://github.com/OpenMediaVa…ob/7.x/debian/control#L16.


    Quote from molnart

    but somehow all the old headers i have removed before crept back it,

    Headers don't cause modules to be built. kernels do. Leave the proxmox headers installed. I am guessing you are removing the proxmox header meta package that the zfs plugin has an 'or' dependency on that I linked to above.

    omv 8.0.8-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.0.2 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from ryecoaaron

    Headers don't cause modules to be built. kernels do.

    well, there where no excess kernels in the OMV menu except the one pve kernel booted.


    Quote from ryecoaaron

    IF you have pve-headers or proxmox-headers installed

    i am quite positive i had those installed. also the installation procedure from the Kernels menu must have installed them.


    Quote from ryecoaaron

    I am guessing you are removing the proxmox header meta package that the zfs plugin has an 'or' dependency on that I linked to above.

    I have removed the kernels from the web interface and then the headers by doing sudo apt remove proxmox-headers-6.5 proxmox-headers-6.8 proxmox-headers-6.11. But in the end it might have removed the meta packages as well, as now when i try to remove the 6.17 kernel (which looks like it does not play with the trixie nvidia drivers) it also tries to remove the meta packages



    i have also noticed that dpkg still lists a lot of kernel residues:


    the big question now is, how to move forward?


    i have a mostly functioning OMV 8 install - meaning ZFS works, docker works, shares work. but i cant get into the web ui because of error 502 and cannot build the nvidia drivers for kernel 6.17, while i also cannot remove it, because it wants to remove the proxmox meta packages too.


    i kinda consider this half-success and am reluctant to go back to OMV7 and start from scratch, given i see little chance to get somewhere under 2-3 hours (twice that much if i have to do it again with all the kernels installed).


    do i have any paths to "fix" the OMV 8 install?

    SuperMicro CSE-825, X11SSH-F, Xeon E3-1240v6, 64 GB ECC RAM, LSI 9211-8i HBA controller, 6x HGST 12 TB ZFS

    Powered by Proxmox VE

    • Official Post
    Quote from molnart

    but i cant get into the web ui because of error 502

    Try: sudo omv-salt deploy run nginx phpfpm

    Quote from molnart

    do i have any paths to "fix" the OMV 8 install?

    The kernel plugin allows installing the 6.14 proxmox kernel on omv8 which should allow compiling the nvidia module. I am running omv8 with the 6.12 kernel on my system with an nvidia gpu but I don't use zfs.

    Quote from molnart

    still lists a lot of kernel residues:

    rc does mean removed. So, not a huge deal. You could purge them. I do that with the following command but it blindly removes every package with rc status. So, if you use it and it installs something you didn't want, no complaints to me.


    dpkg -l | grep ^rc | cut -d" " -f3 | sudo xargs dpkg -P

    omv 8.0.8-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.0.2 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from ryecoaaron

    Try: sudo omv-salt deploy run nginx phpfpm

    Thanks, that was it ! The web ui now works.


    But in the UI all the plugins look to be stuck at their 7.x versions


    Not sure if this is a bug or something related to my install. According to dpkg -l the plugin versions are correct:


    Quote from ryecoaaron

    The kernel plugin allows installing the 6.14 proxmox kernel on omv8 which should allow compiling the nvidia module.

    yep, that is the one i am running, and its working fine, incl. the nvidia drivers.


    but i still have the 6.17 kernel somewhere lingering around, as every apt update tries to build the nvidia drivers for it, basically blocking me from installing or removing anything.


    the 6.17 kernel is not visible from the UI, but also at the same time 6.14 is NOT marked as the default (i assume 6.17 is then)


    for a moment it appeared (after trying to remove non-proxmox kernels) and when i tried to uninstall it from the UI i got this - basically looks like the unbuild dkms module is blocking the removal of kernel

    SuperMicro CSE-825, X11SSH-F, Xeon E3-1240v6, 64 GB ECC RAM, LSI 9211-8i HBA controller, 6x HGST 12 TB ZFS

    Powered by Proxmox VE

    Edited once, last by molnart ().

    • Official Post
    Quote from molnart

    But in the UI all the plugins look to be stuck at their 7.x versions

    this is what the fix7to8upgrade script fixes but I wasn't sure if it would cause other problems on your system due its numerous issues. What until you have your other issues fixed before you run it.


    Quote from molnart

    but i still have the 6.17 kernel somewhere lingering around, as every apt update tries to build the nvidia drivers for it, basically blocking me from installing or removing anything.

    It is installed but in a bad state according to your dpkg output. You will need to remove it from the command line because the kernel plugin only allows you to remove kernels that show up in grub.


    sudo dpkg -P proxmox-kernel-6.17

    Quote from molnart

    the 6.17 kernel is not visible from the UI, but also at the same time 6.14 is NOT marked as the default (i assume 6.17 is then)

    By default, grub doesn't mark a kernel as a default boot kernel. If you really want to see a default boot kernel, click set boot when selecting the kernel you want.



    To fix your system, I would remove all of the nvidia stuff and get everything in a good place. Then add the nvidia stuff back. I really can't wait until the nvidia driver in kernel is further along. These stupid compiling issues are really annoying (I have dealt with them for years not just your system).

    omv 8.0.8-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.0.2 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    thanks a lot for your help, i really appreciate it.


    so i did this:

    - sudo dpkg -P proxmox-kernel-6.17

    - run omv-upgrade to see how things are working, but still got the attempt to build dkms for 6.17

    - so i removed the nvidia drivers by running apt remove nvidia-kernel-dkms

    - ran omv-upgrade again, this removed a bunch of unused nvidia dependencies

    - rebooted

    - ran sudo apt install nvidia-driver but this has again attempted (and failed) to build the module for 6.17


    looking at dpkg -l there are still some remnants of the 6.17 kernel

    i assume these dependencies are the proxmox meta packages.

    - so i did a sudo apt remove proxmox-headers-6.17 followed by a sudo apt autoremove. this time there were no 6.17 packages listed by dpkg anymore.

    - rebooted again

    - sudo apt install nvidia-driver - this time modules only built for 6.14

    - rebooted

    - success


    so now everything seems to be OK, but i don't have the proxmox kernel meta packages installed. i wonder if this will will bite me back in the future.


    Quote from ryecoaaron

    this is what the fix7to8upgrade script fixes

    where do i find this script? does not seem to be installed on the system


    now i should update by secondary server as well. this one is running bare-metal, so no easy rollback. but also no docker, zfs or nvidia drivers, so hopefully will go smoothly.

    SuperMicro CSE-825, X11SSH-F, Xeon E3-1240v6, 64 GB ECC RAM, LSI 9211-8i HBA controller, 6x HGST 12 TB ZFS

    Powered by Proxmox VE

    • Official Post
    Quote from molnart

    but i don't have the proxmox kernel meta packages installed. i wonder if this will will bite me back in the future.

    What is installed at this point?


    Quote from molnart

    where do i find this script? does not seem to be installed on the system

    It isn't installed. You download from github. Lots of posts about it on the forum.

    Post
    RE: How to upgrade openmediavault-omvextrasorg to version 8?
    […]

    omv-extras 8 is definitely available.

    […]

    Something went wrong with your upgrade. Can't tell you what happened without the output from the upgrade. Run the fix7to8upgrade script

    sudo wget -O - https://raw.githubusercontent.com/OpenMediaVault-Plugin-Developers/installScript/master/fix7to8upgrade | sudo bash
    ryecoaaron
    Quote from molnart

    now i should update by secondary server as well. this one is running bare-metal, so no easy rollback. but also no docker, zfs or nvidia drivers, so hopefully will go smoothly.

    I have upgrading all of seven of my servers that fit that description and had no issues. Lots of people are having great upgrade success. Docker is not a problem. zfs and nvidia are the problems. I wouldn't assume you will have issues if you aren't using zfs and nvidia on the bare metal box.

    omv 8.0.8-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.0.2 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from ryecoaaron

    What is installed at this point?

    just the 6.14 kernel. no proxmox-kernel or proxmox-headers meta package - because that installs the 6.17 kernel as far as i understood



    Quote from ryecoaaron

    You download from github.

    Thanks, all fixed now

    SuperMicro CSE-825, X11SSH-F, Xeon E3-1240v6, 64 GB ECC RAM, LSI 9211-8i HBA controller, 6x HGST 12 TB ZFS

    Powered by Proxmox VE

    • Official Post

    pve-headers shouldn't be needed anymore. I need to make the kernel plugin stop installing it. And in reality, I don't know why I have it install proxmox-default-headers either. If someone wants to install the 6.14 kernel, proxmox-headers-6.14 is all they need to get updates. I will have to look at that.

    you can uninstall pve-headers and proxmox-default-headers. As long as you have proxmox-headers-6.14, the zfs plugin shouldn't be uninstalled.

    omv 8.0.8-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.0.2 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Hi, I just upgraded from OMV 7 to 8 without any problems, and everything is working.

    Kernel: 6.5.13-1-pve


    The only thing that bothers me is the message:

    Code
    The following packages have been kept back:
  openmediavault-zfs

    Is there a problem or this can be ignored?



    Thanks and kind regards!

    • Official Post
    Quote from vperas

    Is there a problem or this can be ignored?

    That isn't good. Run the fix7to8upgrade script and post the output of: dpkg -l | grep openme

    omv 8.0.8-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.0.2 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from ryecoaaron

    That isn't good. Run the fix7to8upgrade script and post the output of: dpkg -l | grep openme


    Output after running the script:

    • Official Post

    weird. If you still have the proxmox 6.5 kernel, the upgrade didn't install a newer kernel. I would install the proxmox 6.14 or 6.17 kernel and the zfs plugin should be able to install the dependencies it needs then.

    omv 8.0.8-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.0.2 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Thanks 1

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login