OMV 8 long pause while shutting down.

  • When I monitor the shutdown process in the SOL console it will hang at the the following statement for a considerable amount of time before continuing normally.


    Code
    OMV 8 systemd-shutdown[1]: Waiting for process: 10046 (s6-svscan), 9988 (s6-svscan), 9965 (s6-svscan), 11948 (python3), 882573 (Plex Media Serv), 10031 (s6-svscan), 881402 (s6-svscan), 9984 (s6-svscan), 957120 (s6-svscan), 881448 (s6-supervise), 10737 (s6-supervise)


    I let ChatGPT examine the above Code and it offered suggestions some of which I tried. Eventually I got it solved but the fix introduced a new problem. None of my running dockers would automatically restart when booting the machine. I could manually start each one though, but this is not acceptable. None of what ChatGPT suggested to fix the failure to start the dockers worked so I backed out of the fix that solved the hanging shutdown.


    Is anyone else seeing a long pause on shutdown?

    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.

    OMV AMD64 8.x on headless Tyan Thunder SX GT86C-B5630 1U Server with Intel Xeon Silver 4110 CPU @ 2.10GHz & 32GB DDR4 ECC RAM.

    • Official Post

    I'm not seeing that on either omv 8 system running containers. What did you fix to solve the first problem? I would guess docker is starting before storage is ready if you containers aren't starting automatically.

    omv 8.0.10-2 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.7 | compose 8.1.3 | cterm 8.0 | borgbackup 8.1.6 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • It was Fix 3. here:


    https://chatgpt.com/share/693d8a0d-5914-800b-a711-3852f3355923

    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.

    OMV AMD64 8.x on headless Tyan Thunder SX GT86C-B5630 1U Server with Intel Xeon Silver 4110 CPU @ 2.10GHz & 32GB DDR4 ECC RAM.

    • Official Post

    This seems like something that should be fixed in the image using s6. AI seems to want to attach scripts to systemd units a lot as a fix.

    omv 8.0.10-2 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.7 | compose 8.1.3 | cterm 8.0 | borgbackup 8.1.6 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • This seems like something that should be fixed in the image using s6. AI seems to want to attach scripts to systemd units a lot as a fix.

    I noticed that.

    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.

    OMV AMD64 8.x on headless Tyan Thunder SX GT86C-B5630 1U Server with Intel Xeon Silver 4110 CPU @ 2.10GHz & 32GB DDR4 ECC RAM.

  • I wonder what changed between OMV7/Debian 12 and OMV8/Debian 13 to cause this. I didn't change anything going from one to another.

    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.

    OMV AMD64 8.x on headless Tyan Thunder SX GT86C-B5630 1U Server with Intel Xeon Silver 4110 CPU @ 2.10GHz & 32GB DDR4 ECC RAM.

  • Yes, the shutdown process also takes a long time on my device. I tested it: closing all Docker containers before restarting made the issue disappear. I’m not sure whether it’s also related to AppArmor—I haven’t tested whether keeping the containers running but turning off AppArmor would allow for a quick shutdown.

  • I haven't run apparmor here for many years, making it a non issue for me.


    So far my limited testing with the 19 containers I run shows that Plex is definitely a contributor to the problem.

    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.

    OMV AMD64 8.x on headless Tyan Thunder SX GT86C-B5630 1U Server with Intel Xeon Silver 4110 CPU @ 2.10GHz & 32GB DDR4 ECC RAM.

  • 我已经很多年没在这里使用 AppArmor 了,所以对我来说这不是问题。


    就我目前运行的 19 个容器的有限测试来看,Plex 肯定是造成这个问题的原因之一。

    To locate containers related to S6, I use the following commands to inspect each container:

    docker ps -a

    docker exec <CONTAINER ID> ps aux | grep s6

  • I think this might be likely due to the recent e5b14c7 commit to the openmediavault-compose plugin, that added the Live Restore option to docker's daemon.json file.


    Over at the NixOS forums they discuss this option causing systems to hang at shutdown, with this person explicitly disabling it in their NixOS config to fix it.


    axiauk  gderf Could you see if you notice any difference to the hang time between using sudo shutdown -r now as compared with sudo reboot ?


    I wonder if maybe docker-compose needs a pre-shutdown hook like this to tell docker to gracefully stop containers before the shutdown / reboot command is processed? Because according to this post that is what worked for them.


    You can always manually edit the /etc/docker/daemon.json file and change "live-restore": true, to "live-restore": false, [EDIT] There is a checkbox in Compose > Settings under Docker Config to disable Live Restore. Uncheck it and see if that fixes it, which according to those posts it likely will, but it will change back next time the compose plugin gets an update unless ryecoaaron reverts the change upstream. I don't know enough about Live Restore to know whether this is worth disabling or not, or whether it's better to instead find a graceful way of shutting containers down at shutdown / reboot that doesn't cause them to hang whilst keeping Live Restore enabled.

    OpenMediaVault 7 running on RaspberryPi 5, 16GB RAM model, with Radxa SATA HAT. 2x 10TB HDD in ZFS Mirror. 1x 2TB HDD in ZFS Simple Mode.

    1x USB3 -> NVMe 256GB SSD, XFS formatted, for Docker Installation and Compose Data. 1x USB3 -> SATA 256GB SSD, EXT4 formatted, OMV Boot/Root Drive.

    Edited 2 times, last by gecko ().

    • Official Post

    I don't know enough about Live Restore to know whether this is worth disabling or not, or whether it's better to instead find a graceful way of shutting containers down at shutdown / reboot that doesn't cause them to hang whilst keeping Live Restore enabled.

    I just thought live restore was a cool feature and it hasn't caused any shutdown issues for my containers. If it is, that is why the checkbox is there. If you disable on the Settings tab, it won't come back. I have no interest in writing a shutdown script for s6 containers when the live restore checkbox can be unchecked. The compose plugin has been around a while without live restore enabled and didn't cause a problem. So, it would be nice to know if that fixes people's problems.

    omv 8.0.10-2 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.7 | compose 8.1.3 | cterm 8.0 | borgbackup 8.1.6 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I just thought live restore was a cool feature and it hasn't caused any shutdown issues for my containers. If it is, that is why the checkbox is there. If you disable on the Settings tab, it won't come back. I have no interest in writing a shutdown script for s6 containers when the live restore checkbox can be unchecked. The compose plugin has been around a while without live restore enabled and didn't cause a problem. So, it would be nice to know if that fixes people's problems.

    Yeah I mean it does sound useful, so it's definitely a good option to have it in the settings. (It wasn't until I read further down the commit did I see there was a checkbox in settings. Still being on OMV7 I assume I don't have this setting so didn't check on my web-ui).


    Perhaps the default setting should be off though, the same as it is for an official docker default config? And that way if people want to benefit from it they can always switch it on? But yeah, I suppose only time will tell if it affects lots of people and the containers they run.

    OpenMediaVault 7 running on RaspberryPi 5, 16GB RAM model, with Radxa SATA HAT. 2x 10TB HDD in ZFS Mirror. 1x 2TB HDD in ZFS Simple Mode.

    1x USB3 -> NVMe 256GB SSD, XFS formatted, for Docker Installation and Compose Data. 1x USB3 -> SATA 256GB SSD, EXT4 formatted, OMV Boot/Root Drive.

  • sudo shutdown -r now vs sudo reboot makes no difference in the hang time.


    I also tried setting "live-restore": false, in /etc/docker/daemon.json. This eliminated this shutdown hang but when the machine restarted none of my dockers were running.


    I then ran docker start $(docker ps -aq) to start all the dockers and shutdown -r now again. No shutdown hang and when the machine cam back up all dockers were running.


    I rebooted a few more times without any hangs and all my dockers were running when the machine came back up.


    In my case setting "live-restore": false in /etc/docker/daemon.json seems to have solved it.


    Thanks for the hint where to look.





    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.

    OMV AMD64 8.x on headless Tyan Thunder SX GT86C-B5630 1U Server with Intel Xeon Silver 4110 CPU @ 2.10GHz & 32GB DDR4 ECC RAM.

    • Official Post

    Perhaps the default setting should be off though, the same as it is for an official docker default config?

    Maybe I will change the default but I'm not going to push a release just for that change. Sad that most people won't benefit from the setting because they won't know it exists or to look for it though.

    omv 8.0.10-2 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.7 | compose 8.1.3 | cterm 8.0 | borgbackup 8.1.6 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • No need to manually edit the file. As ryecoaaron said, there is a checkbox in Compose > Settings to change this. I did edit my post, but perhaps I didn't catch you in time.

    OpenMediaVault 7 running on RaspberryPi 5, 16GB RAM model, with Radxa SATA HAT. 2x 10TB HDD in ZFS Mirror. 1x 2TB HDD in ZFS Simple Mode.

    1x USB3 -> NVMe 256GB SSD, XFS formatted, for Docker Installation and Compose Data. 1x USB3 -> SATA 256GB SSD, EXT4 formatted, OMV Boot/Root Drive.

    • Official Post

    In my case setting "live-restore": false in /etc/docker/daemon.json seems to have solved it.

    I assume you a leaving the docker storage field blank to customize your daemon.json? Otherwise, the checkbox is there to disable it.

    omv 8.0.10-2 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.7 | compose 8.1.3 | cterm 8.0 | borgbackup 8.1.6 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Maybe I will change the default but I'm not going to push a release just for that change. Sad that most people won't benefit from the setting because they won't know it exists or to look for it though.

    Yeah, I suppose you could leave it on by default and wait to see how many people are affected by the change? Maybe add some instructions to switch it off to the wiki, or pin it to the forum so people will find it if they have issues?

    OpenMediaVault 7 running on RaspberryPi 5, 16GB RAM model, with Radxa SATA HAT. 2x 10TB HDD in ZFS Mirror. 1x 2TB HDD in ZFS Simple Mode.

    1x USB3 -> NVMe 256GB SSD, XFS formatted, for Docker Installation and Compose Data. 1x USB3 -> SATA 256GB SSD, EXT4 formatted, OMV Boot/Root Drive.

  • I assume you a leaving the docker storage field blank to customize your daemon.json? Otherwise, the checkbox is there to disable it.

    I left that field blank to make custom changes in the file. I did not realize that the checkbox would do the same thing. I have since then restored the docker storage field as it was and am using the checkbox.


    Very powerful checkbox that is. Having it enabled cost me an additional 90 seconds to reboot :(

    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.

    OMV AMD64 8.x on headless Tyan Thunder SX GT86C-B5630 1U Server with Intel Xeon Silver 4110 CPU @ 2.10GHz & 32GB DDR4 ECC RAM.

    • Official Post

    Very powerful checkbox that is. Having it enabled cost me an additional 90 seconds to reboot

    And it didn't change a single thing with my reboots across two systems and 20 images. Hence why I made it the default. I should've just kept to my previous idea of not configuring docker.

    omv 8.0.10-2 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.7 | compose 8.1.3 | cterm 8.0 | borgbackup 8.1.6 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • gderf


    If you have a spare few moments, as a test it would be interesting to know whether, with Live Restore switched back on, shutting down via sudo docker stop $(sudo docker ps -q) ; sudo reboot makes any difference. Because if it did then maybe that could be an easy patch to apply to OMV's shutdown/reboot procedure when Live Restore is enabled.

    OpenMediaVault 7 running on RaspberryPi 5, 16GB RAM model, with Radxa SATA HAT. 2x 10TB HDD in ZFS Mirror. 1x 2TB HDD in ZFS Simple Mode.

    1x USB3 -> NVMe 256GB SSD, XFS formatted, for Docker Installation and Compose Data. 1x USB3 -> SATA 256GB SSD, EXT4 formatted, OMV Boot/Root Drive.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!