OMV and Docker restarts to much

    Zitat von gderf

    When you run out of RAM and have no swap enabled, then this is what is to be expected. But you were already told this. So why do you find this weird?

    Because RAM was never an issue before. I run a Pi4 and it has only a little bit of SWAP Space as default. It should work totally fine. The pi never had so much default SWAP and everything worked fine for 3 Years straight. Now suddenly SWAP should be the issue why OMV or Docker engine is crashing?

    I only enabled more SWAP because Kasm required it but now i already uninstalled kasm so no need for the extra SWAP. Why all of the sudden it should be now "requiret" only because i previously once intstalled kasm?

    If this logic is true then kasm would destroy RaspberryPies

    Zitat von macom

    OR you enabled swap because of kasm, but in the meantime you installed other things that require more RAM. So now, even after you removed kasm, you still need swap

    Thats a good point. I did but i also removed them too because of the stability issues. So it is now same as it was before.

    I will do more testing and if nothing helps and it still crashes i guess i have to enable more SWAP again i guess.

    You can start from the top first:

    Check the performance statistics on diagnostics.

    See the values there to try to find when it goes wrong.


    And a Pi isn't a power house.

    Maybe you're setting too high expectations on it's capabilities: too many services/containers running, for eg

    Zitat von Soma

    You can start from the top first:

    Check the performance statistics on diagnostics.

    See the values there to try to find when it goes wrong.


    And a Pi isn't a power house.

    Maybe you're setting too high expectations on it's capabilities: too many services/containers running, for eg

    Yeah i will check it.


    Yes thats true. I already found the limit of the pi but maybe because of arm64 and everxthing uses more resources so the limit is even lower. Who knows. I will continue testing and see if i can specify the problem more precise

    Code
    systemd[1]: Configuration file /etc/systemd/system/docker.service.d/waitAllMounts.conf is marked world-writable. Please remove world writability permission bits. Proceeding anyway.

    Does someone know this error? I just now found it in OMV Syslog after my NAS crashed again 30mins ago.


    When i open the file i find this


    Code
    [Unit]
After=local-fs.target srv-dev\x2ddisk\x2dby\x2duuid\x2d297c9eb4\x2deb1a\x2d4e95\x2d82cd\x2de56d7e277e22.mount srv-dev\x2ddisk\x2dby\x2duuid\x2d32af1309\x2d18b7\x2d46cc\x2dbbc6\x2dad9ceca9a64d.mount srv-dev\x2ddisk\x2dby\x2duuid\x2da2dcb3d3\x2dd8f9\x2d48b3\x2d92cc\x2d4bee92d1f47f.mount srv-dev\x2ddisk\x2dby\x2duuid\x2dd89126ab\x2dbedb\x2d43f1\x2dab2a\x2d087dfd5be2d5.mount srv-mergerfs-MeinNAS.mount

    but i cannot find anything marked as "world-writable"

    Zitat von Copaxy

    I will continue testing and see if i can specify the problem more precise

    Out of curiosity, what is the output of:

    docker ps -a

    free

    Zitat von Copaxy

    I am still searching for a reason

    You can see the amount of issues on the drives.


    Nonetheless, you said that all was ok with OMV5 but not now that you're on OMV6.


    What SATA HAT are you running? Radxa?!?

    Zitat von Soma

    You can see the amount of issues on the drives.


    Nonetheless, you said that all was ok with OMV5 but not now that you're on OMV6.


    What SATA HAT are you running? Radxa?!?

    Yes thats true. There are not less issues. But it is okey, one o my drives couses some trouble and it will be soon replaced so i don't worry about that.


    I use a quad SATA hat from allnetchina for the Pi4

    https://wiki.radxa.com/Dual_Quad_SATA_HAT


    Yes exactly, on OMV5 never an issue. OMV5 was running stable as a Rock


    But still because OMV is running on the Pis SD card like back then on OMV5 when i first started, whatever problem there was with the drive OMV still was always on because it is on the Pis SD card. Same as now. So why my drives should affect the OMV installation on the installation media?

    I mean i can totally understand the amout of issues because of my one bad drive but why OMV now has running issues? I am still trying to determine the exactly point why OMV thinks it needs to restart. Because whatever my external drives do it should not affect OMV until the point it needs to restart. Usually when there ever was a problem before, the drives where just missing and OMV stayed online and i manually fixed the problem and restarted.

    Zitat von Copaxy

    I use a quad SATA hat from allnetchina for the Pi4

    https://wiki.radxa.com/Dual_Quad_SATA_HAT

    There's some instructions there to use with that HAT and the supported OS are Raspbian Buster 32bit and Ubuntu Server 20.04.


    If you only started having issues with OMV6, perhaps the problem is there.

    I think i know the issue.


    I have completey reset my system and i also still noticed some shutdowns. And they only appeared when RAM is fully used.

    I guess OMV6 uses more ram then before with OMV5. I have to keep that in mind.


    Do you think it could be useful to hook up a ssd drive and use some of that as a emergancy swap space? I know it will be much slower then RAM but just that it won't shut down randomly. Because there isn't much running on my NAS. Just Syncthing and nextcloudpi as a container

    • Offizieller Beitrag

    How much RAM do you have? I've not noticed a significant ram increase. I've ran OMV 6 through a lot of tests in a virtual machines and docker, and it has just over 3.5gigs of RAM and causes no issue.


    Code
    joe0201@omv6-test:~$ free -h -m
               total        used        free      shared  buff/cache   available
Mem:           3.7Gi       406Mi       1.2Gi        51Mi       2.0Gi       3.0Gi
Swap:          974Mi          0B       974Mi
joe0201@omv6-test:~$


    My main server (running 17 containers)


    Code
    ken0201@openmediavault:~$ free -h -m
               total        used        free      shared  buff/cache   available
Mem:           7.5Gi       2.1Gi       167Mi       655Mi       5.2Gi       4.5Gi
Swap:          974Mi       626Mi       348Mi
ken0201@openmediavault:~$
    Zitat von KM0201

    How much RAM do you have? I've not noticed a significant ram increase. I've ran OMV 6 through a lot of tests in a virtual machines and docker, and it has just over 3.5gigs of RAM and causes no issue.

    i use a PI4 with 4G RAM.


    I believe you but then i cannot explain the behavior of my fresh install stetup...

    I just run nextcloudpi and nginx reverse proxy, bitwarden and syncthing. Nothing too big.


    Before on OMV5 i had syncthing twice, bitwarden, jellyfin, sonarr, radarr, grafana with prometheus, guacamole, nginx, synapse, dashmachine. with no issues.


    No i run half of it and it runs out of ram sometimes and it shuts down


    I am out of eplanations...

    • Offizieller Beitrag

    Possible hardware problem? Lot of folks are using Pi 4's and I don't think I've read anything like this.


    crashtest  Soma

    Zitat von KM0201

    Possible hardware problem? Lot of folks are using Pi 4's and I don't think I've read anything like this.

    i mean i think not because on OMV5 everything worked fine. Just in OMV6 when something is using the Memory quite a lot and it goes to its limit then soon after i can see in the OMV surface that it is up since 10mins ago, what means it restarted i guess. I observed the problem quite a while now with OMV6 and had several reinstalls. But it only happens if the RAM for longer time is to high like 95% used or so.


    Very weird..never had this issue before and only on OMV6 for some magic reason...


    Below is the syslog for around that time window. But exept for my full file system i don't get much out of it.

    My OMV6 Pi4 with 4Gb is only running docker qbittorrent and NFS serving the LAN with media share.


    No oom whatsoever..


    But one of the first thing I do is disable swap.

    Don't know if it makes any difference

    • Offizieller Beitrag

    I'm with KM0201 on this. I've never heard of this behavior either. It may point to hardware.

    I have to qualify the following and note that I don't have a lot of experience with OMV6 / Debian Buster on Arm platforms at this point, but I have built 6 on Arm devices more than a couple times.


    - The first thing I would do, if you haven't already, is replace the SD-card. A good quality *new* card is relatively inexpensive. It's worth trying. A single bad location on an SD-card can send you down a rabbit hole. (I know from experience.)

    - The second thing to consider is power. I noticed Soma mentioned examining dmesg for undervoltage messages but, even if dmesg is clean and the power supply is putting out good voltage, that doesn't mean your PS doesn't have AC ripple on the the output. (A dirty output.)

    - Regarding your storage disks, it appears that you have 4 disks that are over 95% full. In that scenario, when there's disk I/O, disk performance is poor to abysmal. There may be system slowdowns and potentially heavy file fragmentation. Adding storage when disks are over 95% full is not an option. I would get more storage ASAP or delete some files.

    - Realize that just one (1) bad hard drive (or a disk going bad) can produce bizarre results and note that SMART stat's do not reveal all. A drive's SATA interface board can develop issues that may not be revealed by SMART.

    - Note that your SATA hat is a SATA to USB bridge. There are issues with connecting hard drives over USB. One drive per USB bus usually works without issues. Two or more drives per USB bus might get into bandwidth contention issues. I'm not saying anything is wrong but note that you're using unconventional "hobbyist" hardware.

    Lastly, note that OMV6 is a web application that allows for easy configuration of the OS. The under laying OS is Debian Buster which, in the case of an R-PI, is a modified "Raspberry PI OS" version of Debian Buster. What I'm getting at is, the issue may be with Buster or an OS "tweak" by the R-PI foundation.
    ______________________________________

    What would I do? Go with the easiest things first.
     
    - First, solve the storage problem. Over 95% full on all data disks is going to create issues, sooner or later. Ideally, I would want storage disks to be well under 85%.
    - Set aside your current build / SD-card and rebuild on a new good quality card - SanDisk, Samsung, etc. If you need something from the NAS, simply reboot on your current build.
    - Don't be afraid to rebuild. Some users will do or try anything to figure out what went wrong with their installation to avoid rebuilding. Trying to forensically figure out what might have gone wrong can result in a huge amount of wasted time. You might even consider rebuilding on OMV5 to find, whether or not, it's an OS problem versus something that has recently developed with hardware.
    - Keep your build clean. When an app is removed, config files are not always reset to the values that existed before. If you want to try out an app, clone your SD-card so you can back out cleanly if you don't want to keep the app.
    - If the reset / restart is predictable and reasonably short, you might try simplifying your setup after a rebuild. For example, on the new build SD-card, setup without Dockers and let it run overnight. When it comes to hardware, start off with no connected drives and add back one drive at a time to see if / when the resets start.

    Finally, note that all Dockers are not created equal. I've seen Dockers get pushed out that had flaws in them. I've never heard of a flawed Docker causing a system reset. That seems unlikely, but it can't be ruled out.

    The above are preliminary ideas for trying to isolate the issue. Hopefully, you can narrow it done to something tangible.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden