High load average caused by multiple instances of omv-engined

iago27 · 31. Mai 2023

Hi everyone, I've been experiencing some repeated behavior on my omv server and I'm hoping to any help or insights as to how to address this.

My setup includes a couple of mergerfs filesystems mounted via NFS. The physical drives are connected via an HBA card and passed through to omv. OMV ver. is 6.4.0-3 (Shaitan) in a debian container. Everything is up to date etc, and persists after reboots/restarting services individually. Mostly video files.

For the most part everything has been stable, but recently I've been having random periods of huge spiking in server load and IO wait, which also makes the NFS shares unresponsive, as well as my VMs which have these shares mounted, which normally results in me having to force kill the process and/or reboot the machine.

During troubleshooting, I noticed a large number of blkid -o full and omv-engined commands appearing in ps aux. iotop sometimes shows a significant IO percentage for mergerfs, although weirdly not for the most recent occurrence.

I noticed multiple instances of the omv-engined daemon running simultaneously, which seemed unusual.

for example: ps -ef

Code

root     1046841  614244  0 18:32 ?        00:00:00 omv-engined
root     1046847 1046841  0 18:32 ?        00:00:00 omv-engined
root     1046850 1046841  0 18:32 ?        00:00:00 omv-engined
root     1046855 1046841  0 18:32 ?        00:00:00 omv-engined
root     1046859 1046841  0 18:32 ?        00:00:00 omv-engined
root     1046862 1046841  0 18:32 ?        00:00:00 omv-engined
root     1046865 1046841  0 18:32 ?        00:00:00 omv-engined
root     1046867 1046841  0 18:32 ?        00:00:00 omv-engined
root     1046871 1046841  0 18:32 ?        00:00:00 omv-engined
root     1046875 1046841  0 18:32 ?        00:00:00 omv-engined

and my dashboard:

votdev · 31. Mai 2023

Zitat von iago27

During troubleshooting, I noticed a large number of blkid -o full and omv-engined commands appearing in ps aux. iotop sometimes shows a significant IO percentage for mergerfs, although weirdly not for the most recent occurrence.
I noticed multiple instances of the omv-engined daemon running simultaneously, which seemed unusual.

The is a forking daemon, so every request is a child process. Additional several requests are splitted into several parallel tasks. In your case i think a RPC is fetching file system information from ~8 disks in parallel.

iago27 · 5. Juni 2023

Thank you for that clarification -- I should explain further that the above is just an excerpt, and there are many many more instances of omv-engined running than posted above.

For full context, I have 6 physical disks mounted, 2 umounted but installed, and 2 virtual mergerfs pools.

Code

root@omv:~# ps aux | grep omv-engined
root     1034513  0.0  0.0  71360 12440 ?        S    11:52   0:00 omv-engined
root     1042542  0.0  0.0  71360 13572 ?        S    11:57   0:00 omv-engined
root     1042543  0.0  0.0  71360 11364 ?        S    11:57   0:00 omv-engined
root     1042545  0.0  0.0  71360 11364 ?        S    11:57   0:00 omv-engined
root     1042548  0.0  0.0  71360 11364 ?        S    11:57   0:00 omv-engined
root     1042551  0.0  0.0  71360 11364 ?        S    11:57   0:00 omv-engined
root     1042554  0.0  0.0  71360 11364 ?        S    11:57   0:00 omv-engined
root     1042557  0.0  0.0  71360 11364 ?        S    11:57   0:00 omv-engined
root     1042560  0.0  0.0  71360 11364 ?        S    11:57   0:00 omv-engined
root     1042563  0.0  0.0  71360 11364 ?        S    11:57   0:00 omv-engined
root     1042566  0.0  0.0  71360 11364 ?        S    11:57   0:00 omv-engined
root     1044187  0.0  0.0  71360 13576 ?        S    11:58   0:00 omv-engined
root     1044202  0.0  0.0  71360 11368 ?        S    11:58   0:00 omv-engined
root     1044204  0.0  0.0  71360 11368 ?        S    11:58   0:00 omv-engined
root     1044206  0.0  0.0  71360 11368 ?        S    11:58   0:00 omv-engined
root     1044208  0.0  0.0  71360 11368 ?        S    11:58   0:00 omv-engined
root     1044210  0.0  0.0  71360 11368 ?        S    11:58   0:00 omv-engined
root     1044214  0.0  0.0  71360 11368 ?        S    11:58   0:00 omv-engined
root     1044218  0.0  0.0  71360 11368 ?        S    11:58   0:00 omv-engined
root     1044219  0.0  0.0  71360 11368 ?        S    11:58   0:00 omv-engined
root     1044223  0.0  0.0  71360 11368 ?        S    11:58   0:00 omv-engined
root     1045384  0.0  0.0  71360 12512 ?        S    11:59   0:00 omv-engined
root     1045654  0.0  0.0  71360 13576 ?        S    11:59   0:00 omv-engined
root     1045655  0.0  0.0  71360 11368 ?        S    11:59   0:00 omv-engined
root     1045657  0.0  0.0  71360 11368 ?        S    11:59   0:00 omv-engined
root     1045660  0.0  0.0  71360 11368 ?        S    11:59   0:00 omv-engined
root     1045663  0.0  0.0  71360 11368 ?        S    11:59   0:00 omv-engined
root     1045667  0.0  0.0  71360 11368 ?        S    11:59   0:00 omv-engined
root     1045669  0.0  0.0  71360 11368 ?        S    11:59   0:00 omv-engined
root     1045672  0.0  0.0  71360 11368 ?        S    11:59   0:00 omv-engined
root     1045675  0.0  0.0  71360 11368 ?        S    11:59   0:00 omv-engined
root     1045678  0.0  0.0  71360 11368 ?        S    11:59   0:00 omv-engined
root     1047300  0.0  0.0  71360 13572 ?        S    12:00   0:00 omv-engined
root     1047301  0.0  0.0  71360 11364 ?        S    12:00   0:00 omv-engined
root     1047303  0.0  0.0  71360 11364 ?        S    12:00   0:00 omv-engined
root     1047306  0.0  0.0  71360 11364 ?        S    12:00   0:00 omv-engined
root     1047309  0.0  0.0  71360 11364 ?        S    12:00   0:00 omv-engined
root     1047312  0.0  0.0  71360 11364 ?        S    12:00   0:00 omv-engined
root     1047315  0.0  0.0  71360 11364 ?        S    12:00   0:00 omv-engined
root     1047318  0.0  0.0  71360 11364 ?        S    12:00   0:00 omv-engined
root     1047321  0.0  0.0  71360 11364 ?        S    12:00   0:00 omv-engined
root     1047324  0.0  0.0  71360 11364 ?        S    12:00   0:00 omv-engined
root     1048164  0.0  0.0  71360 12264 ?        S    12:01   0:00 omv-engined
root     1048485  0.1  0.0  71360 12524 ?        S    12:01   0:00 omv-engined
root     1048486  0.0  0.0  71360 12508 ?        S    12:01   0:00 omv-engined
root     1048487  0.0  0.0  71360 12508 ?        S    12:01   0:00 omv-engined
root     1048488  0.0  0.0  71360 12512 ?        S    12:01   0:00 omv-engined
root     1048489  0.0  0.0  71360 12512 ?        S    12:01   0:00 omv-engined
root     1048493  0.0  0.0  71360 12512 ?        S    12:01   0:00 omv-engined
root     1048496  0.0  0.0  71360 12512 ?        S    12:01   0:00 omv-engined
root     1048499  0.0  0.0  71360 12256 ?        S    12:01   0:00 omv-engined
root     1048502  0.0  0.0  71360 12512 ?        S    12:01   0:00 omv-engined
root     1048953  0.3  0.0  71360 13572 ?        S    12:01   0:00 omv-engined
root     1048954  0.0  0.0  71360 11364 ?        S    12:01   0:00 omv-engined
root     1048956  0.0  0.0  71360 11364 ?        S    12:01   0:00 omv-engined
root     1048959  0.0  0.0  71360 11364 ?        S    12:01   0:00 omv-engined
root     1048962  0.0  0.0  71360 11364 ?        S    12:01   0:00 omv-engined
root     1048966  0.0  0.0  71360 11364 ?        S    12:01   0:00 omv-engined
root     1048968  0.0  0.0  71360 11364 ?        S    12:01   0:00 omv-engined
root     1048972  0.0  0.0  71360 11364 ?        S    12:01   0:00 omv-engined
root     1048975  0.0  0.0  71360 11364 ?        S    12:01   0:00 omv-engined
root     1048978  0.0  0.0  71360 11364 ?        S    12:01   0:00 omv-engined
root     1049288  0.0  0.0   6240   712 pts/0    S+   12:02   0:00 grep omv-engined
root     3569700  0.0  0.0  71204 14080 ?        S    Jun04   0:21 omv-engined

Alles anzeigen

I guess this may still be normal expected behavior for the omv engine, I will explore other reasons why my server load / io wait is ballooning.

High load average caused by multiple instances of omv-engined

Jetzt mitmachen!

Tags