Very slow "Apply", debugging help

1. offizieller Beitrag

    Hi.

    Before I installed OMV5 on target hardware I was doing some testing on VM (Virtualbox on Win10). Through all my test all "Apply" commands ware fairly quick (few seconds) and I configured most of things (including some drives) that I wanted on my target machine (which was still in parts at the time).


    After I assembled my new server I installed OMV5 and start configuring it bit by bit. Save/Apply times ware even better than on VM. I did configured most things including Drives/Filesystem (Snapraid + MergerFS), copied 4TB of data and so on. What I remember at this point it was still quite quick. Then for some unknown reason it started to slow down. I think first I've noticed it it was when I tried to install NUT plugin (and have many problems with it, now it's removed). Recently I did run "omv-salt stage run prepare" and I saw that It is reapplying config for over 253 seconds (logs at the bottom).


    Is there a way to easy debug what is going on? My server hardware config is Ryzen 3600@ 3Ghz, RAM 16GB ECC, 128GB Samsung NVMe for system/docker volumes drive, 3x3TB data drives.

    Tested in local network over 1gbps. No SWAP partition.


    "omv-salt stage run prepare" output

    • Offizieller Beitrag

    I would try the proxmox kernel (kernel tab in omv-extras). I would also check the logs for anything that is going wrong. This system is somewhat bleeding edge which can present problems to the backports kernel.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    During "omv-salt..." there is nothing disturbing in logs.


    Only two things I see in logs that look bad are (today):

    Sep 15 07:46:23 SERVER collectd[20823]: plugin_read_thread: read-function of the `df' plugin took 10.240 seconds, which is above its read interval (10.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.


    Sep 15 18:13:52 SERVER smbd[28643]: process_usershare_file: stat of /var/lib/samba/usershares/git_repositories failed. Nie ma takiego pliku ani katalogu

    "Nie ma takiego pliku ani katalogu" - mean "no file or directory"


    Is there a way to make "omv-salt" more verbose so I could see what is it doing for so long ?

    • Offizieller Beitrag
    Zitat von sebeksd

    s there a way to make "omv-salt" more verbose so I could see what is it doing for so long ?

    Not particularly. It is in verbose mode when you don't use the quiet flag. I would disable monitoring (can enable later if you want it) in the web interface to start. And what is the output of: grep OMV_SHAREDFOLDERS_DIR_ENABLED /etc/default/openmediavault

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Zitat von ryecoaaron

    Not particularly. It is in verbose mode when you don't use the quiet flag. I would disable monitoring (can enable later if you want it) in the web interface to start. And what is the output of: grep OMV_SHAREDFOLDERS_DIR_ENABLED /etc/default/openmediavault

    Ok

    Zitat von ryecoaaron

    And what is the output of: grep OMV_SHAREDFOLDERS_DIR_ENABLED /etc/default/openmediavault

    OMV_SHAREDFOLDERS_DIR_ENABLED="NO"

    • Offizieller Beitrag

    ok. How about:

    sudo /usr/bin/salt-call --local saltutil.clear_cache

    sudo omv-salt stage run prepare

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Zitat von ryecoaaron

    ok. How about:

    sudo /usr/bin/salt-call --local saltutil.clear_cache

    sudo omv-salt stage run prepare

    Same thing. First command took over a minute (not sure how long exactly).

    Second one took similar time like always but result was slightly different


    Code
    saltutil.sync_modules:                                
- modules.omv_conf                                
- modules.omv_utils


    Full output:

    Few fresh info. I did some source code search for omv-salt and found some "salt-call" command call. This command seems to have debug option (found in Salt Stack documentation).

    So I executed this (I hope I didn't damage anything :P):

    salt-call -l debug --local --retcode-passthrough state.orchestrate omv.stage.prepare

    Loking at the output I saw two major "stops" (they are repeated few times):

    [DEBUG ] Elapsed time getting FQDNs: 40.413857221603394 seconds

    [DEBUG ] LazyLoaded zfs.is_supported

    It is stoping on both of them for quite a while (multiply times).


    There is also (every time before those two lines):

    [DEBUG ] /etc/resolv.conf: The domain and search keywords are mutually exclusive.


    Not sure why its trying to get FQDNs. I will probably try to search more tomorrow.

    Full log in attachment.

    First of all there was additional "problem" that I didn't mention (because it wasn't big of a deal), not only apply was slower but also web ui loading was "lagging", for example login page was loading for about 3 seconds, then after logging first page load for additional 3 seconds (other pages to, even "widgets" on dashboard was lagging). When I think about it samba shares also was lagging (first open). But 3 seconds is not very long so I was ignoring it. After fixing the problem all mentioned above start to working blazing fast (less then 1 sec ?)/.


    It's look like the problem is related to domain name/domain mismatch, I'm not sure what domain name I set during installation (probably "home") but I know that after some time I changed it in GUI (probably from this moment everything slowed down).

    After debugging salt-call I looked into /etc/resolve.conf to fix:

    [DEBUG ] /etc/resolv.conf: The domain and search keywords are mutually exclusive.

    And then I saw that in this file I had "home" domain (now I don't remember what I had in GUI). One thing Im sure is that on my router I have domain name "local" so I start changing it everywhere, first in GUI then in "resolv.conf" (GUI change did not overwrite this file so I changed it manually).


    Now after setting GUI domain an in resolv.conf everything work very fast, Apply take about 11s (from 250s), GUI is loading instantly and I think samba shares also sped up.


    Now strangest in all of that is that I was always using local IP (no names, always 192.168.1.200) so not sure why it was trying to resolve any think. I was opening GUI with HTTP (no SSL) also using IP.


    I have local DNS on PiHole (in docker on this exact server), my router point this DNS through DHCP.


    Zitat von votdev

    Could you please add the following line to /etc/salt/minion.d/openmediavault-test.conf and re-run your test. Is it faster now?


    Code
    enable_fqdns_grains: False

    Maybe your issue is related to https://github.com/saltstack/salt/pull/55581.

    votdev Do you still want me to check that ?

    I made few small test:

    1.

    GUI Domain = local,

    /etc/resolv.conf = local

    sudo omv-salt stage run prepare - time 11s


    2.

    GUI Domain = home,

    /etc/resolv.conf = local

    sudo omv-salt stage run prepare - time 130s


    3.

    GUI Domain = local,

    /etc/resolv.conf = local

    sudo omv-salt stage run prepare - time 11s


    4.

    GUI Domain = local,

    /etc/resolv.conf = home

    sudo omv-salt stage run prepare - time 26s


    5.

    GUI Domain = local,

    /etc/resolv.conf = local

    sudo omv-salt stage run prepare - time 11s


    6.

    GUI Domain = home,

    /etc/resolv.conf = home

    sudo omv-salt stage run prepare - time 253s



    Not sure if "/etc/resolv.conf" should change automatically after changing domain in GUI but at least on my server it stays unchanged.

    I have no idea why it is behaving this way but I'm not an "admin" :P

    • Offizieller Beitrag
    Zitat von sebeksd

    Not sure if "/etc/resolv.conf" should change automatically after changing domain in GUI but at least on my server it stays unchanged.

    If your output doesn't look like this, it won't change. If your output does look like this, it will change with changes in the GUI


    aaron@omv5dev:~$ ls -al /etc/resolv.conf

    lrwxrwxrwx 1 root root 32 Jan 13 2020 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    • Offizieller Beitrag
    Zitat von sebeksd

    I did not change any permissions to this files.

    Your /etc/resolv.conf is not a symlink. Fix that with:


    sudo rm /etc/resolv.conf && sudo ln -s /run/systemd/resolve/resolv.conf /etc/

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Einmal editiert, zuletzt von ryecoaaron ()

    • Offizieller Beitrag
    Zitat von sebeksd

    You made a typo "resolved" instead of "resolve"

    Thanks. I mix it up with the systemd-resolved service name. Fixed my post.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden