Huge network latency since 2 days

Im experiencing huge network latency and up and slow download performance, latency goes close to 2000ms. As soon as I disconnect OMV from network everything goes back to normal. Normal is ping = 11 to 17

Interesthingly since exactly these two days I can't change the domain, it end up in an endless loop.

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    895 root      20   0 2411228 196780   7808 S  58.8   0.6     33,22 systemd-worker
   1325 root      20   0  579904  10076   2700 S  23.5   0.0   4:01.98 rrdcached
 276918 root      20   0   13280   5760   3712 R  11.8   0.0   0:00.03 top
 277002 root      20   0  207844  14720  12160 R  11.8   0.0   0:00.02 rrdtool

Im also seeing high cpu demand, not sure if there is a relation.

I used omv-firstaid to reset network interface but it doesn't help.

Any ideas what's wrong with the system.

thanks

Im wondering if changing the domain in config.xml is a viable option. since it doesn't work in workbench

no I haven't changed anything, just thinking out loud

ryecoaaron

from what I observed.

1. Changing domain.

2. confirm setting (yellow banner)

3. spinning wheel for about 20-30 mins

4. back to srcreen with yellow banner

If

5. confirm setting again

6. after 20-30 mins, page 503. error

else

7. revert setting

it looks to me that the changes can't be don't in the modules down the line. and the local domain doesn't get changed.

this is how hosts looks like at step 4. Avahi looks similar no change.

oot@omv:/etc# cat hosts
# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.
127.0.0.1 localhost.localdomain localhost
127.0.1.1 omv.local omv
# The following lines are desirable for IPv6 capable hosts.
# Add hostname to ::1 if there is no other IPv6 interface. This is
# necessary to properly resolve the hostname, otherwise building the Salt
# grains (core.fqdns and core.ip_fqdn) will take a very long time.
::1 localhost ip6-localhost ip6-loopback omv.local omv
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

I just pulled the sysinfo from omv-firstaid, this is what I found:

================================================================================

= Hostname

================================================================================

omv.hq omv omv omv omv omv omv omv omv omv.hq omv

looks like a mess in my eyes

I changed the domain but didnt applied.

here is the output:

root@omv:/# cat /var/lib/openmediavault/dirtymodules.json
[
    "avahi",
    "postfix",
    "proftpd",
    "hostname",
    "hosts",
    "samba",
    "systemd-networkd"
]root@omv:/# 

I meant that the changed to be made in those modules might not happen, therefore the system hung

this

proftpd

is the bad guy. here omv stucks. so the ftp is the issue. the rest I haven't chked so far.

I tried with all my browser, Brave, Safari and Firefox always the same result. I know that service FTP is down but SFTP is up.

And this network delay is accross the hole network. Even Siri complains she has an issue with the internet connection.

Brief networkQuality summery:

netrunner@netsBoard 01 % networkQuality                           
==== SUMMARY ====
Uplink capacity: 26.687 Mbps
Downlink capacity: 90.157 Mbps
Responsiveness: Low (455.614 milliseconds | 131 RPM)
Idle Latency: 746.500 milliseconds | 80 RPM

usually I have up: 36, down: 105 and lactency of around 12-17.

Its a whole questiion mark in my head why this is happening.

I stopped the FTP Service, hence I could change the domain name. But unfortunatly, the network issue is still persistant. Even after reboot.

attached network test:

in your initial post you said something about same ip for multiple systems. I reseted my router to factory setting in order to see if I screw something in the settings, and I've changed from static ip everything to DHCP. but no success so far.

but one thing is still strange, even after being able to change domain name, I cant navigagte to omv-workbench with omv.hq:85 (port 85 because of npm proxy), i still need to use omv.local:85 to access workbench.

ryecoaaron

I had to make a break from this issue for a day.

By accident, I was flashing a router with OpenWRT, I checked the system protocol, I found out literally thousands of conntrack, table full dropping packet entries.

 Sep 29 12:30:05 2024 kern.warn kernel: [41956.550572] nf_conntrack: nf_conntrack: table full, dropping packet
Sun Sep 29 12:30:05 2024 kern.warn kernel: [41956.550732] nf_conntrack: nf_conntrack: table full, dropping packet
Sun Sep 29 12:30:05 2024 kern.warn kernel: [41956.550757] nf_conntrack: nf_conntrack: table full, dropping packet

I ran immediately conntrack -L on OMV and found close to 80k entries.

tcp      6 246 ESTABLISHED src=192.168.1.133 dst=200.14.132.247 sport=32868 dport=22 src=200.14.132.247 dst=192.168.1.133 sport=22 dport=32868 [ASSURED] mark=0 use=1
tcp      6 115 SYN_SENT src=192.168.1.133 dst=84.144.241.12 sport=57282 dport=5555 [UNREPLIED] src=84.144.241.12 dst=192.168.1.133 sport=5555 dport=57282 mark=0 use=1
tcp      6 50 TIME_WAIT src=192.168.1.133 dst=194.156.103.217 sport=52374 dport=22 src=194.156.103.217 dst=192.168.1.133 sport=22 dport=52374 [ASSURED] mark=0 use=1
conntrack v1.4.7 (conntrack-tools): 79689 flow entries have been shown.

Not sure what this all means but most likely this is not as it supposed to be. So I need to go to the OpenWRT Forum and see if some experts help me overthere.

Definetly its call the world and its mostly port 22.

tcpdump: listening on br-lan, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:40:44.969791 IP (tos 0x0, ttl 64, id 47759, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.133.60440 > 181.250.8.232.22: Flags [S], cksum 0x44c7 (correct), seq 3812070887, win 64240, options [mss 1460,sackOK,TS val 26746962 ecr 0,nop,wscale 7], length 0
13:40:44.972074 IP (tos 0x0, ttl 64, id 54717, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.133.43022 > 75.110.166.133.22: Flags [S], cksum 0x54a4 (correct), seq 860516358, win 64240, options [mss 1460,sackOK,TS val 4191878905 ecr 0,nop,wscale 7], length 0
13:40:44.973254 IP (tos 0x0, ttl 64, id 21644, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.133.40518 > 5.215.203.223.22: Flags [S], cksum 0x0143 (correct), seq 3053720998, win 64240, options [mss 1460,sackOK,TS val 2999622422 ecr 0,nop,wscale 7], length 0
13:40:44.973273 IP (tos 0x0, ttl 64, id 25367, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.133.48678 > 44.17.82.6.22: Flags [S], cksum 0xe8e9 (correct), seq 138685906, win 64240, options [mss 1460,sackOK,TS val 2266736754 ecr 0,nop,wscale 7], length 0
13:40:44.973333 IP (tos 0x0, ttl 64, id 61205, offset 0, flags [DF], proto TCP (6), length 60)

I'll keep u posted