Network Saturation Spikes - what's going on here?

McDuff · 23. November 2017

I noticed some weird intermittent problems on my network. At first I thought there was a problem with the router, but then I checked the server usage graphs and saw that the OMV box was periodically saturating the network connection to an absurd degree.

On the hourly graph the baseline noise at the bottom is me doing some file copying. The spikes are something entirely different. Somehow the server has sent out 335 Tb of network traffic this month, in these random spikes.

Has anyone seen this before? Any idea what's causing it?

g2_ufo · 24. November 2017

install iftop: apt-get install iftop

see iftop output in ssh console

McDuff · 24. November 2017

That doesn't really tell me anything. When the spikes happen they are to random IP addresses that I can't ping. There's no way it's actually sending any meaningful data at that rate.

tkaiser · 24. November 2017

Zitat von McDuff

That doesn't really tell me anything

Then you should prepare for the worst: your OMV server is part of a botnet performing DDOS attacks. It needs to be set up from scratch immediately!

Seriously: check what services are installed and which service is talking to 'random IP addresses'. When such a network spike occurs check with the following in parallel:

Code

lsof -i

On my smallest server this happens all the time when somewhere on this planet an 'Armbian image' is downloaded via torrents for example.

McDuff · 24. November 2017

See here for eg. That IP address comes back as destination unreachable when I ping it. But it's a different IP each time.

McDuff · 24. November 2017

OK. That sucks.

How can I prevent that in the future? It's almost a straight out of the box install, I've not installed anything except plugins, how did it get compromised?

tkaiser · 24. November 2017

Zitat von McDuff

how did it get compromised?

I was just kidding. Without some investigation it's impossible to tell. But according to iftop there's a lot of data going out.

To check individual hosts eg. the one above this is suitable:

Code

whois 103.60.164.15
host 103.60.164.15
dig ptr 15.164.60.103.in-addr.arpa

But as already said: 'lsof -i' will tell you also which program/daemon is talking to which address. And that might be important to get an idea what's going on.

subzero79 · 25. November 2017

Zitat von McDuff

OK. That sucks.

How can I prevent that in the future? It's almost a straight out of the box install, I've not installed anything except plugins, how did it get compromised?

On most cases here seen in the forum users have reported to have ssh exposed in wan using port 22 with password authentication. I understand arm images come with ssh enabled and root password predefined. So before any port nat in the router just change to key authentication and elevate the port at the wan line.
On some old threads there were users that reported compromised servers by having plex exposed.
A common pattern of virus seen here a process with random alphanumeric characters sucking all cpu and network.
Just check with top and nethogs.

tkaiser · 25. November 2017

Zitat von subzero79

I understand arm images come with ssh enabled and root password predefined

In the meantime while the above is still true 'permit root login' is disabled by default and when root is logging in the first time a new password has to be chosen (with distro policies in place). So this sort of 'backdoor' is now fixed on all ARM images (since August, on RPI image since Oct IIRC and @ryecoaaron just recently deleted the older RPi image from SF that might have still shown old behaviour)

Jetzt mitmachen!