Something awful happened to my OMV and I don't know what

  • Hi Guys,

    I need your support to understand what happened to my OMV NAS. It's a basic config, 2x 3TB drives + old 128GB ssd for the OS.


    z0ntf7G.png


    Please notice that I don't use my NAS for 24/7 use, but I usually turn it on/off when needed (Plex, SMB backup, some torrent, etc).

    I've been using OMV for almost 2 years and I haven't had a single problem. To be frank, I've set it up and tweaked it in the first 3-4 months and then proceeded to do only small updates.


    Long story short, I was barely home the last month. I left my NAS on for a couple of days to download a poorly seeded torrent. Yesterday I tried to access my folders and it's all empty. The 2x 3TB drives seem to be erased completely:


    4ulmjdQ.png


    SSH was enabled with "Permit root login" on (yep, apparently I'm an idiot). The other services running are MiniDLNA, CIFS/SMB and Portainer running Plex, Qbittorrent, Krusader.

    Also, since the last couple of months I changed my service provider and I'm now under a Public Dynamic IP. I feel like this last detail might be the main issue.


    This shows well when the "accident" happened, around July 15:


    N1BfCd2.png


    What could have happened? Malicious attack? Is there some sort of protection on OMV that could have triggered a disk erasure? An OMV malfunction?

    I now that this is full of rookie mistakes, I feel sh*tty

  • KM0201

    Approved the thread.
  • Yikes.


    Is SSH allowed from outside your network and do you have a crappy password for root (people always call me paranoid for this.. this is why I don't have sudo users as well)


    Second thought is a CIFS client did something intentional or stupid. Maybe not realizing they were deleting files from the server, and went a little crazy deleting files? Is CIFS enabled without a password? If your network was compromised and they managed to mount your CIFS share...


    Final thought, is Plex. Is Plex allowed off your network, or is it inside your network only? Is Plex allowed to delete files? There's been situations in the past where Plex (and Emby for that matter) accidentally deleted a library, usually due to developer error. This is why it is usually recommended you only give Plex, Emby, Jellyfin read only access to your media.


    The IP didn't help any, but if your system was compromised it was likely due to poor security. Unfortunately being 17 days ago it's hard telling what happened.


    So starting with SSH (as root)... This will show any failed SSH attempts


    Code
    cat /var/log/auth.log | grep 'sshd.*Invalid'

    This will show successful attempts... maybe see if something looks weird.


    Code
    cat /var/log/auth.log | grep 'sshd.*opened'

    Then maybe check your CIFS logs in the Logs section of the webUI..


    After that, I'd check Plex (not sure on it's log location) and see if it maybe deleted a bunch of stuff.

    Air Conditioners are a lot like PC's... They work great until you open Windows.


    Edited once, last by KM0201 ().

  • What could have happened? Malicious attack? Is there some sort of protection on OMV that could have triggered a disk erasure? An OMV malfunction?

    I now that this is full of rookie mistakes, I feel sh*tty

    I can't imagine any scenario that OMV would have done this. If I were a gambling man, I'd say someone got access to your Plex account (especially if it's open to the Internet) and did this. Second, is one of your clients was compromised and they done this from there either via SSH or a mounted SMB share.

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • SSH was allowed from inside my network only, but for the past 2 months the dynamic IP was public, can that be the issue?


    I live alone, nobody else has access to my PC & NAS, that's why I was too confident of the security. The password was not too easy, but definitely brute-forceable.


    Plex was in a container, installed using the TechnoDad guide (

    ). As far as I understand (which is not much at this point) it should be able to read/write only onto the folders it had access, definitely nothing beyond the library (certainly not onto the other drive I have).


    I don't think it has been the CIFS client. I'm the only one using it, from a PC that was off at the time of the accident.


    I've found the authentication logs from yesterday, the data was already gone, but look at this:


  • Well, a CIF client could be the client machine was compromised, or if your NETWORK was compromised (maybe a crappy wifi password, etc.).. and they were able to get on your network and mount the share.


    That SSH log is pretty troubling.. Not sure how they'd have gotten there though if you don't have Port 22 open on your router.... Does that IP address look familiar? 209.141.61.174 as that is where all the hits are coming from apparently... Seems they just kept trying random usernames/passwords, but again w/o Port 22 open in your router, I'm pretty sure this had to be done within your network.


    The technodad video is a good guide, but it doesn't (nor should it) tell you every single thing you need to know...


    Just for reference (although from that log I'm not sure Plex is at fault). By making the Plex, Emby, Jellyfin containers have read only access.. IF someone gets access to the UI of them (compromised user/admin password, etc.).. then they still can't delete data.


    PSA for those using Docker and Emby


    That thread links where this has happened on both Emby and Plex.. the Emby thread is dead, but the OP of it started another thread that is archived on Reddit (he was a little upset)....


    https://www.reddit.com/r/emby/…deletes_entire_libraries/

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • massenzio did you require username/passwords to mount your CIFS shares or were they set up as guest shares? (another pet peeve of mine)... if you had a crummy wifi password or someone somehow got on your network... if it was a guest share, they would have been able to mount it no problem.


    Since the CIFS logs aren't showing much however... and you've got a lot going on on that SSH log.. I'd say that is the culprit.


    Maybe ryecoaaron  macom or geaves has an idea.

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • There are many different IPs in the logs, spreaded across the world, to be honest I don't recognize any of them...

    The port 22 is closed in my router. The only enabled one are 80 and 443 for nextcloud -- but I never used nextcloud much, the container itself has been "stopped" for months.


    Plex can delete things beyond the library? I don't think so. The only folder it was allowed to see is not the only one deleted...


    No, I didn't require user/psw for the CIFS mounts, again I am the only one using the network and the only CIFS client...

  • Yeah clearly someone wanted something on your network (they could be usin g a VPN and just as easily be your neighbor rather than wherever the world the IP is located).


    Plex isn't supposed to, but it has happened with Plex in the past. Again usually due to a programming error/bug that wasn't caught, or maybe the person setting it up only set up one "main" share that had everything under it.. you can always be sure someone will figure out a way. Again, this is why they strongly recommend Plex have read only access to your data. Did Plex have access to both drives?


    I understand what you're saying about you being the only user, but if the person somehow was able to join your network, well then they could mount guest shares w/o issue and do whatever they wanted.

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • According to your logs, port 22 is not closed off from the internet. If it was, you wouldn't see people trying to login to sshd from outside addresses. I highly recommend closing the port and/or changing to a non-standard port number.


    You should have old auth.log files that were rotated. What is the output of: ls -al /var/log/auth.log*

    omv 5.6.13 usul | 64 bit | 5.11 proxmox kernel | omvextrasorg 5.6.2 | kvm plugin 5.1.6
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • According to your logs, port 22 is not closed off from the internet. If it was, you wouldn't see people trying to login to sshd from outside addresses. I highly recommend closing the port and/or changing to a non-standard port number.


    You should have old auth.log files that were rotated. What is the output of: ls -al /var/log/auth.log*

    That's what I thought, but was 100% sure.

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • KM0201 a neighbor accessing to my network? That's scarier than any botnet to be honest...



    According to your logs, port 22 is not closed off from the internet. If it was, you wouldn't see people trying to login to sshd from outside addresses. I highly recommend closing the port and/or changing to a non-standard port number.


    You should have old auth.log files that were rotated. What is the output of: ls -al /var/log/auth.log*


    How do I check if port 22 is closed or not? From the router I saw only 80 and 443 and are now closed.


    The output of that command is simply "No such file or directory"

  • How do I check if port 22 is closed or not?

    I don't know anything about your network or router. So, hard to say. Changing the port number might be easier.


    The output of that command is simply "No such file or directory

    I have a feeling you have a typo then. What is the output of: ls -al /var/log

    omv 5.6.13 usul | 64 bit | 5.11 proxmox kernel | omvextrasorg 5.6.2 | kvm plugin 5.1.6
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • I'm not saying a neighbor did it, just saying with a VPN they could look like they were somewhere else in the world, and be right across the street.

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • I don't know anything about your network or router. So, hard to say. Changing the port number might be easier.


    I have a feeling you have a typo then. What is the output of: ls -al /var/log

    I was using the wrong command indeed. Here's the output of the first command:


    Code
    :~# ls -al /var/log/auth.log*
    -rw-r----- 1 root adm 2743768 Aug 2 14:54 /var/log/auth.log
    -rw-r----- 1 root adm 6846685 Aug 1 00:41 /var/log/auth.log.1
    -rw-r----- 1 root adm 503656 Jul 11 00:00 /var/log/auth.log.2.gz
    -rw-r----- 1 root adm 4417 Jul 5 23:53 /var/log/auth.log.3.gz
    -rw-r----- 1 root adm 344431 Jun 27 22:08 /var/log/auth.log.4.gz
  • Code
    -rw-r----- 1 root adm 503656 Jul 11 00:00 /var/log/auth.log.2.gz

    Since that one sees to be around the approx. date this happened...


    Code
    zcat -rw-r----- 1 root adm 503656 Jul 11 00:00 /var/log/auth.log.2.gz

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • I would look through those logs to see if there were any successful logins from the internet. The following should show you logins from july 11th to August 1st.


    sudo less /var/log/auth.log.1

    omv 5.6.13 usul | 64 bit | 5.11 proxmox kernel | omvextrasorg 5.6.2 | kvm plugin 5.1.6
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • Here's a sample of the file, it's like this for lines and lines:

  • Been following this topic to be more mindfull about security. (Although I try to block all I can think off).

    massenzio I'm thinking that you don't have the "fail2ban" service active, correct?


    ryecoaaron Would the "fail2ban" service be enough to prevent the OP situation? I'm almost certain that my port 22 is blocked on my router but am trying to cover all angles I can think of.

    Also, the first thing I do, when configuring OMV ssh access is to disallow "root" access.


    Maybe, crashtest can make an review on the install guide and focus some topics to this issue : a "checklist" of some of the most common "failures" that newbies or not-so-knowlegeable users might have that will lead to something as it happened to OP.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!