Help: OMV decided to delete most of my data.

endstille · 19. Mai 2013

Hey,

i was just watching a DVD, when it suddelny stopped...
then i tried to browse the share but it failed...

i logged in to the admin console and realied, that the system has heavy load...
then in ssh / htop i saw the following process:

after killing it, my disk looked like this:

You can also see the load ...

the server is still running.
what can i do, to recover my data? according to my calc, arrount 1755GB have been wiped within no time ...
i have no baclup, as i never predicted this kind of software-failure!
i just run raid5, to protect against hardware-failure....

thanks in advance for any help.
the server is still running

davidh2k · 19. Mai 2013

First, run this command:

Code

cat /root/.bash_history | grep "rm -rf"

if that spits out nothing, try

Code

cat /root/.bash_history | grep "rm"

i would like to know what command was executed there as it seems to be no simple "rm -rf /media/..." but specific files to be deleted.

Also which filesystem is your raid, and hardware or software raid?

Greetings
David

PS: also it is recommended to turn off your system right away to prevent that the data gets overwritten.

endstille · 19. Mai 2013

Thanks for your answer, David.
I already remounted the FS as RO, so further loss should be prevented...
I can see no evidence in the bash_history:
root@hades:/var/log# mount -o remount,ro /media/4a7464c6-2efa-4e72-807c-6174bcd5085d/
root@hades:/var/log# cat /root/.bash_history | grep "rm -rf"
rm -rf
root@hades:/var/log# cat /root/.bash_history | grep "rm"
rm -rf
rm -r *
rm -r *
rm -r marc/
rm -r marc/
rm -r *
rm -r test/
rm -r *
rm -r *
rm -r .recycle/
rm -r Steam/
root@hades:/var/log#

These should be commands i used on my own.

The system is an Acer H340 Server.

i use the 1TB disk as system and the 3 2TBfiles as Data software RAID5

i have only 1 ext4 filesystem

i got some strange errormessages the last days from my cron, since i installed the last upgrade.
like this
/etc/cron.daily/openmediavault-samba-recycle:
xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option

today i got this (right after killing rm -rf)
/etc/cron.daily/openmediavault-samba-recycle:
xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
rm: cannot remove `media/4a7464c6-2efa-4e72-807c-6174bcd5085d/aquota.user': Operation not permitted
rm: cannot remove `media/4a7464c6-2efa-4e72-807c-6174bcd5085d/aquota.group': Operation not permitted
xargs: rm: terminated by signal 15

i have my samba recycles configured, to delete old data after 30 days... but it never deleted _everything_...

davidh2k · 19. Mai 2013

Sorry, i never used Samba-recycle, but would you mind taking youself a look into /var/log/samba/ if you find anything that could give you a lead to why the deleting was initiated?

I don't know of any Filesystem recoveries for Linux/Ext4 atm, but maybe some others can recommend something for you, to get your Data back. (I only used something a while ago for XFS, and that was under Windows ^^)

Greetings
David

endstille · 19. Mai 2013

Hmm, i cannot get any clue from the logs.
But at this time would like to share them.
Feel free to check them and tell me, what happend.
https://dl.dropboxusercontent.com/u/12896017/log.7z

is it poissible , that the files might not be deleted, but samba cannot access them any more (see syslog or daemon log).

Thank you very much.

davidh2k · 19. Mai 2013

So far i only found one error, but i don't know if that error coud be related to your data loss:

rlimit_max: rlimit_max (1024) below minimum Windows limit (16384)

Samba seems not to log so much as i think it could.

Greetings
David

endstille · 20. Mai 2013

Using extundelete i were able to recover 100-200 gb, but i don't need that stuff, as they are backups anyway, so i have the real data.
ext4magic cannot open the media.

root@hades:/mnt/recover# ext4magic /media/4a7464c6-2efa-4e72-807c-6174bcd5085d/ -j /mnt/journal/file -d /mnt/recover/
/media/4a7464c6-2efa-4e72-807c-6174bcd5085d/ Error 2133571364 while opening filesystem
ext4magic : EXIT_SUCCESS

root@hades:/mnt/recover# umount /media/4a7464c6-2efa-4e72-807c-6174bcd5085d/

root@hades:/mnt/recover# ext4magic /dev/md0 -j /mnt/journal/file -d /mnt/recover/
Filesystem in use: /dev/md0

ext4magic : EXIT_SUCCESS

root@hades:/mnt/recover# lsof | grep md0
lsof: WARNING: can't stat() ext4 file system /home/ftp/home (deleted)
Output information may be incomplete.
lsof: WARNING: can't stat() ext4 file system /export/android (deleted)
Output information may be incomplete.
md0_raid5 823 root cwd DIR 8,1 4096 2 /
md0_raid5 823 root rtd DIR 8,1 4096 2 /
md0_raid5 823 root txt unknown /proc/823/exe
jbd2/md0- 859 root cwd DIR 8,1 4096 2 /
jbd2/md0- 859 root rtd DIR 8,1 4096 2 /
jbd2/md0- 859 root txt unknown /proc/859/exe

root@hades:/mnt/recover#

If someone has some great ideas, i'm willing to try it.
i really cannot understand, how the system can have some trouble with any recyclebin and the decide to rage and delete everything that is on the filesystem.
i used omv and plugins only stably repositories and only GUI-settings, to prevent any mistake... and now all my data is gone...

endstille · 20. Mai 2013

update:
I got ext4 magic to work and if recovered few hundred Gig, but nothing that i needed
My Audio, Video and Software folders seem to be gone.
Any last recover-idea is highly appreciated.
Thanks you in advance.

davidh2k · 20. Mai 2013

You got a PM.

Greetings
David

knumsi · 20. Mai 2013

did you use the "expert mode" of ext4magic?

I had to recover my raid and wrote down how i solved it. The main thing was using expert mode of ext4magic like this:

Code

cd /tmp
wget http://prdownload.berlios.de/ext4magic/ext4magic-0.3.1.tar.gz
tar xfvz ext4magic-0.3.1.tar.gz
apt-get install build-essential e2fslibs-dev uuid-dev libblkid-dev libmagic-dev
cd /tmp/ext4magic
make clean
./configure --enable-expert-mode
make
make install

And then i used this command to revocer:

Code

ext4magic "/dev/md127" -s 4096 -n 4096000 -c -D -d /media/backupdisk

I wrote down how it worked for me (in german http://www.och-group.de/2013/0…dateisystem-raid5-retten/

endstille · 20. Mai 2013

Thank you both,

i'll check both answers, before i give up.
The German Blog makes it even easier for me, to follow

edit: i changed my fstab, to mount the ext4 share ro.
I just changed it that way: UUID=X /media/Y ext4 defaults,ro,acl,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 0 2
so i can continue the restore-attempts, even after shutting down the server.
seems like good idea, so far

KM0201 · 22. Mai 2013

Man... crazy.

Good luck tracking this down.

Tom · 23. Mai 2013

Zitat von "davidh2k"
First, run this command:
Code
cat /root/.bash_history | grep "rm -rf"
if that spits out nothing, try
Code
cat /root/.bash_history | grep "rm"
i would like to know what command was executed there as it seems to be no simple "rm -rf /media/..." but specific files to be deleted.

You won't find anything in his bash history, because from the screenshot you can see that the rm -rf was invoked by cron! Which is rather... worrying... Is this fixed in the new release, because otherwise I will be shutting down my box until it is.

knumsi · 23. Mai 2013

Zitat von "Tom"

You won't find anything in his bash history, because from the screenshot you can see that the rm -rf was invoked by cron! Which is rather... worrying... Is this fixed in the new release, because otherwise I will be shutting down my box until it is.

You are right that this was invoked by cron - its the automatic deletion of the recycle folder (if you use it)

Just disable the automatic recycling of the samba .recycle folder to be absolutely safe I do never trust any bash/program to automatically delete data I dont even trust myself, thats why i additionally use the .recycle

davidh2k · 23. Mai 2013

Lol Knumsi,

never to trust yourself is a good advice. You don't want to know how many times i screamed after a shift + delete under windows....

Greetings
David

knumsi · 23. Mai 2013

Zitat von "davidh2k"

Lol Knumsi,

never to trust yourself is a good advice. You don't want to know how many times i screamed after a shift + delete under windows....

Greetings
David

Thats the reason And before OMV i had a samba share without a .recycle ... That moment when you automatically hit yes on the "Are you sure" - question and you know: THAT WAS WROG °_0 That was the way i once "cleaned up" my Picture collection ... :-/

Always keep a backup

davidh2k · 23. Mai 2013

Zitat von "knumsi"

[...]
"cleaned up" my Picture collection ... :-/
[...]

Thats a good one

And yes, i definitly need a backup solution (for my personal stuff) and i think it is coming in the next few days. (2x2TB Raid1)

Greetings
David

Tom · 23. Mai 2013

Zitat von "knumsi"

You are right that this was invoked by cron - its the automatic deletion of the recycle folder (if you use it)

Just disable the automatic recycling of the samba .recycle folder to be absolutely safe I do never trust any bash/program to automatically delete data I dont even trust myself, thats why i additionally use the .recycle

OK, fair enough to disable the auto pruning. But is this bug Fixed? Is Volker aware, and is it fixed in 0.4.30?

openmediavault 0.4.30

Mantis 0000771: Samba recycle bin pruning script produces errors/warnings.
Please update your system via ‘System > Update Manager’ in the OpenMediaVault WebGUI. Please reload the WebGUI to let the changes installed by the update take effect.

endstille · 23. Mai 2013

According to the Blog, he planed to fix it with the next release (so .30).
http://blog.openmediavault.org/?p=995&cpage=1#comment-311741

However i think it could be marked a little bit more cleary like:
"Beware, the recent version can and will delete everything you have, if you use SAMBA Recyble bin. Please updated asap!!!"
But that is just my impression, as it happend to me.

Btw. got 3 SATA cables now .
There are enough ports on my PC's mainboard and enough power-cables, that i can connect all three disks.
I'll try to recover my data using David's tip, once i got some time...

davidh2k · 23. Mai 2013

Both Thumbs are pressed.

Greetings
David

Jetzt mitmachen!