Big Data Loss with OMV and Snapraid

ryecoaaron · 24. November 2014

Let me boot sysresccd real quick...

ryecoaaron · 24. November 2014

Hmm... That works on my VM. What version of sysresccd? What is the output of:

extundelete --version
mount

This is what it looks like in my VM.

Code

root@sysresccd /mnt/backup/recover % extundelete --restore-all /dev/sdg1
NOTICE: Extended attributes are not restored.
Loading filesystem metadata ... 64 groups loaded.
Loading journal descriptors ... 1 descriptors loaded.
Searching for recoverable inodes in directory / ...
0 recoverable inodes found.
Looking through the directory structure for deleted files ...
0 recoverable inodes still lost.
No files were undeleted.
root@sysresccd /mnt/backup/recover % extundelete --version
extundelete version 0.2.4
libext2fs version 1.42.10
Processor is little endian.
root@sysresccd /mnt/backup/recover %

Alles anzeigen

writzthecat · 24. November 2014

I used this iso :SystemRescueCd-x86-4.4.1

Code

root@sysresccd /mnt/backup/recover/disk1 % extundelete --version

Code

mount
rootfs on / type rootfs (rw)
udev on /dev type tmpfs (rw,relatime,size=10240k,mode=755)
/dev/sr0 on /livemnt/boot type iso9660 (ro,relatime,mode=0644)
/dev/loop0 on /livemnt/squashfs type squashfs (ro,relatime)
tmpfs on /livemnt/memory type tmpfs (rw,relatime)
none on / type aufs (rw,noatime,si=5bb6167003aefe61)
tmpfs on /livemnt/tftpmem type tmpfs (rw,relatime,size=524288k)
none on /tftpboot type aufs (rw,relatime,si=5bb6167003aeae61)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,size=1640680k,mode=755)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=2047722,mode=755)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib/rc/sh/cgroup-release-agent.sh,name=openrc)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
memory on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
net_cls on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
blkio on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
perf_event on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
tmpfs on /tmp type tmpfs (rw)
/dev/sdb1 on /mnt/backup type ext4 (rw)

Alles anzeigen

ryecoaaron · 24. November 2014

extundelete didn't show any output?

writzthecat · 24. November 2014

Yep,

no output.

But i guess i found the problem. my virtual Device with SystemRescueCd disconnected....

Im now trying it again and restarted the server, booting SystemrecueCD with docache command

Will i have to check for the disk again or will they still be the same after restart ?

ryecoaaron · 24. November 2014

You might have to check that the drive letters are the same. They should be but you don't want to get that wrong. Post blkid if you are unsure.

Code

mount /dev/sdb1 /mnt/backup
cd /mnt/backup/recover/disk1
extundelete --restore-all /dev/sde1
cd ../disk2
extundelete --restore-all /dev/sdd1
cd ../disk3
extundelete --restore-all /dev/sdc1

writzthecat · 24. November 2014

blkid

Code

/dev/loop0: TYPE="squashfs"
/dev/sda1: UUID="be149662-abd5-4d5d-b532-14d091321812" TYPE="ext4" PARTUUID="000397bf-01"
/dev/sda5: UUID="5d31da78-38c9-48e7-8aa0-bfa5c4b18235" TYPE="swap" PARTUUID="000397bf-05"
/dev/sdb1: LABEL="Backup" UUID="c7692c8e-2a3e-4ac2-b954-29c71ac43df9" TYPE="ext4" PARTUUID="abbbbcb7-4831-4c62-b4ce-b8eefa757578"
/dev/sdc1: LABEL="DISK3" UUID="858099db-5680-44d7-8cd3-79134be2157c" TYPE="ext4" PARTUUID="571fd11d-2fe9-46a0-a45e-32bfd5a677ef"
/dev/sdd1: LABEL="DISK2" UUID="2f914808-85f6-4b9b-aee3-43cfad784960" TYPE="ext4" PARTUUID="e563015a-431e-488a-93ea-30960db9df19"
/dev/sde1: LABEL="DISK1" UUID="0722de2e-cf4b-4a63-aece-f01f502b7c27" TYPE="ext4" PARTUUID="da3e312c-be1c-42b3-bd40-810f5be6ce4b"
/dev/sdf1: LABEL="Parity1" UUID="a7f543ef-6a8f-41f9-ac83-942085c9bfca" TYPE="ext4" PARTUUID="c206d942-0953-4b93-bb1b-2239a8c42fe7"
/dev/sr0: UUID="2014-11-17-18-17-22-00" LABEL="sysrcd-4.4.1" TYPE="iso9660"

Seems to be the same

ryecoaaron · 24. November 2014

Yes, they are.

writzthecat · 24. November 2014

It's working, but a lot of "Unable to restore inode"

At least i now know how to be better prepared for the future :-/

ryecoaaron · 24. November 2014

Unfortunately, the snapraid fix command may have written the recovered files over deleted files. You can always try photorec. It just doesn't restore the filename or directory.

When I upgrade my server at home, I won't be running any raid or parity program. Just backup using rsnapshot to second server that is only on during backup.

Solo0815 · 24. November 2014

one question:
in your snapraid.conf you excluded "/storage/". I hope your files where NOT in "/storage/"? Because then they can't be restored

writzthecat · 24. November 2014

Hello Solo,

/storage/ is the pool folder created by aufs.

So the content of this folder is splitted onto three disk in the folders D1 D2 and D3.

Therefore i thought i have to exclude /storage/

Solo0815 · 24. November 2014

Ah, OK
Disk 1, 2 and 3 were protected by snapraid, you didn't have to exclude it. Snapraid detects that the drives are covered by another conf-entry

ryecoaaron · 24. November 2014

I would think you would be ok to do that. I would probably put the pool folder on the parity drive though. The next version of the aufs plugin, you won't have to worry about it.

writzthecat · 24. November 2014

i excluded /storage/ because Snapraid always gaves me errors while syncing.

The idea putting the pool folder on the parity drive is a good one.

I will keep it in my mind.

BTW: extundelete was not able to restore any single file this is really bad

Solo0815 · 24. November 2014

Zitat von ryecoaaron

Unfortunately, the snapraid fix command may have written the recovered files over deleted files. You can always try photorec. It just doesn't restore the filename or directory.

Maybe thats why extundelete can't restore anything. Do you have any output example of extundelete?

writzthecat · 24. November 2014

with all the files its: Unable to restore inode "filename" reallocationg free disk space

ryecoaaron · 24. November 2014

I mounted a partition, copy grub.cfg to the partition twice, deleted the files, and unmounted. I ran extundelete and got the following:

Code

root@sysresccd /mnt/backup % extundelete --restore-all /dev/sdg1
NOTICE: Extended attributes are not restored.
Loading filesystem metadata ... 64 groups loaded.
Loading journal descriptors ... 16 descriptors loaded.
Searching for recoverable inodes in directory / ...
1 recoverable inodes found.
Looking through the directory structure for deleted files ...
0 recoverable inodes still lost.
root@sysresccd /mnt/backup % ls
lost+found  recover  RECOVERED_FILES
root@sysresccd /mnt/backup % cd RECOVERED_FILES
root@sysresccd /mnt/backup/RECOVERED_FILES % ls
grub2.conf
root@sysresccd /mnt/backup/RECOVERED_FILES %

Alles anzeigen

Only found one copy.

Tried photorec and it found one copy as well.

writzthecat · 26. November 2014

Hm..

i tried it two times with extundelete and it has restored only a few files.

I now did it with Photorec, and it restored "maybe" everything. But it will take a long time for me to sort all the media files, as they have no name and no directory....

ryecoaaron · 26. November 2014

This page might help with sorting/renaming the recovered files.

Jetzt mitmachen!