Missing Data on Raid1 RPi

Recently my raspberry Pi power was disconnected without warning. When it powered back on, it rebooted in 'emergency mode'. After going through the steps to comment out the extra drive within the /etc/fstab file, I was able to boot up the Pi but my OMV RAID 1 would not mount. When tried to mount I was prompted with an error message. After browsing the Internet I ran fsck on the /dev/md0 mount and it completed successfully. My RAID 1 array was now able to be mounted but I could not see any of my data. The drive space of the array was also correctly reporting so I know there is data on the array. I was hoping you can help me with walking me through how to make my data browsable again. After fixing this, I plan on copying the data to a separate drive not using the RAID setup anymore. I will use Rsync instead.

Here is my output from the commands:

pi@raspberrypi:~ $ sudo cat /proc/mdstat

Personalities : [raid1]

md0 : active raid1 sdb1[0]

3906885440 blocks super 1.2 [2/1] [U_]

bitmap: 0/30 pages [0KB], 65536KB chunk

unused devices: <none>

pi@raspberrypi:~ $ sudo blkid

/dev/mmcblk0p1: LABEL_FATBOOT="boot" LABEL="boot" UUID="4BBD-D3E7" TYPE="vfat" PARTUUID="738a4d67-01"

/dev/mmcblk0p2: LABEL="rootfs" UUID="45e99191-771b-4e12-a526-0779148892cb" TYPE="ext4" PARTUUID="738a4d67-02"

/dev/sda1: UUID="4e020b87-9fc1-db8f-613d-8c256e9f486d" UUID_SUB="6dde817d-aeb2-4cba-818d-0e9cfc4841b6" LABEL="raspberrypi:0" TYPE="linux_raid_member" PARTUUID="a7b373ff-8971-4029-b3fd-987efc076ecb"

/dev/sdb1: UUID="4e020b87-9fc1-db8f-613d-8c256e9f486d" UUID_SUB="15ebe1f2-5611-0d88-3e1e-e014279281ec" LABEL="raspberrypi:0" TYPE="linux_raid_member" PARTUUID="a25a896f-6ba8-be48-8d08-d501dc308228"

/dev/md0: UUID="f7f6036d-aab8-49e7-8f45-2ac1cc870e02" TYPE="ext4"

/dev/mmcblk0: PTUUID="738a4d67" PTTYPE="dos"

pi@raspberrypi:~ $ sudo fdisk -l | grep "Disk "

The backup GPT table is corrupt, but the primary appears OK, so that will be used.

Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram1: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram2: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram3: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram4: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram5: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram6: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram7: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram8: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram9: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram10: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram11: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram12: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram13: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram14: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/ram15: 4 MiB, 4194304 bytes, 8192 sectors

Disk /dev/mmcblk0: 29.8 GiB, 32010928128 bytes, 62521344 sectors

Disk identifier: 0x738a4d67

The backup GPT table is corrupt, but the primary appears OK, so that will be used.

Disk /dev/sdb: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: nal USB 3.0

Disk identifier: 95273E32-0319-6240-8C0F-0DCAE21C8092

Disk /dev/md0: 3.7 TiB, 4000650690560 bytes, 7813770880 sectors

Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: nal USB 3.0

Disk identifier: 9EF89B64-6007-4015-B522-97DCDA041367

pi@raspberrypi:~ $ sudo cat /etc/mdadm/mdadm.conf

# This file is auto-generated by openmediavault (https://www.openmediavault.org)

# WARNING: Do not edit this file, your changes will get lost.

# mdadm.conf

#

# Please refer to mdadm.conf(5) for information about this file.

#

# by default, scan all partitions (/proc/partitions) for MD superblocks.

# alternatively, specify devices to scan, using wildcards if desired.

# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.

# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is

# used if no RAID devices are configured.

DEVICE partitions

# auto-create devices with Debian standard permissions

CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system

HOMEHOST <system>

# definitions of existing MD arrays

ARRAY /dev/md0 metadata=1.2 name=raspberrypi:0 UUID=4e020b87:9fc1db8f:613d8c25:6e9f486d

pi@raspberrypi:~ $ sudo mdadm --detail --scan --verbose

ARRAY /dev/md0 level=raid1 num-devices=2 metadata=1.2 name=raspberrypi:0 UUID=4e020b87:9fc1db8f:613d8c25:6e9f486d

devices=/dev/sdb1

Soma - It is two independent USB HDDs.

ryecoaaron

Thanks for replying.

Here is the output:

pi@raspberrypi:~ $ sudo mount -l

/dev/mmcblk0p2 on / type ext4 (rw,noatime,nodiratime) [rootfs]

devtmpfs on /dev type devtmpfs (rw,relatime,size=1827468k,nr_inodes=97606,mode=755)

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

proc on /proc type proc (rw,relatime)

securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)

tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)

tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)

tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)

cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)

cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)

none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)

cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)

cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)

cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)

cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)

cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)

cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)

cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)

cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)

debugfs on /sys/kernel/debug type debugfs (rw,relatime)

sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)

mqueue on /dev/mqueue type mqueue (rw,relatime)

systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=33,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)

nfsd on /proc/fs/nfsd type nfsd (rw,relatime)

configfs on /sys/kernel/config type configfs (rw,relatime)

tmpfs on /tmp type tmpfs (rw,relatime)

/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro) [boot]

/dev/mmcblk0p2 on /var/folder2ram/var/log type ext4 (rw,noatime,nodiratime) [rootfs]

folder2ram on /var/log type tmpfs (rw,relatime)

/dev/md0 on /srv/dev-disk-by-id-md-name-raspberrypi-0 type ext4 (rw,relatime,stripe=64,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group)

/dev/mmcblk0p2 on /var/folder2ram/var/tmp type ext4 (rw,noatime,nodiratime) [rootfs]

folder2ram on /var/tmp type tmpfs (rw,relatime)

/dev/mmcblk0p2 on /var/folder2ram/var/lib/openmediavault/rrd type ext4 (rw,noatime,nodiratime) [rootfs]

folder2ram on /var/lib/openmediavault/rrd type tmpfs (rw,relatime)

/dev/mmcblk0p2 on /var/folder2ram/var/spool type ext4 (rw,noatime,nodiratime) [rootfs]

folder2ram on /var/spool type tmpfs (rw,relatime)

/dev/mmcblk0p2 on /var/folder2ram/var/lib/rrdcached type ext4 (rw,noatime,nodiratime) [rootfs]

folder2ram on /var/lib/rrdcached type tmpfs (rw,relatime)

/dev/mmcblk0p2 on /var/folder2ram/var/lib/monit type ext4 (rw,noatime,nodiratime) [rootfs]

folder2ram on /var/lib/monit type tmpfs (rw,relatime)

/dev/mmcblk0p2 on /var/folder2ram/var/cache/samba type ext4 (rw,noatime,nodiratime) [rootfs]

folder2ram on /var/cache/samba type tmpfs (rw,relatime)

tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=391912k,mode=700,uid=1000,gid=1000)

I am still unable to browse or see any files or folders on the RAID array. Even though the RAID is mounted.

When I try to create a new Shared Folder, when I browse to the path with the /dev/md0/ device selected, it does not list any folders.

Any ideas on how I can restore making these files visible/browsable again? I would then just copy them to another drive and break the RAID and start over.

geaves - I executed 'sudo mount -a' but it did not provide any output so I ran the mount list command.

ryecoaaron - I disconnected one of the drive to run TestDisk on it to see if I can recovery any of the data.

pi@raspberrypi:~ $ sudo ls -al /srv/dev-disk-by-id-md-name-raspberrypi-0

ls: cannot access '/srv/dev-disk-by-id-md-name-raspberrypi-0/lost+found': Bad message

total 8

drwxr-xr-x 3 root root 4096 Apr 27 13:53 .

drwxr-xr-x 7 root root 4096 Apr 12 2020 ..

d????????? ? ? ? ? ? lost+found

When I run TestDisk it states the filesystem is damaged.

Is there anyway I can repair the filesystem?

geaves - Thank you, I did not know that.

ryecoaaron - I do not recall executing that command. Is that part of the fsck process? Could I do something in order to rebuild the filesystem? TestDisk allows you to rebuild the filesystem using the SuperBlock. Is that something I should do on one of the drives?

I powered down my Pi and my external drive bays. I powered back on my external drive bays and then my Pi. I confirmed within OMV the RAID was there, but it was marked as degraded due to me disconnecting one of the drives before. Do you know how I can enable the mirroring of the drives? Or does this happen automatically?

I ran sudo mount -a  and this time I received the following output:

mount: /srv/dev-disk-by-id-md-name-raspberrypi-0: mount(2) system call failed: Structure needs cleaning.

Any suggestions on what commands I should run to clean the structure?

Thank you in advance for all the help, everyone. I am new at this and I am finding I am running into a lot of issues which I am learning quickly to avoid in the future.

I ran the following commands again and luckily the lost+found directory was created with all of my data within them.

pi@raspberrypi:~ $ sudo fsck -C -p /dev/md0

fsck from util-linux 2.33.1

/dev/md0 was not cleanly unmounted, check forced.

/dev/md0: 94140/244187136 files (2.0% non-contiguous), 765759631/976721360 blocks

pi@raspberrypi:~ $ sudo mount -a

I am very lucky and I am immediately copying this data off to a different drive.

Thank you all for the help and support!