Single raid1 split into two degraded ones

reddy · 22. August 2012

I have rather unusual setup, namely I have two external USB disks connected to old netbook configured in raid1 (mirror). I set it up and it worked well for ~2 days. Today, however, when I turned my NAS on instead one mirror on /dev/md0 I see two degraded ones /dev/md126 and /dev/md127. The same disks are connected to the same USB ports all the time. The only thing which comes to my mind is the fact I was playing with partitions on system disk (not touching the mirror, both disks were disconnected that time) according to the method 4 from http://forums.openmediavault.o…topic.php?f=10&t=192#p734 (but I haven't run "mdadm /dev/md100 --create --force --level=linear --raid-devices=1 /dev/sda3" instruction proposed there to create linear array, I just simply mounted new /dev/sda3 from the Filesytems tab in the UI) - I wonder if that could do something wrong to the raid? Please find below all my current configuration.

What happened and how to fix it?
How to prevent this in the future?

cat /proc/mdstat

Code

root@nas:/home/reddy# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6]
[raid5] [raid4] [raid10]
md126 : active raid1 sdc[0]
	  1953513424 blocks super 1.2 [2/1] [U_]


md127 : active (auto-read-only) raid1 sdb[1]
	  1953513424 blocks super 1.2 [2/1] [_U]


unused devices: <none>

Alles anzeigen

mdadm --detail /dev/md/mirror1

Code

root@nas:/home/reddy# mdadm --detail /dev/md/mirror1
/dev/md/mirror1:
		Version : 1.2
  Creation Time : Sun Aug 12 19:48:25 2012
	 Raid Level : raid1
	 Array Size : 1953513424 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953513424 (1863.02 GiB 2000.40 GB)
   Raid Devices : 2
  Total Devices : 1
	Persistence : Superblock is persistent


	Update Time : Thu Aug 16 19:29:49 2012
		  State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0


		   Name : nas:mirror1  (local to host nas)
		   UUID : 12fde5ff:91c8ff94:e9ad509b:11057223
		 Events : 291


	Number   Major   Minor   RaidDevice State
	   0       8       32        0      active sync   /dev/sdc
	   1       0        0        1      removed

Alles anzeigen

mdadm --detail /dev/md126

Code

root@nas:/home/reddy# mdadm --detail /dev/md126
/dev/md126:
		Version : 1.2
  Creation Time : Sun Aug 12 19:48:25 2012
	 Raid Level : raid1
	 Array Size : 1953513424 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953513424 (1863.02 GiB 2000.40 GB)
   Raid Devices : 2
  Total Devices : 1
	Persistence : Superblock is persistent


	Update Time : Thu Aug 16 19:55:00 2012
		  State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0


		   Name : nas:mirror1  (local to host nas)
		   UUID : 12fde5ff:91c8ff94:e9ad509b:11057223
		 Events : 293


	Number   Major   Minor   RaidDevice State
	   0       8       32        0      active sync   /dev/sdc
	   1       0        0        1      removed

Alles anzeigen

mdadm --detail /dev/md127

Code

root@nas:/home/reddy# mdadm --detail /dev/md127
/dev/md127:
		Version : 1.2
  Creation Time : Sun Aug 12 19:48:25 2012
	 Raid Level : raid1
	 Array Size : 1953513424 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953513424 (1863.02 GiB 2000.40 GB)
   Raid Devices : 2
  Total Devices : 1
	Persistence : Superblock is persistent


	Update Time : Thu Aug 16 19:22:19 2012
		  State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0


		   Name : nas:mirror1  (local to host nas)
		   UUID : 12fde5ff:91c8ff94:e9ad509b:11057223
		 Events : 329


	Number   Major   Minor   RaidDevice State
	   0       0        0        0      removed
	   1       8       16        1      active sync   /dev/sdb

Alles anzeigen

cat /etc/fstab

Code

root@nas:/home/reddy# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/sda1 during installation
UUID=9a37394d-0ebc-4145-857f-2ba3659eb5e1 /               ext4      errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=ea86e3b0-9e3f-4f48-8d19-f0df36580a49 none            swap      sw              0       0
#/dev/sdb1       /media/cdrom0   udf,iso9660 user,noauto     0       0
tmpfs           /tmp            tmpfs   defaults        0       0
# >>> [openmediavault]
UUID=fced0dfe-e95d-402d-8d66-4296b0736393      /media/fced0dfe-e95d-402d-8d66-4296b0736393 xfs      defaults,noexec,usrquota,grpquota 0 2
UUID=ad146658-eed1-4447-8eeb-4659db35086a      /media/ad146658-eed1-4447-8eeb-4659db35086a xfs      defaults,noexec,usrquota,grpquota 0 2
# <<< [openmediavault]

Alles anzeigen

mount

Code

root@nas:/home/reddy# mount
/dev/sda1 on / type ext4 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
tmpfs on /tmp type tmpfs (rw)
/dev/md126 on /media/fced0dfe-e95d-402d-8d66-4296b0736393 type xfs (rw,noexec,usrquota,grpquota)
/dev/sda3 on /media/ad146658-eed1-4447-8eeb-4659db35086a type xfs (rw,noexec,usrquota,grpquota)

Alles anzeigen

df

Code

root@nas:/home/reddy# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              8343480   1174032   6750020  15% /
tmpfs                   510256         4    510252   1% /lib/init/rw
udev                    504076       208    503868   1% /dev
tmpfs                   510256         0    510256   0% /dev/shm
tmpfs                   510256         8    510248   1% /tmp
/dev/md126           1952559564    576696 1951982868   1%   /media/fced0dfe-e95d-402d-8d66-4296b0736393
/dev/sda3            106781756     32944 106748812   1%   /media/ad146658-eed1-4447-8eeb-4659db35086a

All 3 disks: internal system one and both USB drives are configured
the same way to use minimum power, spin down after 10 minutes and
enable S.M.A.R.T.:

I'll post filesystems configuration screenshot in another post due to limit of 3 attachments per post.

reddy · 23. August 2012

And filesystems configuration below. As you see, it's /dev/sda3 I mounted from local system disk and a half of the mirror I created as
/dev/md0 but now visible as /dev/md126

reddy · 25. August 2012

I tried to fix the array in following way:

Code

umount /dev/md126
umount /dev/md127
mdadm –stop /dev/md126
mdadm –stop /dev/md127
mdadm -A /dev/md0 -f –update=summaries /dev/sdb /dev/sdc
mdadm /dev/md0 -a /dev/sdc

And it started re-building the array:

Code

reddy@nas:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdc[2] sdb[0]
      1953513424 blocks super 1.2 [2/1] [U_]
      [===>.................]  recovery = 17.6% (345273472/1953513424) finish=1285.7min speed=20846K/sec


unused devices: <none>

It will take quite a while. Is there any other way avoiding recovery? Both disks should already contain perfectly synchronized data in this case...

reddy · 27. August 2012

When rebuild finished everything seemed to be ok, mirror was mounted as originally defined to /dev/md0. However, after reboot the mirror is still clean but again changed mount point to /dev/md127 this time:

This instability scares me a bit... Why it behaves in such a strange manner?

For reference, current /proc/mdstat

Code

root@nas:/home/reddy#  cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : active raid1 sdb[0] sdc[2]
      1953513424 blocks super 1.2 [2/2] [UU]


unused devices: <none>

and /etc/mdadm/mdadm.conf

Code

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#


# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions


# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes


# automatically tag new arrays as belonging to the local system
HOMEHOST <system>


# definitions of existing MD arrays
ARRAY /dev/md0 metadata=1.2 name=nas:mirror1 UUID=12fde5ff:91c8ff94:e9ad509b:11057223


# instruct the monitoring daemon where to send mail alerts
MAILADDR xxxxxxxx@xxxxx.com
MAILFROM root

Alles anzeigen

reddy · 13. September 2012

Any clue why the mountpoint changes? Shall I be afraid of this?

Spy Alelo · 14. September 2012

No idea what is causing it, and I usually don't worry about mountpoint changes, but I do find XFS scary at times and I have a feeling that this may be related to your issue. If I were you, I would move the data you have there (if possible), and re-arrange the drives with ext4. Unless you have a specific reason to use XFS, stay away from it for a while. What it gains in performance, it looses in reliability (from my experience).

reddy · 16. September 2012

Thanks for the hint on the filesystem, I can still switch as I don't have a lot of data there yet. I built the NAS for reliability only, speed is not an issue (it works on USB drives anyway) so if there's something scary about XFS I'd prefer to switch to good old ext4.
Nevertheless, I'd like to know why those mount points change...

Spy Alelo · 16. September 2012

Not sure, probably when fixes were done it decided to change them, but there's never harm on that to be honest. You did say they are USB drives, moving them from one port to another may cause those changes too.

reddy · 19. September 2012

I marked the ports and kept the same disks connected always to the same ones. OK, I'm away from my system now, but I'll check if there's no side effects in a couple of days. Thanks.

sirevirus · 24. Mai 2013

Hello, I just had the same problem as reddy today.

I started my NAS with approximately the same configuration except that my disk are in SATA inside the NAS and already in EXT4, not XFS.
And my RAID was in the same state all of a sudden, 2 device with strange name, both in degraded status.

I did the same commands as reddy and now my RAID is recovering. I hope I won't have lost any data

Code

umount /dev/md126
umount /dev/md127
mdadm –-stop /dev/md126
mdadm –-stop /dev/md127
mdadm -A /dev/md0 -f –-update=summaries /dev/sdb /dev/sdc
mdadm /dev/md0 -a /dev/sdc

thanks for the help reddy.

martinweiss · 6. Juni 2013

This also happened to me recently, I'm not really sure why.

sirevirus · 4. Juli 2013

Hello,

After rebuilding the raid it works fine again for a few weeks and today i'm back with the same issue : I have two degraded raid.

A litlle precision maybe : If I reboot again without touching anything, the drives resort themselves. (/dev/sda become /dev/sdb, sdb become sdc, etc...)

This doesn't make a lot of sens to me and I begin to think about finding an other solution to create a raid because this one is obviously not reliable.

Too bad, when it works it's a great tool...

reddy · 24. Juli 2013

Just FYI - since I switched to ext4 (more than 6 months now) my mirror is stable with no surprises like this split. I hope it will stay like that I have no idea if and how the file system can affect mdadm, so I don't know how to help anybody or even myself if that happens again...

Jetzt mitmachen!