RAID 5 Degragated - New drive got shrinked

danieliod · 6. März 2016

Hello,

I had a HDD failure and a degragated RAID5.
I have replaced the drive and recovered my RAID.

Everthing worked well for 2 days and today after SMART Tools run, wierd thing happeded:
1. my new drive got shrinked (see attached Picture)
2. my RAID is now degragated.
3. when i try to access File System or RAID Managment then the process: scsi_eh_8 takes all the CPU process power.

Can someone help me rebuild my RAID without data loose?

This is my Degragated RAID info:

Code

Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdc[1] sde[2]
      5855716352 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]


unused devices: <none>
/dev/md127: LABEL="RAID" UUID="7fbde406-4c77-4bed-94c8-b6b045f9e757" TYPE="ext4"
/dev/sde: UUID="332d8084-c2b3-a139-44a4-f8e16865cc49" UUID_SUB="d17728a0-84fe-c504-ffb0-e13849c2282d" LABEL="openmediavault:NAS" TYPE="linux_raid_member"
/dev/sda1: UUID="c4773f30-7a51-4b82-89b2-dc1e8b25f1ff" TYPE="ext4"
/dev/sda3: LABEL="DATA" UUID="395fdb16-d339-4c6d-bbf6-6419f1aaa444" TYPE="ext4"
/dev/sda5: UUID="8cf58870-b3d3-4402-9e95-cbf71c041584" TYPE="swap"
/dev/sdc: UUID="332d8084-c2b3-a139-44a4-f8e16865cc49" UUID_SUB="34c4db9b-ec1b-b61b-2a46-cb372f84d784" LABEL="openmediavault:NAS" TYPE="linux_raid_member"


Disk /dev/sda: 126.0 GB, 126035288064 bytes
255 heads, 63 sectors/track, 15322 cylinders, total 246162672 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006ad32


   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048    29493247    14745600   83  Linux
/dev/sda2        29495294    32567295     1536001    5  Extended
/dev/sda3        32567296   246161407   106797056   83  Linux
/dev/sda5        29495296    32567295     1536000   82  Linux swap / Solaris


Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x1e011e00


   Device Boot      Start         End      Blocks   Id  System


Disk /dev/sde: 3000.6 GB, 3000592982016 bytes
256 heads, 63 sectors/track, 363376 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x1e011e02


   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.


Disk /dev/md127: 5996.3 GB, 5996253544448 bytes
2 heads, 4 sectors/track, 1463929088 cylinders, total 11711432704 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disk identifier: 0x00000000

Alles anzeigen

micdon · 7. März 2016

ssh into your sever and post the output of this command:
mdadm --detail /dev/md127

Assuming that md127 is your broken raid5 array...

One thing that you can try as well is to stop and mount your array in read only mode.
mdadm --readonly /dev/md127

danieliod · 7. März 2016

Thank you, here is the outcome of: mdadm --detail /dev/md127

Code

Version : 1.2
  Creation Time : Mon Jan  4 15:08:44 2016
     Raid Level : raid5
     Array Size : 5855716352 (5584.45 GiB 5996.25 GB)
  Used Dev Size : 2927858176 (2792.22 GiB 2998.13 GB)
   Raid Devices : 3
  Total Devices : 2
    Persistence : Superblock is persistent


    Update Time : Mon Mar  7 08:00:17 2016
          State : clean, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0


         Layout : left-symmetric
     Chunk Size : 512K


           Name : openmediavault:NAS  (local to host openmediavault)
           UUID : 332d8084:c2b3a139:44a4f8e1:6865cc49
         Events : 95354


    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       48        1      active sync   /dev/sdd
       2       8       64        2      active sync   /dev/sde

Alles anzeigen

Since I'm new to Linux, can you help me with commands of mount/unmount as you suggested?

micdon · 7. März 2016

Your array (Raid5) normally needs 3 disks but one is failed. Very likely when you added your replacement disk it wasn't empty which can create a lot of troubles.

To evacuate the data (that's the low risk approach) set it to read only prior to copying data to an other disk set.
To do this you would unmount your file system
umount /dev/md127
Stop the array
mdadm --stop /dev/md127
And start it again in readonly mode:
mdadm --readonly /dev/md127
And mount it again with
mount /dev/md127
Now all of your data should be accessible but of course in readonly mode. You can't rebuild in this mode but you can copy data to an evacuation disk.

If you want to recover / rebuild the array you need to set it to writable first - use the same steps than above (unmount, stop, start, mount) but replace --readonly with --readwrite
mdadm --readwrite /dev/md127

Once your array is set to readwrite it will accept the new disk and it will start to rebuild itself. Prior to adding the new disk please make sure that it is completely empty. You can wipe it with the OMV GUI.
After this is done you can also can also add the disk with the OMV GUI under Raid Management / Recovery. This should fully do the job.

If you want detailed status messages you can can SSH into your box and run
watch -n 1 cat /proc/mdstat

Hopefully this gets you back to an healthy setup
Good Luck
Mic

danieliod · 7. März 2016

Thank you for the detailed info, it's very appreciated.

I have replaced the SATA cable and the drive boot up correctly.
Then I wiped it via the OMV GUI, restarted and started RAID recovery.

It's running now, let's hope it will last long enough...
I will update once finished.

thank you

Jetzt mitmachen!