wrong raid5 size

luke-ws · Feb 4th 2019

hello everybody,
some days ago a disk disappeared from my array.
I physically unplugged and plugged the bay and the disk re-appeared.
I've rebuild the raid but the size is strange: max size 50.94 TiB.
It should be 56 TiB.
Tnx for help.

Here the details:

Display Spoiler

root@pandora:# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdg[8] sda[0] sdh[7] sdf[5] sde[4] sdd[3] sdc[2] sdb[1]
54697266176 blocks super 1.2 level 5, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
bitmap: 0/59 pages [0KB], 65536KB chunk

unused devices: <none>
[10:43:59] /sys/block/md0/md
root@pandora:# blkid
/dev/sdi1: UUID="e2e7ff80-067f-4302-86ab-93835946a049" TYPE="ext4" PARTUUID="d34ab1d1-01"
/dev/sdi5: UUID="61b916a0-8dfa-4176-908d-6cf0fb3f4d3d" TYPE="swap" PARTUUID="d34ab1d1-05"
/dev/sda: UUID="d0b3aec7-c1f2-99e7-73df-4dd8e3c26eba" UUID_SUB="afd7b9ef-0207-79a3-c0d1-7298dbb0081b" LABEL="pandora:pandora" TYPE="linux_raid_member"
/dev/sde: UUID="d0b3aec7-c1f2-99e7-73df-4dd8e3c26eba" UUID_SUB="83ab0586-68cc-766d-72d3-95a0f7a961a8" LABEL="pandora:pandora" TYPE="linux_raid_member"
/dev/sdc: UUID="d0b3aec7-c1f2-99e7-73df-4dd8e3c26eba" UUID_SUB="86cba930-b0c9-efec-5d63-849d84c0fedf" LABEL="pandora:pandora" TYPE="linux_raid_member"
/dev/sdb: UUID="d0b3aec7-c1f2-99e7-73df-4dd8e3c26eba" UUID_SUB="dc2be764-ff22-75a3-c79f-74574795ee40" LABEL="pandora:pandora" TYPE="linux_raid_member"
/dev/sdd: UUID="d0b3aec7-c1f2-99e7-73df-4dd8e3c26eba" UUID_SUB="97d37850-f24d-7f0c-8cde-7ec67679adb8" LABEL="pandora:pandora" TYPE="linux_raid_member"
/dev/sdf: UUID="d0b3aec7-c1f2-99e7-73df-4dd8e3c26eba" UUID_SUB="cacf5c16-df96-ebff-2afb-f50e061c0321" LABEL="pandora:pandora" TYPE="linux_raid_member"
/dev/sdh: UUID="d0b3aec7-c1f2-99e7-73df-4dd8e3c26eba" UUID_SUB="d4796509-cf84-ef00-3780-51bbd08d2e3c" LABEL="pandora:pandora" TYPE="linux_raid_member"
/dev/md0: LABEL="pandora" UUID="e75db39c-9a72-41ac-8d10-0280ea90c9f5" TYPE="ext4"
/dev/sdg: UUID="d0b3aec7-c1f2-99e7-73df-4dd8e3c26eba" UUID_SUB="b7551964-f8fd-b087-486b-d9c079915308" LABEL="pandora:pandora" TYPE="linux_raid_member"
[10:44:06] /sys/block/md0/md
root@pandora:# fdisk -l | grep "Disk "
Disk /dev/sdi: 256.2 GiB, 275064201216 bytes, 537234768 sectors
Disk identifier: 0xd34ab1d1
Disk /dev/sdj: 256.2 GiB, 275064201216 bytes, 537234768 sectors
Disk /dev/sdg: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/sda: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/sde: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/sdc: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/sdb: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/sdd: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/sdf: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/sdh: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/md0: 51 TiB, 56010000564224 bytes, 109394532352 sectors
[10:44:12] /sys/block/md0/md
root@pandora:# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# definitions of existing MD arrays
ARRAY /dev/md0 metadata=1.2 spares=1 name=pandora:pandora UUID=d0b3aec7:c1f299e7:73df4dd8:e3c26eba
[10:44:20] /sys/block/md0/md
root@pandora:# mdadm --detail --scan --verbose
ARRAY /dev/md0 level=raid5 num-devices=8 metadata=1.2 name=pandora:pandora UUID=d0b3aec7:c1f299e7:73df4dd8:e3c26eba
devices=/dev/sda,/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde,/dev/sdf,/dev/sdg,/dev/sdh
[10:44:25] /sys/block/md0/md
root@pandora:# df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 11M 6.3G 1% /run
/dev/sdi1 242G 2.8G 227G 2% /
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
tmpfs 32G 0 32G 0% /tmp
/dev/md0 51T 15T 37T 29% /srv/dev-disk-by-label-pandora

Methy · Feb 6th 2019

In the lines I see he speaks well of 8 disks but he also speaks of a spare disk, the problem would not come from there ?
7 discs + 1 spare

Code

# definitions of existing MD arrays
ARRAY /dev/md0 metadata=1.2 spares=1

My configuration :

Code

mdadm --detail --scan --verbose
ARRAY /dev/md/2 level=raid5 num-devices=4 metadata=1.2 name=NAS:2 UUID=66dde5b9:1996f832:ce496f31:e0589a2f
   devices=/dev/sdb5,/dev/sdc5,/dev/sdd5,/dev/sde5








  Version : 1.2
  Creation Time : Tue Jan 26 17:34:20 2016
     Raid Level : raid5
     Array Size : 5846050368 (5575.23 GiB 5986.36 GB)
  Used Dev Size : 1948683456 (1858.41 GiB 1995.45 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent


    Update Time : Wed Feb  6 16:20:40 2019
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0


         Layout : left-symmetric
     Chunk Size : 64K


           Name : NAS:2
           UUID : 66dde5b9:1996f832:ce496f31:e0589a2f
         Events : 9960


    Number   Major   Minor   RaidDevice State
       0       8       21        0      active sync   /dev/sdb5
       1       8       69        1      active sync   /dev/sde5
       2       8       37        2      active sync   /dev/sdc5
       3       8       53        3      active sync   /dev/sdd5

Display More

luke-ws · Feb 7th 2019

thanks methy.
i've changed mdadm.conf

Code

# definitions of existing MD arrays


#ARRAY /dev/md0 metadata=1.2 spares=1 name=pandora:pandora UUID=d0b3aec7:c1f299e7:73df4dd8:e3c26eba
ARRAY /dev/md0 level=raid5 num-devices=8 metadata=1.2 name=pandora:pandora UUID=d0b3aec7:c1f299e7:73df4dd8:e3c26eba

then

Code

update-initramfs -u && reboot

same problem.
I don't know... I've a lot of space available.
I don't wont to make some mistake and lost data...

Methy · Feb 7th 2019

When you go in raid management do you have a spare disk ?

If yes try to grow the raid.

luke-ws · Feb 7th 2019

nope

Participate now!