So after first thinking there was something wrong with SMB, I realised the RAID FS had mysteriously gone offline after a Shutdown and Powerup.
On a fresh boot, the RAID filesystem doesn't mount, and various commands tell me that /dev/sdc is missing its superblock
On fresh reboot:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : inactive sdd[0](S) sdb[2](S) sda[1](S)
5860147464 blocks super 1.2
unused devices: <none>
/dev/sda: UUID="248114fe-41d4-32a6-fa93-61499686a567" UUID_SUB="d6cc6979-001c-92df-a88d-da91c2f51272" LABEL="h4:TKRaid" TYPE="linux_raid_member"
/dev/sdb: UUID="248114fe-41d4-32a6-fa93-61499686a567" UUID_SUB="c4c9d9f6-76d1-8465-13bb-599285e4d463" LABEL="h4:TKRaid" TYPE="linux_raid_member"
/dev/sdd: UUID="248114fe-41d4-32a6-fa93-61499686a567" UUID_SUB="2bf2a3b0-9d3b-a028-cb06-88d7c218b918" LABEL="h4:TKRaid" TYPE="linux_raid_member"
/dev/mmcblk2: PTUUID="801aac87" PTTYPE="dos"
# definitions of existing MD arrays
ARRAY /dev/md0 metadata=1.2 name=h4:TKRaid UUID=248114fe:41d432a6:fa936149:9686a567
INACTIVE-ARRAY /dev/md0 num-devices=3 metadata=1.2 name=h4:TKRaid UUID=248114fe:41d432a6:fa936149:9686a567
devices=/dev/sda,/dev/sdb,/dev/sdd
root@h4:~# mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Raid Level : raid0
Total Devices : 3
Persistence : Superblock is persistent
State : inactive
Working Devices : 3
Name : h4:TKRaid (local to host h4)
UUID : 248114fe:41d432a6:fa936149:9686a567
Events : 29648
Number Major Minor RaidDevice
- 8 0 - /dev/sda
- 8 48 - /dev/sdd
- 8 16 - /dev/sdb
root@h4:~# mdadm --examine /dev/sd[abcd]
/dev/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 248114fe:41d432a6:fa936149:9686a567
Name : h4:TKRaid (local to host h4)
Creation Time : Sat Oct 31 20:47:11 2020
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=176 sectors
State : clean
Device UUID : d6cc6979:001c92df:a88dda91:c2f51272
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Jul 25 22:28:11 2021
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : bd86b81e - correct
Events : 29648
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 248114fe:41d432a6:fa936149:9686a567
Name : h4:TKRaid (local to host h4)
Creation Time : Sat Oct 31 20:47:11 2020
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=176 sectors
State : clean
Device UUID : c4c9d9f6:76d18465:13bb5992:85e4d463
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Jul 25 22:28:11 2021
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : b32a86b1 - correct
Events : 29648
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
mdadm: No md superblock detected on /dev/sdc.
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 248114fe:41d432a6:fa936149:9686a567
Name : h4:TKRaid (local to host h4)
Creation Time : Sat Oct 31 20:47:11 2020
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=176 sectors
State : clean
Device UUID : 2bf2a3b0:9d3ba028:cb0688d7:c218b918
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Jul 25 22:28:11 2021
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : 2a229932 - correct
Events : 29648
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
So /dev/sdc is missing superblock.
But if I now run mdadm --assemble --scan -v
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sda is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sda is already in /dev/md0 as 1
mdadm: /dev/sdb is already in /dev/md0 as 2
mdadm: added /dev/sdc to /dev/md0 as 3
mdadm: /dev/sdd is already in /dev/md0 as 0
mdadm: /dev/md0 has been started with 4 drives.
root@h4:~# mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sat Oct 31 20:47:11 2020
Raid Level : raid5
Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sun Jul 25 22:28:11 2021
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Name : h4:TKRaid (local to host h4)
UUID : 248114fe:41d432a6:fa936149:9686a567
Events : 29648
Number Major Minor RaidDevice State
0 8 48 0 active sync /dev/sdd
1 8 0 1 active sync /dev/sda
2 8 16 2 active sync /dev/sdb
3 8 32 3 active sync /dev/sdc
root@h4:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active (auto-read-only) raid5 sdc[3] sdd[0] sdb[2] sda[1]
5860147200 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/15 pages [0KB], 65536KB chunk
unused devices: <none>
It seems to assemble fine, with no complaints. It says all drives are online. I can mount it and all my files are there.
Yet still,
mdadm --examine /dev/sdc
mdadm: No md superblock detected on /dev/sdc.
root@h4:~# systemctl status mdadm
● mdadm.service
Loaded: masked (Reason: Unit mdadm.service is masked.)
Active: inactive (dead)
Fdisk -l does not list /dev/sdc
Is there a way to restore the superblock on sdc and have it mount at boot as before?
Thanks.