Mdadm device (/dev/md3) raid5 get lost on every reboot

    • OMV 3.x
    • Mdadm device (/dev/md3) raid5 get lost on every reboot

      Hi to every one.

      Im having an issue, similar to this guy : [RAID5] Missing superblocks after restart

      Didnt want to post on these thread since is quite old.

      Im creating an mdadm with (it defaults to 1.2) :

      Source Code

      1. mdadm --create --verbose /dev/md3 --level=5 --raid-devices=4 /dev/sda /dev/sdb /dev/sde /dev/sdf





      Then after several hours it finnish and i create the fs with mkfs, update the mdadm.conf and im able to mount the raid.

      Source Code

      1. mfks.ext4 /dev/md3 ; mdadm --details --scan >> /etc/mdadm/mdadm.conf ; update-initramfs -u

      Evferything is ok, then I reboot and system cannot mount the fstab entry because md3 is not being initialized, why ? the superblocks are gone (tried several times and occasionally one of the disk still remains intact ) :

      Source Code

      1. mdadm: Cannot assemble mbr metadata on /dev/sdb
      2. root@vlsprnas-omv3:~# mdadm --examine /dev/sdf
      3. /dev/sdf:
      4. MBR Magic : aa55
      5. Partition[0] : 4294967295 sectors at 1 (type ee)
      6. root@vlsprnas-omv3:~# mdadm --examine /dev/sde
      7. /dev/sde:
      8. MBR Magic : aa55
      9. Partition[0] : 4294967295 sectors at 1 (type ee)
      10. root@vlsprnas-omv3:~# mdadm --examine /dev/sdf
      11. /dev/sdf:
      12. MBR Magic : aa55
      13. Partition[0] : 4294967295 sectors at 1 (type ee)
      14. root@vlsprnas-omv3:~# mdadm --examine /dev/sda
      15. /dev/sda:
      16. MBR Magic : aa55
      17. Partition[0] : 4294967295 sectors at 1 (type ee)
      Display All

      The difference with the other guy is that I have another 2 raid 1 running fine in the board (verything create from scratch ) on 2 drives :

      Source Code

      1. root@vlsprnas-omv3:~# lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
      2. NAME SIZE FSTYPE TYPE MOUNTPOINT
      3. sdf 3,7T disk
      4. └─sdf1 25G linux_raid_member part
      5. └─md3 75G raid5
      6. sdd 465,8G disk
      7. ├─sdd2 95,4G linux_raid_member part
      8. │ └─md1 95,4G ext4 raid1 /
      9. ├─sdd3 76,3G linux_raid_member part
      10. │ └─md2 76,3G ext4 raid1 /dataraid1
      11. ├─sdd1 10M part
      12. └─sdd4 18,3G part
      13. sdb 3,7T disk
      14. └─sdb1 25G linux_raid_member part
      15. └─md3 75G raid5
      16. sde 3,7T disk
      17. └─sde1 25G linux_raid_member part
      18. └─md3 75G raid5
      19. sdc 189,9G disk
      20. ├─sdc2 95,4G linux_raid_member part
      21. │ └─md1 95,4G ext4 raid1 /
      22. ├─sdc3 76,3G linux_raid_member part
      23. │ └─md2 76,3G ext4 raid1 /dataraid1
      24. ├─sdc1 10M part
      25. └─sdc4 18,3G part
      26. sda 3,7T disk
      27. └─sda1 25G linux_raid_member part
      28. └─md3 75G raid5
      Display All

      As you can note, it is one raid 5 partition, thats because im testing, now im creating the array with partitions with type raid set instead of the full disk, wish me good luck.

      If any one has any hint please help, im very stucked here.
    • So far , now I rebooted the system and it went ok :
      Display Spoiler

      [ 8.373986] Bluetooth: HCI UART protocol Three-wire (H5) registered
      [ 8.374026] Bluetooth: HCI UART protocol Intel registered
      [ 8.374046] Bluetooth: HCI UART protocol BCM registered
      [ 8.374047] Bluetooth: HCI UART protocol QCA registered
      [ 8.454882] EXT4-fs (md1): re-mounted. Opts: errors=remount-ro
      [ 8.531415] [drm] Memory usable by graphics device = 2048M
      [ 8.531419] [drm] Replacing VGA console driver
      [ 8.532255] Console: switching to colour dummy device 80x25
      [ 8.532395] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
      [ 8.532396] [drm] Driver supports precise vblank timestamp query.
      [ 8.532907] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem
      [ 8.566752] input: PC Speaker as /devices/platform/pcspkr/input/input8
      [ 8.620865] md: md3 stopped.
      [ 8.621817] md: bind<sdb1>
      [ 8.621980] md: bind<sde1>
      [ 8.622130] md: bind<sdf1>
      [ 8.622276] md: bind<sda1>
      [ 8.658449] ACPI: Video Device [GFX0] (multi-head: yes rom: no post: no)
      [ 8.659349] acpi device:09: registered as cooling_device5
      [ 8.659462] input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input9
      [ 8.659537] [drm] Initialized i915 1.6.0 20160711 for 0000:00:02.0 on minor 0
      [ 8.659608] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
      [ 8.698982] async_tx: api initialized (async)
      [ 8.781014] fbcon: inteldrmfb (fb0) is primary device
      [ 8.823303] Console: switching to colour frame buffer device 240x75
      [ 8.851274] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
      [ 8.965874] md: raid6 personality registered for level 6
      [ 8.965877] md: raid5 personality registered for level 5
      [ 8.965878] md: raid4 personality registered for level 4
      [ 8.966964] md/raid:md3: device sda1 operational as raid disk 0
      [ 8.966967] md/raid:md3: device sdf1 operational as raid disk 3
      [ 8.966969] md/raid:md3: device sde1 operational as raid disk 2
      [ 8.966971] md/raid:md3: device sdb1 operational as raid disk 1
      [ 8.967641] md/raid:md3: allocated 4374kB
      [ 8.967721] md/raid:md3: raid level 5 active with 4 out of 4 devices, algorithm 2
      [ 8.967722] RAID conf printout:
      [ 8.967723] --- level:5 rd:4 wd:4
      [ 8.967725] disk 0, o:1, dev:sda1
      [ 8.967727] disk 1, o:1, dev:sdb1
      [ 8.967729] disk 2, o:1, dev:sde1
      [ 8.967730] disk 3, o:1, dev:sdf1
      [ 8.967802] md3: detected capacity change from 0 to 80477159424
      [ 9.013700] ppdev: user-space parallel port driver
      [ 9.047175] iTCO_vendor_support: vendor-support=0
      [ 9.056289] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
      [ 9.056363] iTCO_wdt: Found a Bay Trail SoC TCO device (Version=3, TCOBASE=0x0460)
      [ 9.056588] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
      [ 9.244950] kvm: disabled by bios
      [ 9.286009] kvm: disabled by bios
      [ 9.318498] kvm: disabled by bios
      [ 9.353668] kvm: disabled by bios
      [ 9.551994] intel_soc_dts_thermal: request_threaded_irq ret -22
      [ 9.581186] intel_soc_dts_thermal: request_threaded_irq ret -22
      [ 9.652942] intel_soc_dts_thermal: request_threaded_irq ret -22
      [ 9.866441] intel_rapl: Found RAPL domain package
      [ 9.866444] intel_rapl: Found RAPL domain core
      [ 10.563787] EXT4-fs (md3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
      [ 11.157540] EXT4-fs (md2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
      [ 11.708061] systemd-journald[251]: Received request to flush runtime journal from PID 1
      [ 12.023622] r8169 0000:03:00.0: firmware: direct-loading firmware rtl_nic/rtl8168g-2.fw
      [ 12.068068] r8169 0000:03:00.0 eth0: link down
      [ 12.068372] r8169 0000:03:00.0 eth0: link down
      [ 14.427979] RPC: Registered named UNIX socket transport module.
      [ 14.427982] RPC: Registered udp transport module.
      [ 14.427982] RPC: Registered tcp transport module.
      [ 14.427983] RPC: Registered tcp NFSv4.1 backchannel transport module.
      [ 14.452711] FS-Cache: Loaded
      [ 14.495949] FS-Cache: Netfs 'nfs' registered for caching
      [ 14.521798] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
      [ 14.856497] r8169 0000:03:00.0 eth0: link up




      It has everything :

      Source Code

      1. root@vlsprnas-omv3:~# df -h
      2. S.ficheros Tamaño Usados Disp Uso% Montado en
      3. udev 10M 0 10M 0% /dev
      4. tmpfs 733M 8,9M 724M 2% /run
      5. /dev/md1 94G 1,4G 88G 2% /
      6. tmpfs 1,8G 0 1,8G 0% /dev/shm
      7. tmpfs 5,0M 0 5,0M 0% /run/lock
      8. tmpfs 1,8G 0 1,8G 0% /sys/fs/cgroup
      9. tmpfs 1,8G 0 1,8G 0% /tmp
      10. /dev/md3 74G 53M 70G 1% /dataraid5
      11. /dev/md2 75G 53M 74G 1% /dataraid1
      12. root@vlsprnas-omv3:~# dmesg | less
      13. root@vlsprnas-omv3:~# lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
      14. NAME SIZE FSTYPE TYPE MOUNTPOINT
      15. sdf 3,7T disk
      16. └─sdf1 25G linux_raid_member part
      17. └─md3 75G ext4 raid5 /dataraid5
      18. sdd 465,8G disk
      19. ├─sdd2 95,4G linux_raid_member part
      20. │ └─md1 95,4G ext4 raid1 /
      21. ├─sdd3 76,3G linux_raid_member part
      22. │ └─md2 76,3G ext4 raid1 /dataraid1
      23. ├─sdd1 10M part
      24. └─sdd4 18,3G part
      25. sdb 3,7T disk
      26. └─sdb1 25G linux_raid_member part
      27. └─md3 75G ext4 raid5 /dataraid5
      28. sde 3,7T disk
      29. └─sde1 25G linux_raid_member part
      30. └─md3 75G ext4 raid5 /dataraid5
      31. sdc 189,9G disk
      32. ├─sdc2 95,4G linux_raid_member part
      33. │ └─md1 95,4G ext4 raid1 /
      34. ├─sdc3 76,3G linux_raid_member part
      35. │ └─md2 76,3G ext4 raid1 /dataraid1
      36. ├─sdc1 10M part
      37. └─sdc4 18,3G part
      38. sda 3,7T disk
      39. └─sda1 25G linux_raid_member part
      40. └─md3 75G ext4 raid5 /dataraid5
      Display All
      I dont get WHY, it fails when I tried before, the only change was, full disk in comparisson to using partitions, and the size of course.

      Now im trying again with partitions, typed to raid and full disk :
      Display Spoiler

      root@vlsprnas-omv3:~# fdisk -l /dev/sda /dev/sdb /dev/sde /dev/sdf

      Disk /dev/sda: 3,7 TiB, 4000787030016 bytes, 7814037168 sectors
      Units: sectors of 1 * 512 = 512 bytes
      Sector size (logical/physical): 512 bytes / 4096 bytes
      I/O size (minimum/optimal): 4096 bytes / 4096 bytes
      Disklabel type: gpt
      Disk identifier: 2A15EEF7-8C38-41B6-AF4B-6CF97DEAE8C0

      Device Start End Sectors Size Type
      /dev/sda1 2048 7814037134 7814035087 3,7T Linux RAID

      Disk /dev/sdb: 3,7 TiB, 4000787030016 bytes, 7814037168 sectors
      Units: sectors of 1 * 512 = 512 bytes
      Sector size (logical/physical): 512 bytes / 4096 bytes
      I/O size (minimum/optimal): 4096 bytes / 4096 bytes
      Disklabel type: gpt
      Disk identifier: 9866C6A3-A122-45D3-8537-EBBD00190F50

      Device Start End Sectors Size Type
      /dev/sdb1 2048 7814037134 7814035087 3,7T Linux RAID

      Disk /dev/sde: 3,7 TiB, 4000787030016 bytes, 7814037168 sectors
      Units: sectors of 1 * 512 = 512 bytes
      Sector size (logical/physical): 512 bytes / 4096 bytes
      I/O size (minimum/optimal): 4096 bytes / 4096 bytes
      Disklabel type: gpt
      Disk identifier: D240C38D-27E9-4CD6-B843-D144D2F173F4

      Device Start End Sectors Size Type
      /dev/sde1 2048 7814037134 7814035087 3,7T Linux RAID

      Disk /dev/sdf: 3,7 TiB, 4000787030016 bytes, 7814037168 sectors
      Units: sectors of 1 * 512 = 512 bytes
      Sector size (logical/physical): 512 bytes / 4096 bytes
      I/O size (minimum/optimal): 4096 bytes / 4096 bytes
      Disklabel type: gpt
      Disk identifier: 1970BC1B-2728-4FF4-BC94-B95E3AD04F49

      Device Start End Sectors Size Type
      /dev/sdf1 2048 7814037134 7814035087 3,7T Linux RAID

      root@vlsprnas-omv3:~# lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
      NAME SIZE FSTYPE TYPE MOUNTPOINT
      sdf 3,7T disk
      └─sdf1 3,7T part
      sdd 465,8G disk
      ├─sdd2 95,4G linux_raid_member part
      │ └─md1 95,4G ext4 raid1 /
      ├─sdd3 76,3G linux_raid_member part
      │ └─md2 76,3G ext4 raid1 /dataraid1
      ├─sdd1 10M part
      └─sdd4 18,3G part
      sdb 3,7T disk
      └─sdb1 3,7T part
      sde 3,7T disk
      └─sde1 3,7T part
      sdc 189,9G disk
      ├─sdc2 95,4G linux_raid_member part
      │ └─md1 95,4G ext4 raid1 /
      ├─sdc3 76,3G linux_raid_member part
      │ └─md2 76,3G ext4 raid1 /dataraid1
      ├─sdc1 10M part
      └─sdc4 18,3G part
      sda 3,7T disk
      └─sda1 3,7T part

      root@vlsprnas-omv3:~# cat /proc/mdstat
      Personalities : [raid1] [raid6] [raid5] [raid4]
      md3 : active raid5 sdf1[4] sde1[2] sdb1[1] sda1[0]
      11720658432 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
      [>....................] recovery = 0.1% (3919188/3906886144) finish=448.1min speed=145155K/sec
      bitmap: 0/30 pages [0KB], 65536KB chunk

      md2 : active raid1 sdc3[0] sdd3[1]
      79999936 blocks [2/2] [UU]

      md1 : active raid1 sdc2[0] sdd2[1]
      99988416 blocks [2/2] [UU]

      The post was edited 1 time, last by flechoide ().

    • So far , now I rebooted the system and it went ok :
      Display Spoiler

      [ 8.373986] Bluetooth: HCI UART protocol Three-wire (H5) registered
      [ 8.374026] Bluetooth: HCI UART protocol Intel registered
      [ 8.374046] Bluetooth: HCI UART protocol BCM registered
      [ 8.374047] Bluetooth: HCI UART protocol QCA registered
      [ 8.454882] EXT4-fs (md1): re-mounted. Opts: errors=remount-ro
      [ 8.531415] [drm] Memory usable by graphics device = 2048M
      [ 8.531419] [drm] Replacing VGA console driver
      [ 8.532255] Console: switching to colour dummy device 80x25
      [ 8.532395] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
      [ 8.532396] [drm] Driver supports precise vblank timestamp query.
      [ 8.532907] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem
      [ 8.566752] input: PC Speaker as /devices/platform/pcspkr/input/input8
      [ 8.620865] md: md3 stopped.
      [ 8.621817] md: bind<sdb1>
      [ 8.621980] md: bind<sde1>
      [ 8.622130] md: bind<sdf1>
      [ 8.622276] md: bind<sda1>
      [ 8.658449] ACPI: Video Device [GFX0] (multi-head: yes rom: no post: no)
      [ 8.659349] acpi device:09: registered as cooling_device5
      [ 8.659462] input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input9
      [ 8.659537] [drm] Initialized i915 1.6.0 20160711 for 0000:00:02.0 on minor 0
      [ 8.659608] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
      [ 8.698982] async_tx: api initialized (async)
      [ 8.781014] fbcon: inteldrmfb (fb0) is primary device
      [ 8.823303] Console: switching to colour frame buffer device 240x75
      [ 8.851274] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
      [ 8.965874] md: raid6 personality registered for level 6
      [ 8.965877] md: raid5 personality registered for level 5
      [ 8.965878] md: raid4 personality registered for level 4
      [ 8.966964] md/raid:md3: device sda1 operational as raid disk 0
      [ 8.966967] md/raid:md3: device sdf1 operational as raid disk 3
      [ 8.966969] md/raid:md3: device sde1 operational as raid disk 2
      [ 8.966971] md/raid:md3: device sdb1 operational as raid disk 1
      [ 8.967641] md/raid:md3: allocated 4374kB
      [ 8.967721] md/raid:md3: raid level 5 active with 4 out of 4 devices, algorithm 2
      [ 8.967722] RAID conf printout:
      [ 8.967723] --- level:5 rd:4 wd:4
      [ 8.967725] disk 0, o:1, dev:sda1
      [ 8.967727] disk 1, o:1, dev:sdb1
      [ 8.967729] disk 2, o:1, dev:sde1
      [ 8.967730] disk 3, o:1, dev:sdf1
      [ 8.967802] md3: detected capacity change from 0 to 80477159424
      [ 9.013700] ppdev: user-space parallel port driver
      [ 9.047175] iTCO_vendor_support: vendor-support=0
      [ 9.056289] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
      [ 9.056363] iTCO_wdt: Found a Bay Trail SoC TCO device (Version=3, TCOBASE=0x0460)
      [ 9.056588] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
      [ 9.244950] kvm: disabled by bios
      [ 9.286009] kvm: disabled by bios
      [ 9.318498] kvm: disabled by bios
      [ 9.353668] kvm: disabled by bios
      [ 9.551994] intel_soc_dts_thermal: request_threaded_irq ret -22
      [ 9.581186] intel_soc_dts_thermal: request_threaded_irq ret -22
      [ 9.652942] intel_soc_dts_thermal: request_threaded_irq ret -22
      [ 9.866441] intel_rapl: Found RAPL domain package
      [ 9.866444] intel_rapl: Found RAPL domain core
      [ 10.563787] EXT4-fs (md3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
      [ 11.157540] EXT4-fs (md2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
      [ 11.708061] systemd-journald[251]: Received request to flush runtime journal from PID 1
      [ 12.023622] r8169 0000:03:00.0: firmware: direct-loading firmware rtl_nic/rtl8168g-2.fw
      [ 12.068068] r8169 0000:03:00.0 eth0: link down
      [ 12.068372] r8169 0000:03:00.0 eth0: link down
      [ 14.427979] RPC: Registered named UNIX socket transport module.
      [ 14.427982] RPC: Registered udp transport module.
      [ 14.427982] RPC: Registered tcp transport module.
      [ 14.427983] RPC: Registered tcp NFSv4.1 backchannel transport module.
      [ 14.452711] FS-Cache: Loaded
      [ 14.495949] FS-Cache: Netfs 'nfs' registered for caching
      [ 14.521798] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
      [ 14.856497] r8169 0000:03:00.0 eth0: link up




      It has everything :

      Source Code

      1. root@vlsprnas-omv3:~# df -h
      2. S.ficheros Tamaño Usados Disp Uso% Montado en
      3. udev 10M 0 10M 0% /dev
      4. tmpfs 733M 8,9M 724M 2% /run
      5. /dev/md1 94G 1,4G 88G 2% /
      6. tmpfs 1,8G 0 1,8G 0% /dev/shm
      7. tmpfs 5,0M 0 5,0M 0% /run/lock
      8. tmpfs 1,8G 0 1,8G 0% /sys/fs/cgroup
      9. tmpfs 1,8G 0 1,8G 0% /tmp
      10. /dev/md3 74G 53M 70G 1% /dataraid5
      11. /dev/md2 75G 53M 74G 1% /dataraid1
      12. root@vlsprnas-omv3:~# dmesg | less
      13. root@vlsprnas-omv3:~# lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
      14. NAME SIZE FSTYPE TYPE MOUNTPOINT
      15. sdf 3,7T disk
      16. └─sdf1 25G linux_raid_member part
      17. └─md3 75G ext4 raid5 /dataraid5
      18. sdd 465,8G disk
      19. ├─sdd2 95,4G linux_raid_member part
      20. │ └─md1 95,4G ext4 raid1 /
      21. ├─sdd3 76,3G linux_raid_member part
      22. │ └─md2 76,3G ext4 raid1 /dataraid1
      23. ├─sdd1 10M part
      24. └─sdd4 18,3G part
      25. sdb 3,7T disk
      26. └─sdb1 25G linux_raid_member part
      27. └─md3 75G ext4 raid5 /dataraid5
      28. sde 3,7T disk
      29. └─sde1 25G linux_raid_member part
      30. └─md3 75G ext4 raid5 /dataraid5
      31. sdc 189,9G disk
      32. ├─sdc2 95,4G linux_raid_member part
      33. │ └─md1 95,4G ext4 raid1 /
      34. ├─sdc3 76,3G linux_raid_member part
      35. │ └─md2 76,3G ext4 raid1 /dataraid1
      36. ├─sdc1 10M part
      37. └─sdc4 18,3G part
      38. sda 3,7T disk
      39. └─sda1 25G linux_raid_member part
      40. └─md3 75G ext4 raid5 /dataraid5
      Display All

      I dont get WHY, it fails when I tried before, the only change was, full disk in comparisson to using partitions, and the size of course.

      Well ! SOme hours later (lots of hours)

      Its working ok, it keeps nice and up after every reboot :

      Display Spoiler

      [ 9.264449] kvm: disabled by bios
      [ 9.495186] intel_soc_dts_thermal: request_threaded_irq ret -22
      [ 9.547742] intel_soc_dts_thermal: request_threaded_irq ret -22
      [ 9.607826] intel_soc_dts_thermal: request_threaded_irq ret -22
      [ 9.608408] intel_rapl: Found RAPL domain package
      [ 9.608412] intel_rapl: Found RAPL domain core
      [ 9.749946] EXT4-fs (md1): re-mounted. Opts: errors=remount-ro
      [ 10.359545] md: md3 stopped.
      [ 10.360936] md: bind<sdb1>
      [ 10.361105] md: bind<sde1>
      [ 10.361343] md: bind<sdf1>
      [ 10.361483] md: bind<sda1>
      [ 10.395441] async_tx: api initialized (async)
      [ 10.429058] md: raid6 personality registered for level 6
      [ 10.429062] md: raid5 personality registered for level 5
      [ 10.429063] md: raid4 personality registered for level 4
      [ 10.429613] md/raid:md3: device sda1 operational as raid disk 0
      [ 10.429616] md/raid:md3: device sdf1 operational as raid disk 3
      [ 10.429618] md/raid:md3: device sde1 operational as raid disk 2
      [ 10.429619] md/raid:md3: device sdb1 operational as raid disk 1
      [ 10.430226] md/raid:md3: allocated 4374kB
      [ 10.430288] md/raid:md3: raid level 5 active with 4 out of 4 devices, algorithm 2
      [ 10.430289] RAID conf printout:
      [ 10.430290] --- level:5 rd:4 wd:4
      [ 10.430292] disk 0, o:1, dev:sda1
      [ 10.430294] disk 1, o:1, dev:sdb1
      [ 10.430295] disk 2, o:1, dev:sde1
      [ 10.430296] disk 3, o:1, dev:sdf1
      [ 10.430464] created bitmap (30 pages) for device md3
      [ 10.431459] md3: bitmap initialized from disk: read 2 pages, set 0 of 59615 bits
      [ 10.443826] md3: detected capacity change from 0 to 12001954234368
      [ 11.020116] EXT4-fs (md2): mounted filesystem with ordered data mode. Opts: user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl
      [ 11.544930] EXT4-fs (md3): mounted filesystem with ordered data mode. Opts: user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl
      [ 11.666419] systemd-journald[243]: Received request to flush runtime journal from PID 1
      [ 11.946530] r8169 0000:03:00.0: firmware: direct-loading firmware rtl_nic/rtl8168g-2.fw
      [ 11.990821] r8169 0000:03:00.0 eth0: link down
      [ 11.990841] r8169 0000:03:00.0 eth0: link down
      [ 14.108759] RPC: Registered named UNIX socket transport module.
      [ 14.108763] RPC: Registered udp transport module.
      [ 14.108764] RPC: Registered tcp transport module.
      [ 14.108766] RPC: Registered tcp NFSv4.1 backchannel transport module.
      [ 14.133966] FS-Cache: Loaded
      [ 14.161650] FS-Cache: Netfs 'nfs' registered for caching
      [ 14.203558] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
      [ 14.764899] r8169 0000:03:00.0 eth0: link up





      So, final conclusion is, when using mdadm to raid5 , yo have to use partitions, and also set type to raid, not sure why.

      By the way, hardware :
      Assrock Q1900 DC + PCIE expansion sata card, 2 x 4 sata Tb green , 2 x 4 sata Tb red nas , 2 x 200 sata gb disk