Beiträge von jonni

I have decided to give up and did the following.

Zitat von geaves

The alternative is you start from again, disconnect the drives, reinstall OMV, connect each drive one at a time and wipe it, if you connect all the drives together the probability is it will detect the raid signatures on the drives.

I have also set up notifications as recommended by you:

Zitat von crashtest

With CRC's on all drives, I'd wonder how long that's being going on. Was it a recent event or over a longer period of time? If you had E-mail reports set up, as recommended in the User Guide, you'd have been aware of the problem as it started to develope.

I hope that I will see errors early if they occure in the future.

Thank you guys for your help and time.

Zitat von crashtest

Raid is not my thing and I'm coming in the middle so;

The attributes to look at, for each spinning drive, are:

SMART 5 – Reallocated_Sector_Count.

SMART 187 – Reported_Uncorrectable_Errors.

SMART 188 – Command_Timeout.

SMART 197 – Current_Pending_Sector_Count.

SMART 198 – Offline_Uncorrectable.

The following is usually related to cabling or interfaces, but it's worth mentions becuase an array may kick a drive out with CRC errors.

SMART 199 - UltraDMA CRC errors

Alles anzeigen

My results when executing : smartctl --attribues /dev/sdX (SMART 187 and 188 were missing from the table)

/dev/sdb
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE 
 5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0 
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0                                
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0                                
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       5        


/dev/sdc
 5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0 
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0                                      198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0                                      199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       2

/dev/sdd
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0 
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0                                      198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0                                      199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       18 

Zitat von crashtest

Have you ran mdadm --create ?

I have ran mdadm --create /dev/md1 --assume-clean -l5 -n4 -c512 /dev/sd[bcd] missing with only three out of four drives plugged in.

Zitat von crashtest

Also to confirm - you stay you've tested each drive in the array, with a SMART LONG drive test, correct? smartctl -s on -t long /dev/sd?

That is correct. All drives were tested and these tests completed without error.

Zitat von geaves

There is an option you could try before going down the complete testing route, which will take time.

Recreate/rebuild OMV on a USB flash drive or small hard drive, disconnect the data drives, clean install, update, shutdown, then connect all the data drives, the new install should pick up the array from the drive signatures. If that failed then you're back to testing and the systemrescuecd.

I have reinstalled OMV the way you described. OMV-gui looks the same way it did before. It recognizes my newly created RAID as shown in #25, but I can not access any data. I could only setup a new file system on that RAID, which, as I understand, would delete all my data as well.

I have set up a systemrescuecd usb drive, but I am not sure what to do nor if it is worth trying after all the tests I have already conducted. If you see a chance of still getting it to work that way, I will try it, but I am pretty close to giving up and reinstalling OMV once again, wiping all disks, setting up a new RAID and just losing the data since my backup.

Zitat von geaves

I'm sorry but I am somewhat at a loss as to why you have reposted back here after five days, I suggested you use a systemrescuecd to run commands from the cli in an attempt to ascertain what might be wrong. After suggesting that I was waiting for feedback, to proceed, but as I said this could be hardware related.

I did setup the systemrescuecd, but was not sure which commands I should run.

Because of that I used the following command on all drives via OMV-cli, which took me some days as each execution took about 6 hours.

smartctl -s on -t long /dev/sdX

All of these tests were "Completed without error".

Zitat

If you want confirmation regarding the error you posted in #1 have a look here

I have checked with the Bios and all drives look normal (size of 3tb shown)

smartctl -H /dev/sdX test results for the 3 currently connected drives are "PASSED".

If it is of any help I can post the results of smartctl --atributes -H /dev/sdX as well.

My only reason why I have not given up yet is the fact that I can actually see the RAID in the OMV gui again, where it has not shown up for weeks.

If you would tell me that using the mdadm --create command I have ruined my array anyways, I will just go ahead and do this:

Zitat

The alternative is you start from again, disconnect the drives, reinstall OMV, connect each drive one at a time and wipe it, if you connect all the drives together the probability is it will detect the raid signatures on the drives

I have checked all drives with smartctl and did not find any errors.

I then found the following link: https://unix.stackexchange.com…shed-linux-md-raid5-array

I connected only 3 out of 4 RAID drives and used the following command

mdadm --create /dev/md127 --assume-clean -l5 -n4 -c512 /dev/sd[bcd] missing

The RAID now shows up in OMV RAID management, but I can still not access any files nor can I mount the RAID in the OMV gui.

RAID details in OMV:

           Version : 1.2
     Creation Time : Thu Feb 11 20:12:32 2021
        Raid Level : raid5
        Array Size : 8790389760 (8383.17 GiB 9001.36 GB)
     Used Dev Size : 2930129920 (2794.39 GiB 3000.45 GB)
      Raid Devices : 4
     Total Devices : 3
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Thu Feb 11 20:12:32 2021
             State : clean, degraded
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : nas-Jonathan:127  (local to host nas-Jonathan)
              UUID : cc561cde:2f51df1d:50e87ce5:cff9b919
            Events : 3

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       8       16        1      active sync   /dev/sdb
       2       8       32        2      active sync   /dev/sdc
       -       0        0        3      removed

I would like to either try to repair the RAID or pull a backup and then wipe the drives and build a new RAID.

Zitat von geaves

Do you have a backup of your data?

Sadly I am not back at root@nas-Jonathan:~#

My last backup is about a year old which is not too bad for me, but it would definitely be nicer if I could get the RAID back.

Do you think there is no realistic chance of getting it back?

Zitat von geaves

have you inserted line 2 for reference to what it's doing?

Yes, it is only for reference.

Following the command I have an empty line in my terminal.

I can not put in new commands nor see any progress.

Zitat von geaves

When you say pending what is it displaying, TBH that --readwrite option should be instantaneous, but having no idea what else is happening that is why I said leave it.

It has not shown anything after I executed the command nor can I input any new commands as it is (apparently) still processing the readwrite command.

It looks like this at the moment:

root@nas-Jonathan:~# mdadm --readwrite /dev/md127
"empty line with flashing symbol"

Sorry, my English got a little rusty over the years.

I have left the system running over night (14 hours +) and the state has not changed yet. The command is still pending.

Any advice on how long I should leave it like that before giving up?

I have now left the readwrite command running for about 30-40 minutes, but for some reason it has not finished so far.

How much time should I give OMV to complete that command?

As sdX names changed, I altered the assemble command as well.

root@nas-Jonathan:~# mdadm --stop /dev/md127
mdadm: stopped /dev/md127

root@nas-Jonathan:~# mdadm --assemble --force --verbose /dev/md127 /dev/sd[bcd]
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdb is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdd is identified as a member of /dev/md127, slot 1.
mdadm: added /dev/sdd to /dev/md127 as 1
mdadm: added /dev/sdc to /dev/md127 as 2
mdadm: no uptodate device for slot 3 of /dev/md127
mdadm: added /dev/sdb to /dev/md127 as 0
mdadm: /dev/md127 has been started with 3 drives (out of 4).

root@nas-Jonathan:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : active (auto-read-only) raid5 sdb[0] sdc[2] sdd[1]
      8790405120 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
      bitmap: 0/22 pages [0KB], 65536KB chunk

unused devices: <none>

Zitat von geaves

WHY!!!! did I say reboot,

OK, I will leave it running then from now on.

When it was rebooting the reboot was not going through because of a task that was blocked. I am not too sure whether that task was finished or terminated after 10-15 minutes.

root@nas-Jonathan:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : inactive sdb[0](S) sdc[2](S) sdd[1](S)
      8790402407 blocks super 1.2
       
unused devices: <none>

Result:

root@nas-Jonathan:~# mdadm --stop /dev/md127
mdadm: stopped /dev/md127

root@nas-Jonathan:~# mdadm --assemble --force --verbose /dev/md127 /dev/sd[abc]
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdb is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdc is identified as a member of /dev/md127, slot 1.
mdadm: added /dev/sdc to /dev/md127 as 1
mdadm: added /dev/sdb to /dev/md127 as 2
mdadm: no uptodate device for slot 3 of /dev/md127
mdadm: added /dev/sda to /dev/md127 as 0
mdadm: /dev/md127 has been started with 3 drives (out of 4).

The reboot afterwards took about 10-15 minutes, but it did succeed.

The RAID does still not show up in the OMV gui "RAID management" though.

Thanks macom for the clarification. I was a little confused.

I have booted my OMV with only 4 drives (3 HDD plus SSD with the OS) instead of all 4 RAID HDDs as it does not finish booting with 4 HDDs.

Used RAID HDDs:

TOSHIBA DT01ABA3 (3tb, 3 drives)

Seagate IronWolf NAS HDD (3tb, one drive)

root@nas-Jonathan:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : inactive sdb[2](S) sda[0](S) sdc[1](S)
      8790402407 blocks super 1.2
       
unused devices: <none>

root@nas-Jonathan:~# blkid
/dev/sdb: UUID="1ae7b1fb-004c-10d2-5ee6-31b56d43d6a5" UUID_SUB="c1be727c-965a-bc95-de8f-4c699d4c72c4" LABEL="nas-jonathan:MeinRAID" TYPE="linux_raid_member"
/dev/sdd1: UUID="68bfc5a4-6add-4b8e-8e8a-f6965765bbc8" TYPE="ext4" PARTUUID="6ec3d6f8-01"
/dev/sdd5: UUID="26b08f75-df65-45ea-9e43-8d38352f5629" TYPE="swap" PARTUUID="6ec3d6f8-05"
/dev/sda: UUID="1ae7b1fb-004c-10d2-5ee6-31b56d43d6a5" UUID_SUB="f8ffd68f-5af4-7bb5-2f32-a68089552676" LABEL="nas-jonathan:MeinRAID" TYPE="linux_raid_member"
/dev/sdc: UUID="1ae7b1fb-004c-10d2-5ee6-31b56d43d6a5" UUID_SUB="4e51ff5c-b622-f65f-f29f-137b56169223" LABEL="nas-jonathan:MeinRAID" TYPE="linux_raid_member"

root@nas-Jonathan:~# fdisk -l | grep "Disk "
Disk /dev/sdb: 2,7 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA DT01ABA3
Disk /dev/sdd: 111,8 GiB, 120033041920 bytes, 234439535 sectors
Disk model: Samsung SSD 840 
Disk identifier: 0x6ec3d6f8
Disk /dev/sda: 2,7 TiB, 3000588754432 bytes, 5860524911 sectors
Disk model: TOSHIBA DT01ABA3
Disk /dev/sdc: 2,7 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: ST3000VN007-2E41

root@nas-Jonathan:~# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This configuration was auto-generated on Mon, 21 Sep 2020 15:28:38 +0000 by mkconf

root@nas-Jonathan:~# mdadm --detail --scan --verbose
INACTIVE-ARRAY /dev/md127 num-devices=3 metadata=1.2 name=nas-jonathan:MeinRAID UUID=1ae7b1fb:004c10d2:5ee631b5:6d43d6a5
   devices=/dev/sda,/dev/sdb,/dev/sdc

I was using OMV4 until a couple days ago, but my RAID 5 with 4 hard drives was not showing up in my OMV configurations any longer.

I then upgraded (reinstalled) to OMV5, but my RAID is still not showing up.

I tried to fix it through mdadm assebmling and it actually went through successfully (at least it looked that way to my newbie eyes), but as I tried to restart the machine, it would not even finish booting up. It gets stuck after init ramdisks with 4 errors "ataX: softreset failed (device not ready)".

As soon as I unplug one of the drives it shows only 3 errors and finishes booting afterwards.

These three drives (plus my SSD) show up in OMV gui "Disks".