How are your drives connected, SATA or USB? If a drive dropped out due to connectivity issue, a simple reboot maybe all you that you needed.
What output messages did you see when using that mdadm assemble command?
Please post the output of these commands: cat /proc/mdstat and blkid
As /dev/md127 is currently active with 3 out 4 drives, try failing /dev/sdc, then remove it from the array and add it back and see if it re-syncs OK.
mdadm /dev/md127 --fail /dev/sdc
mdadm /dev/md127 --remove /dev/sdc
mdadm /dev/md127 --add /dev/sdc
If it still complains it's /dev/sdc out of date, then perhaps you're going to have to wipe that drive before it can be added back to the array. But in that case you could wait to see if geaves comes along with an answer.
Display MoreAs /dev/md127 is currently active with 3 out 4 drives, try failing /dev/sdc, then remove it from the array and add it back and see if it re-syncs OK.
mdadm /dev/md127 --fail /dev/sdc
mdadm /dev/md127 --remove /dev/sdc
mdadm /dev/md127 --add /dev/sdc
If it still complains it's /dev/sdc out of date, then perhaps you're going to have to wipe that drive before it can be added back to the array.
OK, and is this to be run with the array stopped?
e.g. run this first? mdadm --stop /dev/md127
AFAIK no, you would not stop the array to remove a drive using the Webui. But let me re-state those commands.
mdadm /dev/md127 --fail /dev/sdc --remove /dev/sdc
mdadm --add /dev/md127 /dev/sdc
follow with a
mdam --detail /dev/md127
Oh, shoot....
Here's the errors on the terminal screen of the VM container when adding the /dev/sdc.
And also the SMART status in the WebUI of /dev/sdc directly after (was operating correctly before adding)
And now when I --fail /dev/sdc and --remove /dev/sdc it all goes back where we were before...
Post the output of cat /proc/mdstat post in a code box please, this symbol </> on the forum bar makes it easier to read
The output from #4 of the above shows the array as (auto-read-only), also to re add /dev/sdc with the 'Possibly out of date' error the drive will have to be securely wiped, this can usually be run to 25% then stopped, then try re adding the drive to the array. Do not add the drive until it has finished rebuilding and the (auto-read-only) is corrected.
Post the output of cat /proc/mdstat post in a code box please, this symbol </> on the forum bar makes it easier to read
The output from #4 of the above shows the array as (auto-read-only), also to re add /dev/sdc with the 'Possibly out of date' error the drive will have to be securely wiped, this can usually be run to 25% then stopped, then try re adding the drive to the array. Do not add the drive until it has finished rebuilding and the (auto-read-only) is corrected.
geaves So, I've already attempted to add the drive back, got errors, and performed a --fail /dev/sdc and --remove /dev/sdc to remove it. now it's back to the previous, non functional state. I hope nothing was damaged. 
How do I get the machine to rebuild the array with only 3 drives... to get the array out of auto-read-only?
Here's the cat /proc/mdstat
root@openmediavault:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md127 : active (auto-read-only) raid5 sdb[1] sde[3] sdd[2]
17581171200 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
bitmap: 0/44 pages [0KB], 65536KB chunk
unused devices: <none>I've allowed the array to finish syncing and the active (auto-read-only) property has disappeared.
Well that's a first
If anyone can chime in on the process of removing the "removed" drive and adding a new one
You can't, all mdadm is telling you is that it has removed a drive from the array, the --detail switch is giving you information
mdadm: add new device failed for /dev/sdc as 4: Invalid argument
if 25% secure wipe did not work I would suggest running a secure wipe on the whole drive then try again
Adding a drive can be done from the GUI, Raid Management, select the array and click recover on the menu and follow the on screen instructions
It's wouldn't hurt to do a long SMART test on that disk to check for bad sectors.
UPDATE 1/30
The /dev/sdc (original) and the /dev/sdf (spare I have from different machine) both finished "Secure Wipe" routines. When I try to add /dev/sdc to the array as a replacement using the WebUI, it returns this error message:
Anyone have any further ideas? I found a thread from another forum where someone battled a very similar issue, but using only mdadm, as whatever system he was using was NOT OMV. I'll add a screenshot of his solution and the URL to the forum below the code box. Is his solution safe for me at all?
OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; mdadm --manage '/dev/md127' --add /dev/sdc 2>&1' with exit code '1': mdadm: add new device failed for /dev/sdc as 4: Invalid argument in /usr/share/php/openmediavault/system/process.inc:197
Stack trace:
#0 /usr/share/openmediavault/engined/rpc/raidmgmt.inc(419): OMV\System\Process->execute()
#1 [internal function]: Engined\Rpc\RaidMgmt->add(Array, Array)
#2 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#3 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('add', Array, Array)
#4 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('RaidMgmt', 'add', Array, Array, 1)
#5 {main}The solution to another RAID5 disaster (like this one), and the URL
Basically it looks like he carefully and manually rebuilt the array and superblock data with a few (precise) commands.
https://www.linuxquestions.org/questions/linux-server-73/mdadm-error-replacing-a-failed-disk-909577/
LukeR1886 It's not a disaster, yet. But please DO NOT use any mdadm create command as per the link you posted at this stage.
Wait untill geaves comes along and confirms what I think. The failure to mount the array device means you need to do a filesystem check on the array device which with luck will reconstruct the necessary superblock info. Your degraded array should then be mountable and you will see what data can be retrieved.
QuoteCodemdadm: add new device failed for /dev/sdc as 4: Invalid argument
I think the RAID is in degraded mode and therefor it is not possible to add another (fourth) disk. You should post the output of
# cat /proc/mdstat
# mdadm --detail /dev/md127Don’t have an account yet? Register yourself now and be a part of our community!