Lost mount for two HDD (mirror)

Hello everyone!

I caught an unexpected "surprise" – I lost (?) a RAID (one of two similar ones).

Two HDDs (4TB + 4TB) in one "mirror" work as before. But the "twin" on two 3+3 drives was beyond me.

The motherboard (and its connectors) are working properly – I checked with a separate HDD. The OMV sees these two HDDs via SMART (it's clean).

I tried to resolve this issue using the "recovery" icon, adding a new, clean HDD (6TB), but... it simply "added successfully" (without any obvious "we'll check and restore it now" actions).

Where should I look? What should I read/study?

Thanks!

ADD

Important clarification.

These two disks (RAID) contain very important data.

P.S.
I didn't make any settings or configuration changes, I just noticed that some folders (visible from the network) were no longer accessible. I compared which folders were on which mirrors – the puzzle came together.

-=-=-=-
6.9.16-1 (Shaitan)

AMD E-350 Processor (ASUS E35M1-I)

Linux 6.1.0-0.deb11.21-amd64
TWO RAID (EXT4) 2x4Tb + 2x3Tb

Gentlemen (especially developers), seriously?

Just yesterday, terabytes of archives containing documents and photos spanning many years existed, and today, suddenly, without any apparent reason or intervention from the OMV system user, it's all gone, and this (judging by the lack of any reaction from the developers and/or other users) is considered "normal" (?!).

Come on!

I repeat – without any so-called "extraordinary reasons," such as physical failure (of one of all hard drives at once). Without any prior notifications (smart status of the hard drive(s) or the system about a sharp deterioration in performance, for example, via email or in a log).

The entire physical assembly (motherboard, SSD, system drive, 2+2 hard drives) appears to be in working order. But... one RAID array (two identical HDDs [WD Red] in a "mirror") simply "disappeared," after several years of operation. And this doesn't spark interest (at least) among other users (the situation could potentially repeat itself for all user`s OMV).

NAS is for home use, it does not work 24/7, it is turned on for a few hours a day.

And the system developers are also (hopefully temporarily) staying "on the side"?!

Sorry, but this seems very strange.

P.S.
Perhaps I'm asking a question about the situation in the wrong thread/section of the site?

Thanks for starting the discussion. PRELIMINARY (currently in draft form).

- Regarding AI. I'm quite skeptical of these trendy "technologies," at least at the level they've reached in the public domain. They're currently offering too much absurd advice and other results.

- Regarding the use of RAID, the option I chose was optimal for the time and the goals I wanted to achieve (two dedicated hard drives (in a mirror) have a good chance of preserving information if one starts losing data).

This is a fairly basic approach. (Maybe not ideal, but...)

Besides, I wrote that one ride (1+1 HDD) in my NAS still works perfectly fine.

and the other (also 1+1 HDD) turned out to be "lost." (and this is one power supply, one motherboard (and possibly one configuration file)

- My NAS doesn't run 24/7. I usually turn it on in the evening, watch a movie or a couple of episodes of a TV series (DLNA), and turn it off in the morning in normal mode (via the web browser interface on my smartphone).

- When I open the NAS interface page, I sometimes see notifications about updates (various), but I usually launch them immediately. This is to say that (IMHO) the operating system should immediately display notifications about malfunctions or any possible errors. (I've never seen this before.)

Although, the equipment I work (in office or my workarea) with regularly can generate messages that I receive (promptly) in a smartphone app and/or by email—and this is already a proven technology for many developers, which really helps "monitor" the status. (Not a complaint, but "maybe a good idea")

Regarding system files (e.g., partition tables, etc.).

Which (possibly) could have been corrupted in my system (which led to this result).

If there are critical entries, then... why not implement their "standard duplication" (not backup with user). Or something similar, like two boot images in industrial switches (not consumer).

Even the microsoft-like option (with restore points), while not ideal, might make you think, "Is there something to this?"

Again, this is just a discussion.

- Why did you add another disk (after the "crash")? Again, I was confused by the icon's caption, which contains the word "recovery" (there was perhaps a ridiculous and naive expectation that the system would recognize this disk as "where to transfer recovered data" after adding it).

Apparently, this is a different perception of the user interface.

Regarding parachuting (as an example/analogy), I understand your thinking, but...

I was a parachuter and then graduated from flight school, and I can say that parachutists (in the true sense) ALWAYS jump with a reserve parachute.

Isn't that a RAID? (Especially when the reserve parachute is triggered automatically (based on altitude above ground) and opens independently of the user's will.)
On the contrary, the user must consciously unlock the automatic opening (when they are conscious and see that everything is proceeding normally).

Obviously, this is "literary" and a topic for discussion elsewhere, but... (Life is an interesting thing, especially when your horizons provide interesting ideas.)

Especially for beginners (regular users) who don't know how (or should they even?) to edit the operating system code

I'll certainly start exploring your proposed solution (thanks again for your response).

But... the situation itself (the fact that it could arise) that happened to probably gives reason to think about improvements/reliability.

I don't know the entire architecture of the operating system, so I can't offer a ready-made solution (even if I wanted to).

But... I hope you add/make changes (?) and make your product even more reliable and popular.

to geaves

Sorry for being "off-topic"
shortly

1.

old time, a city bus with a conductor:

A man enters and addresses the conductor:

- Two tickets, please!

- Excuse my curiosity, but why do you need TWO tickets?

- What if I lose one...

- Well, you might as well lose the other one?!

- I'm not an idiot! I have a season ticket for situations like this!

2.
Parachute systems can be VERY different.

For different payloads (weights), for different speeds (gradual "small to large" opening), for different numbers of main canopies. Even for different atmospheric densities (descent modules on different planets).

With additional braking systems (automatic jet streams in the opposite direction).

Differents vertical or horizontal speed.

Do you see where I'm (maybe) heading with this discussion?

For each task and capability, a specific solution is selected.

The same applies to different generations of file systems and raids.

Everything has its own subtleties and peculiarities, the main thing is that they work WITHOUT SURPRISES (were developed by good engineers and used correctly).

That way, everyone will be happy

P.S.

Then I get home, I'll check your recommendations.

Unfortunately, I'm far from the equipment now.

ADD/UPD (offtop, sorry)
Example solution of multi parachute system`s
Space and...
Includes links to studies (detailed, NASA) and examples of emergency work (Appolo-15).

Quote from geaves

First of all follow this thread to do this you will need to ssh into omv and run each option from the cli, post EACH OUTPUT into a code box this symbol </> on the forum thread bar, this makes it easier to read.

Greetings!

I hope this is done correctly (following the instructions in the link).

Also, below are a couple of screenshots from the OMV web interface (Shared Folders -> Access Control List section).

Two folders located on the "lost" "mirror" (1+1 HDD) display this message. Accordingly, another working RAID, when accessed from this menu (section), displays "other style" working information about properties/access/settings.

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [ra                                                 id10]
md1 : active (auto-read-only) raid1 sdb[0] sdc[1]
      2930134464 blocks super 1.2 [2/2] [UU]
      bitmap: 0/22 pages [0KB], 65536KB chunk

md0 : active raid1 sdd[0] sde[1]
      3906886464 blocks super 1.2 [2/2] [UU]
      bitmap: 0/30 pages [0KB], 65536KB chunk

unused devices: <none>

/dev/sda1: UUID="6ec3493d-af94-4995-b64a-a0ccdad58b92" BLOCK_SIZE="4096" TYPE="e                                                 xt4" PARTUUID="145c8c38-01"
/dev/sda5: UUID="aabc8be4-a666-48e1-82e5-92f21e4a8b88" TYPE="swap" PARTUUID="145                                                 c8c38-05"
/dev/sdd: UUID="400ada41-e92f-a306-7fe2-c9ba8c6f0382" UUID_SUB="34ec2a6a-765f-b4                                                 fa-7b8d-5524ffe11627" LABEL="OMV-NAS:0" TYPE="linux_raid_member"
/dev/sdc: UUID="e2506c62-ef7c-de6a-b5c7-d53a6c9faf56" UUID_SUB="8ab90c83-9494-a4                                                 32-f31a-fa927a83ab8f" LABEL="OMV-NAS:1" TYPE="linux_raid_member"
/dev/sde: UUID="400ada41-e92f-a306-7fe2-c9ba8c6f0382" UUID_SUB="33d685e7-e000-ff                                                 c8-0d6b-664c9cc483d5" LABEL="OMV-NAS:0" TYPE="linux_raid_member"
/dev/sdb: UUID="e2506c62-ef7c-de6a-b5c7-d53a6c9faf56" UUID_SUB="d362dd77-2d73-10                                                 bc-9829-a699cd19b603" LABEL="OMV-NAS:1" TYPE="linux_raid_member"
/dev/md0: UUID="55934491-2036-4b82-a47b-b7e2699deee0" BLOCK_SIZE="4096" TYPE="ex                                                 t4"
/dev/md1: UUID="ebaef05f-8d9b-410e-ae80-e2196623f840" BLOCK_SIZE="4096" TYPE="ex                                                 t4"

Disk /dev/sda: 119,24 GiB, 128035676160 bytes, 250069680 sectors
Disk model: SPCC Solid State
Disk identifier: 0x145c8c38
Disk /dev/sdd: 3,64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFRX-68W
Disk /dev/sdc: 2,73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFAX-68J
Disk /dev/sde: 3,64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFRX-68W
Disk /dev/sdb: 2,73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFAX-68J
Disk /dev/md0: 3,64 TiB, 4000651739136 bytes, 7813772928 sectors
Disk /dev/md1: 2,73 TiB, 3000457691136 bytes, 5860268928 sectors

# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# definitions of existing MD arrays
ARRAY /dev/md1 metadata=1.2 name=OMV-NAS:1 UUID=e2506c62:ef7cde6a:b5c7d53a:6c9faf56
ARRAY /dev/md0 metadata=1.2 name=OMV-NAS:0 UUID=400ada41:e92fa306:7fe2c9ba:8c6f0382
root@OMV-NAS:~# mdadm --detail --scan --verbose
ARRAY /dev/md0 level=raid1 num-devices=2 metadata=1.2 name=OMV-NAS:0 UUID=400ada41:e92fa306:7fe2c9ba:8c6f0382
   devices=/dev/sdd,/dev/sde
ARRAY /dev/md1 level=raid1 num-devices=2 metadata=1.2 name=OMV-NAS:1 UUID=e2506c62:ef7cde6a:b5c7d53a:6c9faf56
   devices=/dev/sdb,/dev/sdc

P.S.
There was a slight delay in response because the root password was written to a file on the "dropped" hard drives.

I had to remember (It's bad when you didn't know, and then you forgot) how to root pass.

Quote from geaves

And no I've no idea what causes the (auto-read-only) in what was originally a working array

Well, I'm completely at a loss here...
Thank you for finding the "moment" and the hint with the commands.

I'll try to more or less explain my "attitude" (toward OMV).

When I was choosing (testing) various systems, OMV was the one that "hooked" me with the impression that it was made by competent people (I work with a wide variety of equipment, and this is a "sixth sense" I've developed over many years).

When I started mastering the basics, I realized that this prediction was 99% correct.

After mastering the basics, when I go to check/configure/add something, I simply take a break from the modern "interfaces" (in other systems) that surround the average user and simply enjoy the thoughtfulness and convenience.

Something similar happens when I work with top-end Canon cameras (the "1s"): there are no "assistants"/"creative programs" or other "enhancers." There's simply a reliable tool that, with the right knowledge, is an extension of the eye and hand (and after which it's very difficult to shoot with a smartphone camera).

Therefore, even in this situation, I remain calm, understanding that something clearly unpredictable has occurred. BUT! One that can be easily (hopefully) corrected.

I have confidence, and that's the most important thing.

P.S.
Let's wait until evening and my return to this NAS (with this situation).

(I'd like to get confirmation of my text above)

Quote from geaves

try mdadm --readwrite /dev/md1 if that does not work, stop the array mdadm --stop /dev/md1 then re run the first command

root@OMV-NAS:~# mdadm --readwrite /dev/md1
root@OMV-NAS:~# mdadm --stop /dev/md1
mdadm: stopped /dev/md1
root@OMV-NAS:~# mdadm --readwrite /dev/md1
mdadm: error opening /dev/md1: No such file or directory
root@OMV-NAS:~#

The first command didn't work.

I tried it again (but after a stop) – no result...

Just for fun, I restarted the entire NAS – but it didn't work.

the system response is "open error + no file or directory", is this full "finish"?
Are specialized data recovery programs the only option? (NAS had no open ports/access to the external network, so it's unlikely there was an "intrusion")

Perhaps there's something else useful in the more detailed screenshots from the web interface to help us understand the situation?

P.S.

I find it strange that the first time I enter a command, there's no response.

And after stopping and entering the same command again, a "opening error" message appears.

There is another "observation" - when the readwright command is entered, the information in the dashboard remains the same as at startup.

But, after entering the stop command (for md1), the cell in the dashboard changes to this:

Perhaps there is some other command (possibly relevant for launching/mounting) in such a situation?

P.S.
mdadm -w /dev/md1

I'm very wary of "experimenting" (hoping that the data can still be saved/viewed), so I'm in no rush to try the options I found on the Internet (for fear of making things even worse)

Quote from geaves

So post the output of cat /proc/mdstat that will tell us the state of /md1

root@OMV-NAS:~# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdd[0] sde[1]
      3906886464 blocks super 1.2 [2/2] [UU]
      bitmap: 0/30 pages [0KB], 65536KB chunk

md1 : active (auto-read-only) raid1 sdb[0] sdc[1]
      2930134464 blocks super 1.2 [2/2] [UU]
      bitmap: 0/22 pages [0KB], 65536KB chunk

unused devices: <none>
root@OMV-NAS:~#

Quote from geaves

so run the mdadm readwrite command, but then post the output

root@OMV-NAS:~# mdadm --readwrite /dev/md1
root@OMV-NAS:~# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sdb[0] sdc[1]
      2930134464 blocks super 1.2 [2/2] [UU]
      bitmap: 0/22 pages [0KB], 65536KB chunk

md0 : active raid1 sdd[0] sde[1]
      3906886464 blocks super 1.2 [2/2] [UU]
      bitmap: 0/30 pages [0KB], 65536KB chunk

unused devices: <none>

Greetings!

I'm a bit "confused," but I did it (even if it's a repeat, just like before).

Or did I miss something in the sequence of actions?

I am certainly grateful for your attention and patience...

P.S.

I looked at various examples (examples of usage mdadm), but I found an article with a large number of commands that I'm still hesitant to use... (not understanding ALL the consequences).

Quote from geaves

now post the output of cat /etc/fstab

 cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# systemd generates mount units based on this file, see systemd.mount(5).
# Please run 'systemctl daemon-reload' after making changes here.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda1 during installation
UUID=6ec3493d-af94-4995-b64a-a0ccdad58b92 /               ext4    errors=remount                  -ro 0       1
# swap was on /dev/sda5 during installation
UUID=aabc8be4-a666-48e1-82e5-92f21e4a8b88 none            swap    sw                                0       0
# >>> [openmediavault]
/dev/disk/by-uuid/55934491-2036-4b82-a47b-b7e2699deee0          /srv/dev-disk-by                  -uuid-55934491-2036-4b82-a47b-b7e2699deee0      ext4    defaults,nofail,user_xat                  tr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
/dev/disk/by-uuid/ebaef05f-8d9b-410e-ae80-e2196623f840          /srv/dev-disk-by                  -uuid-ebaef05f-8d9b-410e-ae80-e2196623f840      ext4    defaults,nofail,user_xat                  tr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
# <<< [openmediavault]

Quote from geaves

but I can't check that because you blanked out the UUID's in your first post

Personally, it's not always "convenient" for me to display various *IDs (when not 100% sure) in the public domain; perhaps it's paranoia