mdadm: Cannot add disks to a 'member' array, perform this operation on the parent container

Well that's a mess, never seen anything like this before, kudos to you for having a go at upgrading the drives but in hindsight, which is great thing you would have better served asking in here

What I think you have done is nested an array within an array, how I don't know but this -> "I removed the 1st 2Tb disk from the Nas and copied the partition on the 1st 5Tb with GParted on another computer" was the first error. I understand why you did it, it makes sense, but that is not the way to upgrade your drives within mdadm. TBH I have no idea how to get out of this

The array is currently in a clean/degraded state (or one of them is or something is) do you have a backup?

How much data is on the array?

You say you are replacing with 5TB drives surely you mean 6TB?

Post the output of the following;

mdadm --detail /dev/md126

mdadm --detail /dev/md127 please the code box </> on the thread menu to paste the results makes it easier read.

Looking at the output of cat /proc/mdstat I am assuming that /dev/md127 is the actual array, two reasons;

1) It's inactive

2) mdadm determines whether an array is local or foreign md0=local md127=foreign, but device nodes work up from md0 and down from md127, so your md126 I believe is the nested one (this is just supposition at this stage)

The output of fdisk would also suggest, but not sure, that the /dev/loop is the remaining space or at least some of it on the current drives

Also mdadm is software it would require some user input somewhere for that nested array to be created

Quote from Youns

My data on the array takes 4.38Tb:

Well that's a bonus

I'm going to answer your questions before I make a suggestion on how you should proceed

Quote from Youns

for the upgrade I've prepared 4disks: 2x5Tb and 2x6Tb like in the picture below, which means that it's like 4x 5Tb....no

Yes, the array would be built upon the smallest drive within the array

Quote from Youns

I my case what would I had to do to upgrade my nas properly and easlily

Sorry there is no easy way, I am more concerned about preserving your data initially

Quote from Youns

As we can see the sdb drive is in the md127 array..... as you said, it looks like md127 (container) is nested in md126.....how is it possible????

Don't know never seen that before

Quote from Youns

apparently not cloning the partitions, are those steps the proper way

Whilst I am aware of cloning partitions within an array this is only ever down when facing a sh!t storm within a non functioning array.

The correct way is to fail then remove a drive from the array, that means using mdadm --fail then mdadm --remove, that way mdadm knows what is happening, it's software it has to have instruction. You then add a new drive and allow the array to rebuild, you do this for each drive that you want to replace.

Downside to this procedure, which has been performed more than once on the forum, is a drive physically failing during one of the rebuilds, this has also happened on the forum.

_________________________________________________________________________________________________________________

This is how I would approach this, you're not going to like it -> reinstall

Remove the drive /dev/sdb from the server, make sure it's that drive, leaving the array in clean/degraded state, install one of the 6TB wipe it then add an ext4 filesystem.

Once complete use rsync to copy the data to that 6TB drive, I use rsync but you'll need to perform this from the root of that array to the 6TB (if you do this I can tag someone to help) this can be done from the GUI.

When rsync is complete disconnect all data drives and reinstall OMV, update, set time etc, then connect the 2x5TB and the remaining 6TB create an array with those 3 drives add a filesystem, shutdown, connect the 6TB with the data on it and rsync it back to the new array. You'll then have to set your shared folders etc.

Once that's done wipe that 6TB drive using secure then grow the array with that drive giving you the 4 new drives.

OK I can see you now saying.......no way!!! and I fully understand that, but this is the safest and cleanest way of doing this, trying to sort your current mess is far riskier, one wrong command and your going to be doing that anyway.

I can't even set this up in a VM to try and work out how to resolve it because I simply don't know how it happened

Quote from Youns

I think that I won't have to reinstall, and I would proceed like so, after reinserting the 4 new drives

Sounds like a plan, good for you never gave that a thought, but>>>>> there's always a but or an however

You'll have to 'clean' your current installation and you can only do that once that data has been backed up, so to this;

remove/delete all smb shares

remove/delete all shared folders

remove/delete the raid array in raid management, if the array is mounted in file systems it will have to be unmounted first before deleting the array check to ensure there is no reference to the array in file systems after it's deletion from raid management

shutdown the server disconnect and remove the old drives, reboot, this is so you can check there are no residual parts of the previous array, even run a cat /proc/mdstat from ssh

Why clean, because your current setup will be stored within OMV's database file

If it all looks good then shutdown install the new drives and start again

I'm going to take a guess as well that the cloning had something to do with the current state of the array, what, I don't know, but it's the only thing that makes sense.

Quote from BernH

I think the re-install recommendation from geaves was meant to ensure there is nothing wrong with the install that would create the issue again and/or cause a problem with the in-place upgrade on the last disk that he was proposing

Exactly a clean install would guarantee there was nothing left from the current setup and a clean install takes what...........30 mins, it's the setting up and updating that takes the time

For a me a clean install and full setup takes a few hours, it's the data restore that is the most time consuming

Quote from Youns

Should I really reinstall?

If you want you can try this. How to regenerate a complete OMV system. If you describe your system to me, I will tell you if you can have complications or not, and I can help you overcome them. I would need to know fundamentally the output of

dpkg -l | grep openmediavault

With that I could give you certain special instructions if necessary.

It can save you all the time reconfiguring your system after a clean install.

You don't really lose anything, you can do this on a new USB, if it doesn't work for any reason go back to your original system drive. And it's quick to do.

Quote from Youns

There is absolutely nothing that I can keep from that database

No, especially as far as shares are concerned as the change of the raid config is going to create new UUID's

Quote from Youns

reinstalling all my plugins and containers (5 installed from Portainer) and configure them and all the others small things running aside will take much more time

This is where the use of Portainer stacks come in to their own, you copy and paste a stack into an editor such as notepad ++ all you do then is to change the newly created bind mounts that point to the newly created shares on the new raid setup

What will take the time is the creation of the new raid, adding a filesystem and putting the data back

Quote from Youns

here is the output for that command:

I have never tried a regeneration with these plugins

openmediavault-apttool -> I don't know what exactly it does. But if they are just utilities to manage apt it shouldn't be a problem.

openmediavault-cputemp -> You will have to install sensors again and configure it. The plugin does not.

openmediavault-diskstats -> I don't know what exactly it does. But if it's just disk usage statistics, it shouldn't be a problem.

openmediavault-fail2ban -> I don't know how this plugin works, but I guess it just creates rules in the frontend so it shouldn't be a problem.

openmediavault-tftp -> No idea. I don't know how it works. In the worst case reconfigure.

openmediavault-usbbackup -> Should not be a problem. It is similar to other disc recognition settings etc.

It seems to me that they shouldn't cause any problems but I can't guarantee it. It's just a matter of trying. The other plugins I know will work fine.

Quote from Youns

all the others small things running aside

Do you have configurations done in CLI? If you have read the link you will know that they are going to disappear. You will have to redo this later.

Quote from Youns

and containers (5)

I don't see openmediavault-compose. Do you have containers installed on CLI and/or with Portainer?

Quote from Youns

Any way to backup the containers database?

Where is docker installed?

Quote from Youns

Hi, As I wrote it I installed them from Portainer, that's why I don't have openmediavault-compose.

Well, in your first post this was not there, you edited it while I was replying. Does not matter. The point is that you will have to reinstall them because you have them in Portainer, if you had them in openmediavault-compose it would be all automatic.

Quote from Youns

I like the idea of geaves to copy the stacks in notepad ++ and rearange them lon the new install.

That is precisely what openmediavault-compose does. If you have the stack of all containers I would suggest using it, after a minimal learning curve it is more useful than Portainer.

Quote from Youns

Docker is installed in OMV extra, see the attached capture.

That will go away. Do you have any reason to make a backup of that? Where are your container configuration volumes?

Quote from Youns

Don't understand your question sorry:

In all (or almost all) containers you will have volumes transferred to the container. The folders where you store the container's configurations, data folders for the container to access, etc.

What I'm interested in knowing is where you have the configuration folders. I just want to make sure they aren't in / as they would be gone. If you have them on any other drive, then all is well.

The /var/lib/docker folder is not important, unless you have some special reason, containers designed by you, etc.

I just tested the regeneration in a vm configured the same as your server. Everything has worked fine.

You will only find the following:

openmediavault-apttool-> If you had packages installed, they will appear in the list, but not installed. Just select and reinstall. Now I know what apttool is for

openmediavault-cputemp-> You will have to reinstall lm-sensors and configure sensors

Everything else works without a problem.

What I did to simulate your case is to create a raid mdadm. So I diverted all shared folders to folders on another drive, you said you have one for docker, you can use that one temporarily.

Then I copied the data with the script that you have in the last post of the thread that I told you about.

Before regenerating, look in the config.xml of the backup for the fstab section. Select the mntent section corresponding to your Raid and delete it. It will be a section similar to this one, use the uuid to identify it.

      <mntent>
        <uuid>0049445f-0dba-4a81-9246-b76f8cb35e26</uuid>
        <fsname>/dev/disk/by-uuid/6e060096-a7c0-4861-a7e6-bdeef1b45b69</fsname>
        <dir>/srv/dev-disk-by-uuid-6e060096-a7c0-4861-a7e6-bdeef1b45b69</dir>
        <type>ext4</type>
        <opts>defaults,nofail,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl</opts>
        <freq>0</freq>
        <passno>2</passno>
        <hidden>0</hidden>
        <usagewarnthreshold>85</usagewarnthreshold>
        <comment></comment>
      </mntent>

Make a copy before. Make sure you don't touch the spaces of other lines, it's an xml file, the spaces are important, if you touch them it will stop working.

Follow the thread procedure. If you have doubts ask me.

One more thing, if you copy the original /etc/passwd and /etc/shadow files you will also have your users with their respective passwords. It will only be necessary to put them in the groups in which they were in addition to users. I keep trying to perfect this procedure.

By the way, I would take the opportunity to install docker on another disk, not in /var/lib/docker and I would also take the opportunity to start using openmediavault-compose.

I have stayed out of this thread because I am totally confused as to what is going on!!

The total clean/new install would be my approach as I stated creating a new raid, filesystem and copying the data back is the time consumption

But Youns #6 got my grey cells working and my #8 lays out how to go about doing that, that is simply to remove all references to the current raid setup, then remove the raid, nothing else changes within the current omv install

Once the new raid and shared folders are running the only thing to change then are the bind mounts within any docker containers , why? because with the creation of a new array there will be the changes to the drive/raid UUID

If a new install was initiated then the regeneration option would be a consideration, but if Youns you don't want to go down that route then any reference to the old raid setup, including the array must be removed using the GUI for the system to hopefully remain stable.