Filesystem Mountpoint Missing

claredad · 12. Dezember 2020

Hi all.

Looking for some help...maybe I made a noob mistake, but one of my file systems is showing up as Missing in the Filesystem section of the web interface. If I use the DF command in SSH, its missing too..

The files system is made up of 3 drives in a RAID array. All three drives are showing us as present in the web interface.

I dont have the option to mount the filesystem in the web interface.

The missing file system is called /srv/dev-disk-by-label-Storage.

Not sure ifs relevant, but in the syslogs which Ive pasted here, it does say :

Lookup for '/srv/dev-disk-by-label-Storage' filesystem failed -- not found in /proc/self/mounts

Here are the contents of /proc/self/mount.

And here are the results of omv-salt deploy run fstab

Can anyone help?

macom · 13. Dezember 2020

Degraded or missing raid array questions

geaves · 13. Dezember 2020

This -> rootfs' space usage 86.3% matches resource limit [space usage > 85.0%] is also a problem and is probably linked to docker, + what hardware are you using.

claredad · 13. Dezember 2020

Zitat von macom

Degraded or missing raid array questions

Code

root@omv-server:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : inactive sdd[1](S) sdc[2](S) sdb[0](S)
      17581174536 blocks super 1.2
       
unused devices: <none>

Spoiler anzeigen

Code

root@omv-server:~# fdisk -l | grep "Disk "
Disk /dev/sdb: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Disk model: WDC WD6001FSYZ-0
Disk /dev/sda: 119.2 GiB, 128035676160 bytes, 250069680 sectors
Disk model: LITEONIT LCS-128
Disk identifier: 0x9685734b
Disk /dev/sdd: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Disk model: WDC WD6001FSYZ-0
Disk /dev/sdc: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Disk model: WDC WD6001FSYZ-0

Code

root@omv-server:~# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This configuration was auto-generated on Mon, 21 Sep 2020 15:28:38 +0000 by mkconf

Alles anzeigen

Code

root@omv-server:~# mdadm --detail --scan --verbose
INACTIVE-ARRAY /dev/md127 num-devices=3 metadata=1.2 name=pool-ipv6-pd-omv:Storage UUID=7686503f:2d11c7ae:48f9f41b:39240c72
   devices=/dev/sdb,/dev/sdc,/dev/sdd

Post type of drives and quantity being used as well

3 Three hard drives for storage (RAID) WD 6TB 3.5" Re 7200 RPM SATA III 128 MB Cache Bulk/OEM Enterprise Hard Drive (WD6001FSYZ)

1 SSD for the OS (one partition) and for storage of downloads before moving to RAID (second partition)

Post what happened for the array to stop working? Reboot? Power loss?I

was trying to free up disk space on OS partition as I couldnt log into WEB Interface. I ran the commands:Here are the commands run.

claredad · 13. Dezember 2020

Zitat von geaves

This -> rootfs' space usage 86.3% matches resource limit [space usage > 85.0%] is also a problem and is probably linked to docker, + what hardware are you using.

Thanks, yes something happened a few weeks ago where i had a power cut and my OS drive filled almost overnight. I was at about 50%, but then it jumped up to 100%. Using docker prune commands I got it down to the 86.3% so I could log in, but then the issues occured with RAID.

I am using a Dell Precision work station with an i5 processor, 4 GB RAM,

3 Three hard drives for storage (RAID) WD 6TB 3.5" Re 7200 RPM SATA III 128 MB Cache Bulk/OEM Enterprise Hard Drive (WD6001FSYZ)

1 SSD for the OS (one partition) and for storage of downloads before moving to RAID (second partition)

geaves · 13. Dezember 2020

Zitat von claredad

I am using a Dell Precision work station with an i5 processor, 4 GB RAM,

The Raid is fine it's just displaying as inactive, post the output from blkid and mdadm --detail /dev/md127

Was this a clean install of OMV5 or it an upgrade from 4?

claredad · 13. Dezember 2020

Zitat von geaves

The Raid is fine it's just displaying as inactive, post the output from blkid and mdadm --detail /dev/md127

Was this a clean install of OMV5 or it an upgrade from 4?

Code

root@omv-server:~# blkid
/dev/sdb: UUID="7686503f-2d11-c7ae-48f9-f41b39240c72" UUID_SUB="177c655c-90be-6e57-305d-a6e1477e4c24" LABEL="pool-ipv6-pd-omv:Storage" TYPE="linux_raid_member"
/dev/sda1: UUID="b470e674-d414-4ae1-b471-7c40a7b544a1" TYPE="ext4" PARTUUID="9685734b-01"
/dev/sda3: UUID="3a9d6220-5d48-4efd-9b36-a0996bb194dd" TYPE="ext4" PARTUUID="9685734b-03"
/dev/sda5: UUID="b78cf05b-c6c7-4a13-b2e3-1ceb42b3689d" TYPE="swap" PARTUUID="9685734b-05"
/dev/sdd: UUID="7686503f-2d11-c7ae-48f9-f41b39240c72" UUID_SUB="a675c434-10e8-4ed7-015d-c6e3341c19b7" LABEL="pool-ipv6-pd-omv:Storage" TYPE="linux_raid_member"
/dev/sdc: UUID="7686503f-2d11-c7ae-48f9-f41b39240c72" UUID_SUB="648cee8c-3f1e-fe3e-ff96-36149a8eae9d" LABEL="pool-ipv6-pd-omv:Storage" TYPE="linux_raid_member

Code

root@omv-server:~# mdadm --detail /dev/md127
/dev/md127:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 3
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 3

              Name : pool-ipv6-pd-omv:Storage
              UUID : 7686503f:2d11c7ae:48f9f41b:39240c72
            Events : 674505

    Number   Major   Minor   RaidDevice

       -       8       32        -        /dev/sdc
       -       8       48        -        /dev/sdd
       -       8       16        -        /dev/sdb

Alles anzeigen

Sorry, I ran blkid in my last message, but forgot to post.

This was a clean install.

geaves · 13. Dezember 2020

Zitat von claredad

This was a clean install.

Interesting as there is no reference to the Raid in mdadm conf, getting it running then sort that out.

mdadm --stop /dev/md127

mdadm --assemble --force -verbose /dev/md127 /dev/sd[bcd]

That should bring the raid back up, you'll have to wait for the rebuild.

claredad · 13. Dezember 2020

Zitat von geaves

Interesting as there is no reference to the Raid in mdadm conf, getting it running then sort that out.

mdadm --stop /dev/md127

mdadm --assemble --force -verbose /dev/md127 /dev/sd[bcd]

That should bring the raid back up, you'll have to wait for the rebuild.

Alles anzeigen

Code

root@omv-server:~# mdadm --stop /dev/md127
mdadm: stopped /dev/md127
root@omv-server:~# mdadm --assemble --force --verbose /dev/md127 /dev/sd[bcd]
mdadm: unrecognised metadata identifier: rbose
root@omv-server:~# mdadm --assemble --force --verbose /dev/md127 /dev/sd[bcd]
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdb is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdd is identified as a member of /dev/md127, slot 1.
mdadm: forcing event count in /dev/sdc(2) from 674505 upto 674512
mdadm: Marking array /dev/md127 as 'clean'
mdadm: added /dev/sdd to /dev/md127 as 1 (possibly out of date)
mdadm: added /dev/sdc to /dev/md127 as 2
mdadm: added /dev/sdb to /dev/md127 as 0
mdadm: /dev/md127 has been started with 2 drives (out of 3).

Alles anzeigen

I ran this, and in Filesystems, in the web interface i saw the new filesystem, /dev/127/. It was showing as normal.

If I do a reboot, it goes back to the old storage filesystem.

I ran the commands again, and md127 shows as online again. I have the option to ount, but when I do that I get the below:

Thanks for your ongoing help and patience!

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; mount -v --source '/dev/disk/by-label/Storage' 2>&1' with exit code '32': mount: /srv/dev-disk-by-label-Storage: cannot mount; probably corrupted filesystem on /dev/md127.
Error #0:
OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; mount -v --source '/dev/disk/by-label/Storage' 2>&1' with exit code '32': mount: /srv/dev-disk-by-label-Storage: cannot mount; probably corrupted filesystem on /dev/md127. in /usr/share/php/openmediavault/system/process.inc:182
Stack trace:
#0 /usr/share/php/openmediavault/system/filesystem/filesystem.inc(738): OMV\System\Process->execute()
#1 /usr/share/openmediavault/engined/rpc/filesystemmgmt.inc(931): OMV\System\Filesystem\Filesystem->mount()
#2 [internal function]: Engined\Rpc\OMVRpcServiceFileSystemMgmt->mount(Array, Array)
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#4 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('mount', Array, Array)
#5 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('FileSystemMgmt', 'mount', Array, Array, 1)
#6 {main}

Alles anzeigen

geaves · 13. Dezember 2020

Zitat von claredad

I ran this, and in Filesystems, in the web interface i saw the new filesystem, /dev/127/. It was showing as normal.
If I do a reboot, it goes back to the old storage filesystem.

why reboot? run those two commands and report back any errors, as I said in my previous post mdadm.conf contains no reference to the Raid and it should.

This -> mdadm: added /dev/sdd to /dev/md127 as 1 (possibly out of date) needs correcting, and now an issue trying to mount after the reboot due to this -> cannot mount; probably corrupted filesystem on /dev/md127

Do you have a backup of the data?

claredad · 13. Dezember 2020

I rebooted because I am an idiot and didnt understand what I was doing

I hadnt seen the error, and when when it showed as online in the file system, I thought everything was OK. 100% my fault. Sorry geaves...hard to help someone that shoots themselves in the foot

I dont easily have a backup of data, but as a last resort can get it, would just take time.

geaves · 13. Dezember 2020

What's the output of cat /proc/mdstat and mdadm --detail /dev/md127

TBH I'm not sure how to proceed with this, due to the out of date error and the filesystem error, what to fix first.

claredad · 13. Dezember 2020

Zitat von geaves

What's the output of cat /proc/mdstat and mdadm --detail /dev/md127

TBH I'm not sure how to proceed with this, due to the out of date error and the filesystem error, what to fix first.

Code

root@omv-server:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : active (auto-read-only) raid5 sdb[0] sdc[2]
      11720782848 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]
      bitmap: 18/44 pages [72KB], 65536KB chunk

unused devices: <none>

Code

root@omv-server:~#  mdadm --detail /dev/md127
/dev/md127:
           Version : 1.2
     Creation Time : Wed Jun  5 15:42:12 2019
        Raid Level : raid5
        Array Size : 11720782848 (11177.81 GiB 12002.08 GB)
     Used Dev Size : 5860391424 (5588.90 GiB 6001.04 GB)
      Raid Devices : 3
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Sat Dec 12 18:42:04 2020
             State : clean, degraded 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : pool-ipv6-pd-omv:Storage
              UUID : 7686503f:2d11c7ae:48f9f41b:39240c72
            Events : 674512

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       -       0        0        1      removed
       2       8       32        2      active sync   /dev/sdc

Alles anzeigen

Whatever you can do to help is great. At this point I am assuming the data is gone, so you cant do any more damage than me, and I wont be blaming you for anything.

geaves · 13. Dezember 2020

Zitat von claredad

At this point I am assuming the data is gone

Not necessarily, we live in hope just do not reboot, do not pass go, no get out of jail free card

See now we have another issue -> md127 : active (auto-read-only) raid5 sdb[0] sdc[2] it appears that sbd has been removed.

mdadm --readwrite /dev/md127 then post the output again of cat /proc/mdstat

claredad · 13. Dezember 2020

Zitat von geaves

Not necessarily, we live in hope just do not reboot, do not pass go, no get out of jail free card

See now we have another issue -> md127 : active (auto-read-only) raid5 sdb[0] sdc[2] it appears that sbd has been removed.

mdadm --readwrite /dev/md127 then post the output again of cat /proc/mdstat

Code

root@omv-server:~# mdadm --readwrite /dev/md127
root@omv-server:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : active raid5 sdb[0] sdc[2]
      11720782848 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]
      bitmap: 18/44 pages [72KB], 65536KB chunk

unused devices: <none>

I must not reboot

geaves · 13. Dezember 2020

Ok that raid is now active with two out of the three drives.

Storage -> Disks select the drive that's been removed sbd, click wipe on the menu, short, -> OK that will wipe the drive and make it available, when finished;

Raid Management -> select the raid -> Recover on the menu, the dialog should display that wiped drive, select it -> OK that should add the drive back to the raid and the raid should display as rebuilding -> WAIT until it's completed in the GUI then post cat /proc/mdstat blkid mdadm --detail /dev/md127 cat /etc/mdadm/mdadm.conf cat /etc/fstab

claredad · 13. Dezember 2020

Zitat von geaves

Ok that raid is now active with two out of the three drives.

Storage -> Disks select the drive that's been removed sbd, click wipe on the menu, short, -> OK that will wipe the drive and make it available, when finished;

Raid Management -> select the raid -> Recover on the menu, the dialog should display that wiped drive, select it -> OK that should add the drive back to the raid and the raid should display as rebuilding -> WAIT until it's completed in the GUI then post cat /proc/mdstat blkid mdadm --detail /dev/md127 cat /etc/mdadm/mdadm.conf cat /etc/fstab

Ok, rebuilding now, but when I click the apply button at the top of the window, it gives me an error (below). I rebooted a few times to try and fix it

Obviously i did nothing except paste error here. Rebuild percent is slowly going up.

Do I need to do anything?

Code

The configuration has been changed. You must apply the changes in order for them to take effect.

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; omv-salt deploy run --no-color initramfs 2>&1' with exit code '1': debian: Data failed to compile: ---------- ID 'update_initramfs' in SLS 'omv.deploy.initramfs.default' contains a short declaration (cmd.run) with a trailing colon. When not passing any arguments to a state, the colon must be omitted.
Error #0:
OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; omv-salt deploy run --no-color initramfs 2>&1' with exit code '1': debian:
    Data failed to compile:
----------
    ID 'update_initramfs' in SLS 'omv.deploy.initramfs.default' contains a short declaration (cmd.run) with a trailing colon. When not passing any arguments to a state, the colon must be omitted. in /usr/share/php/openmediavault/system/process.inc:182
Stack trace:
#0 /usr/share/php/openmediavault/engine/module/serviceabstract.inc(62): OMV\System\Process->execute()
#1 /usr/share/openmediavault/engined/rpc/config.inc(167): OMV\Engine\Module\ServiceAbstract->deploy()
#2 [internal function]: Engined\Rpc\Config->applyChanges(Array, Array)
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#4 /usr/share/php/openmediavault/rpc/serviceabstract.inc(149): OMV\Rpc\ServiceAbstract->callMethod('applyChanges', Array, Array)
#5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(588): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}('/tmp/bgstatusSX...', '/tmp/bgoutputFU...')
#6 /usr/share/php/openmediavault/rpc/serviceabstract.inc(159): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))
#7 /usr/share/openmediavault/engined/rpc/config.inc(189): OMV\Rpc\ServiceAbstract->callMethodBg('applyChanges', Array, Array)
#8 [internal function]: Engined\Rpc\Config->applyChangesBg(Array, Array)
#9 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#10 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('applyChangesBg', Array, Array)
#11 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Config', 'applyChangesBg', Array, Array, 1)
#12 {main}

Alles anzeigen

geaves · 13. Dezember 2020

Zitat von claredad

Do I need to do anything

the error could be attributed to the fact that it's not completed as the apply is doing update initramfs which you would do after the raid has finished rebuilding.

Zitat von claredad

Rebuild percent is slowly going up

That's difficult to quantify as the drives are 6Tb, it should display time remaining.

claredad · 13. Dezember 2020

Zitat von geaves

the error could be attributed to the fact that it's not completed as the apply is doing update initramfs which you would do after the raid has finished rebuilding.
That's difficult to quantify as the drives are 6Tb, it should display time remaining.

Ya, its saying about 700 minutes left. So when thats done I apply the changes?

geaves · 13. Dezember 2020

Zitat von claredad

So when thats done I apply the changes

At this moment, yes, although I have tested/created a raid on a vm and the apply just works, but then again that uses small virtual drives.

Filesystem Mountpoint Missing

KM0201 12. Dezember 2020

Jetzt mitmachen!

Ähnliche Themen

corrupt file system - Please help

File system missing after reboot

FTP: Error when trying to turn FTP OFF!

Filesystem status after disconnecting drive

All my data is missing! ZPOOL says there is data but nothing shows up! HELP!

Filesystem devices are missing after upgrade to OMV 4