Posts by tookys

tookys · Jul 29th 2025

Quote from Krisbee

If you mean mean this https://pve.proxmox.com/wiki/P…sk_to_Virtual_Machine_(VM), then OMV does not have full control of your disks as access still passes through a "virtualisation layer". Disk SMART data is only available on the PVE host. A glitch on either PVE or in the OMV guest can easily upset a MD RAID array with potential filesystem/data loss. Was this your config when suffered previous RAID failures?

If your only use of OMV is to create a storage array and share data, there are alternative ways to do this on PVE.

Yeah, thats what I've been doing, I've been having failures each time i try to expand my raid since i started this method, so that would make since.

I had opted to use a separate vm to handle the docker setup etc. Looks like the solution would be to pass through the SATA controller itself among other options. So I'll need to do some more digging into this to find a better solution.

tookys · Jul 29th 2025

I've tried running the code 3 times now, keeps ending in this error. I guess this means its dead?

Code

Relocating group 119230's inode bitmap to 3906473501...
Relocating group 119230's inode table to 3906473502...
Resize inode (re)creation failed: Inode checksum does not match inode.Continue? yes
ext2fs_read_inode: Inode checksum does not match inode while reading inode 7 in recreate inode
/dev/md0: ***** FILE SYSTEM WAS MODIFIED *****
Recreate journal? yes
Creating journal (262144 blocks): Block bitmap checksum does not match bitmap: while trying to create journal
e2fsck: aborted
/dev/md0: ***** FILE SYSTEM WAS MODIFIED *****

tookys · Jul 28th 2025

root@omv-main:~# fsck /dev/md0

Code

fsck from util-linux 2.38.1
e2fsck 1.47.2 (1-Jan-2025)
ext2fs_check_desc: Corrupt group descriptor: bad block for block bitmap
fsck.ext4: Group descriptors look bad... trying backup blocks...
Superblock has an invalid journal (inode 8).
Clear<y>? yes
*** journal has been deleted ***



Block bitmap for group 8128 is not in group.  (block 14706642446511422411)
Relocate<y>? yes
Inode bitmap for group 8128 is not in group.  (block 2872273462942019452)
Relocate<y>? yes
Inode table for group 8128 is not in group.  (block 9673797327109825064)
WARNING: SEVERE DATA LOSS POSSIBLE.
Relocate<y>? yes
One or more block group descriptor checksums are invalid.  Fix<y>? yes
Group descriptor 8128 checksum is 0xa3ac, should be 0x3d2e.  FIXED.
Block bitmap for group 8129 is not in group.  (block 12019803987863364621)
Relocate<y>? yes
Inode bitmap for group 8129 is not in group.  (block 4498001798006089232)
Relocate<y>? yes
Inode table for group 8129 is not in group.  (block 14735397743365752668)
WARNING: SEVERE DATA LOSS POSSIBLE.
Relocate<y>? cancelled!
Group descriptor 8129 checksum is 0xb09c, should be 0x19d0.  FIXED.



/dev/md0: ***** FILE SYSTEM WAS MODIFIED *****



/dev/md0: ********** WARNING: Filesystem still has errors **********

Display More

Should i just spam "yes" through all these questions?

tookys · Jul 27th 2025

root@omv-main:~# fsck /dev/md0

Code

fsck from util-linux 2.38.1
e2fsck 1.47.2 (1-Jan-2025)
ext2fs_check_desc: Corrupt group descriptor: bad block for block bitmap
fsck.ext4: Group descriptors look bad... trying backup blocks...
Superblock has an invalid journal (inode 8).
Clear<y>?

Should I clear ? Or is there something else I should have done?

tookys · Jul 25th 2025

The reshape / recovery is still proceeding, its currently in the recovery phase. It still has maybe another 20 hours left on the recovery phase

Do you think the File system will start working again once the recovery is complete?

From the gui I used the grow option to add the new drive to the raid.

tookys · Jul 25th 2025

So I currently have an issue where after adding a new drive to my raid 6, and expanding. The File system I had on the raid is no longer working in OMV.

I made a full backup of all the data on the raid on a separate computer before attempting the expansion. (I've had these kinds of issues before) But I would like to avoid the headache and time delays of recovering from backup if possible.

The raid some reason got confused when adding a drive, it lost contact with an existing drive in the process. and now thinks it has lost 2 drives.

I have waited 4 days to allow it to finish reshaping to the new drive, and it is currently in the drive recovery phase.

I plan to let it finish the recovery before starting to try and address the file system issue.

When looking at the file systems tab in OMV shows that the FS still exists, but is missing its data or isn't found anymore.

If i try and have it show more details it simply waits a long time and comes up with an error.

Trying to mount a new drive doesn't give me an option to remount it.

I'm not sure where to start trying to troubleshoot this.

The system is on a custom built server, the server runs proxmox, and runs OMV as a virtual machine within the server to act as the raid manager.

The HDD are passed through directly to the OMV VM. It is a qty of 5, 8tb Seagate Barracuda (Yes not ideal for a NAS/Raid but its where my budget is at)

root@omv-main:~# root@omv-main:~# cat /proc/mdstat

Code

Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid6 sdf[6] sdb[0] sde[5] sdc[3] sdd[1]
      23441682432 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/3] [UU_U_]
      [===>.................]  recovery = 18.1% (1416663108/7813894144) finish=2175.7min speed=49003K/sec
      bitmap: 0/59 pages [0KB], 65536KB chunk



unused devices: <none>

root@omv-main:~# blkid

Code

/dev/sdf: UUID="46adc905-340c-0d45-14c7-713a61913d82" UUID_SUB="f873a232-74e5-8f73-d27b-742f185b26bb" LABEL="openmediavault:0" TYPE="linux_raid_member"
/dev/sdd: UUID="46adc905-340c-0d45-14c7-713a61913d82" UUID_SUB="2efbbc23-2b02-34a2-fc6f-759c6bb5d013" LABEL="openmediavault:0" TYPE="linux_raid_member"
/dev/sdb: UUID="46adc905-340c-0d45-14c7-713a61913d82" UUID_SUB="011ac530-9fbe-8f97-8a40-04dd3a770295" LABEL="openmediavault:0" TYPE="linux_raid_member"
/dev/sr0: BLOCK_SIZE="2048" UUID="2024-02-17-11-39-02-00" LABEL="openmediavault 20240217-12:39" TYPE="iso9660" PTUUID="35659091" PTTYPE="dos"
/dev/sde: UUID="46adc905-340c-0d45-14c7-713a61913d82" UUID_SUB="14356ed5-d411-a762-51e2-063f43b5cf8a" LABEL="openmediavault:0" TYPE="linux_raid_member"
/dev/sdc: UUID="46adc905-340c-0d45-14c7-713a61913d82" UUID_SUB="1ee6b00e-4b3c-7c23-cd98-2d1887634a16" LABEL="openmediavault:0" TYPE="linux_raid_member"
/dev/sda5: UUID="fee293ad-128d-4a5b-9d70-485cb6d3d5dc" TYPE="swap" PARTUUID="8dc0e770-05"
/dev/sda1: UUID="718ac16e-c747-4f3f-bf42-8222b8998950" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="8dc0e770-01"
/dev/md0: UUID="2355b0f0-4f07-4560-936f-b7b0f4e74b6f" BLOCK_SIZE="4096" TYPE="ext4"

root@omv-main:~# fdisk -l | grep "Disk "

Code

Disk /dev/sda: 32 GiB, 34359738368 bytes, 67108864 sectors
Disk model: QEMU HARDDISK
Disk identifier: 0x8dc0e770
Disk /dev/sdc: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: QEMU HARDDISK
Disk /dev/sdd: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: QEMU HARDDISK
Disk /dev/sde: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: QEMU HARDDISK
Disk /dev/sdf: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: QEMU HARDDISK
Disk /dev/sdb: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: QEMU HARDDISK
Disk /dev/md0: 21.83 TiB, 24004282810368 bytes, 46883364864 sectors

Display More

root@omv-main:~# cat /etc/mdadm/mdadm.conf

Code

# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.



# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#



# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions



# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes



# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# definitions of existing MD arrays
ARRAY /dev/md0 metadata=1.2 spares=1 name=openmediavault:0 UUID=46adc905:340c0d45:14c7713a:61913d82

Display More

root@omv-main:~# mdadm --detail --scan --verbose

Code

ARRAY /dev/md0 level=raid6 num-devices=5 metadata=1.2 spares=2 name=openmediavault:0 UUID=46adc905:340c0d45:14c7713a:61913d82
   devices=/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde,/dev/sdf

tookys · Jul 9th 2024

Im not infront of my computer atm so i cant get then for you right this second, i was checking the debugfs right before i left for work this morning, so I shared what I had.

Yes the raid was originally made in omv 6, and i updated it to 7.

No i havent enabled email alerts

What other logs are you needing? I can give you a paste bin from journalctl or dmesg when i get home.

The whole point of a raid is if 1 disk fails the system should keep running, and a raid6 allows 2 disk failures. So a disk with a warning shouldnt be the end of the world.

I dont think it has anything to do with the raid, from the beginning the raid itself has said it was fine, the individual disks have said they were fine, omv says the raid is fine, raid forms fine, everything to do with the raid has said its 100% ok.

.

Is it best practice no, is it what caused this issue, id have to say no. Has it caused my issues in the past? Again id say no.

At this point ive accepted ive lost 4 years of work on my server.

At this point, im going to be starting from scratch and trying to do things better.

Yes i accept my own poor management has caused me to loose my work, and i have no one to blame but myself.

I do have some backups but not complete backups sadly.

tookys · Jul 9th 2024

Before i was able to use debugfs /dev/md127

And ls

I was able to see the internal folder structure of the raid even though it wasnt mounted.

Apparently the

rdump /path/to/copy /mounted/path/to/save

Would have allowed me to syart moving data off the drive.

Unfourtunately im no longer able to access the drive using debugfs im afraid something has happened and the data is lost now.

Debugfs isnt able to mount the drive anymore saying the superblock checksum no longer matches. And i havent found a work around.

tookys · Jul 9th 2024

cat /proc/mdstat

Code

Personalities : [raid0] [raid6] [raid5] [raid4] [linear] [multipath] [raid1] [raid10]
md124 : active raid0 sdb[0] sdj[1]
7813772288 blocks super 1.2 512k chunks


md125 : active raid0 sdm[0] sdl[1]
7813772288 blocks super 1.2 512k chunks


md126 : active raid0 sdk[1] sdd[0]
7813772288 blocks super 1.2 512k chunks


md127 : active raid6 md124[3](S) md125[1] sda[11] md126[2] sdc[10] sdg[6] sdf[5] sdh[9] sdn[8] sdi[7]
54695481344 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/9] [UUUUUUUUU]
bitmap: 0/59 pages [0KB], 65536KB chunk

Display More

The smart status says that all the drives are good, except sdf which is in warning, but its also always said that.

cat /etc/mdadm/mdadm.conf

Code

# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.


# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#


# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions


# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes


# automatically tag new arrays as belonging to the local system
HOMEHOST <system>


# definitions of existing MD arrays
ARRAY /dev/md/2x4tb2 metadata=1.2 name=2x4tb2 UUID=f65108e0:9ef74a7a:bd4890f4:3b434279
ARRAY /dev/md/2x4tb1 metadata=1.2 name=2x4tb1 UUID=2780e0b3:f8cdaf72:56e420c5:d9d08183
ARRAY /dev/md/2x4tb3 metadata=1.2 name=2x4tb3 UUID=b7967fed:0172a102:2d7f5a7f:3b3c8615
ARRAY /dev/md/Archive metadata=1.2 spares=1 name=Archive UUID=4ae4e034:54c4c2cd:8a9132af:c5db2629

Display More

tookys · Jul 9th 2024

omv-salt deploy run mdadm initramfs

Diff

debian:
----------
ID: update_initramfs_nop
Function: test.nop
Result: True
Comment: Success!
Started: 17:11:55.763656
Duration: 0.486 ms
Changes:
----------
ID: update_initramfs
Function: cmd.run
Name: update-initramfs -u
Result: True
Comment: Command "update-initramfs -u" run
Started: 17:11:55.764729
Duration: 19331.86 ms
Changes:
----------
pid:
6616
retcode:
0
stderr:
stdout:
update-initramfs: Generating /boot/initrd.img-6.1.0-22-amd64
----------
ID: remove_cron_daily_mdadm
Function: file.absent
Name: /etc/cron.daily/mdadm
Result: True
Comment: File /etc/cron.daily/mdadm is not present
Started: 17:12:15.099453
Duration: 1.361 ms
Changes:
----------
ID: divert_cron_daily_mdadm
Function: omv_dpkg.divert_add
Name: /etc/cron.daily/mdadm
Result: True
Comment: Leaving 'local diversion of /etc/cron.daily/mdadm to /etc/cron.daily/mdadm.distrib'
Started: 17:12:15.101538
Duration: 18.672 ms
Changes:
----------
ID: configure_default_mdadm
Function: file.managed
Name: /etc/default/mdadm
Result: True
Comment: File /etc/default/mdadm is in the correct state
Started: 17:12:15.120416
Duration: 137.841 ms
Changes:
----------
ID: configure_mdadm_conf
Function: file.managed
Name: /etc/mdadm/mdadm.conf
Result: True
Comment: File /etc/mdadm/mdadm.conf updated
Started: 17:12:15.258349
Duration: 107.784 ms
Changes:
----------
diff:
---
+++
@@ -20,7 +20,3 @@
HOMEHOST <system>


# definitions of existing MD arrays
-ARRAY /dev/md/2x4tb2 metadata=1.2 name=2x4tb2 UUID=f65108e0:9ef74a7a:bd4890f4:3b434279
-ARRAY /dev/md/2x4tb1 metadata=1.2 name=2x4tb1 UUID=2780e0b3:f8cdaf72:56e420c5:d9d08183
-ARRAY /dev/md/2x4tb3 metadata=1.2 name=2x4tb3 UUID=b7967fed:0172a102:2d7f5a7f:3b3c8615
-ARRAY /dev/md/Archive metadata=1.2 spares=1 name=Archive UUID=4ae4e034:54c4c2cd:8a9132af:c5db2629
----------
ID: mdadm_save_config
Function: cmd.run
Name: mdadm --detail --scan >> /etc/mdadm/mdadm.conf
Result: True
Comment: Command "mdadm --detail --scan >> /etc/mdadm/mdadm.conf" run
Started: 17:12:15.366225
Duration: 16.367 ms
Changes:
----------
pid:
11163
retcode:
0
stderr:
stdout:


Summary for debian
------------
Succeeded: 7 (changed=3)
Failed:    0
------------
Total states run:     7
Total run time:  19.614 s

Display More

it still doesn't mount.

tookys · Jul 7th 2024

Quote from Krisbee

tookys Probably need to wait until current activity on /dev/md127 is complete.

Not seen that journalctl error before.

It'll be a little under 24 hours for it to finish running its current check.

I'll try running the omv-salt deploy run mdadm initramfs once its done doing its check and give an update.

tookys · Jul 7th 2024

the journal command is returning different things when i run it

journalctl -g fsck

Code

Failed to iterate through journal: Bad message

Code

-- Boot ec983f550bcb4a62b3c4d716008c1fdd --
-- No entries --

Going through all the individual drives everything looked fine with mdadm --examine /dev/mdX

cat /etc/mdadm/mdadm.conf

Code

# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#


# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers


# automatically tag new arrays as belonging to the local system
HOMEHOST <system>


# instruct the monitoring daemon where to send mail alerts
MAILADDR root


# definitions of existing MD arrays


# This configuration was auto-generated on Mon, 24 Jul 2023 10:14:16 +0000 by mkconf

Display More

cat /proc/mdstat

Code

Personalities : [raid0] [raid6] [raid5] [raid4] [linear] [multipath] [raid1] [raid10]
md124 : active raid0 sdf[1] sdb[0]
7813772288 blocks super 1.2 512k chunks


md125 : active raid0 sdi[0] sdh[1]
7813772288 blocks super 1.2 512k chunks


md126 : active raid0 sdg[1] sdc[0]
7813772288 blocks super 1.2 512k chunks


md127 : active raid6 md124[3](S) md125[1] sdl[9] md126[2] sdk[6] sdm[7] sdj[5] sde[10] sda[11] sdn[8]
54695481344 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/9] [UUUUUUUUU]
[=>...................]  check =  9.3% (732814852/7813640192) finish=1598.7min speed=73815K/sec
bitmap: 0/59 pages [0KB], 65536KB chunk


unused devices: <none>

Display More

Right now it says its doing a check of the raid? should i let it finish before doing anything? could this fix itself?

@Krisbee Should i try the omv-salt deploy run mdadm initramfs still, would it hurt anything if its not the right scenario?

the error currently formed whne trying to moutn drive

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color fstab 2>&1' with exit code '1': debian:
----------
ID: create_filesystem_mountpoint_455bf2d6-cc55-45fc-a887-58640f1f5ca8
Function: file.accumulated
Result: True
Comment: Accumulator create_filesystem_mountpoint_455bf2d6-cc55-45fc-a887-58640f1f5ca8 for file /etc/fstab was charged by text
Started: 11:21:26.596423
Duration: 0.692 ms
Changes:
----------
ID: mount_filesystem_mountpoint_455bf2d6-cc55-45fc-a887-58640f1f5ca8
Function: mount.mounted
Name: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08
Result: False
Comment: mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
Started: 11:21:26.597678
Duration: 1851.704 ms
Changes:
----------
ID: append_fstab_entries
Function: file.blockreplace
Name: /etc/fstab
Result: True
Comment: No changes needed to be made
Started: 11:21:28.449763
Duration: 2.88 ms
Changes:
----------
ID: fstab_systemctl_daemon_reload
Function: module.run
Result: True
Comment: State was not run because none of the onchanges reqs changed
Started: 11:21:28.453358
Duration: 0.003 ms
Changes:


Summary for debian
------------
Succeeded: 3
Failed:    1
------------
Total states run:     4
Total run time:   1.855 s
[ERROR   ] Command 'mount' failed with return code: 32
[ERROR   ] stderr: mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
[ERROR   ] retcode: 32
[ERROR   ] mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.


OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color fstab 2>&1' with exit code '1': debian:
----------
ID: create_filesystem_mountpoint_455bf2d6-cc55-45fc-a887-58640f1f5ca8
Function: file.accumulated
Result: True
Comment: Accumulator create_filesystem_mountpoint_455bf2d6-cc55-45fc-a887-58640f1f5ca8 for file /etc/fstab was charged by text
Started: 11:21:26.596423
Duration: 0.692 ms
Changes:
----------
ID: mount_filesystem_mountpoint_455bf2d6-cc55-45fc-a887-58640f1f5ca8
Function: mount.mounted
Name: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08
Result: False
Comment: mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
Started: 11:21:26.597678
Duration: 1851.704 ms
Changes:
----------
ID: append_fstab_entries
Function: file.blockreplace
Name: /etc/fstab
Result: True
Comment: No changes needed to be made
Started: 11:21:28.449763
Duration: 2.88 ms
Changes:
----------
ID: fstab_systemctl_daemon_reload
Function: module.run
Result: True
Comment: State was not run because none of the onchanges reqs changed
Started: 11:21:28.453358
Duration: 0.003 ms
Changes:


Summary for debian
------------
Succeeded: 3
Failed:    1
------------
Total states run:     4
Total run time:   1.855 s
[ERROR   ] Command 'mount' failed with return code: 32
[ERROR   ] stderr: mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
[ERROR   ] retcode: 32
[ERROR   ] mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call. in /usr/share/php/openmediavault/system/process.inc:247
Stack trace:
#0 /usr/share/php/openmediavault/engine/module/serviceabstract.inc(62): OMV\System\Process->execute()
#1 /usr/share/openmediavault/engined/rpc/config.inc(187): OMV\Engine\Module\ServiceAbstract->deploy()
#2 [internal function]: Engined\Rpc\Config->applyChanges()
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(122): call_user_func_array()
#4 /usr/share/php/openmediavault/rpc/serviceabstract.inc(149): OMV\Rpc\ServiceAbstract->callMethod()
#5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(622): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}()
#6 /usr/share/php/openmediavault/rpc/serviceabstract.inc(146): OMV\Rpc\ServiceAbstract->execBgProc()
#7 /usr/share/openmediavault/engined/rpc/config.inc(208): OMV\Rpc\ServiceAbstract->callMethodBg()
#8 [internal function]: Engined\Rpc\Config->applyChangesBg()
#9 /usr/share/php/openmediavault/rpc/serviceabstract.inc(122): call_user_func_array()
#10 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod()
#11 /usr/sbin/omv-engined(544): OMV\Rpc\Rpc::call()
#12 {main}

Display More

tookys · Jul 7th 2024

Apparenlty this issue is the superblock for the raid is missing.

Code

root@openmediavault:~# mdadm --examine /dev/md127
mdadm: No md superblock detected on /dev/md127.

I am debating if to grow the raid with an additional drive i have installed would trigger it to reform the superblock. But i don't know enough to just start doing things willy nilly.

I'm not familiar with what good architecture for a raid setup would be. I would like to change it, but my server is already maxed out and im to scared to be making major changes.

This post has something similiar, and i think its the right solution, but i'm to nervous to do it without a bit more guidance.

https://serverfault.com/questi…ount-cant-read-superblock

I really thought the raid 6 would save me this headache. I plan to make a offsite backup of everything soon as i can get this back up and running.

tookys · Jul 6th 2024

it still has the same issue after a reboot

tookys · Jul 6th 2024

dmesg | grep "mount"

Code

[    6.210124] EXT4-fs (sdd1): mounted filesystem with ordered data mode. Quota mode: none.
[    6.882780] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
[    6.889418] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    6.890395] systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System...
[    6.891392] systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System...
[    6.892361] systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System...
[    6.912748] systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
[    6.915958] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    6.916107] systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.
[    6.916218] systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
[    6.916311] systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
[    6.919918] systemd[1]: Mounting sys-fs-fuse-connections.mount - FUSE Control File System...
[    6.920818] systemd[1]: Mounting sys-kernel-config.mount - Kernel Configuration File System...
[    6.923327] systemd[1]: Mounted sys-fs-fuse-connections.mount - FUSE Control File System.
[    6.923438] systemd[1]: Mounted sys-kernel-config.mount - Kernel Configuration File System.
[    6.928724] EXT4-fs (sdd1): re-mounted. Quota mode: none.
[    6.929932] systemd[1]: Finished systemd-remount-fs.service - Remount Root and Kernel File Systems.
[    8.105436] EXT4-fs (sdo1): mounted filesystem with ordered data mode. Quota mode: journalled.
[   66.561479] EXT4-fs (sdo1): re-mounted. Quota mode: journalled.

Display More

sudo fsck /dev/md127

Code

fsck from util-linux 2.38.1
e2fsck 1.47.0 (5-Feb-2023)
ext2fs_check_desc: Corrupt group descriptor: bad block for block bitmap
fsck.ext4: Group descriptors look bad... trying backup blocks...
Superblock has an invalid journal (inode 8).
Clear<y>? yes
*** journal has been deleted ***


fsck.ext4: e2fsck_read_bitmaps: illegal bitmap block(s) for Archive


Archive: ***** FILE SYSTEM WAS MODIFIED *****

Display More

tookys · Jul 6th 2024

I tried running

mdadm --stop /dev/md127 and rebooted, it still didn't want to mount after the reboote

Diff

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color fstab 2>&1' with exit code '1': debian:
----------
ID: create_filesystem_mountpoint_475cf9b6-f2cb-40b1-a5d8-3411c143f608
Function: file.accumulated
Result: True
Comment: Accumulator create_filesystem_mountpoint_475cf9b6-f2cb-40b1-a5d8-3411c143f608 for file /etc/fstab was charged by text
Started: 14:16:49.840267
Duration: 0.6 ms
Changes:
----------
ID: mount_filesystem_mountpoint_475cf9b6-f2cb-40b1-a5d8-3411c143f608
Function: mount.mounted
Name: /srv/dev-disk-by-uuid-8dd0c594-9f93-4acc-9008-1bbff4afb177
Result: True
Comment: Target was already mounted
Started: 14:16:49.841508
Duration: 35.812 ms
Changes:
----------
umount:
Forced remount because options (acl,user_xattr) changed
----------
ID: create_filesystem_mountpoint_da7f9a11-fd2d-4082-9ea1-9e07b90d8f36
Function: file.accumulated
Result: True
Comment: Accumulator create_filesystem_mountpoint_da7f9a11-fd2d-4082-9ea1-9e07b90d8f36 for file /etc/fstab was charged by text
Started: 14:16:49.877426
Duration: 0.746 ms
Changes:
----------
ID: mount_filesystem_mountpoint_da7f9a11-fd2d-4082-9ea1-9e07b90d8f36
Function: mount.mounted
Name: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08
Result: False
Comment: mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
Started: 14:16:49.878260
Duration: 79238.621 ms
Changes:
----------
ID: append_fstab_entries
Function: file.blockreplace
Name: /etc/fstab
Result: True
Comment: Changes were made
Started: 14:18:09.117332
Duration: 3.662 ms
Changes:
----------
diff:
---
+++
@@ -14,4 +14,5 @@
UUID=8d8b029b-290c-4b96-8577-2ae9d80e69b5 none            swap    sw              0       0
# >>> [openmediavault]
/dev/disk/by-uuid/8dd0c594-9f93-4acc-9008-1bbff4afb177		/srv/dev-disk-by-uuid-8dd0c594-9f93-4acc-9008-1bbff4afb177	ext4	defaults,nofail,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl	0 2
+/dev/disk/by-uuid/d1ebcbe5-f573-4a7f-becf-a91b7e43be08		/srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08	ext4	defaults,nofail,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl	0 2
# <<< [openmediavault]
----------
ID: fstab_systemctl_daemon_reload
Function: module.run
Result: True
Comment: service.systemctl_reload: True
Started: 14:18:09.121575
Duration: 397.549 ms
Changes:
----------
service.systemctl_reload:
True


Summary for debian
------------
Succeeded: 5 (changed=3)
Failed:    1
------------
Total states run:     6
Total run time:  79.677 s
[ERROR   ] Command 'mount' failed with return code: 32
[ERROR   ] stderr: mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
[ERROR   ] retcode: 32
[ERROR   ] mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.


OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color fstab 2>&1' with exit code '1': debian:
----------
ID: create_filesystem_mountpoint_475cf9b6-f2cb-40b1-a5d8-3411c143f608
Function: file.accumulated
Result: True
Comment: Accumulator create_filesystem_mountpoint_475cf9b6-f2cb-40b1-a5d8-3411c143f608 for file /etc/fstab was charged by text
Started: 14:16:49.840267
Duration: 0.6 ms
Changes:
----------
ID: mount_filesystem_mountpoint_475cf9b6-f2cb-40b1-a5d8-3411c143f608
Function: mount.mounted
Name: /srv/dev-disk-by-uuid-8dd0c594-9f93-4acc-9008-1bbff4afb177
Result: True
Comment: Target was already mounted
Started: 14:16:49.841508
Duration: 35.812 ms
Changes:
----------
umount:
Forced remount because options (acl,user_xattr) changed
----------
ID: create_filesystem_mountpoint_da7f9a11-fd2d-4082-9ea1-9e07b90d8f36
Function: file.accumulated
Result: True
Comment: Accumulator create_filesystem_mountpoint_da7f9a11-fd2d-4082-9ea1-9e07b90d8f36 for file /etc/fstab was charged by text
Started: 14:16:49.877426
Duration: 0.746 ms
Changes:
----------
ID: mount_filesystem_mountpoint_da7f9a11-fd2d-4082-9ea1-9e07b90d8f36
Function: mount.mounted
Name: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08
Result: False
Comment: mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
Started: 14:16:49.878260
Duration: 79238.621 ms
Changes:
----------
ID: append_fstab_entries
Function: file.blockreplace
Name: /etc/fstab
Result: True
Comment: Changes were made
Started: 14:18:09.117332
Duration: 3.662 ms
Changes:
----------
diff:
---
+++
@@ -14,4 +14,5 @@
UUID=8d8b029b-290c-4b96-8577-2ae9d80e69b5 none            swap    sw              0       0
# >>> [openmediavault]
/dev/disk/by-uuid/8dd0c594-9f93-4acc-9008-1bbff4afb177		/srv/dev-disk-by-uuid-8dd0c594-9f93-4acc-9008-1bbff4afb177	ext4	defaults,nofail,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl	0 2
+/dev/disk/by-uuid/d1ebcbe5-f573-4a7f-becf-a91b7e43be08		/srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08	ext4	defaults,nofail,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl	0 2
# <<< [openmediavault]
----------
ID: fstab_systemctl_daemon_reload
Function: module.run
Result: True
Comment: service.systemctl_reload: True
Started: 14:18:09.121575
Duration: 397.549 ms
Changes:
----------
service.systemctl_reload:
True


Summary for debian
------------
Succeeded: 5 (changed=3)
Failed:    1
------------
Total states run:     6
Total run time:  79.677 s
[ERROR   ] Command 'mount' failed with return code: 32
[ERROR   ] stderr: mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
[ERROR   ] retcode: 32
[ERROR   ] mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call. in /usr/share/php/openmediavault/system/process.inc:247
Stack trace:
#0 /usr/share/php/openmediavault/engine/module/serviceabstract.inc(62): OMV\System\Process->execute()
#1 /usr/share/openmediavault/engined/rpc/config.inc(187): OMV\Engine\Module\ServiceAbstract->deploy()
#2 [internal function]: Engined\Rpc\Config->applyChanges()
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(122): call_user_func_array()
#4 /usr/share/php/openmediavault/rpc/serviceabstract.inc(149): OMV\Rpc\ServiceAbstract->callMethod()
#5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(622): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}()
#6 /usr/share/php/openmediavault/rpc/serviceabstract.inc(146): OMV\Rpc\ServiceAbstract->execBgProc()
#7 /usr/share/openmediavault/engined/rpc/config.inc(208): OMV\Rpc\ServiceAbstract->callMethodBg()
#8 [internal function]: Engined\Rpc\Config->applyChangesBg()
#9 /usr/share/php/openmediavault/rpc/serviceabstract.inc(122): call_user_func_array()
#10 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod()
#11 /usr/sbin/omv-engined(544): OMV\Rpc\Rpc::call()
#12 {main}

Display More

tookys · Jul 6th 2024

I unmounted it from the gui, rebooted, and tried to remount. But same thing.

tookys · Jul 6th 2024

Code

Personalities : [raid0] [raid6] [raid5] [raid4] [linear] [multipath] [raid1] [raid10]
md124 : active raid0 sdh[1] sdi[0]
7813772288 blocks super 1.2 512k chunks


md125 : active raid0 sdc[0] sdg[1]
7813772288 blocks super 1.2 512k chunks


md126 : active raid0 sdf[1] sdb[0]
7813772288 blocks super 1.2 512k chunks


md127 : active raid6 md124[1] md126[3](S) md125[2] sdn[8] sda[11] sde[10] sdk[6] sdl[9] sdj[5] sdm[7]
54695481344 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/9] [UUUUUUUUU]
bitmap: 0/59 pages [0KB], 65536KB chunk


unused devices: <none>

Display More

I don't see any auto/read-only, so should i just reboot?

tookys · Jul 6th 2024

I ran the command, there was no output.

Code

root@openmediavault:~# mdadm --readwrite /dev/md127
root@openmediavault:~#

I tried to unmount and remount the raid in the gui, but it still has the same error.

tookys · Jul 6th 2024

here is the error that happens when i try and tell it to mount again.

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color fstab 2>&1' with exit code '1': debian:
----------
ID: create_filesystem_mountpoint_475cf9b6-f2cb-40b1-a5d8-3411c143f608
Function: file.accumulated
Result: True
Comment: Accumulator create_filesystem_mountpoint_475cf9b6-f2cb-40b1-a5d8-3411c143f608 for file /etc/fstab was charged by text
Started: 12:25:29.023131
Duration: 0.617 ms
Changes:
----------
ID: mount_filesystem_mountpoint_475cf9b6-f2cb-40b1-a5d8-3411c143f608
Function: mount.mounted
Name: /srv/dev-disk-by-uuid-8dd0c594-9f93-4acc-9008-1bbff4afb177
Result: True
Comment: Target was already mounted
Started: 12:25:29.024228
Duration: 72.563 ms
Changes:
----------
umount:
Forced remount because options (acl,user_xattr) changed
----------
ID: create_filesystem_mountpoint_3bf98f28-2d1f-4e4c-beda-3edeef19ed0a
Function: file.accumulated
Result: True
Comment: Accumulator create_filesystem_mountpoint_3bf98f28-2d1f-4e4c-beda-3edeef19ed0a for file /etc/fstab was charged by text
Started: 12:25:29.097130
Duration: 2.402 ms
Changes:
----------
ID: mount_filesystem_mountpoint_3bf98f28-2d1f-4e4c-beda-3edeef19ed0a
Function: mount.mounted
Name: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08
Result: False
Comment: mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
Started: 12:25:29.099855
Duration: 543.328 ms
Changes:
----------
ID: append_fstab_entries
Function: file.blockreplace
Name: /etc/fstab
Result: True
Comment: No changes needed to be made
Started: 12:25:29.643660
Duration: 2.316 ms
Changes:
----------
ID: fstab_systemctl_daemon_reload
Function: module.run
Result: True
Comment: State was not run because none of the onchanges reqs changed
Started: 12:25:29.646572
Duration: 0.003 ms
Changes:


Summary for debian
------------
Succeeded: 5 (changed=1)
Failed:    1
------------
Total states run:     6
Total run time: 621.229 ms
[ERROR   ] Command 'mount' failed with return code: 32
[ERROR   ] stderr: mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
[ERROR   ] retcode: 32
[ERROR   ] mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.


OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color fstab 2>&1' with exit code '1': debian:
----------
ID: create_filesystem_mountpoint_475cf9b6-f2cb-40b1-a5d8-3411c143f608
Function: file.accumulated
Result: True
Comment: Accumulator create_filesystem_mountpoint_475cf9b6-f2cb-40b1-a5d8-3411c143f608 for file /etc/fstab was charged by text
Started: 12:25:29.023131
Duration: 0.617 ms
Changes:
----------
ID: mount_filesystem_mountpoint_475cf9b6-f2cb-40b1-a5d8-3411c143f608
Function: mount.mounted
Name: /srv/dev-disk-by-uuid-8dd0c594-9f93-4acc-9008-1bbff4afb177
Result: True
Comment: Target was already mounted
Started: 12:25:29.024228
Duration: 72.563 ms
Changes:
----------
umount:
Forced remount because options (acl,user_xattr) changed
----------
ID: create_filesystem_mountpoint_3bf98f28-2d1f-4e4c-beda-3edeef19ed0a
Function: file.accumulated
Result: True
Comment: Accumulator create_filesystem_mountpoint_3bf98f28-2d1f-4e4c-beda-3edeef19ed0a for file /etc/fstab was charged by text
Started: 12:25:29.097130
Duration: 2.402 ms
Changes:
----------
ID: mount_filesystem_mountpoint_3bf98f28-2d1f-4e4c-beda-3edeef19ed0a
Function: mount.mounted
Name: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08
Result: False
Comment: mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
Started: 12:25:29.099855
Duration: 543.328 ms
Changes:
----------
ID: append_fstab_entries
Function: file.blockreplace
Name: /etc/fstab
Result: True
Comment: No changes needed to be made
Started: 12:25:29.643660
Duration: 2.316 ms
Changes:
----------
ID: fstab_systemctl_daemon_reload
Function: module.run
Result: True
Comment: State was not run because none of the onchanges reqs changed
Started: 12:25:29.646572
Duration: 0.003 ms
Changes:


Summary for debian
------------
Succeeded: 5 (changed=1)
Failed:    1
------------
Total states run:     6
Total run time: 621.229 ms
[ERROR   ] Command 'mount' failed with return code: 32
[ERROR   ] stderr: mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
[ERROR   ] retcode: 32
[ERROR   ] mount: /srv/dev-disk-by-uuid-d1ebcbe5-f573-4a7f-becf-a91b7e43be08: mount(2) system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call. in /usr/share/php/openmediavault/system/process.inc:247
Stack trace:
#0 /usr/share/php/openmediavault/engine/module/serviceabstract.inc(62): OMV\System\Process->execute()
#1 /usr/share/openmediavault/engined/rpc/config.inc(187): OMV\Engine\Module\ServiceAbstract->deploy()
#2 [internal function]: Engined\Rpc\Config->applyChanges()
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(122): call_user_func_array()
#4 /usr/share/php/openmediavault/rpc/serviceabstract.inc(149): OMV\Rpc\ServiceAbstract->callMethod()
#5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(622): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}()
#6 /usr/share/php/openmediavault/rpc/serviceabstract.inc(146): OMV\Rpc\ServiceAbstract->execBgProc()
#7 /usr/share/openmediavault/engined/rpc/config.inc(208): OMV\Rpc\ServiceAbstract->callMethodBg()
#8 [internal function]: Engined\Rpc\Config->applyChangesBg()
#9 /usr/share/php/openmediavault/rpc/serviceabstract.inc(122): call_user_func_array()
#10 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod()
#11 /usr/sbin/omv-engined(544): OMV\Rpc\Rpc::call()
#12 {main}

Display More

Posts by tookys

[Solved] FS error after raid expansion - Proxmox HDD Passthrough

[Solved] FS error after raid expansion - Proxmox HDD Passthrough

[Solved] FS error after raid expansion - Proxmox HDD Passthrough

[Solved] FS error after raid expansion - Proxmox HDD Passthrough

[Solved] FS error after raid expansion - Proxmox HDD Passthrough

[Solved] FS error after raid expansion - Proxmox HDD Passthrough

Raid is refusing to mount

Raid is refusing to mount

Raid is refusing to mount

Raid is refusing to mount

Raid is refusing to mount

Raid is refusing to mount

Raid is refusing to mount

Raid is refusing to mount

Raid is refusing to mount

Raid is refusing to mount

Raid is refusing to mount

Raid is refusing to mount

Raid is refusing to mount

Raid is refusing to mount