Failed to execute XPath query '//system/fstab/mntent[uuid=......... in docker after omv-regen

tdriver · 13. Januar 2024

hello,

after omv-regeneration and opening compose-->files, all files are missing and i'm getting this error .

Code

  Failed to execute XPath query '//system/fstab/mntent[uuid='39608cc1-326e-4e44-8eb1-b26fad5dfe21']'.

OMV\Config\DatabaseException: Failed to execute XPath query '//system/fstab/mntent[uuid='39608cc1-326e-4e44-8eb1-b26fad5dfe21']'. in /usr/share/php/openmediavault/config/database.inc:88
Stack trace:
#0 /usr/share/openmediavault/engined/rpc/sharemgmt.inc(1166): OMV\Config\Database->get('conf.system.fil...', '39608cc1-326e-4...')
#1 [internal function]: Engined\Rpc\ShareMgmt->getPath(Array, Array)
#2 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#3 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('getPath', Array, Array)
#4 /usr/share/openmediavault/engined/rpc/compose.inc(111): OMV\Rpc\Rpc::call('ShareMgmt', 'getPath', Array, Array)
#5 /usr/share/openmediavault/engined/rpc/compose.inc(275): OMVRpcServiceCompose->getComposePath(Array)
#6 [internal function]: OMVRpcServiceCompose->getFileList(Array, Array)
#7 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#8 /usr/share/php/openmediavault/rpc/serviceabstract.inc(149): OMV\Rpc\ServiceAbstract->callMethod('getFileList', Array, Array)
#9 /usr/share/php/openmediavault/rpc/serviceabstract.inc(620): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}('/tmp/bgstatus93...', '/tmp/bgoutputAp...')
#10 /usr/share/php/openmediavault/rpc/serviceabstract.inc(159): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))
#11 /usr/share/openmediavault/engined/rpc/compose.inc(340): OMV\Rpc\ServiceAbstract->callMethodBg('getFileList', Array, Array)
#12 [internal function]: OMVRpcServiceCompose->getFileListBg(Array, Array)
#13 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#14 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('getFileListBg', Array, Array)
#15 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Compose', 'getFileListBg', Array, Array, 1)
#16 {main}

Alles anzeigen

all containers are running, though. and i can connect to them.

i regenerated 2 times (just to be sure), and it seems this happens when i upgrade to proxmox kernel after regenerating.

most of the time ,including kernel in omv-regen ,stops regenerating somewhere in the in the middle , so i regen this way lately.

any ideas?

chente · 13. Januar 2024

Was the shared folder for compose files set to rootfs on the original system?

tdriver · 13. Januar 2024

no, shared folder is on data disk and in there are yml and env files only,because of zfs.(containers should not be on zfs)

chente · 13. Januar 2024

Zitat von tdriver

most of the time ,including kernel in omv-regen ,stops regenerating somewhere in the in the middle

I have tried this many times and have never had any errors. The omv-regen script installs the openmediavault-kernel plugin and only runs the openmediavault-kernel plugin script to install the kernel that was on the original system.

It would be helpful to see the shell output at the time of the error to know what is happening.

Zitat von tdriver

shared folder is on data disk and in there are yml and env files only,because of zfs.(containers should not be on zfs)

So you mean that the compose files are in the ZFS pool after regenerating? And they disappear after installing the proxmox kernel?

chente · 13. Januar 2024

Can you give me more information to try to reproduce this? I would need to know:

- OMV version, 6 or 7

- Define the ZFS pool. How many disks and type of Raid configured.

- Proxmox kernel version installed.

- What file system is docker installed on. Is it in /var/lib/docker or another disk?

- I assume there is a folder for compose files in the ZFS pool and another data folder also in the same pool. Confirm.

tdriver · 13. Januar 2024

i also did omv-regen countless times, never had this error before......

this one is a fresh install ,V6, 1 x zfs mirror, latest proxmox kernel, docker is on rootfs /var/lib/docker

Zitat von chente

So you mean that the compose files are in the ZFS pool after regenerating? And they disappear after installing the proxmox kernel?

the compose files are on the pool, but nothing disappears on the pool, everything is there and the containers even work.

the omv-page services->compose->files is just empty and the above error shows up.

the optional data folder is disabled. the backup folder is in the same location where the compose files are.

as for the regen process with or without kernel: my previous problem was boot/proxmox kernel related with some bug in the bios.

omv would freeze at loading init .ramdisk. this was fixed by installing omv on MBR instead of GPT.

so i assumed (wrongly?), regeneration in GPT mode, did stop sometimes the regen process cause of this bug.

i havn't tried regen with kernel in MBR mode though.

tdriver · 13. Januar 2024

to be more precise, and if gpt/mbr is irrelevant, the only thing changed in docker is: i added persistent volumes in /var/lib/docker/volumes (rootfs)

and also did docker backups via schedule on the pool , which btw are not there (in restore) . empty and same error ones opening the page.

(booting with original disk everything works)

Zitat von chente

It would be helpful to see the shell output at the time of the error to know what is happening.

tell me exactly where and what you want and i will send it when this happens again.

chente · 13. Januar 2024

Zitat von tdriver

i also did omv-regen countless times, never had this error before......
this one is a fresh install ,V6, 1 x zfs mirror, latest proxmox kernel, docker is on rootfs /var/lib/docker

I have tried to reproduce this and the regeneration has gone well. I haven't gotten any errors.

Zitat von tdriver

the compose files are on the pool, but nothing disappears on the pool, everything is there and the containers even work.
the omv-page services->compose->files is just empty and the above error shows up.
the optional data folder is disabled. the backup folder is in the same location where the compose files are.

as for the regen process with or without kernel: my previous problem was boot/proxmox kernel related with some bug in the bios.
omv would freeze at loading init .ramdisk. this was fixed by installing omv on MBR instead of GPT.
so i assumed (wrongly?), regeneration in GPT mode, did stop sometimes the regen process cause of this bug.

Alles anzeigen

This should not influence the omv-regen process. Whether the system supports that kernel or not depending on the hardware or the bios is independent. omv-regen doesn't do anything that can't be done in the OMV GUI.

Zitat von tdriver

the only thing changed in docker is: i added persistent volumes in /var/lib/docker/volumes (rootfs)

Perhaps this could be the cause of some error, although I don't see how it can influence the compose files to disappear. The script reads the information from the original database (those files are there) and copies it to the new database. The relevant salt changes are then applied, causing that information to be written to the compose folder in this case.

omv-showkey compose will tell you if that information is in the current database or not.

Anyway, since you have volumes in /var/lib/docker you should include this folder as part of the omv-regen backup so that it is restored as it was during the regeneration, that way you won't lose that information.

Zitat von tdriver

and also did docker backups via schedule on the pool , which btw are not there (in restore) . empty and same error ones opening the page.

The same as before, that information must be in the database.

Zitat von tdriver

tell me exactly where and what you want and i will send it when this happens again.

At the point where regeneration does not continue. At a minimum we will know at what point in the script it stops if there is no other relevant information.

ryecoaaron Have you any idea what could be happening in this system? I'm out of ideas.

ryecoaaron · 13. Januar 2024

The error says it is looking up the filesystem path for a sharedfolder and it can't find the filesystem. The output of the following might help:

sudo omv-showkey mntent

sudo lsblk

sudo blkid

grep srv /proc/mounts

chente · 13. Januar 2024

Zitat von ryecoaaron

The error says it is looking up the filesystem path for a sharedfolder and it can't find the filesystem

Yeah. But the question is why would one of the file systems be missing from the database?

ryecoaaron · 14. Januar 2024

Zitat von chente

the question is why would one of the file systems be missing from the database?

If it didn't mount for some reason, it would do this. That is why I asked for the output I did.

tdriver · 14. Januar 2024

chente

did another omv-regen, this time with kernel. again it stopped mid way. here is a screenshoot.

chente · 14. Januar 2024

Zitat von tdriver

did another omv-regen, this time with kernel. again it stopped mid way. here is a screenshoot.

For some reason while the proxmox kernel is being installed, the systemd-networkd service cannot be restarted and the connection is lost. This happens while the omv-installproxmox script is running. There seems to be a problem installing the proxmox library libudev1_247.3-7+1-pmx11u1

Code

...
Preparing to unpack .../5-libudev1_247.3-7+1-pmx11u1_amd64.deb ...
Unpacking libudev1:amd64 (247.3-7+1-pmx11u1) ...
Setting up systemd (247.3-7+1-pmx11u1) ...
Failed to try-restart systemd-networkd.service: Transaction for systemd-networkd.service/restart is destructive (network.target has 'stop' job queued, but 'start' is included in transaction).
See system logs and 'systemctl status systemd-networkd.service' for details.
Failed to try-restart systemd-resolved.service: Transaction for systemd-resolved.service/restart is destructive (reboot.target has 'start' job queued, but 'stop' is included in transaction).
See system logs and 'systemctl status systemd-resolved.service' for details.
client_loop: send disconnect: Connection reset

Maybe this would say something else:

systemctl status systemd-networkd.service

Anyway I don't think I can help with this, nothing to do with omv-regen. I'm almost sure that if you stopped the script at that point and installed the proxmox kernel from the GUI the error would reproduce. Maybe ryecoaaron has some ideas.

_______________________________________________________________________

tdriver Can you also publish the commands that ryecoaaron asked for on the system where you don't have the compose files? Maybe that way we can solve something.

tdriver · 14. Januar 2024

first of ,thank you all for your help.

i resolved the issue by creating a new orb-backup. afaik, the old (just yesterday) orb-backup must be corrupted somehow. i did more regenerations (same orb), all with the same error ,so i re-created a new orb-backup ,and voila even with kernel no error or stopping mid way !!

the only thing i have to do usually ,is to reinstall whatever is in apt-tool. (no big deal).

one question though, if i keep a good orb-backup for longer time, and regenerate it, what will the worst case scenario be ? just some newer versions plugins missing?

chente · 14. Januar 2024

Zitat von tdriver

i resolved the issue by creating a new orb-backup. afaik, the old (just yesterday) orb-backup must be corrupted somehow. i did more regenerations (same orb), all with the same error ,so i re-created a new orb-backup ,and voila even with kernel no error or stopping mid way !!

I'm glad it was just that. Anyway, it's strange, I would like to know how that happened and what caused this behavior. How did you copy the backup to the new system? In version 2 of omv-regen I made the backup packaged in tar to facilitate this movement but... And can you publish the content of the ORB_Dpkg and ORB_Unamea files of that backup?

If you could find differences between that backup and the most recent backup it could also be useful, especially in the /etc/openmediavault/config.xml file to somehow guarantee that it was not corrupted.

Zitat von tdriver

the only thing i have to do usually ,is to reinstall whatever is in apt-tool. (no big deal).

I'll look at that part of the script, all apttool packages should be reinstalled automatically during the rebuild.

Zitat von tdriver

one question though, if i keep a good orb-backup for longer time, and regenerate it, what will the worst case scenario be ? just some newer versions plugins missing?

The worst case scenario is that there is an update to the software packages corresponding to openmediavault or some essential plugin, these are those related to the file system: openmediavault-zfs openmediavault-lvm2 openmediavault-mergerfs openmediavault -snapraid openmediavault-md openmediavault-remotemount

In that case the regeneration will stop. If the update has been in any other plugin the regeneration will continue, the plugin will be installed but it will be left unconfigured, that is, just installed and ready for you to configure it manually in the OMV GUI.

chente · 14. Januar 2024

I still don't understand how a corrupt backup can cause the proxmox kernel to not be installed correctly. This makes no sense.

tdriver · 14. Januar 2024

I copy the backup to the new system via winscp.

contents of ORB_Unamea :

Code

Linux nas 6.2.16-20-bpo11-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-20~bpo11+1 (2023-12-01T14:42Z) x86_64 GNU/Linux

contents of ORB_Dpkg is too long, adding attachment.

ORB_Dpkg.txt

i'm checking the config.xml right now.

tdriver · 14. Januar 2024

Zitat von chente

I still don't understand how a corrupt backup can cause the proxmox kernel to not be installed correctly. This makes no sense.

me too, it's just a txt file, i will regen both ,the bad and the good again, later today, just to be sure it is nothing randomly.....

chente · 14. Januar 2024

Zitat von tdriver

the only thing i have to do usually ,is to reinstall whatever is in apt-tool. (no big deal).

Fixed in omv-regen 7.0.1 https://github.com/xhente/omv-…9c2fbb3915bc20eb33e209756

chente · 14. Januar 2024

Zitat von tdriver
I copy the backup to the new system via winscp.

contents of ORB_Unamea :
Code
Linux nas 6.2.16-20-bpo11-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-20~bpo11+1 (2023-12-01T14:42Z) x86_64 GNU/Linux
contents of ORB_Dpkg is too long, adding attachment.

ORB_Dpkg.txt

I don't see anything in all this that helps clarify what happened. Thank you anyway. Let's see if you find any difference in config.xml

Jetzt mitmachen!