Applying pending configuration changes results in 500 internal server errors in webui

rodhull · 5. Oktober 2022

Most of the time (not always) after a manual apt upgrade or dist-upgrade (which will complete without errors) when next visiting the webUI I am greeted with a yellow "Pending configuration changes" banner (despite not having made any changes myself).

If I hit the tick to apply them, after a few seconds I get an internal server error - specifically this:

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color fstab 2>&1' with exit code '1': omv:
Data failed to compile:
----------
Rendering SLS 'base:omv.deploy.fstab.15unionfilesystems' failed: Jinja error: 'NoneType' object has no attribute 'get_dict'
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 497, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "<template>", line 1, in top-level template code
File "/usr/lib/python3/dist-packages/jinja2/sandbox.py", line 465, in call
return __context.call(__obj, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader.py", line 1235, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader.py", line 2268, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader.py", line 2283, in _run_as
return _func_or_method(*args, **kwargs)
File "/var/cache/salt/minion/extmods/modules/omv_conf.py", line 45, in get
return objs.get_dict()
AttributeError: 'NoneType' object has no attribute 'get_dict'


; line 1


---
{% set config = salt['omv_conf.get']('conf.service.unionfilesystems') %}    <======================


{% for pool in config.filesystem %}
{% set poolmount = salt['omv_conf.get_by_filter'](
'conf.system.filesystem.mountpoint',
{'operator':'stringEquals', 'arg0':'uuid', 'arg1':pool.self_mntentref}) %}
[...]
---


OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color fstab 2>&1' with exit code '1': omv:
Data failed to compile:
----------
Rendering SLS 'base:omv.deploy.fstab.15unionfilesystems' failed: Jinja error: 'NoneType' object has no attribute 'get_dict'
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 497, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "<template>", line 1, in top-level template code
File "/usr/lib/python3/dist-packages/jinja2/sandbox.py", line 465, in call
return __context.call(__obj, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader.py", line 1235, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader.py", line 2268, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader.py", line 2283, in _run_as
return _func_or_method(*args, **kwargs)
File "/var/cache/salt/minion/extmods/modules/omv_conf.py", line 45, in get
return objs.get_dict()
AttributeError: 'NoneType' object has no attribute 'get_dict'


; line 1


---
{% set config = salt['omv_conf.get']('conf.service.unionfilesystems') %}    <======================


{% for pool in config.filesystem %}
{% set poolmount = salt['omv_conf.get_by_filter'](
'conf.system.filesystem.mountpoint',
{'operator':'stringEquals', 'arg0':'uuid', 'arg1':pool.self_mntentref}) %}
[...]
--- in /usr/share/php/openmediavault/system/process.inc:197
Stack trace:
#0 /usr/share/php/openmediavault/engine/module/serviceabstract.inc(62): OMV\System\Process->execute()
#1 /usr/share/openmediavault/engined/rpc/config.inc(170): OMV\Engine\Module\ServiceAbstract->deploy()
#2 [internal function]: Engined\Rpc\Config->applyChanges(Array, Array)
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#4 /usr/share/php/openmediavault/rpc/serviceabstract.inc(149): OMV\Rpc\ServiceAbstract->callMethod('applyChanges', Array, Array)
#5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(588): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}('/tmp/bgstatus3N...', '/tmp/bgoutputeO...')
#6 /usr/share/php/openmediavault/rpc/serviceabstract.inc(159): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))
#7 /usr/share/openmediavault/engined/rpc/config.inc(192): OMV\Rpc\ServiceAbstract->callMethodBg('applyChanges', Array, Array)
#8 [internal function]: Engined\Rpc\Config->applyChangesBg(Array, Array)
#9 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#10 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('applyChangesBg', Array, Array)
#11 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Config', 'applyChangesBg', Array, Array, 1)
#12 {main}

Alles anzeigen

I can undo the "changes" (whatever they are!) and it seems to go away until a future time I might do another apt update...

ala.frosty · 7. Januar 2023

I'm having almost the exact same trouble. Except for me, after I do the undo, even the web GUI won't let me make changes, throwing the 500 - internal server error. And the error message is massive! I'm running a Proxmox kernel. Error message is too large to include here due to 10,000 char limit!!!

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color samba 2>&1' with exit code '1': debian:

rodhull · 18. April 2023

This is still a problem on my system. As I originally posted, *sometimes* (not always) after having run an "apt dist-upgrade" when re-visiting the webUI I will be greeted by a warning that some changes need to be applied. When I attempt to apply them it generates a 500 error.

For me it is when it tries to run "omv-salt deploy run --no-color fstab" and the errors are relating to mergerfs or the unionfs plugin or something.

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color fstab 2>&1' with exit code '1': omv:
    Data failed to compile:
----------
    Rendering SLS 'base:omv.deploy.fstab.15unionfilesystems' failed: Jinja error: 'NoneType' object has no attribute 'get_dict'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 497, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3/dist-packages/jinja2/sandbox.py", line 465, in call
    return __context.call(__obj, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1235, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2268, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2283, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/var/cache/salt/minion/extmods/modules/omv_conf.py", line 44, in get
    return objs.get_dict()
AttributeError: 'NoneType' object has no attribute 'get_dict'

; line 1

---
{% set config = salt['omv_conf.get']('conf.service.unionfilesystems') %}    <======================

{% for pool in config.filesystem %}
{% set poolmount = salt['omv_conf.get_by_filter'](
  'conf.system.filesystem.mountpoint',
  {'operator':'stringEquals', 'arg0':'uuid', 'arg1':pool.self_mntentref}) %}
[...]
---

OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color fstab 2>&1' with exit code '1': omv:
    Data failed to compile:
----------
    Rendering SLS 'base:omv.deploy.fstab.15unionfilesystems' failed: Jinja error: 'NoneType' object has no attribute 'get_dict'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 497, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3/dist-packages/jinja2/sandbox.py", line 465, in call
    return __context.call(__obj, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1235, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2268, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2283, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/var/cache/salt/minion/extmods/modules/omv_conf.py", line 44, in get
    return objs.get_dict()
AttributeError: 'NoneType' object has no attribute 'get_dict'

; line 1

---
{% set config = salt['omv_conf.get']('conf.service.unionfilesystems') %}    <======================

{% for pool in config.filesystem %}
{% set poolmount = salt['omv_conf.get_by_filter'](
  'conf.system.filesystem.mountpoint',
  {'operator':'stringEquals', 'arg0':'uuid', 'arg1':pool.self_mntentref}) %}
[...]
--- in /usr/share/php/openmediavault/system/process.inc:220
Stack trace:
#0 /usr/share/php/openmediavault/engine/module/serviceabstract.inc(62): OMV\System\Process->execute()
#1 /usr/share/openmediavault/engined/rpc/config.inc(174): OMV\Engine\Module\ServiceAbstract->deploy()
#2 [internal function]: Engined\Rpc\Config->applyChanges(Array, Array)
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#4 /usr/share/php/openmediavault/rpc/serviceabstract.inc(149): OMV\Rpc\ServiceAbstract->callMethod('applyChanges', Array, Array)
#5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(619): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}('/tmp/bgstatusJ1...', '/tmp/bgoutputgz...')
#6 /usr/share/php/openmediavault/rpc/serviceabstract.inc(159): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))
#7 /usr/share/openmediavault/engined/rpc/config.inc(195): OMV\Rpc\ServiceAbstract->callMethodBg('applyChanges', Array, Array)
#8 [internal function]: Engined\Rpc\Config->applyChangesBg(Array, Array)
#9 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#10 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('applyChangesBg', Array, Array)
#11 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Config', 'applyChangesBg', Array, Array, 1)
#12 {main}

Alles anzeigen

If I run "omv-salt deploy run --no-color fstab" on the CLI manually I get this:

Code

omv:
    Data failed to compile:
----------
    Rendering SLS 'base:omv.deploy.fstab.15unionfilesystems' failed: Jinja error: 'NoneType' object has no attribute 'get_dict'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 497, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3/dist-packages/jinja2/sandbox.py", line 465, in call
    return __context.call(__obj, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1235, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2268, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2283, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/var/cache/salt/minion/extmods/modules/omv_conf.py", line 44, in get
    return objs.get_dict()
AttributeError: 'NoneType' object has no attribute 'get_dict'

; line 1

---
{% set config = salt['omv_conf.get']('conf.service.unionfilesystems') %}    <======================

{% for pool in config.filesystem %}
{% set poolmount = salt['omv_conf.get_by_filter'](
  'conf.system.filesystem.mountpoint',
  {'operator':'stringEquals', 'arg0':'uuid', 'arg1':pool.self_mntentref}) %}
[...]
---

Alles anzeigen

What is causing this? The server/services/mergerfs pool etc. etc. are all working perfectly despite this...

ryecoaaron · 18. April 2023

Zitat von rodhull

What is causing this?

Slow os disk? You really should be on OMV 6.x since 5.x is EOL.

rodhull · 18. April 2023

Interesting - I am already on 6 - I'm running the OS from an mSATA drive (plugged into an adapter in my HP Gen7 Microserver's internal USB port). Transfer speeds are perfectly acceptable.

This only happened since upgrading to 6 some months ago...is it possible something went awry during the migration causing this message?

Here is some interesting output:

Code

root@omv:~# uname -a
Linux omv 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux
root@omv:~# dpkg -l|grep openmediavault
ii  omvextras-unionbackend              5.0.2                          all          union filesystems backend plugin for openmediavault
ii  openmediavault                      6.3.8-1                        all          openmediavault - The open network attached storage solution
ii  openmediavault-backup               6.1                            all          backup plugin for OpenMediaVault.
ii  openmediavault-flashmemory          6.2                            all          folder2ram plugin for openmediavault
ii  openmediavault-keyring              1.0                            all          GnuPG archive keys of the OpenMediaVault archive
ii  openmediavault-mergerfs             6.3.5                          all          mergerfs plugin for openmediavault.
ii  openmediavault-omvextrasorg         6.1.1                          all          OMV-Extras.org Package Repositories for OpenMediaVault
ii  openmediavault-remotemount          6.1.1                          all          Remote mount plugin for OpenMediaVault.
ii  openmediavault-resetperms           6.0.2                          all          Reset Permissions
ii  openmediavault-sharerootfs          6.0.2-1                        all          openmediavault share root filesystem plugin
ii  openmediavault-snapraid             6.1                            all          snapraid plugin for OpenMediaVault.
ii  openmediavault-unionfilesystems     5.1.4                          all          Union filesystems plugin for OpenMediaVault.
root@omv:~#

Alles anzeigen

The "openmediavault-unionfilesystems" and ""omvextras-unionbackend" packages look like suspiciously the wrong versions.

Now I come to think of it, when I migrated from OMV5 to 6 there were some wobbles due to mergerfs plugins I believe and I had to look on the forums for some fixes - I cannot recall exactly what I did unfortunately - I think there was a script I ran - are those 5.x versioned packages now in the way/should be purged?

rodhull · 18. April 2023

OK - I was right - looks like those 5.x versioned packages were remnants of the OMV 5 install that the migration did not clear up.

I ran "apt remove omvextras-unionbackend" which succeeded w/out error (and also removed "openmediavault-unionfilesystems") too.

I then ran an "apt purge omvextras-unionbackend openmediavault-unionfilesystems" which again produced no error.

I manually re-ran "omv-salt deploy run --no-color fstab" on the CLI and it succeeded now.

I expect this will be put to bed now.

Thanks for the tip!

Jetzt mitmachen!