Docker not working after OMV upgrade

MarcS · 1. August 2023

I upgraded to latest OMV and then had to key in my customised docker folder inside the compose Plugin.

Usually a docker restart works but OMV does not like the docker folder.

I end up with this error and systemctl status docker.service does not give any more insight.

Any ideas?

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color compose 2>&1' with exit code '1': raspberrypi:
----------
          ID: configure_compose_scheduled_jobs
    Function: file.managed
        Name: /etc/cron.d/omv-compose-backup
      Result: True
     Comment: File /etc/cron.d/omv-compose-backup is in the correct state
     Started: 10:26:46.575558
    Duration: 430.027 ms
     Changes:
----------
          ID: docker_install_packages
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 10:26:50.425181
    Duration: 121.717 ms
     Changes:
----------
          ID: docker_compose_install_packages
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 10:26:50.547583
    Duration: 36.344 ms
     Changes:
----------
          ID: docker_purged_packages
    Function: pkg.purged
      Result: True
     Comment: None of the targeted packages are installed or partially installed
     Started: 10:26:50.584327
    Duration: 2.909 ms
     Changes:
----------
    .......
FULL ERROR OUTPUT IS ATTACHED

Alles anzeigen

MarcS · 1. August 2023

some additional info:

Code

root@raspberrypi:/srv/JGPool# systemctl stop docker
root@raspberrypi:/srv/JGPool# systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/docker.service.d
             └─waitAllMounts.conf
     Active: failed (Result: exit-code) since Tue 2023-08-01 10:26:59 BST; 20min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
    Process: 6002 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1>
   Main PID: 6002 (code=exited, status=1/FAILURE)
        CPU: 425ms

Alles anzeigen

MarcS · 1. August 2023

anyone any ideas? This is the tail of the OMV error message; full message attached.

Code

Total states run:     18
Total run time:    4.172 s
[ERROR   ] Command '/bin/systemd-run' failed with return code: 1
[ERROR   ] stderr: Running scope as unit: run-r279cad9ed27a422189c3e670996e7095.scope
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
[ERROR   ] retcode: 1
[ERROR   ] Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details. in /usr/share/php/openmediavault/system/process.inc:220
Stack trace:
#0 /usr/share/php/openmediavault/engine/module/serviceabstract.inc(62): OMV\System\Process->execute()
#1 /usr/share/openmediavault/engined/rpc/config.inc(174): OMV\Engine\Module\ServiceAbstract->deploy()
#2 [internal function]: Engined\Rpc\Config->applyChanges(Array, Array)
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#4 /usr/share/php/openmediavault/rpc/serviceabstract.inc(149): OMV\Rpc\ServiceAbstract->callMethod('applyChanges', Array, Array)
#5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(619): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}('/tmp/bgstatusEX...', '/tmp/bgoutput31...')
#6 /usr/share/php/openmediavault/rpc/serviceabstract.inc(159): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))
#7 /usr/share/openmediavault/engined/rpc/config.inc(195): OMV\Rpc\ServiceAbstract->callMethodBg('applyChanges', Array, Array)
#8 [internal function]: Engined\Rpc\Config->applyChangesBg(Array, Array)
#9 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#10 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('applyChangesBg', Array, Array)
#11 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Config', 'applyChangesBg', Array, Array, 1)
#12 {main}

Alles anzeigen

ryecoaaron · 1. August 2023

Did you change the dockerStorage path?

What is the output of: sudo journalctl -u docker

MarcS · 1. August 2023

yes I did. But I updated the compose settings. When confirming that change, then OMV throws that error.

MarcS · 1. August 2023

Last few lines from journalctl -u docker below.

Full output is attached.

Code

Aug 01 10:17:17 raspberrypi systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Aug 01 10:17:17 raspberrypi systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 01 10:17:17 raspberrypi systemd[1]: Failed to start Docker Application Container Engine.
Aug 01 10:17:20 raspberrypi systemd[1]: docker.service: Scheduled restart job, restart counter is at 2.
Aug 01 10:17:20 raspberrypi systemd[1]: Stopped Docker Application Container Engine.
Aug 01 10:17:20 raspberrypi systemd[1]: docker.service: Start request repeated too quickly.
Aug 01 10:17:20 raspberrypi systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 01 10:17:20 raspberrypi systemd[1]: Failed to start Docker Application Container Engine.

MarcS · 1. August 2023

I think the journalctl entries are from a time when the Docker path was still on default and not changed.

After I realised that the compose Plugin had reset the docker path during the upgrade, I tried to save the new path but OMV will not let me.

MarcS · 1. August 2023

The OMV error output is attached and output of systemctl status docker.service shows:

Code

 systemctl status docker.service
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/docker.service.d
             └─waitAllMounts.conf
     Active: failed (Result: exit-code) since Tue 2023-08-01 16:18:06 BST; 3min 51s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
    Process: 35098 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=>
   Main PID: 35098 (code=exited, status=1/FAILURE)
        CPU: 310ms

Alles anzeigen

ryecoaaron · 1. August 2023

What path did you set docker storage to? And what filesystem is that path?

MarcS · 1. August 2023

/srv/Pool2/Docker

Its a BTRFS fileystem

ryecoaaron · 1. August 2023

Did it work on btrfs previously? If you aren't using any named volumes, you could try using /srv/Pool2/Docker2 to see if it works. That would just mean docker would have to download images again.

MarcS · 1. August 2023

It worked fine for the last 6 months, all on btrfs.

I dont know why we had to create all this new compose stuff. Why change everything if its not broken.

Now, nothing on my RPI servers works. All containers down.

ryecoaaron · 1. August 2023

Zitat von MarcS

I dont know why we had to create all this new compose stuff. Why change everything if its not broken.

Guess what? The way docker is installed is exactly the same. It just happens in a different plugin. And docker-compose doesn't do anything with the docker service. Why change? Continuous improvement. Are you still using Windows XP??? People telling how dumb this change is really makes me want to help them... You ever think it was caused by the move from Debian buster to bullseye?

Zitat von MarcS

Now, nothing on my RPI servers works. All containers down.

If the docker service doesn't start, then containers won't work.

This will be the last suggestion I have because the guy that made the horrible change probably can't help you.

Make sure the docker repo is enabled in omv-extras
run: sudo omv-aptclean repos
Run: sudo apt-get dist-upgrade
Reboot
run: sudo omv-salt deploy run compose

MarcS · 1. August 2023

Zitat von ryecoaaron

Continuous improvement

Thats great but then it would be good to do some testing prior to releasing "improvements". We used to have a stable OMV and now its not and unfortunately there is no way to roll back.

As a workaround for others who might encounter this issue:

Downgrade docker:

Code

sudo apt install docker-ce-cli=5:20.10.23~3-0~debian-bullseye 
sudo apt install docker-ce=5:20.10.23~3-0~debian-bullseye 
sudo service docker restart

After that, OMV allows to save the settings without error.

You might need to spend a few hours restoring your containers and not all of them will work 100% (e.g. Nginx and Dozzle throw errors). Of course its no solution but at least containers are up.

Zitat von ryecoaaron

Are you still using Windows XP???

Well, Compose reminds me a lot of Windows XP. Have you tried Portainer? Its much more user friendly, quicker and stable....

chente · 1. August 2023

No one is stopping you from continuing to use Portainer. Install it and go.

Soma · 1. August 2023

Zitat von MarcS

/srv/Pool2/Docker
Its a BTRFS fileystem

Is that path, your docker root?

If yes, you found your problem. Neither docker root nor container configs should be on a merged FS.

EVER.

As for it beeing BTRFS, I only have that FS (except OS) and never had an issue). Heck, even the DATA folder of NC is on a RAID1.

Zitat von MarcS

I dont know why we had to create all this new compose stuff. Why change everything if its not broken.

If I recall well, you always had issues with you RPi, but always assumed it was the software developer's error.

Have you considered that you might be doing something in a wrong way?

ryecoaaron · 1. August 2023

Zitat von MarcS

Thats great but then it would be good to do some testing prior to releasing "improvements". We used to have a stable OMV and now its not and unfortunately there is no way to roll back.

You do realize this plugin has been around for 15 months and hundreds of people are using it now including myself? Do you honestly think I haven't spent dozens of hours on it?

How did this affect the stability of OMV? docker is not OMV.

Zitat von MarcS

As a workaround for others who might encounter this issue:
Downgrade docker:

If that was the solution on your system, you would've hit this issue even if I never changed anything. Where was your backup to rollback to OMV 5? But thanks for blaming me.

Zitat von MarcS

You might need to spend a few hours restoring your containers and not all of them will work 100% (e.g. Nginx and Dozzle throw errors). Of course its no solution but at least containers are up.

How is this my fault? I need a technical reason. If something inside docker doesn't work, it isn't because of any plugin.

Zitat von MarcS

Well, Compose reminds me a lot of Windows XP. Have you tried Portainer? Its much more user friendly, quicker and stable....

Nice feedback. Did you see my 2.5 min video on how to keep using portainer? Portainer isn't limited by the OMV framework. Portainer isn't friendlier. It is just what you are used to. The plugin is stable. Upgrades from 5 to 6 might have issues but there is a major OS upgrade, major OMV upgrade, and plugin upgrade involved in that. I would love to see you make it better.

As for having tried portainer, how could I not since I am the one who made omv-extras start installing it in the first place. I am the one who defended portainer when I didn't port the docker-gui plugin to OMV 5.x. You damn well know I know plenty about portainer. Just because I don't use it myself doesn't mean I didn't test it hundreds of times maintaining the portainer install in omv-extras.

I get it you are mad but out of all the times I have helped you on this forum, for you to act like this to me and my work, it is very disappointing.

BlueCoffee · 1. August 2023

Zitat von ryecoaaron

You do realize this plugin has been around for 15 months and hundreds of people are using it now including myself? Do you honestly think I haven't spent dozens of hours on it?
How did this affect the stability of OMV? docker is not OMV.
If that was the solution on your system, you would've hit this issue even if I never changed anything. Where was your backup to rollback to OMV 5? But thanks for blaming me.
How is this my fault? I need a technical reason. If something inside docker doesn't work, it isn't because of any plugin.
Nice feedback. Did you see my 2.5 min video on how to keep using portainer? Portainer isn't limited by the OMV framework. Portainer isn't friendlier. It is just what you are used to. The plugin is stable. Upgrades from 5 to 6 might have issues but there is a major OS upgrade, major OMV upgrade, and plugin upgrade involved in that. I would love to see you make it better.

I get it you are mad but out of all the times I have helped you on this forum, for you to act like this to me and my work, it is very disappointing.

Alles anzeigen

Ryecoaaron you are a legend and I appreciate everything you have done for OMV and for the big help you have giving me on the forums much love!

MarcS · 1. August 2023

Zitat von chente

No one is stopping you from continuing to use Portainer. Install it and go.

I would but it does not work. After the upgrade docker doesnt work and Portainer is just a docker container. Since docker no work -> all containers no work -> portainer no work

MarcS · 1. August 2023

Zitat von Soma

If yes, you found your problem. Neither docker root nor container configs should be on a merged FS.
EVER.

That is not the issue. Docker worked perfectly fine on mergerfs (for last 4 years). I have also confirmed with the author of mergerfs that there is no issue with docker.

Docker not working after OMV upgrade

Jetzt mitmachen!

Ähnliche Themen

omv-extras 6.3 / openmediavault-compose 6.7

After upgrade from 5 to 6 strange docker behavior and old machines show up that i thought were gone

Docker doesn´t start after last upgrade to 5:23.0.0-1~debian.11~bullseye

Upgraded to 6.4.0 from 6.3.10 and having problems booting

docker not working since omv-upgrade