Another OMV5 Docker sob story... Send Help!!

  • After a routine system update, docker completely stopped functioning. Portainer will not connect/launch and manual container(s) run/start from the CLI is unsuccessful. Pulling my hair out to track down the issue without luck. Monitoring the syslog and output files and I still can't figure out the what or why of the issue. I've read though almost every post that might have relevance to my issue and I sill cannot find a viable solution to get docker working again. My hardware configuration is as vanilla as it gets i.e. a newer Supermicro system with more than enough compute power to run a little Plex server and NAS storage array.


    I would greatly appreciate a second set of eyes from a more experienced deb user versed in the dark-arts of OMV5/Docker.


    **side note** Why do I spend more admin time on OMV5 docker related issues than the accumulated total of the 5 other servers sitting in the same rack? Really? just to run Plex...common.. I experience less hassle from my Nvidia compute nodes running atypical configurations of TensorFlow.

    • Offizieller Beitrag

    Why do I spend more admin time on OMV5 docker related issues than the accumulated total of the 5 other servers sitting in the same rack? Really?

    I have no idea what you updated but seems like you updated docker itself. This has nothing to do with OMV 5 and it is hard for OMV to fix the docker package that comes from docker. What is the output of: systemctl status docker

    omv 7.0.4-2 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.10 | compose 7.1.2 | k8s 7.0-6 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I have no idea what you updated but seems like you updated docker itself. This has nothing to do with OMV 5 and it is hard for OMV to fix the docker package that comes from docker. What is the output of: systemctl status docker

    nah, update preformed via cli omv-update, that's it!


    root@openmediavault:~# systemctl status docker

    Warning: The unit file, source configuration file or drop-ins of docker.service changed on disk. Run 'systemctl daemon-reload' to reload units.

    docker.service - Docker Application Container Engine

    Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)

    Active: active (running) since Thu 2020-07-02 02:19:12 PDT; 1h 54min ago

    Docs: https://docs.docker.com

    Main PID: 26604 (dockerd)

    Tasks: 83

    Memory: 117.2M

    CGroup: /system.slice/docker.service

    ├─ 851 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5800 -container-ip 172.19.0.2 -container-port 5800

    ├─26604 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

    ├─26911 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 19999 -container-ip 172.21.0.2 -container-port 19999

    ├─27051 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6941 -container-ip 172.18.0.2 -container-port 80

    ├─27094 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6080 -container-ip 172.17.0.3 -container-port 6080

    ├─27107 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.18.0.2 -container-port 443

    └─27147 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5900 -container-ip 172.17.0.3 -container-port 5900


    Jul 02 04:11:52 openmediavault dockerd[26604]: time="2020-07-02T04:11:52.037728241-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:12:14 openmediavault dockerd[26604]: time="2020-07-02T04:12:14.951277902-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:12:32 openmediavault dockerd[26604]: time="2020-07-02T04:12:32.230595912-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:12:32 openmediavault dockerd[26604]: time="2020-07-02T04:12:32.968763874-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:12:34 openmediavault dockerd[26604]: time="2020-07-02T04:12:34.642014388-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:13:13 openmediavault dockerd[26604]: time="2020-07-02T04:13:13.973645591-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:13:16 openmediavault dockerd[26604]: time="2020-07-02T04:13:16.269402656-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:13:33 openmediavault dockerd[26604]: time="2020-07-02T04:13:33.560094753-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:13:35 openmediavault dockerd[26604]: time="2020-07-02T04:13:35.925429778-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:13:54 openmediavault dockerd[26604]: time="2020-07-02T04:13:54.901181271-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    root@openmediavault:~#

  • after reloading the daemon via systemctl


    root@openmediavault:~# systemctl status docker

    docker.service - Docker Application Container Engine

    Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)

    Active: active (running) since Thu 2020-07-02 02:19:12 PDT; 2h 5min ago

    Docs: https://docs.docker.com

    Main PID: 26604 (dockerd)

    Tasks: 81

    Memory: 119.6M

    CGroup: /system.slice/docker.service

    ├─21060 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5800 -container-ip 172.19.0.2 -container-port 5800

    ├─26604 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

    ├─26911 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 19999 -container-ip 172.21.0.2 -container-port 19999

    ├─27051 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6941 -container-ip 172.18.0.2 -container-port 80

    ├─27094 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6080 -container-ip 172.17.0.3 -container-port 6080

    ├─27107 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.18.0.2 -container-port 443

    └─27147 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5900 -container-ip 172.17.0.3 -container-port 5900


    Jul 02 04:21:46 openmediavault dockerd[26604]: time="2020-07-02T04:21:46.867233598-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:22:26 openmediavault dockerd[26604]: time="2020-07-02T04:22:26.571775437-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:22:26 openmediavault dockerd[26604]: time="2020-07-02T04:22:26.888823523-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:22:44 openmediavault dockerd[26604]: time="2020-07-02T04:22:44.987013672-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:22:48 openmediavault dockerd[26604]: time="2020-07-02T04:22:48.213382868-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:23:07 openmediavault dockerd[26604]: time="2020-07-02T04:23:07.853849934-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:23:27 openmediavault dockerd[26604]: time="2020-07-02T04:23:27.790205589-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:23:46 openmediavault dockerd[26604]: time="2020-07-02T04:23:46.181473979-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:23:48 openmediavault dockerd[26604]: time="2020-07-02T04:23:48.848349711-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    Jul 02 04:23:49 openmediavault dockerd[26604]: time="2020-07-02T04:23:49.974144411-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

    root@openmediavault:~#

    • Offizieller Beitrag

    nah, update preformed via cli omv-update, that's it!

    Uh, omv-update just does an apt-get update and apt-get dist-upgrade. So, it could have very easily updated the docker package. Did you look to see what packages it updated? This is why you should update from the Updates tab so you can see what you are updating. You should also have a backup of your OS disk.


    after reloading the daemon via systemctl

    Reloaded what daemon? Why reload? Did you try restarting docker? systemctl restart docker Are your filesystems mounted?

    omv 7.0.4-2 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.10 | compose 7.1.2 | k8s 7.0-6 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Uh, omv-update just does an apt-get update and apt-get dist-upgrade. So, it could have very easily updated the docker package. Did you look to see what packages it updated? This is why you should update from the Updates tab so you can see what you are updating. You should also have a backup of your OS disk.


    Reloaded what daemon? Why reload? Did you try restarting docker? systemctl restart docker Are your filesystems mounted?

    I've restarted docker from systemctl, all the obvious has been tried. docker has been uninstalled, purged, apt cleaned, reinstalled (through omv-extras). you name it I've tried it. and yes, all file systems are mounted.. **This is just my opinion but OMV should be updated/managed via CLI. having to navigate to a local Web GUI to update is a poor methodology for managing offsite resources.

    • Offizieller Beitrag

    all the obvious has been tried

    I deal with many levels of linux skills. I never know what people have tried. What I consider obvious is vastly different than most users.


    If docker is running and storage is not a problem then it is very strange that the containers are not working. I have never seen this and I run docker on lots of systems. Are the permissions wrong somehow? Anything in docker logs for any of the containers?


    This is just my opinion but OMV should be updated/managed via CLI. having to navigate to a local Web GUI to update is a poor methodology for managing offsite resources.

    The whole concept of OMV is do things from the web interface. If you are an advanced user, update from the CLI but it isn't any different since the web interface and omv-update do basically the same thing.

    omv 7.0.4-2 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.10 | compose 7.1.2 | k8s 7.0-6 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I deal with many levels of linux skills. I never know what people have tried. What I consider obvious is vastly different than most users.


    If docker is running and storage is not a problem then it is very strange that the containers are not working. I have never seen this and I run docker on lots of systems. Are the permissions wrong somehow? Anything in docker logs for any of the containers?


    The whole concept of OMV is do things from the web interface. If you are an advanced user, update from the CLI but it isn't any different since the web interface and omv-update do basically the same thing.

    We are getting a little derailed from finding a resolution to my issue; however, it is my understanding that the "whole concept" of OMV is actually to provide network attached storage OS shipped with the tools to support it's core competency. The web UI should simply be a utility to make management easier for the less technically adept end user. If updating from the CLI via omv-update does not preform exactly the same function as a clicking button in the WEB UX it should be considered a design flaw with the platform that must be reexamined by the OMV developers.

    getting back to my issue of a mystery malfunctioning Docker... Is there anyone that can help provide a solution? what logs have I over looked, system configs that need modification etc..

  • when I cat the json logs for individual containers I'm getting errors like this:

    {"log":"nginx: [emerg] socket() 0.0.0.0:80 failed (13: Permission denied)\n","stream":"stderr","time":"2020-07-01T20:28:29.057688449Z"}

    {"log":"[01-Jul-2020 13:28:29] ERROR: failed to init signals: socketpair(): Permission denied (13)\n","stream":"stderr","time":"2020-07-01T20:28:29.955074661Z"}

    {"log":"[01-Jul-2020 13:28:29] ERROR: FPM initialization failed\n","stream":"stderr","time":"2020-07-01T20:28:29.955130901Z"}

    {"log":"nginx: [emerg] socket() 0.0.0.0:80 failed (13: Permission denied)\n","stream":"stderr","time":"2020-07-01T20:28:30.059998924Z"}


    or this:

    {"log":"2020/07/02 14:26:12 Templates already registered inside the database. Skipping template import.\n","stream":"stderr","time":"2020-07-02T14:26:12.918967991Z"}

    {"log":"2020/07/02 14:26:12 listen tcp 0.0.0.0:8000: socket: permission denied\n","stream":"stderr","time":"2020-07-02T14:26:12.924010484Z"}

    {"log":"2020/07/02 14:26:12 server: Reverse tunnelling enabled\n","stream":"stdout","time":"2020-07-02T14:26:12.924151737Z"}

    {"log":"2020/07/02 14:26:12 server: Fingerprint 73:1c:db:bf:62:5c:af:67:0e:0b:24:ef:81:22:9c:f4\n","stream":"stdout","time":"2020-07-02T14:26:12.924169962Z"}

    {"log":"2020/07/02 14:26:12 server: Listening on 0.0.0.0:8000...\n","stream":"stdout","time":"2020-07-02T14:26:12.924180853Z"}

  • Update and temporary..ish fix: disable/run docker && containers as unconstrained via apparmor. The update of my system somehow changed the configuration(s) of docker-default. God only knows where docker-default config is located under OMV5 because it was defiantly not in the normal and well documented location of /etc/apparmor.d/docker


    The reason it took so long to find this error is because everything seemed to be running fine. All the sys logs were as normal, dmesg normal etc. I found a random socket error in the docker json log the lead me down a rabbit hole resulting in the temporary solution of "Run docker with apparmor as unconstrained". Sprinkle a little apparmor magic in the CLI and voilà. Thats some hack $h*t my guy!!


    Honestly, why is docker under OMV such a hack?

  • Honestly, why is docker under OMV such a hack?

    I've had zero problems with docker in both OMV 4 and OMV 5. I have 14 containers running.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • I've had zero problems with docker in both OMV 4 and OMV 5. I have 14 containers running.

    Good for you! Go read through all the threads of users experiencing similar issues without solution. My gripe is that docker under OMV is a slightly modified configuration that obscures trouble shooting.

  • Zero problems with Docker either on OMV4 and 5, omv-update regularly updates Docker on my system without issues. I only had issues at the very beginning with permissions with the docker installation folder, but it's another story.


    In your scenario I would have rolled back my system to the previous day backup (use the backup plugin) and try again.

    OMV BUILD - MY NAS KILLER - OMV 6.x + omvextrasorg (updated automatically every week)

    NAS Specs: Core i3-8300 - ASRock H370M-ITX/ac - 16GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker Apps), Several HDDs (Data) w/ SnapRAID - Fractal Design Node 304 - Be quiet! Pure Power 11 350W


    My all-in-one SnapRAID script!

  • Zero problems with Docker either on OMV4 and 5, omv-update regularly updates Docker on my system without issues. I only had issues at the very beginning with permissions with the docker installation folder, but it's another story.


    In your scenario I would have rolled back my system to the previous day backup (use the backup plugin) and try again.

    Why would you roll back an installation when you could just write one small command in a shell via SSH as I did to fix the issue? In my scenario I have mission critical data being served up to a GPU compute cluster running finite element models for extended periods of time. In these types of applications the machine should never go down, ever!! IMO backups should only ever be used in an unrecoverable catastrophic event such as hardware failure.

  • Why would you roll back an installation when you could just write one small command in a shell via SSH as I did to fix the issue? In my scenario I have mission critical data being served up to a GPU compute cluster running finite element models for extended periods of time. In these types of applications the machine should never go down, ever!! IMO backups should only ever be used in an unrecoverable catastrophic event such as hardware failure.

    Well, I use my system as a normal NAS, and if I really break something I prefer to rollback the previous config. I already done this and only takes few minutes.

    OMV BUILD - MY NAS KILLER - OMV 6.x + omvextrasorg (updated automatically every week)

    NAS Specs: Core i3-8300 - ASRock H370M-ITX/ac - 16GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker Apps), Several HDDs (Data) w/ SnapRAID - Fractal Design Node 304 - Be quiet! Pure Power 11 350W


    My all-in-one SnapRAID script!

    • Offizieller Beitrag

    why is docker under OMV such a hack?

    Really? The only thing OMV (and actually omv-extras) does is install docker and set the path in /etc/docker/daemon.json. It is doing nothing else. The install script is doing exactly what the docker instructions on the docker site tell you to do. Which part of that is hacky?

    My gripe is that docker under OMV is a slightly modified configuration that obscures trouble shooting.

    Um, no it isn't. The only thing it will set is the data-root so that you can put the docker files on something other than the OS drive (which may be small). If you want OMV to do *absolutely* nothing to configure docker, clear the path text box and install it.

    The reason it took so long to find this error is because everything seemed to be running fine. All the sys logs were as normal, dmesg normal etc. I found a random socket error in the docker json log the lead me down a rabbit hole resulting in the temporary solution of "Run docker with apparmor as unconstrained". Sprinkle a little apparmor magic in the CLI and voilà. Thats some hack $h*t my guy!!

    OMV and Debian do not enable apparmor by default. So, the fact that it was enforcing means you did something to do that. There is nothing OMV-related that enables it. In fact, the apparmor package shouldn't even be installed on an OMV system installed with the OMV ISO.

    omv 7.0.4-2 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.10 | compose 7.1.2 | k8s 7.0-6 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!


  • ]

    Really? The only thing OMV (and actually omv-extras) does is install docker and set the path in /etc/docker/daemon.json. It is doing nothing else. The install script is doing exactly what the docker instructions on the docker site tell you to do. Which part of that is hacky?

    Um, no it isn't. The only thing it will set is the data-root so that you can put the docker files on something other than the OS drive (which may be small). If you want OMV to do *absolutely* nothing to configure docker, clear the path text box and install it.

    OMV and Debian do not enable apparmor by default. So, the fact that it was enforcing means you did something to do that. There is nothing OMV-related that enables it. In fact, the apparmor package shouldn't even be installed on an OMV system installed with the OMV ISO.

    Oooohhh....wait..... I know what I did wrong... I decided to use OMV, then I treated it like an appliance and not a toy... OMV doesn't even support ZFS natively.. you have to install a plugin and downgrade your Kernel. What is not "hacky" about that?

    • Offizieller Beitrag

    OMV doesn't even support ZFS natively.. you have to install a plugin

    There are only so many things Volker can write. The ZFS plugin took a lot of man hours. And not everyone uses zfs. I don't. So, why do I want zfs installed??

    And once the plugin is installed, it is just as "native" as any other service.

    downgrade your Kernel

    Can you explain why? Because the Debian 5.5 backports can be used. Debian chose for legal reasons to not pre-compile the zfs module but it definitely compiles and works just fine on the debian kernel. It is generally recommend to install the Proxmox 5.4 kernel (basically the ubuntu lts kernel) to avoid the compile issues. Oh the horror that one version back is such a downgrade. In reality, the 5.4 kernel probably have the important features from 5.5 and possibly more since Canonical backports things to their LTS kernel.

    What is not "hacky" about that?

    This is comical. You criticize OMV because you have to install a plugin to get zfs but to get zfs, you have to use code that isn't allowed in the Linux kernel. Which one is hacky again??


    I decided to use OMV, then I treated it like an appliance and not a toy

    I have used OMV as a production NAS in multiple locations since close to the beginning. OMV isn't perfect but it has a couple orders of magnitude less people working on it than some competitors and commercial offerings. Why don't you step and help fix the problems?

    omv 7.0.4-2 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.10 | compose 7.1.2 | k8s 7.0-6 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!