Can't start DOCKER

Sergio · 30. Januar 2016

From the last OMV update I can't start back docker. It happened to be offline while I was replacing a disk.
Now every time I enable and save, throws some error saying "communication failure" or "504 Gateway Time-out"

Does anyone knows whats the problem about?
Thanks.

subzero79 · 30. Januar 2016

Can you start the daemon manually in console? With init?

After it starts just check the daemon is running with docker ps -a or any docker command

Sergio · 30. Januar 2016

Docker was running. Inoticed activity in my disks and PS showed running images.

subzero79 · 30. Januar 2016

Then try again with a browser in private mode or clear the browser cache.

Sergio · 3. Februar 2016

That didn't work. Could be something related to some container not sending the expected response in time to the plugin? Does it make sense?

nicjo814 · 3. Februar 2016

Try to change the api listenig port in the settings page of the plugin, then save. Wait a bit to see if it works, if not change again back to the port it was using before. Do this a number of times (up to 10 times) until it works (if it does). I'm seeing this problem from time to time but have a really hard tie to reproduce the problem at will...

Sergio · 3. Februar 2016

Hi Nicjo. That also didn't work. "Gateway Error"
And every time I do a save, processor usage raises around 20% and also memory.
Process list shows one omv-engined fixed at @20% cpu for each attempt.

Code

top - 22:06:25 up 14:50,  0 users,  load average: 7,66, 5,45, 3,44
Tasks: 279 total,   4 running, 273 sleeping,   0 stopped,   2 zombie
%Cpu(s):  0,9 us,  1,5 sy,  0,0 ni, 96,2 id,  1,4 wa,  0,0 hi,  0,0 si,  0,0 st
KiB Mem:  32973880 total, 13867988 used, 19105892 free,    11844 buffers
KiB Swap:        0 total,        0 used,        0 free,  1076448 cached


  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
26044 root      20   0  213m  21m 6188 S  32,3  0,1   1:02.27 omv-engined
11239 root      20   0  213m  21m 6188 S  25,9  0,1   2:24.41 omv-engined
20410 root      20   0  213m  21m 6188 S  25,9  0,1   0:39.75 omv-engined
21174 root      20   0  213m  21m 6188 S  25,9  0,1   1:59.51 omv-engined
13150 root      20   0  213m  21m 6188 S  19,4  0,1  16:37.42 omv-engined
14891 root      20   0  213m  21m 6188 S  12,9  0,1   0:16.71 omv-engined
   18 root      20   0     0    0    0 S   6,5  0,0   0:14.78 ksoftirqd/2
   28 root      20   0     0    0    0 S   6,5  0,0   0:23.81 ksoftirqd/4
   37 root      rt   0     0    0    0 S   6,5  0,0   0:04.84 migration/6
    1 root      20   0 10656 1632 1496 S   0,0  0,0   0:01.92 init
    2 root      20   0     0    0    0 S   0,0  0,0   0:00.26 kthreadd
    3 root      20   0     0    0    0 S   0,0  0,0   0:20.27 ksoftirqd/0
    5 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kworker/0:0H
    7 root      20   0     0    0    0 S   0,0  0,0   0:29.98 rcu_sched
    8 root      20   0     0    0    0 S   0,0  0,0   0:00.00 rcu_bh
    9 root      rt   0     0    0    0 S   0,0  0,0   0:00.60 migration/0
   10 root      rt   0     0    0    0 S   0,0  0,0   0:00.15 watchdog/0
   11 root      rt   0     0    0    0 S   0,0  0,0   0:00.14 watchdog/1
   12 root      rt   0     0    0    0 S   0,0  0,0   0:00.60 migration/1
   13 root      20   0     0    0    0 S   0,0  0,0   0:14.36 ksoftirqd/1
   15 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kworker/1:0H
   16 root      rt   0     0    0    0 S   0,0  0,0   0:00.15 watchdog/2
   17 root      rt   0     0    0    0 S   0,0  0,0   0:00.56 migration/2
   20 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kworker/2:0H
   21 root      rt   0     0    0    0 S   0,0  0,0   0:00.15 watchdog/3
   22 root      rt   0     0    0    0 S   0,0  0,0   0:00.64 migration/3
   23 root      20   0     0    0    0 S   0,0  0,0   0:14.30 ksoftirqd/3
   25 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kworker/3:0H
   26 root      rt   0     0    0    0 S   0,0  0,0   0:00.14 watchdog/4
   27 root      rt   0     0    0    0 S   0,0  0,0   0:05.02 migration/4
   30 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kworker/4:0H
   31 root      rt   0     0    0    0 S   0,0  0,0   0:00.14 watchdog/5
   32 root      rt   0     0    0    0 S   0,0  0,0   0:04.76 migration/5
   33 root      20   0     0    0    0 S   0,0  0,0   0:19.77 ksoftirqd/5
   35 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kworker/5:0H
   36 root      rt   0     0    0    0 S   0,0  0,0   0:00.14 watchdog/6
   38 root      20   0     0    0    0 S   0,0  0,0   0:19.59 ksoftirqd/6
   40 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kworker/6:0H
   41 root      rt   0     0    0    0 S   0,0  0,0   0:00.14 watchdog/7
   42 root      rt   0     0    0    0 S   0,0  0,0   0:04.84 migration/7
   43 root      20   0     0    0    0 S   0,0  0,0   0:19.66 ksoftirqd/7
   45 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kworker/7:0H
   46 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 khelper
   47 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kdevtmpfs
   48 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 netns
   49 root      20   0     0    0    0 S   0,0  0,0   0:00.02 khungtaskd
   50 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 writeback
   51 root      25   5     0    0    0 S   0,0  0,0   0:00.00 ksmd
   52 root      39  19     0    0    0 S   0,0  0,0   0:00.00 khugepaged
   53 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 crypto
   54 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kintegrityd
   55 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 bioset
   56 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kblockd
   58 root      20   0     0    0    0 S   0,0  0,0   0:07.54 kswapd0
   59 root      20   0     0    0    0 S   0,0  0,0   0:00.00 fsnotify_mark
   65 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kthrotld
   66 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ipv6_addrconf
   67 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 deferwq
  170 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 acpi_thermal_pm
  242 root      20   0     0    0    0 S   0,0  0,0   0:00.89 khubd
  256 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ata_sff
  259 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_0
  260 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 scsi_tmf_0
  261 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 fw_event0
  270 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_1
  271 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 scsi_tmf_1
  272 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_2
  273 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 scsi_tmf_2
  274 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_3
  275 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 scsi_tmf_3
  276 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_4
  277 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 scsi_tmf_4
  278 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_5
  279 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 scsi_tmf_5
  280 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_6
  281 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 scsi_tmf_6
  291 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_7
  292 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 scsi_tmf_7
  293 root      20   0     0    0    0 S   0,0  0,0   0:01.00 usb-storage
  294 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 poll_0_status
  365 root       0 -20     0    0    0 S   0,0  0,0   0:00.15 kworker/2:1H
  366 root       0 -20     0    0    0 S   0,0  0,0   0:00.08 kworker/7:1H
  367 root      20   0     0    0    0 S   0,0  0,0   0:00.93 kworker/5:2
  370 root       0 -20     0    0    0 S   0,0  0,0   0:00.17 kworker/1:1H
  375 root       0 -20     0    0    0 S   0,0  0,0   0:00.13 kworker/5:1H
  376 root       0 -20     0    0    0 S   0,0  0,0   0:03.54 kworker/0:1H
  377 root       0 -20     0    0    0 S   0,0  0,0   0:00.37 kworker/4:1H
  381 root       0 -20     0    0    0 S   0,0  0,0   0:00.11 kworker/6:1H
  387 root       0 -20     0    0    0 S   0,0  0,0   0:00.13 kworker/3:1H
  426 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 md
  442 root      20   0     0    0    0 S   0,0  0,0   0:00.05 jbd2/sdk1-8
  443 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ext4-rsv-conver
  470 openmedi  20   0  209m  14m 8240 S   0,0  0,0   0:00.30 php5-fpm
  483 openmedi  20   0  208m  13m 8240 S   0,0  0,0   0:00.25 php5-fpm
  580 root      20   0 21452 2540 1872 S   0,0  0,0   0:00.10 udevd
  630 root      20   0     0    0    0 S   0,0  0,0   0:00.05 kworker/2:0
...
 4105 root      20   0 18980 2060 1804 S   0,0  0,0   0:00.04 rpcbind
 4125 statd     20   0 23352 2424 1964 S   0,0  0,0   0:00.00 rpc.statd
 4130 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 rpciod
 4132 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 nfsiod
 4139 root      20   0 25304  220    4 S   0,0  0,0   0:00.00 rpc.idmapd
 4498 root      20   0 32872 2388 2092 S   0,0  0,0   0:00.00 zed
 4505 shellina  20   0 29752  412    0 S   0,0  0,0   0:00.00 shellinaboxd
 4506 shellina  20   0 29752  416    0 S   0,0  0,0   0:00.00 shellinaboxd
 4507 root      20   0  4104 1220 1124 S   0,0  0,0   0:00.00 startpar
 4508 root      20   0 69812 4320 3680 S   0,0  0,0   0:00.53 nmbd
 4528 root      20   0  323m 3512 2432 S   0,0  0,0   0:10.70 rrdcached
 4534 root      20   0 98,1m 8460 7520 S   0,0  0,0   0:00.24 smbd
 4553 root      20   0 98,1m 3956 3016 S   0,0  0,0   0:00.00 smbd
 4556 root      20   0  207m  11m 7084 S   0,0  0,0   0:01.14 php5-fpm
 4557 www-data  20   0  207m 7200 2896 S   0,0  0,0   0:00.00 php5-fpm
 4558 www-data  20   0  207m 7200 2896 S   0,0  0,0   0:00.00 php5-fpm
 4559 openmedi  20   0  207m 7200 2896 S   0,0  0,0   0:00.00 php5-fpm
 4560 openmedi  20   0  207m 7200 2896 S   0,0  0,0   0:00.00 php5-fpm
 4572 root      20   0  184m 4000 2412 S   0,0  0,0   0:15.06 rsyslogd
 4616 root      20   0 12996 1552 1408 S   0,0  0,0   0:00.00 mdadm
 4714 root      20   0 37888 3896 3340 S   0,0  0,0   0:12.94 master
 4733 postfix   20   0 40120 3928 3328 S   0,0  0,0   0:08.70 qmgr
 4739 ntp       20   0 34872 4312 3712 S   0,0  0,0   0:01.73 ntpd
 4770 root      20   0 21564 2140 1888 S   0,0  0,0   0:00.20 cron
 4775 nobody    20   0 18816 1884 1692 S   0,0  0,0   0:00.00 noip2
 4776 root      20   0  4124 1568 1416 S   0,0  0,0   0:00.00 acpid
 4782 root      20   0  283m 4492 3456 S   0,0  0,0   0:22.35 collectd
 4794 root      20   0 49944 3448 2840 S   0,0  0,0   0:00.00 sshd
 4799 messageb  20   0 29916 2252 1908 S   0,0  0,0   0:00.00 dbus-daemon
 4814 avahi     20   0 34168 2824 2524 S   0,0  0,0   0:00.05 avahi-daemon
 4815 avahi     20   0 34044  252    0 S   0,0  0,0   0:00.00 avahi-daemon
 4820 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 iprt
 4830 root      20   0 78424 2960 1848 S   0,0  0,0   0:00.00 nginx
 4831 www-data  20   0 79100 4480 3064 S   0,0  0,0   0:00.58 nginx
 4832 www-data  20   0 79096 4480 3064 S   0,0  0,0   0:02.20 nginx
 4833 www-data  20   0 78736 3488 2144 S   0,0  0,0   0:02.22 nginx
 4834 www-data  20   0 78736 3488 2144 S   0,0  0,0   0:02.22 nginx
 4851 root      20   0  203m  50m 8148 S   0,0  0,2   0:17.82 python
 4862 root      20   0  135m  28m 7472 S   0,0  0,1   0:00.67 python
 4863 openvpn_  20   0  163m  36m 8548 S   0,0  0,1   0:09.23 python
 4881 root      20   0  112m  18m 7340 S   0,0  0,1   0:03.28 python
 4882 root      20   0  112m  18m 7200 S   0,0  0,1   0:03.26 python
 4885 openvpn_  20   0 36552 5832 5188 S   0,0  0,0   0:00.26 openvpn-openssl
 4914 root      20   0 19708 3516 2040 S   0,0  0,0   0:00.16 smartd
 4959 postfix   20   0 39956 3816 3276 S   0,0  0,0   0:07.55 scache
 5122 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kauditd
 5179 root      20   0 10004 3376 2396 S   0,0  0,0   0:04.62 autoshutdown.sh
 5224 root      20   0 40784 3076 2388 S   0,0  0,0   0:11.01 monit
...
 7359 root      20   0 21448 2200 1532 S   0,0  0,0   0:00.00 udevd
 8217 root      20   0     0    0    0 S   0,0  0,0   0:00.20 kworker/0:2
 9062 postfix   20   0 39988 3868 3328 S   0,0  0,0   0:00.00 bounce
11476 root      20   0     0    0    0 S   0,0  0,0   0:01.30 kworker/1:0
11785 root      20   0  100m 8068 7084 S   0,0  0,0   0:00.00 smbd
12796 root      20   0  124m 5812 4716 S   0,0  0,0   0:00.02 console-kit-dae
12863 root      20   0  120m 4928 4384 S   0,0  0,0   0:00.01 polkitd
13106 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kworker/0:0
13123 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kworker/2:1
13269 root      20   0 21448 1180  512 S   0,0  0,0   0:00.00 udevd
13328 root      20   0  4104  484  408 S   0,0  0,0   0:00.00 sleep
13481 postfix   20   0 61616 5960 4816 S   0,0  0,0   0:00.05 smtp
14634 root       1 -19     0    0    0 S   0,0  0,0   0:00.00 z_wr_iss
14635 root       1 -19     0    0    0 S   0,0  0,0   0:00.00 z_wr_iss
14834 root       1 -19     0    0    0 S   0,0  0,0   0:00.00 z_wr_iss
15125 root       1 -19     0    0    0 S   0,0  0,0   0:00.00 z_wr_iss
15344 root      20   0     0    0    0 S   0,0  0,0   0:00.02 kworker/7:3
16036 root      20   0     0    0    0 S   0,0  0,0   0:10.53 kworker/2:4
17760 postfix   20   0 39956 3912 3376 S   0,0  0,0   0:00.00 pickup
17971 openmedi  20   0  209m  14m 8240 S   0,0  0,0   0:00.26 php5-fpm
18134 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kworker/5:0
18211 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kworker/7:1
18629 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kworker/6:2
18719 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kworker/3:2
18785 root      20   0     0    0    0 S   0,0  0,0   0:00.93 kworker/u16:0
18830 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kworker/6:1
19735 openmedi  20   0  209m  14m 8240 S   0,0  0,0   0:00.03 php5-fpm
19736 openmedi  20   0  207m  13m 8240 S   0,0  0,0   0:00.02 php5-fpm
20734 root      20   0  213m  20m 5344 S   0,0  0,1   0:00.00 omv-engined
20792 root      20   0  4188  516  448 S   0,0  0,0   0:00.00 sh
20794 root      20   0 24456 2688 2260 R   0,0  0,0   0:00.00 top
21904 root      20   0  4188  520  448 R   0,0  0,0   0:00.00 sh
21905 root      20   0  4188  516  444 S   0,0  0,0   0:00.00 sh
21908 root      20   0     0    0    0 Z   0,0  0,0   0:00.00 mount
21910 root      20   0     0    0    0 Z   0,0  0,0   0:00.00 grep
21911 root      20   0  9004 1836 1716 R   0,0  0,0   0:00.00 grep
21912 root      20   0  4188   92    0 R   0,0  0,0   0:00.00 sh
21913 root      20   0  7120 1644 1544 S   0,0  0,0   0:00.00 wc
25743 root      20   0  213m  20m 5184 S   0,0  0,1   0:02.91 omv-engined
26748 postfix   20   0 39988 3864 3324 S   0,0  0,0   0:00.00 bounce
28675 postfix   20   0 39968 3920 3388 S   0,0  0,0   0:00.00 trivial-rewrite
29730 postfix   20   0 40596 5472 4440 S   0,0  0,0   0:00.04 cleanup
29753 postfix   20   0 40596 5512 4472 S   0,0  0,0   0:00.03 cleanup
29757 postfix   20   0 61616 5872 4728 S   0,0  0,0   0:00.05 smtp
29778 postfix   20   0 61616 5964 4816 S   0,0  0,0   0:00.05 smtp

Alles anzeigen

nicjo814 · 4. Februar 2016

Can you provide output of "ps aux | grep docker" as well as the content of your "/etc/default/docker" file?

Sergio · 5. Februar 2016

Thank you.

Code

ps aux | grep docker
root      4860  0.0  0.0 199752 31436 ?        Sl   08:05   0:02 /usr/bin/docker daemon -p /var/run/docker.pid -H unix:///var/run/docker.sock -H tcp://127.0.0.1:42005 -g /var/lib/docker/openmediavault
root      5243  0.0  0.0  75884 23304 ?        Sl   08:05   0:00 docker-proxy -proto tcp -host-ip 192.168.11.241 -host-port 8080 -container-ip 172.17.0.3 -container-port 80
root      5266  0.0  0.0  75884 23272 ?        Sl   08:05   0:00 docker-proxy -proto tcp -host-ip 192.168.11.241 -host-port 8282 -container-ip 172.17.0.2 -container-port 80
root     20826  0.0  0.0   9008  1980 pts/0    S+   11:58   0:00 grep docker

Code

cat /etc/default/docker
# Docker Upstart and SysVinit configuration file


# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"


# Use DOCKER_OPTS to modify the daemon startup options.
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"


# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"


# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"


### Do not change these lines. They are added and updated by the OMV Docker GUI plugin.
OMVDOCKER_API="-H unix:///var/run/docker.sock -H tcp://127.0.0.1:42005"
OMVDOCKER_IMAGE_PATH="-g /var/lib/docker/openmediavault"
DOCKER_OPTS="$DOCKER_OPTS $OMVDOCKER_API $OMVDOCKER_IMAGE_PATH"
### Do not add any configuration below this line. It will be removed when the plugin is removed

Alles anzeigen

*After "removed" there's not any cr. Command line started just after it. If that has any importance.

nicjo814 · 5. Februar 2016

Both look fine. Is it possible for you to restart your machine? I'm out of ideas right now...

Sergio · 5. Februar 2016

Lately it's powered off every night. So that's an after boot output.

Please, can you tell me if it's possible to force not starting all the containers on boot?
I want to test if I can start the plugin without the containers starting or already started.

I have the emby container. Could that be that the container starting delays too much? Or it being already started?

Sergio · 5. Februar 2016

That's it. It started after stopping absolutely all containers.

nicjo814 · 5. Februar 2016

Ok, glad that you found a solution even though it was an odd one I hope most of these issues will go away in the 3.x version of the plugin that I'm working on.

Sergio · 5. Februar 2016

So it seems it was conflicting trying to start a machine that was already up (emby)

I hope so. Thanks for all your help.

Sergio · 5. Februar 2016

FYI

Zitat

Setting up docker-engine (1.10.0-0~wheezy) ...
Installing new version of config file /etc/init/docker.conf ...

Configuration file `/etc/default/docker'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
==> Using current old file as you requested.
Installing new version of config file /etc/bash_completion.d/docker ...
Starting Docker: docker.
Creating index of upgradeable packages ...
Creating index of openmediavault plugins ...
Hecho ...

Alles anzeigen

And docker-plugin again offline

nicjo814 · 5. Februar 2016

The 1.10 update seems to bring some quite big changes and there is a migration required (should be automatic), but you need to restart the docker service. I haven't had the chance to upgrade to 1.10 myself yet so I can't really help out any more at the moment.

Sergio · 5. Februar 2016

Trying to show docker plugin page.

Zitat

Error: "Operation timed out after 30001 milliseconds with 0 bytes received" - Code: 28

I will reboot the server, kill all the containers and try to enable it again.
Is something needed to get ready after this update like changing configuration files or something?
Now you have a cobaya.

nicjo814 · 5. Februar 2016

I just updated my dev system with docker-engine 1.10 and had the same issue. Waiting for a bit and then manually stopping/starting the docker service fixed the issue for me. I know that they made some changes to how the image ID's are represented, which potentially could cause issues in the plugin. I'll test some more and see if I have to make any fixes in the code.

Sergio · 9. Februar 2016

Thanks! For any help you need, drop me a line.
You deserve an oscar for the development of this plugin.

Can't start DOCKER

Jetzt mitmachen!

Tags