Docker stopped working, would like to understand why

  • Hi :)


    Yesterday i was away from home and was trying to reach my OMV nas with my reverse proxy (nginx and lets'encrypt). It was not working.


    So i logged in via ssh and found that docker was not running, i have restarted it and everything started working fine. I tried to look at logs but found nothing into the single containers logs.


    Where should i look to find what happened? So why docker stopped running?


    Thanks in advance!

  • Search for "error", "warning", "monit", "docker"

    Uhm ok i tried to look after these words but i can't find anything useful, but probably it's me i do not understand it.


    Found this:


    Line 1386: Apr 19 18:17:47 DK systemd[1]: Stopping Docker Application Container Engine...

    Line 2869: Jun 25 07:28:31 DK systemd[1]: Stopping Docker Application Container Engine...

    Line 2877: Jun 25 07:28:31 DK systemd[1]: Stopping Docker Socket for the API.

    Line 4442: Jun 25 22:19:24 DK systemd[1]: Stopping Docker Application Container Engine...



    but i do not think it's it, because it happens also on Apr 19 and yesterday evening at 22:19, and since my docker restart (don't remember what time it was but it was in the afternoon), it never stopped working again.



    Maybe:


    Line 2871: Jun 25 07:28:31 DK dockerd[1403]: time="2020-06-25T07:28:31.151706760+02:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
    Line 3000: Jun 25 07:28:48 DK dockerd[14524]: time="2020-06-25T07:28:48.651408272+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 64ddbf54dfcf4f0679d7c20128904633f7d611361d7fa63f1effcceb00574de2 ef0d6fcf191150c66a09998e2a4da50466f94b0002e9857e9543a26e05342fb1], retrying...."
    Line 3004: Jun 25 07:28:49 DK dockerd[14524]: time="2020-06-25T07:28:49.885625616+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 64ddbf54dfcf4f0679d7c20128904633f7d611361d7fa63f1effcceb00574de2 88c94461239bbbe2206d6b46f2c3c1710329c69a4d8704e79e3ce7c4310a8010], retrying...."
    Line 3008: Jun 25 07:28:51 DK dockerd[14524]: time="2020-06-25T07:28:51.822154855+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 64ddbf54dfcf4f0679d7c20128904633f7d611361d7fa63f1effcceb00574de2 9188227f16f8a91bc74f07db8a028426faa28dd0efeb157faaf03a683937a092], retrying...."
    Line 3012: Jun 25 07:28:52 DK dockerd[14524]: time="2020-06-25T07:28:52.986053751+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6abf8eb069b13d39190da3b9f9451e226f26c68ac27adfbde353b3fbf5b5269 652d8dbfd059de67402c9b8b23edbf89f8c3f5a817f561da6104cd9bb042b5c7], retrying...."
    Line 3017: Jun 25 07:28:54 DK dockerd[14524]: time="2020-06-25T07:28:54.601563008+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 64ddbf54dfcf4f0679d7c20128904633f7d611361d7fa63f1effcceb00574de2 b84ade6703f5e46150ca425359e6ab68bbffba3d7e67c09d4e9644a9e5fecae0], retrying...."
    Line 3021: Jun 25 07:28:55 DK dockerd[14524]: time="2020-06-25T07:28:55.825649128+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 64ddbf54dfcf4f0679d7c20128904633f7d611361d7fa63f1effcceb00574de2 2a2982160333a7299654e9f9a5b5779909335f1096a35f2cbcb10daaad58e878], retrying...."
    Line 3025: Jun 25 07:28:56 DK dockerd[14524]: time="2020-06-25T07:28:56.969579117+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint bf3aeabde4ed8e881762f90fafa51dfd087419ea980a340ee48a26e7f82a6c37 4a4251c8278182ac5280bbf751b67c527376dbcdb2a8b7392eeff161c26320dc], retrying...."
    Line 3029: Jun 25 07:28:58 DK dockerd[14524]: time="2020-06-25T07:28:58.544921433+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 64ddbf54dfcf4f0679d7c20128904633f7d611361d7fa63f1effcceb00574de2 50905476d091428091e88635f4c0984cb672d8ed990ac6cc35e2c98086db3fe0], retrying...."


    and also:


    Line 4650: Jun 25 22:19:39 DK dockerd[14524]: time="2020-06-25T22:19:39.266060135+02:00" level=error msg="Force shutdown daemon"
    Line 4652: Jun 25 22:19:39 DK dockerd[14524]: time="2020-06-25T22:19:39.266498302+02:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=moby
    Line 5175: Jun 25 22:20:39 DK dockerd[17286]: time="2020-06-25T22:20:39.054572504+02:00" level=error msg="getEndpointFromStore for eid 1cd3cf3a1b1cb1a09de83185c2e7d41f00e570a86c9b8f6cf97f827f060634a3 failed while trying to build sandbox for cleanup: could not find endpoint 1cd3cf3a1b1cb1a09de83185c2e7d41f00e570a86c9b8f6cf97f827f060634a3: []"

  • Just an observation from the errors are you running moby

    Nope.


    Apart from the main question, i do not understand why the syslog end at 19 Apr and starts at 25 Jun:


    Apr 19 18:17:47 DK systemd[1]: Stopping The PHP 7.3 FastCGI Process Manager...
    Jun 25 01:32:28 DK systemd-modules-load[288]: Module 'r8152' is builtin


    There is also this:


    Line 3: Apr 19 00:00:03 DK systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
    Line 4: Apr 19 00:00:03 DK systemd[1]: logrotate.service: Failed with result 'exit-code'.
    Line 1359: Apr 19 18:17:47 DK systemd[1]: logrotate.timer: Succeeded.
    Line 5710: Jun 26 00:00:01 DK logrotate[28020]: error: /etc/logrotate.conf:18 duplicate log entry for /var/log/wtmp
    Line 5710: Jun 26 00:00:01 DK logrotate[28020]: error: /etc/logrotate.conf:18 duplicate log entry for /var/log/wtmp
    Line 5711: Jun 26 00:00:01 DK logrotate[28020]: error: /etc/logrotate.conf:25 duplicate log entry for /var/log/btmp
    Line 5711: Jun 26 00:00:01 DK logrotate[28020]: error: /etc/logrotate.conf:25 duplicate log entry for /var/log/btmp
    Line 5712: Jun 26 00:00:01 DK logrotate[28020]: error: stat of /var/log/netatalk.log failed: No such file or directory

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!