Hi 
Yesterday i was away from home and was trying to reach my OMV nas with my reverse proxy (nginx and lets'encrypt). It was not working.
So i logged in via ssh and found that docker was not running, i have restarted it and everything started working fine. I tried to look at logs but found nothing into the single containers logs.
Where should i look to find what happened? So why docker stopped running?
Thanks in advance!
I would start in syslog
I would start in syslog
Thanks! But in Diagnostics -> System Logs i can see the syslog for today only. It's 13 pages but only for today.
Log files are "rotated". You might find previous entries in /var/log.
Log files are "rotated". You might find previous entries in /var/log.
Ok, found them. It's strange, i have a file 'syslog' with the log from today. A file 'syslog.1' that containts logs for 19 April and then jump to 25 June, so yesterday, why this?
In the log from yesterday i'm not sure what to look for. Any idea? Maybe i can post it.
Search for "error", "warning", "monit", "docker"
Search for "error", "warning", "monit", "docker"
Uhm ok i tried to look after these words but i can't find anything useful, but probably it's me i do not understand it.
Found this:
Line 1386: Apr 19 18:17:47 DK systemd[1]: Stopping Docker Application Container Engine...
Line 2869: Jun 25 07:28:31 DK systemd[1]: Stopping Docker Application Container Engine...
Line 2877: Jun 25 07:28:31 DK systemd[1]: Stopping Docker Socket for the API.
Line 4442: Jun 25 22:19:24 DK systemd[1]: Stopping Docker Application Container Engine...
but i do not think it's it, because it happens also on Apr 19 and yesterday evening at 22:19, and since my docker restart (don't remember what time it was but it was in the afternoon), it never stopped working again.
Maybe:
Line 2871: Jun 25 07:28:31 DK dockerd[1403]: time="2020-06-25T07:28:31.151706760+02:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Line 3000: Jun 25 07:28:48 DK dockerd[14524]: time="2020-06-25T07:28:48.651408272+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 64ddbf54dfcf4f0679d7c20128904633f7d611361d7fa63f1effcceb00574de2 ef0d6fcf191150c66a09998e2a4da50466f94b0002e9857e9543a26e05342fb1], retrying...."
Line 3004: Jun 25 07:28:49 DK dockerd[14524]: time="2020-06-25T07:28:49.885625616+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 64ddbf54dfcf4f0679d7c20128904633f7d611361d7fa63f1effcceb00574de2 88c94461239bbbe2206d6b46f2c3c1710329c69a4d8704e79e3ce7c4310a8010], retrying...."
Line 3008: Jun 25 07:28:51 DK dockerd[14524]: time="2020-06-25T07:28:51.822154855+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 64ddbf54dfcf4f0679d7c20128904633f7d611361d7fa63f1effcceb00574de2 9188227f16f8a91bc74f07db8a028426faa28dd0efeb157faaf03a683937a092], retrying...."
Line 3012: Jun 25 07:28:52 DK dockerd[14524]: time="2020-06-25T07:28:52.986053751+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6abf8eb069b13d39190da3b9f9451e226f26c68ac27adfbde353b3fbf5b5269 652d8dbfd059de67402c9b8b23edbf89f8c3f5a817f561da6104cd9bb042b5c7], retrying...."
Line 3017: Jun 25 07:28:54 DK dockerd[14524]: time="2020-06-25T07:28:54.601563008+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 64ddbf54dfcf4f0679d7c20128904633f7d611361d7fa63f1effcceb00574de2 b84ade6703f5e46150ca425359e6ab68bbffba3d7e67c09d4e9644a9e5fecae0], retrying...."
Line 3021: Jun 25 07:28:55 DK dockerd[14524]: time="2020-06-25T07:28:55.825649128+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 64ddbf54dfcf4f0679d7c20128904633f7d611361d7fa63f1effcceb00574de2 2a2982160333a7299654e9f9a5b5779909335f1096a35f2cbcb10daaad58e878], retrying...."
Line 3025: Jun 25 07:28:56 DK dockerd[14524]: time="2020-06-25T07:28:56.969579117+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint bf3aeabde4ed8e881762f90fafa51dfd087419ea980a340ee48a26e7f82a6c37 4a4251c8278182ac5280bbf751b67c527376dbcdb2a8b7392eeff161c26320dc], retrying...."
Line 3029: Jun 25 07:28:58 DK dockerd[14524]: time="2020-06-25T07:28:58.544921433+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 64ddbf54dfcf4f0679d7c20128904633f7d611361d7fa63f1effcceb00574de2 50905476d091428091e88635f4c0984cb672d8ed990ac6cc35e2c98086db3fe0], retrying...."
and also:
Line 4650: Jun 25 22:19:39 DK dockerd[14524]: time="2020-06-25T22:19:39.266060135+02:00" level=error msg="Force shutdown daemon"
Line 4652: Jun 25 22:19:39 DK dockerd[14524]: time="2020-06-25T22:19:39.266498302+02:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=moby
Line 5175: Jun 25 22:20:39 DK dockerd[17286]: time="2020-06-25T22:20:39.054572504+02:00" level=error msg="getEndpointFromStore for eid 1cd3cf3a1b1cb1a09de83185c2e7d41f00e570a86c9b8f6cf97f827f060634a3 failed while trying to build sandbox for cleanup: could not find endpoint 1cd3cf3a1b1cb1a09de83185c2e7d41f00e570a86c9b8f6cf97f827f060634a3: []"
hopefully somebody else will chime in
Crossing my fingers, anyway i have updated my previous post with others infos i found in the log.
Just an observation from the errors are you running moby
Just an observation from the errors are you running moby
Nope.
Apart from the main question, i do not understand why the syslog end at 19 Apr and starts at 25 Jun:
Apr 19 18:17:47 DK systemd[1]: Stopping The PHP 7.3 FastCGI Process Manager...
Jun 25 01:32:28 DK systemd-modules-load[288]: Module 'r8152' is builtin
There is also this:
Line 3: Apr 19 00:00:03 DK systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Line 4: Apr 19 00:00:03 DK systemd[1]: logrotate.service: Failed with result 'exit-code'.
Line 1359: Apr 19 18:17:47 DK systemd[1]: logrotate.timer: Succeeded.
Line 5710: Jun 26 00:00:01 DK logrotate[28020]: error: /etc/logrotate.conf:18 duplicate log entry for /var/log/wtmp
Line 5710: Jun 26 00:00:01 DK logrotate[28020]: error: /etc/logrotate.conf:18 duplicate log entry for /var/log/wtmp
Line 5711: Jun 26 00:00:01 DK logrotate[28020]: error: /etc/logrotate.conf:25 duplicate log entry for /var/log/btmp
Line 5711: Jun 26 00:00:01 DK logrotate[28020]: error: /etc/logrotate.conf:25 duplicate log entry for /var/log/btmp
Line 5712: Jun 26 00:00:01 DK logrotate[28020]: error: stat of /var/log/netatalk.log failed: No such file or directory
Don’t have an account yet? Register yourself now and be a part of our community!
