Four Docker containers suddenly stopped working

Today I realized that I have 4 out of 15ish docker containers that are stopped (grafana, swag, photoprism, Transmissiong / OpenVPN). When I start any of them via the UI they fail immediately with "Failure. Request failed with status code 500". If I ssh into the server to launch the docker compose files I get the same error for all of them:

stat /srv/dev-disk-by-label-Docker/Docker/overlay2/1af8e33685ebdd5949f338938e70a05602a35e6995e81a2abfaef335621449fe: no such file or directory

I found an article on this and tried a bunch of the suggestions that seemed safe. Other suggestions (replacing volumes) seemed a bit risky for someone that is not an expert.

https://github.com/docker/for-mac/issues/1396

I did try to umount the overlay2 - it said that it was not mounted. No idea if that helps.

Quote from raulfg3

revise your filesystem, perhaps, some bind volumes point to a deleted or full path.

Thank you for this. Unfortunately I really don't understand what you mean. Can you either provide some more information on what you mean or point to a resource that I can look at?

So the weirdness extends past the four containers I mentioned earlier. Other containers say that they are running but when I try to look at their logs they give me the same kind of errors ("Unable to retrieve container information"). Again, I had not made any changes or added any new containers for a while. The one event that did happen earlier this week is that I lost power and my server shut down unexpectedly.

I did check to see if I was out of space somewhere but nothing jumps out at me.

I am seeing these errors in the logs.

EDIT

Would running out of space on OS drive cause this? This just occurred to me - I'll try to take a look in a bit.

Quote from raulfg3

Revisa tu sistema de archivos, quizas tu disco de boot este lleno o quizas la ruta (path)= que usas para guardar la configuración de los dockers este llena o no tengas permisos para escribir en esa ruta.

ayudaría que pusieras en este post los archivos que has utilizado para crear los dockers.

Please traslate this onto your language: https://translate.google.es/?hl=es&tab=mT

Display More

Thank you so much for the response. I think the OS drive may have been out of space - I deleted some images and it is now 78% full. I rebooted and the errors are the same. I don't see what posting 15 docker compose files would tell you so I didn't post them.

I am not sure how to check the second part of your recommendation. Docker might be out of space?

Quote from raulfg3

Second part is to check that docker have permission to write on desired config Path.

$PUID=1001 $PGID=1000 $ConfigPath=/srv/dev-disk-by-uuid-6f38f973-7aec-452b-815d-9101878af2e1/Data/dockers

If my config PAHT do not exist or is full, I have the problem you describe, the same if the folder /srv/dev-disk-by-uuid-6f38f973-7aec-452b-815d-9101878af2e1/Data/dockers do not have enought privileges for user UID=1001 or for group=1000.

please post the path your dockers use to store config, and post ls -la of that path to see permission & Owner.

Display More

Ok, this is where I start to go off the tracks a bit. I am pretty weak in this area. I've never used variables in this context so I have no idea where you defined them. Also, are you looking for the permissions for the folders where I keep my docker data (AppData) or the containers themselves?

So lets take Calibre as that is a simple app.

Quote from raulfg3

please post ls -la of /srv/dev-disk-by-label-HOMEMEDIA/AppData/Calibre and /srv/dev-disk-by-label-HOMEMEDIA/AppData/Calibre/config

post ID 1000 to see what user is ID=1000

finally post a screen capture of Storage -> Shared Folders -> Privileges

to see if have read/write permission to /srv/dev-disk-by-label-HOMEMEDIA/AppData

in my case user 1001 is dockeruser =

Display More

Thank you so much, again.

Here are my config folders. I am sorry, I did not follow your comment. Is there a command that I can enter to see which user is 1000?

Quote from raulfg3

ID 1000

your folder are own by root so user 1000 can write on it, please solve it

I am sorry, I don't fully understand what the issue is or how to fix it. I have a working knowledge of how linux file permissions but not enough to solve this problem. I've consumed tons of videos and other resources to learn docker but almost all of them are very light in these areas.

Quote from Wolf2000

In the Console ID 1000 then root should come out

my

root@bigwolf2000:~# id 1000

uid=1000(wolf2000) gid=100(users) groups=100(users),24(cdrom),25(floppy),29(audio),30(dip),44(video),46(plugdev),109(netdev),115(bluetooth),122(scanner),1000(wolf2000),1001(webdav-users)

root@bigwolf2000:~#

Display More

Again, I am sorry. I have no idea what you mean by this: "In the Console ID 1000 then root should come out"

Quote from Wolf2000

Enter the command id root please

Perfect, thank you. What now?

uid=0(root) gid=0(root) groups=0(root)

bump?

Quote from raulfg3

please post ID 1000 and read about linux permission and Owners, is not difficult and you can solve yourself your actual problems.

I did post ID 1000 and as I stated earlier I have done a ton of research and learning in this area. As I noted earlier most guides don't go too deep into explanation and I haven't found a good guide that brings this all together. More often than not I end up throwing things at the wall based on guides that are close to (but not exactly) like what I am trying to do. I am just looking for a final little bump to get me over the hill.