Web UI issues

tannaroo · 18. November 2020

Sorry gents I posted this in another thread but here may be more relevant.

I tried to restart one of my docker container (logitech media server) through portainer and I got one of those red errors (top right corner). It was too quick but something about json error: merged.

So I tried to login to my OMV UI and when I put my username/password, it just says "Loading Please Wait" but then goes back to the login screen.

I have tried rebooting, clearing cache, running omv-firstaid (I can puTTy into the device) but still can not access OMV UI or Portainer UI.

It looks as thought my sdb1 is full but not sure why because my docker and containers are all on sepearate hard drives.Code

Code

root@omv:~# df -h

Filesystem      Size  Used Avail Use% Mounted on

udev            1.9G     0  1.9G   0% /dev

tmpfs           381M  7.1M  374M   2% /run

/dev/sdb1        15G   15G     0 100% /

tmpfs           1.9G     0  1.9G   0% /dev/shm

tmpfs           5.0M     0  5.0M   0% /run/lock

tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup

/dev/sdd1        29G   11G   18G  38% /srv/dev-disk-by-label-Monkey

/dev/sde1       1.4T  1.3T   86G  94% /srv/dev-disk-by-label-Elephant

tmpfs           1.9G  8.0K  1.9G   1% /tmp

folder2ram      1.9G   53M  1.9G   3% /var/log

folder2ram      1.9G     0  1.9G   0% /var/tmp

folder2ram      1.9G  936K  1.9G   1% /var/lib/openmediavault/rrd

folder2ram      1.9G   20K  1.9G   1% /var/spool

folder2ram      1.9G   16M  1.9G   1% /var/lib/rrdcached

folder2ram      1.9G  4.0K  1.9G   1% /var/lib/monit

folder2ram      1.9G  1.3M  1.9G   1% /var/cache/samba

/dev/sda1       4.6T  1.5T  3.1T  33% /srv/dev-disk-by-label-Hippo

Alles anzeigen

Display More

The output of a folder (overlay2) which has 10G, so not sure what is going?

Code

root@omv:~# du -d1 -h /var/lib/docker

4.0K    /var/lib/docker/trust

4.0K    /var/lib/docker/runtimes

20K     /var/lib/docker/plugins

4.0K    /var/lib/docker/tmp

52M     /var/lib/docker/containers

40M     /var/lib/docker/volumes

72K     /var/lib/docker/buildkit

10G     /var/lib/docker/overlay2

120K    /var/lib/docker/network

4.0K    /var/lib/docker/swarm

5.6M    /var/lib/docker/image

20K     /var/lib/docker/builder

10G     /var/lib/docker

Alles anzeigen

I followed another thread which suggested docker system prune but that did nothing as got this error message:

Code

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

tannaroo · 18. November 2020

I think what might have happened is one of my external drives which had the plex appdata may have disconnected somehow as I cannot see that drive with the lsblk command. This may have caused the plex appdat to use up the system drive???

If so, I am trying to figure if I can delete the overlay2 folder (refer to above post) and the command for that docker/if that will fix things?

tannaroo · 19. November 2020

Can anyone please help because my server which runs several dockers is down or do I need a complete rebuild?

ryecoaaron · 19. November 2020

Your root os is too small for the number of containers you are running and/or the data in your containers. You could try docker system prune to get rid of unused images, containers, and volumes. I might backup your system before you do that.

tannaroo · 19. November 2020

I tried docker system prune but got this error:

Code

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Also, do you see from my earlier post I have huge fies sitting in my overlay2 folder, so ,maybe I can delete?

I do not why my root drive is full - its been working fine for about 8 months. All of my dockers and appdata (including plex) are on separate HDD.

ryecoaaron · 19. November 2020

10G of your 15G is in the docker folder. Either it is old stuff before your moved your docker or you didn't change anything correctly. It is hard to tell what you can delete in the overlay2 folder without causing a problem. I would try freeing up space from somewhere else so docker can start and you can use prune.

Does apt-get clean free up any space?

tannaroo · 19. November 2020

No I tried that - it didn't do anything noticeable.

KM0201 · 19. November 2020

Zitat von tannaroo

Can anyone please help because my server which runs several dockers is down or do I need a complete rebuild?

A complete rebuild is likely unnecessary, but you may be rebuilding your containers. I guess it just depends on the amount of effort you want to put into this. If it were me (and it was at one time)... I'd stop docker, delete the overlay folder, and restart docker. The folder structure should rebuild when you restart docker. Then setup my containers and docker again, and make sure my container folders are on a data disk when setting up again.

tannaroo · 19. November 2020

Zitat von KM0201

A complete rebuild is likely unnecessary, but you may be rebuilding your containers. I guess it just depends on the amount of effort you want to put into this. If it were me (and it was at one time)... I'd stop docker, delete the overlay folder, and restart docker. Then setup my containers and docker again, and make sure my container folders are on a data disk when setting up again.

OK so the CLI commands are:

Code

docker stop

Code

rm -rfv /var/lib/docker/overlay2/*

Code

docker start

tannaroo · 19. November 2020

Not sure what is happening now but I deleted the overlay2 folder and managed to sign back in to the WebUI. But was having trouble getting into portainer, so I unsinstalled Docker and reinstalled Docker/Portainer. My OS drive is now back to low storage usage

Some of my containers are still there but stopped. When I try to start them I get this error:

Code

Failure
Cannot restart container 115a1ad003925db7e46c614df57990dcfcb1066b2585790ec1a395e7065b61b8: error creating overlay mount to /var/lib/docker/overlay2/7e1dd00f4e334752a40ba8bc8e1e6b9f528fed3b38096bf0b2d5c08540fec77d/merged: no such file or directory

Does anyone what is happening?

ryecoaaron · 19. November 2020

Zitat von tannaroo

Does anyone what is happening?

You deleted the folder it needed. And like I said, I would try other things to free up space other than deleting docker files. My next suggestion was delete compressed log files in /var/log.

KM0201 · 20. November 2020

Zitat von tannaroo
Not sure what is happening now but I deleted the overlay2 folder and managed to sign back in to the WebUI. But was having trouble getting into portainer, so I unsinstalled Docker and reinstalled Docker/Portainer. My OS drive is now back to low storage usage

Some of my containers are still there but stopped. When I try to start them I get this error:
Code
Failure
Cannot restart container 115a1ad003925db7e46c614df57990dcfcb1066b2585790ec1a395e7065b61b8: error creating overlay mount to /var/lib/docker/overlay2/7e1dd00f4e334752a40ba8bc8e1e6b9f528fed3b38096bf0b2d5c08540fec77d/merged: no such file or directory
Does anyone what is happening?

As I said, you'll end up redoing your containers if you do that. Your containers were apparently not on your storage drive as you thought they were.

tannaroo · 20. November 2020

Zitat von KM0201

As I said, you'll end up redoing your containers if you do that. Your containers were apparently not on your storage drive as you thought they were.

So is this a complete rebuild? Or can it be fixed?

I have deleted my containers and redone them (via Stacks) but I still get this error message

KM0201 · 20. November 2020

Zitat von tannaroo

So is this a complete rebuild? Or can it be fixed?

I'm not following you

Start docker, and it should rebuild your folder structure.

Then reinstall Portainer

Reinstall your containers. I was pretty clear you were going to have to rebuild them when I posted that earlier. Just make sure your containers are on your on your data drives, or a drive big enough to hold your containers.

tannaroo · 20. November 2020

Zitat von KM0201

I'm not following you

Start docker, and it should rebuild your folder structure.

Then reinstall Portainer

Reinstall your containers. I was pretty clear you were going to have to rebuild them when I posted that earlier. Just make sure your containers are on your on your data drives, or a drive big enough to hold your containers.

Alles anzeigen

But I have done all of that.

I have even uninstalled Docker and reinstalled it.

KM0201 · 20. November 2020

Are you on OMV 4 or OMV 5?

tannaroo · 20. November 2020

Sorry shoudl have stated from outset, yes OMV5

KM0201 · 20. November 2020

Zitat von tannaroo

Sorry shoudl have stated from outset, yes OMV5

What is the output of systemctl status docker

tannaroo · 20. November 2020

Zitat von KM0201

What is the output of systemctl status docker

Code

root@omv:~# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: e
   Active: active (running) since Fri 2020-11-20 09:23:21 GMT; 2min 42s ago
     Docs: https://docs.docker.com
 Main PID: 1761 (dockerd)
    Tasks: 30
   Memory: 54.3M
   CGroup: /system.slice/docker.service
           ├─1761 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/contain
           ├─1976 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9
           └─1989 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8

Nov 20 09:23:18 omv.local dockerd[1761]: time="2020-11-20T09:23:18.912692474Z" l
Nov 20 09:23:21 omv.local dockerd[1761]: time="2020-11-20T09:23:21.594093752Z" l
Nov 20 09:23:21 omv.local dockerd[1761]: time="2020-11-20T09:23:21.594194771Z" l
Nov 20 09:23:21 omv.local dockerd[1761]: time="2020-11-20T09:23:21.767432986Z" l
Nov 20 09:23:21 omv.local systemd[1]: Started Docker Application Container Engin
Nov 20 09:24:07 omv.local dockerd[1761]: time="2020-11-20T09:24:07.220856040Z" l
Nov 20 09:24:07 omv.local dockerd[1761]: time="2020-11-20T09:24:07.221166396Z" l
Nov 20 09:24:07 omv.local dockerd[1761]: time="2020-11-20T09:24:07.340172974Z" l
Nov 20 09:24:07 omv.local dockerd[1761]: time="2020-11-20T09:24:07.341265451Z" l
Nov 20 09:24:07 omv.local dockerd[1761]: time="2020-11-20T09:24:07.341344283Z" l
lines 1-22/22 (END)...skipping...
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-11-20 09:23:21 GMT; 2min 42s ago
     Docs: https://docs.docker.com
 Main PID: 1761 (dockerd)
    Tasks: 30
   Memory: 54.3M
   CGroup: /system.slice/docker.service
           ├─1761 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
           ├─1976 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9200 -container-ip 172.17.0.2 -container-port 9000
           └─1989 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8000 -container-ip 172.17.0.2 -container-port 8000

Nov 20 09:23:18 omv.local dockerd[1761]: time="2020-11-20T09:23:18.912692474Z" level=info msg="Loading containers: done."
Nov 20 09:23:21 omv.local dockerd[1761]: time="2020-11-20T09:23:21.594093752Z" level=info msg="Docker daemon" commit=4484c46d9d graphdriver(s)=overlay2 version=19.03.13
Nov 20 09:23:21 omv.local dockerd[1761]: time="2020-11-20T09:23:21.594194771Z" level=info msg="Daemon has completed initialization"
Nov 20 09:23:21 omv.local dockerd[1761]: time="2020-11-20T09:23:21.767432986Z" level=info msg="API listen on /var/run/docker.sock"
Nov 20 09:23:21 omv.local systemd[1]: Started Docker Application Container Engine.
Nov 20 09:24:07 omv.local dockerd[1761]: time="2020-11-20T09:24:07.220856040Z" level=error msg="error unmounting /var/lib/docker/overlay2/7e1dd00f4e334752a40ba8bc8e1e6b9f528fed3b38096bf0b2d5c08540fec77d/merged: invalid argument" storage-
Nov 20 09:24:07 omv.local dockerd[1761]: time="2020-11-20T09:24:07.221166396Z" level=error msg="error unmounting /var/lib/docker/overlay2/7e1dd00f4e334752a40ba8bc8e1e6b9f528fed3b38096bf0b2d5c08540fec77d/merged: invalid argument" storage-
Nov 20 09:24:07 omv.local dockerd[1761]: time="2020-11-20T09:24:07.340172974Z" level=warning msg="error locating sandbox id 2b1f16d0ac409e77bd38720301d524b3117dbd67c761fb518aab16126d95dd6d: sandbox 2b1f16d0ac409e77bd38720301d524b3117dbd6
Nov 20 09:24:07 omv.local dockerd[1761]: time="2020-11-20T09:24:07.341265451Z" level=error msg="115a1ad003925db7e46c614df57990dcfcb1066b2585790ec1a395e7065b61b8 cleanup: failed to delete container from containerd: no such container"
Nov 20 09:24:07 omv.local dockerd[1761]: time="2020-11-20T09:24:07.341344283Z" level=error msg="Handler for POST /containers/115a1ad003925db7e46c614df57990dcfcb1066b2585790ec1a395e7065b61b8/restart returned error: Cannot restart containe
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
lines 1-22/22 (END)

Alles anzeigen

KM0201 · 20. November 2020

OK, so docker is running.... what is your question?

Web UI issues

Jetzt mitmachen!

Ähnliche Themen

On a dual HDD NAS can I use the 2nd HDD as the backup destination of the 1st HDD on Odroid HC4?

rrdcached error messages after reboot

Random Restarts & Docker Containers Not Auto Loading

Possible permission errors with transmission VPN stack?

Portainer & Yacht problem with template