OMV Webgui not reachable

Zoki · Dec 29th 2021

/var/lib/docker/overlay2 stores redd/write layers of the containers own filesystem. Do you clear terminated docker containers or leave them hanging around? You can check with docker ps -a. This will give you a list of all containers defined in your system. Will only work if you restart the docker daemon.

I do not think the inodes are consumed in /var/lib. You only have ~ 60.000 files in there, but you used up 2.8M inodes.

If you have a backup, you can try to move /var/lib/docker onto the data disk (With the docker daemon stopped: sudo mv /var/lib/docker /srv/dev-disk-by...). In OMV change the root path of docker to /srv/dev-disk-.../docker then restart.

After that check free inodes df -i and continue looking, where they are used (many small files or emty directories).

macom · Dec 29th 2021

Quote from Zoki

I do not think the inodes are consumed in /var/lib. You only have ~ 60.000 files in there,

post #28 shows 2.7M files in overlay2

Code

/var/lib/docker/overlay2        2693212

haemburg2021 · Dec 29th 2021

Quote from Zoki

Do you clear terminated docker containers or leave them hanging around?

I think I don´t clear terminated containers.

I would like to restart docker but now I get an error:

Code

~# systemctl start docker.service
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

the status:

Code

docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2021-12-29 16:47:33 CET; 1min 5s ago
     Docs: https://docs.docker.com
  Process: 27095 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
 Main PID: 27095 (code=exited, status=1/FAILURE)

Dez 29 16:47:33 omnas.local systemd[1]: docker.service: Service RestartSec=2s expired, scheduling restart.
Dez 29 16:47:33 omnas.local systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Dez 29 16:47:33 omnas.local systemd[1]: Stopped Docker Application Container Engine.
Dez 29 16:47:33 omnas.local systemd[1]: docker.service: Start request repeated too quickly.
Dez 29 16:47:33 omnas.local systemd[1]: docker.service: Failed with result 'exit-code'.
Dez 29 16:47:33 omnas.local systemd[1]: Failed to start Docker Application Container Engine.

Display More

Zoki · Dec 29th 2021

And this: journalctl -xe?

Run this again: for i in /var/lib/docker/*; do echo -en "$i\t"; find $i |wc -l; done my command only counts files, this one directories too.

haemburg2021 · Dec 29th 2021

Code

~# journalctl -xe
Dez 29 18:35:58 omnas.local postfix/cleanup[29589]: warning: mail_queue_enter: create file incoming/316445.29589: No space left on device
Dez 29 18:35:58 omnas.local postfix/cleanup[15163]: warning: mail_queue_enter: create file incoming/316534.15163: No space left on device
Dez 29 18:35:58 omnas.local postfix/cleanup[29594]: warning: mail_queue_enter: create file incoming/316545.29594: No space left on device
Dez 29 18:35:58 omnas.local postfix/cleanup[30233]: warning: mail_queue_enter: create file incoming/316589.30233: No space left on device
Dez 29 18:35:58 omnas.local postfix/cleanup[28761]: warning: mail_queue_enter: create file incoming/316687.28761: No space left on device
Dez 29 18:35:58 omnas.local postfix/cleanup[30234]: warning: mail_queue_enter: create file incoming/316696.30234: No space left on device
Dez 29 18:35:58 omnas.local postfix/cleanup[15172]: warning: mail_queue_enter: create file incoming/316733.15172: No space left on device
Dez 29 18:35:58 omnas.local postfix/cleanup[28762]: warning: mail_queue_enter: create file incoming/316820.28762: No space left on device
Dez 29 18:35:58 omnas.local postfix/cleanup[14178]: warning: mail_queue_enter: create file incoming/316870.14178: No space left on device

Zoki · Dec 29th 2021

give yourself some relief and move /var/lib/docker to the other disk.

gderf · Dec 29th 2021

Quote from Zoki

give yourself some relief and move /var/lib/docker to the other disk.

Some of the most frequently posted advice in the forums, even right here in this thread.

Zoki · Dec 29th 2021

Quote from gderf

Some of the most frequently posted advice in the forums, even right here in this thread.

yes, by myself including the command.

gderf · Dec 29th 2021

I think the screen where this is set should mention the possible consequences of accepting the default. And yes, I do realize that it must default to something. Otherwise many new users who are unfamiliar with docker concepts couldn't even get started if the entry box defaulted to empty.

macom · Dec 29th 2021

OP is running out of inodes, not out of space. Moving /var/lib/docker is addressing a symptom but not the disease. Something is creating files like hell. Maybe a bot sending emails?

Any idea how to address this? I am leaning to deleting the OS drive and starting with a fresh install.

gderf · Dec 29th 2021

Quote from macom

Any idea how to address this? I am leaning to deleting the OS drive and starting with a fresh install.

Perhaps a bit extreme as a first step. I would delete every container and their data and see the degree of improvement, if any. If the inode count stays stable for a while, I would start adding containers one at a time after running the compose or run files past someone for another set of eyes.

macom · Dec 29th 2021

Yes, might be worth a try.

haemburg2021 · Dec 29th 2021

Is it possible that docker generated error-messages and so we see this in the log-files. and the notification function will send me e-mails to my e-mail account but it cannot send because it is not really configured?

I know that's a bit of a stretch from an unknowingly man like me.

Ok, I will move /var/lib/docker to the other disk with your steps.

haemburg2021 · Dec 29th 2021

Quote from Zoki

In OMV change the root path of docker to /srv/dev-disk-.../docker

Who and how can I change this?

gderf · Dec 29th 2021

Quote from haemburg2021

Who and how can I change this?

In System | omv-extras | Docker

Docker storage

haemburg2021 · Dec 29th 2021

ah ok after the move I have access to the web gui... thanks...

update:

I have access to the gui.

If I can´t change it in OMV:

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; omv-salt deploy run --no-color omvextras 2>&1' with exit code '1': debian: ---------- ID: configure_apt_sources_list_omvextras Function: file.managed Name: /etc/apt/sources.list.d/omvextras.list Result: True Comment: File /etc/apt/sources.list.d/omvextras.list is in the correct state Started: 23:39:17.186576 Duration: 32.048 ms Changes: ---------- ID: configure_apt_pref_omvextras Function: file.managed Name: /etc/apt/preferences.d/omvextras.pref Result: True Comment: File /etc/apt/preferences.d/omvextras.pref is in the correct state Started: 23:39:17.218732 Duration: 6.374 ms Changes: ---------- ID: refresh_database_apt Function: module.run Name: pkg.refresh_db Result: False Comment: No function provided. Started: 23:39:17.225628 Duration: 0.431 ms Changes: Summary for debian ------------ Succeeded: 2 Failed: 1 ------------ Total states run: 3 Total run time: 38.853 ms

Code

rror #0:
OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; omv-salt deploy run --no-color omvextras 2>&1' with exit code '1': debian:
----------
          ID: configure_apt_sources_list_omvextras
    Function: file.managed
        Name: /etc/apt/sources.list.d/omvextras.list
      Result: True
     Comment: File /etc/apt/sources.list.d/omvextras.list is in the correct state
     Started: 23:39:17.186576
    Duration: 32.048 ms
     Changes:
----------
          ID: configure_apt_pref_omvextras
    Function: file.managed
        Name: /etc/apt/preferences.d/omvextras.pref
      Result: True
     Comment: File /etc/apt/preferences.d/omvextras.pref is in the correct state
     Started: 23:39:17.218732
    Duration: 6.374 ms
     Changes:
----------
          ID: refresh_database_apt
    Function: module.run
        Name: pkg.refresh_db
      Result: False
     Comment: No function provided.
     Started: 23:39:17.225628
    Duration: 0.431 ms
     Changes:

Summary for debian
------------
Succeeded: 2
Failed:    1
------------
Total states run:     3
Total run time:  38.853 ms in /usr/share/php/openmediavault/system/process.inc:195
Stack trace:
#0 /usr/share/php/openmediavault/engine/module/serviceabstract.inc(62): OMV\System\Process->execute()
#1 /usr/share/openmediavault/engined/rpc/config.inc(167): OMV\Engine\Module\ServiceAbstract->deploy()
#2 [internal function]: Engined\Rpc\Config->applyChanges(Array, Array)
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#4 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('applyChanges', Array, Array)
#5 /usr/share/openmediavault/engined/rpc/omvextras.inc(180): OMV\Rpc\Rpc::call('Config', 'applyChanges', Array, Array)
#6 [internal function]: OMVRpcServiceOmvExtras->setDocker(Array, Array)
#7 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#8 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('setDocker', Array, Array)
#9 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('OmvExtras', 'setDocker', Array, Array, 1)
#10 {main}

Display More

Zoki · Dec 29th 2021

Didn't you read what I wrote? Move /var/lib/docker to another drive and then we will look at the container running wild.

Stop docker

Move the directory

Make sure you have enough free inodes on the drive

Change docker root in omv

Restart docker or reboot

list docker containers docker ps -a

After that we can continue.

haemburg2021 · Dec 29th 2021

i do the first steps

- stop docker

- move directory

- in post #61 next step In OMV change the root path of docker - doesn´t work

- restart

that inodes:

Code

~# df -i
Dateisystem                Inodes IBenutzt     IFrei IUse% Eingehängt auf
udev                      2002741      470   2002271    1% /dev
tmpfs                     2008968      807   2008161    1% /run
/dev/sde1                 2883584   188421   2695163    7% /
tmpfs                     2008968        2   2008966    1% /dev/shm
tmpfs                     2008968        3   2008965    1% /run/lock
tmpfs                     2008968       18   2008950    1% /sys/fs/cgroup
tmpfs                     2008968       12   2008956    1% /tmp
/dev/mapper/Stube-Media 169472000  2702818 166769182    2% /srv/dev-disk-by-label-med

Zoki · Dec 29th 2021

Quote from haemburg2021

- in post #61 next step In OMV change the root path of docker - doesn´t work

What does it mean: doesn't work? Error message?

Quote from haemburg2021

Code

~# df -i
Dateisystem                Inodes IBenutzt     IFrei IUse% Eingehängt auf
udev                      2002741      470   2002271    1% /dev
tmpfs                     2008968      807   2008161    1% /run
/dev/sde1                 2883584   188421   2695163    7% /
tmpfs                     2008968        2   2008966    1% /dev/shm
tmpfs                     2008968        3   2008965    1% /run/lock
tmpfs                     2008968       18   2008950    1% /sys/fs/cgroup
tmpfs                     2008968       12   2008956    1% /tmp
/dev/mapper/Stube-Media 169472000  2702818 166769182    2% /srv/dev-disk-by-label-med

You should be able to create files on your os disk.

show me cat /etc/docker/daemon.json

haemburg2021 · Dec 29th 2021

Quote from Zoki

What does it mean: doesn't work? Error message?

No, not error message - I had no access to the omv gui

so I restart and now I have access to omv gui after that I tried to change (see post #76

Code

:~# cat /etc/docker/daemon.json
{
  "data-root": "/var/lib/docker"
}

OMV Webgui not reachable

Participate now!

Similar Threads

WebGui not availible

How to redirect subdomain to specific port

No webGUI after Router change

NIC-Bond isn't shown anymore

SMB Speed collapse

Tags