Cant login to the admin panel anymore

j2wtf · Apr 6th 2025

Suddenly I can't login to the admin panel anymore. My server seems to work okay, although I have been getting error messages about disk space (which is why I tried to login in the first place).

I can login with ssh and run the omv-firstaid to change the password, but I still cant login. If I purposely do a wrong password I do get an error message, but if I do the correct password it simply returns to the login screen..

The disk space error messages are a bit weird, since both disks attached had plenty of space the last time I Iooked. (both below 40%). The SSD (i'm running on a Raspberry that boots from SSD was never anywhere near 10%.. I do have SSD backups on harddisks as well, but I feel that would be kind of a hassle and would easily take me a whole evening to do.

Anybody have any suggestions on how to solve this?

gderf · Apr 7th 2025

The most common reason for the exact problem you are seeing is a full system disk.

Login via ssh and run the this command to see your disk usage.

Code

df -h

If / is at 100% then look here for the solution steps:

Thread

[How-To] Fix full OS filesystem - GUI login loop

Symptoms

When you try to login to the GUI of OMV you are not being logged in, but the login page shows up again

Check, if a full root filesystem is the problem
ssh into your server (check the new user guide on how to ssh into your server) and run following command
df df will show you the usage of each filesystem
the output of df will look like this:
(Code, 19 lines)

The filesystem dev/sda1 which is mounted on "/" is the OS filesystem. The use of that filesystem is 100% --> the OS filesystem is full.

…

macom

Jul 23rd 2020

j2wtf · Apr 7th 2025

Super helpful, thanks!

Will need to dig into this later this week, but I indeed suspect rsync to a drive that for some reason didnt mount properly, filling up the SD card (running this on a raspberry with two SSDs attached). Quite annoying though, this has been working for a few months now, and I'm not backing up to remote mounts or anything so was expecting this to more stable...

j2wtf · Apr 7th 2025

Okay, so I took all the steps above and the problem seems to be docker logfile that had grown to 113Gb over the course of one or two weeks..

There was indeed something weird in the contents of that logfile, which seems to have stopped now. But I'll see if I can figure out how to automatically truncate those.. Though I had the impression that stuck process might have filled up those 113Gb faster than the time between two cleanup cycles...

Did truncate, and everything was back to normal immediately. Thanks for that link gderf !

Code

root@314:/var/lib/docker# cd containers/
root@314:/var/lib/docker/containers# du -xhd1
113G    ./5a1edf864e13f5a91723db20a551f3cecc1ae
113G    .
root@314:/var/lib/docker/containers# cd ./5a1edf864e13f5a91723db20a551f3cecc1ae/
root@314:/var/lib/docker/containers/5a1edf864e13f5a91723db20a551f3cecc1ae0# du -xhd1
4.0K    ./mounts
4.0K    ./checkpoints
113G    .
root@314:/var/lib/docker/containers/5a1edf864e13f5a91723db20a551f3cecc1ae# ls -al
total 117872424
drwx--x--- 4 root root         4096 Apr  6 23:21 .
drwx--x--- 3 root root         4096 Feb  4 20:34 ..
-rw-r----- 1 root root 120701229407 Apr  7 07:26 5a1edf864e13f5a91723db20a551f3cecc1ae-json.log
drwx------ 2 root root         4096 Feb  2 11:48 checkpoints
-rw------- 1 root root         5351 Apr  6 23:21 config.v2.json
-rw------- 1 root root         1926 Apr  6 23:21 hostconfig.json
-rw-r--r-- 1 root root           10 Apr  6 23:21 hostname
-rw-r--r-- 1 root root          646 Apr  6 23:21 hosts
drwx--x--- 2 root root         4096 Feb  2 11:48 mounts
-rw-r--r-- 1 root root          243 Apr  6 23:21 resolv.conf
-rw-r--r-- 1 root root           71 Apr  6 23:21 resolv.conf.hash
root@314:/var/lib/docker/containers/5a1edf864e13f5a91723db20a551f3cecc1a#

Display More

Baron1971 · Apr 7th 2025

Sorry, that i jumped in here. I have the same Problem.

Code

df -h

shows:

Code

Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
udev            3,7G       0  3,7G    0% /dev
tmpfs           760M    3,8M  756M    1% /run
/dev/sde2       456G    415G   19G   96% /
tmpfs           3,8G       0  3,8G    0% /dev/shm
tmpfs           5,0M    8,0K  5,0M    1% /run/lock
efivarfs        128K     59K   65K   48% /sys/firmware/efi/efivars
tmpfs           3,8G    3,7M  3,8G    1% /tmp
/dev/sde1       511M    148K  511M    1% /boot/efi
/dev/sda        3,7T    1,4T  2,4T   36% /srv/dev-disk-by-uuid-815394b1-4cdf-491c-a749-b3a964369e64
/dev/sdd        4,6T    1,9T  2,7T   41% /srv/dev-disk-by-uuid-5b6687a7-82a6-4107-923a-d17b8311a4a3
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/c2487613335dfee6e430cfb691136ef2515db1991cc718e4427e059f62d247cc/merged
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/c74af4032242de196d5edc91e919f0a3523eac6b86a6180e7c90da70ccc1f83c/merged
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/1c8e23099d3cf391e9aa0ed8af07a07bfafb770418f1ea622da64ce3aaf0eeb2/merged
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/f76624e703390ba896e81348ec56ac1268bca85df9f07f293e612d0d338443f9/merged
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/73cf791d5c5a592051056545ef5453c40308778835679cb19c6423df0817fd34/merged
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/70f5a71347c0d8352db5fcf9eff03e008c9f73b011a8154fcf21e08e339d7282/merged
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/bfcc4b65b6e0eb0dcdfd7118f1ca8f1b6ee2df21bc409902c262782f337e12a9/merged
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/672f6938555bb8792efab2cd90b896ac52bf3501dba86c1689ad7654cabc521e/merged
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/11b371d549dd507067ef8b875e71df54ac3f17d96b8f257aa05050f6b1794adc/merged
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/d8c7cc92d1fc31618c6db02414c8a78a004eadaa3cdd4657f82d05458ba0b461/merged
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/34856d3d8735265ae22c07d6e377043b191ddf6a5e8c195a6784d69b31366b44/merged
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/fc556a7c0908e403de52ccdad04bb6f442e6c2e787a18b6a9d1d750252639d5f/merged
overlay         456G    415G   19G   96% /var/lib/docker/overlay2/3e2adb8a515394bbc4d4367e8148b0a05030de595bbce056a589ff53b50e9049/merged
root@openmediavault:/# mc

Display More

Code

/dev/sde2

shows 96% But when i du -xhd1 There is no Folder that is extremely big.

Code

697M    ./opt
13G    ./var
2,5G    ./usr
8,0K    ./media
4,0K    ./export
136M    ./boot
4,0K    ./mnt
952K    ./docker
6,8M    ./etc
337M    ./root
16K    ./lost+found
1,7M    ./srv
4,0K    ./home
17G    .

Display More

What can i do? What is the Docker Overlay Folder?

j2wtf · Apr 7th 2025

Dont think this is the same problem... Your setup seems a bit more complicated.

I just followed the steps as outlined in the post linked above. This meant recursively go down the folder structure one step at a time with the 'du' command, until you find the problem.

In my case I found a Docker logfile that used about 80% of the bootdisk. This was easy to fix by just using truncate on that file.

(further looking into the issue it turned out I had not specified the log settings in the compose.yaml file I used. So indeed, skill issue on my side. Hopefully it's fixed now by adding a "max size" to the logging options.. )

j2wtf · Apr 7th 2025

Quote from Baron1971

shows 96% But when i du -xhd1 There is no Folder that is extremely big

Did you go into /var/lib/docker/overlay2/[..]/merged to see if there are files there?

Baron1971 · Apr 7th 2025

Quote from j2wtf

Did you go into /var/lib/docker/overlay2/[..]/merged to see if there are files there?

Yes. For Example the first Merged Folder:

Code

root@openmediavault:/var/lib/docker/overlay2/c2487613335dfee6e430cfb691136ef2515db1991cc718e4427e059f62d247cc/merged# du -xhd1
4,0K    ./opt
4,0K    ./tmp
72K    ./var
167M    ./usr
16K    ./media
4,0K    ./proc
1,5M    ./bin
8,0K    ./mnt
12K    ./run
1,3M    ./etc
80K    ./sbin
12K    ./dev
4,0K    ./root
4,0K    ./srv
4,0K    ./sys
1004K    ./lib
8,0K    ./home
171M    .

Display More

There are no Big Files...

j2wtf · Apr 7th 2025

Did you try ls -al ? I'm not seeing any files, du lists only folders.

Baron1971 · Apr 7th 2025

In merged:

Code

drwxr-xr-x 1 root     root     4096 25. Mär 21:53 .
drwx--x--- 5 root     root     4096  7. Apr 19:36 ..
drwxr-xr-x 1 root     root     4096 20. Mär 12:46 bin
-rw-r--r-- 1 www-data www-data 1560 20. Mär 12:46 Caddyfile
drwxr-xr-x 1 root     root     4096 25. Mär 21:53 dev
-rwxr-xr-x 1 root     root        0 25. Mär 21:53 .dockerenv
drwxr-xr-x 1 root     root     4096 25. Mär 21:53 etc
-rwxrwxr-x 1 root     root      124 20. Mär 12:46 healthcheck.sh
drwxr-xr-x 1 root     root     4096 14. Feb 20:14 home
drwxr-xr-x 1 root     root     4096 14. Feb 00:04 lib
drwxr-xr-x 5 root     root     4096 14. Feb 00:04 media
drwxr-xr-x 1 root     root     4096 20. Mär 12:46 mnt
drwxr-xr-x 2 root     root     4096 14. Feb 00:04 opt
dr-xr-xr-x 2 root     root     4096 14. Feb 00:04 proc
drwx------ 2 root     root     4096 14. Feb 00:04 root
drwxr-xr-x 1 root     root     4096 20. Mär 12:46 run
drwxr-xr-x 1 root     root     4096 25. Mär 21:53 sbin
drwxr-xr-x 2 root     root     4096 14. Feb 00:04 srv
-rwxrwxr-x 1 root     root     1883 20. Mär 12:46 start.sh
-rw-rw-r-- 1 root     root      661 20. Mär 12:46 supervisord.conf
drwxr-xr-x 2 root     root     4096 14. Feb 00:04 sys
drwxrwxrwt 1      777 root     4096 14. Feb 00:04 tmp
drwxr-xr-x 1 root     root     4096 20. Mär 12:46 usr
drwxr-xr-x 1 root     root     4096 20. Mär 12:46 var

Display More

j2wtf · Apr 7th 2025

Sorry.. And if you go into /dev.sde2 ?
Also, I dont think having the disk at 96% would cause the login problem, like I had. I would think it needs 100% for that...

gderf · Apr 7th 2025

If you are running Services|Compose | Settings with its Docker storage set to the default /var/lib/docker you are setting yourself up for a recurrence of the problem. You should be using a location on a large data disk mounted in /srv, not on the system disk.

Baron1971 · Apr 7th 2025

Quote from gderf

If you are running Docker with its Docker storage set to the default /var/lib/docker you are setting yourself up for a recurrence of the problem. You should be using a location on a large data disk mounted in /srv, not on the system disk.

The System Disk is a 500GB Disk. Is it to small? Can i easy Change the Docker Storage?

Baron1971 · Apr 7th 2025

Quote from j2wtf

Sorry.. And if you go into /dev.sde2 ?
Also, I dont think having the disk at 96% would cause the login problem, like I had. I would think it needs 100% for that...

I also had 100%, but in the meantime i deleted the logs of the DNLA. I want to investigate the other blowed up Files...

Code

root@openmediavault:/# ls -al
insgesamt 72
drwxr-xr-x  20 root root  4096  5. Mär 16:08 .
drwxr-xr-x  20 root root  4096  5. Mär 16:08 ..
lrwxrwxrwx   1 root root     7 13. Feb 17:56 bin -> usr/bin
drwxr-xr-x   4 root root  4096 23. Mär 21:36 boot
drwxr-xr-x  20 root root  3820  7. Apr 19:35 dev
drwxr-xr-x   3 root root  4096 13. Feb 21:54 docker
drwxr-xr-x  99 root root  4096  7. Apr 19:38 etc
drwxr-xr-x   2 root root  4096 16. Dez 09:36 export
drwxr-xr-x   2 root root  4096 31. Okt 12:04 home
lrwxrwxrwx   1 root root    33  5. Mär 16:08 initrd.img -> boot/initrd.img-6.12.12+bpo-amd64
lrwxrwxrwx   1 root root    32  5. Mär 16:08 initrd.img.old -> boot/initrd.img-6.12.9+bpo-amd64
lrwxrwxrwx   1 root root     7 13. Feb 17:56 lib -> usr/lib
lrwxrwxrwx   1 root root     9 13. Feb 17:56 lib64 -> usr/lib64
drwx------   2 root root 16384 13. Feb 17:55 lost+found
drwxr-xr-x   3 root root  4096 13. Feb 17:57 media
drwxr-xr-x   2 root root  4096 23. Dez 14:38 mnt
drwxr-xr-x   5 root root  4096 13. Feb 21:57 opt
dr-xr-xr-x 371 root root     0  7. Apr 19:34 proc
drwx------   7 root root  4096  7. Apr 19:20 root
drwxr-xr-x  32 root root  1080  7. Apr 20:17 run
lrwxrwxrwx   1 root root     8 13. Feb 17:56 sbin -> usr/sbin
drwxr-xr-x   6 root root  4096 13. Feb 18:11 srv
dr-xr-xr-x  13 root root     0  7. Apr 19:34 sys
drwxrwxrwt  14 root root   360  7. Apr 21:25 tmp
drwxr-xr-x  12 root root  4096 13. Feb 17:56 usr
drwxr-xr-x  12 root root  4096 13. Feb 17:56 var
lrwxrwxrwx   1 root root    30  5. Mär 16:08 vmlinuz -> boot/vmlinuz-6.12.12+bpo-amd64
lrwxrwxrwx   1 root root    29  5. Mär 16:08 vmlinuz.old -> boot/vmlinuz-6.12.9+bpo-amd64
root@openmediavault:/#

Display More

gderf · Apr 7th 2025

Quote from Baron1971

The System Disk is a 500GB Disk. Is it to small? Can i easy Change the Docker Storage?

A 500GB system disk is overly large, but it makes no difference how big or small it is if you fill it to 100%.

Many users here can get by with a 16GB or 32GB USB or SSD disk. I run on a 32GB SSD in an external USB enclosure.

You can move the Docker storage to another disk fairly easily. There are posts on the forum that offer guidance, so seek them out. But basically it is:

Stop the Docker service.

Copy all of /var/lib/docker to another drive. Be sure to retain all ownership and permissions of all the files and folders.

Change the Docker storage folder in Services|Compose | Settings to the new location.

Start the Docker service.

Once you are very sure the relocation went properly you can delete the contents of /var/lib/docker.

Baron1971 · Apr 7th 2025

Quote from gderf

A 500GB system disk is overly large, but it makes no difference how big or small it is if you fill it to 100%.

Many users here can get by with a 16GB or 32GB USB or SSD disk. I run on a 32GB SSD in an external USB enclosure.

You can move the Docker storage to another disk fairly easily. There are posts on the forum that offer guidance, so seek them out. But basically it is:

Stop the Docker service.
Copy all of /var/lib/docker to another drive. Be sure to retain all ownership and permissions of all the files and folders.
Change the Docker storage folder in Services|Compose | Settings to the new location.
Start the Docker service.
Once you are very sure the relocation went properly you can delete the contents of /var/lib/docker.

Display More

The strange thing is that is the first Time the Disk runs full. The Installation 2 Month old. Previously is used a USB- Stick with 128GB without any Problems with the same config. How can i Investigate which Docker Image runs the Disk full?

gderf · Apr 7th 2025

There are other possibilities that can fill the system disk.

One is that a data disk that you write to is mounted to /srv/dev-disk-by-uuid-xxxxxxxx-yyyy-zzzz-aaaa-bbbbbbbbbbbb. But for some reason the disk didn't get mounted or became unmounted.

All those writes will now go into the mountpoint directory but not into the disk because it is no longer attached. That mount point directory is located on the system disk, which might get filled up.

What can make this difficult to spot is that once the disk becomes properly mounted again, you can't see the files that were written into the mountpoint directory because the contents of the attached and mounted disk overlay what you are looking for. But they are still there and they still take up space.

Baron1971 · Apr 7th 2025

Code

root@openmediavault:~# du -sh /var/lib/docker
18G    /var/lib/docker

The Docker Folder is 18GB. Moving to a bigger Disk make no sense.

gderf · Apr 7th 2025

Quote from Baron1971
Code
root@openmediavault:~# du -sh /var/lib/docker
18G    /var/lib/docker
The Docker Folder is 18GB. Moving to a bigger Disk make no sense.

If your system disk is filled up there is a reason for it. You might want to unmount your disks one at a time and examine their mount point directories. They should be empty. If not find out why.

Baron1971 · Apr 7th 2025

I Found it! The MiniDNLA Log Level was set to Error. I disabled it an now:

Code

root@openmediavault:~# df -h
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
udev            3,7G       0  3,7G    0% /dev
tmpfs           760M    5,4M  755M    1% /run
/dev/sde2       456G     15G  419G    4% /
tmpfs           3,8G       0  3,8G    0% /dev/shm
tmpfs           5,0M    8,0K  5,0M    1% /run/lock
efivarfs        128K     59K   65K   48% /sys/firmware/efi/efivars
tmpfs           3,8G    3,7M  3,8G    1% /tmp
/dev/sde1       511M    148K  511M    1% /boot/efi
/dev/sda        3,7T    1,4T  2,4T   36% /srv/dev-disk-by-uuid-815394b1-4cdf-491c-a749-b3a964369e64
/dev/sdd        4,6T    1,9T  2,7T   41% /srv/dev-disk-by-uuid-5b6687a7-82a6-4107-923a-d17b8311a4a3
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/c2487613335dfee6e430cfb691136ef2515db1991cc718e4427e059f62d247cc/merged
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/c74af4032242de196d5edc91e919f0a3523eac6b86a6180e7c90da70ccc1f83c/merged
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/1c8e23099d3cf391e9aa0ed8af07a07bfafb770418f1ea622da64ce3aaf0eeb2/merged
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/f76624e703390ba896e81348ec56ac1268bca85df9f07f293e612d0d338443f9/merged
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/70f5a71347c0d8352db5fcf9eff03e008c9f73b011a8154fcf21e08e339d7282/merged
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/672f6938555bb8792efab2cd90b896ac52bf3501dba86c1689ad7654cabc521e/merged
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/11b371d549dd507067ef8b875e71df54ac3f17d96b8f257aa05050f6b1794adc/merged
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/34856d3d8735265ae22c07d6e377043b191ddf6a5e8c195a6784d69b31366b44/merged
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/fc556a7c0908e403de52ccdad04bb6f442e6c2e787a18b6a9d1d750252639d5f/merged
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/f69f732c50b50bf9aab24c485832def7a119c419bd0227e8ad592ca4ce710d29/merged
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/d5c5afc2a3d08c21496a734afc18095a6d7e287bac4d45f6cc8e72a4e2867804/merged
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/4feb8666886e0c84ee1dc61372ccba8ba23971b08f8573ab98bdaf92023c191b/merged
overlay         456G     15G  419G    4% /var/lib/docker/overlay2/3e2adb8a515394bbc4d4367e8148b0a05030de595bbce056a589ff53b50e9049/merged
root@openmediavault:~#

Display More

Disk use 15G!

Thanks for your help.

Participate now!