full OS disk

JJakaJonas · Jun 16th 2024

Hello all.

I discovered today that my OS disk is full.

I jumped on to Google to try get a solution for it.

I got so far that I tried to clear my docker images and gained 1.5 GB free storage and access to my webGUI.

The thing is now that I'm stuck on getting more free storage of my 128 GB disk.

When I run df does it looks like the data is to find in my overlay folder.

But when I check the folder in winSCP is it under 1 GB...

What do I do from here?

ryecoaaron · Jun 16th 2024

Thread

[How-To] Fix full OS filesystem - GUI login loop

Symptoms

When you try to login to the GUI of OMV you are not being logged in, but the login page shows up again

Check, if a full root filesystem is the problem
ssh into your server (check the new user guide on how to ssh into your server) and run following command
df df will show you the usage of each filesystem
the output of df will look like this:
(Code, 19 lines)

The filesystem dev/sda1 which is mounted on "/" is the OS filesystem. The use of that filesystem is 100% --> the OS filesystem is full.

…

macom

Jul 23rd 2020

JJakaJonas · Jun 16th 2024

Well that's the thing.

I followed that guide.

And this is the result..

Code

root@nas:/# df
Filesystem                           1K-blocks       Used  Available Use% Mounted on
udev                                  16357452          0   16357452   0% /dev
tmpfs                                  3277088       3304    3273784   1% /run
/dev/sde2                            120983240  113348608    1442812  99% /
tmpfs                                 16385432          0   16385432   0% /dev/shm
tmpfs                                     5120          0       5120   0% /run/lock
tmpfs                                 16385432          0   16385432   0% /tmp
/dev/sde1                               523244        148     523096   1% /boot/efi
/dev/sda1                           3844550452 2885959208  958574860  76% /srv/dev-disk-by-uuid-528c161d-88d4-4868-97da-b44cd199c6d2
/dev/md0                            8719952852 5804151456 2915785012  67% /srv/dev-disk-by-uuid-9cbdb19d-09f7-4c47-b2e2-41d274f5f2f2
jjakajonas_google_drive_1705274571:   15728640   13460280    2268360  86% /mnt/jjakajonas_google_drive_1705274571
overlay                              120983240  113348608    1442812  99% /var/lib/docker/overlay2/07ad6d2ac0af4ae2b79003bce3721d7cd4a3d623070d93b9beea9a9f4461da0c/merged
tmpfs                                  3277084          0    3277084   0% /run/user/0
root@nas:/# du -xhd1
16K     ./lost+found
4.0K    ./export
87G     ./media
2.0G    ./opt
76K     ./mnt
4.8G    ./DATA
1.8M    ./srv
122M    ./boot
4.7G    ./var
88M     ./home
308K    ./root
11M     ./etc
10G     ./usr
109G    .
root@nas:/#

Display More

I don't see a folder to poke in..

Also the numbers when I do "du -xhdi" don't add up.

As I can see there's around 80 GB data not accounted for

ryecoaaron · Jun 16th 2024

You probably deleted a file that a process still has a handle on. Reboot your system and see if the space adds up.

JJakaJonas · Jun 16th 2024

I have done that multiple times now.

Nothing happens.

I have even shut the machine down for over 15 minutes and turned it on again.

ryecoaaron · Jun 16th 2024

Quote from JJakaJonas

I have even shut the machine down for over 15 minutes and turned it on again.

It isn't a car. It won't help to leave it off longer.

You probably have files in the mount point and you can't see them when the other filesystems are mounted. I would unmount each of these filesystems one at a time to check

/srv/dev-disk-by-uuid-528c161d-88d4-4868-97da-b44cd199c6d2
/srv/dev-disk-by-uuid-9cbdb19d-09f7-4c47-b2e2-41d274f5f2f2
/mnt/jjakajonas_google_drive_1705274571

JJakaJonas · Jun 16th 2024

Those 3 is not the OS drive.

So can the problem be on one of them?

gderf · Jun 16th 2024

Quote from JJakaJonas

Those 3 is not the OS drive.
So can the problem be on one of them?

They are probably mount point directories located in the root of the system drive. If there are files in such a mountpoint and a drive is then mounted there the files will be invisible and not accounted for. But they still take up space on the system drive, perhaps enough to fill it.

You need to umount those locations and then look to see what remains inside them.

JJakaJonas · Jun 16th 2024

I will try that tomorrow and see what happens

JJakaJonas · Jun 17th 2024

have now tried

Code

unmount /dev/sde1
unmount /dev/sdd1
unmount /dev/md0

And they all report

Code

-bash: unmount: command not found

Soma · Jun 17th 2024

Quote from JJakaJonas

-bash: unmount: command not found

It's umount

JJakaJonas · Jun 17th 2024

I just found out.

And after I unmounted sde1 and md0 is the problem the same.

JJakaJonas · Jun 17th 2024

And when I try to unmount md0 now it tells me it's busy

Soma · Jun 17th 2024

Will only point this since I see (according to your signature) that you are using OMV with CasaOS which isn't advised.

All it takes is for a bad configured volume on docker for the HOST to have a full OS.

Quote from JJakaJonas

87G ./media

This is where you need to figure out what is wrong.

A normal install would have this on the media folder

Code

USER@HOST:~ $ sudo du -xhd1 /
8.0K    /media
62M     /boot
304K    /root
4.0K    /mnt
1.8M    /srv
715M    /var
21M     /home
16K     /lost+found
972K    /ORBackup
16K     /opt
1.8G    /usr
7.4M    /etc
4.0K    /export
2.6G    /

Display More

So, start going down that folder.

du -xhd1 /media

Also, your overlay is showing ~113G,

Code

Filesystem     1K-blocks      Used Available Use% Mounted on
overlay        120983240 113348608   1442812  99% /var/lib/docker/overlay2/07ad6d2ac0af4ae2b79003bce3721d7cd4a3d623070d93b9beea9a9f4461da0c/merged

Which again, shows that docker root is full and wrong configured.

Normally, means a container went rogue and didn't had the volume mounted when it was running/restarted.

So, go with the above tips.

Good luck

JJakaJonas · Jun 17th 2024

I have totally over looked that media folder.

The reason for the Casaos is purely for docker administration and nothing else.

JJakaJonas · Jun 17th 2024

The problem was a bunch of files from a USB hard drive I hade connected to the server once.

full OS disk

crashtest Jun 16th 2024

JJakaJonas Jun 17th 2024

JJakaJonas Jun 17th 2024

Participate now!

Similar Threads

[SOLVED] Scheduled Tasks giving "500 - Internal Server Error Failed to execute command" while script in terminal and crontab works fine

Recently updated my Openmediavault release, now I can not login even after using omv-firstaid (400-bad request)

Discovered critical bug involving shared folders and reset permissions that can wipe a shared folder

System Partition Full by using Pyload on docker with MergerFS on USB disk

Remote Mount to Windows Share has quota error

Filesystems "gone" After fixing GUI Loop due to Full Root File System