[How-To] Fix full OS filesystem - GUI login loop

Symptoms

When you try to login to the GUI of OMV you are not being logged in, but the login page shows up again

Check, if a full root filesystem is the problem

ssh into your server (check the new user guide on how to ssh into your server) and run following command

df df will show you the usage of each filesystem

the output of df will look like this:

15:01:03 root@server:~# df
Filesystem      1K-blocks       Used  Available Use% Mounted on
udev              8033820          0    8033820   0% /dev
tmpfs             1610128      27588    1582540   2% /run
dev/sda1         39108608   39108608          0 100% /
tmpfs             8050632          0    8050632   0% /dev/shm
tmpfs                5120          0       5120   0% /run/lock
tmpfs             8050632          0    8050632   0% /sys/fs/cgroup
tmpfs             8050632          0    8050632   0% /tmp
folder2ram        8050632      75248    7975384   1% /var/log
folder2ram        8050632          4    8050628   1% /var/tmp
folder2ram        8050632        624    8050008   1% /var/lib/openmediavault/rrd
folder2ram        8050632        996    8049636   1% /var/spool
/dev/sda2        76360744    9428044   63010780  14% /srv/dev-disk-by-label-dataos
folder2ram        8050632      11648    8038984   1% /var/lib/rrdcached
/dev/md127     2884009464 1680001056 1203992024  59% /srv/dev-disk-by-label-pool
folder2ram        8050632          4    8050628   1% /var/lib/monit
folder2ram        8050632       1604    8049028   1% /var/cache/samba
/dev/sdb1      2930265540 1627081640 1300996536  56% /srv/dev-disk-by-label-internalbackup

The filesystem dev/sda1 which is mounted on "/" is the OS filesystem. The use of that filesystem is 100% --> the OS filesystem is full.

Find which folder is filling the filesystem

in the CLI run

cd / to change to the root of the OS file system

du -xhd1 du will show you the size of each folder on the root (x: do not descend into other filesystem; h: display output in human readable format; d1: show only first level of directories)

the output will look like this:

15:08:05 root@hawk:/# du -xhd1
9.8M    ./etc
0       ./media
2.2G    ./var
2.3G    ./usr
1.4G    ./boot
20K     ./home
5.7M    ./root
0       ./mnt
20.2G   ./srv
0       ./opt
0       ./export
4.0M    ./tmp_urbackup
16K     ./.config
0       ./sharedfolders
25.8G   .

(To get only the 10 largest folders/files in order, use du -ahxd1 | sort -hr | head -n 10)

Identify the largest folder and change into that folder with

cd foldername replace foldername by the name of the largest folder

Repeat these steps until you identified the "deepest" folder which is filling your root filesystem.

Change into that folder and check with ls -al what is inside. ls lists the directory content (a: shows you also hidden entries, l: shows in a long list format)

Further steps depend on the reason:

Common Reasons for full OS filesystem

• Docker (e.g. large media library of plex)
  • move docker from the OS filesystem to another one - RE: Installation and Setup Videos - Beginning, Intermediate and Advanced
  • remove old images and volumes that are not needed anymore - Cleaning up Dockers
• a rsync job was running, when the target filesystem was not mounted --> the files were copied to a folder, which was created instead of mount point (usually in /srv/) (see also post #4)
  
  • a symptom of this is, that the results for du and df do not match
    (i.e disk capacity minus reported usage with du is not similar to the result of df)
  • Unmount all drives in /srv from CLI and force the unmount if needed; as an alternative shutdown, unplug all data drives, boot
  • look in each mountpoint directory in /srv and see if anything is in there
  • delete the wrongly created folder (if you are in doubt physically disconnect the related hard drive to not delete the content on that drive)
• log files are spammed --> check the content of the log file and fix the root cause why the log file is spammed; after you fixed the problem you can reduce the size of the log file to 0 with truncate -s 0 logfile.log replace logfile.log

If you have any improvements that should be implemented, please let me know.

A nice tool to support the process of identifying the folder that is filling the OS filesystem is ncdu

Install with apt-get install ncdu

Again move to the root with cd /

Start ncdu with ncdu -x. Again the option x is preventing ncdu from descending into other filesystems (i.e. your data drives) which might take a long time.

Sample output:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help                                                                                                                                                         
--- / ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    2.2 GiB [##########] /usr                                                                                                                                                                                        
    2.2 GiB [######### ] /var
    1.3 GiB [#####     ] /boot
    9.7 MiB [          ] /etc
    5.7 MiB [          ] /root
    3.9 MiB [          ] /tmp_urbackup
    1.2 MiB [          ] /srv
   20.0 KiB [          ] /home

Press q to quite.

If the login loop is not caused by a full file system, try

• force reload of the GUI (check how this is done for your browser, for FIrefox: Ctrl+Shift+R)
• clear browser cache
• open your browser in Privacy/Incognito mode

and see if the problem persists.

This tip is from Zoki :

If you can't find the folder filling the filesystem

Sometimes you see that you OS filesystem is full, but can not find which folder is filling your filesystem with the du command shown above. You may run into this situation, if some process wrote to the place where it expected a mounted disk, but the disk was not mounted at the time of writing. Now, while you are looking at it, the disk is mounted and you can not see the files on the OS file system, because the mounted disk is hiding it.

Excursion: Filesystems are mounted at so called mount points. A mount point is nothing else than a directory. So if a formerly mounted filesystem is temporarily not mounted anymore the directory still exists, but is empty.

OMV mounts all disks to the /srv directory, possibly hiding the contents of some directories used as mount points. To reveal its real contents , mount it to a different place and check:

mkdir -p /mnt/rootfs                  # create an additional mount point for your root file system
mount --bind /srv /mnt/rootfs         # mount /srv to the new mount point
cd /mnt/rootfs                        # go to the place where the real contents of /srv can be seen
du -hxd1 /mnt/rootfs | sort -hr       # check disk usage in the mount point

You will see a list of directories ordered by their size:

root@omv:/mnt/rootfs# du -hxd 1 /mnt/rootfs | sort -hr
6,5G    /mnt/rootfs
4,5G    /mnt/rootfs/docker
2,0G    /mnt/rootfs/docker-apps
1,7M    /mnt/rootfs/salt
32K     /mnt/rootfs/zfs
32K     /mnt/rootfs/pillar
8,0K    /mnt/rootfs/ftp
4,0K    /mnt/rootfs/dev-disk-by-uuid-e887f188-f1ac-4126-9238-32963bb8dced
4,0K    /mnt/rootfs/dev-disk-by-label-docker
root@omv:/mnt/rootfs#

All the directories starting with dev-disk-by- should be at the very bottom of the list, as they should be empty. If not, you have found why your OS disk is filled up. Delete the contents of these directories and see your OS file system regaining free space.

Do not touch the directories not starting with dev-disk-by- or any symlinks , as they contain real data, which is still in use at /srv/.

When you are done, unmount the rootfs again

umount /mnt/rootfs        # unmount the /srv directory from /mnt/rootfs
rmdir /mnt/rootfs         # remove to mount point you created before

In some cases deleting a file does not release the space used on the filesystem. In that case the space will be released during a reboot or a restart of the service that is using that file.

So in doubt, reboot the system.