[How-To] Fix full OS filesystem - GUI login loop

  • Symptoms

    When you try to login to the GUI of OMV you are not being logged in, but the login page shows up again


    Check, if a full root filesystem is the problem

    ssh into your server (check the new user guide on how to ssh into your server) and run following command

    df df will show you the usage of each filesystem

    the output of df will look like this:

    The filesystem dev/sda1 which is mounted on "/" is the OS filesystem. The use of that filesystem is 100% --> the OS filesystem is full.


    Find which folder is filling the filesystem

    in the CLI run

    cd / to change to the root of the OS file system

    du -xhd1 du will show you the size of each folder on the root (x: do not descend into other filesystem; h: display output in human readable format; d1: show only first level of directories)

    the output will look like this:

    (To get only the 10 largest folders/files in order, use du -ahxd1 | sort -hr | head -n 10)


    Identify the largest folder and change into that folder with

    cd foldername replace foldername by the name of the largest folder


    Repeat these steps until you identified the "deepest" folder which is filling your root filesystem.

    Change into that folder and check with ls -al what is inside. ls lists the directory content (a: shows you also hidden entries, l: shows in a long list format)

    Further steps depend on the reason:


    Common Reasons for full OS filesystem

    • Docker (e.g. large media library of plex)
    • a rsync job was running, when the target filesystem was not mounted --> the files were copied to a folder, which was created instead of mount point (usually in /srv/) (see also post #4)
      • a symptom of this is, that the results for du and df do not match
        (i.e disk capacity minus reported usage with du is not similar to the result of df)
      • Unmount all drives in /srv from CLI and force the unmount if needed; as an alternative shutdown, unplug all data drives, boot
      • look in each mountpoint directory in /srv and see if anything is in there
      • delete the wrongly created folder (:!:if you are in doubt physically disconnect the related hard drive to not delete the content on that drive:!:)
    • log files are spammed --> check the content of the log file and fix the root cause why the log file is spammed; after you fixed the problem you can reduce the size of the log file to 0 with truncate -s 0 logfile.log replace logfile.log

    If you have any improvements that should be implemented, please let me know.

  • A nice tool to support the process of identifying the folder that is filling the OS filesystem is ncdu


    Install with apt-get install ncdu

    Again move to the root with cd /

    Start ncdu with ncdu -x. Again the option x is preventing ncdu from descending into other filesystems (i.e. your data drives) which might take a long time.


    Sample output:

    Code
    ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help                                                                                                                                                         
    --- / ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
        2.2 GiB [##########] /usr                                                                                                                                                                                        
        2.2 GiB [######### ] /var
        1.3 GiB [#####     ] /boot
        9.7 MiB [          ] /etc
        5.7 MiB [          ] /root
        3.9 MiB [          ] /tmp_urbackup
        1.2 MiB [          ] /srv
       20.0 KiB [          ] /home

    Press q to quite.

  • macom

    Added the Label core
  • If the login loop is not caused by a full file system, try

    • force reload of the GUI (check how this is done for your browser, for FIrefox: Ctrl+Shift+R)
    • clear browser cache
    • open your browser in Privacy/Incognito mode

    and see if the problem persists.

  • This tip is from Zoki :


    If you can't find the folder filling the filesystem


    Sometimes you see that you OS filesystem is full, but can not find which folder is filling your filesystem with the du command shown above. You may run into this situation, if some process wrote to the place where it expected a mounted disk, but the disk was not mounted at the time of writing. Now, while you are looking at it, the disk is mounted and you can not see the files on the OS file system, because the mounted disk is hiding it.


    Excursion: Filesystems are mounted at so called mount points. A mount point is nothing else than a directory. So if a formerly mounted filesystem is temporarily not mounted anymore the directory still exists, but is empty.


    OMV mounts all disks to the /srv directory, possibly hiding the contents of some directories used as mount points. To reveal its real contents , mount it to a different place and check:

    Code
    mkdir -p /mnt/rootfs                  # create an additional mount point for your root file system
    mount --bind /srv /mnt/rootfs         # mount /srv to the new mount point
    cd /mnt/rootfs                        # go to the place where the real contents of /srv can be seen
    du -hxd1 /mnt/rootfs | sort -hr       # check disk usage in the mount point


    You will see a list of directories ordered by their size:

    All the directories starting with dev-disk-by- should be at the very bottom of the list, as they should be empty. If not, you have found why your OS disk is filled up. Delete the contents of these directories and see your OS file system regaining free space.

    :!: Do not touch the directories not starting with dev-disk-by- or any symlinks , as they contain real data, which is still in use at /srv/. :!:


    When you are done, unmount the rootfs again

    Code
    umount /mnt/rootfs        # unmount the /srv directory from /mnt/rootfs
    rmdir /mnt/rootfs         # remove to mount point you created before

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!