Systemdisk/OS disk almost full

  • Hi.
    I have not spent two days searching for an answer in this and other forums, but have not managed to find a solution. Now I am desperate. :S


    I have a system with a 120 GB SSD drive were OMV has allocated 40 GB space (/dev/sde1).
    The system also contain 6 drives in a Snapraid configuration.


    After installing dockers and setting up a few containers I got a notification a few days ago that the OS disk was getting full, 89%.
    To me it seems that one or a few of the containers filled up the OS drive.
    Also, it seems I might have set up one or more folders erroneously, not pointing to the Snapraid pool of disks, which might have contributed to the problem that data got stored on the OS drive.
    I decided to delete the containers, but this did not solve the problem. I deleted the shared folders that might have been set up erroneously with no changes in disk space.
    I followed up with uninstallation of Docker. Still no solution. Ran rm -r of the suspected folder, files removed, but still no solution. Restart of OMV did not help.


    So, I am stuck. Running df and du has not managed to help me to figure out which folder/file is taking up space on my OS drive after the previous efforts.


    My conclusion: I suspect that somehow containers did take up space, upon deletion of the containers and file/folder, the space on the drive is still occupied but is not visible for the system (or me)? Is this at all possible?


    I need help, please. Alternatively, I have to do a full re-installation of OMV, not a preferred solution.


    Here follows the du and df commands:






    Or this alternative du ...


    • Offizieller Beitrag

    See what you can find under /var/lib/docker The sub-dir /containers might have something interesting.
    Unfortunately, Docker uses a merged directory structure to create containers, that's not unlike UnionFS. Basically the components of a container consist of overlays that are compiled from more than 1 folder on the boot drive.


    The following is only speculation:
    What would be more interesting, and much more likely to contain a lot of data, would the host side path to your deleted containers Volumes and Bind points. If a path to a data drive was not specifically set, and you used a path that might have been part of a containers default set up, the destination was probably set on your boot drive. Depending on what that container is/was, (a downloader?) a lot of data might end up on your boot drive.


    Another Possibility:
    If the meta-data location is not specifically changed, media servers like Plex are known for automatically collecting and dropping several Gig's of meta-data, for media files, on the boot drive.


    As you've noted, Linux has hidden folders and files. While I don't think this will be productive, the following will list all hidden folders.


    find / -type d -iname ".*" -ls
    The following is for all hidden files.
    find / -type f -iname ".*" -ls



    *If you don't have it installed already, WinSCP might make it easier to visualize and navigate your boot drive.* It installs on a Windows client and connects to OMV by SSH.
    ____________________________________________


    A potential patch, if you don't want to rebuild right now:
    Since you have a 120GB SSD, with only 40GB used, Gparted can expand OMV's root partition.


    The next time around, give some consider to backing up your boot drive to an image file, once in a awhile. As you might imagine, it's nice to have a punt option if something goes wrong.

  • See what you can find under /var/lib/docker The sub-dir /containers might have something interesting.

    I have un-installed docker, so there is no docker folder.



    As you've noted, Linux has hidden folders and files. While I don't think this will be productive, the following will list all hidden folders.
    find / -type d -iname ".*" -ls
    The following is for all hidden files.
    find / -type f -iname ".*" -ls


    After running the du -hxd1, which from my understanding will include hidden files, I still can not identify files or folders to occupy 39 GB of data. See my first post.


    Here is the suggested find command:



  • I still have not managed to identify the files that are filling up my OS-disk.


    As a result I have had to boot to GParted and adjust my partitions twice now. If this continues I will run out of space on the disk.


    Does anyone have an idea of what I can do next? Would it be advisable to address this issue on the Debian forum, since OMV is based on Debian?

    • Offizieller Beitrag

    You can install ncdu it will show you a nice list of folders and the space they use (including hidden folders)
    apt install ncdu


    then start with


    ncdu -x


    the -x will prevent to read also other filesystem, so your data drive will not be included


    quit ncdu with q

    • Offizieller Beitrag

    If a rsync job runs without that the target drive is available, it creates a folder in the /media/ or /srv/ folder and puts the data there. The data are then stored on the OS drive instead of the data drive.


    So check out these two folders, if there are folders which are not mount points for drives.

    • Offizieller Beitrag

    By chance do you have Plex or Emby installed? If you do, and you have a lot of media files, the folder where these managers store metadata can grow to enormous sizes. The more media files added (or as these media managers search out media file related data), the larger the metadata folder grows.


    I also had a problem, early on, with UrBackup. Urbackup was trying and failing to backup a UEFI client that had a problem. That resulted in a growing collection of temp files. Once the backup was successful, UrBackup erased the temp files.

  • Code
    root@OMV:/# df -h /dev/sdg1
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/sdg1        91G   49G   38G  57% /


    ncdu -x


    So, using ncdu, what is making up 49GB on my drive?

  • If a rsync job runs without that the target drive is available, it creates a folder in the /media/ or /srv/ folder and puts the data there. The data are then stored on the OS drive instead of the data drive.


    So check out these two folders, if there are folders which are not mount points for drives.

    The more likely scenario is something writing to a mount point with no drive mounted to that mountpoint. When the drive is eventually mounted, the contents of the drive will appear in the mount point directory and what was previously written in the directory will not be visible but is still present and taking up space on the rootfs.


    So, it is best to check by first unmounting the drive, then looking in the mount point directory. It should be empty.


    Reply being delayed because it is wrongly being held for moderation.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • The more likely scenario is something writing to a mount point with no drive mounted to that mountpoint. When the drive is eventually mounted, the contents of the drive will appear in the mount point directory and what was previously written in the directory will not be visible but is still present and taking up space on the rootfs.
    So, it is best to check by first unmounting the drive, then looking in the mount point directory. It should be empty.


    Reply being delayed because it is wrongly being held for moderation.

    This seems like a potential cause.
    I did see at some point thought my docker container Sabnzbd to store data on my OS-disk. This was caused by the data point being pointed erroneously in the docker container (to a mount point/folder in my MergerFS pool). I corrected the path but it did not make a difference. I did uninstall Docker and all the containers, but it did not make a difference. I though this was not the cause - maybe it was.
    Could it be that the data stored by the containers was not deleted on the disk and is therefore taking up space even though you can not identify that by du or ncdu command?


    If data from "ghost" containers still takes up space on the rootfs, how can it be removed?

    • Offizieller Beitrag

    I might be being dumb in my observation here but from the first post:


    This /dev/sde1 47G 40G 5.0G 89% / I assume is the SSD in question


    This /dev/sdg1 3.6T 708G 2.9T 20% /srv/dev-disk-by-label-Disk6 is part of part of MergFS


    If the above is correct what is this:

    • root@OMV:/# df -h /dev/sdg1
    • Filesystem Size Used Avail Use% Mounted on
    • /dev/sdg1 91G 49G 38G 57% /

    or am I missing something, I understand what I see (unless I need new glasses) but I can't understand/decipher it.

  • Typically, dockers write data to two places. One possibility is to a Volume and Bind mount that connects a directory inside the container to a directory outside the container somewhere on the system. That "somewhere" could be anywhere the container has permission to write to, including the rootfs. The other possibility is that the data is written to a point inside the container. This is what happens if a Volume and Bind mount is not specified and the container needs to write data. If the container is located on the system drive, it could fill it up.


    Data written to locations inside a container will be deleted if the container is deleted. But if the data was aimed at a Volume and Bind mount outside the container it will not be deleted if the container itself is deleted.


    Data written onto the rootfs can be deleted. The problem is finding it.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • Code
    root@OMV:/# lsof +L1
    COMMAND     PID                  USER   FD   TYPE DEVICE SIZE/OFF NLINK   NODE NAME
    php-fpm7. 17624                  root    3u   REG   0,40        0     0 918917 /tmp/.ZendSem.mUQiM7 (deleted)
    php-fpm7. 17625              www-data    3u   REG   0,40        0     0 918917 /tmp/.ZendSem.mUQiM7 (deleted)
    php-fpm7. 17626              www-data    3u   REG   0,40        0     0 918917 /tmp/.ZendSem.mUQiM7 (deleted)

    Found some deleted files still being kept "open". These might be a problem. Not sure how I will tackle this further though. Have tried to perform kill -9 PID (e.g. kill -9 17624, but this did not help)

  • I have added two more disks and the disk "names" have changed from the first post.


    sdg1 is now the SSD. I understand that was confusing.

    • Offizieller Beitrag

    sdg1 is now the SSD. I understand that was confusing.

    Not going to need new glasses then :) There seems to be a quite a number of ways to obtain information as to where disk space has gone this appears to be quite extensive, even down to suggesting mc and dutree.

  • I have managed to solve the problem. It seems that some hidden files and folders were present. I posted the problem on the Debian forum and got an answer. Wanted to post the solution here so that other user could troubleshoot "the same problem" using the same approach.


    You need to Mount bind the root ( /) to another mount point. First create the folder mnt:


    Code
    mkdir /mnt


    Then mount bind the root to /mnt


    Code
    mount --bind / /mnt

    Then perform:

    Code
    du -h --max-depth=1 /mnt


    I then found that there was another folder called sharedfolders with lot of files and folders, only visible upon doing a mount bind.
    After removing them on the new mount point i performed:

    Code
    umount /mnt


    Files and folders are now gone and space on the disk is back!


    Thanks to everyone who provided help and suggestions!

    • Offizieller Beitrag

    All's well that ends well, but one would have to wonder; what happened where "sharedfolders" became hidden and what was adding files to it?


    Was there anything on the Debian forum that might explain or suggest how it happened?
    (Just curious.)

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!