Inexplicable Disk Usage

  • Today I went to install updates on my OMV box, and found that my root (/) partition was completely full. It is about 29GB and is "physically" the majority of the space on a 32gb EMMC card in an ODROID XU4. When I run a simple "df -h" from the root folder, I find the following (note that this is after I cleared up a little bit of space on it):


    Filesystem Size Used Avail Use% Mounted on/dev/mmcblk0p2 29G 28G 410M 99% /udev 10M 0 10M 0% /devtmpfs 399M 6.8M 392M 2% /runtmpfs 996M 8.0K 996M 1% /dev/shmtmpfs 5.0M 0 5.0M 0% /run/locktmpfs 996M 0 996M 0% /sys/fs/cgrouptmpfs 996M 8.0K 996M 1% /tmp5a90f259-21b0-4de9-8b75-3b0d173162ce:82342386-dd77-464b-bd8b-ba3ef9c26bb4 5.4T 1.8T 3.6T 34% /export/Share/dev/mmcblk0p1 71M 32M 40M 44% /boot/dev/sda1 2.7T 1.9T 905G 68% /media/dac85fc4-9594-4bdd-877e-bd111753339c/dev/sdb1 2.7T 1.8T 909G 67% /media/5a90f259-21b0-4de9-8b75-3b0d173162ce /dev/sdc1 2.7T 189M 2.7T 1% /media/82342386-dd77-464b-bd8b-ba3ef9c26bb4


    However, when I use a du command (excluding the external storage, returning just the top 25 items, I can only account for about 3GB (even if I include everything, it doesn't go up but about .1 GB), which is about what I'd expect as I don't really use a lot of disk space on that partition for anything.


    root@trunks:/# du --exclude=media --exclude=export -ch -d 10 | sort -hr | head -n25du: cannot access ‘./proc/17757/task/17757/fd/4’: No such file or directorydu: cannot access ‘./proc/17757/task/17757/fdinfo/4’: No such file or directorydu: cannot access ‘./proc/17757/fd/4’: No such file or directorydu: cannot access ‘./proc/17757/fdinfo/4’: No such file or directory3.0G total3.0G .1.5G ./usr796M ./var728M ./var/lib668M ./root566M ./usr/share551M ./usr/lib523M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server523M ./var/lib/plexmediaserver/Library/Application Support523M ./var/lib/plexmediaserver/Library523M ./var/lib/plexmediaserver366M ./root/go1.5265M ./root/go1.4239M ./usr/lib/arm-linux-gnueabihf201M ./usr/local/go201M ./usr/local193M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata178M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies174M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Media/localhost174M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Media164M ./usr/lib/plexmediaserver163M ./root/go1.5/pkg146M ./root/go1.5/.git/objects/pack146M ./root/go1.5/.git/objectsI wondered a little bit about the lines from the du that indicate it can't access certain files in /proc, so I thought maybe they were hung processes that were holding a bunch of disk space, but I tried rebooting the server and nothing changed. I guess I"m just a little bit stumped and hoping for any suggestions.

    Odroid XU4
    8TB Main Data | 3x3TB Backup Data
    OMV 5.6.2-1
    Docker | Plex | MergerFS | Rsync

    2 Mal editiert, zuletzt von jarodmerle ()

  • Here's the command line output in a hopefully more readable format (it didn't want to let me change them to code blocks in "edit" mode of the first post):



    Odroid XU4
    8TB Main Data | 3x3TB Backup Data
    OMV 5.6.2-1
    Docker | Plex | MergerFS | Rsync

  • A couple other tidbits I just found. After I had rebooted the server, I received a whole slew of notifications that I guess had gotten hung up, which may speak to some sort of issue that could relate to this. Amongst them were a couple that may also be pertinent:

    Resource limit matched Service fs_media_6f6e68de-1044-4129-b57a-53b8e38a5bf6

    Date: Sat, 15 Jul 2017 15:22:03
    Action: alert
    Host: trunks.trunks
    Description: space usage 98.5% matches resource limit [space usage>85.0%]



    This one is interesting, because the file system it is referring to is, I believe, that of an external drive I had to remove last weekend because it was failing. I had a 4 (now 3) drive SnapRAID array, of which this was a part. When I try to go to the mount point under /media for this, the directory still appears to be there, but I cannot enter it.


    Then, there's this one:


    The Samba 'panic action' script, /usr/share/samba/panic-action,
    was called for PID 18489 (/usr/sbin/smbd).


    This means there was a problem with the program, such as a segfault.
    However, gdb was not found on your system, so the error could not be
    debugged. Please install the gdb package so that debugging information
    is available the next time such a problem occurs.


    I've never seen a notification like this before, and my Samba share appears to still be working without any issue. I guess this might just be erroneous, but I figured it was worth noting.

    Odroid XU4
    8TB Main Data | 3x3TB Backup Data
    OMV 5.6.2-1
    Docker | Plex | MergerFS | Rsync

  • Since you have no signature of your used stuff, one can just guess...


    Its a Raspi?
    /dev/mmcblk0p2 is a SDCard or something like that you are booting from?


    Shot in the dark: SDCard worn out and broken...

    --
    Get a Rose Tattoo...


    HP t5740 with Expansion and USB3, Inateck Case w/ 3TB WD-Green
    OMV 5.5.23-1 Usul i386|4.19.0-9-686-pae

  • Hi you have to look fast that you have more space
    Otherwise Raspi will not start anymore.
    I would install the Plex media server on a vin your hard disks.


    Ps: I was a copy of your make, with Win32DiskImager

    https://github.com/Wolf2000Pi


    OMV6 Hewlett-Packard HPE-411at - Intel Core i7 CPU 870 @ 2.93GHz - 16GB Ram

    Proxmox omv 7 sandworm Dell OptiPlex 7050 i5-6500 CPU @ 3.20GHz - 32GB Ram (Test!!!!)

    Einmal editiert, zuletzt von Wolf2000 ()

  • Since you have no signature of your used stuff, one can just guess...


    Its a Raspi?
    /dev/mmcblk0p2 is a SDCard or something like that you are booting from?


    Shot in the dark: SDCard worn out and broken...


    Sorry, kind of buried it in my post. Don't post here much (obviously) so never really thought about adding a signature. I'll put some info there now.


    Hi you have to look fast that you have more space
    Otherwise Raspi will not start anymore.
    I would install the Plex media server on a vin your hard disks.


    Ps: I was a copy of your make, with Win32DiskImager


    It's an ODROID-XU4. Plex is only taking up about 500MB from everything I can tell, so it doesn't seem like it should be an issue. In my original build, I had tried moving the Plex library to a different location and it caused me all sorts of grief.



    This only reports 3K of data on /media. Could it actually be more than that?





    Code
    du -shx /media

    should do the job more quickly?


    This, likewise, reports the same 3K the other take on du did. Good to know about these different versions though.


    Code
    root@trunks:~# du -shx /media
    3.0K    /media


    Hi tkaiser
    If he has OMV 3 then
    du -shx / srv


    I am running OMV 3 (3.0.84, to be specific). I assume the space between "/" and "srv" in this one was a typo, so here's the output on it too. Just 3K like the others.


    Code
    root@trunks:~# du -shx /srv
    3.0K    /srv

    Thank you to all for all the ideas. I'm going to try drilling down on media and srv and see if I find anything else.

    Odroid XU4
    8TB Main Data | 3x3TB Backup Data
    OMV 5.6.2-1
    Docker | Plex | MergerFS | Rsync

  • Well, according to df your rootfs is running full. So

    Code
    du -shx --exclude=proc /* /.*

    should report where most disk space is used.

    Right, I agree that it should report where the space is being used, but unfortunately it doesn't. It, again, only accounts for about 3GB out of 29GB that appear to be used, most of which is in /usr and /var. Here's the output of this one, just for completeness sake.



    Odroid XU4
    8TB Main Data | 3x3TB Backup Data
    OMV 5.6.2-1
    Docker | Plex | MergerFS | Rsync

  • A couple other things that I've tried based on googling similar problems:


    1. Remounted the device behind my root FS in a different location and repeated some of the same "du" analysis. A few posts online seemed to indicate this might reveal files that were "hidden" in the "/" mountpoint, but they just reported the exact same results.
    2. I had noticed that some references to the failed drive I had to remove were lingering in configuration (/proc/mounts, /etc/fstab) so I jumped through some hoops to get those cleaned up (deleting/recreating shares and rebuilding my MergerFS pool, primarily), thinking that maybe that had something to do with it, but it made no difference. Recreating my SMB shares does seem to have stopped the "panic action" notifications from Samba at least.

    Odroid XU4
    8TB Main Data | 3x3TB Backup Data
    OMV 5.6.2-1
    Docker | Plex | MergerFS | Rsync

  • Hey Folks,


    strange, started receiving similar notifications and checked my system SDcard (32GB)... web search brought me straight here :D


    Disk usage shows "something" dumped ca. 21GB on my SD last night round 3 o'clock.... can't find the supposed data via sftp or ncdu.


    Syslog from that timeframe attached


    omv_syslog_excerpt.txt


    Maybe somebody can read something from that... after that timeframe it's mainly monit entries about rootfs resource limit.


    thx
    ra

    HP ProLiant MicroServer Gen8, Celeron G1610T, 4GB RAM
    OMV 3.0.59


    32GB SDcard System
    2x4TB WD Red Raid1
    1x2TB Seagate

  • Try disabling all services that can write to /media or srv using Omv GUI. Comment all mount entries in fstab using terminal, including nfs binds. Reboot and check those paths again inside.

    Great call on this. After rebooting, I got this result, showing 25 GB in /media:


    Drilling down, I found that SnapRAID had somehow tried to create a parity file in /media/dac85fc4-9594-4bdd-877e-bd111753339c which is the normal path of my parity drive, but it must not have been properly mounted or something. The date on the file shows the date of when I was having to fight with the failed drive and get things sync'd back up. Deleted that file, set things back up in fstab, rebooted, and turned back on Samba and Plex and I'm all good now.


    Thanks subzero79 and everyone else!

    Odroid XU4
    8TB Main Data | 3x3TB Backup Data
    OMV 5.6.2-1
    Docker | Plex | MergerFS | Rsync

  • OK, seems to be a similar issue in my case. A rsnapshot cronjob that normally goes to an internal hdd wound up on the SDcard. Don't really know why just yet, have to research that. Maybe because usb hdd were not mounted?


    thx
    ra

    HP ProLiant MicroServer Gen8, Celeron G1610T, 4GB RAM
    OMV 3.0.59


    32GB SDcard System
    2x4TB WD Red Raid1
    1x2TB Seagate

  • Just wanted to add that I am also facing this problem with OMV3 and mergerfs.


    I don't use USB Drives, they are all SATA


    Sometimes the mergerfs pool gets disconnected and it seems that transmission-bt writes in a directory under /media/mergerfs-id


    After a reboot mergerfs mounts the pool under /media/mergerfs-id again, so that the local directory /media/mergerfs-id with its content is not visible


    A few times I was able to solve the problem by removing the download in transmission gui and trashing its data, prior to reboot.


    But now I am sitting infront of the problem again, only a few MB left. Already forgot how I fixed it last time, googling brought me to this thread.


    And then a strange thing happened.
    I commented out only the mergerfs pool from fstab
    and disabled bittorrent service
    rebooted
    got a call from pissed off wife because she was watching a TV show, this is not the strange thing :P


    navigated to /media/mergerfs-id ... its empty =O
    checked df
    2 GB free space :thumbup:


    I am not sure what happened, but it seems the problem solved itself.


    Well, not it the long run, in the long run I would prefer that mergerfs would clear the local /media/mergerfs-id directory prior to mounting over it, that would rock, because the problem would be solved just be rebooting, not editing fstab manually.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!