Inexplicable Disk Usage

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • Inexplicable Disk Usage

      Today I went to install updates on my OMV box, and found that my root (/) partition was completely full. It is about 29GB and is "physically" the majority of the space on a 32gb EMMC card in an ODROID XU4. When I run a simple "df -h" from the root folder, I find the following (note that this is after I cleared up a little bit of space on it):

      Filesystem Size Used Avail Use% Mounted on /dev/mmcblk0p2 29G 28G 410M 99% /udev 10M 0 10M 0% /devtmpfs 399M 6.8M 392M 2% /runtmpfs 996M 8.0K 996M 1% /dev/shmtmpfs 5.0M 0 5.0M 0% /run/locktmpfs 996M 0 996M 0% /sys/fs/cgrouptmpfs 996M 8.0K 996M 1% /tmp5a90f259-21b0-4de9-8b75-3b0d173162ce:82342386-dd77-464b-bd8b-ba3ef9c26bb4 5.4T 1.8T 3.6T 34% /export/Share/dev/mmcblk0p1 71M 32M 40M 44% /boot/dev/sda1 2.7T 1.9T 905G 68% /media/dac85fc4-9594-4bdd-877e-bd111753339c/dev/sdb1 2.7T 1.8T 909G 67% /media/5a90f259-21b0-4de9-8b75-3b0d173162ce /dev/sdc1 2.7T 189M 2.7T 1% /media/82342386-dd77-464b-bd8b-ba3ef9c26bb4

      However, when I use a du command (excluding the external storage, returning just the top 25 items, I can only account for about 3GB (even if I include everything, it doesn't go up but about .1 GB), which is about what I'd expect as I don't really use a lot of disk space on that partition for anything.

      root@trunks:/# du --exclude=media --exclude=export -ch -d 10 | sort -hr | head -n25 du: cannot access ‘./proc/17757/task/17757/fd/4’: No such file or directorydu: cannot access ‘./proc/17757/task/17757/fdinfo/4’: No such file or directorydu: cannot access ‘./proc/17757/fd/4’: No such file or directorydu: cannot access ‘./proc/17757/fdinfo/4’: No such file or directory3.0G total3.0G .1.5G ./usr796M ./var728M ./var/lib668M ./root566M ./usr/share551M ./usr/lib523M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server523M ./var/lib/plexmediaserver/Library/Application Support523M ./var/lib/plexmediaserver/Library523M ./var/lib/plexmediaserver366M ./root/go1.5265M ./root/go1.4239M ./usr/lib/arm-linux-gnueabihf201M ./usr/local/go201M ./usr/local193M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata178M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies174M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Media/localhost174M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Media164M ./usr/lib/plexmediaserver163M ./root/go1.5/pkg146M ./root/go1.5/.git/objects/pack146M ./root/go1.5/.git/objectsI wondered a little bit about the lines from the du that indicate it can't access certain files in /proc, so I thought maybe they were hung processes that were holding a bunch of disk space, but I tried rebooting the server and nothing changed. I guess I"m just a little bit stumped and hoping for any suggestions.
      ODROID-XU4
      32GB eMMC Boot Drive | 3x3TB Seagate External Drives
      OMV 3.0.94 | Linux Kernel 4.9.56-odroidxu4
      SnapRAID | MergerFS | Plex Media Server

      The post was edited 2 times, last by jarodmerle ().

    • Here's the command line output in a hopefully more readable format (it didn't want to let me change them to code blocks in "edit" mode of the first post):


      Source Code

      1. Filesystem Size Used Avail Use% Mounted on
      2. /dev/mmcblk0p2 29G 28G 410M 99% /
      3. udev 10M 0 10M 0% /dev
      4. tmpfs 399M 6.8M 392M 2% /run
      5. tmpfs 996M 8.0K 996M 1% /dev/shm
      6. tmpfs 5.0M 0 5.0M 0% /run/lock
      7. tmpfs 996M 0 996M 0% /sys/fs/cgroup
      8. tmpfs 996M 8.0K 996M 1% /tmp
      9. 5a90f259-21b0-4de9-8b75-3b0d173162ce:82342386-dd77-464b-bd8b-ba3ef9c26bb4 5.4T 1.8T 3.6T 34% /export/Share
      10. /dev/mmcblk0p1 71M 32M 40M 44% /boot
      11. /dev/sda1 2.7T 1.9T 905G 68% /media/dac85fc4-9594-4bdd-877e-bd111753339c
      12. /dev/sdb1 2.7T 1.8T 909G 67% /media/5a90f259-21b0-4de9-8b75-3b0d173162ce
      13. /dev/sdc1 2.7T 189M 2.7T 1% /media/82342386-dd77-464b-bd8b-ba3ef9c26bb4
      Display All

      Source Code

      1. root@trunks:/# du --exclude=media --exclude=export -ch -d 10 | sort -hr | head -n25
      2. du: cannot access ‘./proc/17757/task/17757/fd/4’: No such file or directory
      3. du: cannot access ‘./proc/17757/task/17757/fdinfo/4’: No such file or directory
      4. du: cannot access ‘./proc/17757/fd/4’: No such file or directory
      5. du: cannot access ‘./proc/17757/fdinfo/4’: No such file or directory
      6. 3.0G total
      7. 3.0G .
      8. 1.5G ./usr
      9. 796M ./var
      10. 728M ./var/lib
      11. 668M ./root
      12. 566M ./usr/share
      13. 551M ./usr/lib
      14. 523M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server
      15. 523M ./var/lib/plexmediaserver/Library/Application Support
      16. 523M ./var/lib/plexmediaserver/Library
      17. 523M ./var/lib/plexmediaserver
      18. 366M ./root/go1.5
      19. 265M ./root/go1.4
      20. 239M ./usr/lib/arm-linux-gnueabihf
      21. 201M ./usr/local/go
      22. 201M ./usr/local
      23. 193M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata
      24. 178M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/Movies
      25. 174M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Media/localhost
      26. 174M ./var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Media
      27. 164M ./usr/lib/plexmediaserver
      28. 163M ./root/go1.5/pkg
      29. 146M ./root/go1.5/.git/objects/pack
      30. 146M ./root/go1.5/.git/objects
      Display All
      ODROID-XU4
      32GB eMMC Boot Drive | 3x3TB Seagate External Drives
      OMV 3.0.94 | Linux Kernel 4.9.56-odroidxu4
      SnapRAID | MergerFS | Plex Media Server
    • A couple other tidbits I just found. After I had rebooted the server, I received a whole slew of notifications that I guess had gotten hung up, which may speak to some sort of issue that could relate to this. Amongst them were a couple that may also be pertinent:

      Resource limit matched Service fs_media_6f6e68de-1044-4129-b57a-53b8e38a5bf6

      Date: Sat, 15 Jul 2017 15:22:03
      Action: alert
      Host: trunks.trunks
      Description: space usage 98.5% matches resource limit [space usage>85.0%]


      This one is interesting, because the file system it is referring to is, I believe, that of an external drive I had to remove last weekend because it was failing. I had a 4 (now 3) drive SnapRAID array, of which this was a part. When I try to go to the mount point under /media for this, the directory still appears to be there, but I cannot enter it.

      Then, there's this one:

      The Samba 'panic action' script, /usr/share/samba/panic-action,
      was called for PID 18489 (/usr/sbin/smbd).

      This means there was a problem with the program, such as a segfault.
      However, gdb was not found on your system, so the error could not be
      debugged. Please install the gdb package so that debugging information
      is available the next time such a problem occurs.


      I've never seen a notification like this before, and my Samba share appears to still be working without any issue. I guess this might just be erroneous, but I figured it was worth noting.
      ODROID-XU4
      32GB eMMC Boot Drive | 3x3TB Seagate External Drives
      OMV 3.0.94 | Linux Kernel 4.9.56-odroidxu4
      SnapRAID | MergerFS | Plex Media Server
    • Hi you have to look fast that you have more space
      Otherwise Raspi will not start anymore.
      I would install the Plex media server on a vin your hard disks.

      Ps: I was a copy of your make, with Win32DiskImager
      forum-bpi.de Visit on and help us, The Germany Forum :thumbsup:
      github.com/Wolf2000Pi
      Banana Pi /Armbian 3.4.109 / Openmediavault 2.x
      Banana Pi /Armbian 4.11.5-sunxi / Openmediavault 3.0.xx (Test)
      Dell Inspiron One 2205 | Openmediavault 3.0.xx / Kernel 4.7 (Test)

      The post was edited 1 time, last by Wolf2000 ().

    • There might be root data under /media if disk at some point where not mounted.

      Try maybe this

      du -hx --max-depth=1 --exclude=proc /

      x will skip other filesystems

      If /media reports size, i would be suspicious, even if the value is low.
      New wiki
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server
    • Dropkick Murphy wrote:

      Since you have no signature of your used stuff, one can just guess...

      Its a Raspi?
      /dev/mmcblk0p2 is a SDCard or something like that you are booting from?

      Shot in the dark: SDCard worn out and broken...

      Sorry, kind of buried it in my post. Don't post here much (obviously) so never really thought about adding a signature. I'll put some info there now.

      Wolf2000 wrote:

      Hi you have to look fast that you have more space
      Otherwise Raspi will not start anymore.
      I would install the Plex media server on a vin your hard disks.

      Ps: I was a copy of your make, with Win32DiskImager

      It's an ODROID-XU4. Plex is only taking up about 500MB from everything I can tell, so it doesn't seem like it should be an issue. In my original build, I had tried moving the Plex library to a different location and it caused me all sorts of grief.

      subzero79 wrote:

      There might be root data under /media if disk at some point where not mounted.

      Try maybe this

      du -hx --max-depth=1 --exclude=proc /

      x will skip other filesystems

      If /media reports size, i would be suspicious, even if the value is low.

      This only reports 3K of data on /media. Could it actually be more than that?

      Source Code

      1. root@trunks:~# du -hx --max-depth=1 --exclude=proc /
      2. 11M /sbin
      3. 63M /lib
      4. 1.0K /mnt
      5. 12M /opt
      6. 1.5G /usr
      7. 3.0K /srv
      8. 211K /lost+found
      9. 4.3M /etc
      10. 1.7M /home
      11. 1.0K /export
      12. 3.0K /media
      13. 404M /root
      14. 5.8M /bin
      15. 858M /var
      16. 2.8G /
      Display All




      tkaiser wrote:

      Source Code

      1. du -shx /media
      should do the job more quickly?

      This, likewise, reports the same 3K the other take on du did. Good to know about these different versions though.

      Source Code

      1. root@trunks:~# du -shx /media
      2. 3.0K /media

      Wolf2000 wrote:

      Hi tkaiser
      If he has OMV 3 then
      du -shx / srv

      I am running OMV 3 (3.0.84, to be specific). I assume the space between "/" and "srv" in this one was a typo, so here's the output on it too. Just 3K like the others.

      Source Code

      1. root@trunks:~# du -shx /srv
      2. 3.0K /srv
      Thank you to all for all the ideas. I'm going to try drilling down on media and srv and see if I find anything else.
      ODROID-XU4
      32GB eMMC Boot Drive | 3x3TB Seagate External Drives
      OMV 3.0.94 | Linux Kernel 4.9.56-odroidxu4
      SnapRAID | MergerFS | Plex Media Server
    • tkaiser wrote:

      jarodmerle wrote:

      I'm going to try drilling down on media and srv and see if I find anything else.
      Well, according to df your rootfs is running full. So

      Source Code

      1. du -shx --exclude=proc /* /.*
      should report where most disk space is used.
      Right, I agree that it should report where the space is being used, but unfortunately it doesn't. It, again, only accounts for about 3GB out of 29GB that appear to be used, most of which is in /usr and /var. Here's the output of this one, just for completeness sake.


      Source Code

      1. root@trunks:/media# du -shx --exclude=proc /* /.*
      2. 5.8M /bin
      3. 32M /boot
      4. 0 /dev
      5. 4.3M /etc
      6. 1.0K /export
      7. 1.7M /home
      8. 63M /lib
      9. 211K /lost+found
      10. 3.0K /media
      11. 1.0K /mnt
      12. 12M /opt
      13. 404M /root
      14. 6.8M /run
      15. 11M /sbin
      16. 3.0K /srv
      17. 0 /sys
      18. 356K /tmp
      19. 1.5G /usr
      20. 859M /var
      21. 1.0K /.
      Display All
      ODROID-XU4
      32GB eMMC Boot Drive | 3x3TB Seagate External Drives
      OMV 3.0.94 | Linux Kernel 4.9.56-odroidxu4
      SnapRAID | MergerFS | Plex Media Server
    • A couple other things that I've tried based on googling similar problems:

      1. Remounted the device behind my root FS in a different location and repeated some of the same "du" analysis. A few posts online seemed to indicate this might reveal files that were "hidden" in the "/" mountpoint, but they just reported the exact same results.
      2. I had noticed that some references to the failed drive I had to remove were lingering in configuration (/proc/mounts, /etc/fstab) so I jumped through some hoops to get those cleaned up (deleting/recreating shares and rebuilding my MergerFS pool, primarily), thinking that maybe that had something to do with it, but it made no difference. Recreating my SMB shares does seem to have stopped the "panic action" notifications from Samba at least.
      ODROID-XU4
      32GB eMMC Boot Drive | 3x3TB Seagate External Drives
      OMV 3.0.94 | Linux Kernel 4.9.56-odroidxu4
      SnapRAID | MergerFS | Plex Media Server
    • Hey Folks,

      strange, started receiving similar notifications and checked my system SDcard (32GB)... web search brought me straight here :D

      Source Code

      1. root@omv:/# du --exclude=media --exclude=export -ch -d 10 | sort -hr | head -n25
      2. du: Zugriff auf „./proc/19756/task/19756/fd/3“ nicht möglich: Datei oder Verzeichnis nicht gefunden
      3. du: Zugriff auf „./proc/19756/task/19756/fdinfo/3“ nicht möglich: Datei oder Verzeichnis nicht gefunden
      4. du: Zugriff auf „./proc/19756/fd/4“ nicht möglich: Datei oder Verzeichnis nicht gefunden
      5. du: Zugriff auf „./proc/19756/fdinfo/4“ nicht möglich: Datei oder Verzeichnis nicht gefunden
      6. 8,0G insgesamt
      7. 8,0G .
      8. 4,5G ./var
      9. 4,0G ./var/lib
      10. 3,4G ./var/lib/docker
      11. 2,4G ./var/lib/docker/containers/fe949073c429ef24807b0059763d4b36418bb468a4991c5786bdfddc4f0a815d
      12. 2,4G ./var/lib/docker/containers
      13. 1,5G ./root
      14. 1,4G ./root/mydockerbuild
      15. 1,2G ./usr
      16. 821M ./var/lib/docker/overlay2
      17. 676M ./lib
      18. 561M ./usr/share
      19. 502M ./lib/modules
      20. 413M ./var/lib/clamav
      21. 407M ./var/cache
      22. 386M ./usr/lib
      23. 380M ./var/cache/apt
      24. 338M ./var/lib/docker/overlay2/4707eb69b9a36263b5b3f04abb8dc2e836c4a4b28118e94dab023444e4dc6166/merged
      25. 338M ./var/lib/docker/overlay2/4707eb69b9a36263b5b3f04abb8dc2e836c4a4b28118e94dab023444e4dc6166
      26. 329M ./var/cache/apt/archives
      27. 286M ./var/lib/docker/overlay2/4707eb69b9a36263b5b3f04abb8dc2e836c4a4b28118e94dab023444e4dc6166/merged/usr
      28. 285M ./usr/lib/x86_64-linux-gnu
      29. 227M ./usr/bin
      30. 223M ./var/lib/docker/overlay2/4707eb69b9a36263b5b3f04abb8dc2e836c4a4b28118e94dab023444e4dc6166/merged/usr/share
      Display All

      Disk usage shows "something" dumped ca. 21GB on my SD last night round 3 o'clock.... can't find the supposed data via sftp or ncdu.

      Syslog from that timeframe attached

      omv_syslog_excerpt.txt

      Maybe somebody can read something from that... after that timeframe it's mainly monit entries about rootfs resource limit.

      thx
      ra
      HP ProLiant MicroServer Gen8, Celeron G1610T, 4GB RAM
      OMV 3.0.59

      32GB SDcard System
      2x4TB WD Red Raid1
      1x2TB Seagate
    • subzero79 wrote:

      Try disabling all services that can write to /media or srv using Omv GUI. Comment all mount entries in fstab using terminal, including nfs binds. Reboot and check those paths again inside.
      Great call on this. After rebooting, I got this result, showing 25 GB in /media:

      Source Code

      1. root@trunks:~# du -shx --exclude=proc /* /.*
      2. 5.8M /bin
      3. 32M /boot
      4. 0 /dev
      5. 4.3M /etc
      6. 1.0K /export
      7. 1.7M /home
      8. 63M /lib
      9. 211K /lost+found
      10. 25G /media
      11. 1.0K /mnt
      12. 12M /opt
      13. 404M /root
      14. 6.6M /run
      15. 11M /sbin
      16. 4.0K /srv
      17. 0 /sys
      18. 4.0K /tmp
      19. 1.5G /usr
      20. 836M /var
      21. 1.0K /.
      Display All
      Drilling down, I found that SnapRAID had somehow tried to create a parity file in /media/dac85fc4-9594-4bdd-877e-bd111753339c which is the normal path of my parity drive, but it must not have been properly mounted or something. The date on the file shows the date of when I was having to fight with the failed drive and get things sync'd back up. Deleted that file, set things back up in fstab, rebooted, and turned back on Samba and Plex and I'm all good now.

      Thanks subzero79 and everyone else!
      ODROID-XU4
      32GB eMMC Boot Drive | 3x3TB Seagate External Drives
      OMV 3.0.94 | Linux Kernel 4.9.56-odroidxu4
      SnapRAID | MergerFS | Plex Media Server
    • OK, seems to be a similar issue in my case. A rsnapshot cronjob that normally goes to an internal hdd wound up on the SDcard. Don't really know why just yet, have to research that. Maybe because usb hdd were not mounted?

      thx
      ra
      HP ProLiant MicroServer Gen8, Celeron G1610T, 4GB RAM
      OMV 3.0.59

      32GB SDcard System
      2x4TB WD Red Raid1
      1x2TB Seagate