Weird situation - root filesystem filled - please help!

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • Weird situation - root filesystem filled - please help!

      Folks,
      I am on Stoneburner for my main media system. Long story of why I cannot yet upgrade this one including need for TFTP server, iSCSI targets etc.

      Anyway, this system has been going great and has been rock solid for quite a number of years. However, within the last few months the root filesystem has become 100% filled and I cannot seem be able to locate what is causing it.

      I run Docker with three containers including MythTV and Emby and false media plugin. It has NFS exports also.

      Anyway, as mentioned earlier, it has been running all this great for quite some time but it appears something (perhaps a package update) may have caused this.

      If you can help suggest what I can try next, please let me know.

      Thank you.




      Source Code

      1. Filesystem Size Used Avail Use% Mounted on
      2. rootfs 7.8G 7.8G 0 100% /
      3. udev 10M 0 10M 0% /dev
      4. tmpfs 801M 504K 800M 1% /run
      5. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /
      6. tmpfs 5.0M 0 5.0M 0% /run/lock
      7. tmpfs 1.6G 0 1.6G 0% /run/shm
      8. /dev/sdb1 1.9T 1022G 841G 55% /media/9d6
      9. /dev/sdc1 1.9T 1021G 842G 55% /media/004
      10. /dev/sda2 103G 5.1G 92G 6% /media/27b
      11. /dev/sde1 298G 228G 71G 77% /media/665
      12. 9d6b229c-e21f 3.7T 2.0T 1.7T 55% /media/22c8
      13. /dev/sda2 103G 5.1G 92G 6% /var/lib/docker/openmediavault
      14. 9d6b229c-e21f 3.7T 2.0T 1.7T 55% /export/MythTV
      15. 9d6b229c-e21f 3.7T 2.0T 1.7T 55% /export/MythTVRecordings
      16. /dev/sde1 298G 228G 71G 77% /export/EXTERNAL-BACKUPS
      17. tmpfs 1.6G 0 1.6G 0% /tmp
      18. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/lib/php5
      19. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/lib/openmediavault/rrd
      20. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/lib/monit
      21. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/lib/rrdcached
      22. folder2ram 4.0G 964K 4.0G 1% /var/lib/openmediavault/rrd
      23. folder2ram 4.0G 25M 3.9G 1% /var/lib/rrdcached
      24. folder2ram 4.0G 8.0K 4.0G 1% /var/lib/monit
      25. folder2ram 4.0G 0 4.0G 0% /var/lib/php5
      26. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/tmp
      27. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/spool
      28. folder2ram 4.0G 0 4.0G 0% /var/tmp
      29. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/log
      30. folder2ram 4.0G 1.6M 4.0G 1% /var/spool
      31. folder2ram 4.0G 24M 3.9G 1% /var/log
      32. cgroup 4.0G 0 4.0G 0% /sys/fs/cgroup
      33. /dev/sda2 103G 5.1G 92G 6% /var/lib/docker/openmediavault/aufs
      34. 192.168.90.9:/ 90G 1.9G 83G 3% /media/27bd/omv-media-1
      Display All

      The post was edited 1 time, last by 2devnull ().

    • gderf wrote:

      Login via console or ssh and cd /

      Then as root run: du -sh *

      Examine the directory sizes to see which one got filled up.
      This is what I am getting. It doesn't really help me pinpoint the problematic files:

      Source Code

      1. 0 backuppc
      2. 7.2M bin
      3. 285M boot
      4. 4.0K config
      5. 0 dev
      6. 7.8M etc
      7. 2.9T export
      8. 4.0K home
      9. 0 initrd.img
      10. 0 initrd.img.old
      11. 427M lib
      12. 4.0K lib64
      13. 0 lock
      14. 16K lost+found
      15. 16K mnt
      16. 463M opt
      17. 4.0K partial
      18. 23M pkgcache.bin
      19. du: cannot access `proc/21308/task/21308/fd/4': No such file or directory
      20. du: cannot access `proc/21308/task/21308/fdinfo/4': No such file or directory
      21. du: cannot access `proc/21308/fd/4': No such file or directory
      22. du: cannot access `proc/21308/fdinfo/4': No such file or directory
      23. 0 proc
      24. 4.6M root
      25. 504K run
      26. 12M sbin
      27. 4.0K selinux
      28. 23M srcpkgcache.bin
      29. 16K srv
      30. 0 ssdstorage
      31. 12M ssdstorageOLD
      32. 0 storage
      33. 37M storageOLD
      34. 0 sys
      35. 0 tmp
      36. 927M usr
      37. 2.8G var
      38. 0 vmlinuz
      39. 0 vmlinuz.old
      Display All
    • Try unmounting each of your storage disks and then look in their mount point directories.

      These look suspicious:

      Source Code

      1. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/tmp
      2. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/spool
      OMV 4.x - ASRock Rack C2550D4I - 16GB ECC - Silverstone DS380
    • gderf wrote:

      Try unmounting each of your storage disks and then look in their mount point directories.

      These look suspicious:

      Source Code

      1. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/tmp
      2. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/spool
      I did the one for Docker and it didn't show anything different after unmounting that filesystem. I too am beginning to suspect something is up with the flash memory plug in (those that show with folder2ram).
    • 2devnull wrote:

      gderf wrote:

      Try unmounting each of your storage disks and then look in their mount point directories.

      These look suspicious:

      Source Code

      1. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/tmp
      2. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/spool
      I did the one for Docker and it didn't show anything different after unmounting that filesystem. I too am beginning to suspect something is up with the flash memory plug in (those that show with folder2ram).
      Did you unmount /dev/disk/by-uuid/bcb and look there or not?
      OMV 4.x - ASRock Rack C2550D4I - 16GB ECC - Silverstone DS380
    • gderf wrote:

      2devnull wrote:

      gderf wrote:

      Try unmounting each of your storage disks and then look in their mount point directories.

      These look suspicious:

      Source Code

      1. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/tmp
      2. /dev/disk/by-uuid/bcb 7.8G 7.8G 0 100% /var/folder2ram/var/spool
      I did the one for Docker and it didn't show anything different after unmounting that filesystem. I too am beginning to suspect something is up with the flash memory plug in (those that show with folder2ram).
      Did you unmount /dev/disk/by-uuid/bcb and look there or not?
      that is the sda1 disk. I think you meant to stop folder2ram?
    • Source Code

      1. root@OMV-2:~# mount
      2. sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
      3. proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
      4. udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=1013532,mode=755)
      5. devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
      6. tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=819520k,mode=755)
      7. /dev/disk/by-uuid/bcb on / type ext4 (rw,noatime,nodiratime,discard,errors=remount-ro,data=ordered)
      8. tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
      9. tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1639020k)
      10. fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
      11. /dev/sdb1 on /media/9d6 type xfs (rw,noexec,relatime,attr2,inode64,usrquota,grpquota,_netdev)
      12. /dev/sdc1 on /media/004 type xfs (rw,noexec,relatime,attr2,inode64,usrquota,grpquota,_netdev)
      13. /dev/sda2 on /media/27b type ext4 (rw,noexec,relatime,discard,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
      14. /dev/sde1 on /media/665 type xfs (rw,noexec,relatime,attr2,inode64,usrquota,grpquota,_netdev)
      15. 9d6 on /media/22c
      16. type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,allow_other)
      17. /dev/sda2 on /var/lib/docker/openmediavault type ext4 (rw,relatime,discard,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group)
      18. 9d6 on /export/MythTV type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,allow_other)
      19. 9d6 on /export/MythTVRecordings type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,allow_other)
      20. /dev/sde1 on /export/EXTERNAL-BACKUPS type xfs (rw,noexec,relatime,attr2,inode64,usrquota,grpquota)
      21. tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,size=1639020k)
      22. rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
      23. nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
      24. cgroup on /sys/fs/cgroup type tmpfs (rw,relatime,mode=755)
      25. cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)
      26. cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)
      27. cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)
      28. cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
      29. cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)
      30. cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,relatime,net_cls)
      31. cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)
      32. cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)
      33. cgroup on /sys/fs/cgroup/net_prio type cgroup (rw,relatime,net_prio)
      34. /dev/sda2 on /var/lib/docker/openmediavault/aufs type ext4 (rw,relatime,discard,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group)
      35. none on /var/lib/docker/openmediavault/aufs/mnt/f98d67526affaafc8e87a976b82df05e91ee66bb141ba552fb487528f66eaec2 type aufs (rw,relatime,si=5f27b6765f88ba2b,dio,dirperm1)
      36. shm on /var/lib/docker/openmediavault/containers/17f62ae078464f393d52d2025f67a0648af12e75032b9e54c165fb8915e8f24f/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
      37. proc on /run/docker/netns/16322bf7a8ce type proc (rw,nosuid,nodev,noexec,relatime)
      38. none on /var/lib/docker/openmediavault/aufs/mnt/4ae357b7268758f7003fa1e6390127e0117a99b5786b83c85847d1c0c181fa6f type aufs (rw,relatime,si=5f27b674932e7a2b,dio,dirperm1)
      39. proc on /run/docker/netns/default type proc (rw,nosuid,nodev,noexec,relatime)
      40. shm on /var/lib/docker/openmediavault/containers/398b22367a53b17ae16da2780dac331077681616afe82337e10d2a257322e540/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
      41. none on /var/lib/docker/openmediavault/aufs/mnt/0cce45bd495e2ef1108dbb3492b95d70f8a107c27fa6a53a8ad16b60d10ef60b type aufs (rw,relatime,si=5f27b67493f9ea2b,dio,dirperm1)
      42. shm on /var/lib/docker/openmediavault/containers/052fe1351199977189a20d5ba5baa8489aa34614e25b882a6d02b36ebe552f7c/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
      43. 192.168.90.9:/ on /media/27b/omv-1 type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.90.10,local_lock=none,addr=192.168.90.9)
      44. /dev/disk/by-uuid/bcb on /var/folder2ram/var/log type ext4 (rw,noatime,nodiratime,discard,errors=remount-ro,data=ordered)
      45. folder2ram on /var/log type tmpfs (rw,relatime)
      46. /dev/disk/by-uuid/bcb on /var/folder2ram/var/tmp type ext4 (rw,noatime,nodiratime,discard,errors=remount-ro,data=ordered)
      47. folder2ram on /var/tmp type tmpfs (rw,relatime)
      48. /dev/disk/by-uuid/bcb on /var/folder2ram/var/lib/openmediavault/rrd type ext4 (rw,noatime,nodiratime,discard,errors=remount-ro,data=ordered)
      49. folder2ram on /var/lib/openmediavault/rrd type tmpfs (rw,relatime)
      50. /dev/disk/by-uuid/bcb on /var/folder2ram/var/spool type ext4 (rw,noatime,nodiratime,discard,errors=remount-ro,data=ordered)
      51. folder2ram on /var/spool type tmpfs (rw,relatime)
      52. /dev/disk/by-uuid/bcb on /var/folder2ram/var/lib/rrdcached type ext4 (rw,noatime,nodiratime,discard,errors=remount-ro,data=ordered)
      53. folder2ram on /var/lib/rrdcached type tmpfs (rw,relatime)
      54. /dev/disk/by-uuid/bcb on /var/folder2ram/var/lib/monit type ext4 (rw,noatime,nodiratime,discard,errors=remount-ro,data=ordered)
      55. folder2ram on /var/lib/monit type tmpfs (rw,relatime)
      56. /dev/disk/by-uuid/bcb on /var/folder2ram/var/lib/php5 type ext4 (rw,noatime,nodiratime,discard,errors=remount-ro,data=ordered)
      Display All
    • Wanted to update this. It appears omv-mkconf backup backup had made a backup underneath the mount point. Not sure how and and why it did that or was allowed to write to the root area instead of the mount point of sde1. Anyway, it is solved now. Will keep a eye on it. Perhaps sde1 got unmounted at some point and when omv-mkconf backup backup ran it wrote to the root disk. Bazaar!
    • 2devnull wrote:

      Perhaps sde1 got unmounted at some point and when omv-mkconf backup backup ran it wrote to the root disk.
      That is definitely what happened. The backup plugin doesn't do anything that would cause it to write to the root disk.
      omv 4.1.15 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!