System Disk Full - Newbie

  • Hi everyone,


    I realized I was no longer receiving emails about my rsync backup jobs being completed. I tried to login via webgui and was not successful. I then read every thread I could find on this issue and realized my system disk was full. I was able to ssh into OMV and ran the following commands based on the threads that I had read:


    root@openmediavault:/# df -h
    Filesystem Size Used Avail Use% Mounted on
    rootfs 28G 28G 0 100% /
    udev 10M 0 10M 0% /dev
    tmpfs 388M 6.7M 381M 2% /run
    /dev/disk/by-uuid/99a2d57d-c34c-45d5-9554-3f85d32f4544 28G 28G 0 100% /
    tmpfs 5.0M 0 5.0M 0% /run/loc k
    tmpfs 1.1G 4.0K 1.1G 1% /run/shm
    /dev/md0 4.6T 1.1T 3.5T 24% /media/3 db2568c-7b7c-403d-86cd-0a0bac755ab6
    /dev/sdd1 1.8T 1.1T 755G 59% /media/9 a1a30a5-c078-4248-9a16-abb68afbe40f
    tmpfs 1.1G 8.0K 1.1G 1% /tmp


    As you can see, rootfs is full. I have two hard drives (4.6T) in a raid 1 configuration in the case and a USB hard drive (1.8T) that is always plugged into the case that I am backing up to. I am assuming that there was an error at some point and rsync proceeded to back up to my system drive instead of the USB drive.


    However, I cannot figure out where to go to delete the problem files and free up space.


    Based on the other threads I have run this command:


    root@openmediavault:/# du -a /var | sort -n -r | head -n 10
    541204 /var
    242144 /var/log
    178296 /var/cache
    170044 /var/cache/apt
    128104 /var/cache/apt/archives
    125260 /var/cache/apt/archives/plexmediaserver_0.9.16.6.1993-5089475-debian_amd64.deb
    113028 /var/lib
    75616 /var/lib/apt
    75576 /var/lib/apt/lists
    32132 /var/log/syslog


    However, all of these seem reasonable in size.


    Any ideas on where to go next?


    Thanks!

    • Offizieller Beitrag

    It is in /media under the USB drive's normal mount point. You won't see it while the USB drive is mounted. So, disconnect it, delete the files, reconnect it, and remount.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • You are the best!


    That took care of it. Everything is back up and running. I assume the original issue is that I needed to manually remount the usb drive after power outages.


    That took only about 7 hours of researching linux commands (something that my DOS and Commodore experience did not prepare me for) and reading the message boards before I finally decided to post and you solved it in about a minute.


    Thanks for all your help!

    • Offizieller Beitrag

    I filed a bug report to add a step to verify that the destination drive is mounted before starting an rsync job. I need to add the same check to the rsnapshot plugin as well.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Hi, I seem to have similar issue as BigBry. I've spent about 4 hours troubleshooting, and deleting files until I found this thread. However I'm not sure how to follow ryecoarron's instructions: "It is in /media under the USB drive's normal mount point. You won't see it while the USB drive is mounted. So, disconnect it, delete the files, reconnect it, and remount."


    I shut down my server, and disconnected my external usb drives.


    root@Winnie2:/# df -h
    Filesystem Size Used Avail Use% Mounted on
    rootfs 71G 70G 0 100% /
    udev 10M 0 10M 0% /dev
    tmpfs 202M 784K 201M 1% /run
    /dev/disk/by-uuid/9db4a522-5fc9-41ae-9060-5488f7a73a4d 71G 70G 0 100% /
    tmpfs 5.0M 0 5.0M 0% /run/lock
    tmpfs 1.1G 0 1.1G 0% /run/shm
    /dev/sdb1 294G 4.6G 274G 2% /media/b67ba182-bafb-4170-b08b-eac7fc5e8ae8
    tmpfs 1.1G 0 1.1G 0% /tmp


    root@Winnie2:/# du -a /var | sort -n -r | head -n 20
    527860 /var
    286844 /var/lib
    158032 /var/lib/clamav
    122656 /var/www
    117240 /var/www/owncloud
    106588 /var/lib/clamav/main.cvd
    87068 /var/cache
    76704 /var/lib/apt
    76660 /var/lib/apt/lists
    70956 /var/cache/apt
    63640 /var/www/owncloud/apps
    51000 /var/lib/clamav/daily.cld
    28412 /var/cache/apt/archives
    28008 /var/log
    27816 /var/lib/apt/lists/ftp.us.debian.org_debian_dists_wheezy_main_binary-amd64_Packages
    24936 /var/lib/apt/lists/ftp.us.debian.org_debian_dists_wheezy_main_source_Sources
    23580 /var/www/owncloud/apps/documents
    22940 /var/cache/apt/archives/linux-image-3.2.0-4-amd64_3.2.82-1_amd64.deb
    22176 /var/lib/dpkg
    21888 /var/www/owncloud/core


    Where from here?

    Dell Precision 670: 4 XEON CPUs, dedicated system internal hard drive,one internal storage drive, and two external USB harddrives

    • Offizieller Beitrag

    What is the output of: mount

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Here you go


    root@Winnie2:~# mount
    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=255232,mode=755)
    devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
    tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=205904k,mode=755)
    /dev/disk/by-uuid/9db4a522-5fc9-41ae-9060-5488f7a73a4d on / type ext4 (rw,relatime,errors=remount-ro,user_xattr,barrier=1,data=ordered)
    tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
    tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1054040k)
    /dev/sdb1 on /media/b67ba182-bafb-4170-b08b-eac7fc5e8ae8 type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
    tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,size=1054040k)
    rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)


    Thanks,
    Kevin

    Dell Precision 670: 4 XEON CPUs, dedicated system internal hard drive,one internal storage drive, and two external USB harddrives

    • Offizieller Beitrag

    umount /media/b67ba182-bafb-4170-b08b-eac7fc5e8ae8


    if that doesn't fail then look in that directory. There will be no output if it is successful.


    ls -al /media/b67ba182-bafb-4170-b08b-eac7fc5e8ae8/


    There should be nothing in it. I'm guessing there is a lot of files in it. You should delete those if they are files backed up from a data drive.


    rm -rfv /media/b67ba182-bafb-4170-b08b-eac7fc5e8ae8/*

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Unmount returned no errors
    ls:



    root@Winnie2:~# umount /media/b67ba182-bafb-4170-b08b-eac7fc5e8ae8
    root@Winnie2:~# ls -al /media/b67ba182-bafb-4170-b08b-eac7fc5e8ae8/
    total 8
    drwx------ 2 root root 4096 Mar 4 2016 .
    drwxr-xr-x 10 root root 4096 Mar 7 2016 ..
    root@Winnie2:~#


    I don't understand the output. Should I go ahead and run "rm"?

    Dell Precision 670: 4 XEON CPUs, dedicated system internal hard drive,one internal storage drive, and two external USB harddrives

    • Offizieller Beitrag

    Should I go ahead and run "rm"?

    It wouldn't be removing anything.


    You aren't having the same issue as the OP and your space is being used up somewhere else. What is the output of: du -h -d1 /

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • root@Winnie2:~# du -h -d1 /
    142M /lib
    185M /boot
    518M /var
    7.2M /etc
    16K /lost+found
    du: cannot access `/proc/25183/task/25183/fd/4': No such file or directory
    du: cannot access `/proc/25183/task/25183/fdinfo/4': No such file or directory
    du: cannot access `/proc/25183/fd/4': No such file or directory
    du: cannot access `/proc/25183/fdinfo/4': No such file or directory
    0 /proc
    4.0K /lib64
    4.0K /opt
    780K /run
    4.0K /selinux
    0 /tmp
    48K /root
    0 /sys
    625M /usr
    4.0K /export
    7.1M /bin
    0 /dev
    16K /srv
    20K /home
    69G /media
    12M /sbin
    8.0K /mnt
    70G /
    root@Winnie2:~#

    Dell Precision 670: 4 XEON CPUs, dedicated system internal hard drive,one internal storage drive, and two external USB harddrives

    • Offizieller Beitrag

    What is the output of: ls -alh /media/

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Just a FYI, I have two external USBs disconnected right now.



    root@Winnie2:~# ls -alh /media/
    total 40K
    drwxr-xr-x 10 root root 4.0K Mar 7 2016 .
    drwxr-xr-x 24 root root 4.0K Oct 25 14:40 ..
    drwx------ 3 root root 4.0K Oct 19 03:30 38087397087352BC
    drwx------ 2 root root 4.0K Mar 4 2016 b67ba182-bafb-4170-b08b-eac7fc5e8ae8
    drwx------ 3 root root 4.0K Oct 19 03:30 c25195a0-94a1-487d-8288-ceed1aab4f62
    lrwxrwxrwx 1 root root 6 Mar 4 2016 cdrom -> cdrom0
    drwxr-xr-x 2 root root 4.0K Mar 4 2016 cdrom0
    lrwxrwxrwx 1 root root 7 Mar 4 2016 floppy -> floppy0
    drwxr-xr-x 2 root root 4.0K Mar 4 2016 floppy0
    lrwxrwxrwx 1 root root 4 Mar 4 2016 usb -> usb0
    drwxr-xr-x 2 root root 4.0K Mar 4 2016 usb0
    drwxr-xr-x 2 root root 4.0K Mar 4 2016 usb1
    drwxr-xr-x 2 root root 4.0K Mar 4 2016 usb2
    root@Winnie2:~#

    Dell Precision 670: 4 XEON CPUs, dedicated system internal hard drive,one internal storage drive, and two external USB harddrives

    • Offizieller Beitrag

    I guess the files are either in one of the two unused directories.


    du -h -d1 /media/38087397087352BC/ /media/c25195a0-94a1-487d-8288-ceed1aab4f62/


    I assume since they aren't mounted that files can be removed.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • root@Winnie2:~# du -h -d1 /media/38087397087352BC/ /media/c25195a0-94a1-487d-8288-ceed1aab4f62/
    69G /media/38087397087352BC/WDDServerBackup
    69G /media/38087397087352BC/
    23M /media/c25195a0-94a1-487d-8288-ceed1aab4f62/Users2
    23M /media/c25195a0-94a1-487d-8288-ceed1aab4f62/
    root@Winnie2:~#


    If I follow this correctly, these two directories are on my "system" disk (sorry I'm a DOS/Windows/CPM/TOPS10 guy) which wasn't supposed hold any "media" which is mount point if I understand enough Linux. I've been out of the country and we had a power failure. Occasionally my external USB drives (38087397087352BC and c25195a0-94a1-487d-8288-ceed1aab4f62?) fails to mount after reboot. While I was gone, I stopped getting notification emails and (I just found out) my daughter did a full backup of her laptop. I'm guessing the fail over from the unmounted USBs is to the "system" disk (a meager 80g or so if memory serves).

    Dell Precision 670: 4 XEON CPUs, dedicated system internal hard drive,one internal storage drive, and two external USB harddrives

  • Here's the "dir" for them. Looks like its my backups filling it up. Shouldn't be an issue to delete


    root@Winnie2:/media/c25195a0-94a1-487d-8288-ceed1aab4f62/Users2/WINNIEDATA# cd /media/c25195a0-94a1-487d-8288-ceed1aab4f62/Users2/WINNIEDATA/Users2
    root@Winnie2:/media/c25195a0-94a1-487d-8288-ceed1aab4f62/Users2/WINNIEDATA/Users2# dir
    daily.0 daily.1 daily.2 daily.3 daily.4 daily.5 daily.6



    root@Winnie2:/media/38087397087352BC/WDDServerBackup/Data/Winnine# cd /media/38087397087352BC/WDDServerBackup/Data/Winnine/UsersHome
    root@Winnie2:/media/38087397087352BC/WDDServerBackup/Data/Winnine/UsersHome# dir
    daily.0 daily.1 daily.2 daily.3 daily.4 daily.5

    Dell Precision 670: 4 XEON CPUs, dedicated system internal hard drive,one internal storage drive, and two external USB harddrives

  • Thank you sir, I removed the the subdirectories under Media and freed up enough space to be able to log onto the GUI. :thumbup:

    Dell Precision 670: 4 XEON CPUs, dedicated system internal hard drive,one internal storage drive, and two external USB harddrives

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!