System Disk is full

Spectre216 · 9. Dezember 2019

So I have OMV installed on a 128gb SSD. It is full and I can't figure out why. Tried clearing logs, but that was only 25mb. I tried 'apt autoremove' and 'apt clean' and 'du -sh /'. No luck figuring out what is filling the drive. I cleared an old kernel (only 3 of them were installed). Anyone here have any ideas on where I can look?

The result from du -sh /* in case it helps.

Code

root@OMV:~# sudo du -sh /*


0 /bin


109M /boot


0 /dev


6.7M /etc


11G /export


4.0K /home


0 /initrd.img


0 /initrd.img.old


0 /lib


0 /lib32


0 /lib64


0 /libx32


16K /lost+found


8.0K /media


4.0K /mnt


4.0K /opt


du: cannot read directory '/proc/583/task/583/net': Invalid argument


du: cannot read directory '/proc/583/net': Invalid argument


du: cannot read directory '/proc/584/task/584/net': Invalid argument


du: cannot read directory '/proc/584/net': Invalid argument


du: cannot read directory '/proc/585/task/585/net': Invalid argument


du: cannot read directory '/proc/585/net': Invalid argument


du: cannot read directory '/proc/586/task/586/net': Invalid argument


du: cannot read directory '/proc/586/net': Invalid argument


du: cannot read directory '/proc/587/task/587/net': Invalid argument


du: cannot read directory '/proc/587/net': Invalid argument


du: cannot read directory '/proc/588/task/588/net': Invalid argument


du: cannot read directory '/proc/588/net': Invalid argument


du: cannot access '/proc/739/task/739/fd/3': No such file or directory


du: cannot access '/proc/739/task/739/fdinfo/3': No such file or directory


du: cannot access '/proc/739/fd/3': No such file or directory


du: cannot access '/proc/739/fdinfo/3': No such file or directory


0 /proc


60K /root


18M /run


0 /sbin


7.8T /sharedfolders


1.7M /srv


0 /sys


232K /tmp


1.6G /usr


258M /var


0 /vmlinuz


0 /vmlinuz.old

Alles anzeigen

Any help would be greatly appreciated.

votdev · 9. Dezember 2019

Normally this is caused by a not available filesystem (incl. shared folders) which is the target of a backup/rsync/whatever job. So you might concentrate on looking into the /srv directory where all filesystems are mounted. If a filesystem is not mounted and a rsync job or something else is writing to a shared folder the data is written into thr root filesystem instead.

Spectre216 · 9. Dezember 2019

Zitat von votdev

Normally this is caused by a not available filesystem (incl. shared folders) which is the target of a backup/rsync/whatever job. So you might concentrate on looking into the /srv directory where all filesystems are mounted. If a filesystem is not mounted and a rsync job or something else is writing to a shared folder the data is written into thr root filesystem instead.

I don't think I have RSync enabled. Is that something that is on by default?

Adoby · 9. Dezember 2019

And to make matters worse. The junk files may be hidden...

If some job (or you), at one time, wrote a bunch of backup files, or whatever, to a mount point that wasn't mounted, it is possible that the mount point in question now is mounted. then the mounted device "mask" the local filesystem there. You can only see the mounted filesystem, not the local filesystem.

So du can't report correct disk usage.

One way to find out is to boot from other media (a thumb drive?) and examine the full root filesystem when it is not active. Then no mount points are "masked" and du can be expected to show correct local disk usage.

Another common cause for full root fs is that /var/lib is filled with metadata and other stuff from plex/emby or docker. But that seems not to be the case here.

ryecoaaron · 9. Dezember 2019

Your du command should have to the -x flag to not traverse other filesystems. That way, you will still see usage in /srv even in filesystems are mounted on the subdirectories.

sudo du -x -d1 -h / | sort -h

macom · 12. Dezember 2019

Zitat von ryecoaaron

That way, you will still see usage in /srv even in filesystems are mounted on the subdirectories.

I think you do not see usage of other filesystems mounted in /srv (or somewhere else).
But you will see usage in /srv of folders that are not on other filesystems.

ryecoaaron · 12. Dezember 2019

Zitat von macom

I think you do not see usage of other filesystems mounted in /srv (or somewhere else).
But you will see usage in /srv of folders that are not on other filesystems.

Correct but that was the idea. /srv/ shouldn't be large because it just has mountpoints for other filesystems or bind mounts in it. If it is large, you probably had something write to a sub-directory while the sub-directory wasn't mounted.

rk_dd · 12. Dezember 2019

do you use docker?

tannaroo · 19. Dezember 2019

I also have thio problem today as my system disk is full (using urbackup but files are being saved on another hard disk).

So after I enter: sudo du -sh /*

I get 71G in the /srv. Reading this forum I think I understand that for some reason my sharedfolder was down so it began to write the backup into the /srv folder?

if so, is it safe to delete everything within srv folder relating to non system stuff?

tannaroo · 19. Dezember 2019

This is the output from df -h, if someone can kindly point me in the right direction to what to delete (and how)?

Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 388M 7.2M 381M 2% /run
/dev/sda1 31G 31G 0 100% /
tmpfs 1.9G 40K 1.9G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
tmpfs 1.9G 0 1.9G 0% /tmp
folder2ram 1.9G 63M 1.9G 4% /var/log
/dev/sda3 39G 48M 37G 1% /srv/dev-disk-by-id-ata-ST980813AS_5NH1G0Y G-part3
folder2ram 1.9G 0 1.9G 0% /var/tmp
folder2ram 1.9G 852K 1.9G 1% /var/lib/openmediavault/rrd
folder2ram 1.9G 20K 1.9G 1% /var/spool
folder2ram 1.9G 13M 1.9G 1% /var/lib/rrdcached
folder2ram 1.9G 4.0K 1.9G 1% /var/lib/monit
folder2ram 1.9G 0 1.9G 0% /var/lib/php
folder2ram 1.9G 1.3M 1.9G 1% /var/cache/samba
overlay 31G 31G 0 100% /var/lib/docker/overlay2/c27b39a4b79b6318a ca7497baedd2bedb3b0880d87dfb982eab05dd412e2b9f3/merged
overlay 31G 31G 0 100% /var/lib/docker/overlay2/4270ed88df78570c8 d71b8a14b7cb4e8b6dedc4954153c0784375b64d14ba47a/merged
shm 64M 0 64M 0% /var/lib/docker/containers/d624c1fdcbbdb20 7c8db33c154f799ca65bac4d05c83890e86d0664e97ee81b3/mounts/shm

Adoby · 19. Dezember 2019

df is not interesting. You know there is 0% free. That is true for all of sda1.

du is more interesting.

Try:

sudo du -x --max-depth=1 -h /

Then you will get a list of all the folders directly in / and how much space they take up.

Run the command again but replace / with the biggest folder found. Then you will find the problem.

Feel free to post a list of the biggest 3-4 folders you find.

However, see my earlier post. The problem may be hidden...

Edit: It seems some docker image is very big. I assume that you will find it under /var/lib/docker.

You should NOT use /var/lib/docker for docker images, if /var/lib/docker is on the rootfs.

There are many threads on this forum about /var/lib/docker and docker base path and how to fix it. I would reinstall from scratch and set docker base path before I add any docker images. Use a shared folder on a storage drive. Again, there are many threads here about this.

tannaroo · 19. Dezember 2019

Zitat von Adoby

df is not interesting. You know there is 0% free. That is true for all of sda1.

du is more interesting.

Try:

sudo du -x --max-depth=1 -h /

Then you will get a list of all the folders directly in / and how much space they take up.

Run the command again but replace / with the biggest folder found. Then you will find the problem.

Feel free to post a list of the biggest 3-4 folders you find.

However, see my earlier post. The problem may be hidden...

Edit: It seems some docker image is very big. I assume that you will find it under /var/lib/docker.

You should NOT use /var/lib/docker for docker images, if /var/lib/docker is on the rootfs.

There are many threads on this forum about /var/lib/docker and docker base path and how to fix it. I would reinstall from scratch and set docker base path before I add any docker images. Use a shared folder on a storage drive. Again, there are many threads here about this.

Alles anzeigen

thanks adoby. this is how far I get its all in /srv i think but I cant see it. i beginning to think i need to start all over again.

root@omv:~# sudo du -x --max-depth=1 -h /srv
8.0K /srv/ftp
6.0G /srv/dev-disk-by-label-Panda
32K /srv/pillar
18G /srv/dev-disk-by-label-Elephant
48K /srv/dev-disk-by-label-Vulture
775M /srv/dev-disk-by-id-ada-ST980813AS_5NH1G0YG-part3
1.6M /srv/salt
24G /srv

EDIT: Not sure I see anything in /var/lib/docker - it totals about 4.5G and there is plenty of room on the system disk but your point about having it off the system disk is noted!

macom · 19. Dezember 2019

Read post 5, 6 and 7.
Seems like Panda and Elephant were not mounted/availablle when you run a backup job or something.

tannaroo · 19. Dezember 2019

thanks macom. i did read posts 5,6,7 but I still dont understand how to delete whats in /srv - it has 24GB when it shouldn't have anything large?

macom · 19. Dezember 2019

Zitat von tannaroo

it has 24GB when it shouldn't have anything large?

Correct. Are you sure that the two drives/filesystems are not connected. Just to make sure we are not deleting something wrong. Otherwise you have to restore from your backup

tannaroo · 19. Dezember 2019

Not sure I follow but in any event I can't log into OMV control panel because its full. Get this error message:

Error #0:
OMV\Exception: Failed to read file '/var/cache/openmediavault/cache.omv\controlpanel\login_js.json' (size=0). in /usr/share/php/openmediavault/json/file.inc:207
Stack trace:
#0 /usr/share/php/openmediavault/json/file.inc(223): OMV\Json\File->getContents()
#1 /usr/share/php/openmediavault/controlpanel/controlpanelabstract.inc(173): OMV\Json\File->read()
#2 /var/www/openmediavault/index.php(46): OMV\ControlPanel\ControlPanelAbstract->render()
#3 {main}

I don't need the backup on Elephant (which is completely separate from system disk) as I'm only testing urbackup with that drive. I just want to get back into OMV control panel, so does this mean a complete reinstall of OMV?

macom · 19. Dezember 2019

Is Elephant currently connected to your server? From the output above it looks like it is not connected. What I do not understand is the information of G-part3

If you are sure you don't need the data on it, you can delete it with

rm -r /srv/dev-disk-by-label-Elephant/*

rm: remove
-r: recursive
the rest is the path to delete

ian6 · 19. Dezember 2019

tannaroo, as I mentioned in the other urbackup Newbie thread, I had this same login problem. I took the following steps, based on the suggestions above to solve this. there may be easier ways of doing this, but this worked for me.

You need to boot from a different linux disk, (I booted from a linux install on a usb stick), and then mount the systemdisk, for me the original /dev/sda2 . I was then able to look around on my disk using du -h -x -d1 as suggested above. This allowed me to discover a directory in one of my sharedfolders mount points which had a lot of data, 127GB, that should not have been there. Because you do not have your other share disks mounted, the mount points in sharedfolders should have no files. When the disk was running normally those files were there taking up space, but hidden by the mounted share. I'm currently moving the files somewhere else and can see my original system disk getting less full as I move them.

tannaroo · 19. Dezember 2019

No I disconnected Elephant from the server but followed that command and deleted it. The directory /srv is now showing 6.8GB rather than 24GB, so its worked - I think?

Regarding G-Part 3 - its a 60gb drive of which one is a partition of 30gb (the other partition being OMV system for 30gb). I wanted to use a partition for things like app data etc.

So i've cleared my cache but still cant get into OMV portal - still showing same error ?

tannaroo · 19. Dezember 2019

Zitat von ian6

tannaroo, as I mentioned in the other urbackup Newbie thread, I had this same login problem. I took the following steps, based on the suggestions above to solve this. there may be easier ways of doing this, but this worked for me.

You need to boot from a different linux disk, (I booted from a linux install on a usb stick), and then mount the systemdisk, for me the original /dev/sda2 . I was then able to look around on my disk using du -h -x -d1 as suggested above. This allowed me to discover a directory in one of my sharedfolders mount points which had a lot of data, 127GB, that should not have been there. Because you do not have your other share disks mounted, the mount points in sharedfolders should have no files. When the disk was running normally those files were there taking up space, but hidden by the mounted share. I'm currently moving the files somewhere else and can see my original system disk getting less full as I move them.

thanks ian6 per my other post. i just deleted the whole drive and now my system disk seems to be back under 10gb. but rebooted etc but still can't get into the OMV portal.

Jetzt mitmachen!