Server Malfunction - No Usable Temporary Directory - No Space Left On Device

1st Official Post

    After being out of town for about a week, I came back to find my OMV server non-responsive (login prompt for Web GUI eventually just timed out). I tried omv-firstaid initially, but received the following error:

    From there, I decided to try rebooting to see if that made any difference, but of course it did not; in fact it made matters worse (Web GUI just results in a "Bad Gateway" error and none of the services I use are running).


    Looking into this error (specifically the last line about no usable temporary directory) lead me to posts suggesting that the disk could be full or permissions could be set to read-only, but I'm not seeing either of those as an issue in my case. Below are a few things I've checked to see if anything stood out, but I'm out of ideas. Hoping someone might be able to give me some next things to try to get back up and running.


    Here's the output of "df -h" showing that nothing (save one of my backup data drives) is close to full:

    Here's an "ls -ls" showing permissions should allow for write access for /tmp and /var/tmp (I don't have a /usr/tmp):



    When I try to create a file in /root, I get an error, but when I try to create a file in /home I do not, despite both being on the same physical device and both showing as having available space:


    Code
    root@gotenks:/# touch /root/tempfile
touch: cannot touch '/root/tempfile': No space left on device
root@gotenks:/# touch /home/tempfile
root@gotenks:/# cd /home
root@gotenks:/home# ls -ls
total 20
 4 drwxr-xr-x 3 jarod users  4096 Aug 10 11:04 jarod
16 drwx------ 2 root  root  16384 Apr 12 10:47 lost+found
 0 -rw-r--r-- 1 root  root      0 Aug 10 11:06 tempfile

    Odroid XU4
    8TB Main Data | 3x3TB Backup Data
    OMV 5.6.2-1
    Docker | Plex | MergerFS | Rsync

    Ran out of space in the initial post, but was also going to include the output of "mount" showing that none of the filesystems in question are mounting as read-only (though maybe something is failing and actually mounting as read-only, not sure how to check that):


    Odroid XU4
    8TB Main Data | 3x3TB Backup Data
    OMV 5.6.2-1
    Docker | Plex | MergerFS | Rsync

    Looking at syslog, it looks like this probably started a few days ago in the middle of the night. Here's the relevant snippet if this helps at all:


    Odroid XU4
    8TB Main Data | 3x3TB Backup Data
    OMV 5.6.2-1
    Docker | Plex | MergerFS | Rsync

    After doing a bit more digging with du, I think this has (had, hopefully) something to do with Docker. When I would run "du -shx...." at the root level, it would get hung up on /var, so I went into /var and repeated, where it got hung up on /var/lib, and so on until it lead me to the folder /var/lib/docker/overlay2, and one specific GUID-named sub-folder thereof. So I got Docker started up, deleted some things I didn't need anymore, and lo-and-behold things started working again. I don't know if Docker may be taking up disk space that isn't getting reported or what, but I finally let the du command run until it returned and got the results below pointing to a Docker folder that it seems to think exists, but doesn't actually exist as far as I can tell, so maybe that's somehow related.


    For the moment at least though, I'm up and running again. If anyone has ideas on this, let me know.


    Odroid XU4
    8TB Main Data | 3x3TB Backup Data
    OMV 5.6.2-1
    Docker | Plex | MergerFS | Rsync

    Because of no other replies I try to give some hints which you could check (I didn´t read your posts in-depth):

    • If you use the Plex plugin check the free space of your system drive. The Plex database is located on the system drive by default (/var/lib/plexmediaserver).
    • Try to free some space on /dev/sdd1. Some file systems get problems if the disk space is utilized almost completely.

    Maybe this link could be helpfull too: Missing space on boot drive... Gelöst OMV 4.x

    OMV 3.0.100 (Gray style)

    ASRock Rack C2550D4I C0-stepping - 16GB ECC - 6x WD RED 3TB (ZFS 2x3 Striped RaidZ1) - Fractal Design Node 304 -

    3x WD80EMAZ Snapraid / MergerFS-pool via eSATA - 4-Bay ICYCube MB561U3S-4S with fan-mod

    Edited once, last by cabrio_leo ().

    Thank you both for the suggestions. Both of them do sound similar to my problem, with the exception that in my case, it's like the disk space is being used but not actually reported by du or df; it's like it's hidden somehow. I had a similar problem to this last summer that was due to the combination of a failed drive in a SnapRAID array and the use of MergerFS, but nothing like that should be going on now. The next time I pull a new version for my Plex server, I may go ahead and move the Docker home directory to my storage devices though; either that or extend my root partition to use more than 32GB of the 120GB SSD (not sure why I made /home as big as I did, just wasn't thinking).

    Odroid XU4
    8TB Main Data | 3x3TB Backup Data
    OMV 5.6.2-1
    Docker | Plex | MergerFS | Rsync

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login