UnionFS: No space left on device but has 2.5TB free

  • Hi,
    I'm having this annoying problem that for random folders on my share I got the following error when creating files: No space left on device
    My unionFS disk (mounted on /srv/ ) has over 2.5 TB left. The strange thing is this happens for random folders.
    Eg:


    /Movies/USA : no problem writing files
    /Movies/EUR : no space left on device


    What could be the problem please?


    kr
    wouter

  • I'm assuming that you're having the same issue, outlined above, as well.
    _________________________________________________


    Here's what (most likely) happened:
    With "Existing Path, most free space" - when you set up a shared folder, the process establishes a folder (or a path) at the root of the merged drive. "Existing path" sets up a single folder on a single drive, in the merged UnionFS drive. (If a second folder is created, we'll say documents as an example, it would go on another drive but that's not relevant to this issue.)


    We'll say the folder you set up is "Videos" and it stores video files (very large). Since the first directive is "existing path", the second directive "most free space" is ignored because the "Videos" path exists. This means every time a video file is saved to the Videos folder, it goes to that single folder on one drive. If you add a lot of content, the drive fills up rapidly.
    _________________________________________________


    On the fix, you're halfway there. But note that the "most free space" policy is only effective from the time it's implemented. UnionFS will not re-balance your storage for you.


    Using the "Videos" folder as an example:
    You'll have to move some of the content under; /dev/sdd1/Videos/ to /dev/sdc1/Videos/ and - or /dev/sdb1/Videos/ , etc.
    (If the Videos folder doesn't exist on the destination drive, create it but make sure the name exactly matches the folder name as it is on /dev/sdd1, cap's and all.)


    You don't have to completely balance files among all drives. You just need to move enough to take your filled drive down to where it has 10 or 15% free space. Thereafter, the "most free space" policy will direct future content to drives with more space.


    _________________________________________________


    Since "most free space" distributes content among all drives in the Union, you might want to give some thought to dedicating 1 of those 4 drives to SNAPRAID.

  • @flmaxey


    Yeah I read that aswell somewhere, is it possible to remove that file?


    "You'll have to move some of the content under; /dev/sdd1/Videos/ to /dev/sdc1/Videos/ and - or /dev/sdb1/Videos/ , etc."
    I removed some files couple of days ago, and it still telling me no space left now when the drive filled up again, I will try again..
    How does one move easily from one drive to another when using unionFS?

  • I removed some files couple of days ago, and it still telling me no space left now when the drive filled up again, I will try again..

    You'll have to move more, maybe 100's of Gb more. Get it under 10 to 15%. Once the drive is green again, in the GUI, you'll be OK.


    How does one move easily from one drive to another when using unionFS?

    The fastest way to move files is with Midnight Commander. See page 57 of this guide (the current version) for how to install MC, what it is and how it works. You could use WinSCP (how to set WinSCP up is in the guide as well) but it uses a temporary folder which is slow.

  • If you move the files to another drive, inside the drive Union and into an identical folder as noted above, you're done. The new policy will start filling the other drives in the union, with files as you add them, going forward.

    around 200GB, then just move it back

    I'm not sure what you mean by, "just move it back". You'd be filling up the already too full drive again. UnionFS doesn't care which drive the files are on as long as the folder structure is identical. ((**Also note that you DO NOT want two copies of the same files on two different drives, in the same Union.**)


    For example;


    /dev/sdb/Videos/file1
    /dev/sdc/Videos/file2
    /dev/sdd/Videos/file3


    equates to (as you'll see it):


    /Uniondrive/Videos/file1, file2, file3


    Where the files are, on the individual drives, doesn't matter because UnionFS merges them together.
    ___________________________________________________


    With the new policy in place , you could move files out of the NAS, and onto a client, then move them back again. In that case, with the original files on the NAS deleted, when files are moved back the new policy would distribute them to drives with the "most free space".
    In my opinion, this is somewhat risky. Massive file copies, done with Windows explorer is prone to errors. If you do it this way, move small chunks of 5 or 10GB at a time.

  • Have you seen the FAQ entry on https://github.com/trapexit/mergerfs ?


    Why do I get an "out of space" / "no space left on device" / ENOSPC error even though there appears to be lots of space available?

    --
    Google is your friend and Bob's your uncle!


    RAID - Its ability to disappoint is inversely proportional to the user's understanding of it.


    OMV 5.x on ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.

  • @flmaxey


    Im not sure I understand this issue...


    This is my setup, sharedfolder/media/incoming/all fetched
    which is moved upon completion to, sharedfolder/media/completed


    I did set up a new sharedfolder just for this case. sharedfolder/test/ and moved 700gb ish to shared so my unionFS filesystem would be "green" then raised min allowed disk to 415GB and restarted OMV (just to test) Then tried to copy sharedfolder/test back to sharedfolder/media/completed and got the same message "No space left"



    Have you seen the FAQ entry on https://github.com/trapexit/mergerfs ?


    Why do I get an "out of space" / "no space left on device" / ENOSPC error even though there appears to be lots of space available?

    This is my options in UnionFS defaults,allow_other,use_ino,moveonenospc=true Are those not correct?

  • This is my setup, sharedfolder/media/incoming/all fetched
    which is moved upon completion to, sharedfolder/media/completed

    I don't want to add any confusion as this is way I want to go, but if I have read @flmaxey explanation correctly the issue is the policy;


    Existing path -> Video that is why one drive is filling up, irrespective of any sub folders, what he has suggested is to change the policy to Most free space, move 200GB+ off of the UnionFS to an external drive, that's move not copy. Then move it back to UnionFS and it should sort itself out.

  • moving a file back to /srv/123123-123-123123/media from /sharedfolders/test


    and it's super slow, tried smaller files aswell but it's the same speed. can't get over 1mb/s when transfering.
    I've no idea why, someone experienced this also?

  • Since I tried to this above my OMV got really slow.
    Cannot access any docker containers, they time out but they're online in the docker tabs, is there any abnormal in my syslog from the post above?

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!