BTRFS no space left - kernel 5.4 bug

HannesJo · 20. Februar 2020

I am still playing around with btrfs and encountered some kind of that "no disk space available" behavior. Actually it is slightly different. I am getting no errors or so but since today, omv is simply not able to detect the correct amount of total storage space. Current setup is a simple Stripe of 2 16 TB disks. For the last days and weeks, omv's disk usage graphs was showing correct data but stopped doing so some hours ago.

I checked the output of the btrfs disk usage tools and I think it looks good. Any idea?
Edit: Or is it just the common "no disk space bug" and I misunderstood its behavior? I was assuming the unallocated space is auto allocated when needed..

Code

root@NAS:~# btrfs filesystem df -h /srv/dev-disk-by-label-Storage
Data, RAID0: total=14.28TiB, used=14.28TiB
System, RAID1: total=8.00MiB, used=1.00MiB
Metadata, RAID1: total=16.00GiB, used=15.92GiB
GlobalReserve, single: total=512.00MiB, used=0.00B




root@NAS:~# btrfs fi usage /srv/dev-disk-by-label-Storage
Overall:
    Device size:                  29.11TiB
    Device allocated:             14.31TiB
    Device unallocated:           14.79TiB
    Device missing:                  0.00B
    Used:                         14.31TiB
    Free (estimated):             14.80TiB      (min: 7.40TiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)


Data,RAID0: Size:14.28TiB, Used:14.28TiB
   /dev/sdb        7.14TiB
   /dev/sdc        7.14TiB


Metadata,RAID1: Size:16.00GiB, Used:15.92GiB
   /dev/sdb       16.00GiB
   /dev/sdc       16.00GiB


System,RAID1: Size:8.00MiB, Used:1.00MiB
   /dev/sdb        8.00MiB
   /dev/sdc        8.00MiB


Unallocated:
   /dev/sdb        7.39TiB
   /dev/sdc        7.39TiB

Alles anzeigen

HannesJo · 20. Februar 2020

So it seems to be kind of that known "unusual no space left" bug. Complete rebalance is running since half a day now (about 50% finished) and OMV disk usage statistics shows that:

When rebalancing has finished I'm going to try that "btrfs filesystem resize max" command. I found some topics on other web forums where that helped to trigger btrfs allocating free space. Actually, it should do that automatically.

EDIT:
Did some further investigation and it is actually not the common "no space left bug", but a known kernel 5.4 bug. "zero free space issue". Sounds similar but is different. Workaround is to use kernel 4.x or 5.3. Patch is under Review. Maybe this thread can help anyone who is encountering the same behavior.

patman2097 · 23. Februar 2020

Zitat von HannesJo

So it seems to be kind of that known "unusual no space left" bug. Complete rebalance is running since half a day now (about 50% finished) and OMV disk usage statistics shows that:

When rebalancing has finished I'm going to try that "btrfs filesystem resize max" command. I found some topics on other web forums where that helped to trigger btrfs allocating free space. Actually, it should do that automatically.

EDIT:
Did some further investigation and it is actually not the common "no space left bug", but a known kernel 5.4 bug. "zero free space issue". Sounds similar but is different. Workaround is to use kernel 4.x or 5.3. Patch is under Review. Maybe this thread can help anyone who is encountering the same behavior.

Alles anzeigen

Im experiencing the same issue with no space left. i tried using filters with this command

>>sudo btrfs fi balance start -dusage=5 /sharedfolders/documents

but it didnt seem to do anything, but did release so chunnks of lost data.

so i had to rebalanced the full drive using command

>> btrfs balance start /btrfsdrive

took 2 +days. seemed to have worked but after a days use and a heavy copy, the drive and cache have filled up again...

Im on vesion 5.3.2-1 (usul)

I tried lookign at my performance graphs, strangle they all blank, CPU and drive usage.

Any other suggestions, i dont want to be repeating this balancing command and exhausting wait every 2 days

patman2097 · 23. Februar 2020

Zitat von HannesJo

i did a omv firstaid check and i get this error message , im guessing this is related..

Checking all RRD files. Please wait ...
All RRD database files are valid.

ERROR: Command '['monit', 'start', 'rrdcached']' returned non-zero exit status 1

HannesJo · 24. Februar 2020

@patman2097 are you on Kernel 5.4? If yes, just reboot on kernel 5.3 or 4.x. That worked for me. I cant help about your firstaid check error. Have never used that.

patman2097 · 27. Februar 2020

Zitat von HannesJo

@patman2097 are you on Kernel 5.4? If yes, just reboot on kernel 5.3 or 4.x. That worked for me. I cant help about your firstaid check error. Have never used that.

how do i just reboot on kernel 5.3 or 4.x.?

sorry not sure of the procedure?

thanks

HannesJo · 28. Februar 2020

You must install OMV Extras Plugin. There you find a Kernel TAB. Just click "Install Proxmox Kernel", select a Kernel, set as default, reboot.

patman2097 · 5. März 2020

Worked a charm, Thanks HannesJo for that what a relief!

Cusack · 17. März 2020

Thanks HannesJo

I was about to let go OMV because I couldn't find a solution for this problem.

TechLoic · 21. März 2020

Just created an account to say a BIG THANK YOU to HannesJo!!

I rolled back on kernel 5.3 and now I don't have the problem anymore!!

HannesJo · 2. April 2020

I am very happy to see how helpful this thread is. Thank U guys!

Jetzt mitmachen!