Error 28 - No space left on device

    • OMV 4.x
    • Error 28 - No space left on device

      Hello,

      This is on an Odroid HC2, root drive is a 32GB micro SD card, with a 4TB SATA data drive connected.

      When trying to install some extra plugins (shellinabox, for example) I'm getting a "No space left" error (28).

      I'm not sure what or how the root partition is getting used up. I'm including screenshots showing the storage disks, and partitions. The data drive is LUKS encrypted.






      Below is what I get with the du command on the root folder.

      Ideas...?

      Thanks!

      Source Code

      1. root@odroidhc2:/# du -hx --max-depth=1 /
      2. 9.5M /bin
      3. 11M /etc
      4. 0 /export
      5. 0 /home
      6. 107M /lib
      7. 0 /media
      8. 0 /mnt
      9. 0 /opt
      10. 56K /root
      11. 12M /sbin
      12. 0 /selinux
      13. 0 /sharedfolders
      14. 4.0K /srv
      15. 877M /usr
      16. 228M /var
      17. 1.3G /
      Display All
    • So basically: you have only a small amount of space left, but no clue, what occupies the used space. According to my knowledge this can happen, in case a process opens a file for writing, writes data into it and another one deletes the file. In this case the file is not shown in "du", but still occupies space. In case one could simply reboot, to make sure all open files are closed.

      Did you have a look into /tmp, it is missing in the output of "du"?

      Finally: why is the root file system so small, if you provided a 32GB SD card?
    • The root filesystem is the default size on SD cards. I assume that If it is a good SD card wear leveling will distribute wear so that also "unused" parts of the card will contribute to the longevity of the card. I use the same size 7GB rootfs partition on both my 32GB Sandisk Ultra A1 and my new 64 GB Sandisk Extreme A2 cards. So that is most likely fine.

      About the full rootfs.

      There has been a number of threads on this forum about rootfs filling up with seemingly invisible content.

      Typically this is caused by writing to a mount point with the device, that is supposed to be mounted there, missing. The problem is that when the device later is available it is mounted on the mount point and masks the actual contents of the mount point folder.

      Perhaps a rsync push when the remote device was not available?

      Another possibility is some error filling the logs, but that would not be masked like this seems to be.

      Yet another possibility is installing Plex or Emby or other dockers on the rootfs of SD card and then having metadata filling the card up, and severely shorten the durability of the card, but that would also be visible.

      The easiest way to fix this is to boot from other media and examine the rootfs when it is not in use. Then all the mount points will be unmounted and any "junk" files should be visible. For a SD card this means removing it and examining it from some other Linux computer with a SD card reader. It might be a Windows computer temporarily booted from a Linux USB drive.

      Feel free to search this forum for more details.

      Search for something like "system drive full", "rootfs full" or "system partition full"
      OMV 4, 7 x ODROID HC2, 1 x ODROID HC1, 5 x 12TB, 1 x 8TB, 1 x 2TB SSHD, 1 x 2TB SSD, GbE, WiFi mesh

      The post was edited 1 time, last by Adoby ().

    • Adoby wrote:

      The root filesystem is the default size on SD cards. I assume that If it is a good SD card wear leveling will distribute wear so that also "unused" parts of the card will contribute to the longevity of the card. I use the same size 7GB rootfs partition on both my 32GB Sandisk Ultra A1 and my new 64 GB Sandisk Extreme A2 cards. So that is most likely fine.

      About the full rootfs.

      There has been a number of threads on this forum about rootfs filling up with seemingly invisible content.

      Typically this is caused by writing to a mount point with the device, that is supposed to be mounted there, missing. The problem is that when the device later is available it is mounted on the mount point and masks the actual contents of the mount point folder.

      Perhaps a rsync push when the remote device was not available?

      Another possibility is some error filling the logs, but that would not be masked like this seems to be.

      Yet another possibility is installing Plex or Emby or other dockers on the rootfs of SD card and then having metadata filling the card up, and severely shorten the durability of the card, but that would also be visible.

      The easiest way to fix this is to boot from other media and examine the rootfs when it is not in use. Then all the mount points will be unmounted and any "junk" files should be visible. For a SD card this means removing it and examining it from some other Linux computer with a SD card reader. It might be a Windows computer temporarily booted from a Linux USB drive.

      Feel free to search this forum for more details.

      Search for something like "system drive full", "rootfs full" or "system partition full"
      Hi Adoby,

      Thanks for the pointers - very useful, actually. I did what you suggested, booted with a USB linux drive and it was, just as you predicted, a mount point with lot of data filling up the drive. Not sure when or how that happened, but I will pay attention in the future.

      Cheers