AUFS - Drive Balancing Issues **Solved**

    • OMV 1.0
    • Resolved
    • AUFS - Drive Balancing Issues **Solved**

      Hi All,

      I recently got started with OMV (and linux noob) and went the AUFS + SnapRaid route. I originally installed the old AUFS plugin until I realized there was the combined AUFS/Mhddfs plugin.
      When I had the old plugin installed, I could not get the balancing to work but hoped that the latest (combined) plugin would fix the problem.

      The new plugin seemed to somewhat work but I'm noticing that it now filling up the drive that already has the most data. I feel like it is only doing this for one of the folders and not all. After reading some post, I found out that during the uninstall i should have removed the .wh files?

      I did the following..
      1) uninstalled the old AUFS
      2) installed the new plugin
      3) Setup the Pool
      4) Migrated my data over to the new folders.
      5) Removed the old folders


      Here is my current setup and info.



      [IMG:http://s18.postimg.org/jeqzuqwzd/AUFS.png]


      ii openmediavault 1.10 all Open network attached storage solution
      ii openmediavault-backup 1.0.11 all backup plugin for OpenMediaVault.
      ii openmediavault-calibre 1.0.13 all Calibre plugin for OpenMediaVault.
      ii openmediavault-clamav 1.2 all OpenMediaVault ClamAV plugin
      ii openmediavault-couchpotato 1.0.3 all OpenMediaVault CouchPotatoServer plugin
      ii openmediavault-cups 1.0.9 all OpenMediaVault CUPS (network printer server) plugin
      ii openmediavault-downloader 1.0.5 all OpenMediaVault downloader plugin
      ii openmediavault-extplorer 1.0.6 all OpenMediaVault eXtplorer plugin
      ii openmediavault-headphones 1.0.2 all OpenMediaVault Headphones plugin
      pc openmediavault-jdownloader 1.0.8 all OpenMediaVault Jdownloader plugin
      ii openmediavault-keyring 0.3 all GnuPG archive keys of the OpenMediaVault archive
      ii openmediavault-mysql 1.0.13 all MySQL plugin for OpenMediaVault
      ii openmediavault-nginx 1.0.13 all Nginx plugin for OpenMediaVault.
      ii openmediavault-omvextrasorg 1.8 all OMV-Extras.org Package Repositories for OpenMediaVault
      ii openmediavault-owncloud 1.3 all OpenMediaVault ownCloud plugin
      ii openmediavault-plexmediaserver 1.0.9 all OpenMediaVault Plex Media Server plugin
      ii openmediavault-remoteshare 1.0.5 all remote share plugin for OpenMediaVault.
      ii openmediavault-sabnzbd 1.0.5 all OpenMediaVault SABnzbd plugin
      ii openmediavault-sickbeard 1.0.3 all OpenMediaVault Sick Beard plugin
      ii openmediavault-snapraid 1.0.13 all snapraid plugin for OpenMediaVault.
      ii openmediavault-transmissionbt 1.0.13 all OpenMediaVault Transmission (BitTorrent client) plugin
      ii openmediavault-unionfilesystems 1.2 all Union filesystems plugin for OpenMediaVault.
      ii openmediavault-usbbackup 1.1 all OpenMediaVault USB/eSATA backup plugin
      ii openmediavault-virtualbox 1.0.14 all VirtualBox plugin for OpenMediaVault.
      ii openmediavault-wol 1.0.2 all OpenMediaVault WOL plugin


      Regarding the .wh files. The old AUFS setup was using Books1, Books2 etc. The new AUFS is just folder called books.

      4TB1
      drwxrwsrwx 4 root users 4096 Jan 19 19:46 .wh..wh.Books1.1a92
      drwxrwsrwx 4 root users 4096 Jan 19 17:31 .wh..wh.ConsoleGames1.000a
      drwxrwsrwx 4 root users 4096 Jan 19 17:08 .wh..wh.Movies1.0001
      drwxrwsrwx 4 root users 20480 Jan 19 19:51 .wh..wh.Music1.2410
      drwx--S--- 2 root users 4096 Jan 19 17:00 .wh..wh.orph
      drwxrwsrwx 4 root users 12288 Jan 19 22:10 .wh..wh.Photos1.577a
      drwx--S--- 2 root users 4096 Jan 19 17:00 .wh..wh.plnk
      drwxrwsrwx 4 root users 4096 Jan 19 18:03 .wh..wh.SecurityFilm1.000f
      drwxrwsrwx 4 root users 4096 Jan 19 20:39 .wh..wh.Software1.4da5
      drwxrwsrwx 4 root users 4096 Jan 19 21:27 .wh..wh.Tv1.4de5

      3TB1
      -r--r--r-- 2 root root 0 Jan 19 17:00 .wh.ConsoleGames2
      -r--r--r-- 2 root root 0 Jan 19 17:00 .wh..wh.aufs
      drwxrwsrwx 4 root users 4096 Jan 19 19:47 .wh..wh.Books2.1a93
      drwxrwsrwx 5 root users 4096 Jan 19 17:26 .wh..wh.ConsoleGames2.0009
      drwxrwsrwx 4 root users 12288 Jan 19 17:08 .wh..wh.Movies2.0002
      drwxrwsrwx 4 root users 20480 Jan 19 19:52 .wh..wh.Music2.2436
      drwx------ 2 root root 4096 Jan 19 17:00 .wh..wh.orph
      drwxrwsrwx 4 root users 4096 Jan 19 22:10 .wh..wh.Photos2.577e
      drwx------ 2 root root 4096 Jan 19 17:00 .wh..wh.plnk
      drwxrwsrwx 4 root users 4096 Jan 19 18:03 .wh..wh.SecurityFilm2.0012
      drwxrwsrwx 4 root users 4096 Jan 19 20:39 .wh..wh.Software2.4da6
      drwxrwsrwx 4 root users 4096 Jan 19 21:27 .wh..wh.Tv2.4dee

      3TB2
      -r--r--r-- 1 root root 0 Jan 19 17:00 .wh..wh.aufs
      drwxrwsr-x 4 root users 4096 Jan 19 19:47 .wh..wh.Books3.1a94
      drwxrwsrwx 4 root users 4096 Jan 19 19:25 .wh..wh.ConsoleGames3.182c
      drwxrwsrwx 4 root users 4096 Jan 19 17:08 .wh..wh.Movies3.0003
      drwxrwsrwx 4 root users 4096 Jan 19 19:52 .wh..wh.Music3.2437
      drwx------ 2 root root 4096 Jan 19 17:00 .wh..wh.orph
      drwxrwsrwx 4 root users 4096 Jan 19 22:10 .wh..wh.Photos3.5782
      drwx------ 2 root root 4096 Jan 19 17:00 .wh..wh.plnk
      drwxrwsrwx 4 root users 4096 Jan 19 18:03 .wh..wh.SecurityFilm3.0016
      drwxrwsrwx 4 root users 4096 Jan 19 20:40 .wh..wh.Software3.4da7
      drwxrwsrwx 4 root users 4096 Jan 19 21:27 .wh..wh.Tv3.4df0
    • I have never been able to replicate the aufs issue but I have tried to help with it. I haven't been willing to risk a user's data by deleting the wh files. It would be nice to know if that fixes it.
      omv 4.1.15 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • Thanks for the input ryecoaaron.

      I did notice something strange. In the subdirectory that doesn't seem to be balancing, I did notice something different.
      There is a file called .wh..wh..opq.
      I'm currently searching everywhere in the system for that file to see if paints a better picture.

      I'm not sure what this file does even after reading about it in the manual. Any ideas ?

      "The whiteout is for hiding files on lower branches. Also it is applied to stop readdir going lower branches. The latter case is called ’opaque directory.’ Any whiteout is an empty file, it means whiteout is just an mark. In the case of hiding lower files, the name of whiteout is ’.wh.<filename>.And in the case of stopping readdir, the name is ’.wh..wh..opq’ or ’.wh.__dir_opaque.’ The name depends upon your compile configuration CONFIG_AUFS_COMPAT. All whiteouts are hardlinked, including ’<writable branch top dir>/.wh..wh.aufs.’"

      Edit: More Opaque Dir Info.
      Source: zackreed.me/articles/72-snapra…buntu-12-04?view=comments

      ​Next, let's delete aufs_storage/test

      ​All files are gone, the deleted directory on one branch is whiteout-ed but remains on the other two branches. The "test" dir is not visible via aufs of course.
      Now let's recreate "aufs_storage/test

      ​The re-created directory has an opaque flag in it on disk3 (the first branch)

      ​Now if I create some files in aufs_storage/test, they will not be load balanced across branches and will all be created on disk3

      ​I am seeing two issues:

      1. A deleted directory, that was load balanced gets whiteout-ed from the first branch only. All files get deleted, but the entire directory structure (no matter how deep) will be preserved on all but one branch.

      2. If a directory is recreated in the same place, it becomes opaqued and the files written into it will never be load balanced.

      3. Rename works a combination of delete and create and causes the same issue

      The post was edited 2 times, last by cody00 ().

    • The find for that file has finished (find . -name ".wh..wh..opq"), and it only found that one on the whole server. Coincidentally, this seems to be the only directory that is not balancing. Now, we need to figure out if we can simply remove the opaque tag or not.


      Edit:
      From the user on the link above
      I can recover by deleting the whiteout-ed/opaqued dir/marker and all the corresponding dir on all branches and remounting. I can automate this cleanup and a script that runs daily if needed, but there could be a better way.

      What could I be missing here? Thx in advance.

      The post was edited 2 times, last by cody00 ().

    • Okay, I resolved the AUFs not balancing properly.

      1) Searched for the opaque whiteout files. (cd /media then find . -name ".wh..wh..opq")
      2) Removed the ’.wh..wh..opq’ files. (Yes I guess you can do it with one command in the find above.)
      3) Rebooted OMV.

      Warning. : I'm a linux noob and i have no idea what im doing. Do this at your own risk. I will also add, I believe this is a result of deleting/moving/making and just trying to shift data around. (this was also noted from the other link)

      The post was edited 1 time, last by cody00 ().

    • Thank you for your tests. Maybe we can get a few more people to test this and finally solve the aufs balancing problem :)
      omv 4.1.15 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • You will have a serious performance hit using hnotify or inotify. This still seems to be a fairly uncommon problem. So, I would like to try to find a solution without using those.
      omv 4.1.15 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • For Documentation and hopefully helping someone in the future....

      I recently receive an I/O error while trying to write to the pooled drive. At first glances, I thought my drive was failing but it turns out it was another .wh.wh.<foldername> causing the problem. I removed this file, rebooted, and the problem was resolved.

      On a note about erasing .wh.wh files. A friend of mine has the exact same setup with omv aufs. He came across a problem and took it to the extreme. He removed all .wh.wh (not just the opq) files on the system as a test. Nearly a month later and he has not experienced any problems.