AUFS - Drive Balancing Issues **Solved**

Hi All,

I recently got started with OMV (and linux noob) and went the AUFS + SnapRaid route. I originally installed the old AUFS plugin until I realized there was the combined AUFS/Mhddfs plugin.
When I had the old plugin installed, I could not get the balancing to work but hoped that the latest (combined) plugin would fix the problem.

The new plugin seemed to somewhat work but I'm noticing that it now filling up the drive that already has the most data. I feel like it is only doing this for one of the folders and not all. After reading some post, I found out that during the uninstall i should have removed the .wh files?

I did the following..
1) uninstalled the old AUFS
2) installed the new plugin
3) Setup the Pool
4) Migrated my data over to the new folders.
5) Removed the old folders

Here is my current setup and info.

ii openmediavault 1.10 all Open network attached storage solution
ii openmediavault-backup 1.0.11 all backup plugin for OpenMediaVault.
ii openmediavault-calibre 1.0.13 all Calibre plugin for OpenMediaVault.
ii openmediavault-clamav 1.2 all OpenMediaVault ClamAV plugin
ii openmediavault-couchpotato 1.0.3 all OpenMediaVault CouchPotatoServer plugin
ii openmediavault-cups 1.0.9 all OpenMediaVault CUPS (network printer server) plugin
ii openmediavault-downloader 1.0.5 all OpenMediaVault downloader plugin
ii openmediavault-extplorer 1.0.6 all OpenMediaVault eXtplorer plugin
ii openmediavault-headphones 1.0.2 all OpenMediaVault Headphones plugin
pc openmediavault-jdownloader 1.0.8 all OpenMediaVault Jdownloader plugin
ii openmediavault-keyring 0.3 all GnuPG archive keys of the OpenMediaVault archive
ii openmediavault-mysql 1.0.13 all MySQL plugin for OpenMediaVault
ii openmediavault-nginx 1.0.13 all Nginx plugin for OpenMediaVault.
ii openmediavault-omvextrasorg 1.8 all OMV-Extras.org Package Repositories for OpenMediaVault
ii openmediavault-owncloud 1.3 all OpenMediaVault ownCloud plugin
ii openmediavault-plexmediaserver 1.0.9 all OpenMediaVault Plex Media Server plugin
ii openmediavault-remoteshare 1.0.5 all remote share plugin for OpenMediaVault.
ii openmediavault-sabnzbd 1.0.5 all OpenMediaVault SABnzbd plugin
ii openmediavault-sickbeard 1.0.3 all OpenMediaVault Sick Beard plugin
ii openmediavault-snapraid 1.0.13 all snapraid plugin for OpenMediaVault.
ii openmediavault-transmissionbt 1.0.13 all OpenMediaVault Transmission (BitTorrent client) plugin
ii openmediavault-unionfilesystems 1.2 all Union filesystems plugin for OpenMediaVault.
ii openmediavault-usbbackup 1.1 all OpenMediaVault USB/eSATA backup plugin
ii openmediavault-virtualbox 1.0.14 all VirtualBox plugin for OpenMediaVault.
ii openmediavault-wol 1.0.2 all OpenMediaVault WOL plugin

Regarding the .wh files. The old AUFS setup was using Books1, Books2 etc. The new AUFS is just folder called books.

4TB1
drwxrwsrwx 4 root users 4096 Jan 19 19:46 .wh..wh.Books1.1a92
drwxrwsrwx 4 root users 4096 Jan 19 17:31 .wh..wh.ConsoleGames1.000a
drwxrwsrwx 4 root users 4096 Jan 19 17:08 .wh..wh.Movies1.0001
drwxrwsrwx 4 root users 20480 Jan 19 19:51 .wh..wh.Music1.2410
drwx--S--- 2 root users 4096 Jan 19 17:00 .wh..wh.orph
drwxrwsrwx 4 root users 12288 Jan 19 22:10 .wh..wh.Photos1.577a
drwx--S--- 2 root users 4096 Jan 19 17:00 .wh..wh.plnk
drwxrwsrwx 4 root users 4096 Jan 19 18:03 .wh..wh.SecurityFilm1.000f
drwxrwsrwx 4 root users 4096 Jan 19 20:39 .wh..wh.Software1.4da5
drwxrwsrwx 4 root users 4096 Jan 19 21:27 .wh..wh.Tv1.4de5

3TB1
-r--r--r-- 2 root root 0 Jan 19 17:00 .wh.ConsoleGames2
-r--r--r-- 2 root root 0 Jan 19 17:00 .wh..wh.aufs
drwxrwsrwx 4 root users 4096 Jan 19 19:47 .wh..wh.Books2.1a93
drwxrwsrwx 5 root users 4096 Jan 19 17:26 .wh..wh.ConsoleGames2.0009
drwxrwsrwx 4 root users 12288 Jan 19 17:08 .wh..wh.Movies2.0002
drwxrwsrwx 4 root users 20480 Jan 19 19:52 .wh..wh.Music2.2436
drwx------ 2 root root 4096 Jan 19 17:00 .wh..wh.orph
drwxrwsrwx 4 root users 4096 Jan 19 22:10 .wh..wh.Photos2.577e
drwx------ 2 root root 4096 Jan 19 17:00 .wh..wh.plnk
drwxrwsrwx 4 root users 4096 Jan 19 18:03 .wh..wh.SecurityFilm2.0012
drwxrwsrwx 4 root users 4096 Jan 19 20:39 .wh..wh.Software2.4da6
drwxrwsrwx 4 root users 4096 Jan 19 21:27 .wh..wh.Tv2.4dee

3TB2
-r--r--r-- 1 root root 0 Jan 19 17:00 .wh..wh.aufs
drwxrwsr-x 4 root users 4096 Jan 19 19:47 .wh..wh.Books3.1a94
drwxrwsrwx 4 root users 4096 Jan 19 19:25 .wh..wh.ConsoleGames3.182c
drwxrwsrwx 4 root users 4096 Jan 19 17:08 .wh..wh.Movies3.0003
drwxrwsrwx 4 root users 4096 Jan 19 19:52 .wh..wh.Music3.2437
drwx------ 2 root root 4096 Jan 19 17:00 .wh..wh.orph
drwxrwsrwx 4 root users 4096 Jan 19 22:10 .wh..wh.Photos3.5782
drwx------ 2 root root 4096 Jan 19 17:00 .wh..wh.plnk
drwxrwsrwx 4 root users 4096 Jan 19 18:03 .wh..wh.SecurityFilm3.0016
drwxrwsrwx 4 root users 4096 Jan 19 20:40 .wh..wh.Software3.4da7
drwxrwsrwx 4 root users 4096 Jan 19 21:27 .wh..wh.Tv3.4df0

Thanks for the input ryecoaaron.

I did notice something strange. In the subdirectory that doesn't seem to be balancing, I did notice something different.
There is a file called .wh..wh..opq.
I'm currently searching everywhere in the system for that file to see if paints a better picture.

I'm not sure what this file does even after reading about it in the manual. Any ideas ?

"The whiteout is for hiding files on lower branches. Also it is applied to stop readdir going lower branches. The latter case is called ’opaque directory.’ Any whiteout is an empty file, it means whiteout is just an mark. In the case of hiding lower files, the name of whiteout is ’.wh.<filename>.’ And in the case of stopping readdir, the name is ’.wh..wh..opq’ or ’.wh.__dir_opaque.’ The name depends upon your compile configuration CONFIG_AUFS_COMPAT. All whiteouts are hardlinked, including ’<writable branch top dir>/.wh..wh.aufs.’"

Edit: More Opaque Dir Info.
Source: http://zackreed.me/articles/72…buntu-12-04?view=comments

Zitat

Next, let's delete aufs_storage/test

Zitat

All files are gone, the deleted directory on one branch is whiteout-ed but remains on the other two branches. The "test" dir is not visible via aufs of course.
Now let's recreate "aufs_storage/test

Zitat

The re-created directory has an opaque flag in it on disk3 (the first branch)

Zitat

Now if I create some files in aufs_storage/test, they will not be load balanced across branches and will all be created on disk3

Zitat

I am seeing two issues:

1. A deleted directory, that was load balanced gets whiteout-ed from the first branch only. All files get deleted, but the entire directory structure (no matter how deep) will be preserved on all but one branch.

2. If a directory is recreated in the same place, it becomes opaqued and the files written into it will never be load balanced.

3. Rename works a combination of delete and create and causes the same issue

Alles anzeigen

The find for that file has finished (find . -name ".wh..wh..opq"), and it only found that one on the whole server. Coincidentally, this seems to be the only directory that is not balancing. Now, we need to figure out if we can simply remove the opaque tag or not.

Edit:
From the user on the link above

Zitat

I can recover by deleting the whiteout-ed/opaqued dir/marker and all the corresponding dir on all branches and remounting. I can automate this cleanup and a script that runs daily if needed, but there could be a better way.

What could I be missing here? Thx in advance.

Okay, I resolved the AUFs not balancing properly.

1) Searched for the opaque whiteout files. (cd /media then find . -name ".wh..wh..opq")
2) Removed the ’.wh..wh..opq’ files. (Yes I guess you can do it with one command in the find above.)
3) Rebooted OMV.

Warning. : I'm a linux noob and i have no idea what im doing. Do this at your own risk. I will also add, I believe this is a result of deleting/moving/making and just trying to shift data around. (this was also noted from the other link)

It seems that there are some settings to AUFS that prevent opaque files from being created. Is there a chance of adding something like this into plugin?

Zitat

By compiling AUFS with hnotify support. All of these issues seem to go away, and the pool does not leave behind any whiteout or opaque files after a move or deletion.

For Documentation and hopefully helping someone in the future....

I recently receive an I/O error while trying to write to the pooled drive. At first glances, I thought my drive was failing but it turns out it was another .wh.wh.<foldername> causing the problem. I removed this file, rebooted, and the problem was resolved.

On a note about erasing .wh.wh files. A friend of mine has the exact same setup with omv aufs. He came across a problem and took it to the extreme. He removed all .wh.wh (not just the opq) files on the system as a test. Nearly a month later and he has not experienced any problems.