Well, this is embarrassing. Seems like restarting the server was all that it took to solve the issue. Funnily enough, I have restarted the server several times over the past few days due to updates and, well. Because of the hard drive replacements.
The scheduled script diff ran at midnight as expected, and I got notified per email as it should. Seems to me then that a reboot was necessary for the Cron jobs and/or the snapraid plugin configuration to apply properly.
Good day!
I am facing some issues with SnapRaid and I have no idea where the issue might be. My system has one parity drive and two pooled drives. Snapraid is configured as follows:
The scheduled diff is configured as follows:
Finally, the snapraid config looks like this:
# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.
autosave 0
# drives
#####################################################################
# OMV-Name: Standalone1 Drive Label:
content /srv/dev-disk-by-uuid-8c13770e-2445-485a-aaa6-9168205e3498/snapraid.content
data Standalone1 /srv/dev-disk-by-uuid-8c13770e-2445-485a-aaa6-9168205e3498
#####################################################################
# OMV-Name: POOL2 Drive Label:
content /srv/dev-disk-by-uuid-80ecf572-0de3-4f7b-aadd-03db85a12560/snapraid.content
data POOL2 /srv/dev-disk-by-uuid-80ecf572-0de3-4f7b-aadd-03db85a12560
#####################################################################
# OMV-Name: Parity_4_TB Drive Label:
parity /srv/dev-disk-by-uuid-67235391-d4ad-46aa-95cf-726fe23ebec3/snapraid.parity
exclude *.unrecoverable
exclude lost+found/
exclude aquota.user
exclude aquota.group
exclude /tmp/
exclude .content
exclude *.bak
exclude /snapraid.conf*
exclude /Downloads/
exclude /Syncthing/The issue I am having is that the scheduled diff is running, but not properly. The scheduled diff runs every night at midnight, but it is behaving oddly:
- After the diff happens, there is no notification email (notifications are working, I already checked that). Hence n
- The diff is registered in the system logs, but says that no changes were detected (although there are indeed file changes). As a consequence, no synchronization happens.
If I run the diff command from the OMV interface manually, the diff works as expected and detects the file changes. A consequent sync commands also works as expected.
I just recently changed the parity drive and replaced one of the data drives. Before that, the scheduled diff was not working because in the snapraid configuration page I had not selected a default array. After the drive replacements, everything was working as expected. I then checked the scheduled diff and realized that there were two automated tasks to do the same thing: one of them was generated by the scheduled diff option, the other one was perhaps a carry over from my older OMV installation. I removed the old scheduled task and modified the new one to run daily. Since then, I am having the issues above.
Things that I have tried:
- Delete all snapraid scheduled tasks and create new ones
- Check that the correct array is selected under "default array"
Finally, I suspect that the issue is in the command sent by the scheduled tasks:
for conf in /etc/snapraid/omv-snapraid-*.conf; do /usr/sbin/omv-snapraid-diff ${conf}; doneThe correct snapraid conf file is not omv-snapraid-*, but omv-snapraid-XXXXXXXX (where XXXXXXX is a long-ass string identifying the array I assume). Could it be that the scheduled diff fails since the wrong conf file is being addresed? If so, any suggestions how to tackle that?
Thanks in advance, and a happy new year)
Go into Services | SnapRAID | Settings and then select the Default array from the drop down list. Then Save settings.
Dude!!!! I was close to pull the hair out of my head, but this was it! I just realized that my Snapraid stopped working several months ago, and this was the issue. After I updated to OMV 7, this setting was not properly defined (defined array = none). Now snapraid touch works, and so the snapraid commands from the CLI.
Thanks a million times!
The mergerfs balancing tool causes data to be redistributed between the different hard drives in the pool. If you use that tool in this case you could end up with a backup folder split between several different paths. When you go to search for the path of only one of the hard drives, you would be missing files (because they have been moved to some other disk) therefore it would not be complete. So you should avoid using this tool to avoid that situation.
What ryecoaaron is referring to is that instead of creating a pool with entire hard drives you can create a folder on each of those drives and group those folders into a pool. That way you wouldn't have the problem you've raised in this thread. On one of the hard drives you can have the folder that is included in the pool and another independent folder for backups that is not within the mergerfs pool, therefore it would not be affected by the use of mergerfs tools such as balancing, for example.
I was completely unaware that it is possible to pool folders instead of drives. Guess I have homework now. You learn something new everyday. Thanks for the detailed explanation.
If people didn't use the root of each drive for mergerfs and instead used a subfolder, it would be very easy to have a pool and still be able to create folders on individual drivers outside of the pool. Then no worries about using the tool.
Would you mind to formulate further? What do you mean by "using the root for mergerfs"? My data is completely separated from my OS (one drive for OS only, it is more hygienic that way), and the OS drive is not pooled.
You can add that hard drive to the group. Only direct backups to the backup hard drive instead of the pool. This will ensure that backups always go to the same hard drive.
BUT, you should never use the mergerfs balancing tool, that would break your configuration.
Hello, and sorry for the late reply. I guessed that the solution would be something like this, but thanks for confirming.
Regarding the pool balancing, should I avoid balancing due to my particular configuration (1 share not spread on the pool) or should I avoid balancing completely (even if all the shares on the pool are spread across the drives)?
Hello!
My home server has currently 4 drives:
The reason for the standalone backup HDD is due to previous experience. I once had the backup HDD as part of the pool. At some point I had to recover my OMV installation, and finding the appropriate backup files in the pool of drives was a royal pain in the back. Since then, I took the backups drive out of the pool.
Now, the issue is that my backups do not consume the whole standalone drive (roughly 60% is empty), and the storage pool is slowly getting full. Which takes me to my question: is it possible to add the standalone HDD to the pool, but exclude the backups share? Namely, MergeFS will do its magic with all other shared folders except the backups one. If so:
Thanks in advance!
Hello again!
I have an update on the random shut-down issue. After upgrading my server to OMV 7, I had also planned to upgrade several hardware components (read new motherboard, a CPU 4 generations newer, and a new OS SSD). The migration was very painless I have to say, and the server has been online for the last 7 days without a hiccup.
If the system remains without issues, I would then conclude that it could have been one of 3 things:
- Although the upgrade to OMV 7 went smooth, perhaps something got corrupted during the update, triggering the shut downs?
- The hardware was on the old side (3rd generation Intel CPU). Perhaps the new Debian version was not happy with one of the old hardware components? (
- Despite 4 to 5 years of continuous use on the server, the old OS SSD showed no signs of wear and its SMART values were all OK. Perhaps the OS drive started kicking the bucket? But then, why only after upgrading?
I believe that one of the first two options could have been the culprit. I'll report back if the system shuts down again.
OMV is using a watchdog to restart systems automatically when they are in a dead situation.
If your system is under heavy load, it might happen that the watchdog restarts the system because due the load it can’t be updated in time. I have to say that the system must be under very very very heavy load in that case.
You need to check the journal for messages before the system is restarted.
I checked the messages journal, and as said before, the messages start at the time when the server was last started. Everything before is not available. The messages are also sent to my email account, but until now I have not received anything unusual.
The only thing that could produce such a load is Plex. But then again, that happens only when new files are added to the library and the music fingerprints are calculated. Or perhaps Snapraid? But even if that's the case, the system managed it without issues with OMV 6.
Also, when the load reaches such a point, a message is generated and I receive a notification. Hence, I am not really sure that the problem has something to do with it.
I am really confused as to why my signature is not visible. I see it on Firefox on windows, but not in my phone.
I am running a X86 system, kind of old one (third generation core I7).
Hello forum!
As per the title of the thread, I am experiencing random shut downs since I upgraded from OMV 6 to OMV 7. Before you suggest to check the logs, I did, but they are gone. After each shut down, the logs seem to reset. In other words, I can only see logs back to the point when the server was restarted. Anything before that is gone. For example, the last shut down happened somewhere between 21.08 and 22.08. The logs start at 22.08 at 18:01, when I manually turned on the server. I also get no notifications regarding high usage or high system load. I also checked under /var/log/syslog and got exactly the same.
Some additional notes:
So, that's pretty much it. Are any other logs hidden somewhere, or is it normal that the logs reset after an abnormal shut down?
Any help or advice is appreciated.
Last time I had to restore my installation from a dd back up, I found the easiest way was simply to make a clean installation of OMV, boot that up, and then simply restore the system partition out of the dd back up using the dd command line (don't aske precisely what command I used, it was long ago). Do you have a full dd back up, or simply the system partition? If you have the later, try first a clean installation (which takes care of the boot loader and such) and then try to restore the system only.
Nope. This is why the restart button (which should remount) is there. I don't like the idea of restarting the pool with any change because something may be using it.
As a safety measure, I believe a manual restart of the pool is the way to go. In that case you can avoid unintentional changes to the pool. The mistake was on my side, since I ignored that a restart of the pool is necessary to apply the changes. That is not a problem with OMV, that is simply how MergerFS works.
Replying to myself, I found a solution with the help of Reddit. Tl;DR: OMV does not apply the changes from MergerFS on the fly, meaning that the drive pool must be remounted after any changes are done (restarting OMV also works). Therefore, although the Web interface shows the that the pool is only composed of two drives instead of three, the underlying OS has not applied the changes yet. Hence, any changes done to the files in the non-pooled drive will be reflected in the pooled drives.
After mod fing the pool and restarting the system I was able to modify the files in the pooled drive without affecting the ones in the pooled ones.
[…]
Hello! Well yes I did, but my workaround was quite different. I have backups of my OMV installation. My solution was simply to nuke my OS drive and restore to it using one of the back ups. It was virtually impossible to know which docker container got corrupted after running the server without drives. I made the mistake of running some of the containers on a drive pool, hence the container files where spread across 3 different HDDs.
So, long story short, if you have a back up of your OS I would recommend just restoring your installation. It was way easier and faster, at least in my case.
PS: after you restore the OS, please oh please do not forget to connect the data drives before booting into your newly restored OMV. Don't ask me why I am telling you this.
Hello forum!
I am experiencing a very unusual issue that I cannot explain myself. First, my configuration:
1. x4 1 TB hard drives
2. x3 hard drives are pooled together with mergerfs
3. The fourth hard drive works as parity for snapraid
4. All my shared folders (namely samba shares) are spread across the pool, which is evenly distributed (all 3 drives are similarly full)
I wanted to remove one of the drives out of the pool, not from the system, only from the pool. So, I did the following:
1. In OMV UI, I removed the drive (lets call it solo-drive) from the pool. The name of the pool remained the same, but it has only 2 drives now.
2. Log in via SSH, and using MC copy the files from the solo-drive to the pool
3. I did not restarted my system in between
The problem is that the transfer of files went bananas: I first copied a folder with non-critical data. For that I just moved the files from the solo-drive to the pool. Moving the data went fine, but when it was finished it disappeared from both the solo-drive AND the pool. I then tried coping first and then deleting from the solo-drive, with the same result. In short, deleting a folder from the solo-drive deletes also its equivalent in the pool. For whatever reason, although the solo-drive is no longer in the pool, the files inside are still somehow linked.
So, I decided to stop before moving my critical files. Any idea why this is happening?
Hello!
I am in the same boat right now, but I do not really understand your solution. I ran my server without drives and after reinstalling the drives I am not able to mount my merged pool. I am also running docker containers pointing to the pool. I deleted the docker files in the mergerfs mount point, but that did not solve the issue. Would you mind to formulate further?
Thanks in advance!
Hello macom, and thanks for the suggestion. I also considered using it as an off-site backup. However, I don't feel like moving the drive back and forth. It is a mechanical drive after all and they are prone to getting damaged while moved around. On top of that, the files on the drive will change relatively often (mostly pictures synced to the server) meaning that I would need to regularly bring the drive home (i.e. once weekly). That's why I am looking for a solution to set and forget, like automated weekly snapshots of the data. I could also sync over the internet, but the outdated German internet infrastructure with 40 mb/s upload speed is holding me back.
Don't take me wrong! I should have also an off-site backup, but that would happen at a later point in time.
Hello Forum
As part of my backup strategy for my home server, I got myself a refurbished 3TB external drive. The price was very good (45€ for both the drive and Seagate enclosure, Ebay refurbished) so I decided to bite the bullet. The drive seems to be OK (7200 RPM, 0 hours runtime and 4 power on counts) and since it is going to be a secondary backup, I tough I could take the risk.
Now, I am currently using SnapRAID as my first backup (3 data drives plus a parity one) but I wanted to add an additional layer in case the human feces collide with the airflow generating machine. I'll run weekly snapshots of some data in the server (mainly pictures) to the external drive.
My question to you guys is where should the drive hang? I have a secondary server running docker with a bunch of different things. My line of thinking was that it would be "safer" to connect the external drive to the thin client in case my main server suffers a catastrophic failure and, for example, two or more drives kick the bucket. On the other hand, both my OMV server and the thin client sit next to each other in my server rack, which raises the question if it wouldn't be better to connect the external drive directly to my OMV and call it a day.
In addition, if the drive is mounted in OMV as a remote share, do you recommend Samba or NFS?
My apologies in advance if the thread is somewhat absurd, but I have burned more time thinking about it than I would like to admit.
Thanks!
