Beiträge von rowie

Fine, nice work!

Gesendet von meinem SM-G935F mit Tapatalk

Hi Didier,

- sata controller = ahci
- dit you set the pointers with vmkvstools ? (https://kb.vmware.com/selfserv…playKC&externalId=1017530)
- is your gen8 up to date?
- du you have activated a valid esx license?
- and last but not least .. please make automatic backups!

If one if the discs failed .. you have to reboot the gen8 because there is no hotpluf feature.
Hope you doing well.

Best regards,
Ronald

Ps: please excuse the delay ... i am in vienna on a security symposium ... writing on my mobilphone ..

Gesendet von meinem SM-G935F mit Tapatalk

post from 12. Mai 2016 .... nice one!

you have to install etherwake and sendEmail on omv!

hi,

do you have rsync server enabled on QNAP?

i ´m using a small bash-script on my omv for that.
it makes:

- first of all check if qnap is online with wget
- when its online

rsync the hole raid5 to QNAP
- when its offline .. wake it up and wait 5 minutes then check again with wget
- when its online
rsync the .....

- after this send email with:
success
or
failed

copy the script if you like it to your omv ... make it executeable ... and create a crontab entry:

like:

00 22 * * * /bin/bash /root/nas2nas_sync.sh

Script will start at 22 o´clock

On the qnap i have run another script who is checking the discs every 5 minutes. if all discs in sleep ... send qnap to sleep (think S3)

OMV Script:

#!/bin/bash
## rowie - v1.0 - NAS2NAS rsync Backup-Script
############################################
##
#
#
## VARIABLES
#
# Set source location


BACKUP_FROM="/srv/dev-disk-by-label-RAID5/"


#
# Set target location


BACKUP_TO="rsync://nas-backup@192.168.115.17/NAS-Backup" ##<- Replace with QNAP rsync user@IP/rsync FOLDER##


#
#Backup destination


BACKUP_EXCLUDE_BIN="/.*" #Exclude the hidden files, such as recycle bin or other


#
# target URL to check if up


URL="http://192.168.115.17" ##<- Replace it with the QNAP IP##
TIMEOUT="300"
RETRY="2"
reached="0"
retryCnt="1"


#
# Log file


LOG_FILE="/var/log/nas2nas-rsync.log"


###########################################
#
##
## SCRIPT
##
# Check if the log file exists and then delete it


if [ -e "$LOG_FILE" ]; then
        rm -rf "$LOG_FILE"
fi


# Check that the log file exists
if [ ! -e "$LOG_FILE" ]; then
        touch "$LOG_FILE"
fi


# Check that source dir exists and is readable.
if [ ! -r  "$BACKUP_FROM" ]; then
        echo "$(date "+%Y-%m-%d %k:%M:%S") - ERROR: Unable to read source dir." >> "$LOG_FILE"
        echo "$(date "+%Y-%m-%d %k:%M:%S") - ERROR: Unable to read source dir."
        echo "$(date "+%Y-%m-%d %k:%M:%S") - ERROR: Unable to sync." >> "$LOG_FILE"
        echo "$(date "+%Y-%m-%d %k:%M:%S") - ERROR: Unable to sync. Check the log file..."
        echo "" >> "$LOG_FILE"
        exit 1
fi




# Check that Backup destination is up and if not Wake it up (with etherwake)
while [ $reached -eq 0 -a $retryCnt -le $RETRY ]
do
        wget -q -S --spider $URL 2>&1 | grep "HTTP/1.1 302 Found" >/dev/null
        if [ $? -eq 0 ]
        then
                reached=1;
                else
                /usr/sbin/etherwake xx:xx:xx:xx:xx:xx ##<- Insert QNAP MAC-Adresse##
                sleep $TIMEOUT;
        fi
                 retryCnt=`expr $retryCnt + 1`;


done


# Start entry in the log


echo "$(date "+%Y-%m-%d %k:%M:%S") - Sync started." >> "$LOG_FILE"
echo "$(date "+%Y-%m-%d %k:%M:%S") - Sync started... Please wait."


# Start sync


export RSYNC_PASSWORD=lamepassword #<- QNAP rsync-ServerPassword
if rsync -av --exclude "$BACKUP_EXCLUDE_BIN" "$BACKUP_FROM" "$BACKUP_TO" &>> "$LOG_FILE"; then
        echo "$(date "+%Y-%m-%d %k:%M:%S") - Sync completed succesfully." >> "$LOG_FILE"
        echo "$(date "+%Y-%m-%d %k:%M:%S") - Sync completed succesfully."
else
        echo "$(date "+%Y-%m-%d %k:%M:%S") - ERROR: rsync-command failed." >> "$LOG_FILE"
        echo "$(date "+%Y-%m-%d %k:%M:%S") - ERROR: Unable to sync." >> "$LOG_FILE"
        echo "$(date "+%Y-%m-%d %k:%M:%S") - ERROR: Unable to sync. Check the log file."
        echo "" >> "$LOG_FILE"
        exit 1
fi


sendEmail -f from@example -t to@example.com -s smtp.example.com:587 -o tls=auto -xu from@example.com -xp lamepassword -a "$LOG_FILE" -m NAS2NAS sync Successful: $(date)
# End entry in the log
echo "" >> "$LOG_FILE"
exit 0

QNAP Script:

#!/bin/bash
lines=`hdparm -C /dev/sd[a-f]|grep "active/idle"|wc -l|tr -d ' '|cut -f1 -d' '`
if [ $lines == "0" ]; then
   cat /etc/config/sleepmail.txt|sendmail -t
   ntpdate time.euro.apple.com 2>/dev/null >/dev/null
   hwclock --systohc
   sleep 30
   echo mem >/sys/power/state
fi

crontab on QNAP:

*/5 * * * * /etc/config/send_nas_to_sleep_if_idle.sh

the scripts are certainly not perfect ... but they work!

br,
rowie

ups, sorry! 2017.2 natürlich

Hi omavoss,

Also ich nutze den pxs bei mir zuhause hautsächlich wenn ich eine neue vm erstele oder den rechner eines freundes neu install. Hauptgrund ist, ich will mir die ständige usb-stick flasherei ersparen.
Bei debian oder fedora nehme ich dazu immer die netinstall x86_64. 32bit braucht eh kein mensch mehr.
zum festplattenlöschen boote ich eine partedmagic-vm und reiche die Platte per usb3 durch an die vm.
Ich hab auch ein paar kleine kleine alix und minisysteme die ich gerne mit livedistris wie tails oder kali betreibe. Entweder mit oder oder persistenter datenträuger die ich per nfs automatisch einbinde..

Kali2017.1 und der netinstaller von stretch wollen ums verrecken nicht booten .. drum mein post!

Ich hab schon einige images damals erstellt aber die sind halt leider nicht frei. Wie zb. Acronis und eben die bezahlversion von partedmagic.

Gesendet von meinem SM-G935F mit Tapatalk

Zitat von tkaiser

Well, since wasting disks to implement the R in RAID is only about providing availability I would say this simply sucks and is pretty insufficient since the specific implementation thwarts the whole technology

in many things the implementation is flawed but, without raid i had lost data, right!

the mdadm softraid exist since ... lets say 10 years ... maybe longer. From an old Dell server to a miniITX board with ipfire as OS ... from 500GB Disk up to 1TB and now 3TB. ext3 to ext4 and so on
Everthing works fine the last years and i never ever lost any bit of data.

in my castle there are many many things working 24/7 so therefore it is very important to find the right balance between energy consumption, stability and speed.
every day at 2 am omv-vm send wol to my qnap and when its up rsync makes the rest.

in worstcase i stop automatic shutdown of the qnap. bamm!

Zitat von tkaiser

I had the same already 18 years ago when I learned why hardware RAID controllers are so dangerous. Moved a whole internal 8 disk RAID (RAID5 back then since we couldn't do better with one configured hotspare) to another host. RAID controller an insanely expensive thing from AMI (IIRC MegaRAID) sitting in an own 5.25" drive bay and refusing to boot the new server. Since 'hotspare missing' (8 drive bays, one internally damaged). So all the money spent on 'business contuinity' was simply wasted since the RAID controller could not be convinced from forgetting about the 'additional availability guarantee' and we had only two alternatives: exchanging the hotspare with an array member to boot with a degraded array (since I tested a rebuild and measured array performance before that was not an option since +36 hours crappy performance, full restore from tapes would have lasted even longer in case something went wrong) so we moved everything back into the old server (5 in the morning). Lesson learned...

i spend a lot of time in fixing problems on our dell servers and megaraid illness so i know what you mean!

Zitat von tkaiser

It's 2017, every cheap ARM toy with Gigabit Ethernet can do this. That's not a number any more

yes, i know!
but for me was only important to reach the borders of my 1gb network and have a reason to upgrade the whole over the next years to 10gb

maybe I should take a closer look at zfs ..

Nice one!

yesterday was the day!
Backup of 6,4TB data to my qnap finished.
i have simple removed the disk and guess what!

the hypervisor has stopped the openmediavault vm instantly.
The gen8 doesn´t support hotplug so i have to reboot the whole system.
Back online the openmediavault vm starts with one missing disk but clean raid5 and health filesystem.
i added the disk with
mdadm --manage /dev/md127 --add /dev/sdd
and now the raid5 is recovering ... should take a while ..

summary i would say ... it works after a disk failure ... but it is not perfect because the ESX stoped the vm and only a reboot of the whole system brings it back to normal.

at first appearance i would say .. Performace is better then before (native OMV).

SMB write speed 112MB/sec
After raid recover i will make the tests with iozone!

thx

Seas omavoss,

Netinstall: Debian Stretch, SolusOS und Fedora

Live: Kali 2017.1, Tails und Partitionmagic

Lg,
rowie

Gesendet von meinem SM-G935F mit Tapatalk

Zitat von tkaiser

If the VMs were on a RAID5 before and now are on a SSD you gave the answer why already. Easy to test with iozone and '-i 2 -I' (this tests for random IO using direct IO which pretty much simulates what happens in virtualized environments).

OMV3 and the datastore for the 4 virtualbox vm´s was stored on den samsung 850 pro.. but with no seperate partition for the "datastore". Maybe ext4 is not so fine for that..

please could you explain how to test with ".. iozone and '-i 2 -I' .. " ?

Maybe i go back to Gen8 with ESXi6.5 and the SSD for the VM´s and the N40L with OMV3 for the Data or i replace the N40L with a Qnap and enable iSCSi for the VM´s.

Good luck with your storage expansion!

@tkaiser

I agree with you 100% in all!
Let me backup the raid5 to an external drive and then i will try to simulate (or real) a disk fail.

the possibility to cover everything with one box is of course very tempting ... but I want at least two things:

keep it simple as possible
und secure!

before I had openmediavault natively on the hp n40l and virtualbox for the vm´s with a very low performance ...

Don` t know. Think there is no hotplug on the gen8 i think.

Gesendet von meinem SM-G935F mit Tapatalk

HP Prolinant Gen8
ESXi 6.5 on microSD
datastore (for vms) on Samsung SSD(on ODD Port)
4x 3TB WD Red for Data

https://kb.vmware.com/selfserv…playKC&externalId=1017530

RAID5 with the 4x 3 TB Disk (from old OMV-Server --physical HP Server) direct mounted to OMV3 VM .

Works!

Zitat von votdev

7. Juni 2016+4
The core system seems to be ready now, but the plugins need to be adapted now.
i cannot give you an exact date, but it should not take too much time.

Any news since 7. Juni?

hi subzero79,

did you read my first post?
i have that entry in /etc/nginx/openmediavault-nginx.d/syncthing.conf
And i have uses the entrys from you and then from the synthing-team. But it is still not working!

1. you have to disable "admin over secure connection" in synthing - because you can t you 2 different certs
2. then it should be possible to terminate the tsl on the nginx and talk http with the synthing
...but it doesnt work ...

PS: I think it should be default to enable https on all "open web client" bookmarks (and the webinterface too) in OMV3 because its contemporay.
And with Letsencrypt it would be an easy one to serve services halfway "secure" to the internet if required.

br,
rowie

hi subzero79,

i have a working .conf for sabnzbd. Only the syncthing thing doesn t work.
I m connecting over https://nas.xxxx.xxx (https only) with a trusted public Certificate to my nas. And all links should open in this context.

location /sabnzbd {
        proxy_pass        http://localhost:8080/sabnzbd;
        proxy_set_header  Host nas.xxx.xxx;
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_redirect    default;
}

br,
rowie

hi,

there are some other entry´s but they are very old ... so i start a new one.

Plan is to have one url to bind them all ... with one cert.

https://nas.bla.bla/synthing
https://nas.bla.bla/sabnzbd (works)
....

i ve found this but the config doesn t work:
https://docs.syncthing.net/users/reverseproxy.html

openmediavault-webgui_eror.log:

2016/11/16 17:29:03 [error] 2588#0: *12163 open() "/var/www/openmediavault/syncthing" failed (2: No such file or directory), client: 10.10.2.101, server: openmediavault-webgui, request: "GET /syncthing HTTP/1.1", host: "nas.bla.bla"

It would be soooo nice if this automaticly works on openmediavault because ... http is BAD! **pls dont hate me**

PS: Where can i change the url s for example "open Webinterface" in the omv-webgui for sabnzbd?

PPS: this Simple fast daemon host webgui access [Nginx] doesn t solve my problem!!!

br,

rowie

Errorlog:

016-11-16 13:59:25,629::INFO::[assembler:87] Decoding /media/2f594341-147e-4352-8b2c-2a4c9ed49967/Movies/Downloads/temp/ct Magazin fuer Computertechnik 201624.par2/ct Magazin fuer Computertechnik 201624.rar yenc
2016-11-16 13:59:25,711::INFO::[__init__:913] /media/2f594341-147e-4352-8b2c-2a4c9ed49967/Movies/Downloads/temp/ct Magazin fuer Computertechnik 201624.par2/__ADMIN__/SABnzbd_nzo_VmNHKH removed
2016-11-16 13:59:25,711::INFO::[nzbqueue:258] Saving queue
2016-11-16 13:59:25,711::INFO::[__init__:922] Saving data for queue10.sab in /var/opt/sabnzbd/admin/queue10.sab
2016-11-16 13:59:25,712::INFO::[postproc:83] Saving postproc queue
2016-11-16 13:59:25,713::INFO::[__init__:922] Saving data for postproc2.sab in /var/opt/sabnzbd/admin/postproc2.sab
2016-11-16 13:59:25,714::INFO::[postproc:294] Starting Post-Processing on ct Magazin fuer Computertechnik 201624.par2 => Repair:True, Unpack:True, Delete:True, Script:None, Cat:*
2016-11-16 13:59:25,714::INFO::[postproc:595] Starting verification and repair of ct Magazin fuer Computertechnik 201624.par2
2016-11-16 13:59:25,716::INFO::[__init__:879] /media/2f594341-147e-4352-8b2c-2a4c9ed49967/Movies/Downloads/temp/ct Magazin fuer Computertechnik 201624.par2/__ADMIN__/__verified__ missing
2016-11-16 13:59:25,717::INFO::[postproc:616] Running verification and repair on set ct Magazin fuer Computertechnik 201624
2016-11-16 13:59:25,718::INFO::[newsunpack:1089] Quick-check for ct Magazin fuer Computertechnik 201624 is OK, skipping repair
2016-11-16 13:59:25,718::INFO::[newsunpack:1163] Deleting /media/2f594341-147e-4352-8b2c-2a4c9ed49967/Movies/Downloads/temp/ct Magazin fuer Computertechnik 201624.par2/ct Magazin fuer Computertechnik 201624.par2
2016-11-16 13:59:25,719::INFO::[postproc:643] Verification and repair finished for ct Magazin fuer Computertechnik 201624.par2
2016-11-16 13:59:25,755::INFO::[misc:806] Creating directories: /media/2f594341-147e-4352-8b2c-2a4c9ed49967/Movies/Downloads/complete/ct Magazin fuer Computertechnik 201624.par2
2016-11-16 13:59:25,781::INFO::[postproc:366] Running unpack_magic on ct Magazin fuer Computertechnik 201624.par2
2016-11-16 13:59:25,783::INFO::[newsunpack:254] Unrar starting on /media/2f594341-147e-4352-8b2c-2a4c9ed49967/Movies/Downloads/temp/ct Magazin fuer Computertechnik 201624.par2
2016-11-16 13:59:25,783::INFO::[newsunpack:475] Extracting rarfile /media/2f594341-147e-4352-8b2c-2a4c9ed49967/Movies/Downloads/temp/ct Magazin fuer Computertechnik 201624.par2/ct Magazin fuer Computertechnik 201624.rar (belonging to ct Magazin fuer Computertechnik 201624) to /media/2f594341-147e-4352-8b2c-2a4c9ed49967/Movies/Downloads/complete/_UNPACK_ct Magazin fuer Computertechnik 201624.par2
2016-11-16 13:59:26,157::INFO::[newsunpack:814] 2 Datei(en)/Ordner entpackt in 0 Sekunden
2016-11-16 13:59:26,158::INFO::[newsunpack:500] Deleting /media/2f594341-147e-4352-8b2c-2a4c9ed49967/Movies/Downloads/temp/ct Magazin fuer Computertechnik 201624.par2/ct Magazin fuer Computertechnik 201624.rar
2016-11-16 13:59:26,161::INFO::[newsunpack:258] Unrar finished on /media/2f594341-147e-4352-8b2c-2a4c9ed49967/Movies/Downloads/temp/ct Magazin fuer Computertechnik 201624.par2
2016-11-16 13:59:26,162::ERROR::[postproc:532] Nachbearbeitung von ct Magazin fuer Computertechnik 201624.par2 fehlgeschlagen ()
2016-11-16 13:59:26,162::INFO::[postproc:534] Traceback:
Traceback (most recent call last):
  File "/opt/SABnzbd/sabnzbd/postproc.py", line 368, in process_job
    unpack_error, newfiles = unpack_magic(nzo, short_path(workdir), short_complete, flag_delete, one_folder, (), (), (), (), ())
  File "/opt/SABnzbd/sabnzbd/newsunpack.py", line 295, in unpack_magic
    xjoinables, xzips, xrars, xsevens, xts, depth)
  File "/opt/SABnzbd/sabnzbd/newsunpack.py", line 233, in unpack_magic
    xjoinables, xzips, xrars, xsevens, xts = build_filelists(workdir, workdir_complete)
  File "/opt/SABnzbd/sabnzbd/newsunpack.py", line 1686, in build_filelists
    for root, dirs, files in os.walk(workdir_complete):
  File "/usr/lib/python2.7/os.py", line 286, in walk
    if isdir(join(top, name)):
  File "/usr/lib/python2.7/posixpath.py", line 80, in join
    path += '/' + b
UnicodeDecodeError: 'utf8' codec can't decode byte 0x81 in position 9: invalid start byte

Hab die Lösung!

Vorher:

LANG=de_AT.UTF-8
LANGUAGE=de_AT:de

Jetzt:

LANG=en_US.utf8
+ service sabnzbd restart

Gibts ne saubere Lösung auch?

Kann es sein das sabnzbd generell ein problem hat? Wenn ich mit meiner alten SSD boote (OMV 22.3) und dort das Raid (wo die daten runtergeladen und entpackt werden) mounte funktioniert´s.