Huge Speed Drops with Raid 1 and Luks encryption

Hello everybody,

I have an issue where i can´t find the reason:

i build a small NAS a few weeks ago, containing following parts:

-AS Rock J5040 Board with embedded Pentium Gold J5040 Quadcore (AES NI capable) (Speedstep and turbo in bios enabled)

-250 GB SSD for the OS

-a 1 Gig Network

-Two WD Red Plus NAS (CMR) 4TB Drives for the Data

I used the newest Version of OMV6 with the additional Luks Encryption Plugin. I use a Raid 1 Array with the two 4 TB Drives. Then i set up an encrypted Device, unlock and mount and then create a EXT4 File System, set datafolders and SMB Shares. Everything work as expected. Beside that i have USB Backup (not in use), CPU Temp Plugin, Autoshutdown (not in use) and diskstats installed. Otherwise no fancy stuff nor i did any changes to the main debian.

But i have an annoying issue: When writing files to the NAS, i receive full speed (110-115 mb/s), but after 5 to 15 seconds the transferrate get a drop to about 40-70 for a short time, gets back to the 110ish and after 5 to 15 seconds it drops to 0 for around 5-10 seconds. Then it goes back again to the 100ish, a few seconds, and it drops to zero again, and so on. This goes on until the transfer is done. I´m trying with large .img or .iso files from 8 to 24 GB. CPU Usage rise to about 60-75% when writing.

I thought it was a Network/SMB Issue, but i can exclude this:

Read Speeds are continous around 110-115. I also mounted another (unencrypted) Drive via USB. Again, i can reach full speed in read and write, continous.

The WD Red where brand new, SMART Values seems good. I try to disable/enable write cache or changed the power management of the drive, no difference.

I changed the SATA channels from the Drives as well (both on the same controller), as the System have two CPU controlled Channels and two externaly controlled, the same: no difference.

I suspect it has something to do with the encryption or the RAID 1.

Anyone saw this strange behavior before?

Best regards and have a nice weekend!

Hi,

i can now say, that the speeddrop issue is caused by the encryption:

I delete the encryption and set up a new ext4 on top of the raid 1 without encryption. And im receiving a completly constant write (read, too) from about 105 to 115 MB/s; testes with 8 and 100 GB .iso files.

So, why does the encryption has such a huge negative impact? The CPU is a quadcore with native AES support and should easily handle the low traffic from 112MB/s. With encryption and large files, effectivly i just get a transferrate from about 50-70 MB/s.

I used encryption on other systems before, without such a huge impact.

Is there anything i can do, to improve the performance?

Regards

Edit: Cryptsetup Benchmark shows ~1460 MB/s for write/read for aes-xts (which was used)