Ok, thank you again. I feel quite dumb now.
I checked the sata cables and changed the ports on the mobo - no change.
I postponed the change of the sector size, as I was not quite sure how to do it with ZFS.
Then I deleted the zfs pool and checked the drives as single drive individually. The both performed normaly so I assumed the drives are fine.
After this I set up the mirror pool again. While activating compression and dedupe again, I thought maybe they were the culprits.
Disabling compression - no change, the drops in speed while copying were there again.
But after disabling dedupe it turned stable instantly. I first checked with a 74gb image, then with a mixed video/picture folder of 100gb.
They were stable, so I started the 2,2tb transfer via usb again. 5,5 hours later it was done.
After a bit of additional research it seems like that dedupe is rather heavy on the hardware. As my nas system is rather old and low on ram, it was just to much for the system.