I was enable to identify the issue.
iotop showed the smb daemon consuming 100% io for a very short time. This occurs randomly and always happens when my videos freeze.
Since I'm not to keen on troubleshooting smb I now use Plex. Plex also allows you to serve the untranscoded video files to clients. Using Plex everything works fine.
Still seems very strange to me that this particular disk has issues with smb (io), and my other disks don't. If you happen to know the explanation, please let me know.