But what about (security) updates if I disable backports? Enabling backports, looking for potential vbox updates, disabling backports every week is not very convenient.
I don't have a solution for this other than leaving it enabled.
Whats the reason for not using the official virtualbox.org repo for Debian?
The package provided by that repo is much larger and installs unneeded components (like gui client stuff). It is also horrible when it comes to upgrades. We used to use it and switched because of this.
Is there any difference apart from that small version delay?
Yes. Debian patches the code for various reasons and separates the packages like I mentioned above. You would have to look at the patch file for exact changes.