I had the same problem... It's not OpenMediaVault, it's Debian.
I tried to install Debian 9.8 from scratch, same error.
Like Darkarn said the guilty is the amount of ram.
I have 16GB and Debian tries to configure a swap partition of the same size, obviously it can not fit in a 8 or 16GB stick.
I removed 3 modules, and left just one (4GB) and both Debian 9.8 and OMV4 could install on USB flash 8GB flawless.
Thank you very much Darkarn to have pointed us on the right direction. Very smart of you! I was getting crazy for this error.