The Vinagre VNC viewer included in GNOME on Linux does not currently allow more than 8 characters in a password due to historical limitations of the VNC protocol.
Apple has somehow (and I don’t pretend to understand this) extended the protocol to use both a username and a password, including passwords longer than 8 characters.
Someone provided a patch to the Vinagre project to fix this over six months ago, but they blew it off. I went ahead and patched the source and rebuilt the package on my x86_64 Debian Wheezy/Sid system. You can download it from here, and go pester the developers to accept the patch in mainline.