I just got encryption working 5 minutes ago. :D You need to use the msrc4plugin_noreg.dsm plugin file for this to work and bundle that file into your custom.zip file.
BTW, does anyone know why rc4.key has to be put into C:\program files\ultravnc? Even if I use the keygen and specify other ...