can't reverse connect with encryption

[microshnik]

I'm spam-new to UltraVNC, just installed the latest version (1.8.2.4) on two 64-bit machines (Win11 server and Win10 viewer), and went through what I thought was all necessary setup of both to use SecureVNCPlugin64.dsm, but attempting to initiate an outgoing connection from the server results in a hard stop with error message "plugin has wrong arch (x86)". When I click OK to dismiss this, it is followed by an alert with warning message "The Plugin cannot be loaded. Please check its name integrity." When I click OK to dismiss the alert, I get an Information window with message "SecureVNC Plugin authentication failure (no plugin interface available)".

I went back and downloaded the encryptionplugins.zip and put the 64-bit version from there in place (even though it seems to be a solid decade old!) and the outgoing connection still fails in the same way; I also tried downgrading to 1.6.4.0 with the same result. When I initiate a connection from the viewer, it connects without issue unless "Use 56-Blowfish cipher for passphrase" is enabled, in which case the server crashes. An outgoing connection with just the VNC password and no encryption also works fine.

"Reverse VNC" is the only use I have for this software but I really need it to be encrypted, so please help me get this working. I also don't understand why the password or passphrase is still required with exactly one public/private key pair properly placed, so an explanation would be helpful; it was requested by someone else previously (https://forum.uvnc.com/viewtopic.php?p=112399#p112399) but never provided. Thank you in advance for both!

[RudiDeVos]

UltraVNC SecureVNC Plugin Testing
For simplicity, I run server and viewer on the same PC to avoid firewall issues on both PCs.

Step 1 — Test Normal (Direct) Connection
Server: Settings → Security, enable [✓] Use encryption and select SecureVNCPlugin64.dsm (the only plugin in the x64 folder, which is the correct one).
Restart the server — plugin activation requires a restart. (This should be investigated to see if it can be done without a restart.)
Don't configure the plugin yet — just enable encryption with the VNC password.
Viewer: Tab Security, enable [✓] Use encryption and select SecureVNCPlugin64.dsm (same plugin).
Connect.
Result: Status window shows AES-256... — connection works.
Step 2 — Test Reverse Connection
Viewer: Go to tab Listen and press Start Listening.
Server (still running in systray): Select Add New Connection, enter localhost, and connect.
Result: Status window shows AES-256... — reverse connection works.
Step 3 — Create Keys and Configure Plugin
Server: Tab Security → click Configure Encryption.
Set options:
[✓] AES 128
[✓] 256-bit
Set a passphrase (this allows setting a long password).
Click Generate Client Authentication Key — this generates two keys:
20260704_Server_ClientAuth.pubkey ← key for the server
20260704_Viewer_ClientAuth.pkey ← key for the viewer
The caption shows the ini file path: path\ultravnc.ini. Opening this file shows the configuration and long password (scrambled):


UseDSMPlugin=1
DSMPlugin=SecureVNCPlugin64.dsm
DSMPluginConfig=SecureVNC;0;0x00104001;blablabla
Step 4 — Retest with Keys
Repeat Step 2 (reverse connection).
You now need to enter the long password.
Result: Works — connection succeeds with the long password.
The status window shows the selected encryption options.
Both direct and indirect connections work with the configured plugin.

Note on Blowfish-only Mode
[ ] Use 56-bit Blowfish only — configuring the server via the viewer's configure button does not work. This must be done on the server side, as the server needs to know the selections and save them in its own ini file.

[microshnik]

Thank you so much for detailing the steps! I had been using the packaged Windows shortcut for "UltraVNC Viewer (Listen Mode Encrypt)" to start the viewer, and that seems to be the source of the failure. By opening the viewer normally and using the Listen tab as you described, it worked perfectly even without back-tracking from having the passphrase set. Given that the target of that shortcut is:

Code: Select all

vncviewer.exe -dsmplugin SecureVNCPlugin.dsm -listen 5500

is it possible to add to the command whatever other parameters are necessary for the listening viewer to be started with the proper encryption config?

Note on Blowfish-only Mode
[ ] Use 56-bit Blowfish only

This is not the phrasing the in the config dialog, so now I am even more confused as to its purpose! I did have it enabled in the server config, and that's when a normal connection from the viewer caused the server to crash, but I'm not sure if I actually need or want it. Could you please explain what this setting is intended to do?

I also don't understand why the password or passphrase is still required with exactly one public/private key pair properly placed

For anyone still wondering, the answer seems to be: it's a really good idea to make sure no one else can get access just by stealing the keys!

[RudiDeVos]

1) you need to enable encryption on server and viewer
2) server need to configure the encryption
3) and only, when you use a key, you need to copy the viewer key to the viewer PC

Is a passphrase needed when using a key.
No, when you use a key you can even disable the vnc password as the key is used for authentication and encryption.
In all other cases a passphrase is needed when you use vnc over the net, a simple 8 chat vnc password isn't save enough