IBM Connections Mail and Ephemeral Diffie-Hellman key size error

I’m building an IBM Connections 5.5 server to replace our internal Connections server and when configuring the Mail plug-in I came up against problems with the error “Mail server cannot be reached.”

1

The Domino iNotes server is configured to accept SSL and have SSLv3 disabled via DISABLE_SSLV3=1. SSO works in both directions between the two application servers.

I checked the discoveryservlet URL (https://connections.acme.com/connections/resources/discovery/DiscoveryServlet?email=ben.williams@chooseportal.com) which returned valid data so I know the configuration in socialmail-discovery-config.xml was good but there was very little to go on. Even after I enabled *=info:com.ibm.social.pim.discovery.*=all there was nothing much to go on.

I reached out and Michele Buccarello responded and pointed me towards one of his documents http://www.slideshare.net/michelebuccarello/connections-mail-with-exchange-backend. The document is written primarily for an Exchange server but it describes brilliantly what is happening and a bit of trace that came to my rescue.

I enabled *=info:com.ibm.social.pim.discovery.*=all:com.ibm.cre.*=all and all of a sudden I saw what was happening.

[7/12/16 13:49:33:787 BST] 00000220 CREURLConnect 2   An unhandled exception occured connecting to the target host
                                 javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair

Caused by: java.lang.RuntimeException: Could not generate DH keypair

Caused by: java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 256 to 2048 (inclusive)

I read around the various ciphers and I must admit I was a little lost and it’s been a while since I’ve delved deeply into Domino but some googling got me to some Daniel Nashed blogs.

http://blog.nashcom.de/nashcomblog.nsf/dx/first-perfect-forward-secrecy-ciphers-shipped-with-9.0.1-fp3-if2.htm?opendocument&comments#anc1

http://blog.nashcom.de/nashcomblog.nsf/dx/dha-with-more-than-1024-key-size-and-java-still-works.htm?opendocument&comments#anc1

The second had a comment about the Mail plug-in not working so I knew I was getting closer. This put various stackoverflow posts into perspective such as

http://stackoverflow.com/questions/6851461/java-why-does-ssl-handshake-give-could-not-generate-dh-keypair-exception

I stopped Domino and added the following before starting it again and the plug-in started working and I could access my mail and calendar.

SSL_DH_KEYSIZE=1024

I upped the value to 2048 since a previous error said “Prime size must be multiple of 64, and can only range from 256 to 2048 (inclusive).”

On restart of Domino it continued to work. I tried increasing the value to 3072 but this broke the plug-in.

The certificate I was provided was a 4096 bit certificate and not 2048 like I handle more often.

In https://www-10.lotus.com/ldd/dominowiki.nsf/dx/TLS_Cipher_Configuration it states, “By default, these ciphers will use a DH key with a size equivalent to the RSA keysize, so a server running with a 2048 bit SSL certificate would use a 2048 bit DH group.” This means that the DH key being used is 4096 which IBM’s implementation of Java doesn’t support, hence the need to add SSL_DH_KEYSIZE=2048.

I then found the following Domino trace.

DEBUG_SSL_CIPHERS=2
DEBUG_SSL_DHE=2
DEBUG_SSL_HANDSHAKE=2
DEBUG_SSL_IO=0

When I recreate the problem I see in the console.log the following which shows the DH key size.

[11856:00011-1753671424] 07/12/2016 01:50:03.16 PM SSLEncodeDHKeyParams> Server RSA key size 4096 bits
[11856:00011-1753671424] 07/12/2016 01:50:03.16 PM SSLEncodeDHKeyParams> Using a DH key size of 4096 bits
[11856:00011-1753671424] 07/12/2016 01:50:03.26 PM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeServerHelloDone
[11856:00011-1753671424] 07/12/2016 01:50:03.26 PM SSLAdvanceHandshake Exit> State HandshakeClientKeyExchange (11)
[11856:00011-1753671424] 07/12/2016 01:50:03.26 PM SSL_Handshake> After handshake state = HandshakeClientKeyExchange (11); Status = -5000
[11856:00011-1753671424] 07/12/2016 01:50:03.26 PM int_MapSSLError> Mapping SSL error -5000 to 4176 [SSLHandshakeNoDone]
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM SSLProcessProtocolMessage> Record Content: Alert (21)
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM SSLProcessAlert> Got an alert of 0x50 (internal_error) level 0x2 (fatal)
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM SSL_Handshake> After handshake2 state HandshakeClientKeyExchange (11)
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM SSL_Handshake> SSL Error: -6994
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM int_MapSSLError> Mapping SSL error -6994 to 4171 [SSLFatalAlert]

In https://www-10.lotus.com/ldd/dominowiki.nsf/dx/TLS_Cipher_Configuration it also states, “When using Domino 9.0.1 FP3 IF2 one can and should disable DHE_RSA_WITH_AES_128_CBC_SHA (33) which should make those old clients fall back to using RSA_WITH_AES_128_CBC_SHA (2F) instead.”

I tried the below setting which removes “33” to see whether it worked but it did not. I would like to fiddle more with this to try and find a cipher that WAS and Domino can use in common that avoids setting the DH key too low but I suspect I will run out of time.

SSLCIPHERSPEC=9D9C3D3C352F0A39676B9E9F

BTW – I did all this after I had forced TLS1.2 via How to Force IBM Connections 5.5 CR1 to Use TLSv1.2 which is nice to know that Mail is not broken after enforcing TLS1.2 unlike Textbox.io and Surveys…..

Oh, in Domino when it is successful it will look something like this.

[17825:00011-575325952] 07/12/2016 03:12:00.31 PM SSLEncodeDHKeyParams> Server RSA key size 4096 bits
[17825:00011-575325952] 07/12/2016 03:12:00.31 PM SSLEncodeDHKeyParams> Using a DH key size of 2048 bits
[17825:00011-575325952] 07/12/2016 03:12:00.32 PM SSLEncodeRSAServerKeyExchange> Signing ServerKeyExchange using RSAWithSHA256
[17825:00011-575325952] 07/12/2016 03:12:00.36 PM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeServerHelloDone

[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> After handshake2 state HandshakeServerIdle (3)
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> Protocol Version = TLS1.2 (0x303)
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> Cipher = DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> KeySize = 256 bits
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> Ephemeral Diffie-Hellman key size = 2048 bits
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> Server RSA key size = 4096 bits
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> TLS/SSL Handshake completed successfully

You won’t see much difference in trace.log.

If anyone has a better way to get around this without changing the value of the DH key size then please shout.

Additional information

I should mention that a useful tip Michele Buccarello pointed me towards taking a Fiddler trace.

You’ll see a call to /connections/opensocial/gadgets/makeRequest. Within that entry in Fiddler I saw 502 Bad Gateway

throw 1; < ‘invalid javascript’ >{“https://webmail.acme.com/mail/bwilliam.nsf/iNotes/Proxy/?OpenDocument&Form=f_SessionInfo_Data&_icmb=20160425-0501&#8221;:{“rc”:502,”body”:”&amp;amp;lt;HTML&amp;amp;gt;&amp;amp;lt;TITLE&amp;amp;gt;502&amp;amp;nbsp;-&amp;amp;nbsp;Bad&amp;amp;nbsp;Gateway&amp;amp;lt;/TITLE&amp;amp;gt;&amp;amp;lt;BODY&amp;amp;gt;&amp;amp;lt;h1&amp;amp;gt;502&amp;amp;nbsp;An&amp;amp;nbsp;unhandled&amp;amp;nbsp;exception&amp;amp;nbsp;occured&amp;amp;nbsp;connecting&amp;amp;nbsp;to&amp;amp;nbsp;the&amp;amp;nbsp;target&amp;amp;nbsp;host&amp;amp;lt;/h1&amp;amp;gt;&amp;amp;lt;/BODY&amp;amp;gt;&amp;amp;lt;/HTML&amp;amp;gt;”,”headers”:{“date”:[“Mon, 11 Jul 2016 21:30:46 GMT”],”content-type”:[“text/html; charset=UTF-8″]},”DataHash”:”jslu7s57e7d899jbtr7p1d033g”}}

You can also look at the JSON section to see it in a different format.

The above is also seen in the trace.log with *=info:com.ibm.social.pim.discovery.*=all:com.ibm.cre.*=all

[7/12/16 8:49:30:670 BST] 000001bb CREURLConnect 2   IOException caught, response code is 502, Exception was java.io.IOException: Server returned HTTP response code: 502 for URL: https://webmail.acme.com/mail/bwilliam.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries_JSON&PresetFields=FolderName;($Inbox),UnreadOnly;1,UnreadCountInfo;1,hc;$98&Count=1&resortdescendingpn=$70&TZType=UTC&KeyType=time&_icmb=20160425-0501
[7/12/16 8:49:30:671 BST] 000001bb CREURLConnect 2   Retry error while in streaming mode: 502, java.io.IOException: Server returned HTTP response code: 502 for URL: https://webmail.acme.com/mail/bwilliam.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries_JSON&PresetFields=FolderName;($Inbox),UnreadOnly;1,UnreadCountInfo;1,hc;$98&Count=1&resortdescendingpn=$70&TZType=UTC&KeyType=time&_icmb=20160425-0501

Advertisements