IBM Connections Files plugin not working within Notes when TLSv1.2 is enforced

After enforcing TLSv1.2 on our internal Connections 5.5 servers the Files plugin would not work.

In the IHS logs I would see errors such as

[warn] [client 80.229.222.90] [7f9a700a7060] [21173] SSL0222W: SSL Handshake Failed, No ciphers specified (no shared ciphers or no shared protocols). [xx.xx.xx.xx:62899 -> xxx.xxx.xxx.xxx:443] [09:45:11.000102454] 0ms

Enabling trace on IHS showed that the protocol being used was TLSv1.0 which matched Wireshark output. Oddly Status Updates and Activities plugins use TLSv1.2.

“GET /files/basic/api/library/4a7a7240-8f68-44d8-9447-7410cc2bb467/feed?pageSize=300&acls=true&sI=601 HTTP/1.1” 200 168770 TLS_RSA_WITH_AES_128_CBC_SHA TLSV1

I then had to allow TLSv1.0 until I could get an explanation from IBM.

Finally IBM came back with the following two lines to be added to the notes.ini.

SSL_DISABLE_TLS_10
DISABLE_SSLV3=1

Now in access_log I see TLSv1.2 being used.

“GET /files/basic/api/library/4a7a7240-8f68-44d8-9447-7410cc2bb467/feed?pageSize=300&acls=true&sI=601 HTTP/1.1” 200 168770 TLS_RSA_WITH_AES_128_GCM_SHA256 TLSV1.2

IBM also suggested that I check the following was set in plugin_customization.ini, which it was.

com.ibm.documents.connector.service/ENABLE_SSL=true

The notes.ini values have been pushed out to my colleagues via Domino policies.

Old version of Notes Java breaks IBM Connections Files plugin when TLSv1.2 is enforced

I had to raise a PMR on a problem I and others in my company had with the Notes client. After enforcing TLSv1.2 in Connections 5.5 using the following configuration in httpd.conf the Files plugin would not work but the Activities and Status Updates plugins would.

SSLEnable
SSLProtocolDisable SSLv2 SSLv3 TLSv11 TLSv10
SSLCipherSpec TLSv12 +TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA +TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 +TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 +TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 +TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

I kept seeing the following screen and clicking “try again using existing options” did nothing.

Whilst clicking on “try again using existing options” I would see the following in IHS.

[Wed Mar 29 14:30:41 2017] [warn] [client xxx.xxx.xxx.xx] [7f9a480ec800] [30453] SSL0222W: SSL Handshake Failed, No ciphers specified (no shared ciphers or no shared protocols).  [xxx.xxx.xxx.xx:49296 -> xxx.xxx.xxx.xx:443] [14:30:41.000571168] 0ms

The SSL certificate is at 4096 bits and I had previously replaced US_export_policy.jar and local_policy.jar with the unrestricted policy jars so that was not the problem.

I found, oddly, that if I swapped to the IBM Sametime Meetings plugin first and then changed to Files, my files would load…. Also, if I ran Fiddler and restarted my Notes client but went directly to Files it would load too. Weird.

I had a screen share with Elizabeth Hecht and Jacqueline Chewens to show them the odd behaviour and they too were baffled. Liz came across a thought of the version of Java being used may not be allowing connectivity to Files and asked whether I had applied the Java update for FP6? Not having so much focus on Notes and Domino of late I told her I wasn’t even aware that previously you were supposed to update the version of Java being used by the Notes client.

To test this I updated Notes to FP8, which bundles in the Java update and low and behold the Files plugin started working. Also, there was no need to replace the jars with the unrestricted ones!

The version of Java now in play is as follows.

c:\Program Files (x86)\IBM\Notes\jvm>java -version
java version “1.8.0_121”
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) Client VM (build 25.121-b13, mixed mode, sharing)

BTW – if Connections enforces SSL then you need to make sure that com.ibm.documents.connector.service/ENABLE_SSL=true is set in the plugin_customization.ini.

SSL certificates and TLSv1.2 for Sametime (but also valid for WebSphere)

I thought I’d write this entry after assisting a peer and struggling myself to work out why TLSv1.2 was not working for a given node.

I will detail how to add a wildcard certificate to a Sametime 9.0.1 cell and then how to enforce TLSv1.2 for Sametime Proxy and Meeting server nodes.

Import the SSL certificate

There are various ways to go about this but I will detail using a .p12 file (pcks#12 format). The nice thing about getting a .p12 file is that all the certificates should be in there, all intermediary and the root protected by a password.

There are ways to create .p12 files using openSSL and Google is awash with posts so I won’t go into any more detail.

You will want to export the intermediary and root certificates. You can view the contents of the .p12 using openSSL. I am running Cygwin on a Windows laptop hence the .exe.

openssl.exe pkcs12 -in ./wild_acme_com.p12 -info

This will allow you to copy and paste the intermediary and root certificates which are needed. Again there are commands to export the certificates are available from Google or you could down load them from the Certificate Authority (CA).

Once you have your .p12 and intermediary and root certificates log into the ISC and go to SSL certificate and key management > Key stores and certificates > CellDefaultKeyStore > Personal certificates.

Click Add and add the intermediary and root certificates.

Now go to SSL certificate and key management > Key stores and certificates > CellDefaultKeyStore > Personal certificates > Import and click on key store file.

Point it to your .p12 and enter the password. It will then read the contents and give you a ridiculous name for an alias. I suggest you enter something meaningful. Then press apply.

1

At which point you will see the chain in SSL certificate and key management > Key stores and certificates > CellDefaultKeyStore > Personal certificates which should look something like this

3

You can see the chain is complete. This is important otherwise web browsers will show various types of untrusted errors.

If you haven’t done this already you will need to apply the certificate to the nodes that need it.

Go to SSL certificate and key management > Manage endpoint security configurations.

From here you will need to expand the Inbound and Outbound sections for the STProxy and Meeting nodes. If you have a WebSphere proxy in front you will need to apply the certificate to that server. You can also add the certificate to the STProxy or Meeting application server too in case you have users connecting directly.

You need to tick Override inherited values and then press Update certificate alias list at which point in the Certificate alias in key store you should see the alias for the imported .p12. Remember to repeat for both Inbound and Outbound.

4

Now normally you would stop all application servers, WAS proxies, node agents and then the deployment manager and start them back up but because we are enabling TLSv1.2 we need to do a little more…..

TLSv1.2

If you try to enforce TLSv1.2 on a SIP Proxy Registrar then it will not work properly and you’ll get messages like the following when clients try to connect.

[10/12/16 10:37:24:483 BST] 0000008e TelephonyServ I   UserName in Message 
is null
[10/12/16 10:37:31:278 BST] 000000ba SSLHandshakeE E   SSLC0008E: Unable 
to initialize SSL connection.  Unauthorized access was denied or security 
settings have expired.  Exception is javax.net.ssl.SSLHandshakeException: 
Client requested protocol TLSv1 not enabled or not supported

This means that using SSL certificate and key management > Key stores and certificates > CellDefaultKeyStore to control the protocol will not work because it will apply to all application servers in the cell including the SIP Proxy Registrar.

If you have awareness and meetings only then you can get away with it, although you need to take special care with recording of meetings because that will not work if you enforce TLSv1.2. In this case you may need to run the following to add the TLS configuration for recording.

"INSERT INTO mtg.configuration (server_id, CONFIGURATION_KEY,
CONFIGURATION_VALUE) values ('<substitue your server id here>',
'meeting.recording.tlsVersion','TLSv1.2')"

Limitations

Before I go on I will explain what limitation I found. If I enforce TLSv1.2 on the Meeting server I cannot connect to it using a Sametime  (thick) client. Web browser and mobile apps work fine. In the thick client it will not connect and I get errors in the client logs.

The default in QoP is SSL_TLS which enables all SSL V3.0 and TLS 1.0 protocols. This is not terribly useful considering I want to use TLSv1.2 but cannot enforce it across all the cell. You can use SSL_TLSv2 which enables all SSL V3.0 and TLS 1.0, 1.1 and 1.2 protocols so at least I have the option of using TLSv1.2 if the client uses that protocol.

So my steps involve some application servers using SSL_TLS, most using SSL_TLSv2 and the Sametime Proxy using TLSv1.2.

Remember I have WebSphere proxies fronting STProxy and Meeting servers to host HTTP -> HTTPS redirection and I will use them as the TLSv1.2 point.

Import p.12 to NodeDefaultKeyStore

So the steps are threefold, 1) add the .p12 certificate to the STProxy server node, 2) set the node to use the NodeDefaultKeyStore and 3) enforce TLSv1.2.

As I have run through the steps to import the certificate to the cell I do not need to run through that again. You need to go to SSL certificate and key management > Key stores and certificates > NodeDefaultKeyStore > Personal certificates/Signer certificates (choosing the node for STProxy) and repeat the steps above.

Now go back to SSL certificate and key management > Manage endpoint security configurations and go to the Inbound and Outbound sections. I made the change on the WebSphere proxy that fronts STProxy.

Change SSL configuration NodeDefaultSSLSettings click update certificate alias list at which point in the Certificate alias in key store you can select the alias you set. Repeat as required.

5

It will then look something like this. Only was_stpProxy is using the NodeDefaultSSLSettings, all others are using the default, CellDefaultSSLSettings.

10

The reason why you have done this is important in the next section.

Enforce TLSv1.2

I suggest you stop all the application servers, WebSphere proxies, node agents at this point.

Now you need to enforce TLSv1.2 at the node level. Go to SSL certificate and key management > SSL configurations > NodeDefaultSSLSettings (for STProxy) > Quality of protection (QoP) settings and change Protocol from SSL_TLS to TLSv1.2.

6

Go to SSL certificate and key management > SSL configurations and for all the other nodes including CellDefaultSSLSettings and XDADefaultSSLSettings set the Protocol to be SSL_TLSv2 including the SIP Proxy Registrar.

On all the nodes find the ssl.client.props file which is somewhere like /opt/IBM/WebSphere/AppServer/profiles/hostSTPPNProfile1/properties/ssl.client.props on Linux.

Ensure this is set as the following default value

com.ibm.ssl.protocol=SSL_TLSv2

This file instructs the client (the node agent) what protocol to communicate with the deployment manager using. As you have set this protocol in QoP for the cell, all nodes (apart from STProxy) and XDADefaultSSLSettings then all node agents can talk freely to the deployment manager.

If you miss a step here you’ll see from the deployment manager’s SystemOut.log that the node agent seems to stop and then start repeatedly. This is because the node agent cannot communicate properly, mainly because you have not changed XDADefaultSSLSettings appropriately.

Stop and start the deployment manager, run syncNode on all nodes and start the node agents, application servers and proxies and test. Check the SystemOut.log for any exceptions and if you see them check your configuration.

Ciphers

If you run a test against your STProxy or Meeting servers you’ll get marked down for the weak ciphers.

11

You can remove these from SSL certificate and key management > SSL configurations > NodeDefaultSSLSettings > Quality of protection (QoP) settings > Cipher suite settings. You will need to change from Strong to custom and then remove the ciphers listed above, if you so wish.

If you plan to do this for the Meeting server as well as STProxy then you will need to change the Inbound and Outbound options for the WebSphere proxy in front of Meetings so that it uses the NodedefaultSSLSettings which allows you to then use a default set of ciphers.

Finally

I have created a PMR to ask IBM about their support for TLSv1.2 in Sametime. I’ll update things once I get a response.

Forcing TLSv1.2 breaks IBM Connections Surveys and Textbox.io

I had to force TLSv1.2 across all of Connections to fix a problem with RTE as I detailed in Rich Content widget widget stops working due to mix matched SSL protocols but after testing I’ve found that this breaks Textbox.io in Chrome and Surveys.

The process is well documented in How to Force IBM Connections 5.5 CR1 to Use TLSv1.2 but after making the changes the following happens.

Textbox.io

In IE and FF Textbox.io works fine but in Chrome the spell check service fails.

1

In Fiddler trace is saw Spelling server error:  Could not load url “https://connections.acme.com/ephox-spelling/1/correction&#8221;: 500 Internal Server Error

In the SystemOut.log I saw

[6/29/16 10:00:38:507 BST] 00000200 SystemOut     O ironbark-akka.actor.default-dispatcher-17, RECV TLSv1 ALERT:  fatal, handshake_failure
[6/29/16 10:00:38:507 BST] 00000200 SystemOut     O ironbark-akka.actor.default-dispatcher-17, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Received fatal alert: handshake_failure

spray.can.Http$ConnectionException: Aborted
    at spray.can.client.HttpHostConnectionSlot.reportDisconnection(HttpHostConnectionSlot.scala:228) ~[spray-can_2.11-1.3.3.jar:na]
    at spray.can.client.HttpHostConnectionSlot$$anonfun$connected$1.applyOrElse(HttpHostConnectionSlot.scala:161) ~[spray-can_2.11-1.3.3.jar:na]
    at akka.actor.Actor$class.aroundReceive(Actor.scala:465) ~[akka-actor_2.11-2.3.9.jar:na]

IBM asked me to put on SSL trace, *=info:SSL=all. It seems that the client is sending TLSv1.0 which of course is not allowed now TLSv1.2 has been forced.

[7/11/16 9:31:17:286 BST] 00000115 SystemOut     O   ironbark-akka.actor.default-dispatcher-7, READ: TLSv1.2 Alert, length = 2
[7/11/16 9:31:17:286 BST] 00000115 SystemOut     O   ironbark-akka.actor.default-dispatcher-7, RECV TLSv1 ALERT:  fatal, handshake_failure
[7/11/16 9:31:17:286 BST] 00000115 SystemOut     O   ironbark-akka.actor.default-dispatcher-7, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Received fatal alert: handshake_failure

IBM have logged a ticket with Ephox as well as investigating it from there end.

Surveys

When in a community with previous surveys I can not see any of the historical surveys nor could I create new ones.

In the SystemOut.log I saw the following

[6/29/16 10:01:56:542 BST] 0000033b StandardExcep E com.ibm.form.nitro.platform.StandardExceptionMapper toResponse eaa8e54e-7c38-4edb-a5ca-bcbd6d7f6c64
                                 com.ibm.form.platform.service.framework.exception.ServicesPlatformException: com.ibm.connections.directory.services.exception.DSException: com.ibm.connections.directory.services.exception.DSOutOfServiceException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Caused by: com.ibm.connections.directory.services.exception.DSException: com.ibm.connections.directory.services.exception.DSOutOfServiceException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Again, it seems like it’s sending TLSv1.0.

There’s at least one other person I know of who’s logged a PMR for these problems. It’s fairly urgent due to a problem with the RTE application which is only fixed when TLS1.2 is enforced. I’m hoping that these problems can be resolved sharpish so I can resolve the RTE problem for a customer.