iOS push notifications not received due to expired APNs certificates

I was finding in a Connections 5.0 CR4 environment push notifications were not being received on iOS devices but they were for Android.

Errors

I saw the following errors and similar on apps server startup

3/7/17 10:49:16:874 GMT] 00002240 ApnsConnectio I com.notnoop.apns.internal.ApnsConnectionImpl sendMessage Failed to send message Message(Id=1; Token=A907EA2FBD34C6E222B9176FE65AA51576D5100713F3E6061A288F42013287B8; Payload={“1″:”joe.bloggs@dev-acme.com”,”2″:null,”3″:null,”aps”:{“badge”:2,”alert”:{“loc-key”:”N1″,”loc-args”:[“JOE BLOGGS”]},”sound”:”chimes”},”4″:2,”5″:”1″})… trying again after delay
javax.net.ssl.SSLException: Received fatal alert: internal_error
at com.ibm.jsse2.p.a(p.java:20)
at com.ibm.jsse2.p.a(p.java:23)
at com.ibm.jsse2.SSLSocketImpl.b(SSLSocketImpl.java:789)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:397)
at com.ibm.jsse2.SSLSocketImpl.h(SSLSocketImpl.java:320)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:609)
at com.ibm.jsse2.l.write(l.java:24)
at java.io.OutputStream.write(OutputStream.java:69)
at com.notnoop.apns.internal.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:268)
at com.notnoop.apns.internal.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:258)
at com.notnoop.apns.internal.ApnsPooledConnection$2.run(ApnsPooledConnection.java:77)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:790)

These error look network or security related.

Network tests

My tests showed that the network is open.

# telnet gateway.push.apple.com 2195
Trying 17.188.142.148…
Connected to gateway.push.apple.com.
Escape character is ‘^]’.
^]
telnet> quit
Connection closed.
#  telnet feedback.push.apple.com 2196
Trying 17.188.129.25…
Connected to feedback.push.apple.com.
Escape character is ‘^]’.

This is a Java application provided by IBM to test connection to APNs for Sametime servers.

# /opt/IBM/WebSphere/AppServer/java/bin/java -jar ./07700.019.866.cert.jar apns -t -f ./apns-prod.pkcs12
IssuerDN: CN=Apple Worldwide Developer Relations Certification Authority, OU=Apple Worldwide Developer Relations, O=Apple Inc., C=US
SubjectDN: C=US, O=IBM, OU=RBVH72H5WP, CN=Apple Push Services: com.ibm.lotus.sametime, UID=com.ibm.lotus.sametime
From (yyyy-mm-dd): 2016-04-04
To (yyyy-mm-dd): 2017-05-04
MD5: B8:D3:2E:B3:42:04:D5:26:A9:63:68:30:00:15:CA:18:78:A9:AE:20

testing connection to gateway.push.apple.com:2195
passed

testing connection to feedback.push.apple.com:2196
passed

Security tests

I copied /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/Cell01/Mobile.ear/mobile.web.war/WEB-INF/lib/mobile.web.jar locally

I unzipped mobile.web.jar and found \mobile.web\com\ibm\lotus\connections\mobile\push\ios\PushNotificationService.class

I pasted PushNotificationService.class into http://www.javadecompilers.com having failed to get a local application working

The decompiled file lists the certificates as well as the passwords for each .p12

I downloaded the certificates from /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/Cell01/Mobile.ear/mobile.web.war/WEB-INF/classes/certificates/ locally although you could do the following from your server.

Locally, I then ran the following to retrieve the public certificate

$ openssl.exe pkcs12 -in ConnectionsEnterpriseAPNS.p12 -clcerts -nokeys -out ConnectionsEnterpriseAPNSpubliccert.crt
$ openssl.exe pkcs12 -in ConnectionsProductionAPNS.p12 -clcerts -nokeys -out ConnectionsProductionAPNSpubliccert.crt

You can open the certificates or use $ openssl.exe x509 -in ConnectionsProductionAPNSpubliccert.crt -text to get the information on them. I found that the expiration had passed.

        Issuer: C=US, O=Apple Inc., OU=Apple Worldwide Developer Relations, CN=Apple Worldwide Developer Relations Certification Authority
Validity
Not Before: Jan  6 18:08:26 2016 GMT
Not After : Feb  4 18:08:26 2017 GMT
Subject: UID=com.ibm.lotus.connections, CN=Apple Push Services: com.ibm.lotus.connections, OU=RBVH72H5WP, O=IBM, C=US

Remediation

As this is a Connections 5.0 CR4 server and the fix list states that LO87599 (Updated Connections Mobile APNS Certificates) was included. I’m not sure why the certificates were not replaced as the CR4 logs showed BUILD SUCCESSFUL….

I created a PMR and IBM quickly sent me LO90462: UPDATE TO APNS CERTIFICATES FOR 5.5 CR1

After applying the ifix (for 5.0 CR4) I see that PushNotificationService.class has changed matching the file used in Connections 5.5 and the expiration dates are now 20th October 2017. On restart push notifications are now working to iOS devices.

SSL certificates and TLSv1.2 for Sametime (but also valid for WebSphere)

I thought I’d write this entry after assisting a peer and struggling myself to work out why TLSv1.2 was not working for a given node.

I will detail how to add a wildcard certificate to a Sametime 9.0.1 cell and then how to enforce TLSv1.2 for Sametime Proxy and Meeting server nodes.

Import the SSL certificate

There are various ways to go about this but I will detail using a .p12 file (pcks#12 format). The nice thing about getting a .p12 file is that all the certificates should be in there, all intermediary and the root protected by a password.

There are ways to create .p12 files using openSSL and Google is awash with posts so I won’t go into any more detail.

You will want to export the intermediary and root certificates. You can view the contents of the .p12 using openSSL. I am running Cygwin on a Windows laptop hence the .exe.

openssl.exe pkcs12 -in ./wild_acme_com.p12 -info

This will allow you to copy and paste the intermediary and root certificates which are needed. Again there are commands to export the certificates are available from Google or you could down load them from the Certificate Authority (CA).

Once you have your .p12 and intermediary and root certificates log into the ISC and go to SSL certificate and key management > Key stores and certificates > CellDefaultKeyStore > Personal certificates.

Click Add and add the intermediary and root certificates.

Now go to SSL certificate and key management > Key stores and certificates > CellDefaultKeyStore > Personal certificates > Import and click on key store file.

Point it to your .p12 and enter the password. It will then read the contents and give you a ridiculous name for an alias. I suggest you enter something meaningful. Then press apply.

1

At which point you will see the chain in SSL certificate and key management > Key stores and certificates > CellDefaultKeyStore > Personal certificates which should look something like this

3

You can see the chain is complete. This is important otherwise web browsers will show various types of untrusted errors.

If you haven’t done this already you will need to apply the certificate to the nodes that need it.

Go to SSL certificate and key management > Manage endpoint security configurations.

From here you will need to expand the Inbound and Outbound sections for the STProxy and Meeting nodes. If you have a WebSphere proxy in front you will need to apply the certificate to that server. You can also add the certificate to the STProxy or Meeting application server too in case you have users connecting directly.

You need to tick Override inherited values and then press Update certificate alias list at which point in the Certificate alias in key store you should see the alias for the imported .p12. Remember to repeat for both Inbound and Outbound.

4

Now normally you would stop all application servers, WAS proxies, node agents and then the deployment manager and start them back up but because we are enabling TLSv1.2 we need to do a little more…..

TLSv1.2

If you try to enforce TLSv1.2 on a SIP Proxy Registrar then it will not work properly and you’ll get messages like the following when clients try to connect.

[10/12/16 10:37:24:483 BST] 0000008e TelephonyServ I   UserName in Message 
is null
[10/12/16 10:37:31:278 BST] 000000ba SSLHandshakeE E   SSLC0008E: Unable 
to initialize SSL connection.  Unauthorized access was denied or security 
settings have expired.  Exception is javax.net.ssl.SSLHandshakeException: 
Client requested protocol TLSv1 not enabled or not supported

This means that using SSL certificate and key management > Key stores and certificates > CellDefaultKeyStore to control the protocol will not work because it will apply to all application servers in the cell including the SIP Proxy Registrar.

If you have awareness and meetings only then you can get away with it, although you need to take special care with recording of meetings because that will not work if you enforce TLSv1.2. In this case you may need to run the following to add the TLS configuration for recording.

"INSERT INTO mtg.configuration (server_id, CONFIGURATION_KEY,
CONFIGURATION_VALUE) values ('<substitue your server id here>',
'meeting.recording.tlsVersion','TLSv1.2')"

Limitations

Before I go on I will explain what limitation I found. If I enforce TLSv1.2 on the Meeting server I cannot connect to it using a Sametime  (thick) client. Web browser and mobile apps work fine. In the thick client it will not connect and I get errors in the client logs.

The default in QoP is SSL_TLS which enables all SSL V3.0 and TLS 1.0 protocols. This is not terribly useful considering I want to use TLSv1.2 but cannot enforce it across all the cell. You can use SSL_TLSv2 which enables all SSL V3.0 and TLS 1.0, 1.1 and 1.2 protocols so at least I have the option of using TLSv1.2 if the client uses that protocol.

So my steps involve some application servers using SSL_TLS, most using SSL_TLSv2 and the Sametime Proxy using TLSv1.2.

Remember I have WebSphere proxies fronting STProxy and Meeting servers to host HTTP -> HTTPS redirection and I will use them as the TLSv1.2 point.

Import p.12 to NodeDefaultKeyStore

So the steps are threefold, 1) add the .p12 certificate to the STProxy server node, 2) set the node to use the NodeDefaultKeyStore and 3) enforce TLSv1.2.

As I have run through the steps to import the certificate to the cell I do not need to run through that again. You need to go to SSL certificate and key management > Key stores and certificates > NodeDefaultKeyStore > Personal certificates/Signer certificates (choosing the node for STProxy) and repeat the steps above.

Now go back to SSL certificate and key management > Manage endpoint security configurations and go to the Inbound and Outbound sections. I made the change on the WebSphere proxy that fronts STProxy.

Change SSL configuration NodeDefaultSSLSettings click update certificate alias list at which point in the Certificate alias in key store you can select the alias you set. Repeat as required.

5

It will then look something like this. Only was_stpProxy is using the NodeDefaultSSLSettings, all others are using the default, CellDefaultSSLSettings.

10

The reason why you have done this is important in the next section.

Enforce TLSv1.2

I suggest you stop all the application servers, WebSphere proxies, node agents at this point.

Now you need to enforce TLSv1.2 at the node level. Go to SSL certificate and key management > SSL configurations > NodeDefaultSSLSettings (for STProxy) > Quality of protection (QoP) settings and change Protocol from SSL_TLS to TLSv1.2.

6

Go to SSL certificate and key management > SSL configurations and for all the other nodes including CellDefaultSSLSettings and XDADefaultSSLSettings set the Protocol to be SSL_TLSv2 including the SIP Proxy Registrar.

On all the nodes find the ssl.client.props file which is somewhere like /opt/IBM/WebSphere/AppServer/profiles/hostSTPPNProfile1/properties/ssl.client.props on Linux.

Ensure this is set as the following default value

com.ibm.ssl.protocol=SSL_TLSv2

This file instructs the client (the node agent) what protocol to communicate with the deployment manager using. As you have set this protocol in QoP for the cell, all nodes (apart from STProxy) and XDADefaultSSLSettings then all node agents can talk freely to the deployment manager.

If you miss a step here you’ll see from the deployment manager’s SystemOut.log that the node agent seems to stop and then start repeatedly. This is because the node agent cannot communicate properly, mainly because you have not changed XDADefaultSSLSettings appropriately.

Stop and start the deployment manager, run syncNode on all nodes and start the node agents, application servers and proxies and test. Check the SystemOut.log for any exceptions and if you see them check your configuration.

Ciphers

If you run a test against your STProxy or Meeting servers you’ll get marked down for the weak ciphers.

11

You can remove these from SSL certificate and key management > SSL configurations > NodeDefaultSSLSettings > Quality of protection (QoP) settings > Cipher suite settings. You will need to change from Strong to custom and then remove the ciphers listed above, if you so wish.

If you plan to do this for the Meeting server as well as STProxy then you will need to change the Inbound and Outbound options for the WebSphere proxy in front of Meetings so that it uses the NodedefaultSSLSettings which allows you to then use a default set of ciphers.

Finally

I have created a PMR to ask IBM about their support for TLSv1.2 in Sametime. I’ll update things once I get a response.

IBM Connections Mail and Ephemeral Diffie-Hellman key size error – part 2

I wrote about the effects using DHE ciphers can have depending on the size of the SSL certificate used by iNotes when IBM Connections Mail is in play in IBM Connections Mail and Ephemeral Diffie-Hellman key size error

In this blog I suggested the work around was to use the following notes.ini setting.

SSL_DH_KEYSIZE=2048

Our Domino admins weren’t too keen on lowering the key size so I had to look into a way of forcing the server to use a different cipher instead of one of the DHE ciphers.

This is the output from Domino when the DHE cipher is in play.

[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Client requested RSA_WITH_AES_128_CBC_SHA (0x002F)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Best common cipherspec 0x002F (so far)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Best common non-EC cipherspec 0x002F (so far)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Client requested RSA_WITH_AES_256_CBC_SHA (0x0035)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Best common cipherspec 0x0035 (so far)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Best common non-EC cipherspec 0x0035 (so far)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Client requested DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Client requested DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Best common cipherspec 0x0039 (so far)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Best common non-EC cipherspec 0x0039 (so far)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Client requested DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Client requested DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Client requested RSA_WITH_3DES_EDE_CBC_SHA (0x000A)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Client requested DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Client requested Unknown Cipher (0x0013)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Client requested TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00FF)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> TLS_EMPTY_RENEGOTIATION_INFO_SCSV found
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Extensions found in this message
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Processing TLS signature algorithms extension
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> Client supports hash mask 0x007E; server cert chain has mask 0x0030
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> hash/alg in certchain  fSupHasAlg:0000
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessClientHello> We selected cipher DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLProcessHandshakeMessage Exit> Message: ClientHello (1) State: HandshakeServerIdle (3) Key Exchange: 9 Cipher: DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLAdvanceHandshake Enter> Processed: ClientHello (1) State: HandshakeServerIdle (3)
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLAdvanceHandshake client_hello> SGC FLAG: 0   Count = 2
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeServerHello
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLEncodeServerHello> Sending empty renegotiation_info (0xff01) extension
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeCertificate
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLEncodeCertificate> Generating a certificate message with 3 certs
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeServerKeyExchange
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLEncodeDHKeyParams> Server RSA key size 4096 bits
[00403:00011-2285692672] 07/15/2016 11:07:54.94 AM SSLEncodeDHKeyParams> Using a DH key size of 4096 bits
[00403:00011-2285692672] 07/15/2016 11:07:55.01 AM SSLEncodeRSAServerKeyExchange> Signing ServerKeyExchange using RSAWithSHA256
[00403:00011-2285692672] 07/15/2016 11:07:55.04 AM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeServerHelloDone
[00403:00011-2285692672] 07/15/2016 11:07:55.04 AM SSLAdvanceHandshake Exit> State HandshakeClientKeyExchange (11)
[00403:00011-2285692672] 07/15/2016 11:07:55.04 AM SSL_Handshake> After handshake state = HandshakeClientKeyExchange (11); Status = -5000
[00403:00011-2285692672] 07/15/2016 11:07:55.04 AM int_MapSSLError> Mapping SSL error -5000 to 4176 [SSLHandshakeNoDone]
[00403:00011-2285692672] 07/15/2016 11:07:55.06 AM SSLProcessProtocolMessage> Record Content: Alert (21)
[00403:00011-2285692672] 07/15/2016 11:07:55.06 AM SSLProcessAlert> Got an alert of 0x50 (internal_error) level 0x2 (fatal)
[00403:00011-2285692672] 07/15/2016 11:07:55.06 AM SSL_Handshake> After handshake2 state HandshakeClientKeyExchange (11)
[00403:00011-2285692672] 07/15/2016 11:07:55.06 AM SSL_Handshake> SSL Error: -6994
[00403:00011-2285692672] 07/15/2016 11:07:55.06 AM int_MapSSLError> Mapping SSL error -6994 to 4171 [SSLFatalAlert]

The idea was to remove the DHE_RSA_WITH_AES_256_CBC_SHA (0x0039) from the list of supported ciphers.

You can do this by dictating all the ciphers Domino uses using the SSLCipherSpec notes.ini setting.

I stopped Domino and added to the notes.ini the following and then started Domino.

SSLCipherSpec=C030009FC02F009EC028006BC014C0270067C013009D009C003D0035003C02F000A

You can see in the string 0039 is not listed. This means that Domino will not use DHE_RSA_WITH_AES_256_CBC_SHA and another cipher will be negotiated.

On restart you can see that the cipher RSA_WITH_AES_256_CBC_SHA is now selected and that is being used which works.

[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLInitContext> Ignoring invalid SSLCipherSpec value F0
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLInitContext> User is forcing 0xFFF3800 cipher spec bitmask for 15 ciphers
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSL_TRUSTPOLICY>  bits for signature hashes: 0030
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM int_MapSSLError> Mapping SSL error 0 to 0 [SSLNoErr]
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSL_Handshake> outgoing ->protocolVersion: 0303
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessProtocolMessage> Record Content: Handshake (22)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessHandshakeMessage Enter> Message: ClientHello (1) State: HandshakeServerIdle (3) Key Exchange: 0 Cipher: Unknown Cipher (0x0000)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessHandshakeMessage client_hello> SGC FLAG: 0 CTX state = 3 SGCCount = 0
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> clientVersion: 0303
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> SSL/TLS protocol clientVersion 0x0303, serverVersion 0x0303
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> 10 ciphers requested by client
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Client requested RSA_WITH_AES_128_CBC_SHA (0x002F)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Client requested RSA_WITH_AES_256_CBC_SHA (0x0035)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Best common cipherspec 0x0035 (so far)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Best common non-EC cipherspec 0x0035 (so far)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Client requested DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Client requested DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Client requested DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Client requested DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Client requested RSA_WITH_3DES_EDE_CBC_SHA (0x000A)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Client requested DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Client requested Unknown Cipher (0x0013)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Client requested TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00FF)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> TLS_EMPTY_RENEGOTIATION_INFO_SCSV found
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Extensions found in this message
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Processing TLS signature algorithms extension
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> Client supports hash mask 0x007E; server cert chain has mask 0x0030
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> hash/alg in certchain  fSupHasAlg:0000
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessClientHello> We selected cipher RSA_WITH_AES_256_CBC_SHA (0x0035)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessHandshakeMessage Exit> Message: ClientHello (1) State: HandshakeServerIdle (3) Key Exchange: 1 Cipher: RSA_WITH_AES_256_CBC_SHA (0x0035)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLAdvanceHandshake Enter> Processed: ClientHello (1) State: HandshakeServerIdle (3)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLAdvanceHandshake client_hello> SGC FLAG: 0   Count = 2
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLAdvanceHandshake client_hello> Using resumed SSL/TLS Session
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeServerHello
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLEncodeServerHello> Sending empty renegotiation_info (0xff01) extension
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeChangeCipherSpec
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeFinishedMessage
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLCalculateTLS12FinishedMessage Enter> senderID: server finished, PRF using SHA256
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLAdvanceHandshake Exit> State HandshakeChangeCipherSpec (13)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSL_Handshake> After handshake state = HandshakeChangeCipherSpec (13); Status = -5000
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM int_MapSSLError> Mapping SSL error -5000 to 4176 [SSLHandshakeNoDone]
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessProtocolMessage> Record Content: Change cipher spec (20)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSL_Handshake> After handshake2 state HandshakeFinished (14)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM int_MapSSLError> Mapping SSL error -5000 to 4176 [SSLHandshakeNoDone]
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessProtocolMessage> Record Content: Handshake (22)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessHandshakeMessage Enter> Message: Finished (20) State: HandshakeFinished (14) Key Exchange: 1 Cipher: RSA_WITH_AES_256_CBC_SHA (0x0035)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLCalculateTLS12FinishedMessage Enter> senderID: client finished, PRF using SHA256
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLProcessHandshakeMessage Exit> Message: Finished (20) State: HandshakeFinished (14) Key Exchange: 1 Cipher: RSA_WITH_AES_256_CBC_SHA (0x0035)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLAdvanceHandshake Enter> Processed: Finished (20) State: HandshakeFinished (14)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSLAdvanceHandshake Exit> State HandshakeServerIdle (3)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSL_Handshake> After handshake2 state HandshakeServerIdle (3)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSL_Handshake> Using resumed SSL/TLS session
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSL_Handshake> Protocol Version = TLS1.2 (0x303)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSL_Handshake> Cipher = RSA_WITH_AES_256_CBC_SHA (0x0035)
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSL_Handshake> KeySize = 256 bits
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSL_Handshake> Server RSA key size = 4096 bits
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM SSL_Handshake> TLS/SSL Handshake completed successfully
[06035:00011-2986616576] 07/15/2016 12:30:35.85 PM int_MapSSLError> Mapping SSL error 0 to 0 [SSLNoErr]

The string below includes all the ECDHE ciphers which is detailed in https://www-10.lotus.com/ldd/dominowiki.nsf/dx/TLS_Cipher_Configuration but not the DHE cipher that was tripping me up.

SSLCipherSpec=C030009FC02F009EC028006BC014C0270067C013009D009C003D0035003C02F000A

It work’s now and I have tested it with all major browsers. I’m happy and so are the Domino guys too 🙂

Forcing TLSv1.2 breaks IBM Connections Surveys and Textbox.io

I had to force TLSv1.2 across all of Connections to fix a problem with RTE as I detailed in Rich Content widget widget stops working due to mix matched SSL protocols but after testing I’ve found that this breaks Textbox.io in Chrome and Surveys.

The process is well documented in How to Force IBM Connections 5.5 CR1 to Use TLSv1.2 but after making the changes the following happens.

Textbox.io

In IE and FF Textbox.io works fine but in Chrome the spell check service fails.

1

In Fiddler trace is saw Spelling server error:  Could not load url “https://connections.acme.com/ephox-spelling/1/correction&#8221;: 500 Internal Server Error

In the SystemOut.log I saw

[6/29/16 10:00:38:507 BST] 00000200 SystemOut     O ironbark-akka.actor.default-dispatcher-17, RECV TLSv1 ALERT:  fatal, handshake_failure
[6/29/16 10:00:38:507 BST] 00000200 SystemOut     O ironbark-akka.actor.default-dispatcher-17, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Received fatal alert: handshake_failure

spray.can.Http$ConnectionException: Aborted
    at spray.can.client.HttpHostConnectionSlot.reportDisconnection(HttpHostConnectionSlot.scala:228) ~[spray-can_2.11-1.3.3.jar:na]
    at spray.can.client.HttpHostConnectionSlot$$anonfun$connected$1.applyOrElse(HttpHostConnectionSlot.scala:161) ~[spray-can_2.11-1.3.3.jar:na]
    at akka.actor.Actor$class.aroundReceive(Actor.scala:465) ~[akka-actor_2.11-2.3.9.jar:na]

IBM asked me to put on SSL trace, *=info:SSL=all. It seems that the client is sending TLSv1.0 which of course is not allowed now TLSv1.2 has been forced.

[7/11/16 9:31:17:286 BST] 00000115 SystemOut     O   ironbark-akka.actor.default-dispatcher-7, READ: TLSv1.2 Alert, length = 2
[7/11/16 9:31:17:286 BST] 00000115 SystemOut     O   ironbark-akka.actor.default-dispatcher-7, RECV TLSv1 ALERT:  fatal, handshake_failure
[7/11/16 9:31:17:286 BST] 00000115 SystemOut     O   ironbark-akka.actor.default-dispatcher-7, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Received fatal alert: handshake_failure

IBM have logged a ticket with Ephox as well as investigating it from there end.

Surveys

When in a community with previous surveys I can not see any of the historical surveys nor could I create new ones.

In the SystemOut.log I saw the following

[6/29/16 10:01:56:542 BST] 0000033b StandardExcep E com.ibm.form.nitro.platform.StandardExceptionMapper toResponse eaa8e54e-7c38-4edb-a5ca-bcbd6d7f6c64
                                 com.ibm.form.platform.service.framework.exception.ServicesPlatformException: com.ibm.connections.directory.services.exception.DSException: com.ibm.connections.directory.services.exception.DSOutOfServiceException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Caused by: com.ibm.connections.directory.services.exception.DSException: com.ibm.connections.directory.services.exception.DSOutOfServiceException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Again, it seems like it’s sending TLSv1.0.

There’s at least one other person I know of who’s logged a PMR for these problems. It’s fairly urgent due to a problem with the RTE application which is only fixed when TLS1.2 is enforced. I’m hoping that these problems can be resolved sharpish so I can resolve the RTE problem for a customer.

IBM Connections Mail and Ephemeral Diffie-Hellman key size error

I’m building an IBM Connections 5.5 server to replace our internal Connections server and when configuring the Mail plug-in I came up against problems with the error “Mail server cannot be reached.”

1

The Domino iNotes server is configured to accept SSL and have SSLv3 disabled via DISABLE_SSLV3=1. SSO works in both directions between the two application servers.

I checked the discoveryservlet URL (https://connections.acme.com/connections/resources/discovery/DiscoveryServlet?email=ben.williams@chooseportal.com) which returned valid data so I know the configuration in socialmail-discovery-config.xml was good but there was very little to go on. Even after I enabled *=info:com.ibm.social.pim.discovery.*=all there was nothing much to go on.

I reached out and Michele Buccarello responded and pointed me towards one of his documents http://www.slideshare.net/michelebuccarello/connections-mail-with-exchange-backend. The document is written primarily for an Exchange server but it describes brilliantly what is happening and a bit of trace that came to my rescue.

I enabled *=info:com.ibm.social.pim.discovery.*=all:com.ibm.cre.*=all and all of a sudden I saw what was happening.

[7/12/16 13:49:33:787 BST] 00000220 CREURLConnect 2   An unhandled exception occured connecting to the target host
                                 javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair

Caused by: java.lang.RuntimeException: Could not generate DH keypair

Caused by: java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 256 to 2048 (inclusive)

I read around the various ciphers and I must admit I was a little lost and it’s been a while since I’ve delved deeply into Domino but some googling got me to some Daniel Nashed blogs.

http://blog.nashcom.de/nashcomblog.nsf/dx/first-perfect-forward-secrecy-ciphers-shipped-with-9.0.1-fp3-if2.htm?opendocument&comments#anc1

http://blog.nashcom.de/nashcomblog.nsf/dx/dha-with-more-than-1024-key-size-and-java-still-works.htm?opendocument&comments#anc1

The second had a comment about the Mail plug-in not working so I knew I was getting closer. This put various stackoverflow posts into perspective such as

http://stackoverflow.com/questions/6851461/java-why-does-ssl-handshake-give-could-not-generate-dh-keypair-exception

I stopped Domino and added the following before starting it again and the plug-in started working and I could access my mail and calendar.

SSL_DH_KEYSIZE=1024

I upped the value to 2048 since a previous error said “Prime size must be multiple of 64, and can only range from 256 to 2048 (inclusive).”

On restart of Domino it continued to work. I tried increasing the value to 3072 but this broke the plug-in.

The certificate I was provided was a 4096 bit certificate and not 2048 like I handle more often.

In https://www-10.lotus.com/ldd/dominowiki.nsf/dx/TLS_Cipher_Configuration it states, “By default, these ciphers will use a DH key with a size equivalent to the RSA keysize, so a server running with a 2048 bit SSL certificate would use a 2048 bit DH group.” This means that the DH key being used is 4096 which IBM’s implementation of Java doesn’t support, hence the need to add SSL_DH_KEYSIZE=2048.

I then found the following Domino trace.

DEBUG_SSL_CIPHERS=2
DEBUG_SSL_DHE=2
DEBUG_SSL_HANDSHAKE=2
DEBUG_SSL_IO=0

When I recreate the problem I see in the console.log the following which shows the DH key size.

[11856:00011-1753671424] 07/12/2016 01:50:03.16 PM SSLEncodeDHKeyParams> Server RSA key size 4096 bits
[11856:00011-1753671424] 07/12/2016 01:50:03.16 PM SSLEncodeDHKeyParams> Using a DH key size of 4096 bits
[11856:00011-1753671424] 07/12/2016 01:50:03.26 PM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeServerHelloDone
[11856:00011-1753671424] 07/12/2016 01:50:03.26 PM SSLAdvanceHandshake Exit> State HandshakeClientKeyExchange (11)
[11856:00011-1753671424] 07/12/2016 01:50:03.26 PM SSL_Handshake> After handshake state = HandshakeClientKeyExchange (11); Status = -5000
[11856:00011-1753671424] 07/12/2016 01:50:03.26 PM int_MapSSLError> Mapping SSL error -5000 to 4176 [SSLHandshakeNoDone]
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM SSLProcessProtocolMessage> Record Content: Alert (21)
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM SSLProcessAlert> Got an alert of 0x50 (internal_error) level 0x2 (fatal)
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM SSL_Handshake> After handshake2 state HandshakeClientKeyExchange (11)
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM SSL_Handshake> SSL Error: -6994
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM int_MapSSLError> Mapping SSL error -6994 to 4171 [SSLFatalAlert]

In https://www-10.lotus.com/ldd/dominowiki.nsf/dx/TLS_Cipher_Configuration it also states, “When using Domino 9.0.1 FP3 IF2 one can and should disable DHE_RSA_WITH_AES_128_CBC_SHA (33) which should make those old clients fall back to using RSA_WITH_AES_128_CBC_SHA (2F) instead.”

I tried the below setting which removes “33” to see whether it worked but it did not. I would like to fiddle more with this to try and find a cipher that WAS and Domino can use in common that avoids setting the DH key too low but I suspect I will run out of time.

SSLCIPHERSPEC=9D9C3D3C352F0A39676B9E9F

BTW – I did all this after I had forced TLS1.2 via How to Force IBM Connections 5.5 CR1 to Use TLSv1.2 which is nice to know that Mail is not broken after enforcing TLS1.2 unlike Textbox.io and Surveys…..

Oh, in Domino when it is successful it will look something like this.

[17825:00011-575325952] 07/12/2016 03:12:00.31 PM SSLEncodeDHKeyParams> Server RSA key size 4096 bits
[17825:00011-575325952] 07/12/2016 03:12:00.31 PM SSLEncodeDHKeyParams> Using a DH key size of 2048 bits
[17825:00011-575325952] 07/12/2016 03:12:00.32 PM SSLEncodeRSAServerKeyExchange> Signing ServerKeyExchange using RSAWithSHA256
[17825:00011-575325952] 07/12/2016 03:12:00.36 PM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeServerHelloDone

[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> After handshake2 state HandshakeServerIdle (3)
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> Protocol Version = TLS1.2 (0x303)
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> Cipher = DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> KeySize = 256 bits
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> Ephemeral Diffie-Hellman key size = 2048 bits
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> Server RSA key size = 4096 bits
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> TLS/SSL Handshake completed successfully

You won’t see much difference in trace.log.

If anyone has a better way to get around this without changing the value of the DH key size then please shout.

Additional information

I should mention that a useful tip Michele Buccarello pointed me towards taking a Fiddler trace.

You’ll see a call to /connections/opensocial/gadgets/makeRequest. Within that entry in Fiddler I saw 502 Bad Gateway

throw 1; < ‘invalid javascript’ >{“https://webmail.acme.com/mail/bwilliam.nsf/iNotes/Proxy/?OpenDocument&Form=f_SessionInfo_Data&_icmb=20160425-0501&#8221;:{“rc”:502,”body”:”&amp;amp;lt;HTML&amp;amp;gt;&amp;amp;lt;TITLE&amp;amp;gt;502&amp;amp;nbsp;-&amp;amp;nbsp;Bad&amp;amp;nbsp;Gateway&amp;amp;lt;/TITLE&amp;amp;gt;&amp;amp;lt;BODY&amp;amp;gt;&amp;amp;lt;h1&amp;amp;gt;502&amp;amp;nbsp;An&amp;amp;nbsp;unhandled&amp;amp;nbsp;exception&amp;amp;nbsp;occured&amp;amp;nbsp;connecting&amp;amp;nbsp;to&amp;amp;nbsp;the&amp;amp;nbsp;target&amp;amp;nbsp;host&amp;amp;lt;/h1&amp;amp;gt;&amp;amp;lt;/BODY&amp;amp;gt;&amp;amp;lt;/HTML&amp;amp;gt;”,”headers”:{“date”:[“Mon, 11 Jul 2016 21:30:46 GMT”],”content-type”:[“text/html; charset=UTF-8″]},”DataHash”:”jslu7s57e7d899jbtr7p1d033g”}}

You can also look at the JSON section to see it in a different format.

The above is also seen in the trace.log with *=info:com.ibm.social.pim.discovery.*=all:com.ibm.cre.*=all

[7/12/16 8:49:30:670 BST] 000001bb CREURLConnect 2   IOException caught, response code is 502, Exception was java.io.IOException: Server returned HTTP response code: 502 for URL: https://webmail.acme.com/mail/bwilliam.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries_JSON&PresetFields=FolderName;($Inbox),UnreadOnly;1,UnreadCountInfo;1,hc;$98&Count=1&resortdescendingpn=$70&TZType=UTC&KeyType=time&_icmb=20160425-0501
[7/12/16 8:49:30:671 BST] 000001bb CREURLConnect 2   Retry error while in streaming mode: 502, java.io.IOException: Server returned HTTP response code: 502 for URL: https://webmail.acme.com/mail/bwilliam.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries_JSON&PresetFields=FolderName;($Inbox),UnreadOnly;1,UnreadCountInfo;1,hc;$98&Count=1&resortdescendingpn=$70&TZType=UTC&KeyType=time&_icmb=20160425-0501

IBM Sametime HTTPS redirection

Redirection of HTTP to HTTPS for Sametime is made possible by deploying a WebSphere proxy in front of Sametime Proxy or a Meeting server. Once configured you can use a routing rule to redirect a specific URL to another specific URL. What if you want every possible permutation to be directed to HTTPS?

1

It is well documented in http://blog.msbiro.net/2014/02/redir-htp-https-websphere-proxy-sametime-server.html and http://www-10.lotus.com/ldd/stwiki.nsf/dx/Forcing_Sametime_8.5.2_WebSphere_Application_server_to_use_HTTPS_TLS_encryption how to achieve this.

I have used the above method successfully for a while but it got me thinking how I would control a user accessing a meeting room directly as opposed to going to the meeting center which would be captured by the routing rule.

I raised a PMR after testing many scenarios with a WebSphere proxy fronting a Sametime Proxy and Meeting server and IBM told me that it is not possible with a WebSphere proxy but suggested I use IHS. Not his fault, he wasn’t a Sametime guy. But he did suggest that I take a look at using <transport-guarantee>CONFIDENTIAL</transport-guarantee>.

http://docs.oracle.com/javaee/5/tutorial/doc/bncbe.html describes how to achieve this. If you Google <transport-guarantee>CONFIDENTIAL</transport-guarantee> you will find a number of IBM docs on this which helps.

What I wasn’t sure of is whether to make the change in multiple places ie each war’s web.xml where there is a “<security-constraint>” stanza. It may be only appropriate to make the change on the login page and thus the war that relates to it but what if people went directly to a specific page bypassing the login page’s war.

I made the following changes on the SSC and then issued a full sync and restarted the STProxy. I also ensured that I had disabled the WebSphere proxies rule so that it didn’t step in.

[root@st9ssc ~]# cd /opt/IBM/WebSphere/AppServer/profiles/STSCDMgrProfile/config/cells/st9sscSSCCell/applications
[root@st9ssc applications]# cp -r ./SametimeProxy.ear/ /tmp/SametimeProxy.ear.backup

[root@st9ssc applications]# cd /opt/IBM/WebSphere/AppServer/profiles/STSCDMgrProfile/config/cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy
[root@st9ssc SametimeProxy]# ll
total 56
drwxr-xr-x 4 root root 4096 Jan  6  2014 autoaway.war
-rw-r–r– 1 root root 5208 Jun 15 11:49 deployment.xml
drwxr-xr-x 2 root root 4096 Oct 24  2013 META-INF
drwxr-xr-x 4 root root 4096 May 26 17:42 proxyutils.war
drwxr-xr-x 4 root root 4096 Oct 24  2013 screencapture.war
drwxr-xr-x 4 root root 4096 Jan  6  2014 stmobileweb.war
drwxr-xr-x 4 root root 4096 Oct 24  2013 stproxybase.war
drwxr-xr-x 4 root root 4096 Oct 24  2013 stproxymobile.war
drwxr-xr-x 4 root root 4096 Oct 24  2013 stproxyredirect.war
drwxr-xr-x 4 root root 4096 Oct 24  2013 stproxyservlet.war
drwxr-xr-x 4 root root 4096 Oct 24  2013 stproxyweb.war
drwxr-xr-x 4 root root 4096 Oct 24  2013 stwebav.war
drwxr-xr-x 4 root root 4096 Jan  6  2014 workclasses

[root@st9ssc SametimeProxy]# vi /opt/IBM/WebSphere/AppServer/profiles/STSCDMgrProfile/config/cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stmobileweb.war/WEB-INF/web.xml

<security-constraint>
<web-resource-collection>
<web-resource-name>SametimeProxy methods</web-resource-name>
<url-pattern>/*</url-pattern>
<http-method>OPTIONS</http-method>
<http-method>TRACE</http-method>
<http-method>HEAD</http-method>
</web-resource-collection>
<auth-constraint>
<description />
<role-name>AllUsers</role-name>
</auth-constraint>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>

[root@st9ssc SametimeProxy]# vi /opt/IBM/WebSphere/AppServer/profiles/STSCDMgrProfile/config/cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stproxybase.war/WEB-INF/web.xml
<security-constraint>
<web-resource-collection>
<web-resource-name>SametimeProxy methods</web-resource-name>
<url-pattern>/*</url-pattern>
<http-method>OPTIONS</http-method>
<http-method>TRACE</http-method>
<http-method>HEAD</http-method>
</web-resource-collection>
<auth-constraint>
<description />
<role-name>AllUsers</role-name>
</auth-constraint>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>

[root@st9ssc SametimeProxy]# vi /opt/IBM/WebSphere/AppServer/profiles/STSCDMgrProfile/config/cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stproxymobile.war/WEB-INF/web.xml
<security-constraint>
<web-resource-collection>
<web-resource-name>Mobile installation</web-resource-name>
<url-pattern>/*</url-pattern>
<http-method>OPTIONS</http-method>
<http-method>TRACE</http-method>
</web-resource-collection>
<auth-constraint>
<description />
<role-name>AllUsers</role-name>
</auth-constraint>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>

[root@st9ssc SametimeProxy]# vi /opt/IBM/WebSphere/AppServer/profiles/STSCDMgrProfile/config/cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stproxyredirect.war/WEB-INF/web.xml
<security-constraint>
<web-resource-collection>
<web-resource-name>Sametime Proxy Server</web-resource-name>
<url-pattern>/*</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>AllAuthenticatedUsers</role-name>
</auth-constraint>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>

[root@st9ssc SametimeProxy]# vi /opt/IBM/WebSphere/AppServer/profiles/STSCDMgrProfile/config/cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stproxyservlet.war/WEB-INF/web.xml
<security-constraint>
<web-resource-collection>
<web-resource-name>rtc4web based WebApp and GUI</web-resource-name>
<url-pattern>/*</url-pattern>
<http-method>OPTIONS</http-method>
<http-method>TRACE</http-method>
<http-method>HEAD</http-method>
</web-resource-collection>
<auth-constraint>
<description></description>
<role-name>AllUsers</role-name>
</auth-constraint>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>

[root@st9ssc SametimeProxy]# vi /opt/IBM/WebSphere/AppServer/profiles/STSCDMgrProfile/config/cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stproxyweb.war/WEB-INF/web.xml
<security-constraint>
<web-resource-collection>
<web-resource-name>File Share methods</web-resource-name>
<url-pattern>/ajaxproxy/*</url-pattern>
<http-method>GET</http-method>
<http-method>PUT</http-method>
<http-method>POST</http-method>
<http-method>DELETE</http-method>
<http-method>HEAD</http-method>
</web-resource-collection>
<auth-constraint>
<description>All users, registered and unregistered</description>
<role-name>AllAuthenticatedUsers</role-name>
</auth-constraint>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>
<security-constraint>
<web-resource-collection>
<web-resource-name>SametimeProxy methods</web-resource-name>
<url-pattern>/*</url-pattern>
<http-method>GET</http-method>
<http-method>PUT</http-method>
<http-method>POST</http-method>
<http-method>DELETE</http-method>
</web-resource-collection>
<auth-constraint>
<description>All users, registered and unregistered</description>
<role-name>AllAuthenticatedUsers</role-name>
</auth-constraint>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>

[root@st9ssc SametimeProxy]# vi /opt/IBM/WebSphere/AppServer/profiles/STSCDMgrProfile/config/cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stwebav.war/WEB-INF/web.xml
<security-constraint>
<web-resource-collection>
<web-resource-name>WebAV Binaries Install Update</web-resource-name>
<url-pattern>/*</url-pattern>
<http-method>OPTIONS</http-method>
<http-method>TRACE</http-method>
</web-resource-collection>
<auth-constraint>
<description />
<role-name>AllUsers</role-name>
</auth-constraint>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>

Full sync.

[7/24/15 16:52:50:087 BST] 00000072 FileRepositor A   ADMR0012I: The repository epoch is refreshed.
[7/24/15 16:52:50:123 BST] 00000072 FileRepositor A   Repository epoch refresh
[7/24/15 16:52:54:307 BST] 00000664 FileRepositor A   ADMR0016I: User ldap.collaborationben.com:389/server:st9sscSSCCell_st9proxySTPNode1_nodeagent modified document cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stproxyweb.war/WEB-INF/web.xml.
[7/24/15 16:52:54:329 BST] 00000664 FileRepositor A   ADMR0017I: User ldap.collaborationben.com:389/server:st9sscSSCCell_st9proxySTPNode1_nodeagent deleted document cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stproxyweb.war/WEB-INF/.web.xml.swp.
[7/24/15 16:52:54:362 BST] 00000664 FileRepositor A   ADMR0016I: User ldap.collaborationben.com:389/server:st9sscSSCCell_st9proxySTPNode1_nodeagent modified document cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stwebav.war/WEB-INF/web.xml.
[7/24/15 16:52:54:416 BST] 00000664 FileRepositor A   ADMR0016I: User ldap.collaborationben.com:389/server:st9sscSSCCell_st9proxySTPNode1_nodeagent modified document cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stproxyredirect.war/WEB-INF/web.xml.
[7/24/15 16:52:54:456 BST] 00000664 FileRepositor A   ADMR0016I: User ldap.collaborationben.com:389/server:st9sscSSCCell_st9proxySTPNode1_nodeagent modified document cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stproxymobile.war/WEB-INF/web.xml.
[7/24/15 16:52:54:497 BST] 00000664 FileRepositor A   ADMR0016I: User ldap.collaborationben.com:389/server:st9sscSSCCell_st9proxySTPNode1_nodeagent modified document cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stmobileweb.war/WEB-INF/web.xml.
[7/24/15 16:52:54:534 BST] 00000664 FileRepositor A   ADMR0016I: User ldap.collaborationben.com:389/server:st9sscSSCCell_st9proxySTPNode1_nodeagent modified document cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stproxyservlet.war/WEB-INF/web.xml.
[7/24/15 16:52:54:609 BST] 00000664 FileRepositor A   ADMR0016I: User ldap.collaborationben.com:389/server:st9sscSSCCell_st9proxySTPNode1_nodeagent modified document cells/st9sscSSCCell/applications/SametimeProxy.ear/deployments/SametimeProxy/stproxybase.war/WEB-INF/web.xml.
[7/24/15 16:52:55:856 BST] 00000664 NodeSyncTask  A   ADMS0003I: The configuration synchronization completed successfully.
[7/24/15 16:52:56:612 BST] 0000066c AppBinaryProc I   ADMA7021I: Distribution of application SametimeProxy completed successfully.

I still have the WebSphere proxy in front of STProxy which isn’t needed now but when ever I hit the STProxy or WebSphere proxy on their unsecured ports (WC_defaulthost or PROXY_HTTP_ADDRESS) I am redirected to the secure port of the application server (WC_defaulthost_secure).

Early testing looks good. I haven’t tested integration with Meetings, AV or mobile but I will do in time. Mobile may be a bit tricky as this is asking the client to redirect but I would have hoped the mobile app would have been configured to use HTTPS anyway.

One problem would be that each time the STProxy is updated from a fix from fix Central or IBM support these changes will be overwritten and will need to be made again. Also, this would do away with the need for a WebSphere proxy if it is being used solely for redirection to SSL. If you have a cluster of Meeting servers then you will still need WebSphere proxies.

In the circumstance of clustered WebSphere proxies the problem I see arising is that the redirection uses the secure port listed in the virtual host for the application server and not that of the WebSphere proxy. This means that unless the WebSphere proxy is on another host or bound to port 443 on a second NIC on the same node as the Meeting application server then you will not be able to redirect to 443 properly. You can’t have two things listening on 443 on the same host using the same NIC.

Nevertheless, without being able to use Apache or IHS this provides a useful alternative.

 

 

Sametime and POODLE SSLv3 patches

IBM released two Technotes for Sametime and POODLE Security Bulletin: Vulnerability in SSLv3 affects Sametime (CVE-2014-3566) and Security Bulletin: Vulnerability in SSLv3 affects IBM WebSphere Application Server (CVE-2014-3566)

What wasn’t clear (at first) was what actually needed to be done to disable SSLv3 and ensure that Sametime functions properly. Off the back of another PMR relating to the VMCU I managed to get some of Tony Payne’s time to fire off some questions.

The patches available in Security Bulletin: Vulnerability in SSLv3 affects Sametime (CVE-2014-3566) are to resolve problems within Sametime and DO NOT DISABLE SSLV3. These problems are.

  •  In Media servers SSL v3 was still enabled for backend server-to-server connections.
  • After making the POODLE security change on SSC as described in this bulletin, the installers for Sametime products (Advanced, Meetings, Media, Proxy, and Community Servers) are not able to connect to the SSC server and policies are not getting synched from the SSC into the Community Server.

So, you need to apply the patches to your servers and then you need to move onto the steps detailed in Security Bulletin: Vulnerability in SSLv3 affects IBM WebSphere Application Server (CVE-2014-3566)

Before you move onto WAS you might want to know which servers to apply the patches to. The Technote is quite clear but what if you have Edge components? If you do, then the SIP Edge proxy does not need to have the Media Manager code ran against is and nor does the TURN server. If you have an HTTP Edge proxy which sits in front of you Meeting server then this will need the patch applied as it communicates with the SSC, unlike the TURN and SIP Edge proxy. You do need to patch any SIP/HTTP proxies in front of the CM, SIP PR or Meeting servers which may be on their own node and hence their own profile.

After you have installed the patches you then need to disable SSLv3. To do this you can install ifixes or simply turn it off from within the SSC. You should also disable this from within the ISC of your SIP Edge proxy and Video Manager server.

The ifixes remove the ability to set or use SSLv3 so the net effect is that it makes the change within the SSC/ISC.

For my deployment I simply changed the settings within the SSC/ISC. To disable SSLv3 you need to do the following.

  • Log in to the SSC/ISC.
  • Go to Security – SSL certificate and key management – SSL configurations – CellDefaultSSLSettings – Quality of protection (QoP) settings. For VMGR and SIP Edge proxy you can update the NodeDefaultSSLSettings.
  • Change the Protocol from SSL_TLS to TLS.
  • Save and sync the changes to your nodes.
  • Stop all application servers.
  • Stop all node agents.
  • Update the ssl.client.props in each profile replacing “com.ibm.ssl.protocol=SSL_TLS” with “com.ibm.ssl.protocol=TLS”
  • Don’t forget the VMGR and Edge servers.
  • Restart the deployment manager.
  • In each profile run ./syncNode.sh ssc.collaborationben.com 8703 -username adminuser-password ******** to synchronise the node with the deployment manager.
  • Start each node agent and then each application server.
  • Test.

poodle

Testing

To test, find yourself a *nix machine and run the following command “openssl s_client -connect meeting.collaborationben.com:443 -ssl3” and you should get something like the following response.

CONNECTED(00000003)
139961097578312:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 0 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1424780572
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)

If SSLv3 was still enabled you would see something very different. You will see the SSL certificate returned and something like the following.

New, TLSv1/SSLv3, Cipher is DES-CBC3-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DES-CBC3-SHA
    Session-ID:

Problems

I had one problem applying the patch to the VMCU. The instructions say to run ./upgrade.sh but doing so I got the following error.

[root@vmcu SametimeVideoMCU]# ./upgrade.sh
Sametime Video MCU status:SoftMcu service is down
./upgrade.sh: line 15: [: too many arguments
./upgrade.sh: line 17: [: too many arguments
./upgrade.sh: line 20: [: too many arguments
./upgrade.sh: line 23: [: too many arguments
Reading property file /opt/IBM/Sametime/STVideoMCU/console.properties..
Checking Java version:
java version “1.6.0_24”
OpenJDK Runtime Environment (IcedTea6 1.11.14) (rhel-1.65.1.11.14.el6_4-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
Java major version is: 6
Checking for license..
Exited with: 9
License status: 9
License accepted. Proceeding with upgrade:  9
./mcms/Scripts/InstallValidator.sh
313561 blocks
All System requirements met for upgrade. Proceeding with Sametime Video MCU upgrade.
Backing up Sametime Video MCU
There is another operation currently in progress
Unable to backup Sametime Video MCU configuration. Upgrading without a backup may result in loss of data. Aborting upgrade.

I ran “chkconfig soft_mcu off” so the VMCU didn’t start after a reboot and stopped it. On OS restart the same happened. I reproduced this on a customer server and my own.

IBM came back with a few steps, although incomplete, they pointed me in the following direction. I did the following:

  • rpm -qa ‘ibm-sametime*’
  • rpm -e $(rpm -qa ‘ibm-sametime*’)
  • cd /opt/IBM/Sametime/STVideoMCU/
  • Ensure console.properties is correct.
  • ./uninstallVideoMcu.sh
  • rpm -qa ‘ibm-sametime*’ to check the RPMs have been removed.
  • Change directory to the patch.
  • Ensure console.properties is correct.
  • ./install.sh
  • yum update openssl to ensure openssl is up to date.
  • Restart OS due to openssl update.

This effectively uninstalls the VMCU and unregisters it and then installs it again (albeit the new version) and uses the original deployment plan so do not create a new one.

IBM are hosting an open mic on the 11th March 2015 on this subject. I guess they have been getting a few queries from people. I hope this blog means you can get on with this instead of waiting for the 11th.