Sametime and POODLE SSLv3 patches

IBM released two Technotes for Sametime and POODLE Security Bulletin: Vulnerability in SSLv3 affects Sametime (CVE-2014-3566) and Security Bulletin: Vulnerability in SSLv3 affects IBM WebSphere Application Server (CVE-2014-3566)

What wasn’t clear (at first) was what actually needed to be done to disable SSLv3 and ensure that Sametime functions properly. Off the back of another PMR relating to the VMCU I managed to get some of Tony Payne’s time to fire off some questions.

The patches available in Security Bulletin: Vulnerability in SSLv3 affects Sametime (CVE-2014-3566) are to resolve problems within Sametime and DO NOT DISABLE SSLV3. These problems are.

  •  In Media servers SSL v3 was still enabled for backend server-to-server connections.
  • After making the POODLE security change on SSC as described in this bulletin, the installers for Sametime products (Advanced, Meetings, Media, Proxy, and Community Servers) are not able to connect to the SSC server and policies are not getting synched from the SSC into the Community Server.

So, you need to apply the patches to your servers and then you need to move onto the steps detailed in Security Bulletin: Vulnerability in SSLv3 affects IBM WebSphere Application Server (CVE-2014-3566)

Before you move onto WAS you might want to know which servers to apply the patches to. The Technote is quite clear but what if you have Edge components? If you do, then the SIP Edge proxy does not need to have the Media Manager code ran against is and nor does the TURN server. If you have an HTTP Edge proxy which sits in front of you Meeting server then this will need the patch applied as it communicates with the SSC, unlike the TURN and SIP Edge proxy. You do need to patch any SIP/HTTP proxies in front of the CM, SIP PR or Meeting servers which may be on their own node and hence their own profile.

After you have installed the patches you then need to disable SSLv3. To do this you can install ifixes or simply turn it off from within the SSC. You should also disable this from within the ISC of your SIP Edge proxy and Video Manager server.

The ifixes remove the ability to set or use SSLv3 so the net effect is that it makes the change within the SSC/ISC.

For my deployment I simply changed the settings within the SSC/ISC. To disable SSLv3 you need to do the following.

  • Log in to the SSC/ISC.
  • Go to Security – SSL certificate and key management – SSL configurations – CellDefaultSSLSettings – Quality of protection (QoP) settings. For VMGR and SIP Edge proxy you can update the NodeDefaultSSLSettings.
  • Change the Protocol from SSL_TLS to TLS.
  • Save and sync the changes to your nodes.
  • Stop all application servers.
  • Stop all node agents.
  • Update the ssl.client.props in each profile replacing “com.ibm.ssl.protocol=SSL_TLS” with “com.ibm.ssl.protocol=TLS”
  • Don’t forget the VMGR and Edge servers.
  • Restart the deployment manager.
  • In each profile run ./syncNode.sh ssc.collaborationben.com 8703 -username adminuser-password ******** to synchronise the node with the deployment manager.
  • Start each node agent and then each application server.
  • Test.

poodle

Testing

To test, find yourself a *nix machine and run the following command “openssl s_client -connect meeting.collaborationben.com:443 -ssl3” and you should get something like the following response.

CONNECTED(00000003)
139961097578312:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 0 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1424780572
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)

If SSLv3 was still enabled you would see something very different. You will see the SSL certificate returned and something like the following.

New, TLSv1/SSLv3, Cipher is DES-CBC3-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DES-CBC3-SHA
    Session-ID:

Problems

I had one problem applying the patch to the VMCU. The instructions say to run ./upgrade.sh but doing so I got the following error.

[root@vmcu SametimeVideoMCU]# ./upgrade.sh
Sametime Video MCU status:SoftMcu service is down
./upgrade.sh: line 15: [: too many arguments
./upgrade.sh: line 17: [: too many arguments
./upgrade.sh: line 20: [: too many arguments
./upgrade.sh: line 23: [: too many arguments
Reading property file /opt/IBM/Sametime/STVideoMCU/console.properties..
Checking Java version:
java version “1.6.0_24”
OpenJDK Runtime Environment (IcedTea6 1.11.14) (rhel-1.65.1.11.14.el6_4-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
Java major version is: 6
Checking for license..
Exited with: 9
License status: 9
License accepted. Proceeding with upgrade:  9
./mcms/Scripts/InstallValidator.sh
313561 blocks
All System requirements met for upgrade. Proceeding with Sametime Video MCU upgrade.
Backing up Sametime Video MCU
There is another operation currently in progress
Unable to backup Sametime Video MCU configuration. Upgrading without a backup may result in loss of data. Aborting upgrade.

I ran “chkconfig soft_mcu off” so the VMCU didn’t start after a reboot and stopped it. On OS restart the same happened. I reproduced this on a customer server and my own.

IBM came back with a few steps, although incomplete, they pointed me in the following direction. I did the following:

  • rpm -qa ‘ibm-sametime*’
  • rpm -e $(rpm -qa ‘ibm-sametime*’)
  • cd /opt/IBM/Sametime/STVideoMCU/
  • Ensure console.properties is correct.
  • ./uninstallVideoMcu.sh
  • rpm -qa ‘ibm-sametime*’ to check the RPMs have been removed.
  • Change directory to the patch.
  • Ensure console.properties is correct.
  • ./install.sh
  • yum update openssl to ensure openssl is up to date.
  • Restart OS due to openssl update.

This effectively uninstalls the VMCU and unregisters it and then installs it again (albeit the new version) and uses the original deployment plan so do not create a new one.

IBM are hosting an open mic on the 11th March 2015 on this subject. I guess they have been getting a few queries from people. I hope this blog means you can get on with this instead of waiting for the 11th.

IBM Sametime Community server does not upgrade due to SSCEncodedAuthorization

I had a wobble when applying the latest POODLE/SSLV3 (AAZI-9RGLXV) patch to my CentOS 6.5 server. It kept failing and the following exceptions were seen in /opt/ibm/dominodata/SametimeInstall.log

(Feb 19, 2015 10:50:30 PM), Install, com.lotus.sametime.install.product.SSCReg, err, com.ibm.sametime.console.deployment.client.exception.SCClientException: AIDSC0884E: Failed to do IO operation with file.Server returned HTTP response code: 500 for URL: http://ssc.collaborationben.com:9080/console/deployment/getDepIdByDepName
STACK_TRACE: 18
com.ibm.sametime.console.deployment.client.exception.SCClientException: AIDSC0884E: Failed to do IO operation with file.Server returned HTTP response code: 500 for URL: http://ssc.collaborationben.com:9080/console/deployment/getDepIdByDepName
    at com.ibm.sametime.console.deployment.client.util.RestURL.sendData(RestURL.java:358)
    at com.ibm.sametime.console.deployment.client.util.ClientUtility.getDepIdByDepName(ClientUtility.java:1380)
    at com.ibm.sametime.console.deployment.client.api.Deployment.getDepIdByDepName(Deployment.java:1674)

(Feb 19, 2015 10:50:30 PM), Install, com.lotus.sametime.install.product.SSCReg, err, An error occurred and product update failed.  Look at the log file /opt/ibm/dominodata/SametimeInstall.log for details.
(Feb 19, 2015 10:50:30 PM), Install, com.lotus.sametime.install.product.SSCReg, err, ProductException: (error code = 601; message=”err”; additional data = [com.ibm.sametime.console.deployment.client.exception.SCClientException: AIDSC0884E: Failed to do IO operation with file.Server returned HTTP response code: 500 for URL: http://ssc.collaborationben.com:9080/console/deployment/getDepIdByDepName%5D)
STACK_TRACE: 14
ProductException: (error code = 601; message=”err”; additional data = [com.ibm.sametime.console.deployment.client.exception.SCClientException: AIDSC0884E: Failed to do IO operation with file.Server returned HTTP response code: 500 for URL: http://ssc.collaborationben.com:9080/console/deployment/getDepIdByDepName%5D)
    at com.lotus.sametime.install.product.SSCReg.setSSCStatus(SSCReg.java:328)

I also saw the following in the same file.

(Feb 19, 2015 10:50:03 PM), Install, com.lotus.sametime.install.wizard.GetSSCEncodedPassword, msg1, loading /opt/ibm/dominodata/console/console.properties
(Feb 19, 2015 10:50:03 PM), Install, com.lotus.sametime.install.wizard.GetSSCEncodedPassword, msg1, Encoded password found. Decoding…
(Feb 19, 2015 10:50:03 PM), Install, com.lotus.sametime.install.wizard.GetSSCEncodedPassword, msg1, username = wasadmin
(Feb 19, 2015 10:50:03 PM), Install, com.installshield.full.event.dialog.console.PanelSSCUpgradeLoginConsoleImpl, msg1, SC installation detected
(Feb 19, 2015 10:50:03 PM), Install, com.installshield.full.event.dialog.console.PanelSSCUpgradeLoginConsoleImpl, msg1, loading /opt/ibm/dominodata/console/console.properties
(Feb 19, 2015 10:50:03 PM), Install, com.installshield.full.event.dialog.console.PanelSSCUpgradeLoginConsoleImpl, msg1, Encoded password found. Getting previously decoded name and password

I changed the WAS administrator user to use an LDAP based account so I was wondering why wasadmin was still appearing even though /opt/ibm/dominodata/console/console.properties had the new user name and password.

In console.properties I saw that “SSCEncodedAuthorization” was listed. This is the encoded password for the admin account. I removed this from console.properties and ran setuplinux.bin -console again and it upgraded fine.

New Sametime Proxy APNs test application

I have written a couple of posts on this because I find the application extremely helpful in diagnosing network related issues with connection to APNs (Apple Push Notification service) so that iOS devices can receive IMs when the application is “backgrounded.”

Here is the application which includes a text file providing you with the correct syntax to us which would go something like this, for Windows.

D:\support\apnstest>d:\IBM\WebSphere\AppServer\java\bin\java.exe -jar apnstest.jar -k D:\IBM\WebSphere\AppServer\profiles\xxxxSTPPNProfile1\config\cells\xxxx01SSCCell\nodes\xxxxSTPNode1\apns-prod.pkcs12
APNS Test ScriptVersion: 2.0.0
Testing using key: D:\IBM\WebSphere\AppServer\profiles\xxxxxSTPPNProfile1\config\cells\xxxxx01SSCCell\nodes\xxxxSTPNode1\apns-prod.pkcs12
Testing using server: gateway.push.apple.com
Testing using port: 2195
About to attempt to connect to APNS
Initialized SSL Context
SSL Socket Created
Starting SSL Handshake
SSL Handshake Complete
CN=gateway.push.apple.com, O=Apple Inc., L=Cupertino, ST=California, C=US
CN=Entrust Certification Authority – L1C, OU=”(c) 2009 Entrust, Inc.”, OU=www.entrust.net/rpa is incorporated by reference, O=”Entrust, Inc.”, C=US
Successfully Connected to APNS
Test notification will not be sent

D:\support\apnstest>d:\IBM\WebSphere\AppServer\java\bin\java.exe -jar apnstest.jar -k D:\IBM\WebSphere\AppServer\profiles\xxxxxSTPPNProfile1\config\cells\xxxx01SSCCell\nodes\xxxxSTPNode1\apns-prod.pkcs12 -s feedback.push.apple.com -p 2196
APNS Test ScriptVersion: 2.0.0
Testing using key: D:\IBM\WebSphere\AppServer\profiles\xxxxSTPPNProfile1\config\cells\xxxx01SSCCell\nodes\xxxxSTPNode1\apns-prod.pkcs12
Testing using server: feedback.push.apple.com
Testing using port: 2196
About to attempt to connect to APNS
Initialized SSL Context
SSL Socket Created
Starting SSL Handshake
SSL Handshake Complete
CN=feedback.push.apple.com, O=Apple Inc., L=Cupertino, ST=California, C=US
CN=Entrust Certification Authority – L1C, OU=”(c) 2009 Entrust, Inc.”, OU=www.entrust.net/rpa is incorporated by reference, O=”Entrust, Inc.”, C=US
Successfully Connected to APNS
Test notification will not be sent