IBM released two Technotes for Sametime and POODLE Security Bulletin: Vulnerability in SSLv3 affects Sametime (CVE-2014-3566) and Security Bulletin: Vulnerability in SSLv3 affects IBM WebSphere Application Server (CVE-2014-3566)

What wasn’t clear (at first) was what actually needed to be done to disable SSLv3 and ensure that Sametime functions properly. Off the back of another PMR relating to the VMCU I managed to get some of Tony Payne’s time to fire off some questions.

The patches available in Security Bulletin: Vulnerability in SSLv3 affects Sametime (CVE-2014-3566) are to resolve problems within Sametime and DO NOT DISABLE SSLV3. These problems are.

  •  In Media servers SSL v3 was still enabled for backend server-to-server connections.
  • After making the POODLE security change on SSC as described in this bulletin, the installers for Sametime products (Advanced, Meetings, Media, Proxy, and Community Servers) are not able to connect to the SSC server and policies are not getting synched from the SSC into the Community Server.

So, you need to apply the patches to your servers and then you need to move onto the steps detailed in Security Bulletin: Vulnerability in SSLv3 affects IBM WebSphere Application Server (CVE-2014-3566)

Before you move onto WAS you might want to know which servers to apply the patches to. The Technote is quite clear but what if you have Edge components? If you do, then the SIP Edge proxy does not need to have the Media Manager code ran against is and nor does the TURN server. If you have an HTTP Edge proxy which sits in front of you Meeting server then this will need the patch applied as it communicates with the SSC, unlike the TURN and SIP Edge proxy. You do need to patch any SIP/HTTP proxies in front of the CM, SIP PR or Meeting servers which may be on their own node and hence their own profile.

After you have installed the patches you then need to disable SSLv3. To do this you can install ifixes or simply turn it off from within the SSC. You should also disable this from within the ISC of your SIP Edge proxy and Video Manager server.

The ifixes remove the ability to set or use SSLv3 so the net effect is that it makes the change within the SSC/ISC.

For my deployment I simply changed the settings within the SSC/ISC. To disable SSLv3 you need to do the following.

  • Log in to the SSC/ISC.
  • Go to Security – SSL certificate and key management – SSL configurations – CellDefaultSSLSettings – Quality of protection (QoP) settings. For VMGR and SIP Edge proxy you can update the NodeDefaultSSLSettings.
  • Change the Protocol from SSL_TLS to TLS.
  • Save and sync the changes to your nodes.
  • Stop all application servers.
  • Stop all node agents.
  • Update the ssl.client.props in each profile replacing “om.ibm.ssl.protocol=SSL_TLS” with “om.ibm.ssl.protocol=TLS”
  • Don’t forget the VMGR and Edge servers.
  • Restart the deployment manager.
  • In each profile run ./syncNode.sh ssc.collaborationben.com 8703 -username adminuser-password ******** to synchronise the node with the deployment manager.
  • Start each node agent and then each application server.
  • Test.

poodle

Testing

To test, find yourself a *nix machine and run the following command “openssl s_client -connect meeting.collaborationben.com:443 -ssl3″ and you should get something like the following response.

CONNECTED(00000003)
139961097578312:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 0 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1424780572
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)

If SSLv3 was still enabled you would see something very different. You will see the SSL certificate returned and something like the following.

New, TLSv1/SSLv3, Cipher is DES-CBC3-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DES-CBC3-SHA
    Session-ID:

Problems

I had one problem applying the patch to the VMCU. The instructions say to run ./upgrade.sh but doing so I got the following error.

[root@vmcu SametimeVideoMCU]# ./upgrade.sh
Sametime Video MCU status:SoftMcu service is down
./upgrade.sh: line 15: [: too many arguments
./upgrade.sh: line 17: [: too many arguments
./upgrade.sh: line 20: [: too many arguments
./upgrade.sh: line 23: [: too many arguments
Reading property file /opt/IBM/Sametime/STVideoMCU/console.properties..
Checking Java version:
java version “1.6.0_24″
OpenJDK Runtime Environment (IcedTea6 1.11.14) (rhel-1.65.1.11.14.el6_4-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
Java major version is: 6
Checking for license..
Exited with: 9
License status: 9
License accepted. Proceeding with upgrade:  9
./mcms/Scripts/InstallValidator.sh
313561 blocks
All System requirements met for upgrade. Proceeding with Sametime Video MCU upgrade.
Backing up Sametime Video MCU
There is another operation currently in progress
Unable to backup Sametime Video MCU configuration. Upgrading without a backup may result in loss of data. Aborting upgrade.

I ran “chkconfig soft_mcu off” so the VMCU didn’t start after a reboot and stopped it. On OS restart the same happened. I reproduced this on a customer server and my own.

IBM came back with a few steps, although incomplete, they pointed me in the following direction. I did the following:

  • rpm -qa ‘ibm-sametime*’
  • rpm -e $(rpm -qa ‘ibm-sametime*’)
  • cd /opt/IBM/Sametime/STVideoMCU/
  • Ensure console.properties is correct.
  • ./uninstallVideoMcu.sh
  • rpm -qa ‘ibm-sametime*’ to check the RPMs have been removed.
  • Change directory to the patch.
  • Ensure console.properties is correct.
  • ./install.sh
  • yum update openssl to ensure openssl is up to date.
  • Restart OS due to openssl update.

This effectively uninstalls the VMCU and unregisters it and then installs it again (albeit the new version) and uses the original deployment plan so do not create a new one.

IBM are hosting an open mic on the 11th March 2015 on this subject. I guess they have been getting a few queries from people. I hope this blog means you can get on with this instead of waiting for the 11th.

I had a wobble when applying the latest POODLE/SSLV3 (AAZI-9RGLXV) patch to my CentOS 6.5 server. It kept failing and the following exceptions were seen in /opt/ibm/dominodata/SametimeInstall.log

(Feb 19, 2015 10:50:30 PM), Install, com.lotus.sametime.install.product.SSCReg, err, com.ibm.sametime.console.deployment.client.exception.SCClientException: AIDSC0884E: Failed to do IO operation with file.Server returned HTTP response code: 500 for URL: http://ssc.collaborationben.com:9080/console/deployment/getDepIdByDepName
STACK_TRACE: 18
com.ibm.sametime.console.deployment.client.exception.SCClientException: AIDSC0884E: Failed to do IO operation with file.Server returned HTTP response code: 500 for URL: http://ssc.collaborationben.com:9080/console/deployment/getDepIdByDepName
    at com.ibm.sametime.console.deployment.client.util.RestURL.sendData(RestURL.java:358)
    at com.ibm.sametime.console.deployment.client.util.ClientUtility.getDepIdByDepName(ClientUtility.java:1380)
    at com.ibm.sametime.console.deployment.client.api.Deployment.getDepIdByDepName(Deployment.java:1674)

(Feb 19, 2015 10:50:30 PM), Install, com.lotus.sametime.install.product.SSCReg, err, An error occurred and product update failed.  Look at the log file /opt/ibm/dominodata/SametimeInstall.log for details.
(Feb 19, 2015 10:50:30 PM), Install, com.lotus.sametime.install.product.SSCReg, err, ProductException: (error code = 601; message=”err”; additional data = [com.ibm.sametime.console.deployment.client.exception.SCClientException: AIDSC0884E: Failed to do IO operation with file.Server returned HTTP response code: 500 for URL: http://ssc.collaborationben.com:9080/console/deployment/getDepIdByDepName%5D)
STACK_TRACE: 14
ProductException: (error code = 601; message=”err”; additional data = [com.ibm.sametime.console.deployment.client.exception.SCClientException: AIDSC0884E: Failed to do IO operation with file.Server returned HTTP response code: 500 for URL: http://ssc.collaborationben.com:9080/console/deployment/getDepIdByDepName%5D)
    at com.lotus.sametime.install.product.SSCReg.setSSCStatus(SSCReg.java:328)

I also saw the following in the same file.

(Feb 19, 2015 10:50:03 PM), Install, com.lotus.sametime.install.wizard.GetSSCEncodedPassword, msg1, loading /opt/ibm/dominodata/console/console.properties
(Feb 19, 2015 10:50:03 PM), Install, com.lotus.sametime.install.wizard.GetSSCEncodedPassword, msg1, Encoded password found. Decoding…
(Feb 19, 2015 10:50:03 PM), Install, com.lotus.sametime.install.wizard.GetSSCEncodedPassword, msg1, username = wasadmin
(Feb 19, 2015 10:50:03 PM), Install, com.installshield.full.event.dialog.console.PanelSSCUpgradeLoginConsoleImpl, msg1, SC installation detected
(Feb 19, 2015 10:50:03 PM), Install, com.installshield.full.event.dialog.console.PanelSSCUpgradeLoginConsoleImpl, msg1, loading /opt/ibm/dominodata/console/console.properties
(Feb 19, 2015 10:50:03 PM), Install, com.installshield.full.event.dialog.console.PanelSSCUpgradeLoginConsoleImpl, msg1, Encoded password found. Getting previously decoded name and password

I changed the WAS administrator user to use an LDAP based account so I was wondering why wasadmin was still appearing even though /opt/ibm/dominodata/console/console.properties had the new user name and password.

In console.properties I saw that “SSCEncodedAuthorization” was listed. This is the encoded password for the admin account. I removed this from console.properties and ran setuplinux.bin -console again and it upgraded fine.

I have written a couple of posts on this because I find the application extremely helpful in diagnosing network related issues with connection to APNs (Apple Push Notification service) so that iOS devices can receive IMs when the application is “backgrounded.”

Here is the application which includes a text file providing you with the correct syntax to us which would go something like this, for Windows.

D:\support\apnstest>d:\IBM\WebSphere\AppServer\java\bin\java.exe -jar apnstest.jar -k D:\IBM\WebSphere\AppServer\profiles\xxxxSTPPNProfile1\config\cells\xxxx01SSCCell\nodes\xxxxSTPNode1\apns-prod.pkcs12
APNS Test ScriptVersion: 2.0.0
Testing using key: D:\IBM\WebSphere\AppServer\profiles\xxxxxSTPPNProfile1\config\cells\xxxxx01SSCCell\nodes\xxxxSTPNode1\apns-prod.pkcs12
Testing using server: gateway.push.apple.com
Testing using port: 2195
About to attempt to connect to APNS
Initialized SSL Context
SSL Socket Created
Starting SSL Handshake
SSL Handshake Complete
CN=gateway.push.apple.com, O=Apple Inc., L=Cupertino, ST=California, C=US
CN=Entrust Certification Authority – L1C, OU=”(c) 2009 Entrust, Inc.”, OU=www.entrust.net/rpa is incorporated by reference, O=”Entrust, Inc.”, C=US
Successfully Connected to APNS
Test notification will not be sent

D:\support\apnstest>d:\IBM\WebSphere\AppServer\java\bin\java.exe -jar apnstest.jar -k D:\IBM\WebSphere\AppServer\profiles\xxxxxSTPPNProfile1\config\cells\xxxx01SSCCell\nodes\xxxxSTPNode1\apns-prod.pkcs12 -s feedback.push.apple.com -p 2196
APNS Test ScriptVersion: 2.0.0
Testing using key: D:\IBM\WebSphere\AppServer\profiles\xxxxSTPPNProfile1\config\cells\xxxx01SSCCell\nodes\xxxxSTPNode1\apns-prod.pkcs12
Testing using server: feedback.push.apple.com
Testing using port: 2196
About to attempt to connect to APNS
Initialized SSL Context
SSL Socket Created
Starting SSL Handshake
SSL Handshake Complete
CN=feedback.push.apple.com, O=Apple Inc., L=Cupertino, ST=California, C=US
CN=Entrust Certification Authority – L1C, OU=”(c) 2009 Entrust, Inc.”, OU=www.entrust.net/rpa is incorporated by reference, O=”Entrust, Inc.”, C=US
Successfully Connected to APNS
Test notification will not be sent

A customer had a problem with a single user not being able to authenticate with Connections. The user had an active profile and they use Domino LDAP.

The SystemOut.log showed.

[1/5/15 13:27:31:824 GMT] 00000190 LTPAServerObj E   SECJ0369E: Authentication failed when using LTPA. The exception is com.ibm.websphere.wim.exception.PasswordCheckFailedException: CWWIM4529E  The password verification for the ‘juser’ principal name failed. Root cause: ‘javax.naming.AuthenticationException: [LDAP: error code 49 – Failed, invalid credentials for CN=Joe User,OU=xxx,OU=xx,o=xxx]; Resolved object: ‘com.sun.jndi.ldap.LdapCtx@8dd3a48”..
[1/5/15 13:27:31:825 GMT] 00000190 FormLoginExte E   SECJ0118E: Authentication error during authentication for user juser

Looking in the associated FFDC log I found.

Caused by: com.ibm.websphere.wim.exception.AuthenticationNotSupportedException:
CWWIM4530E  The authentication is not supported by the ‘xxx LDAP’ repository. Root cause: ‘javax.naming.AuthenticationNotSupportedException: [LDAP: errorcode 48 – Failed, server access denied]; Resolved object: ‘com.sun.jndi.ldap.LdapCtx@af24747e”

errorcode 48 – Failed, server access denied” interested me but I couldn’t find reference to it anywhere.

The user’s person document looked OK. I asked the customer if the user was in any deny access groups which turned out to be the cause of the problem. The user had been added to a deny access group for the LDAP server. Removing her allowed her to authenticate.

I found that for a customer the meetings icons in the STProxy web client wasn’t bringing up the user’s meeting rooms. After a bit of debugging server trace showed that an LtpaToken was being generated but the browser wasn’t getting an LtpaToken returned to it. It drove me made because the STProxy doesn’t need to have SSO enabled for it to work like the Meeting server does, regardless of that, SSO worked in all directions between the Community server and the Meeting server and the STProxy is in the same cell as the Meeting server so SSO should work!

I raised a PMR and IBM asked me to add the following to the stproxyconfig.xml. After a sync and a restart of STProxy all is well.

<tokenDomain>DOMAIN.CO.UK</tokenDomain>

(replace with your domain)

I’m not sure whether this is missing from the patch they are running which is CKEY-9L9JM5 which is not the latest patch released a couple of weeks ago BPAS-9QSNS7.

The comment from IBM is “for long term the code should be fixed, dev created rtc ticket for it as well as APAR created: LO83144″

stproxy

 

I had come back on site to a customer to deploy some other Sametime 9 services and found that the managed-settings.xml wasn’t being applied to the clients.

I remembered reading Gabriella Davis’ blog http://turtleblog.info/2014/04/02/problems-deploying-sametime-policies-the-missing-link so I found it and at first I enabled POLICY_DEBUG_LEVEL=5 and then updated console.properties adding  SSCUserName, SSCPassword and removing SSCEncodedAuthorization.

I added the following trace to STConsoleServer, *=info:com.ibm.sametime.console.admin.plugins.policy.*=all which allowed me to see the policies read from the STSC.POLICY.TEMPLATE table to make sure they were populated.

The Community server was restarted that evening but still policies were not being applied to the clients. Thanks to the debug I then had some useful output in the Trace directory to look through and found this:

[ 07:58:57.560 | 04.12.2014 | FINEST | 1 ] : DbXmlBlackBox : readSortedData : Policy DTO=> Policy ID: Managed.settings.28/10/14;Policy Weight: 3;Policy Label: null;Policy Type: USER
[ 07:58:57.576 | 04.12.2014 | SEVERE | 1 ] : DbXmlBlackBox : readSortedDataForProducts : PolicyException:
com.ibm.sametime.policy.types.PolicyException: com.ibm.sametime.console.admin.services.exception.ServiceException: com.ibm.sametime.console.common.http.api.exception.HttpBadRequestException: Not Found
Caused by: com.ibm.sametime.console.admin.services.exception.ServiceException: com.ibm.sametime.console.common.http.api.exception.HttpBadRequestException: Not Found

I previously created an IM policy called “Managed settings 28/10/14″ which wasn’t liked. As there is xml involved I removed the slashes to leave “Managed settings 281014.”

I restarted STConsoleServer and then restarted the STPolicy Windows service on the Community server and this triggers activity on the STConsoleServer. Now that the slashes were removed the files policies.server.xml was updated adding the managed settings policy to policies.user.xml which is in the program directory of the Community server. Subsequently, the policy was then applied to the clients.

http://ssc:9080/stpolicy/policy/all

This provide high level information on all policies, IM, audio/video etc.

http://ssc:9080/stpolicy/policy/av.anonymous.policy & http://ssc:9080/stpolicy/policy/av.default.policy

These URLs provide details of the default audio/video policies.

I had a few problems with a customer’s deployment of Sametime 9 which probably come down to deployment plans and the order of the servers being installed.

During installation I had problems detailed in “System version is null” on new IBM Sametime Video Manager installation which forced me to uninstall the VMGR and install again with a new deployment plan. The outcome of this was that I could not administer the default policies nor create new Media Manager policies in the SSC, I saw the following error, “AIDSC#####: Could not connect to Sametime Video Manager. Either VMgr is not installed or server is not up. Please retry after installing VMgr or starting it.”

1

I saw in the deployment manager SystemOut.log “[17/10/14 17:02:04:101 BST] 00000220 SametimeVmgrU E   Forbidden” but nothing much else to write home about.

I raised a PMR with IBM and gathered some trace and sent it off. The PMR ended up with Ankit Vij in L3 who worked as a developer on the propagation of policies from the SSC to the VMGR.

After some to’ing and fro’ing it was identified that there were missing credentials in the DEPLOYMENT table of the SSC database. In the DEPCONF  column of the Conference Manager deployment plan lies XML data. In the data are two fields VMGRUSER and VMGRPASSWORD. In the customer’s data these values were empty, this is why the SSC couldn’t access the VMGR’s policies.

There are few ways in which to edit the data, Data Studio is nice and easy and can export the table, edit it and then import it again in no time at all but as I was accessing their environment using Citrix this wasn’t an option because I couldn’t install any software. Using the CLI was the only way to do it.

My first attempts of using the DB2 EXPORT command failed because the tables have LOBs which are truncated when you export the data to a csv file. The way around it is to export to a csv file but also export all the data to LOB files. This can be achieved using the following command.

C:\Windows\system32>db2 “export to d:\export\deployment.csv of del lobs to d:\export\lobs\ modified by lobsinsepfiles select * from ssc.deployment”
SQL3104N  The Export utility is beginning to export data to file
“d:\export\deployment.csv”.

This produces a csv. Where there was LOB data a .lob file is produced and the csv details which number .lob file holds the information for that particular entry.

Once I had found the .lob file referenced for the Conference Manager deployment plan in the DEPCONF  column I had to copy the contents of the .lob to a new text file.

The VMGRUSER and VMGRPASSWORD values were empty so I then updated them with wasadmin (could be admin/admin) and the password associated with it.

Next I had to add to the beginning of the xml data UPDATE SSC.DEPLOYMENT SET DEPCONF=’ and to the end ‘ WHERE DEPID=’14908a6aa1d-00000000000a-MediaDep’

The DEPID is easy to come about and is listed DEPID column for the Conference Manager deployment plan.

The end result is a single line containing 18000+ characters looking something like this.

UPDATE SSC.DEPLOYMENT SET DEPCONF='<?xml version=”1.0″ encoding=”UTF-8″?>………………………….</parameter></parameters></Config>’ WHERE DEPID=’14908a6aa1d-00000000000a-MediaDep’

As the command was too large to paste into the CLI I saved it to a .sql file.

I stopped STConsoleServer, the node agent and the deployment manager.

Before changing the database I needed to back it up.

C:\Windows\system32>db2 backup database stsc
SQL1035N  The database is currently in use.  SQLSTATE=57019

I then needed to force the application connections from the database.

C:\Windows\system32>db2 list applications

Auth Id  Application    Appl.      Application Id                                                 DB       # of
Name           Handle                                                                    Name    Agents
——– ————– ———- ————————————————————– ——– —–
DB2ADMIN db2jcc_applica 41961      192.168.x.x.49442.141124093130                              STSC     1
DB2ADMIN db2jcc_applica 45374      192.168.x.x.61230.141125142939                              STMS     1
DB2ADMIN db2jcc_applica 45483      192.168.x.x.61666.141125152718                              STMS     1
DB2ADMIN db2jcc_applica 41949      192.168.x.x.49385.141124093116                              STMS     1

C:\Windows\system32>db2 force application(41961)
DB20000I  The FORCE APPLICATION command completed successfully.
DB21024I  This command is asynchronous and may not be effective immediately.

After all applications are disconnected I could run the backup.

 C:\Windows\system32>db2 backup database stsc

Backup successful. The timestamp for this backup image is : 20141125154621

C:\Windows\system32>db2 connect to stsc

   Database Connection Information

 Database server        = DB2/NT64 10.1.0
 SQL authorization ID   = DB2ADMIN
 Local database alias   = STSC

At this point I am going to run the UPDATE command using the .sql file I created.

C:\Windows\system32>db2 -vf C:\DB2\ssc.sql

DB20000I  The SQL command completed successfully.

Normally I would run db2 -tvf but that didn’t work, I think because I didn’t use semicolons for delimiters in the .sql file. Anyway, it worked.

I started the deployment manager, node agent and STConsoleServer and I could now edit the Media Manager policies.

Many thanks to Imran and Ankit at IBM for helping me through this frustrating but interesting problem.

Follow

Get every new post delivered to your Inbox.

Join 88 other followers