Sametime Video MCU problems

When applying the latest version of the Video MCU available on Fix Central 9001-ST-Media-FP-SPIR-9ZTF3Z I faced problems when configuring the VMCU to use a TURN server.

1

I uninstalled and tried again but the result was the same. I found a draft Technote which didn’t help.

Problem
“service soft_mcu status” returns “SoftMcu service is unavailable” although soft_mcu is up.
Symptom
“service soft_mcu status” returns “SoftMcu service is unavailable” although soft_mcu is up.
You will encounter errors in accessing setting of MCU in System Console Server (SSC).
“An error occurred while retrieving the IP service.” in accessing “Video MCU global settings”.
“An error occurred while retrieving Video MCU users.” in accessing “Manage Users”.
“An error occurred while retrieving alarm health status” in accessing “Active Alarms”.
“An error occurred while logger configuration data” in accessing “Configure Logging Settings”

Cause
Due to deletion of necessary files under /tmp by tmpwatch cron.
Environment
Linux OS.
Diagnosing the problem
Video MCU places some files under /tmp directory including httpd.listen.conf that is necessary for status check of Video MCU (“service soft_mcu status”).

Resolving the problem
If you can not find it, tmpwatch cron has deleted files under /tmp unaccessed for more than 10 days as default.  Chek your cron setting and delete or modify the settings.
Stopping and starting re-create necessary files under /tmp.

I also found After updating Sametime System Console not able to see settings for VMCU which I followed even though I had not installed the SSC hot fix SPIR-9VM7XJ. This didn’t work either. I raised a PMR. Waiting for it to make it’s way up to L3 is slow going so I decided to uninstall and use an older version of the VMCU, 9001-ST-Media-FP-SPIR-9RHDAJ, which I could configure and didn’t get “an error occurred while retrieving the IP service.”

There was a bit of too’ing and fro’ing and the long and short of it is that I got the SSC hotfix, which I applied, and then upgraded the VMCU but I still got “an error occurred while retrieving the IP service” when attempting to update the settings.

L3 responded and told me that there is a known problem configuring the TURN server from the SSC and below is how to manually configure these settings.

  1. Open /mcms/Cfg/IPServiceListTmp.xml  (Make the backup of IPServiceListTmp.xml in case something goes wrong, so you can recover)
  2. In SIP_ADVANCED section, set three parameters as below:
    1. Set ICE_ENVIRONMENT to iceEnvironment_standard. By default, value is iceEnvironment_none.
    2. Set STUN_SERVER_IP to TURN server IP.  By default, value is 0.0.0.0
    3. Set TURN_SERVER_IP to TURN server IP.  By default, value is 0.0.0.0
After these changes, SIP_ADVANCED section will look like this.
<SIP_ADVANCED>
<SIP_ADVANCED_USER_NAME></SIP_ADVANCED_USER_NAME>
<ICE_ENVIRONMENT>iceEnvironment_standard</ICE_ENVIRONMENT>
<ICE_STANDARD_PARAMS>
<IS_PASSWORD_SERVER>false</IS_PASSWORD_SERVER>
<PASSWORD_SERVER_IP>0.0.0.0</PASSWORD_SERVER_IP>
<PASSWORD_SERVER_PORT>0</PASSWORD_SERVER_PORT>
<PASSWORD_SERVER_USER_NAME></PASSWORD_SERVER_USER_NAME>
<PASSWORD_SERVER_PASSWORD></PASSWORD_SERVER_PASSWORD>
<STUN_SERVER_IP>9.42.139.62</STUN_SERVER_IP>
<STUN_SERVER_PORT>3478</STUN_SERVER_PORT>
<TURN_SERVER_IP>9.42.139.62</TURN_SERVER_IP>
<TURN_SERVER_PORT>3478</TURN_SERVER_PORT>
</ICE_STANDARD_PARAMS>
</SIP_ADVANCED>
3. Restart the MCU service with “service soft_mcu restart” command. This will copy the TURN configuration from IPServiceListTmp.xml to IPServiceList.xml. MCU is now configured with TURN server. If you have any issues, send us  IPServiceList.xml and  IPServiceListTmp.xml files.
After doing this the the settings were populated correctly BUT you cannot update the values from the SSC, you need to change them in IPServiceListTmp.xml.
IBM tell me that an SSC fix will be available in 9.0.1 release soon.
Advertisements

Sametime and POODLE SSLv3 patches

IBM released two Technotes for Sametime and POODLE Security Bulletin: Vulnerability in SSLv3 affects Sametime (CVE-2014-3566) and Security Bulletin: Vulnerability in SSLv3 affects IBM WebSphere Application Server (CVE-2014-3566)

What wasn’t clear (at first) was what actually needed to be done to disable SSLv3 and ensure that Sametime functions properly. Off the back of another PMR relating to the VMCU I managed to get some of Tony Payne’s time to fire off some questions.

The patches available in Security Bulletin: Vulnerability in SSLv3 affects Sametime (CVE-2014-3566) are to resolve problems within Sametime and DO NOT DISABLE SSLV3. These problems are.

  •  In Media servers SSL v3 was still enabled for backend server-to-server connections.
  • After making the POODLE security change on SSC as described in this bulletin, the installers for Sametime products (Advanced, Meetings, Media, Proxy, and Community Servers) are not able to connect to the SSC server and policies are not getting synched from the SSC into the Community Server.

So, you need to apply the patches to your servers and then you need to move onto the steps detailed in Security Bulletin: Vulnerability in SSLv3 affects IBM WebSphere Application Server (CVE-2014-3566)

Before you move onto WAS you might want to know which servers to apply the patches to. The Technote is quite clear but what if you have Edge components? If you do, then the SIP Edge proxy does not need to have the Media Manager code ran against is and nor does the TURN server. If you have an HTTP Edge proxy which sits in front of you Meeting server then this will need the patch applied as it communicates with the SSC, unlike the TURN and SIP Edge proxy. You do need to patch any SIP/HTTP proxies in front of the CM, SIP PR or Meeting servers which may be on their own node and hence their own profile.

After you have installed the patches you then need to disable SSLv3. To do this you can install ifixes or simply turn it off from within the SSC. You should also disable this from within the ISC of your SIP Edge proxy and Video Manager server.

The ifixes remove the ability to set or use SSLv3 so the net effect is that it makes the change within the SSC/ISC.

For my deployment I simply changed the settings within the SSC/ISC. To disable SSLv3 you need to do the following.

  • Log in to the SSC/ISC.
  • Go to Security – SSL certificate and key management – SSL configurations – CellDefaultSSLSettings – Quality of protection (QoP) settings. For VMGR and SIP Edge proxy you can update the NodeDefaultSSLSettings.
  • Change the Protocol from SSL_TLS to TLS.
  • Save and sync the changes to your nodes.
  • Stop all application servers.
  • Stop all node agents.
  • Update the ssl.client.props in each profile replacing “com.ibm.ssl.protocol=SSL_TLS” with “com.ibm.ssl.protocol=TLS”
  • Don’t forget the VMGR and Edge servers.
  • Restart the deployment manager.
  • In each profile run ./syncNode.sh ssc.collaborationben.com 8703 -username adminuser-password ******** to synchronise the node with the deployment manager.
  • Start each node agent and then each application server.
  • Test.

poodle

Testing

To test, find yourself a *nix machine and run the following command “openssl s_client -connect meeting.collaborationben.com:443 -ssl3” and you should get something like the following response.

CONNECTED(00000003)
139961097578312:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 0 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1424780572
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)

If SSLv3 was still enabled you would see something very different. You will see the SSL certificate returned and something like the following.

New, TLSv1/SSLv3, Cipher is DES-CBC3-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DES-CBC3-SHA
    Session-ID:

Problems

I had one problem applying the patch to the VMCU. The instructions say to run ./upgrade.sh but doing so I got the following error.

[root@vmcu SametimeVideoMCU]# ./upgrade.sh
Sametime Video MCU status:SoftMcu service is down
./upgrade.sh: line 15: [: too many arguments
./upgrade.sh: line 17: [: too many arguments
./upgrade.sh: line 20: [: too many arguments
./upgrade.sh: line 23: [: too many arguments
Reading property file /opt/IBM/Sametime/STVideoMCU/console.properties..
Checking Java version:
java version “1.6.0_24”
OpenJDK Runtime Environment (IcedTea6 1.11.14) (rhel-1.65.1.11.14.el6_4-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
Java major version is: 6
Checking for license..
Exited with: 9
License status: 9
License accepted. Proceeding with upgrade:  9
./mcms/Scripts/InstallValidator.sh
313561 blocks
All System requirements met for upgrade. Proceeding with Sametime Video MCU upgrade.
Backing up Sametime Video MCU
There is another operation currently in progress
Unable to backup Sametime Video MCU configuration. Upgrading without a backup may result in loss of data. Aborting upgrade.

I ran “chkconfig soft_mcu off” so the VMCU didn’t start after a reboot and stopped it. On OS restart the same happened. I reproduced this on a customer server and my own.

IBM came back with a few steps, although incomplete, they pointed me in the following direction. I did the following:

  • rpm -qa ‘ibm-sametime*’
  • rpm -e $(rpm -qa ‘ibm-sametime*’)
  • cd /opt/IBM/Sametime/STVideoMCU/
  • Ensure console.properties is correct.
  • ./uninstallVideoMcu.sh
  • rpm -qa ‘ibm-sametime*’ to check the RPMs have been removed.
  • Change directory to the patch.
  • Ensure console.properties is correct.
  • ./install.sh
  • yum update openssl to ensure openssl is up to date.
  • Restart OS due to openssl update.

This effectively uninstalls the VMCU and unregisters it and then installs it again (albeit the new version) and uses the original deployment plan so do not create a new one.

IBM are hosting an open mic on the 11th March 2015 on this subject. I guess they have been getting a few queries from people. I hope this blog means you can get on with this instead of waiting for the 11th.